Open Lineage Meets Graph: Observability for the Agentic Lakehouse

Summary

As agents query, transform, and feed models, data lineage no longer looks like a chain but like a graph—with producers, consumers, and agents all as nodes—making traditional chain-based lineage tools insufficient for modern agentic architectures.
The open pipeline described in this video uses a Spark plugin that emits OpenLineage events over ConnectRPC with Protobuf schema versioning into Delta Lake and Apache Iceberg tables, capturing plan-level lineage automatically without manual instrumentation.
PuppyGraph enables the lineage data stored in Delta/Iceberg to be queried as a graph using Cypher, unlocking impact analysis questions—such as which pipelines break if a table changes or what training data an AI model used—that relational databases cannot efficiently answer.

Open Lineage Meets Graph: Observability for the Agentic Lakehouse

As agents query, transform and feed models, the lineage they leave looks less like a chain and more like a graph. Scott Haines (Databricks) and Jaz Ku (PuppyGraph) show an open pipeline — a Spark plugin emitting OpenLineage over ConnectRPC into Delta/Iceberg, then queried as a graph with Cypher.
Talk By: Jaz Ku, Solution Architect, PuppyGraph ; Scott Haines, Staff Developer Advocate & OSS Engineer, Databricks ;

Chapters

FAQs

Why is data lineage now a graph rather than a chain?

Traditional lineage tracked a simple upstream-to-downstream chain, but modern architectures involve central nodes where many producers and consumers pull data simultaneously, plus agents that act as both consumers and producers. This n-dimensional graph structure requires different tooling to traverse and understand relationships, especially as agents become first-class citizens in data pipelines.

What is OpenLineage and why is it used in this architecture?

OpenLineage is an open standard for emitting lineage events that captures metadata about data sources, transformations, and outputs in a schema-versioned format using Protobuf over ConnectRPC. In this architecture, a Spark plugin automatically emits OpenLineage events at the query plan level, so lineage is captured without requiring developers to manually instrument their code.

How does PuppyGraph enable graph queries over lineage data?

PuppyGraph allows lineage data stored in Delta Lake and Apache Iceberg to be queried directly as a graph using Cypher query language, without needing to copy data into a separate graph database. This makes it possible to run traversal queries—such as tracing an AI model's training data provenance or finding all pipelines that depend on a specific table—that would be difficult with SQL alone.

What kind of impact analysis does graph-based lineage enable?

Graph lineage enables questions like 'which downstream pipelines will break if this table changes?' or 'what was the exact training data provenance of this AI model?'—answers that require traversing complex dependency relationships across many nodes. This video demonstrates these queries using Cypher against lineage data captured by the OpenLineage Spark plugin and stored in Delta Lake.

Full transcript

[00:08] Thank you everybody for coming out um in the afternoon on essentially day one of summit. Um my name is Scott Haynes. I'm a staff developer advocate uh working for data bricks. Uh and then we got Jazz as well if you want to introduce yourself. >> So I'm Jaz Coup a solution architect at Puppy Graph. Um and we'll be talking
[00:20] about graphs obviously later on. So very excited. >> Awesome. Um so I'm going to kind of kick things off. Um, so is it coming through? >> Okay, awesome. Um, so,
[00:35] okay, perfect. This is good. All right. So, who here um deals with governance? Who here deals with like lineage in general? Probably most people in like the audience as well. Um, who here works with like open source uh or Okay,
[00:48] awesome. This is wonderful as well. So what we're going to talk about today is essentially like what you can do like with open source and then what you can do to essentially kind of tack on like a graph on top of that as well. So if we think about like lineage, lineage of a
[01:00] lot of times has been essentially you know a line over time. Um now with like most modern systems things are becoming more graphlike. Um and especially like with agents uh things now essentially
[01:12] don't necessarily meet the old mold of essentially like A to B to C to D. Everything's more graph-like in terms of that structure as well. And so if we think about it, we went from essentially tables where we had tables, we had
[01:25] relationships, we had an upstream and a downstream dependency. A lot of people have probably dealt with things where you don't necessarily have just an upstream and a downstream. You're kind of central nodes where a lot of people are pulling data from those individual nodes as well. So you have to understand
[01:38] exactly like what the relationships are between what data you're providing, what data people need and then also like the governance and policies on top of all that as well. So in theory, right, we are going from a line uh to more of an
[01:51] n-dimensional graph as well. And so if we think about that as well, right, the relationships are really it's like a first first class citizen now between, you know, us as producers of
[02:03] data, other people as consumers of data and also now like agents as well in the mix. Um, and so we want people to be able to actually, you know, quickly traverse the lineage to understand how data flows through a complex network.
[02:15] And we also want to be able to ensure that you know any kind of change that might affect you know thousands hundreds of thousands many many like very important like downstreams and downstreams being plural nowadays as
[02:28] well but that's not going to necessarily be something that uh we break by accident and so lineage now is a lot more important like it was important before but I feel like over the past like few years it's become something that people just can't really live
[02:40] without and so with that as kind of a setup. We're now going to start talking about how we emit lineage and then that's going to be kind of part
[02:52] one of this talk as well. And then in part two, we're going to talk about how we actually query that lineage. Uh and Jazz is going to go into cipher queries as well. And what does it look like to be able to query things that are end
[03:09] of lineage as a source? Who here is a data bricks customer? Okay, awesome. So you've probably already used Unity catalog. You have autolineage. It's something that just kind of comes for free. For people who aren't a data bricks customer, lineage is something that is probably tacked on.
[03:21] Um and it's either done essentially upfront. It might change over time. Um but you really want lineage to be something that's actually a part of your pipelines that is essentially always updating and keeping like the state of
[03:34] your graph intact. So who here has listen uh heard of open lineage? Awesome. A few hands. This is great. So open lineage is a specification that
[03:47] talks about how we emit things that occur. How do we emit events? Say I run a job. I now have a like essentially I've got an event that keeps track of that job that produces data. Um if I
[04:00] have specific other events that I need to emit as well, there's actually like an open standard. It's all open API. Uh so an open API specification tells me how I can actually emit my lineage so that that can be received by a server
[04:12] that can then essentially create the ground for us to be able to parse that as an actual graph and a relationship uh within a large network as well. So open lineage is a specific
[04:25] um it's also a JSON specification. Uh the project we're talking about today that's also open source um uh all it does is add a gRPC layer on top of that as well. So you can essentially emit the gRPC data from any of your sources. Um
[04:39] which is kind of cool. And so if you don't know about gRPC, that's okay. Um you can take a look at the actual uh project as well at the very end of our talk. Um and then you can use it. It's free to use. Um and it also comes with like uh agent skills. So you don't have
[04:53] to worry about really how it works. You can just start to use it which is really cool. So open lineage we get run uh runle level events column level lineage and essentially you can emit that directly from like an engine like spark
[05:05] you can also emit that from like within like your data fusion scan plan API plan uh behind the scenes andor you can essentially just track this as an API um so if you are creating a data platform or if you work in a data platform group
[05:17] within your company then you can just hit the API if you want to do something that doesn't require an engine to be in place um so there's many different kind of ways of operating within this whole entire graphical interface. Um, and essentially it's all
[05:30] just open API specifications which is really cool. Um, and so given like that we don't have a ton of time to talk about everything today. Um, I'm going to kind of just quickly go over like the essentials. So if I have different
[05:43] events that I need to admit and I don't want those events to break over time in the same way that I don't want my lineage to fall apart over time, I need something that I know I can actually, you know, depend upon that has like backwards and forwards compatibility.
[05:56] And that's essentially where like protobuff and like gRPC comes into play. Protobuff essentially defines how I change my schemas over time which is very important especially if you're keeping track of an end dimensional relationship between lineage within like
[06:08] your actual like uh data networks. Um and so with a lot of this we create messages which is really no different than the way that we think about JSON. Uh only we do it in a format that is essentially type safe. We can compile it
[06:21] down and it's reliable. Um, so what we have on the screen right now shows what a service a service is. And a service can be compiled down to your language of choice. That could be Rust, it could be Go, doesn't really matter. There's
[06:33] compilers for all of that. All we're doing is emitting events. And we don't have to actually think about, well, we only have a JavaScript API from the front end. We only have a Scala API from the back end. Now you can essentially just event you can emit events from
[06:47] anywhere where you can produce a message and then that ends up becoming something that is then uh ingestable into your actual graph as well. So from a spark job uh who here is familiar with spark
[07:00] that's a lot of good hands. Um it is a spark spark is definitely something that we talk about at data summit. Um so with that in mind uh Spark is plugins and so um the official open source like uh open
[07:13] lineage plugin for Spark just as plug a plugin you add it it will automatically read uh all of like your uh any kind of plan. So if you have a data frame and you call any sort of action on it it's
[07:25] essentially creating an unresolved logical plan which then gets resolved and then you have a finite action which might be like you know I'm going to go write this data to iceberg uh to Delta Lake. Uh with all of that behind the
[07:37] scenes, um all the open lineage plugin does is it reads through the whole entire plan. It understands upstream sources. It understands downstream syncs and it creates essentially your relationship for a job that's being run.
[07:52] We don't have to go into much more detail on that right now as well. Um but that's the whole reason for Open AI or not open AI, sorry, open API. It's okay. It's also an AI event, so it's all good. Um, and so anyways,
[08:04] there's a Spark plugin you put into Spark, you run your jobs. You don't really think about it. As long as there's a service running that can accept your emitted uh gRPC payloads, which is Protobuff, you're good to go.
[08:16] Um, the way that we set this up was essentially like a Goingest service because I like Go. I also like Connect RPC because I used to work at a company called Buff and we created Connect RPC and gave it to the CNCF. Um, I also like Rust as well. Um, Rust had a lot more
[08:29] capabilities for iceberg. had a lot more capabilities for Delta Lake than just go itself because Go doesn't necessarily have its own like data frame APIs and so rest sidecar is all part of this architecture. Um but with that said, we
[08:42] can always talk more afterwards. Um and now I'm going to kick it over uh to Jazz. >> Yep. Perfect. So I hope you can hear me. Um so Scott very nicely went over basically how we're trying to collect and ingest that data. And so now that we
[08:55] have all of it very nicely in our tables, we are going to query it as a graph. Um, this is against conventional wisdom to query your tables in SQL. And I'm going to make you raise your hand one more time. How many of you have
[09:07] queried in cipher or Gremlin or know what those languages are? There are some hands. I expected none. I'm very surprised. This is amazing. Let's go. Um, and so before we kind of talk into how exactly we're going to
[09:20] query it as a graph, let's kind of go into why we would want to query it as a graph. And so the thing is that once we have all our lineage emitted and piled up into the tables, um you really start to care about the relationship between
[09:33] the entities in your tables, right? For example, which AI models were trained on data derived from this now revoked source. Um the issue is that you have data sets that are derived from other
[09:45] data sets. Your AI models touch tons of different data and so it's not always very clear-cut where that information is coming from. It's not a single one to two hop query. it can span out to six or seven transformations. Um, so this is
[09:58] very graph shaped. If this table fails compliance, what else is affected, right? What is the downstream blast radius? And hey, maybe uh agents are reading PII that it shouldn't be given access to. Um, what jobs happened along
[10:12] the way and why does it look like that, right? Um, you care about who owns the assets across teams and cataloges and kind of what they're using it for, right? Um I went too far. Um so one
[10:25] other interesting thing is that if graphs are so helpful why aren't people using it more? Um the big thing is that people have tried um but usually traditionally it requires standing up a separate graph database. Um which means
[10:38] that you have to send stand up a whole another database before you can test out the value of graph queries. And that's kind of where puppy graph comes in today as a zero ETL graph query engine. So you actually point at your existing lineage
[10:51] tables um that have come from Scott's half. Um then you tell us what you want which tables you want modeled as nodes and edges and then you can start quering it like a graph as is which is very convenient. You don't have to stand up a
[11:03] separate graph database. You don't have to copy any of your data over. We're connecting directly to the underlying tables. And so this means that you instead of writing um your scrolls and scrolls of SQL, your complex table
[11:15] joins, you can really just write a graph traversal in gremlin or cipher. And one really interesting thing is that we're not running, we're not kind of converting graphs into SQL. Um we are actually handling those traversals on
[11:28] puppy graphs end. So you still avoid the penalty of these table joints. Um what this means for you is that you still get the interactive speed required to answer your governance questions. Um and we also have a DI we are also are a
[11:41] distributed graph query engine um that leverages vectorized evaluation and an MPP architecture in order to ensure that you get your graph graph responses in seconds to subsecond. Um another really
[11:54] interesting side effect of this unified or of this zero ETL approach is that we can actually run federated graph queries very easily. For example, um this data set kind of collects data lineage, but if you want to access your access
[12:07] policies and identities, and they're all in different sources, you just need to point to those cataloges. Um and then you can have one graph traversal to query them all, I guess. Um and so I
[12:19] don't have enough time for a real demo over here, so I've taken screenshots from the puppy graph UI. Um but basically because you're not really certain of how many inputs and outputs a job will produce um open lineage
[12:32] actually kind of uh has them as little JSON. So we're going to flatten the logs first. And then you can see that we have three types which is job inputs and job outputs. And one really interesting I need to stop saying interesting but one
[12:45] really cool thing is that uh the same table can be both a node and an edge. Right? For example, for job inputs, it tells you, hey, what was what data sets were used in the job, but also what direction it's going in, right? It's
[12:58] getting consumed by a job. And then the job output is not only a data set, but it's also produced by that job. Um, so over on the left on the right, you get this really nice um kind of circular
[13:10] thing going on. Um, so that's kind of how the puppy graph schema looks like. Um, and then from there, you can immediately start running your graph queries. So one query any number of hops and you can also collect every intermediate job and it downstream
[13:22] inputs. So if you're not familiar with cipher um this is a very simple cipher query. So what I'm doing over here is saying hey in my path I want to make sure that I start with the data set and I end with the data set and I kind of
[13:35] want to trace the path along it um what was consuming these data sets and what was produced by it. Um and then return me that path right and so over here it's a really simple example. Um, I think it all kind of just filtered into this one
[13:48] mega job. Um, but if you can imagine like how complex it can get as you get more paths and more data and more jobs running. Um, and so yeah, graphs are
[14:00] cool, amazing. Um, and then in this pipeline, we've kind of shown you like this endto-end lineage architecture. Um, where you really just have to think once you emit and transport that lineage, it all goes into your tables. Um, and
[14:12] because of that, you're actually able to kind of run your SQL queries alongside of your graph queries if you wanted to because puppy graph can sit on top of those tables. Um, and you're able to query it straight out, right? So, it
[14:25] kind of makes it very querable. Um, and why is this the future of AI governance? Um, if you think about it, AI is always super unpredictable and it's bringing rapid challenges and you're kind of wrapping your head around, hey, am I
[14:38] giving my am I giving my AI the right information, the right permissions and everything. Um, and so when you are able to trace any model output back it to its source data and policies that govern it, it really provides you a lot more
[14:51] clarity into understanding how your AI is functioning, where it's getting its information from. Um, and so this also ties back to this cross-axis accountability where you're no longer caring about, okay, um, this is the
[15:05] output, right? You're actually trying to trace through and see what exactly they touched. Um, and these model play nicely as a graph. You have your data, your model, your policies and owners that are all entities. And then the edges explicitly relate explicitly show how
[15:19] they relate to one another. Um, and so this lets you answer questions like real-time blast radius. Um, for example, if you revoke a source, you kind of want to see um which AI agents are affected, what prompts you might need to change.
[15:31] Um, and then one of the big things for us is that it is kind of open, not locked in. Um, because you have open lineage definitions, so you know exactly how the data is going to look. Um, you can use connect RPC to kind of just
[15:43] communicate. And then once you have all your data in these tables in their open table formats, you're kind of able to use your compute engine of choice based off of the workloads you need. So for your regular aggregation queries you can
[15:55] continue to use SQL and then once you need to enter that mode for those deeper investigations for those path tracings then you can query in place with um puppy graph so you don't have to stand up a separate graph database you're not
[16:08] locked into a specialized graph um solution right um and so I guess the main takeaways here is emit it openly query it as a graph um once you are able to capture the relationships being able to
[16:22] cap uh being able to query it and get your answers in real time become very important. Um, and it really provides a lot more value to the data that you're kind of shoving all into your data sets, right? Yeah. >> All right, then.
[16:34] >> Awesome. >> Thank you. >> Cool. So, the la the last one was essentially just uh a last minute slide. It's not pretty. Um, if you want to test out everything um this whole entire
[16:46] environment gives you everything to do that create uh it provides all the printabuff provides everything you need uh pre-ompiled as well. Um, so you just have to run it and that's pretty much it. Um, so with that, I know we can't take questions because there's no
[16:59] microphone that I think we can share. Um, but thank you so much for coming out to this. Um, thank you for coming on day one as well and I hope the rest of summit is good. So thank you. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.