Graph-Powered Observability Data Analysis in Databricks With Credential Vending

Summary

• Graph-based analytics: PuppyGraph enables complex traversal of observability data on Databricks, helping organizations trace service dependencies and analyze upstream/downstream impacts in distributed systems.
• Secure access: Coinbase leverages credential vending to enable fine-grained, secure access to sensitive observability data stored on Databricks Data and AI platform without long-lived secrets.
• Operational intelligence: By querying Databricks as graph structures, organizations can uncover recurring error patterns, diagnose issues faster and optimize system performance without extracting data elsewhere.

Graph-Powered Observability Data Analysis in Databricks With Credential Vending

Talk by Danfeng Xu, CTO, PuppyGraph ; Eric Sun, Head of Data Platform, Coinbase
Observability data — logs, metrics, and traces — captures the complex interactions within modern distributed systems. A graph query engine on top of Databricks enables complex traversal of massive observability data, helping users trace service dependencies, analyze upstream/downstream impacts, and uncover recurring error patterns, making it easier to diagnose issues and optimize system performance. A critical challenge in handling observability data is managing dynamic RBAC for the sensitive system telemetry. this video explains how Coinbase leverages credential vending, a method for issuing short-lived credentials to enable fine-grained, secure access to observability data stored in Databricks without long-lived secrets. Key takeaways: Querying Databricks tables as graph structures without ETLing data out Secure access management with credential vending Practical graph-based incident analysis solution at Coinbase, with insights on how PuppyGraph enables this application
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to observability
• 1:41 Data lake architecture
• 9:12 PuppyGraph integration
• 13:44 Credential vending mechanism
• 14:56 Graph query use cases
• 16:42 Advanced analytical insights
• 18:26 Future innovations

FAQs

What is observability data?

According to this video, observability data consists of logs, metrics, and distributed traces—the three pillars for understanding what's happening in modern distributed systems over time.

How does graph analysis help?

Graph query engines enable complex traversal and correlation of observability data, allowing teams to trace service dependencies, analyze upstream/downstream impacts, and uncover recurring error patterns more effectively.

What is credential vending?

Credential vending is a method for issuing short-lived credentials that enable fine-grained, secure access to sensitive system telemetry and observability data on Databricks Data and AI platform without using long-lived secrets.

What are the practical benefits?

This solution helps incident response teams diagnose issues faster, optimize system performance, and maintain security—all while keeping data in the Databricks Data and AI platform rather than exporting to other tools.

Full transcript

0:00 Introduction to observability

Okay, good morning and good noon and thank you for coming to this session. Um, as the title said, this session is going to be observability data on the data lake powered by some graph query engine called Poppy Graph. Um uh I'm from Coinbase and from uh Poppy Graph. Uh my c-orker Shinui cannot come to this uh presentation today. I'm sitting in for him. So um thanks for your interest about so we're going to cover about a few things.
One is what problem we're solving here right? Uh observability is a very interesting and fast growing sector. So we're going to talk about how to leverage data lakeink to complement the other observability solutions like a data dog to give you better business insights about it and we're going to talk about a few use case we are solving at Coinbase and of course we're going to talk about the data bricks technology integration including unity catalog uh credential vending and how do we leverage additional nondatab bricks computation engine on top of a data bricks infrastructure so that you have choice and you can make the most of your data bricks you know investment right and the last and we're going to talk about what we plan to do in the future and hopefully that will give you some of the ideas and inspiration right so first of all what is observability data so observability data mostly are metrics logs and distributed traces These are the three pillar for observability data.

1:41 Data lake architecture

Uh they are all time series based. So what we normally do is using time series to plot them out and see what are the events and uh activities happening at the same time. So you can correlate to that. This is what Splunk and the data um data do up. What happened? Sorry. Oh, am I audible? Oh, it's back again. Okay. Okay. So, let's continue. So, the majority of observability data are plotted and reasoned based on the time series, right?
But then as a business grows, you have a more complicated use case. You want to correlate your particular user ID to see what this particular user has generated the events and what kind of order were getting stuck or which particular service are generating too many retries and generated the storm of connection to the downstream application that cause a congestion right all this like a deep dive analysis you kind of need better analytics which is kind of expensive and hard to provide.
So one of the common problem that we're trying to complement for the existing observability data platform is more like say how can we or augment the rich huge volume of observability data with your business domain data together. Right? So I'm going to give you a little bit volume matrix here to see how big or how complex the problem is. So at a daily basis we can generate 18 to 26 billion just trace data that's in billion not a million.
And uh after the storage optimization we did with partition by time and sub partition by the uh you know trace id and all the prefix and we do the zordering of you know all the frequently used predicates filter. So that is trying to make the query we're going to do at a later time much more performant and efficient because if you look at the volume right you have a 1. 6 six terabytes to almost three terabytes data generate a single day and if you want to query this one very efficiently your data layout need to be optimized accordingly right so this is something we we're talking about so compared with data dog and other system now you are able to keep the data for longer time right typically for cost reason you can only keep very short period of time in data dog or Splunk otherwise your cost is going to skyrocket rocketing right so in this um architecture diagram and you see we just have a mechanism to use the gateway to duplicate or we call dual publish the data dog traces data to both the internal system on data lake in the real time with minimum latency but at the same time we are not affecting the uh data dog you know path itself.
So in this case this is a nondisruptive way to fork your data for additional analytics purpose. Well we're going to talk more at a later time how we can use puppy graph to handle the huge volume of data with affordable cost and uh you know couple of use case here. So now spend a little bit time to explain the traces data structure. This is a kind of like a open standard. This is not specific to data bricks, data dog or puppy graph.
This is an industry standard. Something called a trace is a series of a you know activities happening in your uh micros service ecosystem. Every bar you can see here is kind of like a time span of an API call in one of your micros service and a series of them combined together is one one thing called a call graph or user journey right and we need to kind of combine them together to form the full picture about what happened for that API call coming from a mobile apps enter your API gateway and ex span through your kind of like fleet of a micros service farm and then you can see there's a some complexity about this one and naturally this going to be a huge volume of data your micros service are generating every single second how do we make sense of this data that's the challenge that the data dog splunk try to solve and today we are helping you to think about the third one right which is how can we analyze this data in data bricks with the help with some of the graph query engine right so here we're going to talk about a more specific use case so from that core graph and uh traces telemetry data we can first compute something called service graph right it's a topology about which service call which service and then we can argument the additional data on top of it like how many calls are from service A are to service B that's that's the easy part then the next level is between the call from service A and B how many of them are returned within the latency expectation how many of them were beyond the latency expectation how many of them returned with error code and when We orgument and aggregate these numbers on top of it.
Your topology graph is no longer just the lineage. It is a live data to show you are we operating in the healthy state. Do we have a performance degradation and how are all these problem are propagating through your topology. Now you can zoom in and see where the problem is and trace back where is the root cause of the problem because when you have a service degradation typically it could be widespreaded but you it's not everything is falling apart.
It's about one or two choking point spreading the pressure over your topology. That's where this powerful analytics can really help us to narrow down and figure out what's going on when and what is wrong. All right. Okay. The next one is we're going to talk about how this is happening. We have the data volume. We have a billions of you know tracking point. We have a terabytes of a data every single day. And typically when something wrong, some incident happened, we don't need to scan the entire days of data.

9:12 PuppyGraph integration

We zoom in to maybe up to two to three hours of data and then we connect them to form this graph. Now I'm going to invite from a puppy graph to really explain how this can happen on top of a datab bricks infrastructure and what kind of a analytics they do and what kind of a mechanism and we leverage data bricks to make this one as easy as possible and so that you can reuse all your data lake investment on top of the open table format like iceberg and the delta.
All right, please. Thank you. Thank you. Yeah, so as we've seen, right, we've seen the power of the graph on the screenshot, right? Like the matrix or this like distributed tracing graph helps. The issue is like uh with the tremendous amount of data in data bricks on like terabytes of the like data data volume like generate every day. In many cases actually impractical or it's really hard or it's like not efficient to actually have the ETL like populated data into a different database.
And the way public graph tackled this problem is different. It's not a graph database. It's a graph query engine where it actually qu your data bricks data directly and build a graph model on top of that. So in this like today's session I'm going to talk about two important features. One is like how it supports credential vending. So it it seems to integrate with your data bricks data to securely and uh query your data in an integrated way.
Second is how it supports the same like popular graph query languages that supported by other like graph databases so that all the like tools all the integrations that were working will still continues to work. Um this one oh so the first thing we're going to talk about is the credential vending and this is like a diagram of the credential vending feature of how puppy graph talks to unity catalog and the data actually uh stored in like a data bricks in this example um they are stored in like delta format and iceberg format uh stored in S3 buckets the issue uh with like you know without credential vending is epigraph these to talk to storage and process graph queries.
However, in many cases the authentication or the O of the storage is managed separately. So in this case we need to talk to S3 right to get the data to get for example the actual files like pop files back to do the graph analysis. Um with credential vending what actually happens is when the first step requests come in puppy graph will send a request to unity catalog which is a step two to get the temporary uh credential uh vended back from data uh from data catalog on step three and that credential is down scoped because it only allows public graph to access a specific subpath. path of the S3 bucket that is corresponding to the tables it is looking for and it also uh having like a like time limit it's like not like longterm so that reduce the risk of like what if like some uh credential is leaked or it's like it reduces like the possibility of like uh the credential is like accidentally uh retrieval by other tools.
So this next step probably graph will send a temporal uh use a temporal uh authentication talking to the S3 and buckets and uh accessing the delta table and fetch the data back and process the graph queries and send it back on step five. So this is like a general uh flow of pipraph levering credential vending and this is really like a safe integrated approach because everything is managed by the unity catalog. PGraph doesn't have to worry about a separate setup of the credentials.
This is like the Yeah, this is like the Oh, did I? Yeah, this is actually a concrete uh request has sent to Uni catalog. Um yeah we have the uh credentials uh fetched by sending a post request to the temporary table credentials API on unity catalog and it is actually specifying the table ID whereas like we are interesting this table and we want to actually use the data inside this table to process some like graph queries and the response from unit catalog is a temporary pair of uh access key a security key and there's a session token and what more important is like the expiration time which as indicates it's a shortterm uh credential.

13:44 Credential vending mechanism

It's going to like stay there forever and pepper actually manage the expiration and then when it needs another session token it actually talk to unit catalog again again for the consequent uh like queries. So this is part of the credential vending with this pipraph can securely access the data storing in data bricks. Now let's take a look at what kind of applications we have on top of the uh tracing data. So this is like the first use case we're going to describe.
Uh this is like the service dependencies. The idea is like we have different nodes in the graph and each of the node represents a specific service uh in our use case. uh on the left side is like one of the uh one of the uh graph queries written in uh grammarling. So this is one of the popular uh graph queries. Pepigraph also supports like cipher uh which is another uh popular graph query language. Uh this graph query actually just like starting from some specific service and it treats it as a node and want want to do some like fan out traversing along the uh edges like defined in the graph. um doing some traversal and sending the result back.

14:56 Graph query use cases

And this is like a screenshot of a pipraph like uh query interface. On left side you put in the query and send it out and on left side on the right side it returns a result and visualize the service graph. And this is actually like not the whole not the full service graph but like a portion of that uh corresponding to the query response that we have on the left side. On this one, we can look at the query result and um explore the response of the traverse graph interactively.
I collect on the uh specific nodes and will return the actual information and this output can also be integrated with any kind of like Golan, Python or other like JavaScript like APIs so that it can be uh integrated further with other services of the service graph like and root card analysis system. Once we have the service graph, what we're going to do in the next step is like we have the uh another graph query and we want to argument the service graph with additional information because in a real like reality use case, right?

16:42 Advanced analytical insights

Like beside the trace just tracing, we will have other types of uh data and one things like we use the raw trace data here. uh we have the this query actually computes the error rates uh derived from the uh distribute tracing data. So this on the left side is a grabing query and actually computes the um error rate for this specific service. The idea is like on the left right side we have the uh error rate computed. So imagine um the use case we have we mentioned at the beginning of the presentation that we have uh server graph we have a data populated into data bricks and we want to actually understand what actually happened let's say within last 30 minutes in our uh you know serve infra and what if like some service went down and what's the impact on the downstream what's impact uh what could be the potential causes from upstream and these kind of like uh graph queries integrated with the queries that we we build for the service graph put them together.
It's going to be argument more and more information and provide more and more insights for the uh to build a comprehensive view of what is actually happening uh from observability like use case perspective and it's providing more information. All of this is happening with the data living in the data bricks and built on top of you know uh with the graph capability built on top of the uh data bricks data brought by public graph.
Yeah. So we briefly wrapped up the you know use case that we had at Coinbase and we talk we like to talk about some future innovations. Uh first is like we want to actually you know first two are the use case innovations. So we want to actually expand our use case. We want to uh bring in more and more different type of data uh to argument the existing uh service graph we have built and we want to also in stuff like only supporting the uh analytical use cases we want to actually expand it to like graph driven anomaly detection so that we can proactively see what is actually happening and to sense any potential issues before we actually notice uh the downstream like impact is like growing growing.

18:26 Future innovations

The other part of the innovation we plan to do is like the technical part. We want to actually spend more like the effort improve our uh incremental graph data processing which means instead of like computing all the graph queries from scratch from beginning right we want to actually introduce ways of like saving intermediate results and uh do that in a incremental way so that the next time we do the graph queries will be on top of the existing graph queries.
And we also want to introduce a high efficient sub data structure so that it can be more closely integrated with open table formats like the database like uh delta table like and like the Apache like iceberg so that it'll be uh powering more and more like um use cases and more like uh bigger scale of the um problems. Yeah. So with that I think that's the peritting for today and we yeah we have question time and we have this like delta take uh swag right like yeah for any any questions.

Learn more about the Databricks Data and AI platform.