Sponsored by: Acceldata | Trust at the Speed of Agents: Reliable AI and Data on Databricks

Summary

Acceldata's three-layered trust platform integrates data observability, agentic data management with a semantic context layer, and AI/agent observability into a single operating model for reliable AI on the Databricks Data and AI platform.
Each layer reinforces the next: observability without action is merely alerting, action without observability is guessing, and agents without governance create risk—making all three layers essential for moving toward a self-governing lakehouse.
The agentic layer automates anomaly detection and remediation, allowing agents to write pipelines, trigger quality gap analysis, and create policies autonomously, while the AI observability layer traces agent decisions and costs to maintain accountability.

Chapters

FAQs

What is the trust gap that agentic AI creates in the data lakehouse?

As agents write pipelines, call other agents, and act on data autonomously, organizations lose visibility into what the agents are doing, whether the underlying data is reliable, and how agents arrived at their decisions. Acceldata describes this as the trust gap—trust is hard to build over time and very easy to lose, making observability, governance, and accountability across the entire agentic system critical.

How does Acceldata's three-layered approach work on Databricks?

Layer 1 covers data observability—understanding the reliability of data and pipelines; Layer 2 introduces agentic data management with a semantic context layer that enables automated anomaly remediation and policy creation; Layer 3 provides AI observability that traces agent decisions, costs, and actions. All three layers are integrated on the Databricks Data and AI platform so that each reinforces the others.

What is the self-governing lakehouse concept described in this video?

The self-governing lakehouse is the end goal of Acceldata's approach—a state where data, agents, and policies stay in lockstep without requiring humans to be involved in every decision. This is achieved by combining automated anomaly detection, autonomous remediation agents, and continuous agent observability so the system can maintain its own reliability at scale.

How long has Acceldata been working in the data observability space?

According to this video, Acceldata pioneered the data observability space approximately seven years ago. The company works with some of the largest enterprises globally, including six of the top 12 banks in the world and three of the top five telecom companies in the US.

Full transcript

[00:08] Thank you for coming this morning. First thing in the morning, we're going to talk about trust at the speed of agents. I'm Joe Murphy. I'm principal architect, senior manager at our field engineering team at Excel Data. If you haven't heard of Excel Data, we pioneered data
[00:20] observability space going back seven years. Um and we help some of the largest uh enterprises in the world in the six of the top 12 banks in the world. Uh three of the top five telos in
[00:34] the US uh large CPG large pharma companies come to us because they have big hairy data landscapes as they try to make their transition to the modern spaces of data bricks. um and we help
[00:47] them understand the reliability of their data landscape that includes their data and their pipelines and bring all those signals together. So what we're going to talk about today and we're on a a
[00:59] mission in a 20 minute cycle uh five beats 15 minutes. We're going to talk about the trust gap that's developed as this landscape has shifted under us with agents coming into play and doing just
[01:11] about everything. Um so that what's that trust gap that's developed? We're going to talk about our approach to bring a three-layered platform to understand data observability, bring in the agentic
[01:23] capabilities to help you manage that, but then understand what those agents are actually doing and can we trust them, right? We'll talk about each layer, why integration of all that matters, and I'll get into a demo, and
[01:36] I'll try to squeeze that all in 20 minutes. I talk fast. Maybe we'll leave some time for a question or at the end. Okay. um what's the trust gap? So this
[01:48] has kind of shifted under us very very rapidly. Suddenly agents are writing our pipelines. Agents are calling agents. Agents are acting on data, right? What are those agents doing? What is that
[02:01] code doing? Is that data reliable for us that it's making decisions on it? How did it get to those decisions? The trust gap is a real thing. You have to build trust. It's hard to build over
[02:14] time and very easy to lose. You lose that in an instant and then you have to rebuild it. So it's critical that you understand your data. You understand it's what it's what the data is and how
[02:26] reliable it is. You understand what your agents are doing and how they are operating and can you trust them. So our approach is a three-layered approach. As I said, we pioneered our data observabil
[02:39] in the data observability space. So at layer 1, we really want to understand the data that you're operating on. And that's not just is it of high quality, is it fresh? It's can it be reconciled?
[02:52] Is it shifting on you? Is the distribution of data changing? Is the volume of data rolling and changing? What about the pipelines running it? Can you understand the lineage? Can you infer from that? That's a lot of signal.
[03:05] We collect more signal than any other platform about your data and the processes running on it. how do you start operating? So we built an agentic layer to help you understand that and then now we built a new platform on top
[03:18] of that to understand what those agents are doing. We did it for ourselves but really enabled a platform for all our customers to use it too. So a little bit more about the data observability right so this is continuous visibility
[03:31] in the data flowing into your lakehouse flowing from outside your lakehouse flowing to the next stage. So we really look across your entire landscape. It's critical to know where the data is starting from and going to. That's
[03:43] detecting changes in the data, drift in schema, uh freshness lags, breaks in reconciliation, understanding the profiling of the data, the statistical ranges, how those are shifting, critical
[03:55] for your machine learning applications, especially if you start training on those features and those features start shifting on you, right? Um the lineage, understanding the flow of the data. Now
[04:07] I always talk about two types of lineage. I talk about impact lineage. I need to know everything that touched this asset. And I also want to know about process lineage. If a pipeline's touching that, how is that pipeline run
[04:19] last time? How did it run a month ago? How did it run three months ago? And are there changes in that? So think uh think of the processes and the lineage of that as well as the data and all the things that touch that and then being able to
[04:32] alert on those things, right? Make this operational. One of the biggest things that make our customers successful with our platform is they turn this into operational practices. Right? Oftentimes I hear talk
[04:45] to customers that are coming from a governance perspective and their big gap is trying to take their governance policies and make them operational. And that's one of the things we can do. So if we make this operational,
[04:57] it's just embedded in the fabric of your environment that you can trust your data. >> Now, As I said, we collect so much signal
[05:09] that while we connect and build context from the pipelines to the data, what's happening, the controls you have on it, connect it all together, that's still a lot of signals. And we found that we needed to build a Gentic systems on top
[05:22] of that to help clients understand all that's going on in these complex landscapes. Right? So we built uh an agentic system that uses the powers of the LLMs, but we've built a bridge to a
[05:37] semantic layer to all this signal that we've developed. That's through our data quality agents, our pipeline agents, complex agents to build workflows and take actions for you and ability to reach outside of the system through MCP
[05:49] integration. And I've done these where I'll reach out through MCPS to Service Now or to Salesforce or other systems that I need to get information and take action from. Right? So it's a varied
[06:01] landscape, but this enables us to do what you'd expect. I can plan what we should do. I can reason. I can actually drive action. And that's why we start talking instead of our data observability or our data management
[06:14] platform because we can start taking action and put conclusions and results that the LLM will start giving us into action for them. Okay. Now what we found
[06:27] early on when we went down this path was can I trust that agentic system on all that data that we have underneath it and the conclusions it's bringing to us. And
[06:39] when it came to that conclusion, how did we get there? So we started out early um building this well all in sort of internal traces. How can I explain how I got from A to B? Giving proof points
[06:52] ensuring that our agentic system isn't hallucinating. But we needed to make that more systematic. So we built an AI observability layer so we can see what those agents are doing. We can trace everything through there. We can
[07:04] understand and improve what the agentic system is doing. measure it and evaluate it. When we built that again on an open foundation, integrated into our platform, basically we said you can
[07:16] bring your own agentic systems, agentic applications and observe them and evaluate them here too. So build out an environment where you can observe your data and the agents that are operating on your data all at once. Right? So each
[07:30] layer in this case reinforces itself. Right? And I'm going fast so I can get to demo which I think is more exciting for you guys. Okay, it's important to have these together
[07:42] because if you just have the observability and you're not ready to take action, it's really just alerting right now. That can be very valuable for many customers that make this operational
[07:54] because we're plug into their service now environments. They've got detail. They can take action from it. But basically, this is alerting. Now, flip that around. If you're going to have things that take action, if you
[08:07] don't have the observability, what are you taking action on? You're guessing at what could happen. And again, if you have agentic systems that don't have some sort of governance on top of them, you're just introducing
[08:20] risk. I'm going to come to conclusions that aren't founded in fact or somehow hallucinations have come into the mix. Right? So for us, integrating all these together is critical to building a
[08:33] trusted uh environment that can operate at speed for you. So with that, I'm going to try to get into a live demo for you. What we're going to look at is each one of those
[08:46] layers. Uh understand how we can detect actions that are going on. uh how the agentic layer can start taking action
[08:58] for you based on what it observed and then how we can see what those agentic systems are doing so we can understand that we can trust them improve them going forward. Okay,
[09:11] you can see if I actually cover all those spot those uh spots and um give me a rating at the end. Okay, so I've just come to the sort of homepage of our data
[09:24] observability platform. And if I bring this to up here, you can see this first layer is the data observability cloud where we're able to discover things like assets, pipelines. Uh we can put governance
[09:36] policies, essentially our controls or rules that we put on our assets that say, hey, this has to reconcile from point A to point B when you're bringing data into your data lake. uh that this
[09:49] uh aggregated data has to match up to the unagregated data that you've met all your business requirements for the rules um and we can generate insights. Our agentic data management again integrated
[10:01] into that that gives us opportunities to get into conversations with our data if you will and then our observability layer at the bottom. Right now our clients tend to have this is a little
[10:13] demo system only have 200 assets. Our clients have tens of thousands of assets registered for observation in the system. They may have thousands of pipelines they have under observation. Right? Once we get an asset under
[10:25] observation, this is a process transaction table of sort of 360 degree view of an asset uh in this my delta lake. Right? So I should say lakehouse,
[10:39] my apologies. Um, this gives me an observability view, a consolidated view of my asset. Right? In this case, the performance trend of the asset, the freshness of the asset. I can
[10:52] look into an individual columns in the asset. This is just a table in the lake. And I have the hierarchy to navigate through all these things. Right now, on this, I have a connection context of all
[11:06] the pipelines that are working. These are data bricks pipelines that we're observing and I can see which ones are connected to this asset. Some are uh you know producing it, some are in the middle of it. Um but I have this context
[11:20] of all the pipelines working on the asset. um and the controls that I've put on the asset in this case a data quality set of controls um and a reconciliation but I could have freshness controls
[11:33] anomaly controls drift controls different controls I can build out that I want on the system right now um I can also build a lineage view of the asset
[11:47] now in the middle this guy on the home that's the asset that I'm looking at, but I'm looking upstream and downstream of the asset. We've assembled this view of the asset
[11:59] through our observations of the pipelines that are working on the asset as well as observations of things like query logs uh and other sources of lineage information. So, think of this as impact lineage. This is showing me
[12:14] everything that touches this asset and information about the last time say a pipeline ran whether successful um or a failure. Right? So I can get a lot of
[12:26] insight from this view. Now this is a I flipped the page here to one of the pipelines that is operating on this data. Now this is a DAG of the pipeline
[12:39] a little different than most representations because it just it doesn't just show the task but it's showing the assets. So the blue boxes are actually assets under our observation as well as the tasks within
[12:54] that pipeline and I can link other pipelines that may be running independently that we can resolve operate on the same asset. So I can build out this graph of all the processes that are working on my data
[13:08] and I have a view of the history of all those runs and I can look back in time and see how that process ran that time. Put monitors on things like the time it
[13:20] takes to do any task within it and have SLAs's on when this should run, when it should complete, what should happen if there's errors and I can monitor. Now we're not the orchestrator. We're observing the pipelines, right? This
[13:33] could be your data bricks pipeline. I could be uh it could be airflow, could be any number of tools that we can bring the lineage and the observation of the pipeline into the process. right now. Um
[13:49] that ties back to the policies uh uh excuse me the pipelines we were talking about but the policies are the controls that I then put on place sorry on top of
[14:01] any of these assets right so how do I understand if this meets my quality requirements for example so we build out um these template jobs basically that are the controls that you need that say
[14:14] hey in this case there are 22 different rules I have on on the data and then we keep metadata about the execution of this. So as data flows into this transaction table, I'm running a series
[14:26] of controls on top of it um that check my business rules requirements that could be as simple as this can't be null or as complex as hey I have business rules around posting dates and
[14:38] transaction dates or I could have rules on hey transaction checks uh like I need to check against an OFAC list about a transaction for a regulatory
[14:50] requirement. We can even record for you records that fail these uh validations and store them off for you in your object store. So you have a traceable
[15:02] view of how your data is performing and exception records. So this is deep observability into the state of your data. Right now
[15:14] this brings a tremendous amount of signal. All these become operational. You can see on the left nav all the different runs of this set of controls and I can look back in the history. So I
[15:26] have this huge amount of metadata about what's going on. This is where our agentic system can come in and start helping. I could click analyze this and it could go off and start saying what's happening in the execution of this
[15:40] policy over time. Just fire the agent off. any of these um spots I am in my observability I can say I'm in context I want that agent to go off and analyze it or I can start with right in the agentic
[15:54] system where you'd say hey like any other aentic system how can I help you today right so I can start operating on top of all that metadata with our uh agentic system and that could be as
[16:07] simple as asking for an assessment of the controls on my regulatory reporting assets. Now this is a group of assets and it could start ask it can start doing analysis for me across all that metadata right telling me where I have
[16:21] controls where I have gaps um and what I should do about it right now that's where I can start kicking off the workflows or having the uh aentic system
[16:33] take action for me so the first thing is said go ahead and generate a business context around these assets and then I also asked it later on to say go start building those controls for me that
[16:46] would kick off individual workflows. So this is a business context one and said here's a basic prompt. It actually just went and did this for me. I didn't have to type anything else in but I can track through it. went through a series of
[17:00] different agents to build out a complex uh business context report around those assets that I can take and decide to put a plan in place how to improve my
[17:12] maturity on controls on these assets. again because the agentic system has access to all the metadata that I've collected and generated on the assets, the controls, the processes uh making u
[17:27] up my landscape. Uh sim similarly this one was go ahead and create a policy. So I showed you that shot of a policy and controls but the agentic system can build it up for me. So in this case it's
[17:41] just showing me in the wizard and actually building all individual rules for me giving me fully documented uh controls telling me why it built that and then I can have a human in the loop
[17:53] to actually yes I want that set of controls and I can operationalize it. So think about this is an agentic system which we all know is not deterministic. I can ask an agent the same question twice and get slightly different
[18:06] answers. But here it's building out a deterministic control. This will run exactly the same way every time. So if you're a bank, if you're a manufacturer, if you're a pharma company, you need
[18:18] controls on your data that are deterministic. Um they can evaluate a lot of other actions about trends etc. But needs to be done the same way. So this is the using the agentic systems to build
[18:30] deterministic controls, right? that can even help me understand things like my pipeline. So I showed you before a pipeline that was pretty simple. This is
[18:42] a pipeline like the pipelines we get from our customers, right? And if you walked in and said, "Well, what is that DAG actually doing?" Um, you'd be pretty hardressed. Now, there's ways to navigate it. I can pop to different
[18:55] areas and say go to that part and jump around. But what I really I can do is I can tell the agentic system analyze that and give me the breakdown of what this thing is doing. Bubble it out, break it out for me. I could then
[19:08] even start asking questions in conversation with this agent to say look at the history of the runs and why is this one failing? What's the root cause of the last five failures? It has all the data about the runs. It can operate
[19:21] on all of that. Right now I got 49 seconds left. Okay. So now you get to the a uh AI observability, right? So we've instrumented that AI application.
[19:35] Um and so it omits open telemetry to our AI observability layer and you could instrument other ones but ADM is the application that we uh ADM is our application. Now I can get insights
[19:49] automatically on that application. The volume latency, how is it performing, what's its cost, what's its token usage. So I have a monitor in that if I needed to understand what's going on, I could
[20:02] go in and a thread is think of it a conversation thread. And then the next is a trace is a trace of a prompt. Um and then a span is what that agent is
[20:16] doing in response to the SP uh to that prompt. So I can drill all the way through to say I need to see uh you know this particular conversation I want to view this trace see what happened in
[20:29] there what did the agent do if I'm trying to analyze it um even go down to the prompts say this or spans rather that are the individual actions that it took see where the LLM you did an LM LLM
[20:43] call a tool call what happened in that actual response and understand all those pieces, right? Um, it has the ability, we brought in the ability to let you do
[20:55] evaluations, prompt engineering, uh, optimization and even produ observability of the production clients to see if you have rules in place and
[21:07] evaluations going online. I'm running out of time here. Uh, unless they kick me out, I will take a question or two. Um, but uh hopefully this hopefully I hit
[21:20] all those actions for you in a we're in booth 603. So if you want to come see us, I'll be over there talking all day. Any other questions? Otherwise, thank you and give a give a nice evaluation so
[21:32] my marketing leader will like me.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.