Skip to main content

Why Data-Intelligent Agents Win: A Side-by-Side Benchmark

Summary

  • A live benchmark in this video compares two agents running the same LLM on identical enterprise workloads: one using raw SQL, and one grounded in the Databricks Data and AI platform's Unity Catalog semantic layer, governed metric views, UC functions with LLM-readable docstrings, and Vector Search over enriched business context.
  • The data-intelligent agent outperforms the raw SQL agent on both quality and token efficiency because it does not need to rediscover schema on every request—Unity Catalog primitives act as typed, reusable context that reduces wasted tokens and improves accuracy on ambiguous and multilingual questions.
  • Three core benefits of grounding agents in the Databricks data intelligence stack—quality, cost, and efficiency—are presented as the production standard for any organization evaluating whether to replace traditional data pipelines with pure LLM-driven approaches.

Why Data-Intelligent Agents Win: A Side-by-Side Benchmark

Watch: Why Data-Intelligent Agents Win: A Side-by-Side Benchmark
Vibe coding is fun until it hits production. In this video, we benchmark two agents running the identical LLM on identical enterprise workloads — one armed with raw SQL and good intentions, one grounded in the Databricks Data Intelligence stack: a Unity Catalog semantic layer, governed metric views, UC functions with LLM-readable docstrings, and Vector Search over enriched business context. We run the benchmark live, on real questions including the ambiguous, multilingual ones your users actually ask. Then we step back and look at what the trajectories tell us about where production agent cost actually comes from — and why the platform underneath your agents matters more than the model running on top of it.
Talk By: Austin Choi, Delivery Solutions Architect, Databricks ; Dan Pechi, AI Product Specialist, Databricks ;

Chapters

FAQs

Why do data-intelligent agents outperform raw SQL agents?

Data-intelligent agents built on the Databricks Data and AI platform have access to Unity Catalog primitives—functions with LLM-readable docstrings, metric views, and Vector Search over business context—that eliminate the need to rediscover schema on every query. This reduces wasted tokens, improves answer accuracy on ambiguous or multilingual questions, and lowers per-query cost compared to agents that start from raw SQL each time.

Should I replace my data pipelines with LLM agents or Claude Code?

This video argues against replacing traditional data pipelines entirely with LLM-based agents for workloads like fraud detection, where years of carefully engineered features represent irreplaceable business logic. The recommended approach is to use Unity Catalog primitives as the data layer that agents interact with, so the LLM focuses on reasoning rather than schema discovery.

What is the Genie ontology in the context of agent data access?

The Genie ontology is a component of the Databricks Data and AI platform that helps agents understand the business context behind data, including how tables relate to each other and what business metrics mean. This contextual understanding allows agents to produce semantically correct answers rather than syntactically correct SQL results that violate business rules.

What are UC functions and why do they matter for AI agents?

UC functions are reusable functions registered in Unity Catalog with LLM-readable docstrings that describe their purpose and parameters in natural language. Because agents can read and understand these docstrings, UC functions act as typed primitives that agents can invoke correctly without needing to reverse-engineer behavior from raw code or schema.

Full transcript

[00:07] Yeah, cool. So, I think first a question for the room. Who has hit a Claude code token budget limit? Who has had to just stop working and then say, "Okay, I guess I'll wait like 2 hours."
[00:19] I think I'm in one right now. My budget reset 20 minutes ago. There are ways in which to mitigate this, obviously using Databricks AI Gateway, but there's also some really straightforward practical ways in which
[00:32] we can better manage all of this, and also get better quality usage out of Claude code and agents more generally. And so, what we'll be talking about is this idea of data intelligence as a key
[00:45] component to agents and making them more powerful, more efficient, and more cost-effective. So, we have to put some forward-looking statements that tell you things about,
[00:57] you know, what we will or will not or may hypothetically or may hypothetically not do. So, keep that in mind. Also, please fill out the survey. Tell us what you really think. Great. So, the other big question here when it
[01:10] comes to, "Hey, how am I working with my agents? How am I looking to use them more effectively?" is this idea of just replacing everything with Claude code. So, Austin I talk with tons and tons of
[01:23] our customers, tons of our digital native customers that are all in on Claude code. They're token maxing and everything, right? But, there's kind of this fundamental question that I see a lot of them coming to, which is, "Where is the barrier of
[01:35] Claude code and the beginning of, say, traditional workflow, right?" How agent-driven should a data engineering pipeline be? Do I need to have Claude code just running those pipelines more regularly?
[01:48] And can I maybe just replace Databricks entirely? Right? We think about doing like text to SQL, which is an example that we'll like go into in detail, right? That seems a very logical thing that we could just have Claude Code do for us, right? Um
[02:03] I'll specifically mention this customer that wanted to do fraud detection, right? They have some fraud detection models today that their data scientists probably back a couple years ago spent probably a couple years putting together, right? And they were thinking,
[02:16] "Hm, you know, we're thinking about just using Claude Code for this now." So, what that means is for a traditional fraud detection model, right? We're maybe going through the whole process of defining some features.
[02:29] Maybe we'll look at some unstructured data, right? Putting that into, I guess, Claude and then asking Claude to reason over all that and make some predictions about whether or not there's fraud detected, right?
[02:41] Hopefully, for anyone that has like more of a data science background, this sounds kind of insane, right? Uh we cannot replace these kind of fundamental things with Claude Code. Um and like I, you know, kind of
[02:53] previously iterated, we also see this for things like data engineering pipelines, right? Similarly, we can maybe augment those processes. Maybe there are benefits using language models in the middle, but really want to emphasize that there's tons of challenges uh with replacing your whole
[03:06] stack with Claude Code. Um and hopefully by the end of this presentation, you'll see and understand and appreciate why. So, uh just to get a little nerdy with it for a bit. Um so, with code, there's
[03:19] this idea of a language server protocol, right? We have types. We have kind of verifiable ways of saying, "Hey, this is viable code. This is code that actually runs, right?" When we think about agents, we don't
[03:32] really have the same paradigms, right? Like if we think about MCPs, right? There's kind of this fuzzy layer of what that tool does or what that API does. Um and we don't have a really nice
[03:44] way in which to work with these things more consistently, let alone deterministically. Obviously, I think this idea of determinism, if you're kind of stuck on this idea of like, "Hey, why might Cloud Code not make sense?" That determinism is one of the easiest ways
[03:57] to kind of make it make sense. But, the other piece of this is we lack primitives for what agents should be doing. We'll make the case of Databricks that those UC primitives
[04:09] that are highly governable, highly modular, or and discoverable are the best ways to go about that. But, in a way that a language server protocol prevents coding agent from maybe calling functions that don't exist, making up
[04:34] modular and usable components, whether that's a UC function, vector search, or really anything in between to work with your data, right? So, we will highlight three cases where the
[04:47] use of these things, whether that's a metric view, whether that is a Genie space, ends up leading to three huge benefits around quality, cost, and efficiency, right? When I mentioned the
[04:59] Cloud Code budgets, well, just think about all the stuff that Cloud Code is doing behind the scenes. I'll let Austin kind of speak to that in more detail, but for something like text to SQL, don't use Cloud Code. Use Genie space.
[05:12] It has an ontology. It knows how to work with your data more efficiently. And instead of just recommending this, we'll actually demo this in a bit, but we'll touch on each of these in a little bit more detail. In general, right? I think we saw this
[05:25] before. I saw people that were like, "I'm going to use an LLM to forecast demand." And I was like, "That's insane, right? You have forecasting models." Same exact principle applies for our
[05:37] agents and the problems we're looking to solve with them. Can you do it with Cloud Code? Yeah, but that's kind of stupid and incredibly wasteful, right? You might get a wrong answer or something that's directionally correct, but good luck getting that again like 1
[05:50] week later when you decide to try to detect fraud on an entirely non-deterministic way again, right? Um, the thing is a lot of these things like seem plausible. Um, there is like a lot of thought literally put into them and
[06:02] all those uh reasoning tokens. But, you know, the right answer is much harder to get to and you know, we have existing methods for getting to those answers, right? And hopefully you'll see the detours, the kind of um forking paths that using
[06:16] something like cloud code can take. But, augmenting it with UC primitives makes it a lot more grounded and a lot more actually capable for solving our issues. So, now I'm going to pass it off to Austin to illustrate some of these
[06:28] points. Um, and we'll start with quality. >> Yeah, so we're going to go kind of step-by-step through the three areas that Dad just touched on, but I want to explain the quality piece of this. I think you've all like experienced this
[06:40] where if you have some ambiguous terms like what services do users use to pay rent, like is this rent? Is it rental? Like, or what exactly do we mean by this word rent? Um, if we simply grep it or use like a
[06:54] keyword search or something like that, you may not actually get the answer that you're looking for, but the agent will still give you an answer. Is it right? It could be, but we don't know that. But, if you use something like a vector search and have more of a similarity match, at the very least you can go back to see like, oh, how
[07:07] confident was that like search to figure out what figure out if that uh query was relevant to the original question. So, pretty basic idea, right? Like, uh and that's just this is a big problem that a lot of these uh different agents have.
[07:21] So, what we're going to do is we're going to ask this question, what is the average ticket size for Yappy loans by age cohort? And I want to navigate over to the demo itself. So, this is going to take a while, so I'll talk as it goes,
[07:34] but here's the question, what is the average ticket size for Yappy loans by age cohort? I'm just going to let it run. What you're seeing on the left is the general purpose uh cloud code agents using the cloud cloud agent
[07:47] SDK. It's just out of the box cloud code running as is, just accessing DB SQL to look through the data in my workspace. The one on the right is our data intelligent agent where we've done a
[08:00] little bit of a little more work to set up like metric views and other resources for it to use so that I can find the answer using those resources because it has more information on like what we mean by the actual average ticket size
[08:15] for the up a loan itself. So, this is a little dependent on the internet connection, so we'll give it a few more minutes. Um but you'll you're already starting to see that the general purpose
[08:27] out of the box cloud code agent is taking more tool calls at this time. And is it hanging now? Um We'll give this a few more minutes few
[08:39] more That's as it goes through this. >> Yeah. >> Maybe in the meantime though, right? We can kind of see the difference between these two already. Right? So, this one is executing a bunch of SQL. Right? What
[08:52] does it mean to execute SQL via SQL MCP? Means a couple of things. One, we have to write our SQL query, right? And anyone that's ever written a series of super nasty CTEs knows that A, that
[09:06] really sucks. B, takes a lot of time. And C, is pretty hard to get right. Um we're probably going to spend a lot of time iterating on that or cloud will spend a lot of time iterating on that. And just the generation of those tokens
[09:18] is a huge waste of time uh and money, right? On the other hand, if we define one of these KPIs, so if we go back to our question, right? Which is what is the
[09:30] average ticket size for Yape loans by age cohort? Well, we have something in Databricks called metric views. Metric views help us define different KPIs, helps us define what a cohort even
[09:42] means, right? And instead of Claude code kind of looking to redefine these things on the fly, come up with that by reasoning over thousands of tokens at a time, instead defining a metrics view in
[09:54] Databricks and asking Claude code to instead just use those UC primitives, this ends up leading to ultimately much better performance and you know, that ultimately means we're able to work more effectively and
[10:07] actually trust our outputs more. Um, this is if we have LM connections obviously that are you know, still hitting Databricks but we'll blame it on us being in in the basement here
[10:20] and also everyone else token maxing in our workspace. >> Yeah, let me I'll just double check real quick. Uh, it's is it still good? Um,
[10:37] I think this is Okay, we'll give it a few more seconds for this. Um, Oh, we'll have Okay, we will we'll come back to this and I'll cover kind of the
[10:49] other components of this. I do think it maybe it's right now due to the connection right now. So, anyways, so what what's supposed to happen is that you the agent
[11:03] is going to look at that metric view specifically and then it notices that like there's already information on this particular question that exists in the workspace and so that only existed because your your data
[11:16] team, your data engineers or someone put that information in your workspace already so that context already exists. So, it knows to go down that path and understand like, oh, that's actually directly relevant to what I'm looking
[11:28] for and it has definitions of what this is supposed to be and so will use that instead of actually navigating through all these different SQL queries to figure out what the actual answer is. So,
[11:40] if that query completed, you would have seen that it only took one tool call to finish, and then the other one would have taken like 10 tool calls and like three times the amount of tokens. But, we're going to move on and go take a look at cost because that's directly related to that because the AI-ready one
[11:54] would have taken significantly less time and significantly less tokens to actually complete that task. And all we did was add a metric view. Like, you all don't need anything too fancy. You just need to put in a little bit of elbow grease into this and put in like a
[12:07] metric view or some kind of semantic detail, and it's all ready going to perform significantly better than something that's just Claude code out of the box. And so, once we get the demo working, we'll
[12:19] we'll show that like the vibe the regular vibe version is going to take significantly longer, but you're going to get the same exact answer because of that. So, moving on to the next portion of this, the cost if we ask a more
[12:33] complex question like the one above, you'll find that the AI-ready agent, the one that's actually augmented with data intelligence, uh can handle this question a lot more efficiently than the one that's kind of out of the box with just Claude code.
[12:46] And in the demo itself, you'll find that the one out of the box takes twice as many tokens and twice uh twice as long to actually answer that question. So,
[12:58] it has to continuously try and discover all this new information just so you can answer this question. When if you took the time to actually set that up, it makes that journey that that agent takes
[13:10] a lot simpler for for it to travel through. >> Yeah. And if you're thinking about this, right, from a am I using my Claude code budget up in a good manner, right? A
[13:22] really easy way to see this is using our Unity AI Gateway to make sense of that cost observability, right? Because really what we're getting at is there's a lot of cost coming from just poor tool
[13:34] invocation, usage of MCPs that are unnecessary, right? I've seen Claude code veer off and start trying to reinvent the wheel like so many times and if I have good governance around
[13:46] that, if I prevent it from trying to write every single SQL query from scratch by I don't know limiting the requests per minute on a SQL MCP,
[13:58] I can start to control for these sorts of things and steer Claude code or any other agent in the direction of using tools in a way that's ultimately going to lower our overall bills, right?
[14:10] >> Absolutely. So, and the one last thing that we'll call out here is the efficiency of this. Once you have some kind of layer some kind of layer some again going back to the metric view example, if you have something like that that it already
[14:22] exists, you can afford to use a cheaper model. You don't have to use Opus, you don't have to use the most expensive model available to you now. You can just you can more confidently use something like Claude High-Q and not
[14:34] only will you get faster answers, but it's going to be way cheaper than running Opus. So, this then you know, we can apply this across the board for like open source models and things like that, but just that little bit of effort makes
[14:46] a huge difference. So, again in the demo we would run uh once with Opus and then once again with High-Q and you'll see that they would come up with the same answer, but High-Q is just faster and cheaper. So, the obvious answer is to use High-Q,
[14:59] right? Like why am I getting the same answer and why am I paying more for it? You want you just want you'd want to use the cheaper one or it's the same quality. So, why am I paying more for it? So, that makes a big difference in
[15:11] setting this layer up so that your coding agents know how to navigate through it. >> Yeah, and I think like one helpful analogy with this is you know, some of these like latest breaking models like do think that they can kind of boil the
[15:23] ocean, right? They have a little bit more confidence in their ability to write the perfect CTE, right? And so, they're kind of kind of be steered in this direction of wasting tokens, of reasoning because they're high reasoning
[15:36] models. Um and you know, that is ultimately what drives costs. Most small language models are fine-tuned or post-trained to be particularly effective at calling tools.
[15:49] So, the more that we can kind of abstract away the actual work in the tools themselves, right? Genie is a really great example of this. The more effective our higher-level
[16:01] agents are going to be, right? And so, I think the key area of this, right, is Genie has access to our ontology, right? And it's kind of a meta example of this idea of leveraging the UC primitives in place
[16:15] to more efficiently work, right? Genie is both a combination of a really, really good language model. It's not a terribly large one, but because it has access to the ontology, it performs particularly well,
[16:28] particularly efficiently. Having cloud code invoke a Genie as opposed to try to be Genie and also be a coding agent and also be everything else under the sun, you're ultimately putting the burden on
[16:41] that one agent, which drives some of these costs here. >> Absolutely. Okay. So, I before I show you this slide, we'll try this again. Uh because I think a 504 error is not good. So, we'll try
[16:53] this again. Uh and see if this loads. Um
[17:09] All right. I think we'll just we're just going to move on from this cuz it's definitely not performing right now. It's So, what we were going to show is that the out of the box agent is going to navigate through like, you know, show table. It's
[17:22] going to describe what's in there. It's going to do select star. It's going to go down this route. I think if you all like just looked at the thinking process of your cloud code agent, you can see it doing this, right? And then your token count is like skyrocketing because it's
[17:35] trying to write these queries or it's trying to run these commands just so it understands what it's looking for. But, once you have like a single metric view or once you have a single like vector search index or whatever it is that you
[17:48] create on Databricks or elsewhere, it's just one path that it needs to take. It doesn't need to do this maze through all your information just so it can answer just so just to answer the question. And
[18:00] so, a lot of what we were going to show you in the demo is to show that, you know, this path that it takes ends up costing you three times the amount of tool calls and double the token count
[18:12] and doubles the amount of time it takes to actually execute uh that task that you requested. So, with that, um these are it's just the vibes are always great, you know? I had a lot of fun
[18:25] using cloud code to build things, but at the end of the day, I do build something really cool, but sometimes the journey there is really complicated and it's not always the
[18:37] right way to get to that answer. So, a lot of this we're going to talk about why you would build this on Databricks. So, yeah. >> Well, so just to go back to our original
[18:49] question, can we run our entire data and AI stack through cloud code? The short answer is no. Um I think, you know, as we kind of think about that forking path, as we
[19:01] think about doing our vibe coding work more efficiently, right? This is the question that I think a lot of digital native enterprises are starting to try to answer now, right? If anyone saw the news about Uber having
[19:14] their challenges around inefficient token usage, uh they're a customer that we work with, and their question is, well, how do we do this more efficiently, right? Part of that is like some of
[19:26] these things that we introduced in the gateway around routing, being able to say this query probably doesn't require the latest and greatest model, maybe it can use a smaller model, maybe an open-source one. Um and then routing
[19:39] those queries accordingly. Um but, you know, the other piece of this is, you know, I hope you all see here is it's not just about model selection, it's as much about the tools, as much about the data, and the usage of kind of core
[19:52] Databricks components as it is about the AI, right? This is kind of a thing that we've been touting for a very long time. We're all at the Data + AI Summit, uh because data intelligence is really
[20:04] where Databricks has focused on in the last uh couple of years. And so, in the case of our fraud detection case, right, having cloud code poke around our data is just not the way to do things. It
[20:16] might seem revolutionary to say, wow, we can use GenAI for this. We heard that 3 years ago, right? Um but, you know, as we move into agents that happen to also be IDEs, we're kind
[20:28] of revisiting the same exact ideas and saying again, hey, you know, there's kind of fundamentals here. There's kind of a baseline that makes for more resilient enterprise-grade workflows um
[20:40] that we're kind of ignoring, right? For the sake of innovation, obviously, but the truth is somewhere in the middle, right? There are plenty of things that make sense to kind of drive through cloud code um or through agents more generally. Um it's just a matter of, you
[20:53] know, kind of sitting down with your account teams and working through it, right? So, uh when we have an agent, right, we maybe also don't have things like monitoring, or re-deriving our problems uh from scratch on every single run,
[21:06] makes for very non-deterministic workflows. If I were to ask the same thing and my data doesn't change and then I ask the same thing again, the kind of forking paths that Austin laid out, right, leads to
[21:18] inaccurate results. It leads to inconsistent results and ultimately leads to a workflow that, you know, I'm not going to have faith in, right? So, our coding agents are not replacements for our platforms,
[21:30] but we certainly can use them to interact with that platform, to build on that platform. Austin helped build AI Dev Kit, which is a collection of MCPs and skills to do just that. We wouldn't have built that if we
[21:43] thought Cloud Code could just be all of those tools itself. So, pointing an agent at raw SQL, burn the budget. This is the easiest way to think about it, right? Do I want to write the SQL query every single time or
[21:57] do I maybe want to parameterize it and put it into a UC function? Well, yeah, I mean, that's a really easy path forward here, but you know, that's just one kind of illustrative example of the kinds of efficiencies that we see
[22:12] people not thinking through as clearly, right? So, Databricks SQL over bare SQL MCP is going to have this kind of retrying fumbling process with column names, et cetera.
[22:25] Fundamentally, right, the SQL MCP doesn't have access to the very rich semantic layer that has been part of Databricks forever, right? You can think about this as part of the ontology and that's not something that you just
[22:37] get through trying and retrying SQL. Instead, working with Databricks more directly is going to lead to that. The bill for calling those SQL MCPs and rewriting every single query from
[22:50] scratch every single time is kind of self-evident, right? Every call is refeeding the entire conversation back to the model. I see people complaining about just that context token usage a bunch. So, the
[23:02] more you can smartly manage that, the agent memory as well, right? That's going to lead to lower costs overall. Um, so instead of having your token budget disappear overnight, right? This is part of that solution.
[23:15] The other thing that we've seen more and more of, um, is the idea of re-engineering data intelligence into a prompt. Um, I was talking with a customer last week that basically told me that they are storing all of their
[23:27] semantic layer and skills. And, you know, because of something like graph, they can easily search over that. I think there's some truth and some benefit to working with things that way, right? Uh, if there's a way in which to do a certain workflow,
[23:40] maybe a schema or certain table that's specifically important, sure, put that in a skill. Add that into your context. But, when you start putting in example queries, when you start putting in column names, when you start putting in
[23:53] all of the things that again, are just already native within Databricks into a prompt, into Cloud code and that kind of way of thinking about things, you're really reinventing the wheel here. And, more importantly, just
[24:06] wasting tokens, right? Everything I put into that skill, everything I put into my context in that instant, is going to eat up my token costs. And then you're going to hit your Cloud code budget limits that much faster, right?
[24:19] So, for things like joins, definitions, and guardrails, right? Uh, I assume a bunch of folks here have tried out Genie before. That is just kind of a primitive that we have within that Genie setup, so that you're not reinventing the wheel
[24:32] every single time just for that text to SQL use case. Um, but we see more and more people choosing to avoid that because eh, I don't want to have to bother with defining these things, right? For our
[24:44] engineering teams, um, there's a case where they need to set feature flags, um, to build out releases. Instead, we see people spending $5 in tokens every single time to flip a flag,
[24:57] right? It's kind of a okay, like can we please like take some, I don't know, intentionality, like some more deterministic processes in place so that we avoid these token overruns, right?
[25:10] So, the fix here is ultimately like having our context engineering be solved with things that we all know and love, which is data engineering, right? The more deterministic, the more we can really rely on the fundamentals
[25:23] of Databricks, those UC primitives, we're going to kind of shift our problems over from our language model into things that, you know, are much more reliable, much more enterprise grade. Cool.
[25:35] So, really that this is kind of the the closing message that we want to leave you all with, which is quality, cost, and efficiency. It's not just about the model, it's not just about, you know, swapping out something behind
[25:47] the scenes. That is important, don't get us wrong. We see this as being essential to being able to actually optimize costs for customers, but some of these things are simple,
[25:59] straightforward, and things that, you know, are just being forgotten about because we do have this impression that cloud code can just do everything for us, right? So, try wrapping a query in a metrics view,
[26:11] see the cost benefits, right? We have our gateway, so you can get that beautiful observability in place. Build a vector index instead of expecting cloud code to manage all that for you, and you'll be able to see for yourself some of the benefits there.
[26:24] >> I'm going to try one more time, and hopefully it doesn't >> You can see it's actually the internet and that the demo was working great because that upstream request to literally connect was
[26:36] timed out, so um yeah, unfortunate demo circumstances, but >> Yeah. >> We can maybe share something after after the fact, but I think we're we're just going to be blocked on
[26:49] on internet access most likely. >> I Yeah, I do apologize there. But yeah, making more and more data intelligent, not just smarter. So, um we'll post the
[27:01] repo link to the application yourself if you want to try try it out. Uh this will be posted uh on YouTube later, but that about that will wrap the presentation for today. Yeah. Thank you all.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.