Supercharging Genie Performance With Metric Views

Summary

Databricks' own data platform team deployed Genie spaces internally and found that raw tables were the wrong foundation for natural language queries—Genie was producing accurate results only about 50% of the time before the team restructured the semantic layer.
The team introduced Metric Views—YAML-encoded definitions that embed metrics, fiscal calendars, and business context directly into the data model—and pushed them into Genie spaces via a metric store that enforces uniqueness and consistency across queries.
Nightly regression tests run via the Genie API measure accuracy on an internal benchmark suite, and the approach improved accuracy from approximately 50% to over 95%, with internal adoption reaching 2,100 daily users and 30,000 queries per day.

Supercharging Genie Performance With Metric Views

Databricks Genie allows users to talk to their data, but its answers are only as good as the semantic layer beneath it. The Data Platform team shares how we are "supercharging" our internal Genie deployment by feeding it highly structured Metric Views. We will discuss the art of curating data specifically for GenAI consumption, ensuring that natural language queries yield accurate, trustworthy SQL generation. If you want to deploy AI/BI that actually works, you need to see how we structure our data for it.
Talk By: Krishna Bhupatiraju, Sr. Engineering Manager, Databricks ;

Chapters

FAQs

Why were raw tables the wrong foundation for Databricks Genie?

Genie generates SQL from natural language queries and needs explicit business context—such as metric definitions, fiscal calendars, and exclusion rules—to do so accurately. Raw tables lack this context, so Genie was essentially guessing at business intent, resulting in only about 50% accuracy on business questions in the months before the improvements described in this video.

What are Metric Views and how do they help Genie generate accurate SQL?

Metric Views are YAML-encoded definitions that encode metrics, fiscal calendar logic, and business context directly into the data model in a form that Genie can use when generating SQL. By treating Genie like a new hire who needs explicit context rather than tribal knowledge, the team gave Genie the structured information it needed to answer business questions reliably.

How does the Databricks data platform team measure Genie accuracy?

The team built an internal benchmark suite of business questions with known correct answers, then runs nightly regression tests via the Genie API to track whether accuracy is improving or regressing as the underlying data model evolves. This systematic measurement approach was key to moving from approximately 50% to over 95% accuracy.

What level of internal adoption has Genie reached at Databricks?

According to this video, Databricks' internal Genie deployment now serves 2,100 daily users generating approximately 30,000 queries per day. This scale was reached after the data platform team restructured the semantic layer with Metric Views and established the accuracy measurement framework described in this video.

Full transcript

[00:07] So for the next 20 minutes, I'm going to talk about our journey at Databricks of how we implemented Genie spaces and how we made it successful. Uh I'll quickly introduce myself. My name is Krishna. I'm a senior
[00:19] engineering manager at Databricks. Uh I work on the data platform team. Uh think of us as a customer of Databricks. Uh we actually internally launch Databricks to run internal data analytics. And prior to Databricks, I was at Airbnb
[00:32] and Amazon. Uh each of these places I worked for about 7 years. So my wife says I do have a 7-year itch.
[00:44] Um so let's start with uh with the screenshot on on your left. Uh this is a post from Andy Kofoid, who basically heads Databricks uh global field operations. So he posted on LinkedIn 3 weeks back and he said uh his success story with
[00:57] Genie spaces. So he was asking a question about the performance of Databricks in the LATAM region. And typically this would be a question for a data analyst or a data engineer, right? But instead he went to Genie spaces, asked the question, got
[01:10] the answer back, validated in about 2 minutes. So his cycle of iteration for analytics was extremely quick and he's super impressed. And Ali Ghodsi, who's our CEO, he responded back to him and said this is exactly his
[01:23] his working style now, right? He doesn't actually go to an analyst anymore. Now hold this image for a few minutes because this is not how it started for us at Databricks. Uh we didn't launch Genie and then everything it started
[01:35] working. It was not the case at all. This was what we had 6 months earlier. This is David Gojo, my colleague, who runs sales analytics at Databricks. And this was 9:00 p.m. in the night 6 months
[01:47] back. He was like, "Genie is not accurate, right? What do we do? Please help." Um so this talk is all about how we got from this stage to the to the earlier slide, right? How How did that success happen?
[02:01] As I said, 6 months back it was a coin toss. We may have gotten some answers right through Genie, some not. Right? Uh and again, as I said, the talk is about getting to more than 95% accuracy from 50% coin
[02:14] toss to about 95%. And this 95% is based on an internal benchmark and I'll talk a little bit about that. All right. Quick raise of hands, how many of you have used an LLM for a data question?
[02:26] Yeah, at this stage almost all of us did. The more interesting question is this, how many of you got a hallucinated answer back? Right? And it'll almost always uh say that, "Oh, I was right." Right?
[02:38] "Don't argue with me." Right? Sometimes it does do that. And the final question is rhetorical, but how many of you wish it was better, more accurate, grounded in truth?
[02:51] And this is where Genie Spaces comes into play. Right? Uh a a quick introduction about with about Genie Spaces. Um a Genie Spaces is a product within Databricks. Um so, a data engineer or a data scientist, they might curate a bunch of
[03:04] tables. They might add it to this Genie Space. And they'll add a bunch of context. The table columns, the description, business context, and then they would release it. And then the users of that Genie Space
[03:17] will ask questions in English. They'll get an answer back. They'll get both the SQL back, the result, and also sometimes visualization. So, it really depends on the question you ask, you get an answer back. But again, it's validated and it's only
[03:29] based on the tables that you added to the Genie Space. Right? It doesn't go outside and that's what makes it very effective. Let's walk through a very simple example. Let us say you're running a business like Databricks, usage-based.
[03:42] So, all of you most of you probably know DBUs, right? That is usage. Let us say this table usage events contains our usage, DBUs. We multiply that with skew prices, and we we arrive at consumption, right?
[03:55] That's what we get billed on. Let's say we add these three tables to a Genie space. We then add a bunch of context. This might look like a wall of text, but a lot of this context I just generated through Genie code. I didn't write it
[04:08] myself. So, yes, it might look intimidating, but you just have to nudge Genie code, and it'll actually generate that uh that for you. You'll have to validate it, but a lot of the heavy lifting is done by Genie code.
[04:20] And then finally, you get an interface like this. This is a Genie space. In this case, I called it consumption analytics. You ask a question in English, and it returns the result, right? It's based on on the tables that we specified earlier.
[04:32] All right. Now, let's get into the journey at Databricks. For the motorheads in this group, this analogy will hopefully land well. So, when we started, we were in gear one. A lot of gas, not a lot of acceleration,
[04:45] right? What was the issue? The first issue that we ran into was that of data quality, or table quality. All of us have hundreds and thousands of tables. A lot of them are not documented. Table
[04:57] names are hard to understand. What did we do? We took those exact same tables. We gave it to a Genie space, and we looked for answers. Did we get the right answers?
[05:09] No. It was a coin toss, right? And this is not surprising if even if let's say you were a new hire, and I gave you a question, and if you had no context, you would give a wrong answer, too.
[05:21] This is an example of This is an example of a question that we asked. We asked Genie for the total consumption at Databricks, right? We spoke about usage and consumption earlier. Usage is DBUs, consumption is
[05:33] dollars. And it gave the answer at the top. It basically looked at the It looked at this found this column called usage, named it total usage, and said, "Hey, that's your dollar usage." As clearly not not correct. Usage multiplied by
[05:47] unit price is what gives us consumption. Everybody hallucinated, gave this answer. So, this this wrong answer made us realize that we have to actually invest in core data first.
[06:00] Before you get to the LLM layer, you have to invest in a core data layer. These are basically a set of tables that you trust. Table tables with good names, column names, column descriptions, right? You need this. And our sales team, David
[06:14] Goja, introduced you to earlier. His team spent a lot of time building sales tables for us, the core data. And when we added those tables to the Genie space, the accuracy went up fairly significantly. Right? From 50% to about 65%.
[06:28] It's great. We'll give ourselves maybe a B minus, but still progress. Obviously not enough. But we were at least clearly in gear two. Gaining some momentum, not right there not there yet,
[06:41] but getting there. We had a new set of problems, and here's a great example. We asked Genie for Q1 consumption. I mean, all of us know Q1 is what? Jan through March.
[06:53] But at Databricks, our fiscal year starts in February. There's no way for Genie to know that. Right? So, instead, it gave us answers from Jan through March, and then we were like, man, this is still giving us wrong
[07:05] answers. So, we realized we have to start treating Genie like a new hire. Give it a lot of context. What would you do when a new hire joins your company? Let's say they're a data engineer or data analyst. You would give
[07:17] them queries and tables. You would give them a lot of documentation. You'd give them a lot of business context. How does your business operate? Right? The example that I gave of the quarter will be different for your company. It'll be something else that is very specific to your company.
[07:31] But you have to give the new hire all of that context for them to be able to perform. And the way we did this was through metric views. So, a quick introduction of metric views. Metric views is a Unity catalog
[07:43] object where you define metrics. So, in this case I define two metrics, usage and consumption. As you could see, they are well documented along with the formulas for deriving those those metrics.
[07:55] If you see to the left, you see the sources and the joints. That is basically me telling uh me telling the metric view, get get this get these metrics from these tables and this is how you join them. Right? And then I also give it the
[08:07] dimensions by which you can break it down. There is one thing I want to zoom in on, which is this one. Recall we had an issue with with
[08:19] quarter. That is now predefined within the metric view. So, there is no more guessing. Right? So, you you you basically now have both your table context and your business context packed into a metric
[08:31] view. And you then expose this metric view to a Genie space. Right? So, Genie is now not guessing. It's asking the metric view. Metric view gives it back the context and the query, and then it returns that result.
[08:43] Accuracy got much better. We got from 65 to 85. Probably give us a A minus. Things are looking good. But it's still not enough.
[08:55] We were in gear three. We had momentum, for sure, but we were seeing a new set of problems. And that is duplication. A lot of you will relate to this. You have a metric in your company. You ask 10 different people for the same
[09:08] metric, you get 10 different answers. Here's a very specific example. Usage. Right? We spoke about this usage as DBUs. The usage on the top is what maybe an engineering team might might care about. They look at all usage across
[09:20] Databricks. It doesn't matter whether it's internal or external usage. Like my team's usage get included at the top, where an internal customer of Databricks, right? But finance, which cares about money money hitting the bank, they care about the the definition
[09:33] of usage at the bottom because it excludes internal usage. Right? But again, both of them are valid. But based on which which metric view got added to your Genie space, you might get a very very different answer.
[09:45] Right? So, there is a problem with duplication. Has nothing to do with an LLM. Right? How do you solve this? We actually solve this through a tool called the metric store. A metric store is an internal tool that
[09:57] we built. It allows us to define data pipelines through a YAML config. Uh but more importantly, it actually allows it forces a single definition per metric. So, when we check in this the this YAML
[10:09] file with the metric and the metric definition, if we if it finds another metric already being defined in the in the in in our repo, CI actually fails. CI says, this metric already exists. Here's the owner
[10:21] of that metric. Please go and align on a definition. And we go through a certification process where we figure out what the actual metric should be. You might be realizing a lot of this talk is not about LLMs itself, but it's a lot of that context that you have to
[10:34] that you have to provide. Now, you might be wondering how is metric store how does that relate to metric views and then how does it relate to uh Genie spaces? So, for us, we define our pipelines and metrics in the
[10:46] metric store. As I said, it enforces uniqueness of metrics. That get pushed into metric views through DAGs or Terraform. It's automated. And then metric and then uh Genie spaces then reads metric views.
[10:59] Right? This was great. Accuracy got up to more than 95%. This is where this is This is this is A plus. This is great. But there is one more problem.
[11:15] The problem here is the evolution of your business. Let's say the Genie space performs today, how do you know it's going to perform 6 months down the line? You add a new business line. A new table gets added. A column semantic changes. Like how do you know your Genie space is
[11:28] still going to perform? Right? How do you measure it? And that is when we get into regression testing. Let us say you have a master branch for your for your code. You test it every night. You run all your test cases. You
[11:41] have to do something very similar for Genie, right? You basically write test cases for it and run it every day to make sure it actually runs. So, the way we did that is we actually created a test harness. Now, the harness is a term that is
[11:53] overloaded nowadays, but it's very simply a bunch of Genie questions that we expect from users, the sequel that we expected to write and the result, right? We put that in a file,
[12:05] and we basically call the Genie spaces API every night. We produce the questions. We basically give it the same question. We get the answer back. We compare it against the harness, and we make sure it is accurate. If it
[12:17] is not, the author of the Genie space gets a message to go and fix because it's most likely either a context problem or a column problem and so on. Because otherwise, what will happen is your Genie space is going to get worse
[12:29] over time, right? As your business evolves. Now, let me start summarizing. At the very bottom of the stack, we built core data. Right? These are canonical data sets
[12:41] that we can trust. We layered on top of that business context. Remember with the fiscal example, the fiscal quarter example? That we made it happen through metric views. On top of that, we built a metric store
[12:54] to ensure that we have no duplication of metrics. And finally, do not forget regression testing. You bring this all together, you will hit that 95% accuracy, or at least we did.
[13:09] That is what got us to the message on the top where frantic frantic David Gojo messaged me at 9:00 in the night versus Ali being very satisfied with the product, right? Now, this is just one example of success at Databricks, but our most popular
[13:22] Genie at Databricks gets about uh 2,100 users every day, and about uh 30,000 queries, 20 27,000 queries a day. All right, so it's extremely popular.
[13:34] So, my call to action for you is when you go back to your offices on Monday, find three tables that are your core data sets. Clean them up. Give table comments. Make sure your columns actually do make sense.
[13:47] Find three metrics that are core metrics for your company. Align on a definition. Please specify those regression tests because once you create the Genie space, 6 months down the line it may not perform, right?
[13:59] And then create that Genie space and you see that magic happen. This is my takeaway for for this group. Fix the data Fix the data, not the model. The temptation is let's get to
[14:11] the next model, right? It will help a little bit, but context is far, far more important and more impactful than the model itself. There are a few other talks that my team is doing. If you're interested, please
[14:24] please attend them. I'll flash them on the screen for a bit. Again, thank you for taking the time. Uh really appreciate it. Please provide any feedback you have.
[14:36] I'll be uh available offstage for any questions you have about our journey. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.