Behind the Curtain: How We Do Eval in Genie

Summary

The Genie engineering team runs large-scale, real-world benchmarks after every model update to prevent regressions, pairing automated LLM grading with expert human review to keep evaluation datasets reliably grounded.
Genie is designed to refuse ambiguous queries rather than guess, inheriting Unity Catalog governance so users can only access data they are already authorized to see on the Databricks Data and AI platform.
This video details four evaluation workflow types—standard metric, complex SQL, functional, and skills—and explains how production feedback traces feed directly into the model fine-tuning pipeline.

Behind the Curtain: How We Do Eval in Genie

Evaluating AI systems is notoriously challenging—especially when correctness isn’t binary. In this talk, we’ll walk through how the Genie engineering team approaches evaluation at scale, from defining what “good” looks like to building reliable, automated eval pipelines. We’ll cover our mix of offline benchmarks, human-in-the-loop validation, and production feedback loops, along with the tradeoffs we’ve encountered. The goal is to share practical patterns and lessons learned that can help teams move faster while maintaining trust in their models.
Talk By: Shanshan Zheng, Sr Manager, Engineering, Databricks ;

Chapters

FAQs

How does Genie prevent hallucinations in enterprise data analytics?

Genie inherits Unity Catalog governance so it can only access data a user is already authorized to see. When a query is too vague or ambiguous, Genie asks for clarification rather than generating a potentially incorrect answer—treating a confident wrong answer as more dangerous than an honest 'I don't know.'

What evaluation workflow types does the Genie team use?

The Genie team uses four workflow types: standard metric evaluation, complex SQL evaluation, functional evaluation, and skills evaluation that validates tool calls and API responses. Each type maps to real enterprise usage patterns observed in production data from the Databricks Data and AI platform.

How does the Genie team ensure evaluation datasets remain accurate over time?

The team pairs LLM grading with expert human-in-the-loop review to verify that benchmark datasets stay grounded in real business logic. Continuous rigor means every model update or performance improvement triggers a large-scale benchmark run before release, preventing regressions from reaching customers.

How do production traces feed back into Genie model improvement?

Production interactions are harvested as traces and fed into a fine-tuning pipeline, creating a feedback loop between live usage and model improvement. This allows the team to catch regressions and adapt to evolving enterprise query patterns without relying solely on static offline datasets.

Full transcript

[00:11] So um so the promise of Jai is undeniable. It offers rapid insights u streamlined workflows and massive productivity gains. However for our enterprise customers those stakes are
[00:23] much much higher. Um the potential for hallucinations for incorrect SQL or fraud business logic creates a risk of um business crisis. This tension between
[00:36] the speed we want to get for data insights and the accuracy we demand um is real and our engineering mission for the Genie team is to bridge this gap and
[00:48] make sure the speed we are getting on data insights does not come as cost of accuracy. So if we um took a look at how standard JII benchmarks are defined, it's usually
[01:02] measuring possibility. Um for example, does the chatbot sound convincing um creative and articulate? But in enterprise data, possible is probably
[01:14] the most dangerous word in the room. Um a business cannot run on numbers that just look right. So with Genie everything starts with Fes U schema
[01:26] mapping. If an LM misinterpret your meta data it went through a syntax error the database will happily exit SQL and uh maybe render some beautiful charts and
[01:38] this looks perfect for business users but since then the underlying logic is that wrong. That's why we are treating um u genie more than just a consumer um
[01:50] chatbot and genie cannot do guess work. It have to stay within your business domain and it need to um choose a I don't know answer over a potential
[02:02] hallucination. Um but staying within your domain doesn't just uh isn't just about knowing what to say. It's al also about what's allowed to see. So, Genie inherits the Unity catalog governance so
[02:16] that uh if the user don't have access to uh a data cell in the warehouse, Genie won't be able to see it or query it. Now that we've defined our standard on
[02:30] um accuracy, let's look at how do we actually enforce it. First is continuous rigor. So um every time we update model or we um launch some performance
[02:42] improvement we will run uh a large scale real world benchmarks to prevent uh regressions any regressions um potentially in the future and second is
[02:55] a multi-layered auditing uh we pair LM grading together with expert human in the loop to uh make sure our um benchmark data set are grounded. it. Finally, uh adaptive guardrails will um
[03:11] govern every live interactions. Um because Genie inherits governance through Unity catalog and also if a user's prompt is too vague to um or too
[03:23] ambiguous, Genie will ask for clarification instead of trying to rushing through for answer. Achieving this level of rigor is not easy. And uh now we will look at the non-trivial
[03:35] engineering um challenges we run into. So to power the continuous rigor we have to construct a large scale offline evaluation set and to understand the
[03:47] reality at scale we have to um move past generic test and u develop a rigorous approach to engineering. a ground truth
[03:59] and also use LM as judge to scale our evaluation. But we also don't just rely on offline uh evaluation sets. Uh we heavily rely
[04:11] on online production feedbacks to monitor um our real world performance as well. Uh at the same time we have to handle the inherent model complexity um
[04:23] because um LMS are naturally stoastic. Um but enterprise reporting demands determinism. So we have tight bias management to filter out online noises.
[04:37] We have strict confidence levels to validate true pro um progress from the engineering team and we also um handle non-determinisms through system mitigations. Now let's double click into
[04:50] exactly how we tackle those challenges to uh simulate reality at scale. We don't just um guess what users want. We look at the data. So we run extensive
[05:04] case study on Genie production traffic to map out exactly how enterprise users interact with Genie. So for example, we discovered that a real world um usage
[05:17] breaks down into four different workflow types from standard metric reporting root cause investigations to more open-ended data discovery and also
[05:29] multi-turn on strategic conversations that's moving from what happened to why and what next. Similarly, um the traffic analysis allowed us to classify the
[05:41] incoming queries into four different difficulty levels. So the simple queries usually focus on basic aggregations with clear time points and the moderate tasks
[05:53] usually um like introduce abstract business concepts like market segments. Complex queries can require Genie to handle mathematical correlations and
[06:06] maps the output to advanced um visualizations like scatter plots and expert props can be highly open-ended strategic questions that force Genie to search across accounts on trends and
[06:20] schemas all at the same time. So we use this metrics to make sure that our offline benchmark sets are just are not just a random set of questions but a statistically accurate reflection of the
[06:34] real enterprise usage. Once we mapped out the production traffic the next um and critical step for us is to engineering ground truth.
[06:47] First we leverage academic foundations. So we uh selectively integrate uh industry standard benchmarks like spider and bird mapping them to specific
[06:59] functional categories to validate genius uh structural logic. Second we lead in the internal expert curation. So uh we use uh our own
[07:11] internal experts to handcraft test cases based on the messy reality of production. for example, fragmented schemas, non-standard naming conventions or complex business specific logics. So
[07:26] this is to make sure that Genie can also handle real world business um edge cases that academic tasks miss. Third, to scale without internal bias, we partner with third party um vendors.
[07:40] So we use external subject matter experts to execute um blind audits and also provide ground truths of thousands of query pairs at scale
[07:53] to skills evaluation even further. We also rely on LI message judge to handle complex um queries where traditional exact match or numeric metrics would
[08:05] fail. However, we don't usually treat an LM score as a absolute metric uh because uh our internal evaluations have uh shows
[08:18] that on strictly numeric benchmarks, LM judges usually exhibit a nonpositive bias and often um grading um accuracy 10 to 20% higher than our human expert. But
[08:33] the main value of LM judge or LM grading is a relative delta has been consistently has been like very consistent between iterations um over the different versions. So um this makes
[08:47] LM judge very effective for pair wise comparison. So if we have uh um version a a versus version b for performance optimization uh lm judge give us um like
[09:00] help greatly improved our um optimization cycles because it can identify which direction is better. Now that we have all the different pieces in place the core objective
[09:13] becomes building the representative evaluation set and ground truth. So we maintain a regression um guard of over 2,000 golden queries to make sure that a
[09:25] performance or speed optimization we post today does not break a critical revenue report tomorrow. We also include thousands of queries dedicated to edge case stress testing. So these are
[09:38] intentionally ambiguous questions designed to test genie's boundary logic. We use those to verify that um genie knows exactly how and when to ask for
[09:50] clarifications rather than making an unguided guess. Thirdly, we test for multi-turn um conversations. So data analysis is rarely a single um question.
[10:03] It's usually a dialogue. So we run multi-step conversations to make sure Genie successfully maintains contacts over long chat history. Finally uh the evalat isn't a static repository. It
[10:17] function as a iterative loop where we are continually updating eval cases based on the live traffic patterns we observe from production with a representative test set uh the
[10:31] infrastructure run it becomes just as critical as the data itself. So we force a strict production eval parity for our eval platform. Um it executes the exact
[10:43] production um code basis within our evaluation sandbox. By entirely eliminating this testing gap, we make sure our um offline validation is a one one mirror to the actual user
[10:57] experience. Secondly, we maintain a continuous daily cadence for the evaluation. So accuracy is really treated as a living heartbeat. Uh our platform would trigger automated high
[11:10] coverage benchmarks every 24 hours to monitors end to end accuracy. Finally um our platform also act as an automated regression guard. It's not just a
[11:22] passive dashboard, but uh the moment a daily d um ben benchmark drops under our threshold, it will um trigger real-time alerts routed to the engineering team.
[11:34] So we can intercept and fix regressions before they reach a user. Um while that daily eval um give us great value, we have to acknowledge its
[11:46] limit. offline evals can go still overfit and miss important cases that matters in production. Um that's why we also have to look beyond offline evals.
[11:58] So first we learn from operational signals when a user or a customer flags an edge case fail. We deep dive to analyze the root causes identify fix and turns that into a test case so that it
[12:12] can never regress again. Secondly, we also monitor engagement metrics. When a user exports data or share a report, sets our strongest engagement signal
[12:24] that genius response was accurate. So people simply don't share or export broken numbers. Thirdly, we run LM grading and arrow category analysis uh where we reveal errors from logs to
[12:37] understand root causes and nuances. LM grading also flags sudden anomalies drastic format shift or potential semantic drift for us. So finally we
[12:49] track sentiment monitoring uh via um basic UI sums up sums down but uh it it's been great for us to isolate immediate friction points and pulling
[13:01] these raw examples right back into our evaluation set. Now processing all the incoming data um brings us to the final hurdle um as of handling inherent model
[13:13] and pipeline complexity. So the first thing we account for are um data pipeline skills. We have to make sure that our collection methods are not
[13:26] artificially infraing our accuracy numbers or hiding regressions. We also have to isolate external variables that um create random fluctuations in our performance metrics. We see things like
[13:40] the novelty effect where new users behave different from existing users as well as um temporal um variance etc. Because sampling bias and noise can
[13:52] swing any single measurement by as much as 10% we have to measure our performance using a straight confidence level. So this give us the statistical rigor to know for sure that our
[14:04] performance gain is a real improvement and not just a random fluctuation in the data. This leads us to the final hurdle in maintaining consistency and handling non-determinism.
[14:16] So for a business user determinism isn't a future it's a requirement. If a CEO asks for revenue growth for May 2026, they need the exact same answer every time they ask the question, a roughly
[14:30] similar number is the incorrect answer. So the challenge is that LMS are inherently probabilistic. They can be influenced by unrelated contacts or noise um in the promp or they might
[14:43] simply find um two different equally valid looking solutions. Sometimes the variance is just a re reaction to the ambiguity in user questions. To reduce the variance, we use concrete
[14:57] fuel short examples and specific instructions to anchor the model's response spaces so it doesn't um render excuse me. We also keep um genie spaces
[15:09] narrow scoped and the SQL queries very clear for the model to follow. More importantly, if a question is too vague for a deterministic result, Genie is
[15:21] designed to ask for more detail rather than lowering the dice.
[15:34] so that your teams can focus entirely on uncovering the underlying um value of your data, trust the numbers, scale your insights and move forward with total
[15:46] confidence. Thank you. Uh and please let me know if you have any questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.