How We Built Agent Mode for Genie

Summary

The Genie engineering team found that a supervisor-plus-sub-agents architecture caused context loss and wasted tokens, leading them to rebuild around a single agent with composable SQL tools that share context across every analytical step.
Stating variable and metric definitions upfront before running queries improved both business user trust and LLM code quality, cutting down on hallucinations caused by ambiguous schema metadata in the Databricks Data and AI platform.
This video covers four architecture principles for production-grade data agents: start simple, define state upfront, use dynamic checklists for multi-hypothesis exploration, and implement progressive error disclosure for self-healing behavior.

How We Built Agent Mode for Genie

Agentic analytics promises to go beyond answering questions enabling AI to investigate, reason, and synthesize insights across complex data. But building systems that do this reliably is far from simple.
Genie Agent Mode introduces a new approach to conversational analytics, where AI can explore multiple hypotheses, interact with data iteratively, and generate structured explanations. Under the hood, this requires careful orchestration of planning, validation, and context management to ensure results are both accurate and trustworthy.
In this video, we’ll go behind the scenes to explore how Genie Agent Mode works and what it takes to make agentic analytics production-ready:

How Genie Agent Mode plans and executes multi-step analytical workflows
How the system maintains context and evaluates intermediate results
How accuracy and reliability are enforced through validation and safeguards
Key lessons learned building agentic systems for real-world data use cases

Talk By: Alan Lee, Software Engineer, Databricks ; Ryan Ziegler, Software Engineer, Databricks ;

Chapters

FAQs

Why did the Genie team move away from a multi-sub-agent architecture?

The original supervisor-plus-sub-agents design caused individual agents to lose shared context between steps. For example, if one agent discovered a customer did not exist, the next agent had to rediscover this independently, wasting time and tokens while delivering a lower-quality answer compared to a single agent that maintains all context throughout.

What is Principle 1 for building production data agents on Databricks?

Principle 1 is to start simple: when multiple analytical steps have complex relationships, use a single agent rather than multiple sub-agents. A single agent with composable SQL tools can maintain shared context across all steps, eliminating the redundant discovery that plagues sub-agent architectures.

How does stating definitions upfront improve agentic analytics reliability?

Defining variables, metrics, and business terms before executing queries gives the LLM a clear semantic contract to work from. This reduces hallucinations caused by ambiguous column names or schema metadata and increases trust among business users who can see exactly what each term means before results are returned.

What is progressive error disclosure in the context of Genie Agent Mode?

Progressive error disclosure means the agent surfaces and attempts to fix errors incrementally rather than failing silently or all at once. When a SQL query fails, the agent exposes the error, adjusts its approach, and retries—creating a self-healing loop that improves answer quality without requiring human intervention on the Databricks Data and AI platform.

Full transcript

[00:08] All right. Hey everyone. It's 4:30, so I'll go ahead and get started. If you can't hear me, please make some signal. Otherwise, I will assume that people can hear us. So, this is a talk about how we built Genie Spaces agent. I think I'm required
[00:23] to have this. So, I'm going to skip past it because we are not really looking forward. We are kind of going through this talk to share some of the learnings along the way building Genie Spaces and some of
[00:36] the things that we wish we knew when we first started building the Genie Spaces agent. So, we'll cover some of our lessons learned along the way, the architecture that we first had versus what is
[00:49] ultimately shipping in Databricks today, and then some principles that you can use to build your own agents to help analyze and understand data. When we first started building the Genie
[01:02] Spaces agent or Genie agent, we really heavily leaned into this idea of sub-agents. Here's a very early screenshot where we ask Genie, "Is Nick Bruce one of our
[01:16] best customers?" And you can see here that Genie's come up with a concrete plan to first validate the existence of Nick Bruce, and then compare Nick Bruce's
[01:29] performance to that of other customers, and then conduct some analysis to assess Nick Bruce's loyalty tier. There's a few problems in this picture, though. Individual sub-agents are responsible
[01:42] for each step of this process, and these individual sub-agents might not have all of the shared context that we would like them to have.
[01:55] The architecture looks a little something like this. We started with a supervisor agent that talked to these individual SQL execution agents.
[02:07] But there's a problem. Nick Bruce doesn't exist. Now the first agent who is validating his existence might find this. The second agent trying to compare his performance to other customers is going
[02:20] to have to rediscover this and then report a negative result. Same thing with the third agent. This ends up wasting a lot of time and a lot of tokens and we would prefer to get you a
[02:33] higher quality answer faster. So we decided to start from first principles and start with what is the simplest possible data agent
[02:47] we can build and starting from this data agent, how can we improve it into something that is useful for everyday developers to ask and answer data questions on
[02:59] Databricks. >> Yeah, so principle one of building these agents is when multiple steps have complex relationships, just use a single agent. Start simple.
[03:14] So we started from scratch essentially. We got rid of our sub agents. Instead, we created these tools. These are the most important tools our agent has which are just executing SQL
[03:27] and output a response. So our agent just ex- will execute SQL consecutively, see the results of every single SQL statement and then put it all together and give you
[03:39] a natural language business answer. So going back to the Nick Bruce example, let's say Nick Bruce does exist but his real name is Nicholas Bruce at least in
[03:51] our database. Um so, as you can imagine, the agent will first look for a Nick Bruce. Uh but the SQL statement might not return anything. Um so, in our previous architecture, this
[04:04] information would have not been forwarded to our next sub agent, and our next sub agent would have had to somehow work off an empty Nick Bruce result, and try to identify a non-existent
[04:17] customer's revenue, which could could lead to just ridiculous hallucinations that we don't want any of our customers to ever see. But now instead, our agent is able to spend multiple SQL statements
[04:29] on locating a Nick Bruce, and the agent is able to discover, um in this example, that Nick Bruce is actually a Nicholas Bruce, and then only move on to the next, like, find financial metrics after
[04:41] identifying his real name. So, again, one advantage of this architecture is that information is completely shared between SQL
[04:53] statements. Um so, if we spend a couple of SQL statements finding Nick Bruce, identifying what his name is in our database, this information is automatically passed to
[05:05] the next stage of our research. We don't need to worry about, you know, recording that Nick Bruce's name is actually Nicholas Bruce, and passing that to the next sub agent, which we tried. I mean, it could work for
[05:17] uh a basic example like this, but you can imagine for a very complex problem like root cause analysis for why did our revenue fall in this month, that between sub agents, information just starts
[05:29] getting lost, and context lost is very bad, causes hallucinations, um and you just get much worse quality answers. Um one thing to note is that these agents, or these foundation models from
[05:42] Anthropic and OpenAI, are trained to operate in these agent in this agentic loop, where they're able to connect uh results between different tools and draw complex correlations and relationships
[05:55] between them. So, you want to play to the foundation's model the foundation model strength and what they're trained to do. Um yeah, there it's also faster.
[06:08] If you're trying to pass these information between sub agents such as, you know, Nick Bruce's actual name is Nicholas Bruce, that requires asking this LLM agent in addition to generating tool calls to
[06:20] generate instructions to next sub agent, which you know, it it requires output tokens to do this, which wastes your time and wastes your money. >>
[06:34] >> Yeah, another advantage of the single loop agent is that the agent is able to use long-term planning to create the best report not just in terms of
[06:46] quality and honesty, but also in terms of appearance. So, our agent after it finds the data it wishes to present to you, it actually goes back and execute SQL that
[06:58] is best for generating visualizations versus in our previous architecture, visualizations occurred as sort of a side effect that our report writing
[07:10] agent has to incorporate into its report, but now we have one agent that handles everything. So, it you know, conducts research accurately and then um it goes back and execute SQL to
[07:24] present data in the most cohesive way as well. >>
[07:36] know, I'm saying tools are generally better than sub agents at least as a starting point. Well, what's the difference between a tool and a sub-agent? Well, tools generally um are deterministic. Sub-agents rely on
[07:49] LLMs, so you know, for the same question, you might get something different um every time. Uh tools generally receive structured inputs and structured outputs, while uh sub-agents generally receive natural
[08:01] language queries that are unstructured. Sub-agents are generally slower, um while tools are generally pretty fast to execute. Uh and tools generally just conduct one action versus sub-agents
[08:14] um have access to a composition of actions in the form of different tools that it has access to that the main agent may not have access to.
[08:28] Next up, we built a basic agent. It runs some SQL and it generates a report. But, when you get to a really long context window in a model, you start to see
[08:41] performance degradation, even in state-of-the-art models. So, the more SQL that the model is executing, the more comprehensive a report could be, but the more opportunity there is for
[08:54] inconsistency in that report. Naturally, as models are kind of random, one of the SQLs might apply a slightly different definition. And so, we want to solve this problem of
[09:07] getting an agent to be more consistent. This helps build trust and helps our output quality. And what we found most effective to do this is something very simple. It is to have an agent explicitly state
[09:23] the implicit assumptions that you care about for whatever problem you're trying to solve. For us in Genie, that looks like stating the definitions that we're going to apply upfront. So,
[09:36] instead of just having the LLM silently pick a definition for engagement or what it means to receive a shared chat, we ask the LLM to state that definition to
[09:50] the user. This improves the user's trust because the a non-technical user can read these definitions and decide whether or not they agree with them, and it improves quality. A state-of-the-art
[10:03] model, when given a specific definition, is going to be better at generating code that fits that definition. This is something that you probably see in your agentic coding workflows. If you give
[10:16] Claude or Codex a specific, well-structured prompt up front, it can autonomously execute that prompt. And in essence, what we're doing here is exploiting that same capability to have
[10:30] the model itself come up with that specification and improve output quality.
[10:42] >> So, the third principle is just considering multiple angles to answer a question. So, if you imagine a real business question, such as why was there a spike in my revenue? You know, it's actually very hard for a human to answer this question. Well,
[10:56] first you need to validate that there is a spike in your revenue. And if there is a spike, when was it? What is the definition of a spike? Now, to actually answer the question, there's so many different relationships that you should
[11:08] check for, right? Was this spike in revenue just from a certain region? Was it from a certain, say, age demographic in your customers? What if it's both? What if it's only customers
[11:20] from California that are suddenly spending twice as much as usual? Um So, our previous architecture had this, you know, planner agent that orchestrated these small sequel sub
[11:33] agents. The advantage of having a planner agent is that it's able to uh dedicate itself solely on planning. So, how do we bring that planning advantage into our single agent
[11:46] architecture? Uh the answer is actually very simple. We use a checklist. So, while the agent context is clean, we ask the agent to draft uh some initial hypotheses on how best to approach this question. Um for
[12:00] example, uh these for example, these check XY relationships, check XZ relationships might be some um initial guesses the agent uh thinks might might be promising. But, we have to be very
[12:13] flexible when it comes to this checklist, or else we get into the prob- the rigid um the rigid architecture we had previously that we're trying to solve. Uh so, as we're going through executing SQL, we
[12:25] complete these tasks, but we're also giving the agent the flexibility to cancel tasks. Uh for example, um if the data shows that an assumption was incorrect, we should just cancel the
[12:37] task and not spend SQL chasing something pointless. Um as we discover interesting things in our data, we should update our checklist with new tasks. Um if there is a gap in our data, we should mark that
[12:50] and explore that uh and make sure it's not uh there's not a deeper trend to be discovered there first before responding to the human.
[13:03] >> So, this is all great, but inevitably models are going to make mistakes. They're going to have SQL syntax errors. They're going to write a query that simply takes too long because maybe they underestimated how large your table was.
[13:17] Maybe they wrote a join that tried to join two columns of the incorrect type. There's a whole wide range of errors that can happen in production. And to
[13:29] build a production grade agent, one of the most important things to do is to use these errors as an opportunity to steer the agent and provide guidance and
[13:42] provide suggestions for improvement. We don't want to just tell an agent that the query failed. It is better to give agents specific and actionable error
[13:54] messages. So, instead of just saying the query failed, I should tell the agent that the query was canceled due to a timeout and provide a suggestion. In this case,
[14:07] I might say consider selecting less rows. Another perfectly reasonable suggestion might be consider a different type of join operator. And this is very domain specific and one
[14:21] of the best ways to improve an agent's performance in your domain is to really think about how you handle these kind of errors and treat each error that you see in production as a new opportunity to
[14:35] provide guidance to the agent. These error messages are a really kind of useful tool because we don't want to have to account for every single type of
[14:50] obscure SQL syntax in our system prompt. That would make our system prompt really long and that would consume a lot of tokens, which would make our agent really expensive. But
[15:02] by only revealing this kind of expert guidance when the agent actually makes a mistake, we're able to progressively disclose only the instructions that the agent currently needs to achieve its
[15:16] current goal. And this is more token efficient and creates better performance. Another example of this is even in the case of a successful tool call,
[15:29] we can structure the output of that tool call to steer the agent. For example, instead of just showing the agent the first 50 rows of the table, it might be better to show the agent the
[15:43] first 50 rows of the table and explicitly call out that the last 10 rows have been truncated. This will guide the agent toward selecting less rows in in future tool
[15:57] calls so that it can see the full result set. And this is something that is especially important for Genie, where we don't want to give users a 5,000-row table to
[16:10] scroll through in a final report. We want to give them a table that has maybe 50 rows, something they could actually read. And by using our successful tool call outputs
[16:22] as a way to steer the agent, we're able to do this. And this kind of brings us to the current state of Genie, where I've asked this pretty complex
[16:36] question about gold tier customer LTV dropping after Q2. This is really hard. This is far more than asking about Nick Bruce. But Genie's able to answer this question
[16:49] by starting with definitions, executing SQL statements, refining its checklist and planning, executing more SQL statements,
[17:02] and self-repairing from errors before generating a final response. And as a production team, this self-repairing is really awesome because
[17:15] anytime you're deploying an agent, if an error makes it into the final response, some user is eventually going to encounter it. So, by moving to the single agent architecture,
[17:28] we've been able to really reduce our latency and reduce our cost and improve the output quality for Genie users. So, if we were to leave you with kind of
[17:40] four big takeaways for our talk today, they would be to start simple. Start with a single agent with small, composable tools. And
[17:52] as you start to discover failure modes, guide the agent toward sharing thoughts or definitions or concepts that are relevant to those failure modes. From there,
[18:05] as you continue to discover new failure modes, particularly with long context conversations, figure out why your agent is drifting and ask it to state what it needs to
[18:18] stay on track. Often, this comes in the form of a checklist. And then finally, treat every error in production as an opportunity to progressively disclose
[18:30] new information to the agent. This will make your agents self-healing. They can understand errors and make concrete changes during their execution to
[18:43] recover from errors and get an answer to the user. So, we finished about 2 minutes ahead of schedule somehow, but I think
[18:55] we can leave time for a couple questions if anybody has them.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.