Behind the curtain: Key learnings from building Genie

Summary

Genie 1 introduces a unified multi-agent interface on the Databricks Data and AI platform that allows anyone to ask complex questions across their entire data and agent estate without needing to understand the system's underlying architecture.
Rather than relying on generic LLM knowledge, Genie 1 reconstructs business context automatically by building a semantic index over the Databricks platform and deriving authoritative definitions from Unity Catalog and actual usage patterns.
The engineering team shares key lessons on routing, orchestrating, and combining results from specialized Genie agents while balancing low-latency token optimization with the precision required for enterprise analytics.

Behind the curtain: Key learnings from building Genie

As organizations scale, the real challenge isn't just single talk AI — but designing a system like Genie that can coordinate multiple agents, domains, MCP connections, structured data, unstructured data, etc. while keeping the user experience effortless.
We've introduces a unified, multi-agent interface that allows anyone to ask complex questions across their entire data and agent estate and get answers without needing to understand how the system works underneath.
In this video, we’ll pull back the curtain on how Genie was designed and share the best practices we developed to make multi-agent analytics usable, reliable, and scalable.
In this video, we’ll cover:

How to design a unified experience across multiple agents and data domains
How to route, orchestrate, and combine results from different agents
How to balance flexibility with simplicity for business users
Key lessons learned building and deploying multi-agent systems in production

Talk By: Chung Wu, Principal Software Engineer, Databricks ; Yi Liu, Engineer, Databricks ;

Chapters

FAQs

What is Genie 1 and how does it differ from earlier Genie on Databricks?

Genie 1 is a new AI co-worker on the Databricks Data and AI platform that can answer data questions, look at calendars, catch up on communications, and schedule recurring analyses. Unlike earlier Genie, it automatically reconstructs business context—understanding what specific company terms like 'weekly active users' mean—without requiring subject-matter experts to manually curate every agent.

Why do large language models struggle with enterprise data questions out of the box?

Large language models are trained on general text and do not know the specific definitions, metrics, or data structures a company uses internally. Without understanding what business-specific terms mean in a particular organization's context, even a capable LLM cannot reliably answer enterprise data questions on the Databricks Data and AI platform.

How does Genie 1 build business context automatically?

Genie 1 builds a semantic index over the Databricks platform, deriving authoritative metric definitions from Unity Catalog metadata and actual usage patterns. This allows it to resolve conflicting definitions by identifying which source has the most authority, rather than requiring humans to curate every business term manually.

How do Genie Spaces fit into the Genie 1 multi-agent architecture?

Genie Spaces—now known as Genie agents—serve as first-class sub-agents within Genie 1's multi-agent hierarchy. Subject-area experts can still curate a Genie agent around a specific business topic, and Genie 1 routes complex queries to the appropriate specialized agent while combining results into a unified response.

Full transcript

[00:09] Hi everyone. Uh, welcome to Genie1 behind the curtains. My name is Ch. I'm an engineer uh, working on Genie1 and Genie spaces or Genie agents. And >> hi, my name is E. I work with Chan together on Genie.
[00:22] >> Yeah. So we uh announced Genie1 uh general availability earlier today at the uh at the keynote and today right now in this session we're going to uh talk a little bit about how we're
[00:34] approaching thinking about the problem of Genie one and how we built it. All right so you saw the demo earlier today in the keynote but what is Genie1? Uh it is a new agent uh on data bricks.
[00:48] You know, it's your AI co-worker that helps you understand your enterprise and get your work done more easily. Uh it'll be able to do all the things you expect these things to do. You can answer your questions, look at your calendar, help
[01:00] you catch up on Slack, uh schedule recurring data analyses, and so on. But and there's like a lot of these kind of AI co-workers nowadays. So, what is so special about Genie1? Um, well, one
[01:13] thing that sets it apart is that it will, we're really trying to build it so that it will understand your business and your work really deeply. We want to share a little bit about how we're building it and how we're approaching the problem and what we're working on.
[01:28] So nowadays, large language models are getting pretty good. Uh, they're getting better all the time, too. you know, it can get everything right right in one shot, but given a good harness with the right capabilities, uh, it can be pretty
[01:41] great. Building a good harness for Gen1 involves a lot of really nuance trade-offs, and that's what E will be talking about later in this talk. But even with a good harness, these things still suck a lot out of the box in the
[01:54] enterprise context. And that's because they just don't understand your business. They don't know what you mean when you say weekly active users. and no amount of uh model parameters will change that. So to help you do your work, it really needs to understand your
[02:07] work. So we see this as the biggest challenge and blocker for having an effective work agent. So the first problem we need to solve is how do we provide the right context to the agent so that it can do the right things.
[02:22] So we started this journey about two years ago at data bricks by building genie spaces which uh now will uh now known as genie agents. Uh so with genie spaces or genie agents uh these are
[02:36] products where subject area experts will curate an agent around a specific topic uh to be used by business users to ask questions and get answers. Okay. And in order to fill this knowledge gap from
[02:48] between the model and your business, uh these agent authors spend a lot of time writing instructions, adding examples and definitions and uh evaluating agent performance and they do all this work to
[03:00] make sure that the agents understand your business and how your business maps to the uh data sources that you have the sources truth data. Um so it may seem obvious but it's
[03:12] worthwhile to consider what is it about this curation from the author that makes these uh agents so effective. So first uh it is this curation is high recall
[03:25] right the author adds the instructions about everything the agent needs to know and it's also high precision whatever information is given to that agent it is correct and it's appropriate right there's nothing wrong or misleading in
[03:38] all these instructions that the author has written uh for the agent and then finally it is high authority right there's this single source of authority which is the agent author right there's no ambiguity at all. You know, whatever
[03:50] the author says is the truth. So with Genie spaces or Genie agents, we kind of cheat a little bit on this business context problem, right? We have a really motivated author who is there
[04:02] to manually help you fill in this gap. This is great, but it also doesn't always work well, right? It's a lot of work to curate these agents, and you can't build an agent for every possible topic you have in your company. And even
[04:15] curated agents often are missing important contacts and so on. So with Genie1, we wanted to start kind of from the opposite end of the spectrum. We want to build an agent that
[04:28] understands your work out of the box without an author that's doing all this curation up front. But you know, without that curation, how can the agent know how to answer your questions, right? We'd be taking away all these high recall, high precision,
[04:42] high authority instructions from the agent. Now you're back to just the plain vanilla model. What can we do there? Even a simple question like you know what is my revenue last quarter has
[04:54] already challenged you to answer. You know what do I mean by revenue? When does the quarter even start? Which tables and views am I supposed to use? There's like 400 tables with revenue in the name. And how do I use these tables
[05:06] together? You know, how do I join them? how do I aggregate the data and so on. So kind of the primary challenge for genie one is when the user asks a question we need to go and dynamically
[05:18] and automatically reconstruct all this relevant context somehow right on the spot and to replace this you know really high recall high precision high authority curation that the genie agent author used to be providing for us. So
[05:32] let's dig into each of these challenges. So first you know where can we automatically find curation where can we automatically find all this knowledge? Well first place place to start as always unity catalog you know this is
[05:45] where data and domain experts they put their curation and their instructions in the centralized governed way. So this is a really important source of truth but of course it's only a subset of all the truth that that exists in your enterprise.
[05:58] There's a much messier source of truth which is the data bricks platform itself. Right? The platform is a living breathing system where people already do a lot of work with their data. Every time a user interacts with the platform,
[06:11] the knowledge come pouring out of their head through their fingers and into the keyboard and into the system. Right? This happens whenever they are, you know, building dashboards, building data pipelines and notebooks or when you're
[06:23] when they're just talking to agents, right? Every time they do these things, they're explaining their business concepts over and over again to us and that's something that we can leverage. So one trick is we want to harness all
[06:36] this power from all this activity. We want to harness all this knowledge happening through all these activities across the platform and surface this bottom up distributed creation of knowledge in some way that's useful and
[06:48] of course layered with UC governance uh unity catalog governance on top. So let's try to recreate our three conditions high recall high precision and high authority right. So first uh
[07:02] high recall you need to know what you need to know before you can answer the question. So first we need to be able to find that knowledge. Uh there may be a lot of knowledge encoded on the platform but of course we
[07:14] can't if we can't search over them then you know we're lost. So we need to build that semantic search index over everything that might be useful. These might include tables, columns, uh, comments, views, dashboards, notebooks,
[07:27] and also the conversations that you have with your agents. We might enrich that data by generating a bunch of keywords and topics uh based on the content of those assets uh so
[07:39] that these relevant assets can be found by users relevant questions. And then we use some distillation techniques to kind of distill these things down so that the agent every time
[07:51] you ask a question the agent is not going out there and ingesting like millions of tokens just to understand what you're trying to ask. And we need to do all this work of course while somehow preserving uh
[08:04] access control and privacy so that you know you never end up with any knowledge that uh you're not supposed to have. So as we do all this uh expansion and distillation, we need to keep track of
[08:17] the lineage of where we're getting each piece of information so that we can make sure you only have access to that uh distilled knowledge if you have access to the underlying assets.
[08:30] Okay, so now let's say we have this index and we can find a lot of stuff but how do you know if it's any good, right? How do we replicate this high precision that we used to get from expert curation and genie spaces and genie agents? Well,
[08:42] again, many truths are essentially governed by Unity catalog and that's a good source. Uh, but many truths are not, right? Especially if we're trying to automatically uh generate or distill a lot of this knowledge from user
[08:54] activity across the platform. You know, some users are right, some users are wrong. So, how do we tell them apart? Some truths are even kind of uh contextual and subjective, right? Different teams will define weekly
[09:06] active users differently. every campaign will define the definition of a lead differently and so on. So uh and of course these definitions are also changing all the time over time, right?
[09:18] Nothing ever no definition ever stays the same. So we can't just do the simple things of oh let's do a search and rank it by semantic distance and then just show it to the agent and cross our
[09:30] fingers and let the agent pick the best thing, right? We need to somehow uh be intentional about resolving these conflicts. uh and figure only showing the agents the things that are most likely to be correct and relevant.
[09:48] So core to this precision problem is understanding authoritiveness. So this is like who can we trust? What are the assets that are authoritative? So uh where the where can we find
[10:00] authority like where where does authority lie? Well, of course there's unity catalog as always. This is like the centralized governed source of truth. So a lot of authority there, careful team carefully curating the content uh in Unity catalog. But what
[10:13] about all this knowledge that we're sourcing from activity across the platform? How do we derive uh authority from that? Well, um we can't treat all these assets
[10:26] as equal sources of truth, right? Uh we might consider popularity. Hey, a popular thing is more likely to be right, but not all authoritative things are popular either. like you might have a dashboard that only I don't know a few
[10:39] executives look at and it's not popular but it's very authoritative. So we have a few ideas here. You know, one is uh you know that intuition is right. I think that popular assets are more likely to be trustworthy assets.
[10:53] But also popular assets are built more likely to be built by by experts, right? Experts are likely to build trustworthy assets. They are likely to become popular. And also experts are more
[11:05] likely to use trustworthy assets, trustworthy assets because you know they know what's right and they're not going to open some random person's uh dashboard to they're going to be using authoritative dashboards. So some
[11:19] authors are more important than others and some viewers are more important than others also. So from all this activity of users who are creating assets and consuming assets and asking questions
[11:32] around all these topics and all these use cases, we can kind of there's like a graph of all this activity and there are some sources of authoritative sign author sources of authoritative signals
[11:46] like popularity and so on. And we can use an algorithm that is similar to page rank to kind of propagate this authoritiveness around and figure out you know what are the where authority tends to uh lie in this graph and that
[12:00] can give us some definition of what are the most authority the most authoritative uh assets for these topics.
[12:16] Yeah. So all all this work that I'm talking about trying to understand uh your enterprise context is all kind of happening under the banner of geneontology which is our system of governed and discovered knowledge right when you ask a question we want to surface the agent context that is you
[12:29] know high authority uh sorry high recall high precision and high authority so that that's the goal um we're just starting our journey there but we're already seeing some exciting results and
[12:42] we'll have a lot of uh exciting things uh to share in the future. But yeah, I'll hand it over to E. >> Um thanks a lot Chan uh for covering how we surface trustworthy knowledge and
[12:54] also give genie useful context. So once we have that context, the next question is how to use it effectively. We made several design decisions in the harness architecture to support that. So I'll start here with the agent architecture.
[13:07] A simple question. Should they stay flat or should we introduce hierarchy? So a flat architecture keeps like routing simple and agent state centralized. This makes control flow easier and al to
[13:20] understand and debug. It also gives the model access to full context at every step with minimal coordination overhead. The trade-off is of course the context pressure because as task gets more
[13:32] complex two results metadata intermediate reasoning user guidance along the way all of this accumulate in the same context window. Um the other pattern is a hierarchal architecture
[13:45] with sub aents. This lets us isolate skills, tools, prompts and event failures into specialized agents. Each sub aent works with a smaller, more focused uh prompts um uh context and
[14:01] also help the complex task decomposition and improve the overall reasoning success rate. But hierarchy also adds cost here. handoff latency, coordination overhead and possible contact loss
[14:13] across boundaries. These two architectures um are not mutually exclusive as we have already seen in some of the popular agents implementations. Our goal is to build a
[14:25] flexible framework that can scale with task complexity and other factors. This table just like highlights some of the most obvious ones. of course task complexity for a simple question that do
[14:37] not require deep reasoning. We want to answer quickly without the overhead of setting up running sub aents but as the complexity goes up sub aents with isolated contacts will definitely help.
[14:50] Second when we have relevant agents already exist such as genie spaces we want to reuse them as sub aents. This let the customers get a consistent experience from agents they have already spent time building and tuning. Um but
[15:05] overall for default cases we still prefer a flat agent. It is more token efficient with prompt caching especially and also it avoids complex potentially lossy contact passing between sub aents.
[15:18] Recent models are also becoming much more capable on long flat traces which makes this approach increasingly practical. So I also want to double click a little
[15:30] bit on the relationship between Genie 1 and other Genie agents. As we mentioned, when a user already have set up a special purpose agent like Genie space, we want to reuse it whenever applicable. These agents often come with rich
[15:43] instructions and benchmarks. So they can serve as a strong sub aent just inside the genie one. But the opposite direction is equally interesting. Imagine if a user spent some time
[15:55] exploring a new topic with Gen1 chatting, generating queries, visualizations and giving guidance along the way. We want we wanted that you can just with one click convert it convert
[16:07] the session into a new genie agent and you can deploy it for broader use. So this lets others benefit from the context and iteration that has already happened. We think it will make agent
[16:19] authoring much more flexible and make contact reuse a natural part of how new agents are created. So stay tuned. We will be bringing this capability to you very soon.
[16:31] Um next I want to go a little bit deeper into Genie 1's asset search strategy. In open-ended workspace setting finding the right data asset is one of the most critical steps. So to narrow the search
[16:45] quickly, we combine several strategies, exploring asset types in parallel using type specific ranking models, running iterative agentic search and then updates it search focus based on what it
[16:58] sees, what it learns. We also use usage graph to understand asset relationships. When the data is complex or ambiguous, we also use sub aents to drill down or fan out. So while ontology ontology
[17:13] knowledge helps guide both search and dis exploration process um in addition we also improve index quality with LM generated summaries and userdriven embeddings. So together this knowledge
[17:26] uh this technology together uh make asset search an agentic discovery process not just the typical retrieval and beyond search data exploration often
[17:38] needs to examine many sources at once like reading metadata inspecting table details running SQL and comparing solutions in parallel. So to make this
[17:50] efficient, Gen1 maximize parallel two execution. Um but two latencies vary wildly. A SQL execution can take minutes in some cases, right? And reading a
[18:02] table metadata may finish in seconds. If each alarm runs wait for all the two calls, the exploration is gated by the slowest two call. So to avoid this, Genuine uses non-blocking parallelism.
[18:16] It collects results in best effort batches using fast results immediately while slow tools can continue running in the background and then are incorporated when they are ready. So this lets agent
[18:29] keep progressing explore more paths concurrently reduce the end to end latency and makes data exploration feel more responsive. Um next I want to connect back a little bit to our flat agent architecture. For
[18:43] most Gen1 stages, we intentionally prefer a flight flat design because it's faster, simpler for context handling and more token efficient than routing through nested agents. At the same time,
[18:56] Gen1 is a general purpose super agent. So, it needs to connect to many data bricks features, skills, tools, and specialized workflows. So, loading all of this information, all of this
[19:08] knowledge upfront would make the contacts u very bloated. So it's noisy and harder for the LM to reason from. So instead we build a um framework. So we bundle the skills and tools into groups
[19:21] and we we defer load them only until when we need them. This gives us the speed of a flat architecture and also allowing deep specialized capabilities on demand.
[19:37] Um, and also personalization is another important access when Genie1 can make a big difference. In an open-ended workspace, the same question may mean very different things depending on what the user has worked on, what assets they use, what what context they already
[19:50] have. So first we have personalized asset discovery. A recent activity like uh exploring a data dashboard, querying a table or working on in a business
[20:02] domain can help genfer the intent and prioritize the relevant assets. Personal memory is another feature that coming very soon. This can help reuse prior solutions such as a refined query
[20:14] analysis pattern or preferred workflow instead of starting from scratch every time. uh we can further enrich contact through personal MCP connections like your Google Docs, your email, calendar,
[20:26] chat, tickets and other enterprise systems while still respecting all ACL controls. So together these layers make Gen1 more context aware, efficient and align with how each users work.
[20:40] Uh the last critical access I want to mention for Gen1 is anti-h hallucination and grounding. Since Gen1 operates in an open-ended workspace and can reason across many assets, we need to make sure that the final answer is not just fluent
[20:53] but actually grounded in reliable computation and trusted sources. The first principle is always force computation into code not into model's head. Uh for anything that requires aggregation, filtering, joining,
[21:06] numerical reasoning, Genuine generates and execute SQL or Python rather than asking the model to mentally infer the result. This makes the answer reproducible and easier to verify. Second, citations are mandatory.
[21:19] Whenever uh Gen1 presents a conclusion, chart or business insights, it should be directly connected back to the SQL results uh the source tables, documents or other retrieved evidences. So the
[21:32] user should be able to trace where each important claim came from. Um third, Gen1 should always validate instead of assuming. We do not take user input as a correct value by default. Uh and we
[21:45] never guess on the identifier semantics. Uh always verify against actual table or other sources. We also integrate authority as Chan mentioned and corroboration signals. Not all assets
[21:57] are equally trustworthy. Um so we talk about this a certified dashboard garvin table or widely used metric definition may deserve more weight than ad hoc notebooks or rarely used tables.
[22:12] Finally we fence untrusted content so gen1 apply injection resistant sanitization and always match extracted information back against original source uh before using it in reasoning or
[22:24] presentation. So together, these grounding strategies help Genuine produce answers that are not only helpful but reliable and auditable. So to wrap up, Genuine is designed to
[22:37] help users with grounded, trusted knowledge across their entire workspace. We're combining agentic search, flexible exploration, personalization, and strong grounding principles to make that
[22:49] experience to be both powerful and reliable. This is still a very fast evolving journey and we're excited about what's coming next. So thank you all and we look forward to hearing from you about your experience with Gen1.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.