Skip to main content

Beyond Simple Q&A: Building an Agent Orchestrator for Enterprise Analytics

Summary

  • The World Bank built a LangGraph-based multi-agent orchestrator on the Databricks Data and AI platform that routes natural-language queries to domain-specific Genie spaces, decomposes multi-step questions, and synthesizes unified answers across structured and unstructured data.
  • The architecture combines metrics-grounded Genie spaces for answer consistency, a RAG-powered document retrieval agent spanning over 10 million unstructured documents, and an auto-visualization agent that surfaces relevant BI reports alongside text responses.
  • This video covers how MLflow tracing, LLM judges, and an online scoring layer provide full observability and hallucination detection, plus a continuous improvement flywheel using evaluation and user feedback.

Beyond Simple Q&A: Building an Agent Orchestrator for Enterprise Analytics

Watch: Beyond Simple Q&A: Building an Agent Orchestrator for Enterprise Analytics
What does it take to go from multiple separate Genie spaces to a single agent that can coordinate across them all? How do you ensure they reference context from your unstructured data, pull up relevant BI reports, and loop in downstream reporting agents? In this breakout session, we’ll share how the World Bank built a multi-agent system that coordinates Genie spaces, document retrieval, and downstream reporting behind a single interface.
What we'll cover:
Orchestrator architecture: Designing a LangGraph-based orchestrator that routes natural-language questions to the right domain Genie, decomposes multi-step queries, and synthesizes unified answers.
Consistency through the metrics layer: Introducing metrics-grounded Genie spaces to reduce answer variability.
Bridging structured and unstructured data: Combining Genie's structured query capabilities with a RAG-powered agent for document retrieval and an auto-visualization agent that can show users plots and surface relevant BI reports.
Talk By: Max Marcussen, AI Engineer, AI FDE, Databricks ; Suresh Kaudi, Data Leader & AI, World Bank ;

Chapters

FAQs

How did the World Bank build a multi-agent system for enterprise analytics?

The World Bank built a LangGraph-based supervisor that classifies user intent, routes questions to the correct domain Genie space, and synthesizes answers across structured and unstructured sources. The system was built in collaboration with Databricks to coordinate multiple Genie spaces behind a single natural-language interface.

What are metric views and why do they matter for Genie spaces?

Metric views are metrics-grounded definitions that enforce deterministic, consistent answers when Genie queries structured data, reducing answer variability across different users or query phrasings. The World Bank introduced them to ensure that domain Genie spaces return the same number for the same question every time.

How does the World Bank handle structured and unstructured data in one agent system?

The orchestrator combines Genie's structured SQL query capabilities with a RAG-powered agent for retrieving information from over 10 million unstructured documents. An additional auto-visualization agent can surface relevant BI reports and generate plots alongside the text responses.

How is the World Bank multi-agent system monitored for hallucinations?

The system uses MLflow tracing for full observability of every agent action and LLM judges for online scoring to detect potential hallucinations in production. A continuous improvement flywheel collects user feedback and evaluation results to retrain and improve agent behavior over time.

Full transcript

[00:08] Good morning everybody. Thanks for joining the session. My name is Suresh Kauldi. I work at the World Bank Group. And with me I have Mark Max who's a Databricks consultant. He's awesome. He was the one who helped us with a lot of implementation
[00:21] with Genie and rest of the stuff. So, you've heard a lot in this conference about Genie, Genie 1, Genie orchestration. It's not a flip or a flip of a button that you can do it. There was a lot of
[00:34] history that went behind this. There was a lot of research that was done in the last last 2 years. And today we have Genie 1. But before that, how we did that journey, how did we unpack things is what we're going to
[00:47] talk about in this session where we'll primarily talk about building building an agent orchestration for enterprise analytics. So, I'm from the World Bank Group. What is World Bank Group? World Bank Group is a multilateral part of a
[01:00] multilateral consortium where we have World Bank, IMF, we have Asian Development Bank, many other banks. It's a consortium of 10. What is the mission? The mission of the World Bank is to
[01:12] eradicate poverty by 2030 on a livable planet. What do you mean by livable planet? We want to do impactful development being resilient, being sustainable, and being inclusive.
[01:25] What do you mean by all this? What why What does it translate to? It translate to providing jobs across developing countries, provide better sanitation, provide better air, and provide overall better quality of life, and be able to
[01:39] be agile to react to fragile situations that happen across the globe. Like COVID being one of them, the war situation, the conflict situation, and all that. So, how did we as a World Bank Group
[01:51] leverage Databricks to answer some of the questions is what we'll be talking about. So, before we we get into the to the demo and we talk about things, we've Over the last 2 days, we've heard
[02:06] a lot of talk about Genie 1, um Genie ontology, ML Ops, a lot of things. In today's session, we'll talk about how we we got there. What are the four C's
[02:20] that they talked about, right? Context, control, cost, compliance, and choice. Compliance is the fifth one, but choice is the fourth one. So, we'll talk about that briefly. So,
[02:34] what was the problem that we were trying to solve at the bank? Right? We had multiple Genie spaces. We had rich information on structured data. How did we get there? I mean, it was not
[02:46] like we had the structured data overnight. First thing is we had a legacy platforms. We had to migrate to Lakehouse, which was Databricks. And we embarked on the journey in 2019. We started modernizing the platform. We
[02:58] wanted to be cloud native. We made sure all our data is on the cloud. Cloud ready. When we say AI ready, what cloud ready is making sure your data is cloud, your data is described, your data is governed, and you have all the lineage
[03:12] associated to that, so that your Genie space can crawl and get to all the information. We had a lot of structured data that we mastered. We had it. We were running analytics. We were running dashboards. Yes, that was giving the
[03:24] answers. But, the world was evolving around us. People were talking about natural language. People were talking about asking questions. We have a Google where we can ask questions. We have ChatGPT, we could ask questions. Why not on structured data?
[03:38] We have unstructured data through that. So, the other we had at bank, we execute projects in developing countries. And we've executed many thousands of projects. We had rich information of documents. We had
[03:52] like 10 10 million documents. Each um month we download close to 3 million documents across the developing countries, across countries which have low bandwidth, across countries which
[04:04] have no proper access to information. So, our goal was to democratize this information, be able to provide this information on fingertips to our clients.
[04:16] And as part of every organization, we have built several dashboards, reports, hundreds of reports, hundreds of dashboards, everywhere, all over the place. And very few people knew how to
[04:29] get there and get that information. Only a typical BI user would know how to navigate, how to get to that thing, how to get to that report, and do that. Right? This was the problem statement that we started off with.
[04:43] Now, the vision. What was the vision as part of this, right? We wanted to do provide our users with a simple interface where they could come, ask questions, not worry about where the
[04:56] data is, where the documents are, and where all the reports are, right? We didn't want them to navigate across the structured data or across unstructured data, or across our our BI repository
[05:09] that we had tens of millions of dashboard. Today, without Genie space, what we would do and how Genie solved our issue. If we Imagine we didn't have a Genie space, I would go to structured data and
[05:21] I want to know what are the list of projects in this country. Okay, I got the list of projects. Then I would take that project, go into a document repository, look at tens of millions of documents, find out where they are. Okay, I mean,
[05:33] it'll take for me a consultant who would come in and do that job specifically to get there. And then with the hope that you'll find the information. Sometimes you don't, sometimes you do, sometimes it takes a
[05:46] lot of time. And you have to read each individual document out of it to understand what happens. Right? So, how did Genie help us with that? With Genie, I could just come in and ask a
[05:59] question. What are the countries that World Bank is What is the In a country, let's say Africa, how many projects is World Bank executing?
[06:11] Number one question. Number two, what is the risks associated to this project? To answer this question, it would take us months and in several cases years
[06:23] because of so much information. With Genie, we indexed all our unstructured data in Google vector search. We put all our data for structured data in Genie and we
[06:35] built an agent on top that would travel through this information. So, today when I ask a question, the Genie goes, figure out it's on the on the unstructured uh structured data, get the list of
[06:47] projects, go to the unstructured content, reads through that specific thing, and pops it up in quick second. This was the information that people were struggling to have for ages.
[06:59] And this was the problem that we solved. And how did we get there? If you look at the the presentation there, the first question is we get how many of our projects that we have. We have like four
[07:11] projects that that came up with the lineage associated to that. It will go and pick the corresponding Genie. And then why you What's the question that you asked? The risk assessment. And
[07:24] then it will the rag will go to the next context thing. And finally, it will provide the visual associated to that. Right? How Now, architecting the solution,
[07:37] how did we do that? Right? How did we unpack this? This was the problem we easier said than done. Right? It was a needle in the haystack. How would people find so much information? There is a lot of information. There's
[07:50] good information, but how do we find it? Now, this is this was our architecture. The first thing is if you go bottom-up, you need guardrails. Before you do the genie, which in the four C's that Ali
[08:02] talked about, the control. Right? You need to have control of row-level security. You need to have control of cost limitations, logging, caching, feedback. Unity catalog was our
[08:15] main driver, where we had all of our data in the Unity catalog. We had vector search index for all our documents in Google using the Google vector search. We had a BI catalog, where we had a
[08:27] plethora of reports, which we built over the time. We had the NCP server. In order to do any of the genie things, you need ontology or you can call it taxonomy, where you define the genie
[08:40] spaces. You domain the genie based on finance, based on sales, and then you have additional sub genies, where you can do uh revenue or uh in an HR term, staff on board. You you
[08:53] can go to uh your recruitment and all those pieces. At the top, you have the web app, where you chat. What happens when when you chat through this, right? I chat I ask a question, what I said, "What are the list of
[09:06] projects I'm executing in in Africa? What is the risk assessment for this project? Why is this negative?" Right? So, that question first it the orchestration using LangGraph
[09:18] supervisor. It the intent classifier is the primary one. The intent classifier takes the questions, understands the domain, decouples it, and then pass it on to the domain
[09:30] router. The domain
[09:42] single question. If it's a single question, yes, it goes to particular genie and picks up. It's a multi-pronged question. It could be two, it could be three, and sometimes it could be four as well. So, the query decomposer comp-
[09:56] decomposes the question into multiple things, does parallel processing, and then synthesizes and provide the output to the response synthesizer. The response synthesizer, you can decode it any which way you want. You can ask a
[10:10] specific prompt to the user. You want to say, uh there is a BI report that you found. You want to see the BI report, or you want to see the output that is coming out from the genie, or you want both. That's how you code it using your um
[10:25] your chat GPT engine, which is very internal to your thing, and that can be coded and configured the way you want. And that's the report synthesizer that finally gets the information and publish it onto you. This is the architecture
[10:38] behind how the genie worked. Just to give a step back, genie one, they talked about everything being in lakehouse. You can do all the processing. But, remember, metrics view has just recently
[10:52] launched. Before we embarked on the metric view, we had to take data from our gold certified layer, which is there, and we had to train the model and train the model and train the model using our
[11:04] dashboards. But, there is always uncertainties, the questions that are being asked. You don't know what kind of question. In the unstructured world, you can hallucinate. You have a sentence that could be written in English in a
[11:17] way the next token can give you a different interpretation. In structured data, you cannot go wrong. What is my age? Who am I? It cannot show I'm 42 once, 50, and 60. Same thing with
[11:32] your sales revenue. Your sales revenue, what is it? It can say you made 10 million next this year, 10 million. And that was happening. That was happening when we did the genie on the gold layer without the metric view. Before the
[11:44] metric view came in, we did that. We tried to train the model, tried to put in a lot of language around it to make sure it it provides the right input, but that was not possible. And we were not ready to
[11:56] launch with that uncertainty because our users were really really scared. Even though it gave 80% of the right results, you cannot have 80% right in structured content. You have to be right on the
[12:08] dot. So, there there was no go on that. So, we didn't have to we didn't go forward with that. So, now >>
[12:20] address this non-deterministic? It was we were lost and we had Databricks on our side, thanks to them, and they put in a lot of effort working with us to get through this. And boom, there came
[12:32] the innovation where they opened up genie space to metric view using AIRBR. So, the first thing we did was we built a metric view to begin with, a small metric view. On top of it, we built the
[12:44] genie space and we were getting deterministic answers. So, we were getting the answers what we wanted. We tested against all of the dashboards, tested against some of the reports. We couldn't do all the testing, but we did
[12:57] give it to our end users to do as a pilot. How do How does this solution work as opposed to what we did previously? So, we did the non-deterministic cleared the non-deterministic using
[13:11] metric view with consistent naming standard, quality thresholds, and shared business glossary.
[13:24] Finally, I said gold layer and metric view standards. We talked about why gold layer was not I think everybody knows what the gold layer is in data in data bricks. When you move the data through the medallion architecture, you have bronze, silver, gold. Gold is the one where you where you process all your
[13:37] users hit the data and you can get the information out, right? So, how when metric view came in, we had to build metric we can build metric view, but we had to build it with a purpose with a way that it scales. Not just building
[13:51] metric view. So, we came up with naming standards. The first thing is table naming standards, column naming standards, the metrics. Because remember when Ali talked about the ontology, when the lang graph goes through and creates
[14:05] the graph, this is what is pivotal. This is the one that drives how your knowledge graph connects to each other. And then we have data contracts. Data contracts is is a contract that we established because this is a metric
[14:18] view. We have questions being asked. We want to make sure that we have a a proper contract to ensure that what data goes into that. We need to have schema definitions,
[14:30] SLAs, and also breaking the change politics. The quality threshold is the next one. We wanted to put completeness targets. How much percentage of data is available or
[14:44] are we reaching our quality threshold? Is the data ready 100% accurate? Do we have the data from yesterday or today? All that thresholds need to be checked in the quality threshold. And finally, business glossary. This is a very
[14:56] important aspect of it. People use Collibra, Avantra, a lot of things. With Unity Catalog, you can dump all of the data into your metric view, and in the
[15:08] metric view, there is an attribute that you can populate your business definition. That will give you the complete picture of your data because your data is incomplete without metadata. What
[15:20] you're doing with by populating that attribute is you're putting the metadata associated to that. Finally, we had to build the What do we need to do in order to configure the
[15:32] genie spaces? We need to build a metric view, create the certificate create the certified views per domain based on however you want it, embed the embed the scores, write business-friendly
[15:44] definitions, register in Unity Catalog. That's the most important the last but the most important thing. And then configure your genie spaces. Create your genie spaces, write instructions, 20-30 verified pairs that you check with it.
[15:57] You make sure that you get consistent answers where you can write it in your instructions, and then finally set the data permissions associated to that.
[16:10] Orchestration, I think my colleague Max is going to talk about this. Max, over to you. >> Okay. So, like Suresh said, I'm Max. I'm an AI FDE here at Databricks. Um and this was a project that we worked on uh a few months ago related to connecting genie spaces to each other.
[16:23] Um per domain, each genie becomes an expert answering questions on its own, um but they don't really connect to each other super well. Um with genie one, uh which I guess a
[16:35] few weeks ago was called genie unified chat, um genie can support genie one can support cross-domain questions, but you really want for more complicated use cases a custom supervisor agent that allows you to fine-tune your own domain logic.
[16:47] So, per domain, you have a certified metric view back genie. You have all your business rules and instructions. Um you have all of your verified Q&A pairs as seed examples and your example queries in your genie space.
[17:00] You have permissions inherited from Unity Catalog, so a person querying a genie space can't query things they're not supposed to. And with all this configuration, you get a deterministic answer every time. So, if I ask a finance question to the
[17:12] finance genie, like what is our commitment in India, the finance genie will answer the same way every time with an expert configured answer. But the problem is that real questions often cross domains. So if I ask a
[17:25] question like, "What is our commitment in India and what are our actions?" Commitments live in one genie space, actions live in another. The user shouldn't have to know which one to ask which part of which question to. So the answer and the other problem is that
[17:39] genies perform best when they're domain specific. You don't want to stuff all of your domains into a single genie because then you just won't get coherent answers. So the answer is a supervisor agent that can route across genies, decompose questions, and stitch together a final
[17:52] response. >> Thanks, Max. >>
[18:06] the session, is genie ontology. Right? How do you define and map your domain, break your subdomains, establish relationship, establish hierarchies,
[18:18] identify shared domains, like in our case it's projects, it's country, it's region, assign domain owners for continuous maintenance. You need to do this because end of the day, why it it
[18:30] matters is this is the one you have to take time to put together this metadata so that your ontology graph when it builds formulates the right picture, diverts it to the
[18:42] right genie, diverts it to the right cross domain questions to answer the questions. Max. >> So the overall pattern for orchestration, um we used a react
[18:56] pattern, which is a common pattern for multi-agent supervision. Um basically you don't want to have to encode at which genies to route to for every single question because there are just too many. You might have dozens of genie spaces that you want your
[19:08] supervisor to cover. You might have supervisors of supervisors. And LLMs are strong enough at this point that if you give them that enterprise domain taxonomy, they can break down questions properly and do the routing themselves. Um so, an agentic orchestrator will
[19:22] decompose your questions, route subqueries to the correct Genies, and it does that through this react pattern. So, the first step in the react pattern is the supervisor will think about what it should do, how should decompose a question to multiple parts. Um so, if I
[19:34] ask a two-part question, it'll break it up into a finance question, and then maybe into an ops question. Then it will act, it will plan out the actions that it should take. It'll go to the first Genie space, call it. Um it'll
[19:46] read the result from that, update its internal state, say I got this information from this Genie. Now it's time for me to route to another one. And then once it does that over and over, once the supervisor LLM decides that it's finally gathered all the
[19:58] information it needs to synthesize a coherent answer, it will do so. So, it's very reactive. It's not really following an exact structured pattern every time. It's doing what is best to answer the user's question, um and it's
[20:11] trained with that enterprise domain taxonomy. Overall design principles, it's modular, so doing this react pattern allows you to add or swap in sub-agents without retraining or rewriting your routing logic. And any tool-shaped agent plugs
[20:24] in. So, for example, an agent bricks with a supervisor API can plug in Genie spaces, knowledge assistants, vector search, etc., Unity catalog tools. It's transparent, so you can inspect every react trace. You can see exactly
[20:37] why the LLM made the decisions it did. Um and when the answer is wrong, you can see which step failed, whether it was a routing error or whether it was a Genie error. It's incremental, so you can ship the supervisor with one or two sub-agents on day one, add additional Genies or
[20:50] knowledge assistants as you go. And then finally, it's governed. So, each Genie and each sub-agent inherits Unity catalog permissions. It does this via on behalf of user authentication. So, the supervisor never bypasses the
[21:02] access controls that you have or that an individual user has. >> I I in the interest of time, um, I think we we'll move a little bit faster. This is the same thing that we spoke about, implementing multi-agent
[21:15] orchestration. One of the questions that I want to answer is you you guys can have a doubt saying with Genie 1, yes, we can do all this. Yes, you can do all that provided your data is all in
[21:28] lakehouse. Not all our data is in one place. Yes, you want to eventually get there, but your data today your structured data can be in in lakehouse, it could be in some other legacy databases, it could be your
[21:40] documents could be in some other repository. You need a supervisor agent if you want to traverse across multiple data hubs. And that's the one while Genie 1 focuses on one thing,
[21:52] this supervisor agent that Max and I are talking about navigates across many multiple data sets that we do. And that's the same architecture, the intent classifier, the domain router, the query decomposer, and
[22:04] the disambiguation that uh we spoke about. All right, additional agents. Why did we do this supervisor agent? We have the data in one place, you can ask Genie,
[22:16] you can give the question, but that was not enough for us. We had a lot of knowledge context agents pulls, why behind the numbers from a project that was key for us. Yes, you get the projects, but what answers? We are doing
[22:29] a project in an X country and the project is at risk because of what? Because of this political environment, because of the geopolitics, because of the environment, because of the current election that are coming. How do we know all that? That's
[22:42] all in the document repository. You get the project list, you get that why from the document repository using your knowledge context agent, and then you stitch them together. Finally, you also have the visualizations and report
[22:54] discovery agent where you've done a lot of reports, a lot of curated reports, curated dashboards which you rely on over and over day to day for your answers. You also can hook it up with a
[23:06] visualization report discovery agent and get the answer right. So, if your user goes and runs your dashboard and publishes on a quarterly basis, you go and run the number, the numbers are consistent. If they're wrong, they're consistently wrong, but at least you
[23:18] have one single truth and not multiple sources of data. We'll embark on the demo now. This is a real demo that we did and Max is going to showcase that on how we used uh Genie
[23:31] Metric View uh and how we got our answers. Over to you, Max. >> Yeah. So, I'm going to start with this. This is a lightweight Data wrangling app that is built off of a multi-agent supervisor that coordinates multiple
[23:43] Genies. So, this is all based on synthetic data, but the Genie spaces themselves are built on top of metric views. I'll show that in a second. Basically, I've asked a question here. For all high-risk active Africa projects, what is the total current
[23:56] budget and total actuals? This is a multi-hop question and requires going to one Genie then to another. So, my supervisor here is decomposing what it has to do. Um so, first it assesses user intent,
[24:09] recites the question, make sure it understands it. User wants to know the total current budget and total actuals for all high-risk active projects in the Africa region. And then it's going to do domain detection. So, this is how it maps Genies to the enterprise domain taxonomy
[24:21] and how it maps the question to that. It says, "This is a multi-domain question. It requires the operations Genie first >>
[24:33] budget Genie to determine the budget information for those projects that it pulls from the first Genie." It's going to map out key entities. So, region Africa, status active, environmental and social risk high, etc.
[24:46] And then it's going to write a plan for what it's going to do. Um it's important to note that this plan is not set in stone. If, as the supervisor is going, it decides that it wants to do something else um
[24:58] or that it needs to do something else to answer the user's question, it can do so. But, the plan currently is to call operations genie to get the list of project IDs for all high-risk active projects, and then to call the budget genie.
[25:10] So, it does that, takes an action to call the operations genie, gets some answers back. Operations genie returned two high-risk active Africa projects. Now, we need to go call the budget genie with the these project IDs. Calls the budget genie, gets some
[25:23] information. Budget genie returns budget and actuals for those two high-risk projects, and now it can synthesize its final answer. So, how we got there, most of this was built and evaluated
[25:36] with LangGraph and with MLflow. So, here at the top of this notebook, I'm just setting an MLflow experiment. My experiment is right here. This is what it looks like. Um, I have traces, judges, etc.
[25:48] And here I'm querying my genies directly. So, my genies exist here in, you know, Unity Catalog or in my Databricks workspace. I ask them instructions about them. And then here they are going over a
[26:00] metric view. So, this is what enforces the answers are the same and consistent every time. I won't go through all three, but here I am using this genie agent class, which comes from the Databricks AI brick
[26:13] library and allows you to query agents via code, or to query genies via code and compose them into a larger agent. So, here I'm just going to query each genie through code. I just have a helper function to invoke that genie agent
[26:26] class on a user question. And here I'm just asking the operations genie, which countries have the most active projects with high environmental and social risk, what sectors are those projects in? It pulls back an answer.
[26:38] Uh, and here I can see the full trace in MLflow of how the genie answered. So, you get this really nice traceability. You get a timeline. You can see how long the genie took. In this case, it looks
[26:50] like it took a little long, 35 seconds. but you can also see how long each step took. So, it took a few seconds to ask the LLM how it should actually structure a query, then it ran the query, executed it. Um
[27:02] it looks like it did that a couple of times, but as part of the final answer, you get all of the traceability of everything Genie has done internally, the query it ran, attachments, conversation ID if you want to ask
[27:14] follow-up questions. So, the supervisor also enables you to ask follow-up questions within the same Genie as part of the same conversation. Doing the same thing for two other Genies, and then we can get to how we structure
[27:26] our supervisor. So, the supervisor is done in LangGraph. Basically, each Genie is connected to the supervisor. It can route to them as it wants to. Um it doesn't necessarily need to go in the exact structured format every time, but
[27:40] it's able to intelligently make choices with that enterprise domain taxonomy about which Genie best answers which question. So, here I have some supervisor prompt text. This is the prompt for my
[27:52] supervisor. Um I'm going to actually register that to the MLflow prompt registry. So, here I'm doing that. It's super straightforward. You can do that with a commit message. And in MLflow in the prompt registry, my
[28:04] prompts all exist right here. So, this is my supervisor prompts. Um team description is encoded in here programmatically when I call the Genie space. So, you don't necessarily need to
[28:17] hardcode all of your definitions ahead of time. This is what makes it modular. You can add new Genies as you want to. Um state, there's a user question here. Genie's already invoked, so it maintains the full state of what it's done already
[28:29] to get to the current answer and the information it has so far. And then available next options for Genie spaces. And this is where I'm encoding that intent domain detection, key entities,
[28:41] and planning steps that uh the supervisor uses to actually decompose a user's question. There are some basic rules here as well, but after that, I'll run my supervisor on a cross-domain question. So, here I'm
[28:54] asking for all high-risk active Africa projects, what is the total current budget and total actuals? It's the same question I asked through the app. And you can see it's kind of following the same steps, but in MLflow, since I set up auto logging, you get to see all
[29:07] of the steps and a timeline here. So, first I ask the supervisor, you can see the supervisor's internal tracing, um and then domain detection, etc. Same thing again. Then after that, it calls the operations
[29:20] genie. You can see how long that took. It'll then call the supervisor again, saying what it actually has, the info it has, um it'll keep going, and then eventually
[29:32] it'll provide that final answer at the bottom here. So, this is what allows you to really debug your agentic workflow. You can go into each step, you can get total token count, cost as well. Looks like this whole question costs about 2 cents,
[29:44] um and then total latency. Here I'm just deploying that Databricks app. I asked the same question in the app already, and then finally I'm going to do some offline evaluation. So, it's
[29:56] important to evaluate your supervisor agent against real-world questions that people will actually ask, um build out an evaluation set over time continuously, and make sure with configured properly LLM judges that it
[30:09] actually answers the questions properly that you want to answer. So, I have three evaluation questions. I have them here inside a evaluation data set.
[30:29] So, each question has, um the actual question I want to ask, some expected expected facts that I want to come through, so I want the genie to always pull back these exact things in its answer for this question, and then expected genies that that going to route to. So, you can actually also evaluate
[30:42] those routing decisions. Once I have my eval set, I'm going to configure some LLM judges. Um so, I have a judge that um is basically assessing correct routing.
[30:54] I have a built-in one for correctness. I have one for relevance as well. And then I also have a custom judge for no fabricated numbers. So, basically what this judge will do is it will go through those traces I showed earlier, make sure
[31:06] that any numbers that the supervisor sites in its answer actually came from a Genie space and it's not hallucinating. And then finally, this is the results of my eval run. It looks like against my three evaluation questions, my
[31:18] supervisor is doing pretty well. So, you can see the judge assessment here. Final answer states that there are 200 projects in total. It knows that this figure is directly grounded in a Genie space. So, this is how I guard against hallucinations.
[31:30] And then I can actually register these judges as online scores as well. So, here I have my online judges. I've registered that no fabricated numbers judge as an online judge. Whenever new traces come in, I will actually be able to see my Genie space
[31:44] assessed against those judges. I'll be able to see how my supervisor is doing against them live for each new question that someone's asking in my app or through a notebook, through an API, however they're talking to my agent.
[31:56] So, this is the question I just asked through the app right here. As soon as it comes through, I get live judge evaluation against it and I can make sure that my agent is continuously doing well, that it's not suddenly drifting, that it's not suddenly giving the wrong answers.
[32:18] >> Thanks, Max. This was a live demo, by the way. This was done on a data set. We did the evals. We constantly provided the instructions to make it get to where we wanted to. This was not a journey like
[32:30] off the bat. We had to really work through it and we had we did it was a journey for 2 years. So, how do we do finally all this is done? How do you do the incremental maturity model? Build, test, single
[32:43] capability at a time. First, take one genie at a time. See if it's hallucinating. Give the instructions. Provide the eval's. Then move on to the multi-domain genies.
[32:56] See how the genie is responding. How's your supervisor diverting the questions? Then in include your knowledge context graph. Then include your BI visualization gap. So, context enrichment, which we put that in my
[33:08] opinion it should go first. After all listening to all the the sessions and the demos here, your context enrichment, which is nothing but your ontology or taxonomy, is what drives your genies. If you don't have
[33:20] your proper taxonomy, your genie is going to give you flawed answers because it won't know how to get to because you have trained it based on your taxonomy. Knowledge augmentation, which is where the rack comes into play, you need to
[33:32] make sure that you hook it. You hook one at a time. Don't try to build a complicated model. Do slowly, incrementally until you get the answers what you want to. And finally, you need to scale enterprise at scale, where you do cross-domain genies.
[33:46] You should be able to match between genies. Do all the answers. Validate it. Put the eval's. Put the judges. Answer the scores. Put the instructions and then get there. Uh we have some time for questions after this slide is done and
[33:58] then we'll open up for any questions. Go ahead, folks. >> So, this is just a basic slide about the evaluation loop that I showed. Um basically, the way to keep your agent good is to constantly review it and make it good. So, whenever a user query comes
[34:11] through, um you can provide feedback live, especially in a genie space, there's a little thumbs up, thumbs down button. It's important to have an SME that constantly reviews those. Make sure your agent is still performing, especially as there's data drift or as
[34:23] new kinds of questions come in. So, if an agent writes the wrong SQL, it's straightforward enough to add the corrected sample Q&A to a genie space as a new example. the outcome is that the same mistake will never occur.
[34:35] If your Genie space is missing data, you can flag that gap to a domain owner. The outcome is that your gold layer coverage expands. Your Genies and your supervisors are able to answer more and more questions. If the Genie provides a confusing answer, which can happen when
[34:48] instructions get too long or too verbose, um you can tighten instructions, provide more business context. The outcome is that clarity improves for everyone, both people directly using a Genie space and people interacting with it through supervisor.
[35:01] And then finally, if there's a new pattern of question, um if people are using the Genie to ask new kinds of things about the data that you didn't anticipate, you can promote that new pattern into your sample library, and then the outcome is that verified coverage grows organically.
[35:13] Um so, constant iteration, constant improvement is the way to make a Genie space and a supervisor performant. >> All right. That's it. Open for any questions.
[35:26] >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.