How to Build and Manage the Full Agent Lifecycle

Summary

This video walks through the complete agent lifecycle on the Databricks Data and AI platform, covering how to build modular agents using Agent Bricks, AI Functions, and Genie Code, then orchestrate them with a Supervisor Agent that dynamically routes tasks across tools and MCP servers.
Enterprise agents must pass what presenters call the CFO test: demonstrating both control (governance, permissions, and access) and effectiveness (measurable quality) before any autonomous or customer-facing deployment can be trusted.
The video demonstrates deploying a full multi-agent application on Lakebase with Unity AI governance, showing how to measure quality, trace behavior, and use trace log topic modeling to identify optimization patterns before and after deployment.

How to Build and Manage the Full Agent Lifecycle

Taking agents into production—coordinated, measurable, and grounded in enterprise data—is where most teams hit the wall. Fragmented components, inconsistent evaluation, and brittle orchestration turn promising prototypes into projects that never ship. This hands-on session walks through the full agent lifecycle on Databricks. You'll build modular, production-ready agents using Agent Bricks, AI Functions, and Genie Code—then orchestrate them with a Supervisor Agent that dynamically routes tasks across tools and MCP servers. We'll deploy a complete agentic application on Lakebase, fully governed through Unity AI. You'll see how to measure quality, trace behavior, and build confidence before and after deployment. Through real-world examples, you'll leave with clear patterns to build, evaluate, deploy, and scale intelligent multi-agent applications on Databricks' latest agentic platform.
Talk By: Amber Roberts, Staff Tech Marketing Engineer, Databricks ; Craig Wiley, VP of AI, Databricks ;

Chapters

FAQs

What is the CFO test for AI agents?

The CFO test asks two questions before deploying an agent to production: can you control it (governance, permissions, and access), and does it work (measurable quality). This framework, introduced in this video, helps enterprise teams demonstrate that an agent is safe and effective for autonomous workflows before taking it to production.

What tools does Databricks provide for building production-ready agents?

The Databricks Data and AI platform offers Agent Bricks, AI Functions, and Genie Code as modular building blocks for creating production-ready agents. These components can be orchestrated using a Supervisor Agent that dynamically routes tasks across tools and MCP servers.

How does agent evaluation work on Databricks?

This video demonstrates an agent evaluation framework that measures quality at scale using trace logs and topic modeling to identify optimization patterns. Agents can be evaluated both before and after deployment to build confidence in their behavior and catch regressions early.

What is Lakebase and how is it used for agent deployment?

Lakebase is the Databricks platform layer for deploying agentic applications with full governance through Unity AI. This video shows how to deploy a complete multi-agent application — a demo called Boba Bricks — on Lakebase with end-to-end traceability and governed access controls.

Full transcript

[00:11] Thank you. Uh, no, really excited to have all of you here today. Uh, really excited to to talk through uh, what I think is probably one of the most important topics. You know, we've heard a lot so far at this conference around
[00:24] how to build these agents, how to defend these agents, how to evaluate these agents, etc. But like fundamentally, how do you get from that moment where you have that good idea into that prototype
[00:36] into that that system that can be safely deployed uh in in challenging situations? Uh this is I'm required to show you this. This says that you're not allowed to trade our stock uh based on
[00:49] the things that I say, which is interesting because you're also not allowed to trade our stock at all uh because we're not a public company. So uh but you know anything I say you're not allowed to think of financially about our company. Um so how to build
[01:03] and manage the full agent life cycle. This is what we're hearing from customers right we're hearing I need my agent to access data and internal systems but it has to respect permissions and there's new models and
[01:15] new frameworks and you know I got the demo working but it's harder to get to production myself right now. I live in a world in talking to AI leaders across
[01:28] companies every day that says I think there's only two things you need in order to successfully launch a model into production or launch an agent into production. Only two things. I call it
[01:41] the CFO test, right? And really what this is is if you built an agent and you want to go deploy it into some autonomous workflow or some customerf facing system. Listen, if you built the
[01:54] agent just to help you figure out what you did last year, uh I don't care. Do whatever you want. But if you are going to go build something that the company is going to be relying on, then the CFO
[02:08] or someone in IT risk or someone in financial risk is going to come to you and they're going to ask you two questions. The first question they're going to ask you is, can you control it? Right? what's it going to do if it uh
[02:21] you know what does it have access to? What's the worst that could happen? What are the what are the downsides of this thing? Right? So they're going to ask you can you control it? Which is really a way of saying is it governed? Right?
[02:34] What does it have access to? What can it do? Are those critical critical pieces that that organization needs to understand before we take it to production. The second thing they're going to ask is does it work? Right? So,
[02:48] they're going to ask you, can you control it? Don't take too many pictures yet because there's a hidden third thing. Okay? Like, uh, they're going to ask you, can you control it? And can you
[03:00] can you tell me whether or not it actually works? And when it doesn't work, what happens? And then they're also going to say, "And how much does it cost?" Right? Now, the good Yeah. Now, take the pictures. Yes. Thank you. Uh the good news about the
[03:13] cost side of things is if you went to the keynote this morning, uh not only did you see uh our next presenter on stage, but you also learned a lot about cost management. And if you didn't go to
[03:25] the keynote, it will stream and you can watch it after the conference and see what we're doing for cost management there. Now, uh I I mentioned those two things, but I want to take a moment and look back at
[03:38] 2024, right? We all talk about how the world is changing so fast and you know the tech cycle is faster than the business cycle now and things like this and you know what what what the tech companies
[03:51] and the vendors are focused on is changing every two or three months and all of these kinds of things. This was my slide at data and AI summit in 2024. Our colors back then were yellow and purple instead of this like black and
[04:04] red thing that we're doing. uh accuracy, control, ROI. I think this just says governance, evaluation, and cost,
[04:17] right? Despite the fact that things are changing so fast, the key fundamental pieces of what it's going to take for you to be successful
[04:29] deploying these things has not right. what model you use, what orchestration framework you use, what harness you use, what methodology
[04:41] you use, those are interesting things, but they're not important from a perspective of whether or not I can deploy this thing, right? Whether you choose to use a GPT or a Gemini or a
[04:55] Grock or what have you, it's great. you should use the one that performs best, but that's not what they're going to be worried about when you go to try and p push this into production. Those are really the implementation details. Now, we got to get the details right, but
[05:09] fundamentally, these are the pieces that are going to block your organization or or where your your organization will block you from being able to move forward. So, today everyone is building
[05:21] AI and they're building AI with more or less everything, right? I mean, every system you can build AI with, you know, the clouds will let you build AI, the SAS systems will let you build AI,
[05:33] right? There's AI being built all over the place. Uh, arguably, we're at a place now where most companies I I had a meeting a couple of weeks ago with a the CTO of a large manufacturing company. And I looked at him and I said, "How
[05:46] many agents do you have running at your company?" And he looked at me and he said, "Craig, I don't know." And I said, 'Well, that's not important. What's important is of the agents that you have, if we looked
[06:00] at the top 10 from a traffic perspective, how accurate are they? And he looked at me and he said, "Craig, maybe you didn't understand. I don't even know how many agents I have running." And he actually he actually we
[06:12] we had an hourlong meeting with he and his team. He actually stopped the meeting. He said, "Craig, if it's okay with you, can we stop the meeting and come back next week? I want to wrap my arms around a little bit of this before we continue." Right? That's where the
[06:25] leaders are right now is we have we have spent decades building software and we've built tremendous overhead on top of this software so that we know exactly what we're launching and exactly how it
[06:38] works and exactly how it can fail. Yet we're giving the business users the ability to build agents and maybe deploy them. Right? And so what's happening is agents
[06:51] sprawl. Right? We're seeing agents show up all over the place. This is a good one to take a I mean just crazy little red lines but uh agent sprawl. So this is the problem right is that
[07:03] organizations a year ago if I went and sat down in our CIO forum I would have heard you know our challenge is how do we how do we get this one PC we're excited about to a place where we have
[07:17] something functional. a year later, just a year later, and the conversation is, I don't even know how many agents I've got.
[07:30] So with that agent sprawl, we come back to this. How do we help organizations manage this complexity, right? And so it really comes down to a platform that does three things, right? First, it has
[07:42] agents that know your data, right? So you're not you're not dealing with just some model that your competitor also has access to and a prompt and you trying to
[07:54] guess which data should I put into the a into the agent at this time. I will say, you know, a quick aside, if uh if 2026 or 2025 was the year when everyone talked about building agents and 2026
[08:08] was the year that uh everybody started building more agents than they knew what to do with, my guess is 2027, it's going to be the year when we realize a lot of this is just a search problem, right? A lot of this is looking
[08:20] at your pabyte of of enterprise data and trying to find the 500 kilobytes that would be most useful for the next uh for the next prompt that the agent sends.
[08:32] Right? That's the challenge we're facing is how do we find that most optimally suited piece of context. Right? The second piece of this is full stack developer tools and infrastructure. Right? the when I say hey the hard part
[08:46] is governance and and uh evaluation I'm sure there are folks who are in here who are like you know actually the hard part is getting access to the models in my organization or finding a safe runtime environment to run it or something those
[08:58] those pieces I don't want those to be to be the challenges for you anymore and so a full stack developer experience and then finally unified AI governance right hopefully we've uh we've almost bored
[09:10] you to death with that I I often joke that my job is to be the one who takes agents, which are like the hottest thing to talk about in tech right now, and marry it with governance, which I'm not sure has ever been the hottest thing to
[09:23] talk about in tech, but you know, it's critical all the same. So, let's dive into agents that understand your data and what that means, right? It means having agents that have retrieval techniques and and reasoning
[09:36] capabilities to to be able to travel across your ontology and your semantic. and they can learn that semantic. You'll see that that you know when we launched this ontology capability that we've got.
[09:49] We didn't ask you to write it, right? We had the agents write the first draft of it. It's very likely that they're going to be able to do this a lot better than we are in the fullness of time. The business semantics, right?
[10:01] How do we ensure that we know the definitions of the of the terms your organization uses or your acronyms or what have you and how do we trace those back to the queries and the columns and the the algorithms that help us
[10:14] calculate them. And then lastly, how do we help manage those those strange custom semantics that you have, right? That make your business different even from others in your industry, right? So
[10:27] all of this needs to be done in a trusted platform. And when I say a trusted platform, it's not enough to say I've got the agent running. I, you know, that we all know of the kind of, you know, early in
[10:40] the in the days of, I don't know, 18 months or two years ago when all we knew how to talk about was rag and it was, you know, oh, did you rag it? You know, did you rag this? I ragged these things and all this. You know, we had we had
[10:52] organizations putting together different different chat bots and you know, there was the famous like, hey, we should make an HR chatbot that has all of our HR policies in it, right? So that anyone could ask for the HR policies. And then
[11:05] but then that team, they got greedy and they said, "We don't just want the policies in here. We want it so that if you ask a question like how much vacation time I have, we actually go
[11:17] pull back how much vacation time you have and give it to you. Right? And then they got even further and they said, "And we should do that with pay slips and W2s and other tax forms." Right? But did they have the right
[11:30] plumbing and the right controls? We we talked to more than one company who came to us and said, "Hey, we use vendor X. We built a really cool HR system and guess what? I can ask it and it will give me my W2.
[11:43] And not only will it give me mine, it will give me anyone else in the organizations, >> right? that that is not trusted management, right? Trusted management is being able to manage that trusted
[11:56] relationship all the way through, right? It's all the way from the app stack and the agents you're building down to the tracing and the debugging and and the the automated improvements, right? being
[12:08] able to manage and ensure that not only do I have this system that is functional but this system can manage the not happy pass by the way that story
[12:21] it's a real story and it was it was classic because you would say like how did this happen and they would say well we had the list of questions we wanted we knew it we wanted it to answer one of them was can you give me my W2 and it
[12:33] worked right again one of the challenges was been nonsoftware developers building software, right? Is they're not always thinking around around the edge cases
[12:45] the way those who have who have been trained often think through those pieces, right? We've made it as easy as possible for you to create new agents, right? For you to get up and running quickly and easily with different shapes
[12:57] and sizes of agents and different capabilities with their agents. Welcome to take a picture. You can also screenshot it inside data bricks. It's right there for you every day if you need it. uh and then AI functions,
[13:09] right? The ability to run these AI models and and agents within SQL queries, right? This is one of the the fastest areas of growth that we see is,
[13:23] hey, I have a table of thousands of utterances. I want to know what they're, you know, maybe these are are customer service uh responses or maybe they're reviews or maybe they're they're uh
[13:35] scraping off of Twitter or or something like this. And I want to know, you know, hey, are they are they angry or or happy? Are they what is it about? What's the root cause? All these things. I can
[13:47] just build a prompt. I can hit that prompt against any model or against any agent and go and and uh get those responses for every row of that table. Right? So as we move from agents that
[14:00] know your data, agents that are comfortable having to try and look at that pabyte and find that 500 kilobytes, then we move to the full stack developer and the infrastructure required from that. Right? And so what does that look
[14:13] like? Well, fundamentally, hopefully you've you've uh you've caught on with our four C's, and one of them is choice. And it comes back to this idea that what I care about is that you are operating
[14:27] in a padded playground, right? That you are in a controlled environment where you can't go do things with data that you're not supposed to, where we can track what it is you are doing and all
[14:40] of these things. But where you also have choice, right? You can choose whatever model you want, right? Really exciting to add Grock to today's listing of models available as well as Kimmy. Uh you have access
[14:54] control so we know what exactly you can access and others can access and we can make sure that's that's operating the way we needed to. You can use any framework, right? any of these kind of you know libraries or marginal decisions
[15:07] that you want to pull in we're happy to help support that right I think the I had read somewhere that 85% of the world's agents are built with like two or three orchestration frameworks it's
[15:19] like you know lang chain and uh crew and vanilla python uh we support 22 of them right so we're not we're not messing around like we really are committed to this idea of choice and and make it as
[15:34] easy as possible to orchestrate these systems, not just across the kind of, you know, the LLM and the the the sub agent and what or and the MCP, but but make it as easy as possible to get all
[15:47] the different pieces, right? The meeting transcripts, the the PowerPoint decks. Often when you ask for a piece of data, you know, we all think that like, oh, when you ask Genie for a piece of data, Genie should go run a query. But what if
[16:00] that piece of data is actually in a PowerPoint deck somewhere, right? It was it was part of a board board presentation or something along those lines, right? How do we make sure that we can access and orchestrate across all
[16:12] of it? And then fundamentally, one of the most important and underused pieces evaluation, right? Is this mushroom edible? You know uh the
[16:24] you know we we had a strategy for a long time around this which was if you got an answer like that then what you if you got a wrong answer there was a very simple solution go add a sentence to the prompt that says when asked this question don't say this stupid thing
[16:38] right but that ended up with these prompts that were a mile long and ended up basically being a shadow version of what we were trying to build and what have you right so instead trace logs judges, right? Uh AI assisted judges,
[16:53] right? Hum the ability to drive human feedback. Data Bricks has an award-winning a uh a industry leading according to the one analyst uh organization that has done an evaluation
[17:06] of of uh agentic evaluation systems. You know, we are the leader here. So, and we've open sourced most of this as part of ML flow. Uh my hope is that this is
[17:18] the piece of the stack that gets that we get the most usage of. And the reason I say that is fundamentally this is what differentiates an agent that is good from an agent that is remarkable. One of
[17:31] the companies I get the opportunity to work with is Flow Healthcare. Flow Healthcare is the largest women's healthcare app in the world. And they wanted to build a chat assistant that could answer questions that were
[17:44] medical in nature. I'm not supposed to say a medical assistant. So medical in nature questions, right? uh they had teams of of doctors as their their human
[17:56] experts, you know, going through the not going through the trace logs, but but the engineers could very easily say, "Hey, this set of of questions, ship them off to the subject matter experts to review." That all takes place within
[18:08] the data bricks ecosystem. And by the way, that subject matter expert review capability is a totally separate UI because we're assuming that that person may or may not be someone who wants to actually enter into data bricks, right? So, if you are tasked with building an
[18:22] agent, if you are the one on the hook for building the HR chatbot and you've got the HR team reviewing all the, you know, uh, you don't have to review all the answers. You can push that off to the HR team and have them review all the answers, right? And the evaluations,
[18:35] they run with ML flow, right? This makes comparison easy. This makes root cause tracking easy. My favorite part of it is if you click on one of the little fails, it doesn't just that little fail, the
[18:47] little red square there, you can click on it. We bring up a whole page that tells you why we think it failed and what you should do to go fix it. As someone who built the ML stacks at other
[19:00] clouds, that was the golden moment, right? Because it's one if I say, "Hey, your model's not very good or your agent doesn't work well." There's not a lot that that does for you. But if I say, "Hey, your agent got
[19:13] this question wrong and we think that's because there's information that's incorrect inside your vector database. That makes it much clearer. That gives you those actionable needs to improve the quality of this system." And finally, unified AI governance. This is
[19:27] where I make governance exciting. All right. So, here we are, right? You already saw it. You don't need to take another Well, go for it. Uh this is where we want to get to for those company. This is a good photo
[19:40] side. Yeah. Uh for those companies who want to really protect their entire agent estate, we have a really e clear and
[19:52] easy way to do it. Put it all in UC. And I don't just mean put your agents in UC or put register your agents in UC. I'm saying every single thing that agent touches goes through Unity Catalog. And
[20:04] the reason for that is that now we have we have one surface and we can say hey the agent is operating. The only way to get to the agent is through Unity catalog and the agent is operating and so I can tell you authoritatively every
[20:17] single thing in Unity catalog that this agent has access to. Right? It's no long I'm no longer writing commands for it to go out and hit other SAS systems. What I'm doing is I'm taking those SAS systems MCP servers registering them in
[20:30] Unity catalog. uh just like data we don't make a copy of the MCP server it sits back where it was we just virtualize it behind Unity catalog and now you can access any system right access any data right you can ensure
[20:45] that you know what it's going what every single agent is capable of what every single agent has access to now not only do we talk about what agents have access to but we also can talk about agents
[20:58] working in a on behalf of case Right. So, hey, oftentimes if I'm using that HR chatbot, that HR chatbot is no longer the HR chatbot. It is the HR chatbot on
[21:10] behalf of me, right? And I have very different permissions than the head of HR does, right? And so I should see a very different set of outcomes than the head of HR does, right? Right. And so
[21:22] those kinds of capabilities are just super easy to build into all of this if you have all of the the objects or all of the entities that that agent is going to touch or interact with behind one
[21:34] common governance system, right? Monitoring endpoint health, monitoring your agents, knowing exactly how many agents you've got. And not only are they are they healthy, are they
[21:48] doing what you expect them to be doing, but you know, we actually give you, many of you may know, but we actually give you a trace log of every single time that agent is called. Now, some of the coolest stuff I've seen are customers
[22:01] who start running topic models on top of their trace logs, right? They run topic models to find out what exactly are the questions that we're getting asked. They run those same topic models on top of
[22:13] their evals and say not only am I get here are the questions that I'm getting asked here are the areas where I've done evaluation do those things line up or am I getting questions about things I haven't looked at in evals and then finally running that same topic model
[22:26] over their knowledge stores and saying hey these questions I'm getting do I actually have those topics in my in my context layer or am I getting questions that have nothing to do with
[22:38] the context layer I'm running Right. When we worked with Fox Sports to build uh the Cletus, so Fox Sports, for those of you who don't know, uh that during NFL football games here in the States,
[22:50] they have a dancing robot. Uh and that dancing robot you can talk to on your phone. Uh it's all powered by data bricks. Now, the challenge for Fox Sports in this moment was that if they
[23:03] get something wrong about your favorite team or your favorite player, you know, it takes it takes years to build trust and seconds to lose it. And fans are not rational humans. And man oh man, if you
[23:16] if you tell a fan something wrong about their favorite team, that is they are they are probably getting rid of the app at that point. And so Fox Sports, not only did they need to ensure they were getting correct answers, but they also
[23:29] did not want to become a GPT interface for the world, right? They didn't want you coming and asking, you know, coding questions of their app. So, so you know, guard rails both on the front end and
[23:41] the back end, both to filter out the kinds of questions we don't want and failing gracefully on those, as well as being able to ensure that we were only outputting responses that we'd be proud of. Right. fine grain safety and uh
[23:55] security controls. the ability to apply these safety and security controls not just to LLM but to agents to agent harnesses
[24:07] and to MCP servers right so imagine if I were want didn't want PII from my company exposed and so I said hey I've got a guardrail
[24:19] that I'm going to put in front of my LLM saying hey you should not allow uh PII to go through. I'm going to put the same one between my harness and my LLM. So,
[24:31] if I'm using cloud code between claude code and claude, I'm going to have that same policy layer injected in there and a service policy so that they can't send PII through there. And then lastly, I
[24:43] can apply it to MCP servers. And so I can say, hey, you cannot send that PII through to that MCP to this specific MCP server if I wanted. And I'm saying PII,
[24:55] it could be any policy of your determination that you could apply to these these different systems. So before we had all this sprawl and hopefully this is what it looks like afterwards. Your organization has fleets
[25:08] of agents. Those fleets of agents point to Unity Catalog. Unity catalog on their behalf points to different agents, different models, different MCPs so that you can really ensure that you're always
[25:20] getting the context that you want. So with that, I would like to hand it over to Amber to show you what this all looks like in action.
[25:56] Okay, here. Yes, I just made a little added. Yes, I just added a little bit of a architecture diagram because I think sometimes these demos are a bit easier
[26:09] when you understand what's happening under the hood. And so we're going to talk about a bit about agents, but overall it's going to build into a broader ecosystem.
[26:21] So we have agent bricks, we have genie spaces, we have apps, we have this powered by lakebase. And so just quickly what this demo is going to be and what it isn't going to be. This demo is pretty much the L100 for the data bricks
[26:35] AI platform. If you're looking for Geneontology, if you're looking for omnient, those will be in different sessions. And there's also an L200 version of this. Um, and that one is
[26:47] Elise Gonzalez's talk. All right, let's get into it. And just real quickly here, how many people like use data bricks at least once a week? I know we're a packed room,
[26:59] so yeah. Uh, how many people use uh AI on data bricks? Okay, great. That's awesome. All right, so we're just starting in
[27:12] data bicks apps and welcome to Boba Bicks. We are a premium boba tea company with 30 locations around the world and this is really to help us with uh inventory drink recipes and so we built
[27:24] a lot of capabilities on top of our application. Let's see how good the internet is here. Um, how do I make brown sugar boa for example? And so this the way this demo is going to work is
[27:36] we're kind of dive into all these capabilities. And so if this is our final product, we might see like, okay, so we have our supervisor agent, that's our Boba Bricks HQ, we have our Opspbot,
[27:49] that's our Genie Space, and then we have like, okay, this is how you build the brown sugar boba tea recipe. That looks pretty good. And just a little bit more about like what's really powering this
[28:01] Boba HQ supervisor, our Opspot or Opspot is the knowledge assistant. So that's our recipes, training, food and safety. We have our sales analyst which is our genie space. And we also have an MCP
[28:13] server. Um so if we need any additional information there and maybe I can ask a more a little bit more details. What temp
[28:25] do you brew matcha at?
[28:37] So, it's going pretty fast. I'm going to say that and then it's going to stop loading. All right. So, matcha's brewed at 75 degrees Celsius. Eight grams of matcha. Okay. Now, let's go in and see exactly
[28:49] how this is powered under the hood. All right, first just go into the sidenav. We're in lakehouse. If I wanted to just access my
[29:01] app, we now have a new app page. So this is that Boba bricks that you just saw that's currently running. And then everything that it is currently connected to. If we have a lakebased instant, we have Genie spaces, knowledge
[29:13] assistants, um, MCP servers. And what I'm going to now do now is really like break it down in the platform where and how you can build these capabilities. So I'm just going to click into any of
[29:26] these. Uh okay, that's not important. So this is what I just want to focus on now. This side nav. All right. There's a lot of AI cap AI capabilities kind of thrown
[29:39] in. So we're going to start with genie spaces. We're going to look at the Boba Genie
[29:51] and I can ask it, what is your bestelling drinks or bestselling drink?
[30:06] And so, how many people have used Genie on Data Bricks? All right, so we can just kind of skip through this one a bit. It's it's pretty self-explanatory. It's going to go and look at all your uh structured capability and get your
[30:18] questions answered around that. Let's see if it can give me the answer though. Skip to a faster answer.
[30:31] See, it's always more fun live if things start happening. All right. Awesome. Okay. So, it's able to pull that information for us from those Genie tables. The next part with our playground. Now
[30:45] you guys all pretty much raised your hand for genie spaces. I'm sure you've seen the playground space of where you can select kind of any LLM, any agent brick. Uh so agents that you build within a uh data bricks, external agent
[30:59] capabilities, custom agents, and it's just a nice little interface for you to test things out. And you can open up multiple windows to test against models.
[31:15] and then into our agent capabilities which Craig briefly showed. Like when you click create agent, this is where you get that screen that Craig showed a little bit. Zoomed in. And these are all the agents, but what's really nice is it
[31:27] breaks into the type of capability that you're looking for. So if you're looking for specific function capabilities, this is where pretty much everyone who's wanting to build an agent should start. And that's really just with the data.
[31:39] You want to be able to use uh information extraction, classification, uh document parsing to extract the information you need from your data sets and then kind of build your agents on top of that. our structured capability
[31:53] like our genie spaces and then our unstructured capability like knowledge assistant rack chat bots and then of course custom if you want to use uh link
[32:05] chain for example and use custom frameworks you're able to do so on data bricks as well all right and we will get into AI gateway a little bit later but let's go into our supervisor agent just so we can
[32:18] get a better look okay so This is our boba bricks HQ agent. This has a new interface. So if you've used agent bricks before, you're now noticing um differences in the nav,
[32:31] but now you can easily search for capabilities that you want to add into the space. So if I have, let me see what boba capabilities are here.
[32:43] Oh, if I didn't add like my boba glean, for example, I can add that capability um and then have it available. So now those are searchable and you can add them in along with descriptions and instructions.
[32:56] But what Craig was saying and what I want to get to is that ability to click into your evaluations. So when you run these experiments and if you saw the Genie code demo in the
[33:08] keynote this morning, Genie Code can run all these experiments for you actually be um under the hood. And when you run those, they'll link directly to your experiments here or any agent you run
[33:20] them with, the experiments will link. So on the sidenav, this is where you have all your trace capabilities. So here we're seeing what temperature do you brew your matcha at? We just ask this question. And so that's the live
[33:34] trace. We have sessions. So if I interacted with my application back and forth for a while, saves all those as session capabilities. looking at judges. We'll get back to
[33:46] judges in a second. You could build specific data sets and evaluation runs. So, this is what Craig was showing with an actual eval run here looking at questions and seeing the
[34:00] assessment for evaluation. So, things like correctness, relevance, ground uh retrieval, groundedness. And I'm going to just select one of these questions here that we got wrong.
[34:12] So, How do we prepare matcha uh for the matcha latte? What's the ratio? Can anyone spot before I go to correctness? Just you can just raise your hand. Do you see what it did wrong
[34:24] here? Any tea drinkers? Okay. So, what it got wrong was the conversion of Celsius and Fahrenheit. So, if you're a tea drinker um or familiar with uh Fahrenheit,
[34:39] you might notice that's a pretty cold cup of tea. And so this is something pretty straightforward that like okay this could happen and there's a lot of different ways that this can happen if you know we have uh boa brices all
[34:53] around the world right so we're in locations that use Celsius some that use Fahrenheit so we might have instructions where some are just in Celsius and some are in Fahrenheit and that gets a little bit confusing we might have one data
[35:06] leakage issue we might you know it might have been better like oh instead of us creating like Boba bricks for everything. We should have created regional Boba Brises. And with that, you can really test out a lot of different
[35:19] capabilities. And you can even have Genie Code actually run them for you. So if you already have an agent created, that's a lot of times when I use Genie Code because I'm like, "Hey, I already ran these evals. Take that eval, take this app, take anything I've already
[35:31] built and make it like make it do this instead." Right? Because when you start at zero, that that's always the hardest part. So, if you have an agent capability built, you can replicate it and start editing it. But I'm going to
[35:43] show you something really neat that you can do right now with Genie Code if you have evaluations and especially if you have ones that are failing. Okay, going back to these judges. Right
[35:55] now, I just cleared all my judges. So, I took away and I'm going to build a new judge. Just get rid of old tabs here.
[36:07] Oh, new chat. All right, so we are entering Genie code. I copied and pasted this one because I wasn't sure if I'd be able to just spell
[36:19] Fahrenheit on stage in front of you. Okay. So, essentially, when you're setting up judges, it will kind of run through um
[36:33] Oh, wait. I want it I want it to actually tell you the things that are going on. Let me see. Okay, how do I explain this? So, I think
[36:45] what's really interesting with Genie Code and that a lot of people don't know and don't really talk about is when you're in different parts of the data bricks platform, those are actually specialized agents and they're all running under Genie Code. So when you're
[36:57] in dashboard capabilities and you start asking genie code questions and it's editing things for you, it's because that is a specialized agent for dashboard. And that's why we also have like you know genie code zero ops for
[37:10] data engineering why we have things specific for ML because we have all these specialized capabilities basically per page um and then uh but it's like we put it on genie code like over top. So
[37:22] if you're in a specific part of the platform or you say like oh create an alert it takes you to a different page and then you open Genie code there you can still see all the threads from Genieode but note you're getting specialized capabilities and that's a
[37:35] lot of times what I say when people ask me about well why don't I just use like cloud code uh for all these all these things and it's like well we have these purpose-built um spaces and agents that are connected to those genie code
[37:47] instances and that's why it's so good on data bricks you know but I wouldn't take like genie code and use it somewhere not on data bricks. I don't think that would make a whole lot of sense for your use case. Okay. So, when I click create judges, it
[38:02] pretty much runs through like all the kind of judges I can create. Okay. But I just want it to only create um Celsius and Fahrenheit temperatures.
[38:14] But yeah, these these are the scores that it's going to recommend to you because we have out of the box scores and then we have um ones you can make custom. Is there something specific?
[38:29] We getting temp.
[38:49] We don't have much time. Also like just debugging these things live because it's like I've tried this thing once. Let's see if it does it again.
[39:03] I feel like I confused it some. Oh, I didn't even tell it to make a judge. Let's just see what it comes up with.
[39:15] I'm just going to clear that chat. Let me just try this one more time.
[39:27] Oh, it does it and it's really cool. See me after. Okay. Okay. So, now it's gonna Yeah. run through it and give all the guidelines.
[39:39] But uh we have 30 seconds left and I still wanted to be able to show you lastly which it the last thing which is this is our dashboard capability out of the box in AI gateway and with this
[39:52] capability you have your full overview. So right now I just showed you like okay this is an app that we built on data bricks with data bricks and so all those capabilities are governed but like Craig was saying what if you take something from the outside and bring it in and so
[40:05] that's when you get like the full overview you get performance aspects you get usage you get coding agents this is a real workspace so it's like actually how much um I won't name the team but oh
[40:20] you can see it on there let's see how much they're spending
[40:34] Okay, that's not too bad, Anthony. It's basically a leaderboard that we have if you're top of the if you're top of the charts. All right, so we are out of time. Um, thank you everyone for coming. Uh, and if you have any
[40:46] questions, Craig and I will be over there for a few minutes. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.