An Intro to Building and Scaling Agentic Apps

Summary

Agent Bricks, Databricks Apps, and Lakebase form the core trio for building production-ready agentic applications—providing the intelligence layer, the application surface, and the transactional backend respectively.
This video demonstrates a TechMart customer chatbot built with Unity Catalog Functions as governed tools, LangGraph for session memory management, and Lakebase for persistent agent state across user sessions.
An automated agent ops evaluation loop scores agent responses against quality benchmarks, enabling iterative improvement with human-in-the-loop safeguards before agents are exposed to real users.

An Intro to Building and Scaling Agentic Apps

Agents have promised a lot lately, but most people just have a job they want done and don't want to spend time on endless science experiments. Teams stall on integration, struggle with scale, and wrestle with how to deliver agents to users safely and reliably inside a real application. This intro session shows how to build and scale agentic apps end-to-end on Databricks. You'll see how Agent Bricks lets you create managed, production-ready agents; how Databricks Apps delivers them to users with governance and observability built in; and how Lakebase powers the transactional backend so your app can act — not just answer. You'll walk away with a clear blueprint for turning your first agent into a production app your whole company can use, and the confidence to move from prototype to scaled deployment without rebuilding your stack.
Talk By: Robert Mosley, AI Product Specialist, Databricks ; Tim Lortz, Lead AI Product Specialist, Databricks ;

Chapters

FAQs

What is the difference between building an agent locally and shipping it to production?

Building an agent locally in a notebook is easy, but production requires the agent to interact with systems of record, persist history across sessions, enforce permissions on behalf of users, and provide audit trails and observability. This video addresses that gap using Agent Bricks, Databricks Apps, and Lakebase as the core production stack.

What does each component of the agentic app trio do?

Agent Bricks covers the intelligence and context layer—including LLMs, MCP tools, and UC Functions; Databricks Apps provides the consumption surface where users interact with the agent in daily workflows; and Lakebase is the transactional database backing both the application state and the agent's own memory. Together they form a complete production stack.

When should teams use Agent Bricks versus custom code with LangGraph?

This video presents Agent Bricks as the preferred path for managed, production-ready agents that need governance and observability built in, while custom LangGraph code is appropriate when teams need fine-grained control over agent routing, session memory, or specialized orchestration patterns. The demo uses LangGraph for session memory management alongside Agent Bricks.

How does the agent ops evaluation loop work in the TechMart demo?

The agent ops loop automatically scores agent responses against predefined quality benchmarks after each iteration, flagging responses that fall below acceptable thresholds for review before they reach production users. This continuous evaluation cycle allows teams to improve agent quality systematically rather than relying on manual spot-checks.

Full transcript

[00:08] There we go. Okay. Great turnout. It's Wednesday afternoon of Summit Week. Energy is still high. Super excited to have these next 37 minutes with you all. Yes, the obligatory note before we start about
[00:20] we don't speak officially for product. Um And on with the show. So, does this resonate with anybody? Right? It's getting easier and easier to build agents, right? You can build it locally, play with it locally, it works
[00:33] well for you. But how do you ship it so that it's usable for the rest of your team? Has anybody struggled with this before? Right? And how do you embed it inside an application where it's not just a simple chat interface, right? That's where we
[00:45] want to go. So, we will talk about architectural patterns with a concrete example within the next 30 minutes. Right? So, first we'll talk about the architectural concepts. We'll talk about the basic primitives that you use to
[00:58] build these agentic apps. And then Robert here is going to actually do it for you live. Okay. So, we want to pivot from this world again where you're building a a basic Q&A bot, maybe in a notebook, maybe in
[01:11] your terminal, um which is super useful and I don't want to to to denigrate the idea of building an agent for yourself. But what if you need to make it interact with systems of record and take action, right? What if you need to have an agent
[01:24] that persists, has history across sessions, that isn't just for you, but has to actually enforce permissions on behalf of others? Can you do that? Do you have audit trails? Do you have observability? Do you have tracing? Um
[01:37] and can you deploy it in a way that's reliable? Those are the things that we're going to work towards. All right. So, our trio of stars to make this happen. One, Agent Bricks. Now, when I say Agent Bricks, I really mean
[01:49] the whole suite of agent building tools inside of Databricks. So, this includes things like the the building blocks like LLM's, the MCP's, the tools. Um so we call this like the intelligence
[02:01] and the context layer, right? Databricks apps uh is our surface for actually well for apps obviously. Uh but this is where the consumption happens. Where you actually point your users to come uh
[02:14] invoke the agent and and put it inside their daily workflows. And Lakehouse is a a quiet star but it's powerful because Lakehouse um you know, if you build an application using this a database underneath it,
[02:27] right? Lakehouse is that layer for the database of the application but it's also system of record for the agent itself in terms of uh storing its memory uh and and context.
[02:39] Um underlying these we won't focus on these as much in this talk but Unity AI Gateway is how you get governed access to the resources, right? LLM's, MCP's, etc.
[02:51] Uh super important to run it through there so that you have all the benefits of enterprise grade governance. And then MLflow uh will be implicit. Actually Robert will show you a bit of a deep dive on MLflow. Uh but MLflow is what powers the
[03:04] observability and evaluation. Okay, want to talk about tools. We break it down into three basic categories. We're trying to expose everything via MCP, right? That's been
[03:17] the product shift over the past year and a half or so is to make MCP the standard tool layer. Um in this demo you'll be seeing some of the managed Databricks MCP servers. So if you're using
[03:29] um Genie, if you're using AI search, if you're writing uh functions in Unity Catalog like utilities for your agent, those are all automatically uh de- uh provisioned as MCP servers. You don't
[03:41] have to do anything for it. Um those may not be everything you need though. So there are other options too, right? Databricks may not be the only place where you have the context for your agent. that might be sitting in
[03:53] APIs scattered throughout your enterprise. Maybe you have subscriptions to other products that have like like Confluence or Glean or or Slack where you need to interact. They actually surface those APIs now as
[04:07] MCT servers and we can very easily integrate them into Databricks for more holistic workflows. So those are other options. We will not be showing the the custom and external routes today, but they are very popular.
[04:23] Okay. So Databricks apps, we're not going to go deep on this today, but the point is that apps solved what was for a long time kind of a gap in the Databricks stack, which is like yeah, we can build data pipelines, we can deploy models, but there's not really that surface where users come to
[04:35] interact with a a polished application, right? Databricks apps from the time we launched it was one of our fastest growing products ever, right? Now it's in GA and it's building on all kinds of extra goodies in the app stack. So this is where we're going to
[04:48] house the agent, right? It handles both the user surface as well as the APIs to invoke the agent under the hood. It's It's how you package the agent.
[05:00] Okay, I mentioned Lake Base. This is kind of the the the quiet part of the trio. Um but this is it's managed Postgres. Postgres is a standard that every enterprise knows how to work with. All right, so the benefits here of Lake Base compared to like rolling your own,
[05:14] you've probably heard about this in the keynotes, but a lot of it has to do with auto scaling, right? So scale to zero. Um the ability to branch. So for agents that need to um you know, try to make improvements to or
[05:27] sorry, make improvements to the agent, right? Being Being able to branch on the on your data model super important so you don't break what's in production. And also the fact that it's Unity Catalog native with zero copy makes it a
[05:40] no-brainer. So that's what we'll be using under the hood for the context layer management. One thing to call out cuz a lot of you might be wondering, "Wait a second, you're going to show me some agents in Databricks, but can't Genie do all
[05:53] this?" Um so, I want to be careful here to call out that much of what uh much of agent building on Databricks can be done through Genie. And that's probably like the best place to start most of the time.
[06:05] We're going to show you a different path today for custom agents, and there's some nuanced reasons why. Um for one thing, you might have developers who have standards they've already adopted for agent building,
[06:17] right? They have you know, if they're if they're big fans of LangGraph or some other open-source framework, we want to make sure that works because you might want more control, and they might already have existing agents they want to bring in, right? So,
[06:29] Genie can't quite do all that. So, you might want to bring your your uh your other frameworks in. Um two is that sometimes you want to steer agent behavior a little more, give it very specific actions it can take.
[06:41] And again, with custom codes, you have full control over exactly which tools are exposed to the agent and how they're used. Um So, long story short, it's not an either-or. It can often be a
[06:53] better-together because you can use Genie to expose it via MCP. So, if you have a very complex workflow uh that you want to control carefully, you can build a custom agent that invokes Genie under the hood.
[07:06] All right. So, there, we'll put it in one stack working from the user all the way down to your data. I'll leave this up if you want to take a picture, but we've already covered everything on here.
[07:19] All right. So, this week uh there's a customer presentation from TC TBC Bank that puts a exclamation point on sort of why we're building this stack. So, it's a bank,
[07:31] right? They're doing uh anti-money laundering, really important job. So, they've done a lot of classical ML, uh but they needed to put it inside of a more cohesive workflow, and actually allow an agent to help um steer that so
[07:44] it's not always a human having to intervene. So they used exactly this stack uh to deploy their agent. And there was a there's a blog on that this week as well as a presentation here at Data and AI Summit. Um we use it internally as well. Like
[07:57] we've rolled out so many agent-powered apps in Databricks they've actually replaced some of our legacy SAS. SAS isn't dead, right? But you can replace some things with these uh agent-powered apps on Databricks.
[08:10] All right. And with that, I'm going to hand it off to Robert to show you how it's done. >> All right. Okay, good. My mic's good. So yes, I get to show you guys how to actually take action and and do all
[08:23] this. So uh in here we're going to have a a fictional company called TechMart. Uh what they want to do is they want to build a support agent. And in theory, I'm going to deliver it to you guys in 16 minutes, right? Um so the they want
[08:36] to have a a chatbot. It's going to be a customer-facing chatbot. Uh it's going to be very limited in what it can do because it it it does things like ask them for their email or their order number to verify who they are. But you
[08:49] know, you don't want to expose a lot of PI information or something through that uh because of the risks that would be there uh you also want to be able to make uh let people ask questions about product availability or or product
[09:01] questions, things like that. So you're going to make it a accessible to like product details and so on. So uh we have four four tables, four data sets. As usual, everything starts
[09:14] with data, right? Um and so we have product information um that that gives you all the kinds of stuff that's there. We have orders and and so on and so forth that that goes on there. But just like what he was talking
[09:28] about, we could attach Genie to all four of these tables and then allow like just generic text to SQL against it. Um and while text to SQL's powerful, it does come with trade-offs. Once again, we're planning on making this customer facing.
[09:40] We don't want people to start asking about all sorts of information uh and have it just to go and explore all the data uh and and expose us to a whole bunch of risk. So, we want to narrow that window. Um text to sequel also has additional
[09:52] latency and cost concerns, right? That come with it. So, instead what we're going to do is we're going to build UC functions. And so, this will be our way to build like very narrow tools that have a a specific pro or specific inputs
[10:07] with specific actions that retrieve specific sets of data. So, we're very much limiting the type of information that customers will be able to access. So, these become available automatically through our MCP servers. Now, there are
[10:20] other ways to do this, right? Uh our vector search or AI search product has managed MCP server on top of it. So, you can give a little bit more exposure to general product lookup. Uh you can also build custom MCP servers where you're
[10:33] writing your own Python code to go and do something else or register an external MCP server if you want. For our purpose here today, we have all the data. We're going to wrap it in the UC function and then use the MCP servers on
[10:45] top of that. So, we have our tool sets. And then next, we need to build the agent harness, right? So, we're going to use LangGraph here. We're going to pair it with these MCP servers. We're also
[10:57] going to use LakeBase for agent memory. Um one of the things that's really great about LakeBase is because it's Postgres, most of these agent harnesses have tie-ins for memory into Postgres. So, LangGraph for instance has Postgres
[11:10] saver built right in. You're able to tie that directly into LakeBase. And automatically now, we have uh memory and things like that long term. So, you can uh view all the conversations, you can retrieve sessions and and bring that up,
[11:24] right? Um and then we're going to pair it with MLFlow to be able to log all the traces, to be able to evaluate the the agent, see how it's doing, create our eval data sets, and then improve the agent and use that to measure it. All
[11:37] built on top of a Databricks app for delivery to our our end customers. Um one thing I will note, you know, we are using Vibe coding to build all this. This is all actually very easy to build
[11:50] out with Vibe coding. Uh we have the AI dev kit. That's what it's talking about there. So, if you haven't explored the Databricks AI dev kit yet, I absolutely would. Uh it just Google it, find the
[12:03] the GitHub repo for the AI dev kit. I think we had a a URL or a link at the end. Um and then just ask your coding agent to install it. And it has a whole bunch of MCP servers and skills that
[12:15] teach you how to do a wide plethora of things across Databricks, but including all the steps that we're going to do here. Um I'm also going to share all the code with you guys that we used to build
[12:27] this demo. It's actually a workshop that we do with our customers. And so, you'll be able to pull that that thing down. It has Claude MD files along with it that'll walk you step-by-step through everything that we're doing today. And
[12:39] you can also take it and customize it for your own use case. So, a lot of ways for you guys get accelerated off of this.
[12:51] So, after we've built it, chances are everything you build, every agent you build, you can already think about the weaknesses, right? What are the types of questions that this is going to fail on? Um so, these are three questions that we already know that the agent might struggle with. So, one, our product
[13:04] catalog has discontinued items in it. So, will the model say that or will the agent say that something's available when it's actually discontinued? Um and then also warranty information. Our data
[13:16] set, we know includes things in the in the product info about manufacturer warranties, but that's different than our warranty that we want to provide. And so, if somebody asks a question about our warranty, will the agent
[13:28] accidentally retrieve that information and and give them the wrong answer? So that's that's very possible. And then lastly, sometimes these agents or these models try to be super friendly with the end customer. And so if somebody tries to
[13:42] ask it to break a policy because I'm a long-standing customer or something or other, will the agent violate that? How can we try to ensure that or catch that, right? So what we're going to do
[13:54] to sort of accelerate this and and the code base does this as well. Instead of having somebody sit there and ask all these questions over and over again and have to brainstorm them, we're actually going to use a new feature called conversation simulator that's built into
[14:08] MLflow. I can give it test case or concerns that I have and I can tell it how many conversations to spin up and it will go through and explore the data set and and the agent and it'll generate
[14:20] synthetically its own conversations and then talk to the agent and get responses back from the agent. And so it'll build things out for me so I have a bunch of traces that I can use for evaluation purposes. You know, you can either send
[14:34] it around for testing and collect traces that way or you can do something like this which simulates a whole bunch of testing and you can specify what kind of test you want it to perform, right? So we did this, we built out an eval data
[14:47] set using these conversation simulations. Next is to build out judges to evaluate those simulations, right? So once again, all through cloud code, able to describe
[15:01] you can do this with Genie code as well. You can describe your concerns and say, "Hey, please go out and create my judges inside the experiment that's tied to this agent." And it'll go and it'll
[15:13] create these LLM judges for you and register them into the system. So now whenever you go into MLflow, you'll see judges that are curated for this, right? And then we can run an eval using that
[15:27] data set and running the the judges against that data set and that agent. So, it'll run it against the agent, score it, and give us our results, right? So, based on how it's
[15:39] performing, we can know what we need to change. And then lastly, what we're going to do is we're going to modify the system prompt. This is actually sort of an older version of the prompt. You'll see in there there's actually eight versions
[15:52] that that I went through, but it can continue you can continue to loop through, refine things. You know, this might be the the worst way to do it, but just basically building out a better prompt, you know, prompt engineering our way through it. But then
[16:04] you can also do other things as far as tweaking which model you use, tweaking the tools and and things like that to try to refine your process. But this was the fastest way to get there. You'll see it's very successful, too.
[16:17] And then afterwards, you can run the exact same evaluations. And so, as you continue to iterate, as you optimize your prompt, you're able to score and get there. We actually have processes inside MLflow that do exactly this. You
[16:29] can provide an ML eval data set. You can provide a prompt in our prompt registry, and it will iterate, improve the prompt, run it against the eval set, improve the prompt, run it against the eval data set
[16:41] until it tells you, "Okay, I've optimized for your problem." Oop. So, that is our agent ops loop, right? When we talk about agent ops, sometimes that can feel mysterious or ethereal
[16:56] ephemeral, but this is what it is, right? So, we're going to explore our data, understand what our data is, we're going to build our agent, we're going to chat with it or simulate conversations with it. We're going to build out LLM judges.
[17:09] We're going to measure based on our eval data set. And then we're going to fix it and adjust and and loop right through, right? And continue to measure until we have a an agent that meets our requirements.
[17:22] >>
[17:36] that we have these tables out there. We got products, we got policies. Uh we have order information. We have product docs. And then from these product documents,
[17:48] we actually created our vector search or AI search index. So, I'm not sure if you're aware of how our AI search index works, but when you have a base table that has information like this, so you
[18:01] see it has my product information along with an index doc that's sort of composed a lot of this together. I can actually just come over here and click create. And I can say, "Hey, create an AI search index."
[18:15] And then I just come through here and I fill out this information. What embedding model do I want to use? What column do I want to index? So on and so forth with a few options. And boom, it generates my my index for me. And it also automatically builds out a pipeline
[18:29] that Oh, head's going down. Uh that that refreshes it for you. So, once you have your index, >>
[18:43] can click it to refresh it whenever you want. Um and you can actually test and and query it whenever you would like just to see how it's going to perform, right? So, uh this becomes your your
[18:56] search to be able to find the products based on different terms that are being used, right? So, if I ask about laptop availability, I can search from for laptops from here. Um beyond that, we're going to use this
[19:08] Lake Base instance. Once again, easily created through APIs, through MCP servers, and the skills that we have. So, just in the background, it goes and it generates uh this this lake-based
[19:20] instance. So, the way this code works, because it's a a workshop, it actually created uh a a table schema for me. Um and then it created all the tables
[19:34] that you would need for LangGraph to be able to manage all your all your session memory, right? Um and then so we can come in here and we can see all those sessions over time.
[19:48] And so, here are all the all the conversation checkpoints and things throughout. So, if you're familiar with LangGraph, if you're familiar how LangGraph memory works, these are all those tables that are used in the background to store all your conversations, right?
[20:00] Um and then lastly, we have apps, right? So, we also we actually have a whole bunch of app templates that are available. If you create a new agent through the UI, you can choose from them
[20:12] there. All those templates actually sit in a GitHub repo. And once again, AI DevKit knows how to navigate to them and and pull them. So, there's a LangGraph advanced agent template that's out there.
[20:24] Easy to to pull over and then modify and get it deployed literally in a few minutes, right? So, having all that all those resources available gets you from zero to 60 really quick uh to be able to
[20:36] work with it. But this is deployed, you can see that it's running. That's the app. This is the source code. If I go into the source code, one of the ways in which I modified this is it you'll see here this is the UI
[20:49] code. This is the agent server code. It's part of the template there. But what I did for modification is I added a configuration file in here where I'm able to give it a prompt alias. So, it
[21:01] makes it easy for me to switch back and forth. Um you can also change the aliases that are on your on your prompts. And we'll get into MLflow and how you can do the prompt registry, prompt versions, and then assign aliases to them. But this is what
[21:14] I did so I could easily toggle back and forth between different prompt versions to be able to to do this for you guys, right? So, you can you can make these kinds of modifications. Once again, I didn't I didn't write this code myself.
[21:27] All through Genie code, all through Claude code. And then lastly, I have the app that it's spun up. And so, this is the final conversation app. It's it's somewhat basic, right? I styled it with Databricks colors. Um
[21:41] it's easy to just say, "Hey, reference a URL and say use this pattern." And it'll go and grab that website. Um and it'll it'll use that as a template as it spins something out, right? So, coming in here, uh gave a a few
[21:55] sample questions. So, what laptops do you carry? Might have to wake some things up behind the scenes, but uh it's going to go out. The agent is going to use those tools that are available to it. It's going to collect it and bring it back to us.
[22:08] And this is where we get ourselves in trouble, right? Cuz it's really easy to get here. I'm sure a lot of you have built an agent, built a simple UI, um maybe shown it to a few people internally, and they go, "Great. Now
[22:20] let's get it out in prod." Or "Great. I'm scared what happens if a customer gets a wrong answer, right?" As I said, there are those those issues that you know are in there behind the scenes. And it's what's worse is when it's not
[22:33] consistent. So, you ask a question, sometimes it gets it right, and sometimes it makes a mistake, right? So, that's actually where this whole evaluation process becomes super valuable.
[22:45] So, I'm going to step into MLflow here and show you how it works. So, by default, whenever you build an agentic app in Databricks apps, it ties into an experiment behind the scenes. If you try to create one in manually, it'll
[22:58] prompt you for an experiment ID and give you an option to create a new one. If you're building it externally, all the templates have an experiment ID in the environment variables. So, it it looks for you to collect traces right
[23:11] from the get-go. So, these traces are landing right here inside this experiment. So, I have all the conversations that are coming from my agent. So, if I refresh this,
[23:28] live refresh. Hopefully, the internet works great here. Maybe
[23:40] do you carry, right? And so, I get the full trace. I see all the activity that's going on in the agent and LLM running inference against it, right? And it's using that information to answer
[23:53] the question for me. So, as I told you, I used a conversation simulator to create an eval data set. So, this is my eval data set that's out here. As you
[24:06] see, it's actually a Unity catalog table. So, it's persisted out there in UC. And I can run evals against this whenever I want. And in there, it includes things like the input prompt, expectations,
[24:19] tags, right? So, it it includes things like concerns that it was written for. So, that can really help educate the process like hey, as part of my simulation, I generated this trace to test for this kind of this
[24:32] kind of concern, right? And then as I told you, we also created judges. So, these
[24:44] You can see that it came out here. It created its own prompt. And so, you can read through the prompt. You can modify the prompt. We one of the key things that can actually help improve this is using prompt
[24:57] optimization against your LLM's. So, you can use human feedback to guide these LLM judges in those type types of processes to improve your LLM judges so that they score evals just like you would, right? So,
[25:10] that's that's part of this maturity process is improving these things for that that kind of purpose. Um it it these right now don't run in a monitoring state. These are really only
[25:23] for offline evals. But, you could really come in here. Um you can choose the sampling rate. You can tell it, "Hey, I want to run on all future traces." So, if I click that on, every single time somebody's chatting
[25:36] with the interface and one of those traces lands here, it'll perform that eval and score it so that you can get dashboards and stuff that show you how this this model is performing over time or how these agents are performing over time.
[25:52] So, with that information, I I performed two different evals. So, one I performed on the original prompt that was created. Uh I forgot to show you the prompt down here. So, down here I have this prompt in my prompt registry.
[26:05] I have version one, which is a very basic prompt, right? Right? Like, you're a customer support agent. Do your job. Um and then there's lots of different versions of it iterating up until I have a finalized like champion improved uh
[26:19] prompt here. And so, this has a lot of guidance that's provided to this LLM to help it be able to be a better customer service agent for me, right? Um so, I have these two different
[26:32] versions and I've created these two eval runs. So, you can see a summary here. So, you can see a little bit of how this is how the story's going to end. But, if I click into the eval run, I see all 16
[26:46] traces. And then, I'm going to change the columns here. Turn off the expectations, turn on the assessments. And so I can see how this how the these
[26:59] judges assessed the responses here. And then I can come over here and I can see, okay, this guy failed both the return policy and the warranty uh process. And so I can come in here and I
[27:11] can see why it feels like it did that. Um and look at the uh assessments. I'm having trouble seeing here, so forgive me. Uh so it's no and
[27:25] where's the justifications? Sorry. So yeah, I'm going to I'm having a lot of trouble seeing the screen. Maybe I'll just look over here.
[27:44] But it has both the the uh what was the result and then it has also the justification for it as well, so you can see the breakdown of what's going on there. Um I ended up clicking on the the wrong thing. That's what I get.
[28:02] And all right, so I'm I'm getting into this uh And I clicked on the wrong eval. Anyway, so this is the finished eval. So as you
[28:14] can see here, it's 100% complete. Uh so running on the improved loop now you see that we have a 100% score across the board. Um one of the fun things that you
[28:26] can do with the eval runs is you can actually click in here and you can grab both of them and then you can click on actions compare and then
[28:41] how does it perform. Um so I tied in. I guess I uh ran one more time and I shouldn't have done that. But uh so I'm still just seeing the the improved solution there.
[28:58] So this failed. There we go. So I'm seeing the justification at least through this part of the UI. On the other screen it's there too, but I'm having trouble seeing up here. So my apologies. So the customer explicitly states that they bought the headphones 14 months ago, which is beyond the
[29:11] one-year return policy. Yet the agent still said that you can still return the the product. So the agent was probably looking at the manufacturing warranty and and said, "Oh, it's 3 years. So
[29:23] yeah, it falls within that that scope." And so now the new and improved prompt actually really corrects for that and and educates the LLM that there are two different types of policies and you
[29:35] should only reflect the one that's for our company. So that new and improved prompt allows it to be able to process like that.
[29:48] So as you guys can see, as you tie all this together, right? That's how you get to that level of confidence, right? Trying it once, trying it twice doesn't get you there. But if you can build out
[30:01] these eval data sets, and I only built out 16 prompts, I would build out more for a real customer-facing use case. I would work with the business to come up with what are our list of concerns, you know, what what's this going to perform?
[30:14] And I would build out eval data sets that do more than just try something once or twice, right? If you're just prompting for a certain issue a couple of times, that's running on vibes. It's not going to get you where you're going. But if you create these evals and you
[30:27] have them running, um and you build out this eval data set, you're able to go through this loop to get yourself to a place where you're like, "Okay, I know this is going to be be performing pretty well." And then when you turn on the monitoring, what you're able to do is
[30:40] you're scoring real production questions in real time. So, there's always going to be those types of questions that you didn't anticipate ahead of time. But, in monitoring, you can see how this model is performing over time. You can
[30:52] identify, oh, it flagged one is red. Go into it, understand it. Okay, I didn't I didn't expect that question. Now, I can create eval data sets around that. And I can score it, I can improve again, and go through my cycle, and get my agent
[31:05] back on track. So, these types of loops gives you that confidence that you can feel to be able to put something that's out there in front of customers, and know that you have that that improvement loop down, and you're not going to get
[31:18] phone calls from the CEO because, you know, you ended up on Twitter somewhere, right? Or X. So, yeah. I'm going to hand things over back to Tim.
[31:32] >> Okay. So, we told you what we're going to see, we showed it to you, and now we're telling you what you saw, right? We saw these three trios, uh this trio working together, right? Between the intelligence and the context with Agent Bricks, uh the surface area of Data Bricks apps,
[31:45] and Lake Base as that persistent memory layer. In conjunction with getting our models from Unity AI Gateway, and using MLflow extensively for observability and evaluation and monitoring. And uh it was
[31:57] an awesome job by Robert for that last part. So, it works. And we have customers doing this in production. And so, if you want to try this yourself, go for it. Uh
[32:09] on the right, that is for the AI Dev Kit that Robert mentioned. So, again, curated skills and uh MCP servers and plugins that we use internally. They're tested uh to work
[32:23] really well with all the Data Bricks APIs that you might want to automate with. >> If you're using coding agents at all, regardless if you're building what we're we built today, you if you're interfacing with Data Bricks,
[32:35] you absolutely should add this. Like it it it accelerates you significantly to be able to begin building. >> And on the And on the left, this is Robert's baby, the the workshop that he's run that you saw here live today.
[32:48] And if you have a Databricks account, uh you can run this yourself. >> Yeah. >> want to add Robert? >> Yeah, so feel free to to play with it. You can point your coding agents to it. As I said, it has guides built into the
[33:00] repo, so it'll walk you through step-by-step. It does not have to be this data, but we have this data provided as part of it. So, you can go through exactly what we went through today, or you can apply it to your own
[33:12] use case. These are also workshops that we offer our customers, right? So, if you are interested, if you have a use case, and you're like, "Hey, we want to do something like this." Um come talk to us. Talk to your AE. Try
[33:25] to engage us. We will come out. We can work with you for a day or two specifically on accelerating your use case, or educating your people on how to build things using using the use case that we have here. So. >> And lastly, a shout-out to a couple of
[33:37] other talks going on this week that if you're if you're interested in this topic, might be of more interest to you as they'll go a little bit deeper on some of the new product releases. So, thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.