Build and Monitor AI Agents with MLflow: End-to-End Observability

Summary

MLflow has evolved from an MLOps tool into the largest open-source AI ops platform with more than 25 million monthly downloads, now covering the full agent development lifecycle from tracing and feedback collection to prompt optimization and production monitoring.
Developers can add MLflow tracing to more than 40 libraries across Python and TypeScript with a single line of code, enabling root cause analysis, domain expert feedback collection, and automated LLM judge evaluation.
The MLflow AI gateway provides organization-wide governance and cost controls for access to models and MCPs, while experiment tracking and stakeholder sign-off workflows support iterative improvement before production deployment.

Build and Monitor AI Agents with MLflow: End-to-End Observability

Building production-quality AI agents requires more than a working prototype. MLflow provides comprehensive observability and governance tools to guide agents through development, testing, and production deployment. From capturing execution traces to collecting expert feedback, building automated evaluators, and optimizing prompts, MLflow enables developers to systematically improve agent quality.
Learn the full agent development lifecycle: implement MLflow tracing for root cause analysis, gather feedback from domain experts, build and calibrate LLM judges for automated quality evaluation, optimize prompts to fix issues, and set up continuous production monitoring. Discover how the MLflow AI gateway manages access to models and MCPs across your organization while maintaining cost controls and governance.
📂 MLflow documentation: https://mlflow.org/genai
🤝

Chapters

FAQs

What is MLflow and how has it evolved to support AI agent development?

MLflow was created by Databricks approximately 8 years ago as an MLOps platform focused on training classifiers and regressors and deploying them to production. It has since expanded into LLM training, experimentation, and agent development, becoming the largest open-source AI ops platform with more than 25 million monthly downloads.

How does MLflow tracing help developers debug AI agents?

MLflow tracing captures detailed execution logs for each agent run, allowing developers to inspect which tools were called, what responses were returned, and where the agent deviated from expected behavior. With support for more than 40 libraries in Python and TypeScript, tracing can be added to an existing agent with a single line of code.

What is the MLflow AI gateway and why does an organization need it?

The MLflow AI gateway manages organization-wide access to models and MCPs, providing centralized cost controls and governance so that individual teams do not independently connect to external services without oversight. It ensures that all AI components across the organization operate within approved usage boundaries.

How do you build and calibrate an LLM judge with MLflow for agent evaluation?

Building an LLM judge in MLflow involves defining the quality criteria the judge should measure, creating test cases that represent known-good and known-bad agent outputs, and iterating on the judge prompt until its scores align with human reviewer labels. MLflow's evaluation tooling supports this calibration workflow and stores results in the experiment tracking system for stakeholder sign-off before the judge is used in production.

Full transcript

[00:01] With a single line of code, I can add MLflow tracing to more than 40 different libraries across the Python and TypeScript ecosystem. The developers who are building agents are going to take a new tool off the shelf every day. As a result, each and every one of these components needs to be compatible with the entire ecosystem of programming languages, models, authoring frameworks,
[00:18] and beyond. Hi folks, I'm Corey. I'm an engineering lead here at Databricks focused full-time on MLflow and agent development. And I'm super excited to show you guys how to build high-quality, reliable agents with MLflow.
[00:35] Do you want to start with a little bit of background on what MLflow is and has been? So, Databricks started developing MLflow about 8 years ago, and we were initially focused on MLOps. So, how do we enable folks to build a great
[00:50] classifier or regressor, get that thing packaged up and deployed to production. See a few folks nodding, you may be familiar. Well, it's come a long way since then. We've gotten into LLM training and experimentation, and most recently agents development. And as a
[01:06] result, MLflow is now the largest AI ops platform in open source with more than 25 million monthly downloads. We also offer it as part of the Databricks GenAI platform with extra enterprise features like governance and
[01:22] massive production scale. So, with that background taken care of, with respect to agents, our goal is to give every organization, individual practitioner, research team, the ability to build agents as quickly as possible on an open platform that gives them everything they need to
[01:38] succeed. Now, before we dive into the specifics of that platform, it's useful to talk about the challenges in agent development. And I think to really anchor that, we should contrast them with something that we're all probably familiar with, software
[01:53] development. The best practices around developing high quality software have become more or less standard the last 15 or 20 years. Right? Folks are going to write and run code locally. Once they think it's reasonable, they'll write some unit tests and iterate until
[02:09] they pass. If they're really doing a good job, they might write the tests first. After that, they'll hand it off to an internal team for QA or they'll send it to their colleagues. They'll add some production telemetry, ship the thing into production, and get stack traces when things crash, and hopefully they
[02:24] have some evidence that folks are using it. But what's the equivalent for agents? And why doesn't this same workflow apply to agents just as much as software? Well, it turns out there are several layers of complexity here that make building an agent a lot more challenging.
[02:41] First and foremost, agent outputs are often free-form and unpredictable. I take the same query, send it to the same agent five different times, I can end up with five different answers. Measuring the quality of output is also subjective. And typically requires domain expertise.
[02:57] If I deploy an agent, I might have a user expecting some certain answer, and I would have no idea that that's what they wanted or how to give it to them. I could ask my colleague if they think an answer is reasonable, and they may give me a totally different answer than I would have thought. This necessitates collaboration. So,
[03:13] your developers who are building agents need to go and talk to those domain experts to understand fundamentally how that agent is performing. And then you have platform teams that need to make sure the agent remains available, fast, and cost-effective at scale.
[03:30] While folks grapple with these challenges, every single agent that's deployed to production creates new risk. Risk of leaking PII, offending users, or creating a bunch of other hazardous outcomes. And finally, folks are constantly playing this game of a trade-off between
[03:46] cost and latency and quality. They may pick a cheaper, faster LLM, and unbeknownst to them end up degrading quality to the point where it's unacceptable. So, how do organizations fare in the face of all of these new problems?
[04:02] Well, unfortunately, we see a pretty common anti-pattern emerge. Folks write an agent locally. They may even test it with a couple of examples. They'll ship it to prod and they really will hope for the best. You know, they start getting bug reports and user feedback and they think that's good enough. All the while they've exposed
[04:17] themselves to all the hazards that I just talked about. Now, unfortunately, there's a better way. And through the process of working with hundreds of enterprise customers and open-source developers, we've created a more well-defined risk-free agent
[04:32] development life cycle and a platform to support it. First and foremost, developers are still going to build that working prototype. They'll try a few examples and they'll get it to pass basic checks. But instead of just shipping it to production, they're going to bring in those experts
[04:48] to test it. They're going to go through that QA process. Inevitably, they'll end up identifying new quality issues. These are their test cases. They can then implement fixes in prompts, agent code, switch models, and beyond.
[05:03] And then they can verify that those fixes work using those reproducible test cases. Finally, they get stakeholder sign-off, deploy to production, and monitor quality. Now, this all sounds pretty intuitive and it sounds simple. So, so what's the big deal? Why do you need a platform for this?
[05:19] Well, there are actually several pieces of infrastructure, several components that are extremely necessary to enable developers to go through this workflow. First and foremost, you need observability. It's the name of our meetup here. You need to be able to understand what's going on at each step
[05:35] within an agent as it processes a query. Where is it retrieving knowledge from? Which context is it passing to models? What code is being executed? And beyond. You also need evaluation capabilities. So, how can I collect feedback from my
[05:50] domain experts and my end users to understand whether my agent is meeting their needs? How can I automate the process of quality issue detection using techniques like LLM as a judge, which we'll talk about shortly? Because prompts, even in 2026, remain
[06:06] extremely important to dictating the performance of an agent, folks need to manage their prompt engineering. They need to register, version, and share prompts with one another in order to structure the development process. And finally, for platform teams, every single LLM call is creating that risk of
[06:23] PII leakage. It's creating the risk of runaway costs. Folks need to manage access to LLMs, MCPs, tools, and beyond across their organization in order to avoid bankruptcy and and other serious hazards. There's this emerging category
[06:40] of AI gateway technologies or platform components that provide out-of-the-box rate limits, cost controls, budget grouping, and all sorts of other stuff for LLMs and MCPs across an organization. But just having these components is not enough.
[06:55] The developers who are building agents are going to take a new tool off the shelf every day. They're constantly iterating and experimenting because the space is moving too quickly not to. As a result, each and every one of these components needs to be compatible with the entire ecosystem of programming
[07:10] languages, models, authoring frameworks, and beyond. This is why building a robust platform that accomplishes all this is such a challenge. And this is why we've purpose-built MLflow to have all of these components with total ecosystem compatibility.
[07:26] Now, let's move into that agent development life cycle and take a closer look at how MLflow's tracing or observability component is going to enable developers to get a prototype built faster. With a single line of code, I can add MLflow tracing It's more than 40
[07:42] different libraries across the Python and TypeScript ecosystem. MLflow tracing is also OpenTelemetry compatible, which means that I can stream in traces from any programming language or any authoring framework that I'm already using. Let's jump right into a demo. For the purposes of the demo contents
[07:57] throughout today's talk, we'll be looking at an example agents to answer questions about a telecom company. We call this Telco agent. So, I'll go ahead and I'll import the MLflow library in my Python notebook. And I'm going to go ahead and add the
[08:12] MLflow autolog API, enabling tracing for my agent in a single line of code because I used one of these authoring libraries. Then, from the agent interface perspective, I can go ahead and issue an example query to my telecom agent. I'll go ahead and submit it. The query is
[08:29] asking about device upgrades without increasing a customer's bill, and I'll get a response. Now, the beauty of this is on the developer side in the MLflow UI, I get a step-by-step detailed execution trace that I can use to inspect exactly how
[08:44] the agent handled that query. We'll be looking at this a lot more closely in a moment, but what this lets me do as a developer is, you know, if I'm dissatisfied with the response, I can dig in, root cause it, fix the problem, and keep iterating on my prototype. If I don't have this, I'm left guessing as to
[09:01] what went wrong. So, because trace tracing helps me get to a working prototype faster, now I can start test driving with domain experts inside my organization. This is where MLflow's feedback collection capabilities are super important. We offer several components
[09:18] for this. First and foremost, we have a built-in chat style review app. You can take an agent or a collection of traces from anywhere in your organization and send them to an expert reviewer. All you need is their email address, and they'll get a nice chat UI, which we'll see in a moment, to review those traces. You can
[09:35] also integrate feedback collection using MLFlow's APIs and SDK into any agent regardless of programming language. And all of this feedback is stored directly alongside the traces, as we'll see in a moment. So I can come back into this UI, and I know the text might be a little small,
[09:51] but if we look closely, we'll see the agent's response wasn't actually all that useful. It recommends reaching out to a human. It doesn't know what to do. Because I've connected this with MLFlow's feedback collection, I'll hit thumbs down, submit some explanation for why I'm dissatisfied, and then from the
[10:07] developer's perspective, on the right side of the screen, they see that feedback recorded. So not only do I get the step-by-step execution information, I can also search and filter for all of these traces that have negative user feedback and hone in more deeply on my
[10:22] search for quality issues. It's also quite common to, you know, end up with ill-specified explanations like this, or in some cases no explanations at all, and to require more information from those domain experts. This is where MLFlow's labeling
[10:39] capabilities come in. So I can filter for all of those traces that have user feedback where I want more information. I can then build an MLFlow labeling session to request additional input from any colleague within my organization. So I'll go ahead and I'll hit add to
[10:55] labeling session here. And then I will share that labeling session with another member of my organization. Once I share this, they will be given this chat style UI where they can review each trace and provide more context. So this time, in addition to asking for
[11:11] that binary thumbs up, thumbs down feedback, we can also ask them to provide a more detailed comment or explanation, which gets persisted back to the trace. This helps the developer hone in even further and understand why exactly the domain expert or internal
[11:26] tester was dissatisfied and what to do about it. So now that we've gotten feedback from human it's time to hone in and discover some quality issues. Looking more closely at this particular trace, it's going to be important for us to
[11:42] understand what went wrong. So, the way our agent works is it reads this query and it comes up with the topic or category. In this case, it thinks the topic is billing related, so it sends the query to a specialist agent that only knows about billing. And as a result, this
[11:58] billing agent says, "You know, I really can't help you with a device upgrade." And you end up with the not-so-helpful end user response. Now, for a small trace like this, it's pretty easy for me to visually read through this and get to the bottom of what's going on. But, it's also common to have much more
[12:13] complex agents with hundreds or even thousands of different operations. And how could I possibly read all of that? So, I want to show you another approach that we support here in MLflow. Instead of manually reading through this trace, I can leverage the brand new MLflow assistant. Open up the trace in
[12:29] the UI and click a button and ask this assistant to root cause and debug using this trace for me. And after a few seconds to a minute or two, I will ultimately end up with the exact same root cause without having to do any clicking or any manual
[12:45] explanation. We'll see that uh the assistant gives me the root cause, comes up with the exact same conclusion that there's a routing problem. It's even going to give me a recommendation about where the fix likely lies, so in the prompt in the
[13:01] agent, which we'll see in a bit. This helps take the debugging process to the next level, helps me understand exactly what's going wrong, regardless of how large my trace is. So, now we've found one issue in one trace, but the natural question that comes up is
[13:18] is this worth spending my time on? How many How many different traces have this issue? Uh often times with agents, there's always something going wrong. There could be 5, 10, 15 different things that really just don't matter that much. Or there could be one or two huge things that really merit time investment. And how do I
[13:33] tell? This is where I need to be able to automatically look for more instances of this issue. It's going to be really hard for me to read 100, 1,000, 10,000 additional traces and see how prevalent this is. So, I'm going to build a function to automate that process for us.
[13:50] This is referred to in industry as an LLM judge. Based on a description of the issue, for example, uh formality in in this screenshot here, but you know, soon we'll see based on this particular use case, I can pick a model and ask an LLM to
[14:05] find all the traces that have the exact same problem. In addition, MLflow offers a variety of built-in judges, more than 50 plus ecosystem integrations for a variety of popular use cases, including measuring the correctness of a response, the
[14:21] relevance, safety. We also have judges for chatbots. They can look across an entire conversation and determine whether a chatbot is repetitive, whether a user is frustrated, whether that agent remembers something that you said to it 15 messages ago.
[14:36] They can look at tool calls, retrieve documents, and beyond. MLflow also has capabilities for judge optimization. So, instead of iterating on the quality criteria and the instructions and then trying again and again until it agrees with a human, you can pass in labeled traces and will
[14:53] leverage state-of-the-art optimization techniques like DSPI and JEPA to automatically write those instructions until they detect the exact same thing that humans do. But we're going to hone in on that custom judge use case. After all, we've identified a particular issue that
[15:08] pertains to our particular agent. So, we're going to build a routing accuracy judge in the MLflow UI. I'll call the judge routing accuracy and I'll select an LLM to use from our AI gateway, which we'll talk about later. I can then go ahead and insert some
[15:23] instructions that judge should follow as it reads each trace. So, I'm going to tell it about how my agent works. I'll tell it how it classifies a query. I'll tell it how it determines whether or not that classification was reasonable, and I'll
[15:38] ask it to look at the trace and give me its assessment. Is the correct sub agent being selected here or not? I can then go ahead and select an example trace to run this on. So, in this case, I'm going to pick that trace that we just looked at, and I'm going to
[15:54] go ahead and run this judge. And after just a few seconds, we see that we've built an automated function that agrees with us. It looks at that same trace, and it says, "You know what? This should have been routed to a product sub agent. The billing agent was the wrong one." So, now I've built this automated
[16:10] function. I don't need to look through hundreds or thousands of traces. I can take this function, and I can run it across all of them in just a matter of minutes, and see just how prevalent this issue is. So, I'm going to go ahead and do exactly that. Fortunately, MLflow gives me an SDK and a UI where I can plug in some
[16:27] traces, plug in one of these judges or multiple, and we will execute them in massively parallel fashion. So, I can go ahead and hop into a notebook. I'll load the latest 200 traces, and I'll pass them in to the MLflow GenAI evaluate API along with that judge
[16:44] that we just built. And after a couple of minutes, we get an evaluation report, which I'll go ahead and open up. And what we see is that of those 200 traces, only 79% of them have correct routing. Which means this wasn't a one-off, it wasn't an isolated incident. 40 traces
[17:02] here, just of the last 200, have the exact same problem, which obviously merits more attention. And now I know that I need to dig in and spend actual time fixing this problem. So, now, in order to fix the problem, I need to be able to run test cases. otherwise
[17:18] I'm not going to know whether I'm making progress. This is where I can snapshot my traces, capture the inputs, which can be used to test a new version of my agent. So we'll go ahead and do this in the MLflow UI. I will filter for all of those traces
[17:33] that have the negative routing accuracy judge results, same way I filtered for user feedback, except for this time, instead of adding the traces to a labeling session, I'm going to go ahead and add them to an evaluation data set. So I'll select all of my traces, I'll
[17:49] hit uh add to evaluation data sets, and I'll create a new one. This can be backed by S3, Azure blob storage, your local file system, Databricks, and a whole variety of other storage locations. I'll give this data set a nice name, and
[18:05] uh in just a moment hit the export button to snapshot these traces. And what I'm ultimately going to be given in just a moment is a list of all of the inputs from those traces that had routing accuracy issues. They link back to the trace they came
[18:21] from, and I can also tag and annotate them and search and filter based on different criteria if I only want particular rows as well. Now the power of these test cases is I'm just going to be able to use them to verify that I've fixed the issue or not as I iterate.
[18:39] So now that we've identified a quality issue, we've confirmed that it's a real problematic quality issue, and we've created a set of test cases, it's time to implement a fix. Now, as I mentioned previously, prompts are still fundamental to agent performance and agent quality. So
[18:54] they're a natural first place to look. Here we're going to see how MLflow's prompt registry makes it very easy for me to debug this particular issue. Hopping into the MLflow prompt registry UI, we see that I have a prompt for each major component of my agent. I have the
[19:12] routing prompts as well as sub agent prompts for billing and products and beyond. As I scroll through this prompt and its various versions, I can come down and see that there's some very well-specified routing decision criteria in my agent.
[19:27] And it seems pretty obvious at this point that this is where I would start if I were to implement a fix. But prompt engineering is real slow and I don't have time to do it up here with all of you live. Um and you know, even if I had several hours, there's no guarantee that I would make consistent progress. So rather than spending all this time
[19:43] tweaking the prompt, checking if it's working better, iterating, why can't I automate this process as well? And that's where MLflow's prompt optimization capabilities come in. I can take that data set that I just built. I can take that judge
[19:58] that I just built, drop that information along with my prompt into an optimization function, and an LLM will try new prompts, run the agent on that data set until the judge is happy. It's a much faster approach to this problem than me sort of guessing
[20:14] and checking, which is really what prompt engineering tends to be. Hopping into my notebook, let's do exactly that. I'll go ahead and load my data set, and I'll load my routing accuracy judge that we just built. Then I'm going to call MLflow's optimize
[20:29] prompts function. I'll pass in my agent with the new prompt. I'll pass in my data set. And I'll specify the location of my prompt that I want to optimize. I'll use GPT-5 as the LLM to come up with new prompts.
[20:45] And I'm going to iterate until the judge agrees that the new agent is really producing a better result. I'll let this run for several minutes and then fast forward. And what we'll see is we have four new versions of our prompt culminating in the best one that the judge determined
[21:03] to have a higher accuracy or more quality on our test data set. But now I need to verify this, right? I ran optimization. How can I be sure that, you know, it really produced better results? This is where it comes time to test my fixes.
[21:20] That exact same evaluation API that I used to find the issue can also be used to test the fix. Instead of passing in the traces, I'm just going to pass in my data set, my inputs, and my updated agent. It's the only difference. So, I'll come into the notebook. It's
[21:36] going to look very similar. And I'll run the evaluation of that judge on the data set and the updated agent. And I'm going to get a brand new evaluation report. Looks a lot like what we saw last time. When I open it up in the UI,
[21:53] I see this claim, 100% routing accuracy. We can really trust this. I mean, it sounds too good to be true. It's kind of an outlandishly high number. Uh so, why don't I go ahead and open up the trace for how the new agent handled that same query about the device upgrade.
[22:09] Well, this time it's routing to the product sub agent. It's no longer going to billing. We can see that the judge takes a look at this new trace and says, "You know what? Product still makes sense to me." So, I've confirmed here with checking a single trace that the new agent is performing better. The judge still
[22:26] agrees with me. So, I can have much more confidence that that outlandishly high figure might actually be real. I can also go in and compare the performance of the updated agent with the performance of the old agent. And I can see that 21% uplift in routing
[22:43] accuracy in just a few minutes through this prompt optimization run. I can also see how this impacts the end user experience. So, on the right, we have the old version recommending escalation to a human. And on the left, we have concrete device
[22:59] upgrades. Not only did we fix the routing problem, we actually made the end user experience better as well. This had real tangible quality impact that any human talking to our agent can appreciate. Now everything that I just walked
[23:14] through, finding a quality issue, creating a test case, optimizing a prompt to fix it, and verifying my fix was a UI and kind of API based workflow. But I know that a lot of you are probably using coding assistance each and every day, and you're probably using them to build
[23:30] agents as all of these capabilities that you just saw are available available through MCP, CLIs, and agent skills, which we publish on our MLflow GitHub repository. You can point your agent at MLflow at point your coding assistant at MLflow
[23:46] and your agent and have it identify issues, build test cases, fix prompts, and autonomously iterate to improve agent quality while you grab a cup of coffee. So what we saw was one iteration of life cycle, right? I'm going to get this
[24:02] feedback from human testers, I'm going to find some bugs, I'm going to fix them, I'm going to test the fixes, and I'm going to throw that agent back over the fence and have the humans test again and again. But eventually, I will have squashed all the important bugs, and it's time to go towards production. MLflow is also going to help here
[24:20] through its experiment tracking capabilities, which have existed for quite some time. But we revamped them and brought them into the agent world. So instead of sending my stakeholder this report of 2,000 traces with thumbs up, thumbs down, good judge feedback, bad judge feedback, I can just
[24:35] give them a dashboard that gives them overall metrics. What is the routing accuracy on average for this agent as compared to this agent on the same data? What's the per query relevance? What does latency look like? Just by sharing the single overall report, folks can
[24:51] have a lot of confidence that a particular agent version is actually ready for production. Now once I've gotten that sign-off, it's time to deploy to production and monitor quality in production. After all, users are going to ask
[25:06] different questions than my domain experts. I'm bound to find more issues. So, I really need that quality monitoring. Fortunately, to actually deploy to production, nothing really changes. I already added tracing to my agent. So,
[25:21] when I deploy to prod, I get traces from end users instead of domain expert internal testers. I already added feedback. Now, the feedback just starts coming from end users instead of those internal testers as well. Because I already built an LLM judge, I can start running that judge online, as
[25:37] we'll see in a moment. And the exact same approach that I took for building data sets from development and internal testing applies to production as well. Set this all up once, and you can identify quality issues across different environments, dev, staging, prod.
[25:55] So, let's hop back into the UI, and we'll see how we can set up online monitoring for that routing accuracy judge. I'll go ahead and open the judge, and I'm just going to select a single checkbox here. When I hit automatically evaluate future traces, this judge will start to run on
[26:13] every single new trace that comes in. Now, I can set a sampling rate to reduce costs. I can apply a filter, so I'm only interested in, you know, looking at traces from a premium user tier, for example, and much more. I can also add new judges directly from this UI. So, I'm going to go ahead and
[26:29] get a safety judge running as well. It's going to look for things like PII or offensive messages, which is real important. And I'll run it on 100% of traces, cuz turns out safety is pretty much always important. I already have a query relevance judge running. And so, as new traces come in,
[26:46] coming back into the traces tab, I can see results for each of those judges in real time along with aggregate statistics. You know, 89% routing accuracy is probably much better than where we would have been if we had just deployed to prod initially. This gives me some confidence that our
[27:01] agent is legitimately meeting the needs of our users. And you know, for examples where it isn't, I can now search, filter, and dig into those particular examples to further improve. I have complete observability and I have this recurring quality evaluation process.
[27:18] Now, I did want to briefly revisit how MLflow enables organizations to manage access to LLMs, tools, and MCPs. With the MLflow AI gateway, every single request that is targeted towards an LLM or an MCP is rate limited, is subject to
[27:36] permissions checked, cost controls, and is usage tracked. So, you can see all of the logs for all LLM API calls across an entire organization. This supports more than 50 different LLM providers. You can also bring your own model. Hopping into the MLflow UI, we
[27:52] previously built a judge by selecting this GPT 5.2 endpoint. Because I was using the AI gateway to host this endpoint, we can take a closer look at the usage tracking functionality. Now, because I was only doing this in development when
[28:09] building the judge, I only issued a few queries to GPT 5.2, but we see them tracked here in a usage dashboard. I get request counts, token counts, error rates. All of this scales up to production as well. Regardless of how many different developers in my organization are
[28:24] building agents with GPT 5.2, I could understand my spend from a single dashboard. I can also view the traces. So, I can see the exact LLM call that was initiated when I tested that judge. Here we see a the judge trying to reason
[28:39] about the routing of our agent. This provides total observability and governance along with additional useful features like traffic splitting and fallbacks in case LLMs become unavailable. So that's the overview of the end-to-end
[28:54] agent development life cycle with MLflow and the wide variety of platform components that support it. And I guarantee you you don't want to try to build this yourself. I want to briefly cover the new releases and roadmap. So we recently launched more than 30 different built-in and
[29:09] customizable judges for chatbot evaluation. They can look across long horizon conversations of hundreds of messages and detect for subtle problems in agents in ways that simple turn-by-turn judges just can't. We also are launching supports in just a couple of days for user simulation. So
[29:27] you can use an LLM to act as a user and talk to your chatbots and uncover issues. Normally with a chatbot, you have to send that agent out in your organization and wait for people to talk to it to understand what's going on. User simulation solves a lot of that
[29:42] problem. It also enables techniques like red teaming for more sophisticated quality testing. And everything that I just showed you, we want to make 10 times simpler. The end of the day, we're targeting an automated issue discovery product where you can just log traces. We'll tell you what's wrong with your agent and how to
[29:58] fix it. Sounds like science fiction? We're this close and we're very excited to deliver that shortly. I also do want to briefly plug the managed offering on Databricks. One of the benefits you'll get is the ability to ingest terabytes of traces every hour or every day directly into Databricks
[30:14] Unity Catalog. You can use MLflow to redact PII, which you can also do in open source, but here you can take copies of the traces, share them with other members of your organization, query them with SQL, and build rich AI BI dashboards. You can also use Databricks Unity Catalog to
[30:30] govern access to human feedback evaluation data sets and more. Now to get started with MLflow, you can scan the QR code, check out mlflow.org/genai, or visit us on GitHub.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.