Closing the Agent Quality Loop Through Observability in Production

Summary

The agent quality loop on the Databricks Data and AI platform uses MLflow tracing to log every production interaction to Unity Catalog Delta tables, out-of-the-box LLM judges for automated evaluation, and a review app for subject-matter expert labeling that generates the human feedback needed to align judges to real user corrections.
Automated issue detection opens trackable GitHub bugs when quality regressions are found, and coding agents automatically apply fixes—only involving a human reviewer when there is an actual improvement to approve—creating a continuous improvement flywheel around production agents.
Databricks Asset Bundles gate each agent release with a CI evaluation set drawn from production traffic, ensuring every new version is validated against real-world queries before deployment.

Closing the Agent Quality Loop Through Observability in Production

You built an agent. It passed dev testing and shipped. But production is a different world, and now the real work begins: proving it stays high quality and making it better with every release. this video walks the agent quality loop end to end on Databricks with MLFlow, from the seat of the AI Ops engineer supporting a production agent. We start with tracing and observability: real-time trace logging into Unity Catalog, out-of-the-box LLM judges, and a review app for SME labeling. We turn that human feedback into trustworthy, scalable evaluation by aligning LLM judges to users' corrections. We surface problems through automated issue detection that opens trackable bugs, then feed production traffic into larger, more targeted evaluation sets as a CI gate, and ship repeatably with Declarative Automation Bundles. In the end, we are left with an Agent flywheel that de-risks the upcoming releases and discovers problems in existing ones. Through this video, we demonstrate this whole loop live on a real composite agent.
Talk By: Arthur Dooner, Sr. Specialist Solutions Architect, Databricks ; Euirim Choi, Software Engineer, Databricks ;

Chapters

FAQs

How do I monitor agent quality in production on Databricks?

The Databricks Data and AI platform supports real-time trace logging into Unity Catalog Delta tables via MLflow, giving teams a queryable record of every agent interaction. Out-of-the-box LLM judges run automated evaluation against those traces, and a review app lets subject-matter experts label responses to ground the evaluation in actual user expectations.

What is LLM judge alignment in MLflow?

Judge alignment is the process of calibrating automated LLM evaluators so their scores match what human reviewers consider correct, using a .align() method in MLflow that incorporates subject-matter expert feedback. Without alignment, LLM judges can drift from real user quality expectations and give misleading pass or fail signals on agent outputs.

How can coding agents automatically fix production agent regressions?

This video demonstrates a workflow where automated issue detection identifies quality problems in a production agent and opens a trackable GitHub issue, after which a coding agent proposes a fix. The fix only reaches a human reviewer once it is ready for approval, keeping most of the remediation loop automated and minimizing engineering interruption.

What is the agent quality flywheel on Databricks?

The agent quality flywheel is a continuous improvement cycle where production traces feed evaluation, evaluation surfaces regressions, automated agents propose fixes, and Databricks Asset Bundles gate each new release with a CI evaluation set before deployment. This loop de-risks upcoming releases and discovers problems in existing agents without requiring constant manual oversight.

Full transcript

[00:08] Let's talk about closing the agent quality loop through observability and production. Wow, isn't that a mouthful? Um, so when I think about agents that you build, think about the the set of all the agents that you build and you think about the very very small subset
[00:21] that reach production, right? And a lot of you are probably coming from Adam G's talk on how he how you think about the inner dev loop. How you build this AI loop where you're actually talking about like how do I iterate on my AI systems?
[00:34] How do I make those AI systems higher quality rapidly with some of the capabilities in MLflow. So now you've been through Adam's talk and if you haven't seen it, I encourage you to watch it on YouTube or the portal later.
[00:46] um your agent exists, you were forced to put into production suddenly and you don't have all the monitoring pieces that maybe you want. So we want to make sure that you can make this a virtuous loop of closing the quality gaps by
[01:01] automating a loop around this AI life cycle. And I mean that quite intentionally where by the end of this talk you'll see how we use coding agents over the agent that you've made to
[01:14] automatically open up issues and help us discover the issues that exist in your agent and automatically improve it and only involve you when there is actually a change for you to review an
[01:26] improvement that it's made. So um let's get started with some intros. Why sh why not? So I'm Arthur Duner. a specialist architect at data bicks. I work in the manufacturing vertical. I uh work with
[01:39] all manufacturing customers from logistics to um semiconductors to automotive to energy. I I'm in the trenches building the building these sort of systems with enterprises across
[01:51] industries and I collaborate with other industry partners all the all the time. With me is Yurum. >> Hey, I'm Yurum. Uh I'm a software engineer working on AI evaluation and observability here at Data Bricks.
[02:03] Awesome. So, when we think about these AI systems, note how I have all these boxes and they get progressively smaller. Everyone's really excited to build this AI AI agent. Everyone's very enthusiastic. And then you're like, ah,
[02:16] we've got to test it. And you're like, oh no, I got to get feedback from my users. What? Like all the and then wait, I have to have confidence gateways before I move to production. So what
[02:28] ends up happening is people end up asserting correctness. THEY ASSERT THAT IT'S HIGH QUALITY. They don't necessarily know that this AI agent is high quality. So what do we do? We have
[02:41] to think about
[02:54] to actually solve these problems. And this loop is not something that you can say, "Oh, I'm just going to build." Um, I like to draw the distinction between data scientists who have a lot of difficulty shipping things to production
[03:07] and app developers who have no sense of how to know if their AI system is high quality. We need skill sets from both of these backgrounds to have high quality systems. And it requires us to have this evaluation data data uh gathering
[03:21] mindset collect this prep new data for evaluation and continue this loop. So I have a case study that I'm going to dive into throughout this presentation today. It is a supply chain agent and it's all
[03:34] about tracing and observability and how we layer this repeatable motion in data bricks and use some of the primitives of MLFlow to help us get there and help us make this AI agent higher and higher
[03:48] quality automatically. So to give you a little sense of what this agent is, we've got all these supply chain planners. They're all thinking about, you know, we've got all of these shortages at different factories, different plants. How do we resolve
[04:00] these shortages? So, I threw together a few genie spaces for them. Everyone knows what a genie space is. Text to SQL, ask questions about my data, get SQL back. One that looks into all the inventory available across all the
[04:13] different plants. One that looks at all the transportation lead times and one that looks at and then like a unified AI search, which is the new term for our vector search of like the business processes. So, maybe we shouldn't be shipping stuff from Mexico to the United
[04:25] States for whatever reason per this company's policy. So, we're going to manage and uh deal with that. So, so to give you a sense of this, I want to stress not every agent needs to be a
[04:40] chatbot. Say it with me. Not every agent needs to be a chatbot. Note how I don't have a lovely UI for this. One, I'm lazy. Two, this not what it's for,
[04:52] right? You want to have this actual system looking over all the part number shortages and then sending your analysts an email. This is the resolution we found for different sol different um part shortages. Can you approve this?
[05:05] Here's the research that we did. And then maybe there's a chatbot to investigate further. So you think about like what would this generate again in the spirit of laziness? Let's talk about it looks something like this like this is what the parts that we have on hand.
[05:17] This is the safety stock. These are the different the highest risk parts of it. And then like these are the best options to manage this shortage. If you're familiar with agent bricks is a supervisor agent that connects all of these together. Uh so you have this idea
[05:30] of like oh take a look at all of these different agents and or all these different genie spaces and get us an answer. So how do we know if THIS IS GOOD? I HAVE no idea. That's what Yurum's going to talk about. He's going to start with traces.
[05:43] >> Mind anyone with click? >> Yeah, of course. So we talked about the demo. There you go. >> Great. Yeah. So phase one is about tracing and observability. Now what are the problems that we're trying to solve in tracing and observability? So you
[05:55] have two problems, but let's start with the first one, right? So you have an agent in production, but you're not sure how it's performing. It might be performing poorly, but you just generally don't know. And so this is where real-time trace logging for production agents becomes necessary.
[06:09] Now, we have a whole bunch of folks in this room with um different backgrounds. And so I want to make sure that everyone is on the same page with respect to what is a trace. Now you can think about a trace as something that tells you what
[06:21] your agent did and when. Right? So this includes things like tool calls, LM calls, errors, etc. And so it'll tell you exactly when an agent called a tool, exactly when it called the LLM, and when
[06:33] things fail, it'll tell you as well. And each step in this process, um, these are like the components of a trace. This is known as a span. So on your right, you can see a nice little trace viewer here, um, that shows you what these spans look
[06:45] like. So here you have a full trace. It's a coding agent. You can see that the coding agent is calling uh the LM via these chat completion calls. Then you have like this glob tool. You have this bash tool. For those of you who are, you know, developers and working
[06:58] with coding agents, you know what these tools are. And it'll also tell you what the inputs to these tools are and what the outputs to the tools are. And for the chat completion spans, it'll tell you what you inputed to the LM and what the LM outputed. So it provides
[07:10] essentially endtoend observability. And it also gives you metrics things like like things like aggregate metrics like token counts, the cost, latency, etc. So how do you actually get your agent to
[07:23] generate a trace? Well, with MLflow, it's super easy, right? So you can use any agent framework you want. It includes things like lang chain strands forcelli SDK, claude agent SDK, etc. Whatever you want, any major provider,
[07:36] MLFlow can trace it for you. And all you need to do is the code on the right. Now obviously you'll need to make some small changes to it depending on the agent framework but essentially it's always the same right so you install MLflow you
[07:48] set an experiment and setting experiment what it does is you tell it where you want to send your traces to and in this case you want to send your traces to a Unity catalog location so you specify a catalog name a schema name and a table prefix and and this will sort of tell
[08:01] you this will generate a few tables and you'll store those traces into those delta tables and how you actually trace get your agent to submit traces to the delta table is via this nice little decorator here, right? MLflow at
[08:14] MLflow.trace. And if your agent is a function, that's all you need, right? So once you do that, your traces start getting into delta. So whenever someone submits a query to your agent, your traces end up in a delta
[08:28] table in the location that you specified. And it it does this really quickly, right? So the latency is really short. It's like under 5 seconds the vast the most of the time and under 30 seconds the vast majority of the time, right? So you'll see your traces as rows
[08:41] in this table or your spans as rows in this table. And the beauty about this is that this is like any other delta table that you're used to in Unity catalog. You can take advantage of all the UC features like governance, lineage, etc.
[08:53] And so things like permissions, you can specify who gets access to your data. You can do, you know, row based masking. So if you don't want certain users to be able to see certain traces or certain spans or whatever, you can mask those uh
[09:05] using SQL. And of course um just like any other delta table that you're used to you can analyze the delta table using SQL right so you can also build AI BI dashboards on top of the delta table to
[09:17] actually visualize your traces just like any other business state data. So this becomes really powerful right so like you can you can ask uh for example Genie uh Genie 1 if you use Genie 1 Genie 1
[09:29] can access these traces in these tables because these traces are in delta. So, of course, you can view your traces using SQL, but we also want to provide you a good experience when viewing the
[09:41] traces. So, we've actually built a UI um in MLflow um that can help you visualize the traces. So, it's a nice little table here. You can see the request response, the number of tokens, execution time, etc. And if you click on one of these
[09:53] rows like the the request column here, um then you can actually see the trace viewer that we talked about before. And you can, you know, set filters like the time range and other filters like whether an agent called a specific tool. And you can narrow down your traces um
[10:06] to that um specification. The other important thing to note is that these traces are not proprietary, right? We're not we're not we're not saving them in a proprietary format. So if you really want to export your traces
[10:19] to a different system, we obviously want to keep you within data bricks, of course. But if for some reason you want to analyze your traces in data dog or some other competitor, then these traces are all follow the open telemetry standard, it's fully open. So there is
[10:32] no vendor lock in. You can export your traces uh with little to no friction. So now that you have the traces in the delta table, you can of course get aggregate metrics from those traces, right? Remember when we saw that in the
[10:45] trace viewer you saw metrics like token counts etc. Well, you can see an aggregate view of all that data. So in this little visualization here, let's wait until this GIF goes to the usage uh tab. You can see the total number of
[10:58] traces, latency, errors, uh token usage, tokens per trace. You can see, you know, which models are generating which tokens or how how many tokens and etc. And you can also see this nice little tool call section that details a breakdown of how
[11:10] how often each tool call is being called and what the error rates are. Great. So we've solved this problem. We've solved the first problem of providing visibility into agent
[11:22] performance. But now we want to evaluate quality more efficiently, right? Nobody has the time to read thousands, tens of thousands, millions of traces. Some of our customers, they have literally millions of traces per day. It's crazy,
[11:35] right? No one has the time to read all that. And you all you also might not want to monitor the dashboards all the time, right? And so the solution is something we call out of the box LM judges for offline and online
[11:47] evaluation. Now a lot of you are probably familiar with LM as a judge right and so MLflow provides some out-of-the-box options that have been developed with the help of our research team as well as our engineering staff and there are sort of
[11:59] two types of judges which we also call scores. So there's a single turn judge. So what is a single turn judge? Well it's for evaluating one message sent right. So if you have someone talking to an agent the first me message they send or if they only send one message that's
[12:12] for the single turn judge right? So we have things like safety retrieval relevance correctness etc. a lot of other built-in judges that we've specified in the docs link docs link up there. We also have multi-turn judges. So, a lot of users, they interact with
[12:24] your agent over the course of multiple turns. And what does that mean? Well, they're having a conversation with that agent, right? They're sending multiple messages. And so, if you want to evaluate the conversation, we also have some built-in options for you like user
[12:36] frustration. So, measuring, you know, whether a user gets frustrated while interacting with your agent. Uh some of you might have had that experience when interacting with a customer service agent. We also have conversational completeness, conversational safety,
[12:48] etc. So many options. And the important thing to note is it's really easy to use these out of the box judges, right? So you can you can you can see here on your right, all you need to do is install MLflow, of course, fetch your traces uh using MLflow and then uh you can just
[13:02] pass those traces along with the the the judges that you want like correctness and safety, which again can also be referred to as scores into MLflow geni evaluate. and you'll see the evaluation results in the results variable there.
[13:14] So it's super easy to use and of course out of the box judges they don't you know always work and so you might want to have some custom judges that um you know provide quantitative metrics like the amount of you know characters that are that are outputed by a judge uh and
[13:28] you can always define those judges uh yourself or with the help of a coding agent including genie code. So you have these judges and you've used them for offline evaluation. So what does offline evaluation mean? you have a
[13:41] set of traces and you've evaluated them using these judges but of course a production agent it's not static right you keep getting traces um because users are using your agent in real time and so you don't want to have to remember to
[13:53] you know run MLflow geni evaluate each time and so we have a solution for you and this is what is known as online evaluation right so when you're creating one of these or editing uh one of these LM judges or editing or or creating a
[14:05] custom one you can actually enable something we call automatic uh eval EV valuation. So what this means is that once you enable that that judge will be run automatically on new traces as they come in. And the important thing to note
[14:18] is it's really convenient. You you don't have to make any changes to the judge in order for for this to happen. You can reuse the same scores, the same judges um without needing to make any changes in between off the that you use for offline evaluation. You can use that for
[14:32] online evaluation as well. And of course LM as a judge, it can be pretty expensive, right? So let's say you're using for example like claude opus 48 right super expensive model uh and maybe when fable comes back you can use that
[14:44] um super expensive model so you don't want to use that on all your traces it can be super expensive right so you can also adjust the sampling rate uh to be under 100% so that you can select a a select portion of your traces to actually evaluate
[14:58] so once you've done the offline evaluation once you've done the online evaluation you can see the results in the traces table that we showed before and it'll show up as like a column Right. So here you're comparing two different runs or or two different
[15:10] prompts. But essentially here um you can see that you know you can see aggregate metrics as to how your your your agent is performing on various uh judges uh like correctness um at the top or you can also see the individual results um
[15:23] per row um below. Okay. So you've created these judges right? How do you know the judges are working as you expect? Who is judging the judges? This is where alignment
[15:35] comes in. This is phase two of the problem that we're dealing with today. So what is the status quo that we've reviewed so far? So we've seen that you can define LM as a judge, right, VML flow. Um you know you can define a
[15:48] custom one, you can define an out of the box one. Um you also have heristic metrics, right? So um remember when you're talking about we could you could build custom judges that output, you know, quantitative metrics, right? So these are things like tool use
[16:00] consistency, token usage, etc., right? But the thing is um you also havememes in your business who really know your domain um and they are like the ultimate judges like they know how to evaluate these traces. The problem is unlike LM
[16:14] as a judge and unlike these heristic metrics they can't be automated, right? You can't just like you can't just have them evaluate a million traces. It would be very cruel. Um no one wants to do that. Um and so you need to find some way to make sure that your LM as a judge
[16:26] that you defined align or is or behaves similarly to uhmemes that can actually deeply evaluate your traces. And so this is what the alignment problem this phase two is all about.
[16:40] So the first step is of course if you really want to align your judges, your elements of judge, your automated judges with um your theme feedback, you got to collect the SM feedback, right? And so we provide you the tools to do that here in MLflow. Um so you can see here in the
[16:54] trace viewer itself, there's this nice little column called assessments, right? And so if you use like something like LM as a judge before, you can see the results in the assessments column, but you can also have a human addual assessments to the uh to the trace
[17:07] itself. And so you can have a human specify, you know, whether a trace is correct, whether it's relevant, etc. And it can they can also provide some expectations for the trace like expected facts that um it should know um expected
[17:19] uh documents that it should retrieve, things like that. Beyond this, we we have an even nicer UI for non-technical users, right? A lot of thememes, they're not they don't want to deal with like a complicated interface. So we have a really simple interface.
[17:32] It's called the review app that allows uh users to really focus on uh the the content itself and and provide their feedback um via like a distinct schema. So like you can specify like a form
[17:44] schema that they can follow to apply feedback on these traces. And so the important thing is that these feedback this feedback thatmemes provide they get stored on the traces themselves.
[17:58] So what does this allow you to do? Well, it allows you to start visualizing how often human feedback feedback aligns with your judges. And so, we don't have a great UI for this today because um it's very, you know, it's very path dependent for a lot of users. Um but
[18:11] here you can see um an open source um piece of software that we developed. It's called Vivecaler that actually visualizes this for you. It's not on platform, but it's a completely separate repo. And so you can see here that um
[18:23] you know for for specific traces you can see whether a human al whether the SM feedback aligns with the judge feedback and if it doesn't it's a good signal that you need to align your judge and you can of course play around with your
[18:35] prompt here um adjusting it so that to to try to make sure that your judge your LM as a judge uh the prompt for the LM as a judge can uh in fact align with human feedback you can try to do that manually but it's a little bit of um you know
[18:49] it's a little bit of a tricky process. doing it manually is is not always easy and often times you're just going to you know you play around and spend a lot of tokens to try to align a judge manually with feedback and so we provide an
[19:02] automated mechanism to do this and the important thing to note here is I know that some of you may be thinking oh I'm going to need so many traces labeled by an theme it's going to be a very difficult process but it turns out you
[19:14] can based on our research you only need a handful of traces uh labeled by anme to start start with this alignment process process and to get good results. Um, and so the code that you need is on your right. It's an automated process
[19:26] that you can do via the SDK. And so again, you fetch your traces. These are the traces with your feedback. And then you specify a judge that you want to align, right? The judge to align. Um, and then you want to align the judge to
[19:39] SM feedback. And how you do that is via this dot align uh method that we provide via MLflow. And all you need to provide is the traces that have the feedback, the SM feedback attached. And of course once you're done with that process you
[19:51] have an aligned judge. I think there is a typo here but um there is an aligned judge and then you can save that judge via this register method so that you can use it for future use cases. So once you save that judge you can use it for the offline evaluation via mlflow.jai
[20:04] evaluate as well as the online evaluation that we talked about a few slides ago. So you have documentation here if you want to take a look. So how does how exactly does this alignment process work? It's actually, you know, you know, fairly technical to describe.
[20:17] Um, and so just in the interest of time, at a very high level, what it does is it dynamically adjusts the judge prompt when evaluating new traces so that it more accurately aligns with SM preferences. And we call this process
[20:29] mem align. Um, so essentially, you know, it's it it finds some highle standards that it needs to adopt. It appends that to the prompt. And on top of that, whenever it's examining a new trace, it tries to find examples that it's seen
[20:41] before that closely match that new trace. examples that have been labeled by anme and appends that to the prompt with the proper verdict. Um so that the judge is more grounded in what anme would expect and it turns out it gives
[20:53] you uh very good results even with a small number of traces. Great. And so we also provide a UI experience via Vivecaler that allows you to visualize how this is working. Um and
[21:05] you know you can see here you can specify the evaluation judge that you use. You can use any model that you want. You're not locked into data bricks at all. You can also specify the the the language model that you use to actually align the judge uh as well as the judge
[21:17] name and things like that. And you'll see the output here. Great. And so, you know, as I mentioned before, um you know, all this alignment is doing is is judge adjusting the prompt, right? We don't fine-tune the model. The reason why we don't do that
[21:29] is very few customers want fine-tuned models. It's very expensive to host fine-tuned models. And so, we try to do the best that we can as rigorously as possible by tuning the prompts. And so you can see that the prompt has changed here uh via this really long prompt and
[21:42] um it's treated like any other judge um that we've worked with before. Okay. So what have we done so far? Well, in phase one, we've worked on tracing and observability, right? We've traced
[21:54] every request um and uh from your agent. We've stored those traces on Unity catalog and we started applying judges on those traces um both for offline and online evaluation. And in phase two, we
[22:07] started doing judge alignment. We started collecting feedback. We've started aligning the judges. And once we've aligned the judges, we have a rigorous judge that we can depend on for online and offline evaluation. And it also can be used for the next phase,
[22:20] which is what my colleague Arthur is going to talk about. >> Awesome. Thanks, Yum. Um, I want you to know how much effort it took to try to get all these arrows to work. And I could not get that last one in the corner to be right. Uh, attempts were
[22:32] made. Uh, let's talk about issue detection. So now that you've got these judge, I can't be too close. Now that we've got these issu these judges aligned, now that we have a good way of assessing how high quality our AI system
[22:45] is, we need to recognize that when we detect these issues, we there are three different ways that we could use to event a novel issue on Git or Jira or
[22:58] GitHub. I I don't care where just track it, right? Please. So there are three ways, two of which MOFlow helps you with out of the box. One is capturing user feedback and then saving that alongside.
[23:10] Yurim covered that in depth, how it's capturing that and how we can then event off of those feedback as a signal that we can log to get like, hey, this is an issue. This is something that's wrong. This is something that we need to fix.
[23:23] Similarly, when you have all of these really well- aligned AI judges that behave as if they were the subject matter experts sitting there, you have the same sort of thresholds that you can
[23:35] set on these judges to then open an issue on Git as an example to help you see like, oh, we can improve this. We can this is a quality gap. And these things happen over time. You might have
[23:48] a lot of high quality answers early on, but as weeks go by, there's a concept drift in your project or new users approach it. Now you're discovering these issues as you have the system live
[24:01] in production. And there are there are insights that are more novel, more nuanced, maybe something you didn't plan for that. We've seen that people use coding harnesses to review all of the
[24:13] traces of your agent to see like, huh, where could this have done better or, you know, take a very critical lens in an unbounded way without clearly defined judges to assess, huh, maybe there's
[24:26] something we could fix about this or more nuanced senses of quality that we should be opening issues for. So instead of you manually tracing and looking for all of these issues across your data
[24:38] estate and the agents that you make, you can now say there are automated ways to find these signals for you and raise them into your git repos, raise them into the places that you have these
[24:51] agents saved is a huge deal. So, how we would apply this in issue tracking is we take this agent that we've made. We've got all of these
[25:04] traces and we did the work for align judges and like let's be real, you might not align judges on the first version. That's where we want you to be. But you might not have perfectly aligned judges and you'll be realigning judges. You'll have a version of your judges. You'll
[25:16] have all these traces. You'll have all this user feedback. you maybe run it through a job to have a coding agent look at it and say like, "Hey, can you find any issues in my traces?" Right? And from these signals, we can log them
[25:29] all into GitHub, Azure DevOps, GitLab, whatever. And then we trigger a runner on something like GitHub to quite literally spin up a coding agent, look
[25:42] at all of your code, look at the trace that's failing or the set of traces, traces that are low quality, they might not be universally failing, and then it attempts a solution at the problem. And
[25:54] through that attempt, it opens a PR. So this is a self you know people I've had a lot of people approach me on this and be like is this reinforcement learning like no but fundamentally the same sort
[26:06] of idea just using agents to bridge the gap of quality. So how does this look in the real world? Um and when I say real world I mean this nice demo that I prepared for y'all. Um
[26:18] we have we have a lot to review. So, I want to bring you up to speed with how our demo has implemented all of the stuff that Nurum has talked about and then some of this issue tracking. So, let's let's remember where we left off.
[26:31] Remember, we had this agent. It's kind of mediocre. It's getting us a first pass answer at these shortages. And you can see the trace of the question that I asked. I think we presented for longer
[26:44] than 10 minutes. So, it showed up here and it was available. And here's the question that I asked the inputs, the outputs, and then on the right side, you're seeing this LM judge. You can see I'll try to zoom in a little bit more, but it doesn't really scale nicely. You
[26:58] can see like, you know, simple judges like safety, completeness, and then as we get uh further down, you can see all the custom judges that we defined like whether or not a substitution policy was within a guideline that we accepted and
[27:11] the rationale for that. And then you can see a little bit further down the workshop judge that we have defined. So we've defined a custom judge to grade this. And we'll talk about that custom judge and how we made it. So this is one
[27:24] trace. Let's zoom out. We've got a lot of these traces that are too zoomed out. You can't actually see them. >> Lots of traces. You believe me? From these traces, we can take a look and see
[27:37] this more summary of all the different questions that have been asked of this agent. I'm going to zoom in again to make it a little bit easier to read. Like all the simple judges that are a little bit more baked in and more universal and then more specific as we
[27:49] get further on like how many impacted customers were identified for this use case. Every time I build an agent, I always sit down and I think about what are the inputs for it? What are the outputs that my LLM generated or my
[28:01] agent generated? And what would I expect that response to be? And from those three columns, I pull out pieces of of a rubric to grade it. So I just keep adding columns on in like Excel and that
[28:13] becomes what I design in MLflow, right? So from these traces, this is where I would recommend you lean into Vibescaler because it helps make
[28:26] the first time that you build this easier. You can go to all of your data bricks uh account teams and ask them, I want Vibescaler. I want help getting this set up. This is something that we will help you set up and run on any
[28:40] agent that you already have existing on data bricks or not. So vibescaler is my entry point as like a user and someone who doesn't really understand all these moflow abstractions and how they work
[28:52] well together. So this is where you get started to assess your agent quality with your subject matter experts. So this is the data bricks the repo for it vibescaler. It's out just came out like a week ago. We've done this in like
[29:04] casual like smaller groups with a few customers before, but now it's public and folks can use it. So ask your account team about this. So this is what Vibescaler looks like. We intake a lot of the questions that folks might ask.
[29:17] You can see this workspace that I have. Um all the traces that I'm going to be ingesting from MLFlow. I discover all of the different pieces of this rubric that I'm actually looking to build. Um, so we
[29:30] can see like what different responses and what questions do I want to design from the questions and answer pairs that that are there. But we're not really stressing about that. That's really baked into MLflow and that's already there. So this is kind of like
[29:42] boilerplate recreating what MLflow has. What gets really cool is this evaluation comparison. And Yurum showed a screenshot of this, but I I really need to stress how amazing this is talking about the side by side like here is my
[29:55] judge. This is the input of the question that was asked. Here's the output that the agent was able to make. And then this is how a user would grade it and then how a judge would grade it. And you
[30:07] get a sense of how accurate are the judges grades compared to how a subject matter expert would grade it. And you can get a sense of that um difference whether or not they match. And then based on how much they match, uh that
[30:21] judge alignment process runs. And that is quite literally just calling that MLflow. fine under the hood, but it's making a nice GUI over all of it. And we end up with these
[30:34] uh optimized judges. So, here's all these pre-baked judges that are already built into MLflow. Here's some custom ones that we started to make like substitution, policy, okay, options,
[30:46] cost aware. But then we we have aligned judges. Where's my aligned judge? Further down. Come on, scroll. Too many. Yeah, that's it. uh the responsible promise no overcommit judge that you can
[30:58] see the evaluation criteria that has been optimized through this alignment process and then we can turn on automatic evaluation so that it runs on all future traces. So remember your
[31:10] judges are a combination of all of these different uh overlay judges that are baked into MLflow, the out of the box ones, the custom ones you design and then you align them to how a user would
[31:22] grade them. From here, couple more things I wanted to show you. Okay, I already showed this. You just build a really simple git runner where you have a coding agent on
[31:35] that git runner that has your repo loaded in. And from these issues that are triggered from your agent, these are all autogenerated. I've got three issues that are autogenerated on this repo that
[31:48] you can see. um different types of quality bugs that were discovered just through users using it, just through quality thresholds, all automatically sent to git. And some might interpret this as spam. I interpret it as signals
[32:02] because you can always filter these out through these tags, right? You don't you don't need to look at them, but they help you refine and figure out where there are quality gaps in your agent from all of these signals that exist in ML flow and elsewhere. And then we we
[32:15] actually act on it. So this is the issue. This is where it's identifying like this is our recommendation, what the agent said, this is what the planner asked for. The responsible promise judge flagged. So there's a problem with that quality in the responsible judge uh
[32:29] flag. So we need to like raise that and look, hey, can we make sure we're giving more responsible promises when it comes to supply chain guarantees? And then we say like okay let's run this
[32:41] um CI runner or be this runner where we run this coding agent and we run all these actions and it's quite literally just looking at all of this code and saying like okay I should probably change the agent py in this way. So a
[32:53] coding agent knows that we're trying to approach this this problem knows the context about the problem just like Ali said a lot of these agents are just lacking the context they need about the business task. This is what we're doing with git here. We're giving it all that
[33:06] context and giving it where it's low quality. And then finally, you can see the pull request that the coding agent made. Uh, hey, here are the changes that it actually wants to apply. And we can
[33:18] guard rail and gatekeep this through different CI processes. So, I'm sure there are questions on this and we've got about seven more minutes of the presentation. I'm going to get into the CI and CD pieces, but if you want to
[33:31] start lining up in the back, if you have any questions, do not hesitate to do that. Let's talk. Let's come back to the presentation and figure out what we've learned. We built monitors.
[33:44] We built judges. These judges should back off over time. You shouldn't run every judge for every question across all these judges. These are sentinels to tell you the quality of your AI system.
[33:57] But you maybe don't have great ways of moving this to production. Right? As I talked about at the beginning, we just kind of threw this in production. Arthur had a fire drill. He had to go and throw this in production. So, it's not really
[34:10] the most stable production deployment. How do we make this more stable? And how do we guarantee that this doesn't happen again, that I don't have to deal with a fire drill, that I don't have to manually fix all of these things? So,
[34:22] that's where CI comes in. And I feel like I don't need to say it out loud because it's on the screen. Um, so how this doesn't happen again is scalable evaluation sets. You start from your
[34:34] development life cycle. You've got this like 10 question. Yeah, I think I want the agent to do this, right? And it's not that wellformed. Typical, right? A lot of these like chat bots people don't really have a great model of all the
[34:47] questions it's going to ask. So you build the first version. You need to ship that first version out. You need to get it past a threshold to get start getting these much higher scale data sets of how people are using this agent
[34:59] and how they expect to interact with it. Then as you have these much richer data sets, what I like to talk about is I use the analogy of self-driving cars. Um it's very easy to make a self-driving
[35:12] car that's 95% reliable. It's not that easy for the last 5%. You've got all these edge cases that you're chasing, right? So, what are these self-driving car companies doing? Um, they're saying
[35:25] like, "Okay, let's make these really targeted data sets and find the places in the world where it's legal to do a a left on red," which is like, "Oh my god, right? You wouldn't think it's ever allowed, but there are actually some
[35:37] places, I think in Ohio, that you can actually do that, and it's totally allowed." So, how do you model that? You have to find the data where there is that left on red scenario. and how do you train for that? How do you test for
[35:50] those scenarios? Right? So, you're finding all of those edge case scenarios that are important to your business and in that example important to driving, right? But they aren't necessarily the most important and and something that
[36:03] you see every day. So, you need to test for these edge cases. You need to consider them. You don't necessarily have them always in front of you. So, you take all these dynamic AI judges and guard rails to evolve as you're building
[36:15] this. So you have focused evaluation sets, you have largecale evaluators, and then your judges evolve as you have more data. You're considering more to the problem space. There's going to be
[36:27] concept drift on what the problem is actually trying to solve. There's going to be changes to what the subject matter expert is expecting. So obviously built automation to capture all three of these
[36:40] pieces rather than all of us ad hoc testing it. So thinking about how this applies to that supply chain agent, we add a CI runner. We rerun all of these nicely aligned judges that we have. And
[36:53] I don't want you I want you to take this as an example of a state-of-the-art. I want you to do something like this. Even if it's mediocre and build like a a kernel of every single piece of this so
[37:05] you have the process down because you only have to build the process once. If you build this process once, right? then you can use it for every single AI agent you move you make into the future as this self-reinforcing loop.
[37:18] So there's a gate that whether or not this becomes this high quality thing like okay is it better than the previous version that's typically the threshold that we set. So this can be really interesting because you'll have a lot of different judges. You say you got 15
[37:31] judges some are more important than others right? So you might get higher quality safety answers, but you get lower quality quality of the answer or or groundedness or correctness. So those
[37:43] two things aren't created equal. So can combining those together gets really important. And then merging or blocking uh on that PR decision is really a check that we have baked in from this this gate at the end. So finally the last
[37:58] phase is obvious obviously deployment. I'm going to take a uh 30 seconds aside on you really need to care about declarative automation bundles. If you don't care about declarative automation bundles, DABs, they're really data
[38:12] bricks asset bundles renamed. You should care about them because they allow you to do I hate all these transitions. I should have taken them out. Um different deployment bundles to different versions of this uh resource that you have
[38:24] deployed. So what do I mean by that? Targeting different deployment environments. You could even target the same environment three different times pointing at three different sets of assets. You could deploy across different clouds. You could deploy
[38:36] across different uh git repos. Doesn't really matter. You need bundles to deploy to different targets across data bricks estates. So it's very important we do this deploy across uh it's really
[38:48] straightforward to do data bricks bundle deploy. You would always run this on like a git runner. You can run it locally for testing but you run this on a git runner typically. Um, and the challenge here, the hardest part is
[39:00] really realizing like all of the things that you need to bundle together in that asset bundle to be able to recreate across environments. And I made this slide because I knew no one this slide, this like concept doesn't exist out there. All the things that you have to
[39:12] think about when you bundle an ML asset or an AI asset out there. So, it might be worthwhile to screenshot this. And I think about all the different monitoring, coding, those sorts of pieces here. So when I bring all this
[39:25] together, I think about Unity AI gateway as a means for governing and reusing all of these coding assets and all these traces. We use these templates, we build it once and we reuse it across our
[39:37] entire AI agent estate and we think about AI observability at the very end to say like, oh, these are the high quality systems. This is how we improve it. This is how we promote new judges across our AI estate. And to close it
[39:51] out, we have Yurum to talk about the conclusion here. >> Great. Yeah, thank you. So, um, we covered a lot of information today, but I just want to, you know, reiterate that, um, this is a a five-phase process
[40:05] that is effectively a loop, right? So, you have phase one. We started off today talking about observability. We want to understand the performance of your agent in production. We want to have end to end observability there. Uh, and we want to get that through tracing in Unity
[40:18] catalog, right? And so once you have those traces in Unity catalog, you can start um evaluating those traces using out of the box LM judges or judges that you define uh and through both online and offline evaluation. But we also want
[40:32] to know whether your judges are actually working as they expect, whether they align with your theme feedback. And so there is this alignment process where you can collect theme feedback and align the LM judges to make them more robust, to make them more um correct uh with
[40:46] your theme feedback. So now you have these aligned judges. You can start doing things with these aligned judges. And one of the things that you can do is automatically detect issues with your agent. Um have there is this automated process where you look at your traces
[40:59] through uh this align judge and detect whether there are some um regressions in agent performance. And so these issues they get filed as GitHub issues as Arthur pointed out or any other uh any
[41:12] other version control system that you you prefer. Uh and of course you can start fixing those issues whether yourself manually or via the help of a coding agent um as Arthur has shown uh and once you have those fixes you want
[41:24] to make sure that those fixes don't cause further regressions and agent performance in some other unrelated domain and so that's where continuous integration plays a role uh and of course once you once you pass the CI checks you want to deploy your agent and
[41:37] this is where declarative automation bundles can help you deploy your agent in an automated fashion without needing to do a bunch of manual And it's important that it's automated because whenever you want to, you know, make automated fixes, you don't want to have
[41:49] to manually deploy these changes all the time. And so once you've deployed your fix, you're all the way back to phase one. You have to start continuing to observe your agent. And that's why this is the full loop here. This is why it's
[42:03] a continuous pro process to improve the quality of of your agent. you can't sit still but we are providing you the tools here at data bicks to do uh this in a much more automated way so that for for virtually all these phases can be fully
[42:16] automated. So that is our presentation. Uh thank you for attending. If you have any questions please feel free to approach us or approach the mic uh right now. Thanks. >> Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.