Building Production-Ready Agents with a Self-Evolving Test Suite

Summary

This video diagnoses the vibe-check regression loop—where each coding-agent fix risks breaking previously passing behaviors—and introduces MLflow's self-evolving test suite as the solution, automatically converting every feedback event into a persistent automated test.
MLflow 3.14's new command instruments a LangGraph agent with a single line of code, capturing every tool call, model invocation, cost, and input-output pair as a trace that the test harness uses to build and run a growing pytest suite against each subsequent fix.
The video demonstrates the full workflow from a prototype with no tracing to a CI/CD-ready agent backed by a real test suite, including domain expert feedback collection via trace review and Genie-powered analysis to surface top quality issues across dozens of feedback entries.

Building Production-Ready Agents with a Self-Evolving Test Suite

Building agents is challenging. Most teams iterate on agent quality by vibe-checking. Try a question, fix what looks wrong, ship. It works at first and breaks down fast. Each fix can revive an old bug. Coding agents make this worse by rewriting the agent in seconds while verification stays manual.
this video introduces a new best practice. With one MLflow command, your coding agent turns into a rigorous builder that grows a test suite from inside the vibe-check loop. Every piece of feedback on a bad answer becomes an automated test. Every coding-agent fix runs against the accumulated suite, scored and recorded in MLflow. By the time your agent is ready to ship, you have a real test suite that grew alongside the build.
We walk through this on stage end to end. We start with a prototype agent that has no tracing and no eval, and we end with a production-ready agent backed by a test suite that grew alongside the build.
Talk By: Adam Gurary, Senior Product Manager, Databricks ; Yuki Watanabe, Senior Software Engineer, Databricks ;

Chapters

FAQs

What is the vibe-check regression loop problem in AI agent development?

The vibe-check regression loop is the pattern where developers informally test their agent, fix a problem they notice, and then discover that the fix breaks a behavior that was previously working. This video explains that coding agents make this worse because they can rewrite the entire agent codebase in seconds without checking against prior test cases.

What is MLflow's self-evolving test suite and how does it work?

The self-evolving test suite is a capability in MLflow 3.14 that intercepts developer feedback on bad agent answers and automatically converts each feedback event into a pytest test case, building a growing test suite alongside the development process. This video shows how a single MLflow command instruments an agent to capture every tool call, model, cost, and input-output pair as a trace that feeds into the test harness.

How does MLflow tracing help improve AI agent quality?

MLflow tracing records a complete execution trace for every agent run, including which tools were called, which models were used, what each step's inputs and outputs were, and what the total cost was. This video demonstrates that having these traces available allows coding agents to make context-aware fixes rather than guessing from code alone, and enables domain experts to review and annotate traces at scale.

How do you move an AI agent's self-evolving test suite into CI/CD?

This video walks through the full path from a manual feedback loop to an automated CI/CD pipeline, where every new coding-agent fix is automatically scored against the accumulated MLflow test suite before being merged. The Genie-powered analysis step surfaces the top quality issues across large batches of feedback so the most impactful problems are addressed first.

Full transcript

[00:10] Uh, okay. So, 5:20 p.m. last session of the day. Hopefully, everyone's alert and awake. Thanks for coming at uh at such a late time. We appreciate it. I'm Adam. I'm a product manager on our AI team here at Data Bricks. >> And I'm Yuki. I'm a tech lead for OSS
[00:24] team. And today we are going to be talking about how to build production ready agents with our new self-evolving test suite. So show of hands, how many of you have
[00:37] fixed one thing with your AI agent only for it to break something else? Okay, that's literally what the whole talk is about today uh in essence and uh and it's the problem that uh we're hopefully solving for you guys. So
[00:51] here's this loop that we've been seeing in our own workflows and in the wild with a lot of customers. So you have an agent and you're building that agent using a coding agent and you vibe check
[01:04] the agent and maybe something isn't quite right with the answer. Maybe it's like too verbose, maybe it's hedging or maybe it's not leading with the right thing. But anyway, you fix it. And these days that means pasting the feedback
[01:18] into cloud code and saying hey like fix my agent it's not doing the right thing. Claude code goes and fixes the agent and then you try the question again and it's better. And then fast forward a little bit. Maybe a stakeholder tells you hey I need
[01:32] another change in the agent and you do the same exact flow. You go to cloud code. You paste the feedback into cloud code and then cloud code goes and edits the whole agent code again to try to reflect the stakeholders feedback. And then you try it and it works against the
[01:46] stakeholders feedback, but then you try it against the original question and it fails. So one fix actually overrode the other. And here's the thing with this loop.
[01:59] Every fix is reviving is potentially reviving old bugs. And coding agents are making this so much worse because they can just rewrite your whole agent in like a few seconds without checking
[02:11] anything while you're checking is staying completely manual. You're still vibe checking. And in a lot of cases actually the coding agent isn't even looking at any of the traces. It's just looking at the code and kind of making some educated guesses.
[02:24] So it never saw the execution. It never saw what tools got called. It never saw what models got used. It didn't know the inputs and the outputs of every step of the agent. And meanwhile, you don't have time to literally revibe check
[02:36] everything you've already vibe checked before. And your fixes are competing with each other instead of compounding. And you can kind of stay in this loop forever and never really feel comfortable to get to production.
[02:48] So here's what we built. It's pretty much exactly the same motions you're already doing, but a slightly different loop. You still do the vibe checking. You still give feedback to your coding agent about what
[03:01] your agent is doing wrong, but now we're putting MLFlow inside of that loop. We're going to make sure that every single piece of feedback you give to your coding agent about your agent becomes a permanent test.
[03:16] And so every time your coding agent ships a fix, it's going to run against everything you've ever marked as a bad answer before. So old fixes are going to stay green. And that's the difference between the new loop and the old loop. The old loop,
[03:29] a lot of us have been running, your fixes are competing. And in the new loop, your fixes are going to compound. And the reason you should care is very simple. You're just going to get to production faster. And not because any
[03:41] individual fix is going to be faster. It's just because you're going to stop moving backwards and revive checking the things that you've already vibe checked before. And so by the time the agent is ready to ship, you will have accumulated a test
[03:54] suite for the agent. And when that suite is green, you're ready to ship. So we're going to do this hands-on today and show you exactly how to do this. Uh we're going to do a real agent
[04:06] with cloud code with MLFlow and implement this exact new self-evolving test suite loop. Uh but real quick before we do the demo, I just want to level set about ML flow.
[04:18] If the last time you saw MLflow was your like scikitlearn pipeline and like model registry and all that, it's definitely not the ML flow you remember. Over the last couple years, we've made MLflow into a first class Python package for
[04:32] Genai and agents. It works with Python, with TypeScript. It has native integrations to trace all of the major agent frameworks and harnesses and anything else can just plug in with open telemetry. So, it's
[04:45] all open source. You can run it anywhere. Uh, and what you're looking at here is MLFlow tracing. So, anytime your agent runs, MLflow is going to capture literally every step of the interaction and execution of the agent. So, every
[04:58] single tool call with the inputs and outputs of every tool call, the latency of every step, the token count of every step, the cost of every step, what model got called, literally everything in the agent execution. And so when an answer
[05:11] is bad, you're going to look at the actual trace to see what happened instead of just guessing. Or even better, you're going to have your coding agent look at these traces uh to figure out what's going on. So keep that in
[05:23] mind with the coding agent. This is a very very big deal. And that's that's going to play into the self-evolving test suite that we're going to show. And the second thing, we've made MLflow into a really sophisticated way to grade quality for agents, too. So we have this
[05:37] whole library of built-in judges uh that you can use with MLFlow and run out of the box. Uh and of course you can define your own custom judges uh and scores. And so what you're looking at here actually is a set of judges are
[05:50] grading a set of traces. And some of these judges are like out of the box like generic things like correctness uh and uh contextual relevance and some of these are going to be custom judges. Uh but basically all of these judges are
[06:02] going to run against your traces. So that quality becomes more of a number instead of like a feeling. Um so I'm actually going to skip that part. So that was ML flow until
[06:14] yesterday. And then literally today in MLflow 3.14 we shipped one last thing. Uh and that is this self-evolving test suite. And uh yeah, the whole thing is set up by one command in your terminal.
[06:27] Uh it's uvx mlflow agent setup. And this is going to make your coding agent into an absolute genius at using MLFlow and it'll onboard your coding agent into
[06:39] that very opinionated loop to create the self-evolving test suite. So anytime you're going to give feedback to your coding agent about your agent, it is going to go to the traces, the actual
[06:51] traces. It's going to do a root cause analysis to see what went wrong. And then it's going to write a test that locks that fix in. And then every single change you make to your agent is going to run against that full suite of test
[07:05] of tests that you're accumulating before you ship anything. And so you keep doing exactly what you've been doing. You use your coding agent to build your agent. You vibe check it and you give feedback to your coding agent of what's going wrong. And you're going to have this
[07:17] whole test suite just building up underneath you automatically. So, there's going to be three steps to the demo here. Uh, number one is you're just going to we're going to show you how to set up your coding agent to do
[07:29] this whole loop. And then number two, we're going to show you how every piece of our feedback is going to become a test. And then every change you make is going to run against those tests. And then finally, as you scale up, maybe you want to get feedback from your peers,
[07:41] from some subject matter experts. You can take all of their feedback and turn it into tests, too. And then you have a test suite that you can take to CI/CD. So, I'm just going to set the scene real quick. Uh, imagine me and Yuki are
[07:53] engineers at a company called Omni Bricks Supply Company, uh, where we sell bricks online. Uh, so we sell bricks like the Cinder block classic and the Legacy Red Brick and the Brick Father.
[08:06] Um, so our job is going to be to build a customer support agent for our website. So, if our customer asks us like, "Hey, what brick should I use for my red wall?" uh you know hopefully we're going to recommend them the right brick and we
[08:18] want to make this agent good without breaking it along the way. So I'll just briefly show you what this app is going to look like.
[08:30] So we have this app here uh and we have the chatbot here and uh and yeah so Yuki uh oops
[08:44] yeah Yuki if you want to uh take us away and uh and set up the agent and take it away from here. >> Sure. Okay thanks Aram. Uh so here's my screen but um I'm a developer in Omni bricks supply core and I'm building a
[08:57] customer support agent and I want to be clear this is just a prototype right now we have like single just a run graph agent sorry I'm I choose a wrong action here so this is a lang graph agent it's very simple there's no tracing no
[09:10] there's no evil there nothing about m4 right now so this is the starting point the first thing I do actually first thing I do is actually make it full screen so that you can see the more clean. Okay, the first thing I do is run
[09:23] the MLFlow agent setup command. So just go back to my terminal here. I'm using ghosti and just type things. So I'm using u run but m agent set app.
[09:38] So what this command does is um so there's some download happening here uh because uh it's running a UV it's install mfro dependency right now and after that happens there's some question
[09:50] pop up like this it first ask which coding agent I should use this this command support multiple coding agent but here I just use cl code and if asking whether if I should if you
[10:03] should use install mflow skill or not so I just click yes And here's a magic happening actually it launches a cl code and is instruct mro to do all the setup. So let me show actually uh explain
[10:15] what's happening here. Another foot there's a three steps happening. So first it on boards my coding agent cloud code now knows you can see the prompt there right. So you
[10:28] can cl code now knows ml's best practice how to query traces and how to do eval. And second it install mro skill just like say yes. So install the mro skill is a kind of instruction set of
[10:41] instruction that describe all the best practices and also the loop we are trying to demonstrate here. And the third is instrument my agent with tracing. So my agent is lang graph. So
[10:54] it figures out how to trace langraph agent how to instrument that and yeah so and in during this execution I didn't need to write any code because coding agent does all things and after
[11:06] the execution done then now I have a tracing setup already. So let's take a look at how it goes right now.
[11:18] Okay, it's still executing. So let me show the empty experiment first. This is ML4 experiment. So right now we don't have anything. This is empty. This dashboard doesn't have any trace. This doesn't have any latencies, errors because there's no tracing. This is the
[11:30] starting point. And if I go back to coding agent hopefully it does something. Okay. So it editing my code, it set up tracking URI. It call log which is a command to command to
[11:43] instrument langraph agent. So since it does a few more verification but let me go back to the agent to actually see if tracing works or not. I'm go back to this agent and ask question or maybe
[11:56] just say hi. Hi agent and agent return something. Hey welcome to brick sub call. So let me go back to the mro experiment
[12:11] and if I launch this is actually traces. There are the two because actually in the coding agent runs verification to generate some domy traces but the one of them are actually my trace. So let's take a look. Okay, this page shows all
[12:23] the traces. So each row corresponding to one execution of agent. So the first one high agent if you open this the trace records all the details
[12:35] under the hood. It runs all the langraph node. It has input corporate input prompt and output response. It has token count. It has cost. Every details are here. So this is a foundation of
[12:47] everything what comes next. So when the coding agent root calls the issues that I'm saying now it looks at those traces and what's happening inside what kind of prompt pass what kind of tools are called instead of guessing what's
[12:59] happened. And one quick note here is everything you are seeing here in this demo is open source. So you can just run it on your machine or you know on premise server but if you're using database data bricks
[13:11] maybe I think many of you guys are data customers then uh you can actually point this to unity catalog. So if you're using the data bricks these traces MFO traces are stored under catalog. So you can get all the governance for example
[13:23] access control PI reduction or downstream analysis for example DBSQL or lake cost monitoring everything is all set as soon as you in you expose a trace to uni catalog so if you're using data
[13:36] bricks please check it out and I highly recommend enabling that feature all right so now we have trace so let me go back to the slide
[13:50] so now we have a tracing so all the setup is So step two, this is a core part of this whole talk. Feedback becomes a test. So let me check my agent just like um meas do. So let me go back
[14:04] to the agent again here and refresh the conversation and that's something like um what do you recommend for my red wall? I recommend red ball. I send it and my
[14:18] agent is running and it answers something. Okay. So it recommend legacy red recommends cinderbrook cas it recommend brick father and it shows some sort of
[14:31] pricing. This is not incorrect but if you're a customer this is too long right I just wanted to know like one bricks that fit with my red wall but it's actually hinging a lot of details are
[14:43] here and two verbers. So I want to change this and now normally what would I do is I just paste my feedback my this conversation to cl code and ask something like hey please make this
[14:55] consist and maybe also please only recommend one thing right and without ML flow what going to happen is it grow code just jump straight into the solution because I ask make it concise
[15:08] it just directly say hey yeah I can make it concise and go to system prompt and edit something like hey you should answer concept or something. It doesn't verify about trace. It doesn't verify anything
[15:21] because it doesn't also know about test. So it never it never check whether that system prompt edit breaks anything else. This is like how exactly the red Adam
[15:33] showed in the like last slide. The block of red happens and next your next regression actually comes in. So, oops. Oh, yeah. I need to go back to
[15:45] slide. So, here's what I do instead. And okay, I actually going need to go back to the console here.
[16:00] Okay. So, the tracing setup was also done. What I going to do is just same. I didn't do anything else, anything more. I just ask, "Hey, my brick agent response was too verbous. Please make it concise and lead with one
[16:12] recommendation." Okay. Hopefully it's transcribed pretty well. Nice. So I just send it that this is same, right? I didn't do anything
[16:24] else. I didn't tell it about like all tracing. I didn't tell it about ML flow. I didn't take anything about a variation. But if you see this, it actually load one skill fix agent issue. So let's take a look at what actually it does while it's executing.
[16:43] First step explore plan implement and verify. This is what ML for skill instruct close code to do this for. And if oh sorry and if you have ever done like testdriven development this is pretty similar maybe you might be
[16:56] familiar with that. The first step is explore. Instead of jumping into a solution, now clone code explore traces explore the codebase to understand what exactly happened. So in
[17:08] this case, I just say hey answer two berbers. So now cl code look at the answer to understand what was burbus and why it is called. Next step is plan. This is important part. So before touching anything any code any system
[17:21] prompt it first plan how to verify the fix how to define the goal and how to verify it. So at this phase the crawl call is okay. So this is the test case I need to run. This is the test how how I judge to
[17:34] verify this and only after that step start to implement the fix maybe system prompt edit maybe to go edit but only after verification uh plan it start implement test and after that it run the
[17:46] test case I generate and also runs every test that is already existing in the repository. So it doesn't cause regression and CL only say done after all the test cases becomes green. So now when this
[17:59] says done it's not just editing system prompt but it says okay here I verified a fix works and all the regression is checked. So let me take a look at the console and it's probably still executing but
[18:12] it's executing some verification step I guess and going to yeah plan step and going to implement step. So let me take a look at the how the test actually looks like.
[18:24] So this is some test case I kind of pre-generated yesterday with all the same flow and under the foot this is just pi test maybe most of you guys some like touch
[18:37] python in some way if you're you doing agent development maybe this looks familiar there's a single test function for each test case it says h maybe small but test should lead with one recommendation when customer is
[18:49] undecided there's a test prompt what do you recommend for building a red Oh, this is just exactly my question I asked and there's a mro API for evaluation with some guidelines which is LM judge
[19:03] which that which defines a criteria this test case validates for example response should lead with one ex one specific bricks product and if you look at the second one response should be concise no more than three four sentences.
[19:16] So when the agent or maybe us run the pi test this is what uh what exactly validated the agent is executed with a prompt and verified with this natural rankage criteria using mlfm judge
[19:30] so and then since this is piest it's also naturally integrated with all the pyest x system so you can use like pyest plugin you can parize that or you can
[19:42] parameterize that with different questions which just adding this mro test decorator you can get all of this combination of pyest and mfro in your codebase and this is not blackbox like
[19:54] this is this sits in your repository so if you think prompt is a bit different from what you want to test or guideline is a little bit different you can just go and edit so this is totally not blackbox all right so hopefully the the job is
[20:07] finished let me showcase some the test k test actually test result so if you if you decorate your test with mfra test decorator The result is direct logged into MLFlow as a run. Every test run basically pi
[20:22] test session corresponding to run ML for run and if you open it you can see all the test example executed in the session and you see this column this is a where
[20:35] the result of each assertion like natural rangage criteria. So in this case it checks like whether the response should be concise whe the response should be should have one recommendation or not and recall the result inflow. You
[20:49] can open each result to see that what agent input and output is. You can also see the LMJ relational why this judge think this test case pass or not. And you can even see the traces to drill down more into why this test case pass
[21:02] or fail and only after this test test case is bring the agent say done. And now I can review the agent code like fix. All right. So now we have these three
[21:15] tests but uh and we just make the agent concise. But let's see why this matters. Okay. Let me go back. Sorry. Let me go back to my agent.
[21:28] >> Hey. >> Sure. >> Of course. And this one. Okay. So here's what's make what make the work interesting. When the sales team comes in and ask a question to me, hey brick
[21:42] father this the bottom gray one is our highest margin product. So can we actually push brick further more? So basically asking my asking me to kind of make my agent recommend brick further in
[21:55] general. This is total reasonable ask right? This is sort of reasonable business ask and I guess like majority of us have encountered this kind of ask and once or twice. However, without ML Pro, what is
[22:08] happen is the coding agent just added one blanket rule of recommend brick father and then my last question if you remember asking hey what do you recommend for my red wall start to recommend brick father
[22:22] which is great but nobody can catch this because nobody remember my question nobody in the team remember about my red wall question so it just go ship to production and then one day customer comes in and ask about red wall or brew
[22:35] wall or something and then get brick fuzzer and then get angry. This is how those regression happens. So let's take a look at the what happened instead in MRO.
[22:52] So in order to just save time I just showcase how this tests uh what happened yesterday. So this test test runs are generated by executed by coding agent. And let's take a look at one result here.
[23:09] So for this one, this test case, I'm looking for bricks. What do you recommend? The agent answer is great choice. Brickfather is our premium sound dumping brick blah blah blah. This is correct, right? Customer doesn't have any like specific requirement. So we
[23:22] just want to recommend breakfast. So this is a result of coding agent edit system prompt to recommend breakfast and it looks good. If we take a look at next test case, the
[23:36] question is I need red bricks for heritage restoration project and the recommendation is for heritage restoration brick brick father is a great default pick but brick father is great. So imagine if we don't have this
[23:48] test suite this just go ship to production and customer get red a brick father. However, now we have MFO test case and the agent coding agent only says done only shape it when all the test case green and this test suite
[24:02] catch this exact regression. So this shows fail. So now code can go back read the test result read the traces and iterate until all the test becomes green. So you can see a few more run uh
[24:16] cro code is running a few more tests. It just climb the test pass rate 0.5 0.8 eight and one and only after that close says it's done. It's now all the test cases are green. If you take a look at
[24:29] the red bricks recommendation, now it says for heritage restoration I recommend for region red bricks. So it's a correct answer. So with this kind of like fixed goal, cold call is pretty good at hill climbing and make the correct uh edit to the uh to the agent.
[24:47] Let me take a back uh let me take back and maybe test like test this. This is break fuzzer. So for example, if I ask something like what you recommend by uh sorry uh
[24:59] what is the best bricks and recommend breaker if you refresh
[25:11] what do you recommend for my red bull and now it has a legacy So yeah now sales teams are happy
[25:23] customer asking legacy red bricks is happy. So now this is how like regression is captured and this is how ship ship like safe is f uh ship ready feels like. All right so this is the wrong way but
[25:36] uh like go back to my slide. Now we just we just fixed one duration. So which bring us to the step three scaling because so far all the test case
[25:50] come from me but I'm a developer and I'm honestly not so familiar with bricks but in the company there's a lot of bricks expert so their feedback is worth 10 worth more than mine but the question is
[26:02] how we gather those feedback effectively and how we harness that. This is where mlf can help. So with MLFlow if you already have traces
[26:14] you can go back to experiment like this you can select the traces like this and directly assign to the people bricks expert for example this case Erin or Dave or Carol assuming they are bricks
[26:28] expert we can directly assign from here and and once you log into ML flow they can find the list of traces assigned to them from Here for example, Alice has
[26:40] the five traces assigned. They can just click start review to see to see the input and output or for each traces to see the all the like per computer question they want to uh
[26:52] answer. So in this case like is a product recommended is a large choice they can say like yes or no and then a so all the like uh they can see all the focus review experience from MMO.
[27:05] Alternatively, sometimes you also want to let those expert directly interact with MFOR no sorry agent rather than fixed set of traces. In this case, you can use review app in data bricks. You
[27:19] can deploy your agent to ser end point. Then it you you can deploy this review app which is interactive uh playground with your agent, right? And if you if if the domain expert logs in and asks
[27:32] something, they can see the feedback UI like this. Similarly to the review app in ML tracing and they answer like read the feedback like this and then all the feedback is logged to ML tracing. So now
[27:44] as a developer I can see all the feedback in the experiment and assuming the week goes by and now I have all the feedback logged in into ML flow. The problem is maybe I have like
[27:57] 50 or 60 different feedbacks. How can I deal with that? I don't I don't want to like read all the feedback one by one, categorizing them, prioritize them. It just takes a lot of time. But now,
[28:12] >> oh, >> can you zoom in? >> Okay. Sorry. All right. So now if you are in data bricks, one good thing is you have genie. Genie understand traces. Genie
[28:24] understand if Genie understand feedbacks. Now you can so if you go if you go to MF experiment and just open Genie and ask something like hey please find top five quality issues. Genie just
[28:36] iterate on traces, read the feedbacks and summarize, give a very comprehensive report of quality issues like this. Issue one agent fabricating, issue two, safety, issue three. And what I going to
[28:49] do is you can maybe predict just I just copy paste this encoding agent and then all the issues now go through the explore plan explore plan implement verify flow. So
[29:02] we can verifiing
[29:14] All right. So let me go back to the slide. Okay. And then after that you probably have a very very good test suite. Maybe it was just like three or four test
[29:26] cases. But with all the feedback from domain expert you probably have like tens of more test cases. And now you can put it to your CCD because it's just pi test. So whenever you can run pi test you can run mro test as well. So maybe
[29:38] you can put into the github cscd and run it on daily or maybe on the PR to make sure all the feedback from doing expert are constantly checked and does not regress anymore.
[29:51] All right. So hopefully this gives you some idea of what we are trying to do and how we prevent the regression. And with this just back to Adam. Thank you, Yuki. So, let's recap what just happened
[30:04] because a lot just happened. So you vibe check your agent, the thing that literally everyone is already doing. And any bad answer, you just give a sentence of feedback to your coding agent, a few sentences of feedback to
[30:16] your coding agent, and your coding agent is going to go into the traces, root cause the issue, and it'll fix the issue and make a test so you can detect if that issue ever happens again. And then
[30:30] from then on, every change you make to that agent will run against all of the tests that you've accumulated. And so old fixes are going to stay green. That's how the brickfather rule shipped without uh without breaking the red wall
[30:42] fix, right? And then it scales up. So your experts are going to give you feedback. They're going to try the app. They're going to give you feedback. And you're going to have your coding agent look at all their feedback root cause any issues there and
[30:54] make those into tests, too. And so by the end, you have this like massive suite of tests that you can use in CI/CD. And I just want to call this out for what it is. This is eval. It's just
[31:06] maybe not the eval that you've been dreading. So, traditionally, Evval has been before you get any value back, you got to do a golden data set. You got to
[31:18] set up custom judges. Maybe you got to align those judges to humans uh with the experts. And uh and all that is real. It's valuable. It's just a huge upfront cost, right? And it's why a lot of teams
[31:32] stall in that doom loop uh of vibeing forever. And so we just inverted it. Basically, you start evaluate
[31:51] set on day one. You don't have to do custom judges or or judge alignment on day one or anything crazy like that. And this is a suite that can actually grow and potentially actually get you all the way to production. And these are going to be real PI tests
[32:04] based on real MLflow scoring. And then maybe later as you learn more what matters for your agent, you can do things like use the built-in judges and the custom judges and do things like judge alignment and production monitoring. So you're not skipping eval.
[32:18] You're just bootstrapping it from the very first time you try your agent. And then you can grow into this test suite and you can grow into eval
[32:30] gate your progress. So to be clear, we're not saying that all the rigorous stuff doesn't matter. It really does matter. And once your agent ships, production is going to be its own discipline.
[32:42] Maybe you got to do monitoring at scale. Maybe you'll end up doing jud judge alignment. And maybe you'll do observability across a whole fleet of agents, right? And there's actually a whole session on this tomorrow. It's called how to observe how to implement
[32:54] observability to reduce agent sprawl with Arthur and Yurum at 150 uh tomorrow. And the line in between our two talks is simple. Today the talk was
[33:06] about getting your agent to ship and then tomorrow is everything that happens after. And one more thing I want to mention just because I think it's super super cool is this isn't just something we built for you and we're not using ourselves. So if you've ever used like
[33:19] the MLFlow assistant or Genie code, this is exactly how we build them. We vibe check them. Every feedback becomes a test and that test suite grows. And yes, over time we've created like a super
[33:31] robust Evaluite for all of these things. But we're actually still in that loop today. We literally just shipped improvements to the MLFlow assistant today and MLFlow uh 3.14 that came from
[33:43] doing the vibe checking and doing the self-evolving test suite that we showed you today. So, our agents are getting better this way. I think your guys's agents will, too. So, here's the thing I want you to do tonight. Go to your terminal, run UVX
[33:58] MLFlow agent setup, and run it inside your projects with whatever coding agent you use. It's cloud code or cursor or open code. Uh, it's going to make your coding agent a genius at using MLflow,
[34:10] and it will force you to go through uh that exact flow by default. So all you have to do is build your agent the exact way you're doing it already, vibe check it, give feedback to your coding agent, and then you'll watch this test suite
[34:23] grow up underneath you automatically and it'll get you to production faster. So before we wrap, I thought it'd be great to share some of the roadmap stuff. Some of you guys are building some really crazy stuff with your
[34:35] agents, so uh we have to keep up. So first thing is one crazy thing you guys are doing is like one agent has like 30 or 40 tool calls in a trace. Uh and so the standard trace view that we've been
[34:48] showing to you guys uh is maybe getting a little bit out of hand. Uh so one thing we're investing in right now is custom views over traces so that you can view your traces in the way that makes most sense for your use cases. The second thing is tracing coding agents. A
[35:01] lot of you guys have already started doing this uh and uh and a lot of you guys wanted to start doing this. So things like how the coding agents are executing their skills, where they go wrong, or maybe if you're an admin or a
[35:13] manager, you want to see how people are using their coding agents and how they're getting value out of it. Uh so we're also investing here uh in terms of tracing the coding agents. That's already uh and we're already working on potentially how to do eval here as well.
[35:25] And then finally, judge alignment. So the tests you saw us create today were checking pretty narrow concrete criteria. you know, the kinds of things LLM judges are going to be pretty good
[35:37] at, like out of the box. But as you mature and as you maybe want your judges to grade the traces in a very similar way to your experts would grade those traces, you maybe want to start doing
[35:50] judge alignment. Uh, and historically, this has been pretty hard. So, we want to make this way, way easier. Uh, and we're investing in that. And tomorrow's talk is going to go super deep on this. So, uh, so definitely check that out. So, that's the talk. Run this command.
[36:03] UVXML flow agent setup. I'm Adam >> and I'm Yuki. >> Thank you guys so much for coming. Hopefully this was useful.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.