Evaluating AI in Production: A Practical Guide to Continuous Improvement

Summary

AI evaluations consist of three components — a task, test data, and a scorer — and should begin after an MVP launches so production data can inform meaningful test cases rather than block initial deployment.
Agent evaluations operate at three levels: end-to-end outcome quality, step-level correctness, and trajectory efficiency, with multi-agent systems requiring additional measurement of routing accuracy and handoff quality.
Platforms like Braintrust combine observability and evaluation in one interface, enabling engineers, product managers, domain experts, and data teams to collaborate on continuous improvement throughout the AI system lifecycle.

Evaluating AI in Production: A Practical Guide to Continuous Improvement

Evaluation is a continuous process, not a one-time checkpoint. It becomes critical once you launch an MVP and production data starts flowing. AI systems demand different evaluation strategies than traditional software because outcomes are uncertain, data is noisy, scores drift over time, and teams need clear roles to define success criteria, run experiments, and iterate rapidly.
This practical guide covers the three components of any eval (task, test data, scorer), clarifies when to start evaluating and what data volume you actually need, and details how engineers, product managers, domain experts, and data teams collaborate throughout the evaluation lifecycle. Learn how to structure agent evaluations at three levels (end-to-end, step-level, trajectory efficiency), evaluate multi-agent coordination through routing accuracy and handoff quality, test multimodal agents with audio and video, and use remote evals to safely validate changes against your production system without deploying. Understand the tradeoffs between code-based scores, LLM judges, and human calibration, plus how platforms like Braintrust and MLflow integrate observability with experimentation for iterative improvement at scale.
🤝

Chapters

FAQs

When should a team start evaluating an AI system?

Teams should start evaluating once the MVP has launched and production data begins flowing through the system. Using evaluation as a prerequisite for launch is discouraged because real production traffic provides the most relevant test cases and reveals failure modes that synthetic data often misses.

What are the three components of an AI evaluation?

Every eval has three components: the task, which is the AI system or agent being assessed; the test data, which is the set of inputs used to exercise the system; and the scorer, which is the mechanism that judges the quality of each output. Scorers can be code-based, LLM judges, or human reviewers depending on the nature of the task.

How do you evaluate a multi-agent system's coordination and handoff quality?

Multi-agent evaluation requires measuring routing accuracy — whether the supervisor agent delegates to the correct sub-agent — and handoff quality — whether the context passed between agents is complete and accurate. These coordination-layer metrics complement end-to-end outcome scores to pinpoint where failures occur in the system.

What are the tradeoffs between code-based scores, LLM judges, and human evaluation?

Code-based scores are fast, deterministic, and inexpensive but limited to tasks with objectively verifiable outputs such as JSON validity or exact string matches. LLM judges can assess more nuanced qualities like tone or reasoning quality at scale but require calibration against human labels to remain reliable. Human evaluation is the ground truth but is expensive and does not scale, making it best suited for calibrating automated scorers.

Full transcript

[00:02] As long as you're continuing improving product, as long as there new versions of models coming every day, you have to do emails. We always recommend folks to start evaluating once you have launched your the MVP of your product. Once you will start seeing the production data coming through your AI system. And
[00:18] because we we don't want to use email as a kind of a blocker for you to launch your product. Hello everyone. Uh thank you so much for having me at this meet up. So my name is Mengying. Today I'm so excited to be here to share
[00:34] a topic that is very close to my heart, which is how do you do uh how do you evaluating AI in production? And also I have never heard about Koresh's presentation. This is my first time. So you might hear some very similar concept and similar content
[00:50] compared to his presentation, but from a different angle and using a different UI because I'm from Trust. Uh but hopefully after you heard about the same the the same concept twice, you are able to apply, you know, the concept the skills to your day-to-day workflow
[01:07] afterwards. Little bit about me. So my name is Mengying. I'm currently leading data and product growth at Brain Trust. Previously I was at Mother Duck, Notion, Meta, and my and Microsoft leading their data and growth team, too. So for folks who have never heard about Brain Trust, Brain Trust is the AI observability and
[01:25] the evaluation platform. By combining observability and evaluation into one platform, we allow builders to really to have a good understanding and the visibility into what's going on in AI in their AI system and also give them the tools to keep improving their AI system.
[01:43] So teams at Notion, Dropbox, Lovable, uh even OpenAI, Replit, they're all using our platform to improve and make their AI AI system better. So I'm going to start with a very quick introduction to the foundation and then going to about a
[01:59] common eval process and then lastly I will talk about AI agentic eval flow. You probably have heard about evals a lot. So, evals is basically just structured experiments that help you assess and improve AI application performance. So, I'm from data background. So, every time I try to
[02:15] explain evals to people, I always want to use the AB testing analogy because it's basically the same. You compare two versions of AI system. You want to know which one is which one is is performing better. So, they consist of three parts.
[02:31] The task, the data, and the score. So, the task is basically your AI system. It can be your agent or it can be a simple prompt. And the data just the test examples. Um and the score is basically the functional model that mirrors the
[02:48] quality of the output of your AI system. So, think about your basically apply a task on your data and then you are going to come up with some response and then you use a score or or curator to really to measure the quality of that response. That's eval.
[03:04] It's very straightforward. Why evals matter so much? To be honest, before I joined Braintrust last May, I never heard about evals. But I feel like since last year evals becomes a very popular concept in in the AI world. The reason why evals is so
[03:21] important becoming more and more important is one is AI interactions are very uncertain. You probably have similar experience as me even you ask the same question to your chat GTP, they might have different answers. And secondly, the data is very noisy. You
[03:37] know, previously you are working on the you know, the typical like product features is all like kind of like structured data like whether the user click into certain things, whether the user has engaged with your platform. But in the AI world, the data is is very
[03:52] noisy. They have a lot of unstructured data. And also, you know, as uh your AI system keeps evolving, developing, you might have new data stream coming the all the time. And the third one, the scores are just not perfect. And also, a lot a lot of scores are powered by a
[04:08] model as well. And the And the same problem applied to scores, which means, you know, the model can change. So, that's why you have to keep evaluating and keep monitoring the performance of your experiment. And then, people also ask, "Well, I should start eval. Uh I'm
[04:24] a startup founder. I haven't launched any product yet. Should I eval right now?" Uh the answer is no. And in terms of when to eval, uh we always recommend folks to start evaluating once you have launched your the MVP of your product. Once you start seeing the production
[04:40] data coming through your coming through our AI system. And because we we don't want to use eval as a kind of a blocker for you to launch your product, right? Because we don't want you to feel like, "I need to be so perfect. I I need my AI system to to to to be so great before I can before I can
[04:57] open it up to uh uh to my customer." Uh instead, you should launch your product first, and then you collect data, and then you start iterating and and improving your your AI product. And when you are done with evals, so unfortunately, there is no done here
[05:14] because evals are just a CD process. As long as you continue improving product, as long as there are new versions of models coming every day, you'll have to do evals. And also, people asking, "Do you need a lot of
[05:29] data to do evals? Do I have to, you know, have like millions of rows in order for me to understand what's going on with my AI system?" The answer is no as well. You know, of course, from a data perspective, from statistical perspective, the more data points you have, the more confident you are in the
[05:45] eval result. However, even with limited data, you can still treat evals as a QA tool. You can still manually check individual traces, really understand what what might be going wrong with your AI system. And also, you can always generate synthetic data as well
[06:02] to strengthen your analysis. Uh the funny thing is last week I I was at a a party. I have I have met three founders who are trying to build a synthetic data engine for reinforcement learning. So, I'm pretty sure there's so many ways you you can generate synthetic
[06:18] data these days. And also, another question I also got is who should be involved in this eval process. Uh in our opinion, there are four major roles who should who should be who should be involved in in this process. The first one is our dear AI
[06:35] engineer or previous called software engineer. So, what they do is because they are most technical folks among all other roles here. So, what they do is really help you get the real-world data into your your evaluation platform. Either you're using a
[06:50] Bring Trust or ML Flow or any kind of evaluation platform. Because you already has to, you know, use a like use a SDK to send your production traces in. Uh they can also tweak some advanced tasks such as tool calls or rag. These
[07:07] things probably need a more technical background. The second role is product manager. They're still very important. Uh they are the people they help you to develop hypothesis. So, think about to be honest, I like again, I'm I'm from the traditional like data product
[07:22] background. I think the process of eval evaluating AI agent is the same as traditional product development cycle. Because you still want to before you make any changes to your AI system, you still want to develop develop hypothesis about why I want to change to begin
[07:39] with, right? So, PMs, they will go to Twitter, they will do UX research, they will look into data, look at your dashboard, develop hypothesis about why my AI system is not working as expected. And then they will help decide on success criteria, whether
[07:54] I want to improve the reliability, I want to improve the tone of friendly of my chatbot. They can also take simpler task such as prompt because prompt is basically just plain English, right? And they they probably know better about user experience than AI engineers.
[08:11] And lastly, they can also help analyze results from from experiments. And the third role is subject matter expert. As Corey mentioned, they know the data the best. They understand what is a good response versus a bad response. They can go there
[08:27] and help label the sample data. And the last role is the data team. So, the data team will help develop the scores and graders. So, they're the people who can help you, you know, build a a a such a score and they can also help you analyze the results afterwards. And
[08:44] then in terms of common eval process, so as I mentioned, the first thing is you have to develop hypothesis to begin with. And then you need to gather signals of room for improvement. The first signal is just common issues. You can try to find errors from the traces.
[09:01] You can look at their response time, whether they have too many turns in a conversation. This is a little small, but this is our AI assistant. So, our assistant is actually able to help you to find common failure models and also help you find the common like
[09:17] commonly used cases and topic of your choices. And also something I really like is this is in green trust is once you find a interesting trace, you're like, okay, this trace seems interesting. Can I find similar traces in my log? You want to understand is
[09:33] this pattern actually common enough for me need tackle to begin with. And the second is of course human review. You You know, I expert come in and tell you which trace looks bad, which means you can actually go into individual traces as you can see on the left side is the
[09:48] trace tree and you can assign a human review directly in the trace and you can also assign domain experts to help you to evaluate your your score as well. And of course like we are also use user interaction signal here like you know the like it's very common especially in
[10:04] the chat apps. The users can give thumb up thumb down to the response from the chat app and also have have CTA clicks abandonment of conversation. So as long as you make sure you log this actions with the same like the same the same
[10:20] stable ID like a conversation ID or message ID is we are able to actually to to map this user interaction signal back to your trace. And then the next step is really to decide what is success criteria. So our recommendation is you don't need
[10:37] to have too many success criteria. Our recommendation is probably fewer than than three. And statistically speaking, if you have so many metrics, some metrics will definitely going up, some metrics will definitely going down. And how do you decide, okay, should I actually launch the feature or not?
[10:54] So instead you should decide what your success criteria beforehand. So in this example, if you want to tweak your chat app performance, so for example here you only want to like improve their friendly tone. So you can say, okay, my goal of
[11:10] the change is to increase the friendly tone by 10%. So the category of metrics is we called checking metrics. So in this example, for example, if you want to you know, track want to improve the the friendliness of your AI system, you
[11:25] can you can track whether the response has some friendly keywords like whether they have understand sorry or something like that. And the last is a guardrail matrix. So, we want to make sure there is no major degradation on the quality of response such as like error rate or
[11:41] response time. So, at Braintrust we think there are three different scores. The first one we called code-based scores. So, they are pretty fast. They're deterministic. They are cost-effective. In the example I gave before, you can actually just detect whether the response have any of
[11:59] the words. That is a code-based score. The second one is LLM as a judge, which is just a subjective quality assessment empowered by model and the and the prompt. And the last one is a human review. It's basically the ground truth calibration. As you can imagine,
[12:16] this is going to be the most accurate one. However, it's really hard to scale and can be very costly if you hire some external expert to uh score every trace in your logs. So, in our playground, we are able to actually ask Loop to optimize your prompt to generate
[12:33] a score for you and also generate synthetic data for you as well. So, to be honest, like I'm not I'm not a AI engineer. So, every time when I try to evaluate my AI system, I always ask Loop to give me a kind of a first draft of scores to
[12:49] begin with. After you have decided on, you know, okay, this is the thing I want to change. This is success criteria. I have my scores already. And on the left side here is just a uh view is it's a little bit small, but it's the view of our playground. So, what is a playground? Playground just allows you
[13:05] to compare two versions of AI system. Even this is small, but you probably can see the color. On the on the left side is one version, on the right side is is another version. And you compare the these two versions, and then it will give you the scores.
[13:20] They're basically able to score every single row of your data set. So, every row is kind of a trace, and every column is a score and then you can actually compare to see okay which version of my AI system is actually getting better. For playground we always think is a kind
[13:36] of a temporal like kind of just a trial thing for you to begin with and once you are happy with your prompt with your changes you can actually turn it into like into experiment. So the experiment is basically more permanent that's kind
[13:52] of point of time snapshot about what's going on in your AI system in your evaluation. The reason why we want to keep monitoring the performance of the experiment is as I mentioned your experiment scores can can drift like even the model hasn't released a major
[14:08] version yet but because the underlying version might be changing the underlying model might be changing that's why you want to keep monitoring to make sure the performance of the experiment sustain. Beyond this kind of offline experiment playground you can also keep
[14:25] scoring the incoming traces or have similar capability as ML flow where you can keep you can actually score every single trace coming into your system and keep keep monitoring the performance and also set
[14:41] alerts that can send into your slack channel if some if something is up. And then let's talk about a little bit about a little bit about AI agent eval. So the typical agent eval is one we want to make sure you trace everything you trace the calls the state change to make sure
[14:57] you kind of have all the data you need for for your eval. And we we recommend you eval at three levels of your agent because compared to a single prompt like agent is a little bit more complicated. So first of all we want you to eval whether it's working or not so it's kind
[15:13] of end to end so I give you input do you give you the output I expect. Second is a step level eval so which means in every single step does that use the right tool? Does it use the right like right arguments? And then the third level is the trajectory. Is even even
[15:31] it's able to actually to, you know, give you the right right response, even it seems fine at every single step, does that actually takes the most efficient path? So, for example, if it's taking a more efficient path, it probably going to take 2 seconds. But if it if it
[15:47] doesn't, it's going to take it's going to take 15 seconds. And how do how do you make sure it's working in a more efficient fashion? And afterwards, it's going to be the same flow as we mentioned before is you you look at the failed emails, a kind of like like inspect trace and fix and the re
[16:03] you know, eating again. And here are some like typical metrics on the task level and on the performance. This is all needs business domain experts to help to help you to customize the metrics that is tuned to your own specific use case. And the second is
[16:21] multi-agent email. So, multi-agent, of course, has become more popular and beyond what you can do with a single agent, multi-agent is a team. So, which means you have a lot of agents together. So, which also means you need more metrics and steps to evaluate the team
[16:40] performance as well. So, for example, you want to understand the routing accuracy, the handoff quality and coordination. And another thing I want to talk about is a special case for multi-turn emails. So, a very simple one to do multi-turn emails is just, you
[16:56] know, you just get you just gather all the data just right before the step you want to run emails on and then you just saying, "Okay, you know, now I have all the contacts. Only want to change the last step. I want to evaluate the the response from the from the last step
[17:14] to to see whether it's working or not." This is pretty simple. And this And the second way to do this is more comprehensive. It's basically you simulate a user. So, you you still get the whole conversations of your AI
[17:30] agent, and then you you kind of uh extract the incense from this conversation. You know, you know the tone, the goal, you know, what the user trying to do. And you give this information to the simulated user, and the user and the agent will produce the
[17:47] whole conversation. And then you actually evaluate on the whole conversation from there. So, which is going to be more kind of real world uh than this kind of uh like arbitrary kind of the last step like multi-turn evaluation approach. And uh the last one
[18:03] to talk about is the multimodal agent eval. Uh multimodal is also very popular these days. So, at BrainTrust, you can actually evaluate all the multimodal agents like image, audio, video files. For example, Naveen is using us to
[18:18] evaluate their audio eval their audio agent, and Netflix is using us to eval all their eval all their videos as well. So, for multimodal uh agent evaluation, so you always have a like have a like extraction uh step.
[18:34] You are using OCR as TT to make sure you parse the files correctly. Uh and then you kind of send it to agent to kind of to to eval like the intent, the task completion. And afterwards, the the output is going to be a more structured
[18:50] output here. And in terms of the the metrics, uh every, you know, every format has their own special metrics. Sometimes, for example, if a you are uh evaluating some like audio agent, you also want to make sure you can evaluate their kind of multi-cultural
[19:06] consistency if you have different languages, for example. And the last thing I want to talk about here is not about the special agent, uh but a special way to evaluate your agent. So, so this is a special feature at Bring Trust. It's called remote emails. So,
[19:24] what does remote emails do? So, you know, AI agent flow can be very very complicated and in the offline fashion, it's kind of hard to mimic the exact experience and process in your AI agent flow. So, instead, what what what we do
[19:40] is we have a local proxy that can talk to your AI agent in production, which means, you know, you you only need to expose the things you want to change in our playground, so which is example over here. And you don't have to care about other kind of state change
[19:56] or, you know, other part of your agent flow. We will tweak the part in our playground, so this changed part will be able to interact with your AI system in production and give you the response as if you already launch the change like into the production already. So, in this
[20:13] way, it enables especially like less technical folks to be able to to evaluate agents in a more efficient and simpler way. Eval the goal is never about perfection. It's always about iterative improvements.
[20:28] And also, there's another Bring Trust the website bringtrust.dev. So, if you want to search for us, feel free to go here.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.