Building AI Agent Evals: Designing Graders That Measure Outcomes

Summary

Effective AI agent evaluations must measure outcomes rather than execution paths, accounting for non-determinism and compounding error rates that accumulate over long agent runs.
Graders calibrated against human judgment are essential because subtle scoring bugs can dramatically skew results, as illustrated by a Core Bench case where fixing a decimal-precision issue raised measured accuracy from 42 percent to 96 percent.
Distinguishing capability evals, which measure model improvements, from regression evals, which catch performance drift, and regularly reading agent transcripts are foundational practices for reliable agent quality assurance.

Building AI Agent Evals: Designing Graders That Measure Outcomes

Building effective evaluations for AI agents is fundamentally different from testing traditional software. Evals must measure outcomes rather than execution paths, account for non-determinism in agent behavior, and incorporate human oversight at scale to stay aligned with changing models and evolving use cases.
In this talk, Marius from Anthropic covers the three core challenges in agent evaluation: verifying complex outcomes like multi-step documents or Excel spreadsheets, managing shifting environments in reproducible tests, and leveraging LLM judges calibrated against human graders. Learn how to distinguish between capability evals that measure model improvements and regression evals that catch performance drift, plus practical strategies for scaling human oversight through effective tooling and transcript analysis.
🤝

Chapters

FAQs

What is the difference between capability evals and regression evals for AI agents?

Capability evals measure whether a model has improved on a task, while regression evals detect performance drift that occurs when new model versions or code changes degrade existing behavior. Both types are necessary to maintain agent quality as models and use cases evolve.

Why are AI agent evaluations harder to build than traditional software tests?

Agent evaluations are harder because agents exhibit non-deterministic behavior, accumulate compounding errors over long runs, and operate on tasks with under-specified success criteria. Unlike unit tests, agent evals must capture whether complex, multi-step outcomes are correct rather than whether specific code paths were followed.

How did the Core Bench story reveal the importance of reading agent transcripts?

A grader in the Core Bench evaluation was penalizing correct answers that lacked a precise number of decimal places, causing measured accuracy to appear as 42 percent when the true performance was closer to 96 percent. Reviewing the raw agent transcripts exposed this grader bug, which would have been invisible in aggregate metrics alone.

How do you scale human oversight in AI agent evaluation?

Human oversight can be scaled by building effective tooling for transcript analysis that lets reviewers quickly spot grader errors or unexpected agent behaviors. Calibrating LLM judges against human grader labels allows automated scoring to operate at scale while remaining aligned with human judgment.

Full transcript

[00:05] My name is Marius. I'm on the Applied AI team at Anthropic. And today, I want to talk to you about a blog post that we published a couple of months ago, which is called Demystifying Evals for AI Agents. It's peppered with
[00:22] very cool stories, customer stories or learnings from our evals and and research team. A group of our researchers was working on an eval called Core Bench. Core Bench is no longer
[00:38] relevant, it's being saturated, but at that time it was pretty cool. It was reproducing experiments from research papers. They were working really hard. We were launching a new model and they were still at 42% accuracy. We thought that's pretty good, you know,
[00:54] it's it's it's it's not too bad. It's a hill to climb, you know, some some room for our models to get better. Until our research really dug into the transcripts of those runs and and realized that the grader was
[01:09] very particular about the number of decimal places in the outputs. So, like a 96.12 that we report was scored as failed because the grader was expecting, you know, extra few digits at the end.
[01:25] After we solved that, the researcher got to 96% on the on the eval. So, a massive jump from 42 to 96, which means that, you know, the eval is solved. And that's only because somebody paid attention to the transcripts, which is a lesson that
[01:42] I want you all to take away. And I do have another reminder at the end. Please look at the transcripts. Read the transcripts is the best thing that you can do for your agent. I want to take take a step back and uh, understand like why why are we talking
[01:57] about agent evils, right? Uh, why are we all here? Uh, and a big reason uh, for that is because uh, building agent agents is uh, really hard. And the longer we uh, let these agents run, like
[02:14] we have uh, customers that run their agents for I don't know, 10 hours uh, at a given time, the uh, more difficult it is for us to uh, to control the quality of the agents. So, there uh, three reasons why I think uh, building agents
[02:31] uh, is hard. First is uh, the compounding error rate. If a uh, given prompt and a respond has a small drift, you maybe wouldn't notice, but compound that over time, over the course of 10 hours, and um,
[02:47] maybe your agent gets to place where you uh, haven't really intended. Number two reasons why I think building agents is hard uh, is because of uh, the task uh, under specificity.
[03:02] And what do I mean by that? I'm a power user of Claude Code uh, and I I love, you know, talking to Claude every single day, and it does uh, you know, 80% of my job, but when I prompt it, I think of Claude Code as a
[03:18] uh, mind reader. I just want to type in a sentence, and I want Claude Code to just give me back, read my mind, and give me back exactly what I want in in in a single a single shot. And uh, that is not realistic. I'm
[03:33] asking uh, Claude Code too much. I'm not giving him enough context or enough specificity for my task uh, for for it to be able to solve the thing. And if you ever push an agent in production, uh, that's probably true with your agent as well. Your users are going to do the
[03:49] exact same thing that I'm doing with Claude Code. They're going to go and type in uh, or you know, a task and expect magic from the agent while the the specifics are not well stated. Number three, why I think that the agents are are hard to build is
[04:06] because of the unknown unknowns. If you put a text box in front of someone, they will just type in anything. If you you know, think of yourself as you know, solving the same task as the agent, you probably have some refinement
[04:21] about you know, any question that you're getting asked and use your logic to decide whether you can do it or you cannot do it, but the AI models are not in that that place right now, so they can confidently say they can solve the thing where they can't. So, you got to
[04:37] be very careful around guiding the users when they ask the thing that is not yet supported. All right, so now that we know that the agents are really hard to build and we know that we have to build evals for them, there's another
[04:54] you know, thing that we have to worry about which is that evals are pretty hard to build. And if they weren't, we wouldn't be sitting here talking about them and we would just have CloudCode you know, write me evals
[05:09] and and we're done. Kind of like we do with our unit tests. I still have Well, I remember colleagues from from the past when I was writing evals and they were asking me like, I'm just writing unit tests and I was like, it's kind of like unit tests, but it's a little bit
[05:24] harder. Let's see, why are agent evals so hard to build? I guess for the most part is that the outcome is very difficult to verify. And we're going to dive into this a little bit later,
[05:40] but if you can just think of one example, let's say I have an agent that builds a uh, produces a nice uh, spreadsheet with uh, tons of formulas or a very long document that has to be consistent and well researched that just based on the final output is going to
[05:57] be, uh, incredibly hard to, uh, kind of grade it and be consistent and and be correct about it. It's a it's a pretty gnarly task. Second reason for for why for why evals are so difficult to build is, uh, that the environment or or rather uh,
[06:15] the the ground under you is is ever shifting. You want your evals to be, uh, very easily reproducible. Uh, so in case you ever touch, uh, databases or Slack or or the web, which,
[06:30] you know, for the most part if you're building a a useful agent, you will, uh, do touch those external, uh, sources. Uh, you would want to kind of freeze those in time and we're going to get to that, uh, a little bit later. That's kind of like, uh, a lift that you are
[06:47] kind of taking on and and it's a responsibility when you start building e- evals. Third thing uh, is that, uh, unfortunately, there is no replacement for, uh, human oversight. Uh, and what I what I mean by that is
[07:04] you will still have to, uh, dig in and read the, uh, transcripts of your agent and understand what the behavior is. And this is a pretty gnarly task. Uh, if you have a long-running agent with 200, 300, uh, loops and you have, I know,
[07:22] thousands of runs per hour of of those, uh, transcripts, uh, it's kind of humanly impossible to to actually go ahead and and read those. So, uh, the best thing you can do is find ways that you can scale, uh, that you can just only look at the important
[07:39] thing, only sample the the thing that is, uh, actually important. Here's like two reasons why you should actually invest in your evals and not just, you know, slap some unit tests there and and
[07:54] call it a pass. Number one reason is that you would without evals you don't know when your agent is regressing. So if you ever hear customer complaints that, you know, the agent maybe it's a little bit dumber this
[08:10] morning, there's no way to tell what what caused it and there's no way to tell whether that that user is actually correct or maybe it's just a feeling that they have. And number two here that's like pretty expensive is that
[08:25] it's going to be very hard for you to kind of stay on the frontier of the new models and not only models but capabilities if you don't have a very solid eval harness. Things move pretty fast, unfortunately.
[08:41] Not a lot of moat to go around for building agents and the best moat that you can have for for your startup or for business is being very quick to iterate and adopt the latest changes. Drawing form those
[08:57] two reasons to to build evals, we can kind of put evals in two buckets. Number one are the capability evals. So these are evals that are not yet saturated. These are the
[09:12] evals that will give you a sign when a model drops that the model is better at your task. Maybe it knows more about your domain or maybe that it fits better with your harness. A few months ago we launched the 4.5 family Sonnet and Opus and um
[09:31] we're talking to the folks at Cognition and actually posted a a post about this. Uh they took in the model, then spent like one weekend with it trying out their harness. They realized that uh you know, their harness was not
[09:48] fit for the model. Uh and you know, instead of saying your model is worse than the previous one before because our eval's are bad, they say actually let us look into our harness, uh debug it, try to come up with a
[10:04] better harness. And they did and they changed a whole thing in in one weekend and then their agents got way way better. Really cool cool blog post about this that I I recommend folks. So, that's what it takes to be on the frontier and to stay on the
[10:21] frontier. And uh number two category of eval's are regression eval's. There's a kind of like a migration from my capability to regression once you saturate. Uh so, these are the eval's that uh will tell you like will give you a sign whether, you know, your your agent
[10:38] is drifting in in one direction or or the other. There's a kind of like an inherent reason why uh your eval's are very kind of tailored to your product. Uh and that's because uh there's like a high variance of of agents that you can build. I listed four
[10:54] here which are really different. There's coding, there's conversational, there's research, and computer use agent. Uh and for each of these uh you have to be uh very creative about what kind of eval's to build. There's not a
[11:09] one-size-fits-all. There's only guidelines. Uh that's why I encourage you to think from first principles and uh really understand what's meaningful for your specific agent. There are things out there in the wild that can help you get started.
[11:25] And those are the the benchmarks uh that are listed in uh in the in the last column here. So, if you're building uh something with coding, maybe a good idea is to get started on the coding benchmarks. Take those, adopt them, change them,
[11:42] tailor them for your product. What do we actually verify when we run an eval? There two things that the agent produces. One is the actual outcome, and this come in like different shapes and and and and
[11:58] sizes. And two is is the transcript. The outcome, I think it's something that we encourage people who start with agents to to to to focus on right away. This is the most important thing because this is
[12:14] what your agent produces. This is what your users eventually see. Focus on this first. Get your greater of the outcome to be the best in in class. And then you can migrate over to look at the transcript,
[12:31] which is like the behavior of the agent, all the steps that it took. My recommendation for grading the transcript is to be less rigid about it and to have this idea in mind that maybe the model and the agent is better at the
[12:49] task that you you would think. So, if you have some preconceived ideas of like, "Hey, what are these are, you know, steps 1 to 10 that my agent needs to take in order to to achieve the the thing." Likely
[13:06] you would be a little bit too rigid for for your agent, and your agent might surprise you. We had like a cool story that we published about our agent kind of hacking Tow Bench, which is a customer support
[13:22] agent eval or benchmark. And we realized that the uh eval is is wrong because our agent found a loophole in their flight policy and was able to use
[13:37] that to refund a a ticket. Uh, so be very mindful of the fact that the agent can take turns that are unexpected in order to achieve its result. But you should be all set if you're just focusing on the outcome. I'm
[13:53] going to give you a maybe like a heads-up of how to think about like verifying the outcome and maybe understanding like why this is hard. The easiest thing to verify would be writing code. And if you can
[14:10] actually write an eval or like a grader just with code, uh, you're lucky. You're probably, uh, you know, uh, you don't need to think too much about evals if it's just a matter of like comparing two numbers or doing like, I don't know, static,
[14:26] uh, analysis of code. On the other side of the spectrum is having humans looking at the outcome and grading it, which is probably the best that you can do, but unfortunately it doesn't scale, uh, and it's it's very expensive and slow. So, what we're stuck with is the
[14:42] thing in the middle, uh, which is having an LM judge, and I think I need to update the slides there. It's actually, uh, an LM agent that will uh, evaluate your your outcome. So, in the
[14:57] case in the example that I gave you about, uh, an agent that generates an Excel spreadsheet, you would have an agent that uh, has to use tools to read that Excel, has to read the formulas, ensure a bunch of things, maybe like, uh, check
[15:13] a bunch of rubrics, and, uh, you know, sum sum things up uh, in order to eval. But, you can think that there's one problem with this, right? If uh you have an agent for eval your agent, it's problematic in the sense that the grading agent also needs an
[15:31] eval, right? How do you actually How do you actually make sure that the grading agent is doing the right thing? So, we call that calibration, and that is being done by comparing the LM grader with human grader,
[15:49] you know, every so often whenever you can just to make sure that those two are aligned. And the last thing that I wanted to cover is the human oversight. As we uh talk about this all throughout, and
[16:04] I do want to double down on this. You will have to look at transcripts. You will have to read thinking blocks of of your agent and tool calls. But, you want to make sure that you do it in a way that is very efficient. What
[16:20] I mean by that is investing a lot in your tooling for for for viewing and and, you know, comparing these evals. And there's a lot of things that you can do. There's search, there's comparing screens,
[16:37] there's nice visualizations for your agent result. Uh it's very very very easy right now to spin up a front end with with cloud code or any other coding agent, and just have this this thing that is is
[16:53] tailored for your agent and and keep investing to that to make your life easier.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.