Agent as a Judge: Scaling AI Evaluation for the Agentic Era

Summary

Agent as a Judge uses evaluation agents to perform open-ended discovery on raw production traces—identifying recurring failure modes, clustering them into issue categories, and generating targeted evaluators—addressing the scale limitation of manual trace review.
Arize Phoenix's in-app evaluation agent Pixie is modeled after coding agents with a file system, bash runtime, tools, and a trace debugging skill that applies open coding and axial coding to surface issues at a scale human reviewers cannot match.
This video contrasts Agent as a Judge with LLM-as-a-Judge: LLM judges are single-pass and score predefined criteria, while agent judges are iterative, multi-step, and designed for open-ended discovery of previously unknown failure modes.

Agent as a Judge: Scaling AI Evaluation for the Agentic Era

Agents have arrived. Evaluation workflows need to catch up.
The bottleneck is knowing what to evaluate. You can’t catch failures you don’t know exist, and in production agents, they’re subtle: unclear tool definitions, bloated context, brittle handoffs, and inefficiencies that compound across long sessions. Manual trace review is the only way to discover them, but it doesn’t scale. Blind spots accumulate.
Agent as a Judge closes this gap. Where LLM-as-a-judge scores predefined criteria, evaluation agents do open-ended discovery. They work directly on raw traces, surface recurring failure modes, and turn those findings into targeted, high-signal evaluators — a step toward agents that improve agents.
We’ll show how this works in Phoenix’s open-source evaluation agent, running on Phoenix’s own production traces. You’ll see what it surfaced, how it compares to human reviewers, and what it takes to trust agents with the work of evaluating agents.
Talk By: Elizabeth Hutton, Senior AI Engineer, Arize ;

Chapters

FAQs

What is Agent as a Judge and how does it differ from LLM-as-a-Judge?

Agent as a Judge uses an autonomous agent to perform open-ended discovery on raw production traces, surfacing recurring failure modes without predefined evaluation criteria. LLM-as-a-Judge, by contrast, is a single-pass approach that scores traces against criteria you have already defined—making Agent as a Judge the right tool when you don't yet know what to evaluate.

What is Pixie and what did it find when analyzing its own traces?

Pixie is Arize Phoenix's in-app evaluation agent, modeled after coding agents with a file system, bash runtime, and a trace debugging skill. When run on Phoenix's own production traces, Pixie identified tool-calling inefficiency, missing instrumentation where failed tool calls were incorrectly reporting status OK, context management issues, and instruction gaps.

How does the Agent as a Judge workflow operate under the hood?

The agent starts with raw traces, orients itself to the trace structure, collects a representative sample, and applies open coding and axial coding to take open-ended notes on any issues it finds. It then clusters those notes into issue categories and produces a ranked table of findings with evidence and links to representative traces.

Why doesn't manual trace review scale for production agents?

Production agent traces are long, messy conversations full of tool calls, agent handoffs, and bash commands that are difficult and time-consuming to read. At the scale of hundreds or thousands of daily traces, manual review creates a bottleneck that prevents teams from discovering the failure modes needed to write effective downstream evaluators.

Full transcript

[00:07] Hi everyone. Thank you so much for joining me. You can put on your headphones. Going to get started.
[00:19] All right. Everyone, my name is Elizabeth. I'm an engineer on the Phoenix team at Arize. We're an open-source observability platform for evaluating AI. And my background is in machine learning. I used to train models, and while I
[00:32] don't do that anymore, I find that the skills have generalized well. So, today I'm going to talk about agents evaluating agents and agent as a judge. I'll talk a bit about how it works and
[00:45] how it compares to other evaluation techniques. I'm guessing many of you are building agent systems or thinking about it. So, you've probably heard this advice before
[00:57] that you should look at your data. And when we say look at your data, what we mean is reading traces and doing error analysis on them. But, I'm also guessing that most of you are not reading traces every day because
[01:11] your data looks like this. Agent traces are messy. They're long conversations full of tool calls, agent handoffs, bash commands. Nobody wants to read that.
[01:24] It's hard to read just one session, uh but then consider reading hundreds. So, the reason we tell you to look at your data, though, is because this is how you
[01:38] discover the issues that fuel your downstream evals and your improvement workflows. This leaves us with a big gap. Trace review is how you discover issues
[01:50] and helps you identify what evals to write, but it doesn't scale. As usual, humans are the bottleneck.
[02:02] So, I wondered what would happen if we let an agent do this trace review for us. The Phoenix team recently released our own in-app agent called Pixie. It's modeled after coding agents, so it has a file system, a bash runtime,
[02:16] tools, and skills. So, we gave it a trace debugging skill and wanted to see what it could do. So, here it is analyzing its own traces,
[02:28] identifying issues, and categorizing them for us. Under the hood, Pixie uses a school uh skill that has best practices for error analysis baked in.
[02:45] You start with the raw traces, the agent will orient itself to the trace structure, collect a representative sample, and then do what's called open coding and axial coding, which is basically taking open-ended notes on any issues
[02:57] that it finds, and then clustering them into issue categories. It'll then summarize its findings into a table with a ranked list of issues that
[03:09] it found, evidence for them, and links to representative traces. What you do with the findings is up to you. You can use them to scope a targeted evaluator,
[03:22] uh ship a fix directly, or use them to seed a hill climbing data set. You also don't have to use Pixie for this. Uh you can use your coding agent of
[03:34] choice with the Phoenix CLI and some of our own skills. So, when we pointed Pixie at its own traces, we found some interesting
[03:48] issues, things that we you not have found otherwise or as quickly. For example, it found this very specific tool calling inefficiency where Pixie was calling our GraphQL API over
[04:01] and over again when it would have been more efficient to pull that data down once and query it from the file system. It also found an important gap in our tracing and instrumentation.
[04:13] The tool call spans that had failures still showed status okay, which meant it was harder to filter and identify any errors. Seems like every time we ran this skill,
[04:26] uh it found new issues with the context management, the instructions, efficiency, and instrumentation. So, we could see that yes, in fact, uh you can have an agent do this trace for
[04:39] you for you, and it will surface real meaningful issues. The next question we had was, well, is an agent really necessary? Can't you use LLM as a judge for this kind of thing?
[04:56] So, let's compare the two. LLM as a judge is a single pass. You give it input, output, your eval criteria, and it gives you a score. It's really great at making subjective judgments when you all know exactly what you're trying to measure.
[05:09] I like to think of it as an approximation of human annotation. Just labeling examples according to some predefined criteria.
[05:23] Agent as a judge looks different because the task is different. Error analysis involves intelligent sampling, navigating nested and heterogeneous data, and noticing patterns across multiple sessions.
[05:39] It returns more ground more grounded and valuable insights into the behavior of your application. And this type of iterative exploration really needs agentic tools, a file system, bash, sub agents,
[05:52] and the ability to manage context dynamically. They're not competing, though. LLM as a a judge still has a time and place. It's really good for catching the issues that you already
[06:05] know to look for. Agent as a judge is how you learn what failure modes to look for in the first place. So then one of the problems with LLM as a judge is that it requires pretty
[06:19] careful alignment to human judges in order to be trusted. So, what about agents? Can we trust agents to do this type of job? Or do they need careful alignments as well?
[06:37] Let's consider this example, a conversation between Pixie and a user. The user has asked Pixie to create an LLM evaluator. Pixie calls some tools and hits an error and says, "Sorry, I can't do that."
[06:50] Now I as a human look at this and immediately can tell something feels off. I know that Pixie is able to create this type of evaluator, and it should have recovered from the error.
[07:02] We humans are really great at understanding the user experience, the product and its capabilities. But an agent catches what we miss.
[07:17] There's a lot of tool calls in here, and I'm not great at reading and understanding complex bash queries. But agents are tool calling experts, and they can really quickly identify any issues with the tool calls or the sequencing.
[07:31] They're really good at attention to detail. In my experience, agent trace review actually complements human review nicely. The agent will catch what we miss, and vice versa.
[07:47] Research on the alignment between human and agent judges in evaluation tasks is still fairly limited, but it leans positive. One Microsoft study showed that agent judges align with human judges, or agree with them, as
[07:59] often as humans agree with each other, which is about as good as it gets. And another Meta AI lab study showed that LLM judges have 90% agreement with
[08:12] human judges, which is a 20% increase over plain LLM as a judge, which tells me there's something to the dynamic context management of agents that really helps close this gap in
[08:24] alignment. So, yes, I'd say you can trust agents with your evals, with your error analysis tasks. You can keep humans in the loop for anything high-stakes, of course, if that
[08:36] makes you feel more comfortable, but use agent as a judge to surface these issues at a scale. Which brings me to where I think this actually fits into your evaluation workflow.
[08:53] Agent as a judge is just one of many tools in your evaluation toolkit. And ultimately, the goal of all evaluation is to gain insight into the behavior of your application. You start with a wall of traces,
[09:05] and the evals you write help you to filter it down into something more manageable. First line of defense are automated evals, like LLM as a judge and heuristic
[09:17] code evaluators, which run on your production traces at scale. And these are great at catching the issues you know to look for. Agent as a judge sits in the middle. It's more expensive than LLM as a judge,
[09:31] but less expensive than human trace review. And it helps you find the unknown unknowns. Combining these evaluation signals is where the magic happens. It helps you
[09:43] narrow down the set of traces that you actually want a human to review. And at the end of this process, you should have a sense of where your agent is struggling and a set of examples you can use to improve it.
[10:00] None of this is a one-time thing, unfortunately. Evaluation is a process. Your agent is constantly evolving and any evals that are tied to really specific failure modes will have a short half-life.
[10:12] The issues that you see today in your agent are not going to be there in 6 months because you're continuously improving. And so one of the nice things about agent as a judge is that it doesn't go stale as quickly.
[10:24] It's always exploring the frontier of what your agent is capable of.
[10:36] The way we we evaluate AI has always evolved alongside the technology, growing ever more complex. We had F1 for classification models, LLM as a judge for generative AI, and now
[10:49] agents. And They're good at evaluation for the same reason that they're good at anything. The tools, the iteration, the context
[11:01] management all help it to accomplish this open-ended evaluation task. An agent as a judge isn't a replacement for LLM as a judge in the same way that error analysis isn't a replacement for
[11:13] data annotation. They're both valuable. So you use agent as a judge to discover and prioritize new issues.
[11:25] This keeps humans in the loop without letting them become the bottleneck. That's all I had. Thank you. You can learn more about this work at Phoenix repo or the docs.
[11:39] And everything we do at Phoenix is open source. So, you can take a look at Pixie and even its its evil harness is open source. So, I encourage you to check that out if you want to learn more. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.