Sponsored by: Snorkel AI | The Art and Science of Benchmarking Agents

Summary

Snorkel AI's Charles Dickens argues that great benchmarks are not just measuring sticks but should actively inspire and shape the frontier of AI research, requiring both scientific rigor—task quality, distributional control, and model headroom—and an artistic vision for what capabilities to advance.
This video examines landmark benchmarks including GPQA, MMLU, ARC-AGI, TaoBench, SWE-bench, HELM, and Terminal Bench, drawing lessons for next-generation agent benchmarks that must handle environment complexity, autonomy horizon, and nuanced output complexity.
Snorkel AI's Open Benchmarks Grants program is a $3M initiative with partners including Hugging Face, Together AI, Prime Intellect, and Factory, designed to fund enduring benchmarks that close the evaluation gap between how fast AI is developed and how rigorously it is measured.

Chapters

FAQs

What is the evaluation gap in AI and why does it matter?

The evaluation gap is the asymmetry between how quickly AI capabilities are being developed and how rigorously they are being measured. This video explains that while model cards show quantitative progress and coding agents are clearly improving, enterprises still hesitate to deploy agents in high-stakes settings because existing benchmarks do not capture whether agents can be trusted in domain-specific, real-world scenarios.

What makes a benchmark scientifically rigorous according to Snorkel AI?

This video identifies three table-stakes properties for scientific rigor: task quality (expert-validated, adversarial questions like those in GPQA), distributional control (taxonomic diversity across domains like MMLU), and model headroom (benchmarks must not saturate quickly, as illustrated by ARC-AGI's design to remain challenging). Robust evaluation processes that assess intermediate steps rather than just final outputs are also highlighted.

What is Snorkel AI's Open Benchmarks Grants program?

The Open Benchmarks Grants program is a $3M initiative run by Snorkel AI in partnership with Hugging Face, Together AI, Prime Intellect, Factory, and others to fund the creation of enduring, open benchmarks that advance AI evaluation. This video presents it as a call to action for researchers to submit benchmark proposals that address next-generation challenges in agent evaluation.

What are the key dimensions Snorkel AI identifies for next-generation agent benchmarks?

This video identifies three emerging dimensions for next-generation agent benchmarks: environment complexity (the richness and realism of the setting in which agents operate), autonomy horizon (how many sequential steps an agent must take to complete a task), and output complexity (how nuanced and multidimensional the evaluation rubric needs to be, including preference collection rather than simple correctness scoring).

Full transcript

[00:08] All right, everyone. Thank you so much for joining. I'm Charles Dickens and I'm a research scientist at Snorkel and today I'm going to be discussing a synthesis of what we think makes
[00:21] benchmarks great. So, this is the art and science of making benchmarks for agents. This is a consolidation of what we've learned from both first-hand experience and also from
[00:33] our academic and industry collaborators. So, we have a really unique privilege to work with researchers both internally at Snorkel and academic research labs, industry labs, open source community and
[00:46] we've learned a lot from all of these partners and that's what I'm going to share today. Um but if there's one takeaway that you take away from this talk, um if you left
[00:58] right now, I'd be sad, but I want you to leave with the idea that great benchmarks are not just measuring sticks. They are sure, they're calibrated and reliable, but they don't just measure things.
[01:10] Benchmarks are at the risk of sounding pretentious, benchmark designers are artists. I think that our job is to actually inspire and shape the frontier of AI research.
[01:22] Um All right. So, first a little bit about us, Snorkel. We're a frontier AI data lab and our team and founders have labs at Stanford, University of Washington,
[01:34] University of Wisconsin and then we have a internal team of really talented researchers, applied engineers, um FDRs, FDEs and our team is constantly contributing
[01:46] to open source research and projects and that's all to say that our team is built for focusing on creating the best data possible to define and advance AI.
[01:59] And we're in a fortunate position to play and by play I mean build and deploy data sets and environments. And this is at the intersection of frontier academia and industry research as well as
[02:11] grounded in the reality of enterprise deployments. So today there's an asymmetry that we see from our vantage point of working at this intersection of frontier research
[02:25] and experiencing and working with enterprises to deploy agents in the real world. And the excitement around agentic AI is backed by quantitative progress that we see on model cards, of course. And there is
[02:39] genuine leaps um especially in coding. We see it and use it at least I do um every day. But ask individuals at enterprises um if they feel ready to actually deploy
[02:52] agents, do they trust them in the high stakes domain specific settings outside of coding? And you're going to see hesitation. At least that's what we see when we work with enterprise customers. And that's not to say capabilities
[03:05] aren't there, but the our ability to measure these capabilities I think is lagging actually a bit. And this is one of the challenges that I think is one of the most interesting
[03:17] problems in research and what we're working on actively at Snorkel. Closing this gap. So closing this gap requires a holistic view. A full evaluation toolkit
[03:30] that'll give you that view. And we're strong believers in field deployment. So actually getting your hands dirty, deploying agents in finance, insurance, health settings, and analyzing use case specific
[03:42] performance in these real environments. Cuz this gives you an understanding of what actually matters at the end of the day. We're also strong believers in red teaming, private human evals, crowdsourcing, but
[03:54] what this talk is about is open benchmarks. And they remain a critical piece of our measurement toolkit. And the best benchmarks don't just measure a snapshot of AI performance, but they actually
[04:06] give new vectors of AI progress. And I'll celebrate some of the great benchmarks in more detail, but to start the shout-outs, Terminal Bench, Meter Long Horizon, Arc AGI, I think these are all great examples of benchmarks, the type of
[04:20] benchmarks I'm talking about. So, something that we launched and is motivated by this view, so we actually are putting this view and making it a part of our roadmap is
[04:34] the open benchmarks grant. So, this program's in partnership with Hugging Face, Prime Intellect, Together Factory, Harbor, and more. And this is a really fun project because we get to work with, again, academic and industry researchers
[04:47] to support really ambitious goals, new benchmarks that shape frontier AI. And we've had a really wild, exciting reception, a lot of submissions, and we've seen
[04:59] already amazing output from this project. So, Continual Learning Bench, Agent Last Exam, Terminal Bench, Science, Judgment Bench, Slot Code Bench. I'm name-dropping, but you can check this all out on our open
[05:12] benchmarks grant page. Um, so, back to the main premise of the talk, the art and science of benchmarks. So, we've reviewed tons of applications, and this is from groups from a range of
[05:25] domains and capabilities. Today, I'll share what's now considered table stakes, so like minimum requirements for creating a useful benchmark, but also get into what makes a benchmark stand out,
[05:37] rise above the noise. And these are benchmarks that actually shape the frontier, rather than just measure the current state. So, first I'll walk through the science. This is uh items including pillars including
[05:50] task quality, distributional control, model headroom, robust evaluation methodology. On the art side, what separates a benchmark is having a thesis on the frontier,
[06:03] road maps for the field, a new view of progress, and a researcher user experience. It's actually easy to use this benchmark and run evaluations um of your model and agent harness. So, let's go into the details of what I
[06:15] mean by each of those pillars, starting with the science. So, the first pillar, task quality, individual tasks need to be high quality, and what I mean by that is real
[06:27] world, for example, well-posed instructions, and I think one of the highest bars you can set is actually having quality validated by real human experts, domain experts that are going to be using your
[06:39] system. GPQA, graduate level proof question answering, did a great job at this. Uh they set a standard for quality assurance that allowed uh the benchmark
[06:51] to remain relevant for a long period of time, and the ideas from this uh work remain today. And a contribution that stood out to me that was uh novel was adversarial quality control to actually
[07:03] um ensure that tasks were not just valid, but also solvable and difficult for valid reasons, not contrived. And this was done through multi-expert uh reviews,
[07:15] as well as a panel of non-experts that attempted to solve the problem, um but the difficulty was defined by the pass rate of these non-experts. So, this idea
[07:27] was uh really interesting. And if you read the appendix, they actually aligned incentives payouts with this process to encourage challenging, but still valid data points. So, this is a nice idea for getting task quality. And this figure
[07:40] illustrates um the process to get to the diamond set. MMLU, the next one, for representing distributional diversity. And diversity is important because this can be uh
[07:52] important for realism. But it also is important for ensuring representation of areas where the model is performing poorly. Um or it's rare, but disproportionately
[08:04] important. Maybe an example is a pedestrian crossing the road at a improper uh location. This is less likely, but you got to get that right. Um so defining a taxonomy to measure and
[08:16] define the diversity of the benchmark is a hallmark, I think, of uh great benchmarks. And MMLU is a great example of this. The next thing is model headroom. So you don't want to publish a benchmark that's
[08:29] already saturated. You want to actually measure um where models are at the current state and show that there's room for improvement. So I think ARC-AGI is a really good example of this. Um This benchmark uh was published on
[08:42] saturated. In fact, it shipped with initially less than 1% model scores when um 100% of the tasks were already solvable by humans. And a few months in, ARC-AGI 3 um I
[08:54] think it's still one of the most anticipated benchmarks and is uh really showing that there's still room to improve. Um a figure at the top right here illustrates um correlation with the uh
[09:07] reasoning boom over the last 24 months that resulted in a spike on that benchmark and it correlates well with um capabilities and model changes in uh reasoning.
[09:19] Next thing is robust eval methodology. So metrics, I think, for benchmarks should go beyond just accuracy, capturing cost, latency, reasoning quality. Um the benchmark should also measure what it claims to, and the
[09:32] methodology is reproducible and robust to potential contaminations. Uh TaoBench is a good example of this. Um It's interesting that there's multi-turn interactions and in fact a
[09:44] model that might book the right flight but violates fair class violations. That's what's illustrated here is still a failure. So the process is also what matters, not just the final state.
[09:58] Now we'll get into the art. So a good benchmark that is rising above the noise is something that's going to be interesting to researchers should have a strong thesis on the frontier.
[10:10] I think terminal bench is a really good example of this. Terminal bench made a bet that the CLI is not only going to be important for coding agents, but for general purpose computer use and tasks.
[10:22] And I think this is actually been a really consequential bet at least as folks big names are actually building their general general enterprise tools on top of coding or terminal use agents.
[10:38] Next is road maps for the field. So I think um what I'm trying to get out here is benchmarks should produce road maps, next steps, ways to develop on top of the benchmark. It shouldn't be thought of as the final product. Sweet bench is a good example
[10:50] of this and it spawned a whole family of benchmarks like verified, multilingual, multimodal and its evolution shaped how teams build coding agents. So this is really an example of a benchmark inspiring
[11:03] design and progress. I think this is uh this was successful because the idea just made sense and it's something that is practical. The agent's given a code
[11:16] base, a problem description, a GitHub issue and a set of tools and the model needs to generate a patch. So this is something useful for engineers day-to-day. Next is a new view of progress.
[11:29] So great benchmarks will establish a way to measure a prog- measure progress or specific capabilities. And this gives us a common way to communicate progress and ideas and model performance improvements. And this is a
[11:42] exemplified in continual learning bench. So, this is a new benchmark uh came out recently and is designed to measure whether agents generally improve with experience rather than um
[11:54] uh remain at a baseline performance. And this spans diverse domains. And um the novel view of progress here is a metric measuring gain. So, performance improvement over a
[12:07] stateless baseline. So, this measures continual learning capabilities. And finally, research user experience. So, this is the user experience uh with actually interacting with the benchmark. Are you
[12:21] able to run your agents on the benchmarks easily? Um and if you do this, you'll get adoption, of course. And I think this is something that's overlooked often times, but you need to get it right before you release your benchmark. Otherwise, people are going
[12:33] to be uh there's a barrier to entry. And um I think good examples of this are HELM. So, this was uh something that pioneered a standardized and modular harness for reproducible evaluations. Something more
[12:46] recently is Terminal Bench 2.0 that shipped with Harbor. And this is becoming the de facto tooling for teams who are building uh command line use agents. So, here's the full picture.
[13:00] On the science side, what is necessary to make the benchmark scientifically useful uh and rigorous? We have task quality, distributional control, difficulty headroom, uh metrics and methodology. And then on
[13:13] the art side, what's going to differentiate you and um make things uh really interesting, going above and beyond the table stakes. There's a thesis on the frontier, roadmaps for the field, a new view of progress in
[13:27] research or user experience. So, um I want to give our point of view not just on um what makes benchmarks great, but also where benchmarks are going, how they're
[13:39] evolving, and what the next steps are. And this is our point of view on that, uh what the design what the design space is for the next generation of uh great benchmarks. The first is environment complexity. I don't think I need to sell
[13:52] this group on how complex environments can be, and uh I think benchmarks need to align with that and represent um uh dynamic environments uh that agents we want them operating in. The next is
[14:05] autonomy horizon. So, agents are tested at horizon lengths that users, professionals, scientists expect. Uh and finally, output complexity. I think we are making progress here, but there's still a lot of room for
[14:17] improvement, and it's currently, in my opinion, still under explored. Um so, let's see what I mean by this. I won't go into like excruciating detail on each of these, but diving in, environmental uh complexity,
[14:31] ways that I'm seeing uh people improve and progress being made is on domain specificity. What do we mean by uh that? So, unwritten knowledge is um
[14:44] ever-present in enterprise environments. Um there's going to be uh context that you need to consider beyond just a single code base repository. Uh consider coding agents. Um these are
[14:56] the most advanced agents today, but there's still areas where benchmarks don't capture every complexity there is. Uh policies, context across Slack, screenshots could be important, design
[15:08] docs, flaky toolchains, human reviewers, um and I think this theme resonates even more in enterprise environments, specifically like insurance and healthcare. Um so, most benchmarks only capture capture
[15:21] a fraction of this. Next is autonomy horizon. So, autonomy horizon how independent is the agent over a long horizon? And what I mean by this is does
[15:33] the agent actually stay aligned with my preferences over multiple turns and make the right decisions when it inevitably has to because your specification, your initial instruction isn't going to cover every detail and it
[15:46] shouldn't. We shouldn't have to be so prescriptive. So, take a customer experience agent onboarding and enterprise customer over several weeks. It might lose track of the customer needs. Uh there might be changes in our
[15:59] policy. Um there might be major judgment calls that were made that are leading to downstream uh risks, issues. And all of these things should be represented somehow in the benchmark. And each of
[16:12] these reflects, I think, a distinct dimension um that should be considered when designing benchmarks for measuring autonomy horizon. Finally, output complexity. This is the
[16:25] last access I'll cover. Output complexity is all about how sophisticated the deliverable is. So, I think so far a lot of benchmarks have been focused on code or single uh text
[16:38] documents and progress on this front is being made. Multimodal um outputs um for example, but it can be even more nuanced and sprawling. So, as the output becomes
[16:54] more nuanced and complex, so must evaluations. And I think this goes all the way back to the figure where I showed the gap between measuring capability and evaluation. To really um measure the progress on agents in output
[17:09] uh output complex tasks, we need to match it with evaluations. And I think for example, there's no unit tests whether this type of deliverable um is good or bad. And I think nuanced rubrics
[17:22] is one way to perhaps approach this type of problem. And these rubrics should be designed with domain experts who actually have this tribal knowledge, domain experience, and can capture that. Um
[17:34] Preferences is another great way to collect this type of nuanced judgment. And tomorrow's benchmarks need to capture all of these aspects. And finally, I'll close with our
[17:49] um another plug for proposing a benchmark to our open benchmark grant. And hopefully this inspired ideas for benchmarks based on what you're working on in your day-to-day.
[18:01] Um and if it did, make a submission. Um it's been really fun reviewing and supporting really ambitious benchmarks and seeing them through and come out and actually be used to evaluate frontier
[18:13] models and be presented in model cards. Um you're making a huge contribution to um the open source research community when you uh do this so that we can have open source models um and start to build
[18:25] trust in them and uh be able to actually deploy them in enterprise environments. So, we're really excited about building benchmarks. Um and I hope you are, too. And uh thank you for your time and attention.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.