Skip to main content

How should I evaluate new AI models before rolling them out broadly?

Summary

  • Test on your own work, not on launch claims. Databricks found that models marketed as frontier often aren't, so it scores each release on private benchmarks, user feedback and measured cost before promoting it.
  • Use three signals together: benchmark results on your tasks, feedback from a pilot group, and cost measured per task or session from logs.
  • Measure cost at the task level. Token price is a poor guide to end-to-end cost, because models differ in how much they read and reason to finish a task.
  • Expose the model gradually, with a capped pilot for people or a small traffic split for applications, and keep a fallback ready.
  • On Databricks, MLflow evaluation scores app versions against the same dataset, and Unity Gateway provides the traffic split, budgets and usage logs.

How should I evaluate new AI models before rolling them out broadly?

Run every new model through a short, repeatable gate: score it on a private benchmark built from your own tasks, give it to a small pilot group with a spending cap, and compare cost per task with the model it would replace. Promote it only when quality, user feedback and cost tell the same story. Anything less, and you're betting the organization on a launch announcement.
That gate matters because the newest model isn't automatically the best one. In its rollout write-up, Databricks warns that models marketed as frontier often aren't, that moving workloads to a model that regresses quality can hurt a company, and that naive use of a new model can explode costs.

What a pre-rollout evaluation should cover

Check What to measure How to get it
Task quality Pass rate or scorer results on representative tasks A private benchmark or evaluation dataset from real usage
Cost per task End-to-end spend to finish a task or session Usage logs or traces with token and cost data
User experience Whether pilot users prefer the new model for their work Surveys and structured feedback from a pilot group
Safety and policy Whether guardrails still hold on new outputs Logged requests and responses, plus safety scorers
Operational fit Latency, error rates and rate-limit pressure Per-request usage metrics
Regressions Whether anything that worked before now fails The same dataset run against old and new versions

Pitfalls that skew results

  • Public benchmarks. Their tasks are public, so solutions leak into training data over time, and they may not resemble your workload. Databricks' coding benchmark post gives both reasons for building its own.
  • Token price as a cost proxy. The same post found token price a poor indicator of end-to-end task cost: a model that's cheaper per token can work longer and read more, and end up costing more per task.
  • Biased pilot cohorts. Early adopters tend to be power users. Databricks compared the same group's usage before and after access, not early adopters against everyone else.
  • A shifting task mix. Pilot users try harder problems with a new model. Databricks stratified sessions by whether they were single- or multi-turn and whether they edited files, then reweighted before comparing cost.
  • Judges that reward sounding right. For code, Databricks grades correctness with held-out tests and doesn't use an LLM judge, which it found rewards sounding right over being right.

How Databricks approaches pre-rollout evaluation

Pilot access with a spending cap. Databricks gives new models to employees through Unity Gateway on an experimental basis and caps usage with per-user budgets, then promotes or drops each model once the three signals agree. Unity Gateway budgets support per-user thresholds that alert, block further requests, or both. To keep a model with a smaller group first, restrict EXECUTE on its model service to that group.
Small traffic splits for applications. A custom model service can split traffic across several model backends. The docs describe traffic splitting as a way to gradually roll out new models and run A/B tests. Session affinity keeps each session on one destination, fallbacks retry failed requests against backups, and routing decisions are logged so you can confirm the split.
Measurement in system tables. system.ai_gateway.usage records token usage, latency and requester for each request, and request tags let you attribute usage to a project or experiment. Inference tables log full request and response payloads to Unity Catalog Delta tables for debugging and audit.
Quality scoring with MLflow. MLflow evaluation scores traces with scorers: LLM judges such as Correctness and Completeness, and code-based checks. The evaluation tutorial builds an evaluation dataset from real usage, runs mlflow.genai.evaluate(), and compares versions to confirm that a change improved quality without causing regressions. To compare a new model, rerun the same dataset and scorers against it and read the results next to the old model's.

FAQs

How long should a new model stay in evaluation?

Until benchmarks, pilot feedback and cost data agree. Because the pilot and the measurements run at the same time, Databricks has reached a decision within days of a model's launch.

Are public benchmarks enough?

No. They're a starting signal, but their tasks can leak into training data and may not match your work. A benchmark built from your own tasks is the stronger test.

Should I use an LLM judge?

MLflow offers LLM judges such as Correctness and Completeness for open-ended output, alongside code-based checks. For code, Databricks grades with tests that check behavior, since it found judges reward sounding right over being right.

How do I roll back if a model disappoints?

Move its traffic share back to the previous model in the service's routing configuration, or remove the pilot group's access. Applications that call the service name don't need to change.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.