Which evaluation framework should I use to decide whether to ship or iterate an LLM product?

Summary

• Use an evaluation framework that turns "good enough to ship" into explicit pass/fail gates, scored on a fixed evaluation dataset with the same scorers every time a prompt, model or retrieval step changes.
• Measure each failure mode separately. Correctness, relevance, groundedness, tool use, safety and product rules each need their own scorer, alongside latency and cost.
• Combine LLM judges for coverage, code-based checks for deterministic rules, and expert labels to confirm that the judges agree with people.
• On Databricks, MLflow 3's mlflow.genai.evaluate() runs your app over an evaluation dataset with built-in or custom scorers and records an evaluation run you can compare with the previous version.

Which evaluation framework should I use to decide whether to ship or iterate an LLM product?

Use a framework that scores your application against explicit gates, on a fixed evaluation dataset, with the same scorers every time something changes. The ship-or-iterate call then becomes two questions: did this version pass every gate, and did it regress on anything the last version got right? On Databricks, MLflow 3 provides that loop: evaluation datasets in Unity Catalog, LLM judges and code-based scorers, and evaluation runs you can compare side by side.

What should the framework measure?

LLM applications fail in ways unit tests don't catch: a fluent answer that's wrong, a response that ignores the retrieved context, a tool call with the wrong arguments, an unsafe reply. A useful framework measures each failure mode separately and fixes the threshold for each one before anyone sees results.

Dimension	Question to answer	How to score it
Correctness	Does the answer match the known-good answer?	LLM judge with ground-truth expectations
Relevance	Does the response address the request?	LLM judge, no ground truth needed
Groundedness	Is the answer supported by the retrieved context?	LLM judge on the retrieval step
Tool use	Are tool calls and arguments right for the query?	LLM judge on tool calls
Safety	Is the content free of harmful or toxic material?	LLM judge plus refusal test cases
Product rules	Tone, format, required disclaimers	Guidelines judges or code-based checks
Latency and cost	Is it fast and affordable enough at expected volume?	Measured from traces

How do you build the framework?

1. Set the gates first. Write the pass threshold for each dimension before development starts, so results can't move the goalposts.
2. Build an evaluation dataset. Each record holds inputs and, where you have them, expectations: the ground-truth answer or facts. Include edge cases and known failure modes.
3. Pick scorers per dimension. Use LLM judges where quality is a judgment call, code-based scorers where the rule is deterministic, and natural-language guidelines for product-specific rules.
4. Calibrate with people. Have domain experts label a sample, then align the judges with that feedback so automated scores track what good means for your product.
5. Run, compare, decide. Evaluate each candidate and compare it with the current version, request by request.

How does Databricks support LLM evaluation?

MLflow 3 on Databricks scores traces with scorers: LLM judges and code-based checks that attach a quality assessment to each trace. A scorer can return pass/fail, true/false, a number or a category. Running mlflow.genai.evaluate() executes your app on each record, creates traces with scorer feedback, and records the run under the experiment's Evaluation runs tab. The Databricks tutorial, which evaluates a sales-email generator, follows this pattern:

import mlflow
from mlflow.genai.scorers import Guidelines, RelevanceToQuery, Safety

eval_results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=generate_sales_email,
    scorers=[
        RelevanceToQuery(),
        Safety(),
        Guidelines(
            name="professional_tone",
            guidelines="The email must be in a professional tone.",
        ),
    ],
)

Built-in judges cover most of the table above. Correctness, RetrievalSufficiency and ToolCallCorrectness need ground truth; RelevanceToQuery, RetrievalGroundedness, RetrievalRelevance, Safety, Guidelines and ToolCallEfficiency don't. Multi-turn judges such as ConversationCompleteness and UserFrustration score whole conversations.
Evaluation datasets are stored in Unity Catalog, which adds versioning, lineage, sharing and governance, and you can build them from interactions captured by MLflow Tracing. Each trace records the inputs, outputs, latency, token usage and cost of every intermediate step, so the traces from an evaluation run also show latency and cost. Domain experts can label traces directly, and a trace that carries a ground-truth expectation becomes a ready-made test case. To choose between versions, select two evaluation runs and compare them side by side.

When should you ship, and when should you iterate?

Ship when:

• Every gate passes on the current evaluation dataset.
• The side-by-side comparison shows no regressions on requests the previous version handled well.
• Known failure modes have a fallback, such as a refusal or a hand-off to a person.

Iterate when:

• Any gate fails, especially correctness or safety in a regulated domain.
• Judges and expert labels disagree, a sign the judges need alignment before you trust their scores.
• Latency or cost in the traces is outside your budget at expected volume.

FAQs

How big should the evaluation dataset be?

Big enough to cover your main request types and known failure modes. Start with a set you can review by hand, then grow it with production interactions that exposed problems.

Should I use LLM judges or human reviewers?

Both. Judges score every record on every run; expert labels on a sample tell you whether to trust them, and MLflow can align judges with that feedback.

How often should I re-evaluate?

After every change to a prompt, model, retrieval step or tool, and whenever new traces reveal failures your dataset doesn't cover. Add those traces to the dataset so the next run tests them.

Can I switch to a smaller model to cut cost?

Evaluate it against the same gates and dataset. If it passes every gate and the comparison shows no regressions, it's a ship candidate; if not, keep iterating.