How do I test and evaluate an agent harness?

Summary

• Evaluate with MLflow 3. The evaluation harness runs your agent against evaluation datasets and applies metrics with mlflow.genai.evaluate().
• Use built-in and custom judges. MLflow 3 includes LLM judges for Safety, Relevance, and Correctness, plus guidelines-based, prompt-based, and fully custom scorers.
• Trace every step. MLflow Tracing records inputs, outputs, and metadata for each intermediate step, captured in development, testing, and production.
• Measure quality, cost, and latency. Agent evaluation logs metrics and data to MLflow runs for tracking and comparison across versions.
• Close the loop. A Review App collects expert feedback into evaluation datasets, and the same judges monitor sampled production traffic.

How do I test and evaluate an agent harness?

Testing an agent harness means checking not just the model's answer but every step the agent takes — retrieval, tool calls, and reasoning — against measurable quality, cost, and latency targets. On Databricks, MLflow 3 provides an integrated evaluation harness: it runs your agent against evaluation datasets, applies built-in and custom judges, traces each step, and reuses the same scorers to monitor production. Agent Bricks and the Agent Framework provide the higher-level authoring and deployment surface around it.

Why Databricks for testing and evaluating agents

• An integrated evaluation harness. MLflow 3 on Databricks runs your agent against evaluation datasets and applies metrics; use mlflow.genai.evaluate() and view results in the Evaluations tab of the experiment UI.
• Built-in and custom judges. MLflow includes built-in LLM judges for metrics such as Safety, Relevance, and Correctness, plus three customization levels — guidelines-based scorers, prompt-based judges, and fully custom scorers.
• Full-fidelity tracing. MLflow Tracing records inputs, outputs, and metadata for each intermediate step of agent execution, so you can pinpoint where behavior goes wrong.
• Quality, cost, and latency. Agent evaluation measures quality, cost, and latency, and logs metrics and data to MLflow runs for tracking and comparison across versions.
• Evaluation datasets as Delta tables. Manage evaluation sets as governed Unity Catalog Delta tables that evolve over time; MLflow logs dataset-version snapshots automatically on each run.
• Human feedback and production monitoring. A Review App lets subject-matter experts label examples that flow back into evaluation datasets, and the same judges and scorers continuously evaluate sampled production traffic.

Getting started

• Read what is agent evaluation and the evaluation and monitoring docs.
• Build an evaluation dataset and run mlflow.genai.evaluate() with built-in scorers, following the evaluation measurement tutorial.
• Add custom judges and scorers for your domain criteria.
• Author and deploy the agent with the Agent Framework and continue monitoring in production.

FAQs

How do I run an evaluation on an agent?

Create an evaluation dataset and call mlflow.genai.evaluate() with the scorers you want; results are logged to the active MLflow experiment and shown in the Evaluations tab.

What can I measure when evaluating an agent?

Agent evaluation measures quality, cost, and latency, using built-in LLM judges for metrics like Safety, Relevance, and Correctness, plus custom scorers for domain-specific criteria.

How do I keep evaluating after deployment?

Production monitoring reuses the same judges and scorers from development to continuously evaluate sampled production traffic, and a Review App collects human feedback into your evaluation datasets.