How do I monitor and debug AI agents running in a harness?

Summary

• MLflow Tracing gives end-to-end observability. Every step of an agent's execution is captured as a trace — the full span hierarchy, each tool call with its inputs and outputs, latency at each step, and token usage — so you can pinpoint the source of unexpected behavior. See Observability for any agent, anywhere.
• Turn tracing on with one line or instrument manually. Autologging captures traces for supported frameworks automatically, and the @mlflow.trace decorator lets you instrument any custom function. Traces are stored in Unity Catalog for governance and querying.
• Evaluate quality with LLM judges and scorers. MLflow evaluation measures quality, cost, and latency using built-in and custom LLM judges against evaluation datasets, with review apps and labeling sessions to collect human feedback.
• Monitor production with the same judges. The evaluation configuration you use in development runs automatically against live production traces, so you detect regressions, drift, and emerging failure patterns as they happen.
• Debug interactively and log every request. Genie Code answers natural-language questions over your trace data to find failing traces and bottlenecks, while AI Gateway inference tables log every request and response from agent endpoints into a Unity Catalog Delta table.

How do I monitor and debug AI agents running in a harness?

On Databricks, you monitor and debug AI agents built with Agent Bricks and the Mosaic AI Agent Framework using MLflow. MLflow Tracing records what happens inside every agent request, MLflow evaluation scores quality with LLM judges, production monitoring applies those judges to live traffic, and Genie Code plus AI Gateway inference tables let you interrogate and log every interaction. Traces and evaluation results are stored in Unity Catalog, so observability data is governed alongside the rest of your data. See Observability for any agent, anywhere.

Why Databricks Agent Bricks makes agents observable

• MLflow Tracing captures every step. A trace records the inputs, outputs, and metadata of each intermediate step in an agent's run: the full parent-child span hierarchy and execution flow, every tool invocation with its inputs and outputs, latency at each step, and token usage across all spans. Because each step is captured, you can quickly find the source of unexpected behavior. See MLflow Tracing.
• Enable tracing automatically or manually. Autologging captures traces for supported agent frameworks with a single call, and the @mlflow.trace decorator lets you instrument any custom function with a chosen span type. Traces are stored in Unity Catalog so they can be governed and queried with standard data tools.
• Evaluate agent quality with LLM judges. MLflow evaluation measures quality, cost, and latency using built-in and custom LLM judges and scorers that you tailor to your use case, run against evaluation datasets you build. During development, review apps collect feedback through a chat UI for quick vibe checks and structured labeling sessions for expert feedback. See Evaluate, debug, and optimize agents and Agent Evaluation.
• Production monitoring with the same configuration. Production monitoring applies the same LLM judges and custom metrics you used in development to live production traces. MLflow automatically evaluates those traces, helping you detect regressions, drift, and emerging failure patterns and turning evaluation into an ongoing practice as the agent evolves. See production monitoring.
• Log every request with AI Gateway inference tables. The AI Gateway inference table captures incoming requests and outgoing responses from agent production endpoints into a Unity Catalog Delta table. When MLflow tracing is enabled, it also logs the inputs and outputs of each component inside the agent, so you can analyze traffic with existing data tools and combine it with agent evaluation for quality monitoring, debugging, and optimization. See Build compound AI systems faster.
• Debug interactively with Genie Code. Genie Code gives natural-language access to the observability data in an MLflow experiment. You can investigate failing traces and examine span trees to pinpoint root causes, identify latency bottlenecks, analyze token-usage and cost patterns, review scores from human feedback and LLM judges, and surface common failure patterns across traces. See Genie Code for agent observability.

Getting started

• Enable MLflow Tracing with autologging, then test locally to confirm traces appear before deploying.
• Build an evaluation dataset and configure LLM judges with MLflow evaluation and monitoring.
• Set up production monitoring so your judges run automatically against live traces.
• Use Genie Code to debug traces conversationally and trace, evaluate, and monitor AI as one workflow.

FAQs

What does an MLflow trace capture for an agent?

A trace records each intermediate step of a run: the span hierarchy and execution flow, every tool call with its inputs and outputs, latency at each step, and token usage across all spans — all stored in Unity Catalog.

How do I evaluate an agent's quality?

Use MLflow evaluation with built-in and custom LLM judges and scorers against an evaluation dataset to measure quality, cost, and latency, and collect human feedback through review apps and labeling sessions.

How do I monitor an agent in production?

Production monitoring applies the same LLM judges and custom metrics from development to live traces automatically, so you can detect regressions, drift, and emerging failure patterns as they occur.

How do I debug a failing agent interactively?

Use Genie Code to ask natural-language questions over your trace data — finding failing traces, latency bottlenecks, high token usage, and common failure patterns — and query AI Gateway inference-table logs of every request and response.

What are best practices for debugging agents?

Enable MLflow trace autologging early, document each tool clearly so the agent understands how to use it, add timeouts and token limits to LLM calls, and test locally to verify traces appear before deploying to production.