How do I monitor AI agents in production?

Summary

• Monitor AI agents in production on three layers: traces that show what the agent did on every request, quality scores that show whether it got the answer right, and operational metrics (latency, errors, tokens, cost, access) that show whether it's healthy and affordable.
• Agent monitoring differs from service monitoring because agents fail quietly. A request can return a fluent, wrong answer with a success status, so quality needs its own automated checks.
• Run the same scorers in production that you used in development, so a regression shows up as a score change you can trace back to specific requests.
• On Databricks, MLflow Tracing stores traces in Unity Catalog, MLflow production monitoring (Beta) scores live traces, and Unity Gateway (generally available) tracks usage, cost and access for every model and MCP call.

How do I monitor AI agents in production?

Instrument every agent with tracing, store the traces somewhere you can query and govern, and watch three things. Behavior: what each request did, step by step. Quality: whether the answers were correct, safe and grounded, measured by automated judges on live traffic plus human feedback. Operations: latency, error rates, token use, cost and who called what, measured where the traffic flows. Then alert on regressions and turn the failures you find into test cases, so each fix is checked before the next release.

Why is monitoring agents different?

A web service fails loudly: errors, timeouts, status codes. An agent can fail politely. It returns a well-formed answer that's wrong, calls the wrong tool, or loops through retries and still reports success. Application logs show that a request finished; a trace shows which model call, tool call or retrieval step went wrong and what it received. And because agents are multi-step and non-deterministic, quality needs continuous measurement, not a one-time test before launch.

What should you monitor?

Signal	What it tells you	Where it comes from
Step-level traces	Which step failed or slowed down, and with what inputs	Tracing on every request
Quality scores	Correctness, safety and groundedness on real traffic	Automated scorers (LLM judges or code checks) on sampled traces
Human feedback	Whether experts and users agree with the agent	Ratings and expectations recorded on traces
Latency and errors	Service health and reliability	Request metrics at the gateway
Tokens and cost	Spend per user, agent and model, and where it's wasted	Usage and billing records
Tool activity	Which tools are called, how often and how often they fail	Tool spans in traces and gateway MCP usage records
Access	Which person or agent called which model or tool	Audit logs

How Databricks approaches production agent monitoring

Tracing. MLflow Tracing, the open-source tracing layer in MLflow, records the inputs, outputs, latency, token usage and cost of every intermediate step. On Databricks you instrument supported frameworks with one mlflow.<framework>.autolog() call. Databricks recommends storing traces in Unity Catalog as Delta tables in OpenTelemetry format, governed by schema and table permissions and queryable with SQL, AI/BI dashboards and alerts. In an experiment's Traces tab you can filter traces in plain English, open a span tree, or click Analyze trace.
Quality on live traffic (Beta). Production monitoring runs MLflow 3 scorers on a configurable sample of incoming traces and attaches the results as feedback to each one. Scorers can be built-in judges such as Safety, custom LLM judges, code-based checks or multi-turn judges for whole conversations. You register a scorer, start it with a sampling configuration, and can filter which traces it evaluates. Because the same scorers run in development and production, evaluation stays consistent.
Feedback loop. Reviewers record feedback and expectations directly on a trace, and a trace with a ground-truth expectation can be added to an evaluation dataset. That's the loop the docs describe: find an issue in production, write a scorer for it, fix the agent, evaluate the fix, and keep monitoring.
Gateway metrics and audit. Agents that call models and MCP servers through Unity Gateway log every request to system.ai_gateway.usage. Model requests record token counts, latency, status code and requester; MCP requests record call and service metadata. The built-in usage dashboard's Performance tab tracks latency percentiles, time to first byte and error rates, and each model or MCP service has a Metrics tab in Catalog Explorer. Inference tables log request and response payloads for a model service. The unified trace table (Beta) puts gateway and client-side spans for all services in one table. system.access.audit records who accessed which AI service, and its run_by and run_as fields separate a person from an agent acting for them.
Resilience. Fallbacks retry failed requests against backup models to keep agents available, and rate limits on model and MCP services cap consumption per service, per user or per group.

FAQs

What's the difference between evaluation and production monitoring?

Evaluation tests an agent against a fixed dataset before release. Production monitoring runs the same scorers continuously on live traffic, so you catch regressions and new failure modes after release.

Do I need to score every request?

No. Score every trace for critical checks such as safety, and sample at a lower rate for expensive LLM judges. Databricks production monitoring lets you set a sampling rate per scorer.

Can I monitor agents built outside Databricks?

Yes. MLflow Tracing works with many agent frameworks and stores traces in OpenTelemetry format, and any agent or coding tool that sends traffic through Unity Gateway shows up in its usage table and dashboards.

Which metrics should trigger an alert?

Start with error rate, latency, a drop in quality scores and a jump in tokens per request. Traces stored in Unity Catalog tables can drive alerts and AI/BI dashboards directly.