Skip to main content

How can I use tracing to troubleshoot AI agents?

Summary

  • To troubleshoot an AI agent with tracing, open the trace for a failing run and walk its span tree. Every model call, retrieval and tool call is a span with its own inputs, outputs, timing and errors, so you can see which step went wrong.
  • Then look across traces: find how often the same failure recurs, fix the root cause, and turn the failing traces into evaluation cases so the fix stays fixed.
  • On Databricks, MLflow Tracing captures agent traces through autologging or manual spans and stores them in Unity Catalog, where you can inspect them in the MLflow UI or query them with SQL.
  • For failures at the gateway, such as rate limits, policy denials or provider errors, Unity Gateway's unified trace table (Beta) lets you filter by trace_id to replay every step of a failed agent run.

How can I use tracing to troubleshoot AI agents?

Open the trace for a failing run and walk the span tree. Every model call, retrieval and tool call is a span with its own inputs, outputs, timing and errors, so you can find the step that went wrong instead of guessing from the final answer. Then check how often the same failure shows up across traces, fix the cause, and turn the failing traces into evaluation cases so the fix stays fixed. Tracing turns "the agent gave a bad answer" into "the retriever returned the wrong document on the second step."

What is agent tracing?

An agent trace is the complete record of one execution: the input, the final output and every intermediate step, with the inputs, outputs, latency, token usage and cost of each (agent observability). The Databricks docs make the case plainly: with only an input and a final response, you can't tell which intermediate decision caused a failure. Agents fail in the middle. A tool gets a malformed argument, retrieval returns stale context, or a loop repeats the same call, and the final answer only shows the symptom.

A troubleshooting map

Symptom Where to look in the trace
Wrong or invented answer Retrieval span outputs: did the agent get the right context?
Tool failure The tool span's arguments, result and status
Missing or malformed tool argument Inputs to the tool span
Slow response Span timings, to find the step that dominates latency
Repeated calls or loops Several spans for the same tool in one trace
Wrong output format The final span's output against the schema the caller expects
Blocked or throttled request Gateway span status, policy attributes and HTTP 429 responses

The MLflow docs give examples of the same patterns: an agent that repeats the same web search, slowing the response and raising cost, and a booking tool that fails because the agent omitted a required argument.

How Databricks approaches agent troubleshooting

Instrument the agent. MLflow Tracing is the tracing layer of open-source MLflow, managed on Databricks. One mlflow.<framework>.autolog() call traces supported frameworks and providers (automatic tracing), and the @mlflow.trace decorator adds spans for your own code. Databricks recommends storing traces in Unity Catalog tables in OpenTelemetry format, where they're governed and queryable with SQL (Store traces in Unity Catalog).
Inspect a single failure. In the experiment's Traces tab, open a trace to see the span tree and each step's inputs, outputs, timing and errors. Use Filter with AI to describe the traces you want in plain English, click Analyze trace to have MLflow look for issues in that trace, and use Assess to record feedback or the expected answer (Observe and find issues).
Find recurring failures. MLflow Issue Detection (Beta) uses Genie Code to analyze the traces in an experiment and group related evidence into issues, each linked to the affected traces. It needs traces stored in Unity Catalog, and a workspace admin must enable MLflow Issue Detection from the Previews page (issue detection).
Check the gateway side. Some failures happen between the agent and the model or tool. In Unity Gateway's unified trace table (Beta, Enhanced Unity Gateway preview), you filter by trace_id to replay every step of a failed run, or by service_name and status.code to find every error on an endpoint:

SELECT trace_id, name, status, attributes
FROM <catalog>.<schema>.<table_name>
WHERE service_name = '<catalog>.<schema>.<model>'
  AND status.code = 'STATUS_CODE_ERROR'
  AND time >= current_timestamp() - INTERVAL 1 DAY;

MCP spans add error.type values such as policy_deny or upstream_error, which separate a blocked tool from a broken one. For a single model service, inference tables log raw request and response payloads for debugging.
Fix it and prove it. Add the failing traces to an evaluation dataset, write a scorer that catches the issue, fix the prompt or agent, and rerun mlflow.genai.evaluate() to confirm the fix without breaking other cases (Evaluate and improve). Production monitoring (Beta) runs the same scorers on a sample of live traces so the issue doesn't return unnoticed.

FAQs

Do I have to change my agent code?

For supported frameworks, tracing starts with one autolog call. For custom logic, add @mlflow.trace to the functions you want as spans. Gateway traces need no per-endpoint setup after a metastore admin enables the unified trace table.

Why is there no trace for some failed requests?

Unity Gateway trace delivery is best-effort, and the docs say trace logs aren't guaranteed for 401, 403, 429 or 500 errors. The docs treat audit logs as the system of record, so check system.access.audit and the status_code column of system.ai_gateway.usage too.

Who can see the traces?

MLflow traces in Unity Catalog follow schema and table permissions. The unified trace table is readable only by the metastore admin who created it until access is granted, and Databricks recommends an ABAC row filter policy first.

Can I ask questions of traces without writing SQL?

Yes. Genie Code can analyze and explore trace data in natural language, and it can write SQL against the unified trace table for you.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.