How do I trace AI agent activity end to end?

Summary

• The unified trace table captures every request and response across all Unity Gateway services into a single Unity Catalog table in OpenTelemetry format, providing complete visibility with no per-endpoint setup required.
• Once the unified trace table is enabled by a metastore admin, you can filter by trace ID to replay every step of a failed agent run, analyze token usage patterns, and identify error patterns in LLM calls.
• MLflow tracing integration automatically logs agent invocations to Delta tables by passing a trace_destination parameter, capturing full request payloads, responses, latency, and status codes for debugging.
• The Unity Gateway usage dashboard shows request volume, token consumption, and latency metrics; the system.ai_gateway.usage table enables deeper analysis by requester, endpoint, time range, and request tags.
• Use natural language queries in Genie or data lineage tools like Lakewatch to explore agent logs, identify costly workflows, understand how agents spend time, and detect suspicious activity.

How do I trace AI agent activity end to end?
Use the unified trace table to capture every AI agent request and response in a single Unity Catalog table. This provides complete end-to-end visibility without per-endpoint setup. For detailed debugging, enable payload logging to inspect exact prompts, model responses, and latency. Then query the trace data using SQL, use natural language with Genie, or integrate with analysis tools like Lakewatch to identify errors, optimize costs, and detect security issues.
The most effective approach combines the unified trace table (for completeness), payload logging (for debugging), and MLflow tracing (for end-to-end correlation across application and agent code).

What is end-to-end AI agent tracing?

End-to-end tracing means recording every step of an agent's execution: which LLM was called, what prompt was sent, what the model returned, which tools or MCPs were invoked, what data was retrieved, and whether the agent succeeded or failed. Tracing lets you debug failures, optimize costs, audit who called what, and detect policy violations.

Unified Trace Table (Primary Method)

The unified trace table is the recommended approach for comprehensive visibility [1]. It captures every request and response across all Unity Gateway services into one OpenTelemetry-formatted table in Unity Catalog.
Key features:

Feature	Benefit
Complete	All activity lands in one place; no blind spots
Enforceable	Metastore admin sets it up once; applied across all services
Open standard	Built on OpenTelemetry, so any analysis tool can consume it
Requires no code changes	No per-endpoint setup; automatic for all Gateway traffic

Setup

A metastore admin creates the unified trace table with CREATE TABLE and USE SCHEMA permissions [1]. Once enabled, all agent activity is automatically logged.

Querying for Troubleshooting

Filter by trace ID to replay every step of a failed agent run. For example, if an agent failed at 3:45 PM, find its trace ID in the table and reconstruct the full execution flow: which prompt was sent to the LLM, what was returned, which tool was called, what error occurred, and where execution stopped.

Payload Logging with Inference Tables

For detailed request and response debugging, enable inference tables to capture complete payloads, latency, status codes, and errors to Delta tables. When an agent fails, you can trace exactly what prompt was sent, what the model returned, and where it broke. This is essential for debugging complex multi-step agent workflows.

MLflow Tracing Integration

Agent invocations are automatically logged to MLflow experiments using OpenTelemetry standards. Enable tracing by passing a trace_destination parameter in the request:

from mlflow.tracing import get_tracing_context_headers_for_http_request
import mlflow

with mlflow.start_span("agent-run") as span:
    span.set_inputs({"input": "Analyze sales data"})
    trace_headers = get_tracing_context_headers_for_http_request()
    response = client.responses.create(
        model="databricks-claude-sonnet-4-5",
        input=[{"type": "message", "role": "user", "content": "Analyze sales data"}],
        extra_body={
            "trace_destination": {
                "catalog_name": "my_catalog",
                "schema_name": "traces",
                "table_prefix": "agent"
            }
        },
        extra_headers=trace_headers
    )

This correlates application-side tracing with agent execution, making it easy to trace from user request to final agent result.

Usage Dashboard and System Tables

Access the Usage Dashboard at Unity Gateway > Govern > Usage Dashboard in Databricks to track request volume, token consumption, and latency at a glance. For deeper analysis, query the system.ai_gateway.usage table directly to analyze metrics by requester, endpoint, time range, request tags, and cost.

Analysis and Troubleshooting Tools

Once tracing is enabled, you have multiple paths to insight:

• SQL queries: Write SQL to analyze the trace table, find slow agent runs, or identify error patterns.
• Genie: Use natural language to explore logs ("Show me failed agent runs today", "Which endpoints consumed the most tokens?"). Genie automatically generates the right SQL and visualizations.
• Lakewatch: Analyze Unity Gateway traces to detect suspicious activity, policy violations, and unauthorized access attempts.

Each tool is suited to different users: data engineers prefer SQL, business analysts use Genie, and security teams rely on Lakewatch.

Common Tracing Use Cases

Debug failures: Filter by trace ID to see every step. Did the LLM call fail? Did a tool invocation time out? Where did the agent get stuck?
Optimize costs: Find which requests consume the most tokens. Are certain agent workflows inefficient? Which endpoints have high latency?
Audit and compliance: Review which user called which endpoint, when, with what parameters. Generate a complete audit trail for governance reviews.
Reduce token waste: Identify common error patterns in LLM calls or MCP invocations (e.g., repeated failed retries, malformed prompts) and fix them upstream.
Security investigation: Trace unauthorized access attempts, detect policy violations, and investigate suspicious patterns.

FAQs

Does tracing add overhead or latency to my agents?

No. Tracing is non-blocking. Logs are written asynchronously, so agent performance is not affected.

What permissions do I need to set up tracing?

Metastore admin role and CREATE TABLE, USE CATALOG, and USE SCHEMA permissions in the target Unity Catalog and schema. Metastore admins typically set this up once; developers then query the results.

Can I combine unified trace table with MLflow tracing?

Yes. Both can be enabled simultaneously. The unified trace table captures all Gateway activity; MLflow tracing adds application-level correlation. Use both for the most complete observability.

How long is trace data retained?

Trace data is stored in your Unity Catalog, so retention is controlled by your Delta table policies. You can keep it indefinitely or implement a retention policy based on your compliance needs.

Can I export trace data for external analysis?

Yes. The trace table is a standard Delta table in Unity Catalog. You can export it to external data warehouses, visualization tools, or AI analysis platforms for deeper investigation.