How can AI agents diagnose model performance issues?

Summary

• AI agents fail silently in production. Context windows decay, tool calls become confused, and latency spikes go unnoticed until the agent stops working.
• Diagnosis requires execution traces. Modern observability captures every step and decision in real time so root causes become visible, not guesses.
• Genie Code analyzes traces in natural language. Ask questions about performance issues ("why did costs spike?") and Genie Code automatically discovers latency bottlenecks, error patterns, and cost drivers.
• MLflow tracing and inference tables capture everything. Every LLM call, tool invocation, and retrieval gets logged with timing, tokens, and latency, making multi-step agent workflows fully transparent.
• Seven common failure modes are now diagnosable. Context rot, tool overload, brittle tool wiring, latency creep, irrelevant retrieval, weak verification, and unmonitored actions can all be caught within hours.

How can AI agents diagnose model performance issues?
AI agents fail quietly in production. Context windows grow stale, tool descriptions get ignored, multi-step chains stall mid-execution, or token costs spike unexpectedly, yet the agent keeps running and returning plausible-sounding nonsense. By the time you notice the performance cliff, the damage is already done.
The core problem: agent behavior is opaque. Traditional application logging captures request-response pairs. Agent logs need to capture reasoning traces (the branching decisions, tool calls, retrieved context, and intermediate results) at every step. Without visibility into that trace hierarchy, "the agent failed" is all you know.
Databricks Genie Code solves this by making agent diagnosis fast and natural. Describe what went wrong in plain language ("why are our costs spiking?" or "why did the agent miss this step?"), and Genie Code automatically analyzes your execution traces to find root causes. It discovers latency bottlenecks, error patterns, context decay, tool confusion, and cost drivers. The 7 common failure modes that sink production agents become visible.

What is an agent execution trace?

A trace is a structured record of every decision an agent makes. It captures the span hierarchy: top-level requests spawn sub-spans (tool calls, retrievals, reasonings), each with timestamps, latency, token counts, and outputs. Modern observability requires capturing these traces automatically, in a format that tools and AI can query.
MLflow 3.0 Tracing does this out of the box. When you instrument an agent with @mlflow.trace or enable autolog, every step lands in a trace without manual logging. Unity Gateway's Inference Tables go further: they automatically capture HTTP status, request/response JSON, model latency, and token usage for every LLM call, creating a queryable audit trail.

Key capabilities to evaluate

Capability	Why it matters
Automatic span capture	Every tool call, retrieval, and LLM invocation is logged with timing and metadata. No code changes required.
Trace hierarchy inspection	Understand which sub-spans caused latency, failed, or consumed unexpected tokens.
Cost attribution	Track tokens and costs per tool, per step, per version. Find where the budget is actually going.
Latency percentiles	Identify P50/P95/P99 slowdowns and their root causes (retrieval lag, LLM timeout, tool error).
Error discovery	Catch silent failures: tool calls that return empty results, chains that halt mid-step, or context misses.
Natural language analysis	Query your traces in plain English ("why did this request take 15 seconds?") without writing SQL or parsing JSON.
Quality scoring	Compare agent output quality against human feedback, LLM judges, or programmatic checks over time.
Version tracking	Monitor performance changes across agent versions or model updates.

How Databricks approaches agent diagnostics

Genie Code is Databricks' natural language AI agent for diagnosing agent performance. Point it at your traces, either directly in MLflow or stored in Inference Tables, and ask questions in natural language. It returns root-cause analysis with evidence.
Genie Code works in three layers:

1. Automated trace instrumentation (MLflow 3.0): Your agent code declares spans with @mlflow.trace or mlflow.start_span(). Every LLM call, tool invocation, and retrieval is captured. No vendor lock-in: traces are JSON and queryable via SQL.
2. Unified observability (Unity Gateway + Inference Tables): Route agent calls through Databricks Unity Gateway and inference tables auto-capture HTTP status, request/response payloads, model run times, and token consumption. This creates a queryable audit trail for compliance and debugging.
3. Natural language root-cause engine (Genie Code): Genie Code reads your traces, recognizes common failure patterns (context rot, tool overload, brittle tool wiring, latency spikes, irrelevant retrieval, weak verification, missing guardrails), and explains what went wrong and why.

Example: A team saw unexpected token cost growth on their RAG agent. Instead of manually parsing logs, they asked Genie Code: "Why did spending jump?" Genie Code traced the issue: retrieval was now returning more documents per query due to an index change, so the agent was sending longer context to the LLM. Cost spike explained and actionable. Reduce retrieval limit or improve ranking.

Common production agent failure modes

Once you can see your traces, you can diagnose:

• Context rot: Conversation history grows; early context gets ignored; reasoning degrades.
• Tool overload: Too many tools cause wrong tool choices or repeated calls.
• Brittle tool wiring: Tool descriptions or schemas change silently; agent calls tools with stale parameters.
• Latency creep: Multi-step agents drift toward 10+ second response times; end users leave.
• Irrelevant retrieval: Search returns wrong documents; agent hallucinates.
• Weak verification: Agent marks work complete without checking; downstream systems get bad data.
• Unmonitored actions: Reversible decisions (database writes, emails sent) execute without audit trails.

Getting started

Start with MLflow tracing. Instrument your agent code:

import mlflow

@mlflow.trace(name="agent_step")
def run_agent_step(context):
    # Your agent logic
    return result

Or enable autolog for frameworks like LangChain:

mlflow.langgraph.autolog()

Traces flow to Unity Catalog. Query them via SQL or point Genie Code at them. If you use Unity Gateway for LLM calls, inference tables auto-populate without extra instrumentation.

FAQs

Do I need to change my agent code to get tracing?

No. MLflow autolog works with popular frameworks (LangChain, CrewAI, LangGraph) with one decorator. Manual @mlflow.trace adds control but is optional. If you use Unity Gateway, you get inference-table traces for free.

What if I'm not using Databricks infrastructure?

Genie Code works on any MLflow traces or OpenTelemetry-compatible exports stored in Unity Catalog. You can export traces from other systems and ingest them.

How quickly can Genie Code diagnose an issue?

Seconds to minutes for trace analysis of hundreds or thousands of runs. Real diagnostics catch issues that would take teams days to uncover manually.

Can I use this for non-agents?

Yes. MLflow tracing and Genie Code work on any multi-step AI workflow: RAG systems, model evaluation pipelines, LLM chains, or data transformation jobs. Any system where you need to see what's happening inside.