What metrics should I track for AI agents and MCP tools?

Summary

• Track four families of metrics for AI agents: reliability (error rates by type), performance (latency percentiles and time to first byte), usage and cost (requests, tokens, spend by owner), and quality (judge and feedback scores).
• MCP tools need their own view: call volume and error rate per tool, the JSON-RPC method behind each request, policy denials, and who calls which tool on whose behalf.
• Infrastructure health isn't enough on its own. An agent can return fast, error-free answers that are wrong or that took needless tool calls, so pair operational metrics with tool-call correctness and efficiency scores.
• On Databricks, Unity Gateway logs model and MCP requests to system tables and a built-in usage dashboard. MLflow judges and production monitoring cover quality.

What metrics should I track for AI agents and MCP tools?

Track reliability, performance, usage and cost, and quality for every agent. Break each one down by tool for MCP servers. In practice, that means error rates by error type, latency percentiles and time to first byte, request and token volume with spend attributed to an owner, and judge scores for correctness, groundedness and tool use. For MCP tools, add per-tool call volume and failure rate, policy denials and caller identity. The goal isn't a bigger dashboard. It's being able to answer, for any agent, whether it's working, what it costs, and whether its answers can be trusted.

The metrics that matter

Metric family	Example metrics	Why it matters
Reliability	Error rate, HTTP status distribution, error type, fallback attempts	Shows broken tools and flaky providers before users report them
Performance	Latency percentiles (P50 through P99), time to first byte	Long tails hurt agents, because each step adds wait
Usage	Requests, active users, tokens per request, cache reads	Shows adoption and spots context bloat
Cost	Spend by agent, model, user, team or project tag	Ties spend to an owner and a budget
MCP tool health	Calls and errors per tool, JSON-RPC method, server type	Finds the tool behind an agent's retries
Governance	Policy denials and approvals, rate-limit hits, on-behalf-of calls	Shows guardrails working, and agents testing their limits
Quality	Correctness, groundedness, relevance, safety, guideline adherence	Catches confident wrong answers
Tool use	Tool-call correctness, tool-call efficiency	Catches wrong tools, bad arguments and redundant calls
Conversation	Completeness, user frustration, knowledge retention	Judges the whole session, not one turn

Two habits make these metrics useful. Slice everything by agent, tool, model and owner, and keep enough trace detail to drill from a spike to the run that caused it.

How Databricks approaches agent and MCP metrics

Usage, performance and cost from Unity Gateway. Unity Gateway governs model services and MCP services from one control plane. Each request lands in the system.ai_gateway.usage system table with latency, time to first byte, tokens, status code, requester, tags and routing details for fallback attempts. The built-in usage dashboard turns this into tabs. The Performance tab tracks latency percentiles, time to first byte, error rates and HTTP status distributions. The Usage tab shows consumption by service, workspace and requester, plus cache hit ratios. The Cost Observability tab breaks down spend by model service, target model, user and tags. The External MCP Server tab shows request volume, error rates, users and connections.
Per-tool detail for MCP. For MCP requests, the usage table's mcp_metadata struct records the tool_name, the server_type and the json_rpc_method. That lets you separate tools/call traffic from tools/list discovery and initialize requests. The unified trace table (Beta) goes deeper. Each MCP span records the tool's arguments and result, rpc.response.status_code (the docs call it the primary MCP failure signal), and an error.type such as policy_deny or upstream_error. Each service policy evaluation is recorded as an event with its action (ALLOW, DENY or ASK).
Governance metrics. Rate limits on MCP services are set in requests per minute, and on model services in requests or tokens per minute, so limit hits are a capacity signal. The system.access.audit table logs each MCP request as an mcpCall event, and run_by and run_as separate a human caller from an agent acting on their behalf.
Quality from MLflow. MLflow built-in judges cover Correctness, RetrievalGroundedness, RelevanceToQuery, Safety and Guidelines. For tools, ToolCallCorrectness checks whether the tool calls and arguments were right for the query, and ToolCallEfficiency checks for redundancy. Multi-turn judges such as ConversationCompleteness, UserFrustration and ConversationalToolCallEfficiency score whole sessions. Production monitoring (Beta) runs these scorers on a configurable sample of live traces, so quality sits next to the operational metrics.

How to put it into practice

1. Start with the built-in usage dashboard. An account admin can create it from the Govern menu in Unity Gateway. It covers reliability, performance, usage, cost and MCP traffic without custom SQL.
2. Tag for ownership. Use service tags when each team has its own model service, and the Databricks-Ai-Gateway-Request-Tags header when teams share one.
3. Pick a few alerts. Error rate per tool, tail latency per agent and spend against budget catch most problems early.
4. Add judges where errors are invisible. Run tool-call and correctness judges on sampled production traffic.

FAQs

What is the most important metric for an MCP tool?

Error rate per tool, read alongside call volume. A tool that fails often makes agents retry and work around it, which costs tokens and time even when the task eventually succeeds.

Should tools/list calls count as tool usage?

Track them separately. The json_rpc_method field distinguishes tools/call, which invokes a tool, from tools/list discovery and initialize, which starts a session.

How do I measure agent quality, not just uptime?

Score traces with LLM judges and code-based scorers. Use the same scorers in development and production so results stay comparable as the agent changes.

Are trace tables complete enough for compliance metrics?

Not on their own. Unified trace delivery is best-effort, and the docs say trace logs aren't guaranteed for some error responses. Use the audit log system table as the system of record.