Skip to main content

where to find LLM analytics

Summary

  • LLM analytics lives in two places: in your application's traces, which show what each request did, and at the model gateway, which shows who called which model and what it consumed.
  • For one app or agent, open-source MLflow's experiment Overview tab charts requests, latency, errors, token usage, quality scores and tool calls from logged traces.
  • For usage across every app on Databricks, the system.ai_gateway.usage system table records requests through Unity Gateway, with token counts, latency and the requesting principal.
  • Unity Gateway also has a built-in Usage Dashboard with a Cost Analysis page, and inference tables log requests and responses to Unity Catalog Delta tables.
  • When traces are stored in Unity Catalog, they become SQL-queryable tables you can chart in AI/BI Dashboards or join with business data.

where to find LLM analytics

You find LLM analytics in two layers. The application layer is your traces: what each request did, how long each step took, how many tokens it used and whether the answer was any good. The gateway layer is the record of every model call across all your apps: who called which model, how often and at what cost. In open source, MLflow covers the application layer through its experiment Overview tab. On Databricks, Unity Gateway system tables cover the gateway layer, and traces stored in Unity Catalog let you query both with SQL.

What counts as LLM analytics?

LLM analytics is the measurement of how LLM-powered applications are used and how well they perform. Typical questions are operational (request volume, latency, error rate), financial (tokens and spend by team, project or model) and qualitative (are answers correct, grounded and safe?). The first two come from logs. The third needs a scorer, an LLM judge or a code-based check, attached to each trace.

Where each answer lives

Question you're asking Where to find it
How is this agent performing today? MLflow experiment Overview tab: Usage, Quality and Tool Calls views
Why did this one request fail or run slow? The trace's span tree in the MLflow Traces tab
Which team or project is using the most tokens? system.ai_gateway.usage, grouped by request tags
What are we spending on models? Unity Gateway Usage Dashboard, Cost Analysis page
What exactly did users send and receive? Unity Gateway inference tables in Unity Catalog
Are answers getting worse? Quality charts built from scorer assessments on traces

Application analytics with open-source MLflow

MLflow's Dashboard (Overview) tab shows three views for an experiment. Usage covers Requests, Latency, Errors, and Token Usage & Token Stats. Quality shows charts "dynamically generated based on the assessments available in your traces." Tool Calls covers tool performance, usage, latency and error rate. Every tab has time range and time unit selectors. It needs a SQL-backed tracking store (SQLite, PostgreSQL or MySQL).
Per the token usage and cost docs, MLflow automatically tracks token usage and estimated cost for LLM calls within traces. Token tracking needs MLflow 3.2.0 or later and cost tracking 3.10.0 or later. Trace-level totals sit in the token_usage field of TraceInfo. For custom analysis, mlflow.search_traces() returns a pandas DataFrame by default.
MLflow Tracing is fully compatible with OpenTelemetry, so agents built in different frameworks can report to the same place.

How Databricks approaches LLM analytics

Gateway usage as a system table. Unity Gateway usage tracking writes to system.ai_gateway.usage, which captures request and response details for model services, model provider services and MCP services. Documented columns include destination_model, requester, latency_ms, input_tokens, output_tokens, total_tokens and request_tags. This query is from the docs:

SELECT
  request_tags['project'] AS project,
  COUNT(*) AS request_count,
  SUM(total_tokens) AS total_tokens
FROM system.ai_gateway.usage
WHERE request_tags['project'] IS NOT NULL
GROUP BY request_tags['project']
ORDER BY total_tokens DESC;

By default, querying the table requires both account admin and metastore admin roles.
Built-in dashboards. Account admins can create the Usage Dashboard from the Govern menu on the Unity Gateway page. Its Cost Analysis page shows Databricks cost and external model spend. Analyze Unity Gateway cost names the system tables behind those figures: system.billing.usage for Databricks-hosted models and system.ai_gateway.external_model_spend for external models.
Request and response logs. Inference tables log requests and responses from model services to Unity Catalog Delta tables, with fields such as status_code, latency_ms and requester.
Traces as tables. Databricks recommends storing traces in Unity Catalog, where they land as OpenTelemetry-format Delta tables (<table_prefix>_otel_spans, _otel_logs, _otel_metrics, _otel_annotations). You can query them through a SQL warehouse or chart them in AI/BI Dashboards, and the tracing overview points to Genie Code for asking questions of trace data in natural language.

Use cases

  • Chargeback: tag requests by project and report token totals per team from system.ai_gateway.usage.
  • Agent health review: check the Overview tab's Tool Calls view to find the tool with the highest error rate, then open its traces.
  • Quality trend: run judges on a sample of traces and watch the Quality view for drops after a release.

FAQs

Is there a prebuilt LLM analytics dashboard?

Yes. Open-source MLflow has the experiment Overview tab, and Unity Gateway has a built-in Usage Dashboard that account admins create from the Govern menu.

Why can't I query system.ai_gateway.usage?

Querying it requires both account and metastore admin roles by default, so ask an admin for access.

Can I get LLM analytics for agents running outside Databricks?

Yes. MLflow Tracing is OpenTelemetry-compatible, and Databricks documents trace ingestion that doesn't depend on where the agent runs.

Where do quality metrics come from?

From assessments on traces: scores written by LLM judges, code-based scorers or human reviewers. Without them, you get usage analytics only.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.