Skip to main content

tools for genai observability 2026

Summary

  • In 2026, GenAI observability tools have to handle agents, not just single prompts: step-level traces of model, retrieval and tool calls, sessions for multi-turn conversations, and token and latency data on every span.
  • OpenTelemetry is the baseline. MLflow, the open-source option, ingests and exports traces in the OpenTelemetry GenAI semantic conventions, so instrumentation isn't tied to one backend.
  • Observability now includes quality. Look for LLM judges that run automatically on a sample of live traces, plus human feedback recorded on the trace itself.
  • Traces are increasingly treated as governed data. On Databricks, MLflow stores them as Unity Catalog Delta tables with SQL access, built-in dashboards and production monitoring (Beta).

tools for genai observability 2026

As of October 2026, a GenAI observability tool needs to trace multi-step agents, follow OpenTelemetry, score quality on live traffic, and keep traces in storage you can query and govern. Logging prompts and responses alone no longer explains why an agent picked the wrong tool or looped on a retrieval. The open-source baseline is MLflow, whose current release is 3.17.2 (published October 21, 2026); its tracing is OpenTelemetry-compatible and its evaluation runs the same judges in development and in production.

What GenAI observability means now

GenAI observability is the practice of seeing how an LLM app or agent behaves on real requests and whether its answers are any good. Databricks describes agent observability and quality as "measuring, debugging, and improving your AI agent from your first trace through production". Traces are the foundation: each one records the input, the output and every intermediate step, with latency, token usage and cost for each step.

What to look for in a 2026 GenAI observability tool

Capability What to look for
Agent-aware tracing Spans for model calls, retrieval, tool calls and sub-agents, with sessions for multi-turn conversations
OpenTelemetry support OTLP ingestion plus export, using the GenAI semantic conventions
Cost and token tracking Token usage and cost per step, not only per request
Online evaluation LLM judges that score a sample of live traces in the background
Human feedback Ratings, comments and expected answers stored on the trace
Traces as data SQL access, long retention and access controls on prompt data
Coding-agent access An MCP server so coding agents can search traces and log feedback

What open-source MLflow provides

Here is how open-source MLflow handles the core items. Autologging and the @mlflow.trace decorator produce agent traces, and MLflow documents sessions, PII redaction and sampling for high-volume apps. Its automatic evaluation runs LLM judges on traces as they're logged, which MLflow says is "often referred to as online evaluation". A built-in judge is registered once, then started with a sampling rate and an optional filter:

from mlflow.genai.scorers import Safety, ScorerSamplingConfig

safety_judge = Safety().register(name="production_safety_check")
safety_judge = safety_judge.start(
    sampling_config=ScorerSamplingConfig(
        sample_rate=0.1,
        filter_string="metadata.environment = 'production'",
    ),
)

Recent releases include changes for teams that run a shared MLflow server. The MLflow 3.17.0 release notes list fine-grained permissions that control access to contained resources "including runs, traces, and versions", and opt-in daily summaries that cut SQL analytics overhead on large trace datasets. Since MLflow 3.7.0, a new MLflow server uses SQLite by default in place of file-based storage, and the MLflow docs mark the file store as deprecated as of MLflow 3.6.0.

How Databricks approaches GenAI observability

On Databricks, MLflow is "the same open source product, just managed and integrated with the Databricks platform". The managed pieces that matter for observability:

  • Traces in Unity Catalog. Databricks recommends storing traces as Unity Catalog Delta tables in OpenTelemetry format, with schema- and table-level permissions, no per-experiment trace limit and direct SQL access.
  • Built-in dashboards. The MLflow experiment UI includes observability dashboards for traces in Unity Catalog, covering trace volume, errors, latency, token usage and cost.
  • Natural-language analysis. Genie Code answers questions about trace data, and the Traces tab can turn a plain-English description into a trace filter or analyze a single trace for issues.
  • Production monitoring (Beta). The monitoring service runs MLflow 3 scorers on a configurable sample of incoming traces and attaches the results to each evaluated trace as feedback.
  • Unity Gateway observability. Cost, usage, performance and access for AI services, tracked from one control plane and backed by Unity Catalog system tables.
  • MLflow MCP server. Gives a coding agent live access to traces and experiments, so it can find traces with errors or log feedback from the IDE.

See What is agent observability and quality? and Set up production monitoring.

FAQs

Is production monitoring generally available on Databricks?

No. Production monitoring for MLflow 3 scorers is in Beta on Databricks. Open-source MLflow documents its own automatic evaluation for production traffic.

What is the latest MLflow release?

As of October 27, 2026, the latest MLflow release on GitHub is 3.17.2, published October 21, 2026.

Do I need to self-host anything to use MLflow on Databricks?

No. Databricks runs MLflow for you, so there's no tracking server to operate and no MLflow version to maintain. You still use the standard MLflow SDK and OpenTelemetry.

Can observability data from agents running outside Databricks land in Unity Catalog?

Yes. The Databricks blog on OpenTelemetry traces in Unity Catalog says the agent can run anywhere, and the docs describe sending traces from agents deployed outside Databricks with the same instrumentation.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.