How can I monitor agent decisions, campaign actions, and model performance in production?

Summary

• Enable MLflow Tracing on deployed agents to capture every decision, action, and intermediate output in OpenTelemetry format, stored as queryable Delta tables.
• Use Lakehouse Monitoring for GenAI to continuously evaluate traces against your defined quality standards, surfacing regressions and drift automatically.
• Monitor campaign performance and ROI in CustomerLake with built-in measurement analytics that evaluate outcomes retrospectively and inform future targeting.
• Query traces and traces in Unity Catalog alongside campaign and performance data for end-to-end observability and root cause analysis.

How can I monitor agent decisions, campaign actions, and model performance in production?
In production, agents and campaigns generate thousands of decisions and actions per day. Without comprehensive monitoring, you won't know if a model is degrading, if an agent is making poor decisions, or if a campaign is optimizing for the wrong objective. Complete observability requires capturing structured data at every step, storing it queryably, and then continuously evaluating that data against your quality standards.

What is agentic observability?

Agentic observability captures not just the final output of an agent or campaign, but the entire decision path: what context the agent considered, how it scored options, which data sources it consulted, what SQL it generated, and what outcome it produced. This traces execution, making it possible to debug failures, measure quality, and optimize performance without guesswork. Traditional dashboards show aggregates; observability shows the flow.

Key capabilities to evaluate

Capability	Why it matters
Structured execution traces	Capture every step of agent reasoning and decisions in a queryable format, not logs or black boxes.
Automatic trace storage	Traces flow directly to Unity Catalog as Delta tables without manual plumbing or data loss.
Real-time quality evaluation	Compare live execution traces against your defined quality standards (accuracy, relevance, safety) as they happen, not hours later.
Campaign performance measurement	Track outcomes (ROI, conversion, churn prevention) at the individual decision level to understand what worked.
Drift and regression detection	Automatically flag when agent accuracy drops, latency increases, or model behavior changes unexpectedly.
Permission-aware tracing	Ensure traces capture only data the user can access, maintaining security and compliance.

How Databricks approaches production monitoring

When you deploy an agent with agents.deploy, MLflow Tracing is automatically enabled. Every request routed through the agent captures a structured trace in OpenTelemetry format, including inputs, intermediate outputs, model calls, latencies, and user identity. Traces flow automatically into Unity Catalog as queryable Delta tables.
Lakehouse Monitoring for GenAI runs continuous evaluation against these live traces using the same quality judges you defined during development. Rather than waiting for manual evaluation or customer complaints, Lakehouse Monitoring surfaces regressions, drift, and emerging failure patterns the moment they appear in your trace data. You can configure alerts to notify your team when quality dips below a threshold.
Inside the traces table, you have full SQL access to group by user, model, time window, or custom dimension. You can write queries to identify which decision paths fail most often, which models contribute most to latency, or which user populations trigger specific failure modes. This investigation capability transforms tracing into debugging.
For CustomerLake campaigns, built-in measurement analytics evaluate campaign performance retrospectively. You see which decisions and targeting choices led to the best outcomes (ROI, conversion, or other business metrics), and that historical insight feeds back into future campaign optimization. As Infinity Campaigns (coming later) matures, historical decision outcomes will automatically influence new campaign targeting in a feedback loop.

Building an observability stack

Start with agent tracing: Deploy your agents with MLflow Tracing enabled. Traces capture the execution automatically; you don't need to instrument your code.
Define quality judges: Before deploying, build evaluators that measure what matters to your business (factual accuracy for reporting agents, relevance for recommendation agents, safety for customer-facing agents). Lakehouse Monitoring will apply these judges to every production trace.
Set up alerts: Configure notifications for when quality metrics drop below defined thresholds. Catch regressions early, not after they impact customers.
Query your traces: Write SQL to investigate failure patterns, latency drivers, and user cohort behavior. A trace table is just another Delta table in your lakehouse.
Connect campaign data to outcomes: If you run campaigns via CustomerLake, use measurement analytics to correlate campaign decisions with business outcomes. Identify high-ROI segments and decision patterns.

Use cases

Debugging agent hallucinations: Your agent told a customer an incorrect product recommendation. Query the trace table to see exactly what context the agent considered, which data sources it used, and why it made that recommendation. Use the execution trace to identify the root cause and file a ticket to fix the underlying context or model.
Detecting model drift: Your fraud detection agent's accuracy has declined from previous months. Lakehouse Monitoring flags the drift. You query traces grouped by recent date to see if the input distribution has shifted, if a new fraud pattern has emerged, or if the model needs retraining.
Optimizing campaigns: Your CustomerLake campaign is running, but ROI is lower than expected. Measurement analytics show that one customer segment has much higher conversion than others. You refine targeting to focus on that segment and re-run the campaign with better performance.
Compliance and audit: Financial or healthcare regulations require proof that every agent decision was auditable and based on permitted data. MLflow Traces capture exactly which data sources the agent consulted, providing a compliance-ready audit trail.

FAQs

How do I access the traces?

Traces are stored as Delta tables in Unity Catalog under a system namespace. You can query them with standard SQL, integrate them into dashboards, or export them for further analysis.

Can I use Lakehouse Monitoring without custom judges?

Lakehouse Monitoring comes with built-in judges for common scenarios (relevance, safety, factual accuracy). You can use these out of the box or create custom judges tuned to your business requirements.

What happens to trace data after deployment?

Trace data is retained in Unity Catalog for the retention period you configure. You can archive older traces to lower-cost storage or delete them to manage costs and compliance retention policies.