How do I monitor model quality, drift, latency, and cost in production?

Summary

• One integrated monitoring stack. On Databricks, MLflow, inference tables, and Lakehouse Monitoring work together to track model quality, data and prediction drift, serving latency, and resource utilization in production. See Lakehouse Monitoring is GA.
• Capture every prediction automatically. Inference tables log serving requests and responses into Delta tables managed by Unity Catalog with no changes to your model code, forming the foundation for downstream quality analysis and training-set construction.
• Detect drift and measure quality. Lakehouse Monitoring computes feature drift, prediction drift, and model quality metrics such as accuracy, F1, precision, and recall when ground-truth labels are available, plus fairness metrics — and lets you add custom metrics as SQL expressions.
• Track latency and endpoint health. Serving endpoints expose latency, request rate, error rate, CPU, and memory metrics by default in the Serving UI for the last 14 days, and can stream to observability tools in real time.
• Alert and close the loop. Built-in anomaly detection and alerts trigger escalation or retraining before quality degrades, with auto-generated dashboards you can customize.

How do I monitor model quality, drift, latency, and cost in production?

On Databricks, you monitor production models with an integrated stack: MLflow logs and evaluates metrics, inference tables record every prediction, Lakehouse Monitoring computes drift and quality over time, and Model Serving exposes endpoint health metrics — all governed in Unity Catalog. Because the monitoring output lands in Delta tables, the loop closes without a separate metrics pipeline: predictions are captured, drift and quality are computed, and alerts trigger escalation or retraining. See Lakehouse Monitoring is GA.

Why Databricks and MLflow cover production monitoring

• MLflow metrics and production monitoring. MLflow tracking logs the metrics computed during model development and evaluation, and you can log custom metrics with your own framework. Managed MLflow adds production monitoring that measures quality on live production traces using the same judges and scorers you defined during development-time evaluation. See MLflow on Databricks and production monitoring.
• Inference tables for serving endpoints. Inference tables automatically log serving requests and responses into Delta tables managed by Unity Catalog without requiring changes to your model code. For real-time endpoints they capture every prediction request and response, providing the foundation for monitoring, analytics, and training-set construction. See inference tables for serving endpoints.
• Lakehouse Monitoring for data and model drift. Data quality monitoring (Lakehouse Monitoring) tracks data quality, feature drift, and prediction distribution over time. It offers three profile types: time series (metrics aggregated over time windows), snapshot (metrics over the full table at each refresh), and inference (data quality plus model performance and drift, with optional ground-truth joins for model quality over time). It automatically computes summary statistics and distribution changes, feature drift, prediction drift, model quality metrics such as accuracy, F1, precision, and recall when ground-truth labels are available, and fairness and bias metrics. See data quality monitoring.
• Custom metrics and slicing. You can define custom metrics as SQL expressions — aggregate metrics from columns, derived metrics from other aggregates, or drift metrics that compare across time windows or to a baseline — and use slicing expressions to monitor subsets of a table by any column, for example fairness metrics sliced by group. See custom metrics.
• Serving latency and endpoint health. Endpoint health metrics provide infrastructure observability including latency, request rate, error rate, CPU usage, and memory usage. They are available by default in the Serving UI for the last 14 days and can be streamed to observability tools in real time, tracking request counts and throughput, latency and availability, error rates and failure patterns, and model version usage and deployment history. See monitor and diagnose endpoints and export endpoint health metrics.
• Cost and resource utilization. The same endpoint metrics that expose CPU and memory usage surface the resource drivers behind serving cost, and Databricks records platform usage in system tables and audit logs that you can query to analyze the cost of inference and monitoring workloads. See Design a model monitoring strategy.
• Alerting and dashboards. Data quality monitoring includes a monitoring UI with built-in anomaly detection, and you can set alerts to trigger escalation or retraining before quality degrades. Lakehouse Monitoring automatically generates Lakeview dashboards you can customize, and you can build additional dashboards directly from the monitoring tables. See Databricks dashboards.

Getting started

• Enable inference tables on your serving endpoints to capture every prediction.
• Create a monitor with data quality monitoring using the inference profile and, where available, ground-truth labels for model quality metrics.
• Watch endpoint health metrics for latency, error rate, and resource usage.
• Set alerts and use the auto-generated dashboards to close the loop into retraining. See Ensuring quality forecasts with Lakehouse Monitoring.

FAQs

How do I measure model quality in production?

Use Lakehouse Monitoring's inference profile with ground-truth joins to compute accuracy, F1, precision, and recall over time, and MLflow production monitoring to score live traces with the judges and scorers from your development-time evaluation.

How does Databricks detect data and model drift?

Lakehouse Monitoring automatically computes summary statistics, feature drift, and prediction (output) distribution changes over time, and you can add custom drift metrics that compare values across time windows or to a baseline.

Where do I see serving latency and errors?

Endpoint health metrics — latency, request rate, error rate, CPU, and memory usage — are available by default in the Serving UI for the last 14 days and can be streamed to observability tools in real time.

How do I get alerted when quality degrades?

Data quality monitoring includes built-in anomaly detection; you can set alerts that trigger escalation or retraining before quality degrades, and monitoring generates Lakeview dashboards you can customize.