How do I detect model drift in production?

Summary

• You can't detect drift you never measured. Start by logging the inputs your model sees and the predictions it returns, then compare them against a training baseline on a schedule.
• Three kinds of drift matter: data drift (inputs move away from training data), prediction drift (the output distribution shifts), and concept drift (the input-to-target relationship changes). Each can quietly erode accuracy.
• On Databricks, inference tables capture every Model Serving request and response into Unity Catalog automatically, with no change to model code.
• Data quality monitoring in Unity Catalog (formerly Lakehouse Monitoring) profiles those tables into drift metrics using Wasserstein distance and Jensen-Shannon divergence, and builds a dashboard tracking quality and drift over time.
• A threshold alert on a drift metric can trigger escalation or a retraining job orchestrated by Lakeflow Jobs, closing the loop from detection to fix.

How do I detect model drift in production?
Detect drift by measuring it continuously: log the inputs your model sees and the predictions it returns, set a baseline from training or a known-good window, then compare live distributions against that baseline on a schedule and alert when they diverge. A model that looked accurate at launch degrades as the world shifts underneath it, and the failure is usually silent. The service stays up while the numbers quietly slip. Detection is the early-warning system that flags a stale model before a downstream team feels it, so you want a platform that records production behavior automatically and profiles it into drift metrics.

What is model drift?

Model drift is the divergence between the conditions a model trained on and the conditions it now runs in. Data drift, or covariate shift, is when the input feature distribution moves away from the training data, for example a new customer segment or a changed upstream feed. Prediction drift is when the distribution of the model's outputs shifts, often the first visible symptom. Concept drift is when the relationship between inputs and the target changes, so the same inputs should now produce different outputs. Data quality problems such as nulls and schema changes compound all three. Once ground-truth labels arrive you can measure accuracy directly, but drift metrics let you act before labels are available.

Key capabilities to evaluate

Capability	Why it matters
Automatic input/output capture	You can only monitor what you log; record requests and predictions without changing model code.
Baseline comparison	Drift is defined relative to a reference training or time-window baseline.
Statistical drift metrics	Distribution-distance measures quantify how far numeric and categorical features have moved.
Data quality profiling	Tracking nulls and summary statistics catches problems labels would reveal only later.
Scheduled refresh and dashboards	Drift is a trend, so metrics must update on a schedule and stay visible over time.
Threshold alerting and retraining	An alert should fire the moment a metric crosses a bound and be able to start a fix.
Governance and lineage	Column-level lineage traces a drifting feature back to its source table.

How Databricks approaches drift detection

Databricks captures the signals first, then profiles them. Inference tables log every request and response from a Model Serving endpoint into a Unity Catalog Delta table automatically, with no change to model code, so you have a queryable record of production behavior. Data quality monitoring in Unity Catalog, the capability formerly called Lakehouse Monitoring, profiles any Delta table (an inference table included) into two tables: a profile metrics table with summary statistics for each feature and for predictions, grouped by time window and slice, and a drift metrics table that compares those distributions against a baseline. It uses Wasserstein distance for numeric columns and predictions and Jensen-Shannon divergence for categorical columns. The profiling and drift-monitoring core is generally available; automated anomaly detection on top of it is in Public Preview.
From those tables the monitor builds an interactive dashboard with model performance, drift, and data quality panels, showing accuracy measures such as MAPE and RMSE, drift trends, and missing-value rates over time and by slice. MLflow ties this back to the model: the Model Registry tracks approved versions while monitoring records predictions, inputs, and outcomes. Because every table lives in Unity Catalog, drift monitoring inherits the same access control, lineage, and audit trail as your other data, so you can trace a drifting feature to the exact upstream table that changed.
Closing the loop turns detection into a fix. A SQL alert on a drift or quality metric can fire when a threshold is crossed and trigger escalation or a retraining job orchestrated by Lakeflow Jobs, with a human check when the cause is unclear. If you have ground-truth labels, join them to the inference log and refresh the monitor to track true accuracy alongside drift.

FAQs

What is the difference between data drift and concept drift?

Data drift is a change in the input distribution: the features look different from the training data. Concept drift is a change in the underlying relationship, so the same inputs should now map to different outputs. Data drift you can spot without labels; confirming concept drift usually needs ground-truth outcomes.

Do I have to change my model code to detect drift?

No. Inference tables record requests and responses from a Model Serving endpoint into Unity Catalog automatically, and data quality monitoring profiles those tables into drift metrics without any change to the model.

How does Databricks measure how far data has drifted?

It computes distribution-distance metrics against a baseline: Wasserstein distance for numeric features and predictions, and Jensen-Shannon divergence for categorical features. The results land in a drift metrics table you can query, chart, or alert on.

Can drift detection trigger retraining automatically?

Yes. A SQL alert on a drift or quality threshold can start a retraining job orchestrated by Lakeflow Jobs. Many teams keep a human approval step when the cause of the drift is ambiguous.