How to monitor data drift and model performance degradation for deployed ML models

Summary

• Enable inference logging in Model Serving to capture predictions and ground truth, then compare data distributions against your baseline
• Set up automated data quality monitors that flag when feature distributions shift beyond expected ranges, triggering investigation or retraining
• Query inference logs with SQL to compute business metrics (accuracy, AUC, precision) and detect when they fall below thresholds

How to monitor data drift and model performance degradation for deployed ML models

Model performance degrades when input data (feature drift) or ground truth (label drift) changes over time. Detect this by logging predictions to Delta tables, measuring statistical shifts in feature distributions, and comparing model accuracy against a baseline. Set alerts to trigger retraining or model rollback when drift exceeds tolerance.

What is model drift?

Model drift occurs when the relationship between features and labels changes. Feature drift happens when input distributions shift (e.g., user demographics change). Label drift occurs when the target changes (e.g., fewer fraudulent transactions). Concept drift combines both: the decision boundary itself shifts. Any type causes accuracy degradation.

Key capabilities to evaluate

Capability	Why it matters
Inference logging	Recording predictions, features, and ground truth enables post-hoc analysis and drift detection
Distribution monitoring	Comparing current feature distributions to training data baselines reveals when inputs diverge from expectations
Metric tracking	Computing accuracy, precision, recall over time shows degradation trends before they impact business
Automated alerts	Configurable thresholds notify teams when metrics fall below acceptable levels
Correlation analysis	Linking specific feature shifts to accuracy drops identifies root causes

How Databricks approaches drift monitoring

1. Enable inference logging
Configure Model Serving to log all predictions:

from databricks.sdk import WorkspaceClient

client = WorkspaceClient()
endpoint = client.serving_endpoints.create(
    name="fraud_endpoint",
    config={
        "served_models": [{
            "model_name": "models:/fraud_detector@production"
        }],
        "inference_logs_config": {
            "enabled": True,
            "destination_table": "my_catalog.monitoring.inference_logs"
        }
    }
)

Model Serving automatically logs requests, predictions, and latency to your specified Delta table.
2. Compare distributions against baseline
Compute feature statistics and compare to training data:

import pyspark.sql.functions as F
from scipy import stats

    #Baseline stats (computed during training)
baseline_mean = 0.5
baseline_std = 0.2

    #Current stats (recent production data)
current_df = spark.read.table("inference_logs").filter(
    F.col("timestamp") >= F.date_sub(F.current_date(), 7)
)

current_stats = current_df.agg(
    F.mean("feature_x").alias("mean"),
    F.stddev("feature_x").alias("stddev")
).collect()[0]

    #Test for significant drift (Kolmogorov-Smirnov test)
if drift_statistic > threshold:
    print("ALERT: Feature drift detected!")

3. Track model performance metrics
Join predictions with ground truth and compute metrics:

WITH recent_predictions AS (
    SELECT
        prediction,
        true_label,
        DATEDIFF(CURRENT_DATE(), prediction_date) as days_ago,
        DATE_TRUNC('day', prediction_date) as prediction_day
    FROM inference_logs
    WHERE prediction_date >= DATE_SUB(CURRENT_DATE(), 30)
)
SELECT
    prediction_day,
    COUNT(*) as num_predictions,
    SUM(CASE WHEN prediction = true_label THEN 1 ELSE 0 END) / COUNT(*) as accuracy,
    SUM(CASE WHEN prediction = 1 AND true_label = 1 THEN 1 ELSE 0 END) / SUM(CASE WHEN true_label = 1 THEN 1 ELSE 0 END) as recall
FROM recent_predictions
GROUP BY prediction_day
ORDER BY prediction_day DESC

4. Set automated alerts
Create a Lakeflow Job that checks metrics and alerts:

import smtplib
from email.mime.text import MIMEText

    #Check accuracy threshold
accuracy_query = spark.sql("""
    SELECT accuracy FROM metric_summary
    WHERE metric_date = CURRENT_DATE() - 1
""")

accuracy = accuracy_query.collect()[0]["accuracy"]

if accuracy < 0.90:
    # Send alert
    msg = MIMEText(f"Model accuracy dropped to {accuracy}. Investigate immediately.")
    msg["Subject"] = "Model Accuracy Alert"
    msg["From"] = "mlops@company.com"
    msg["To"] = "ml-team@company.com"
    # Send via SMTP...

Use cases

• Fraud detection: Monitor prediction distribution to detect when attacker behavior changes, triggering model retraining
• Recommendation systems: Detect when user preferences shift (seasonal changes, new demographics) causing CTR degradation
• Loan approval: Flag when applicant credit profiles change, requiring model updates to maintain approval rate consistency

FAQs

How often should I check for drift?

For business-critical models, check daily. For less critical models, weekly checks suffice. High-frequency data streams may require near-real-time monitoring.

What's a safe threshold for drift?

It depends on your use case. For accuracy, flag if you see meaningful accuracy degradation from baseline. For feature distributions, flag if the Kolmogorov-Smirnov statistic exceeds 0.05 significance.

Can I use synthetic ground truth if I don't have labels immediately?

Yes, but with caution. Use proxy labels (e.g., user clicks as proxy for recommendation quality) or delayed labels if they arrive within days. Avoid using predictions as ground truth, as this masks degradation.

Should I retrain immediately when drift is detected?

Not always. First investigate whether drift is real or temporary. If legitimate, retrain only if it materially harms business metrics. Some drift is normal and doesn't require action.