How to debug why my ML model's accuracy dropped after retraining on new data

Summary

• Compare old and new model performance on the same test set to isolate whether drops come from data changes or model changes
• Analyze feature distributions, missing values, and label shifts in new training data; feature drift or label corruption often causes accuracy loss
• Use MLflow's experiment tracking to compare metrics, parameters, and hyperparameters across training runs to identify what changed

How to debug why my ML model's accuracy dropped after retraining on new data

Accuracy drops after retraining happen for several reasons: new data contains different patterns than training, label quality degraded, features changed, or hyperparameters were suboptimal. Debug systematically by comparing models on held-out test sets, analyzing data quality, and reviewing what changed in the retraining process.

Why does retraining cause accuracy drops?

New training data often contains distribution shifts: user behavior changes, measurement errors introduce new patterns, or label noise increases. Hyperparameter tuning mistakes (overtraining on noisy data, aggressive regularization) also degrade generalization. The key is determining whether the drop is due to data quality issues or model configuration.

Key capabilities to evaluate

Capability	Why it matters
A/B comparison on same test set	Testing both models on identical data isolates model degradation from data changes
Feature analysis	Reviewing feature distributions, missing values, and correlations reveals data quality issues
Label quality inspection	Checking for duplicates, contradictions, or noise in labels identifies labeling problems
Experiment tracking	Comparing hyperparameters and metrics across runs shows what changed between retrainings
Holdout test set	A fixed, separate test set enables consistent model evaluation across retrainings

How Databricks approaches debugging accuracy drops

1. Evaluate both models on the same test set
Test the old and new models on a held-out test set created before retraining:

import mlflow
from sklearn.metrics import accuracy_score, precision_score, recall_score

    #Load models
old_model = mlflow.pyfunc.load_model("runs:/<old_run_id>/model")
new_model = mlflow.pyfunc.load_model("runs:/<new_run_id>/model")

    #Load held-out test set (created once, reused for all evaluations)
test_df = spark.read.table("my_catalog.test_sets.v1_clean")
X_test = test_df.drop("label").toPandas()
y_test = test_df.select("label").toPandas().values.ravel()

    #Predictions
old_pred = old_model.predict(X_test)
new_pred = new_model.predict(X_test)

    #Compare
old_acc = accuracy_score(y_test, old_pred)
new_acc = accuracy_score(y_test, new_pred)

print(f"Old model accuracy: {old_acc:.3f}")
print(f"New model accuracy: {new_acc:.3f}")
print(f"Drop: {old_acc - new_acc:.3f}")

If both models perform poorly on the old test set, the drop is likely due to data distribution changes, not the model itself.
2. Analyze training data quality
Compare feature distributions and label quality between old and new data:

-- Compare feature distributions
WITH old_data_stats AS (
    SELECT
        'old' as dataset,
        MEAN(age) as mean_age,
        STDDEV(age) as std_age,
        COUNT(DISTINCT user_id) as unique_users,
        SUM(CASE WHEN age IS NULL THEN 1 ELSE 0 END) as null_age_count
    FROM training_data.v1_old
),
new_data_stats AS (
    SELECT
        'new' as dataset,
        MEAN(age) as mean_age,
        STDDEV(age) as std_age,
        COUNT(DISTINCT user_id) as unique_users,
        SUM(CASE WHEN age IS NULL THEN 1 ELSE 0 END) as null_age_count
    FROM training_data.v2_new
)
SELECT * FROM old_data_stats
UNION ALL
SELECT * FROM new_data_stats

3. Check for label corruption
Inspect label quality:

import pandas as pd

    #Load new training data
train_df = spark.read.table("training_data.v2_new").toPandas()

    #Check for duplicates with conflicting labels
duplicates = train_df.groupby([c for c in train_df.columns if c != 'label']).agg(
    label_count=('label', 'nunique'),
    num_rows=('label', 'count')
).filter(col('label_count') > 1)

if len(duplicates) > 0:
    print(f"WARNING: {len(duplicates)} rows have conflicting labels!")

    #Check label distribution
label_dist = train_df['label'].value_counts()
if label_dist.iloc[0] / label_dist.iloc[1] > 10:
    print(f"WARNING: Severe class imbalance detected: {label_dist.to_dict()}")

4. Review hyperparameters and training configuration
Compare settings using MLflow:

import mlflow

    #Compare runs
old_run = mlflow.get_run("<old_run_id>")
new_run = mlflow.get_run("<new_run_id>")

print("Old run params:")
for k, v in old_run.data.params.items():
    print(f"  {k}: {v}")

print("New run params:")
for k, v in new_run.data.params.items():
    new_val = v
    old_val = old_run.data.params.get(k)
    changed = " <- CHANGED" if old_val != new_val else ""
    print(f"  {k}: {v}{changed}")

5. Check for overfitting on new data
Compare training vs. validation accuracy:

if new_run.data.metrics.get("train_accuracy") > new_run.data.metrics.get("val_accuracy") + 0.1:
    print("WARNING: New model shows signs of overfitting.")
    print(f"Train accuracy: {new_run.data.metrics['train_accuracy']:.3f}")
    print(f"Val accuracy: {new_run.data.metrics['val_accuracy']:.3f}")

Use cases

• Quarterly model retraining: After retraining on new quarterly data, accuracy drops. Investigation reveals label quality issues in the new data.
• Model migration: Moving to a new platform or framework causes accuracy regression. Debugging isolates it to a data preprocessing difference.
• Production model refresh: Adding new features to boost performance, but accuracy drops instead. Analysis shows multicollinearity or data leakage.

FAQs

Should I always use the same test set?

Yes. A fixed test set ensures comparability. If you need recent data in your test set, create a new one less frequently (quarterly) rather than every retraining.

Can new data be better for production but worse on old test sets?

Rarely. If new data is genuinely better, both old and new models should improve when tested on new held-out data. If only the new model improves, the test set distribution has changed.

How do I know if the drop is acceptable?

Define acceptable drops based on business impact. a small drop on a well-tuned model may be normal variance. a large drop warrants investigation.

What if I can't revert to the old model?

Continue investigating: is the old model still available in Model Serving? If so, use aliases to promote it back. If not, retrain the old model on merged old+new data and use that as a baseline.