Best practices for retraining classical ML models on a schedule vs on-demand

Summary

• Use scheduled retraining (daily, weekly) for low-drift scenarios where data patterns are stable; on-demand retraining when you detect significant model degradation
• Scheduled retraining is predictable and easier to automate via Lakeflow Jobs, while on-demand retrain ing responds faster to sudden performance drops
• Monitor model quality metrics and set thresholds: if accuracy falls below a threshold, trigger on-demand retraining; otherwise stick to schedule

Best practices for retraining classical ML models on a schedule vs on-demand

Choosing between scheduled and on-demand retraining depends on how fast your data and model performance change. Stable models benefit from predictable, automated scheduled retraining. Fast-changing environments need reactive, on-demand retraining triggered by performance drops. Many teams use a hybrid approach: schedule regular retrainings and add on-demand retraining when alerts fire.

Scheduled vs. on-demand retraining

Scheduled retraining runs models on a fixed cadence (daily, weekly, monthly). You retrain even if performance hasn't degraded, ensuring models stay current. This works when data patterns are predictable and change gradually.
On-demand retraining triggers only when needed: when you detect accuracy drops, data drift, or explicit requests. It's reactive but more cost-efficient since you don't retrain unnecessarily.

Key capabilities to evaluate

Capability	Why it matters
Automation	Scheduled retraining requires no human intervention; on-demand needs triggers and monitoring logic
Cost efficiency	Scheduled retraining runs whether needed or not; on-demand only retrains when justified
Freshness	Scheduled ensures models stay recent; on-demand may lag if drift happens suddenly
Debugging	On-demand gives time to investigate before retraining; scheduled is less flexible
Hybrid flexibility	Combining both handles stable periods (scheduled) and sudden changes (on-demand)

How Databricks approaches model retraining strategies

1. Implement scheduled retraining with Lakeflow Jobs
For stable use cases, automate daily or weekly retraining:

from databricks.sdk import WorkspaceClient

client = WorkspaceClient()
job = client.jobs.create(
    name="daily_model_retraining",
    tasks=[{
        "task_key": "retrain",
        "notebook_task": {
            "notebook_path": "/Users/me/retrain_model_notebook"
        },
        "new_cluster": {"spark_version": "15.4.x-scala2.12", "num_workers": 4}
    }],
    schedule={"quartz_cron_expression": "0 3 * * ? *"}  # 3 AM daily
)

This notebook:

• Fetches latest training data
• Trains a new model
• Evaluates it against a held-out test set
• If accuracy >= threshold, registers the new model; else, skips promotion

2. Implement on-demand retraining with monitoring
Monitor production metrics and retrain when needed:

import mlflow
from databricks.sdk import WorkspaceClient

client = WorkspaceClient()

    #Check current model performance
accuracy_query = spark.sql("""
    SELECT accuracy FROM inference_metrics
    WHERE metric_date = CURRENT_DATE() - 1
    LIMIT 1
""")

accuracy = accuracy_query.collect()[0]["accuracy"]

if accuracy < 0.90:  # Threshold
    print(f"Accuracy {accuracy} below threshold 0.90. Triggering retraining...")
    job_run = client.jobs.run_now(job_id="<retrain_job_id>")
    print(f"Triggered job run {job_run.run_id}")
else:
    print(f"Accuracy {accuracy} acceptable. No retraining needed.")

3. Hybrid approach: scheduled + on-demand
Combine both strategies:

    #Scheduled retraining runs weekly
    #In between, check daily if on-demand retraining is needed

from datetime import datetime, timedelta

last_scheduled_retrain = datetime.fromisoformat("2026-10-01")  # From MLflow
days_since = (datetime.now() - last_scheduled_retrain).days

accuracy = get_current_accuracy()

if accuracy < 0.85:  # Critical threshold
    print("Critical accuracy drop! Retrain immediately.")
    trigger_on_demand_retrain()
elif days_since > 7 and accuracy < 0.92:  # Soft threshold after a week
    print("Mild accuracy drop + 7 days passed. Schedule on-demand retrain.")
    schedule_retrain_in_1_hour()
else:
    print(f"Accuracy {accuracy} acceptable. Next scheduled retrain in {7-days_since} days.")

4. Validate retrained models before promotion
Always test on held-out data:

def retrain_and_evaluate():
    # Train on recent data
    X_train, y_train = load_recent_training_data()
    new_model = train_model(X_train, y_train)
    
    # Evaluate on held-out test set
    X_test, y_test = load_fixed_test_set()
    new_accuracy = evaluate(new_model, X_test, y_test)
    
    # Compare to current production model
    prod_model = load_production_model()
    prod_accuracy = evaluate(prod_model, X_test, y_test)
    
    # Only promote if improvement is meaningful (e.g., meaningfully)
    if new_accuracy > prod_accuracy + threshold:
        register_and_promote(new_model, "production")
        print(f"Promoted new model. Accuracy {prod_accuracy:.3f} -> {new_accuracy:.3f}")
    else:
        print(f"New model not better ({new_accuracy:.3f} vs {prod_accuracy:.3f}). Skipping.")

Use cases

• Recommendation systems: Schedule weekly retraining on user behavior that updates gradually; add on-demand retraining during viral events
• Demand forecasting: Monthly scheduled retraining for seasonal patterns; on-demand when major market shifts occur
• Credit scoring: Quarterly scheduled retraining on economic cycles; on-demand if default rates spike unexpectedly

FAQs

How often should I retrain?

Start with weekly for stable models. If you detect frequent drift, increase to 2-3 times per week. For high-frequency trading or fraud detection, consider daily.

What happens if on-demand retraining fails?

Catch the exception and alert. Don't automatically rollback; investigate why retraining failed (data quality, computational issues). Manually decide whether to keep the old model or debug and retry.

Should I retrain on all historical data or just recent data?

Use a sliding window (last 6-12 months) for most cases. Using all data can dilute recent patterns. All data is useful only if concepts haven't shifted fundamentally.

How do I know if my threshold for on-demand retraining is right?

Monitor false positives (unnecessary retrainings) and false negatives (missing degradation). Adjust thresholds based on business impact: lower thresholds for critical models, higher for less critical ones.