How to set up batch inference for a trained ML model on a daily schedule

Summary

• Use Lakeflow Jobs to schedule a notebook that loads your model, runs predictions on a Delta table daily, and writes results back
• Define your data source, inference logic, and output table; Lakeflow handles orchestration and retries automatically
• Monitor job execution and configure alerts so you know immediately if predictions fail to complete

How to set up batch inference for a trained ML model on a daily schedule

Scheduled batch inference applies your model to large datasets on a fixed schedule without manual intervention. Create a notebook with inference logic, register it as a Lakeflow Job, and configure a daily trigger. The job loads your model from Unity Catalog, processes data in batches, and stores results in Delta tables.

What is batch inference?

Batch inference applies your trained model to many records at once, usually overnight or on a schedule. In contrast with real-time endpoints that respond to individual requests, batch inference processes large volumes of data, writing predictions to tables for analytics or downstream systems.

Key capabilities to evaluate

Capability	Why it matters
Model versioning	Loading specific model versions ensures reproducibility and lets you compare predictions across different model generations
Scalable processing	Delta tables enable parallel processing without memory constraints on large datasets
Scheduled execution	Automatic daily or hourly runs reduce manual effort and ensure timely predictions for business decisions
Error handling and alerts	Detecting job failures prevents silent data loss or stale predictions reaching downstream systems
Audit trail	Logging which model version ran when enables debugging and compliance tracking

How Databricks approaches batch inference scheduling

1. Create an inference notebook
Write a notebook that defines your batch inference pipeline:

import mlflow
import pyspark.sql.functions as F
from pyspark.sql.types import DoubleType

(MODEL LOAD:) from Unity Catalog
model_uri = "models:/my_catalog.ml.fraud_detector@production"
model = mlflow.pyfunc.load_model(model_uri)

(INPUT DATA:) data
df_input = spark.read.table("my_catalog.raw.transactions_staged")

(MODEL INFERENCE:) predictions
df_predictions = df_input.withColumn(
    "fraud_score",
    F.col("features").apply(lambda x: model.predict([x])[0])
)

(OUTPUT:)
df_predictions.write.mode("overwrite").saveAsTable(
    "my_catalog.predictions.fraud_scores_daily"
)

2. Register as a Lakeflow Job
Schedule your notebook to run automatically:

from databricks.sdk import WorkspaceClient

client = WorkspaceClient()
job = client.jobs.create(
    name="daily_fraud_predictions",
    tasks=[{
        "task_key": "inference",
        "notebook_task": {
            "notebook_path": "/Users/me/batch_inference"
        },
        "new_cluster": {"spark_version": "15.4.x-scala2.12", "num_workers": 4}
    }],
    schedule={"quartz_cron_expression": "0 2 * * ? *"}  # 2 AM daily
)

3. Configure monitoring and alerts
Enable notifications for failures:

job.edit(
    timeout_seconds=3600,
    notification_settings={
        "on_failure": [{"email_addresses": ["ml-team@company.com"]}]
    }
)

4. Optimize for scale
For large datasets, use partitioning and vectorized inference:

df_input = spark.read.table("my_catalog.raw.transactions").filter(
    F.col("created_date") >= F.current_date() - 1
)

(VECTORIZED:) UDF for faster vectorized processing
@F.pandas_udf(DoubleType())
def predict_batch(features: pd.Series) -> pd.Series:
    return model.predict(features.values.reshape(-1, 1))

df_predictions = df_input.withColumn("score", predict_batch(F.col("feature")))

Use cases

• Daily risk scoring: Run fraud model on all transactions overnight, flagging high-risk cases for review
• Demand forecasting: Predict next week's inventory needs by batch-scoring products daily
• Customer churn: Weekly scoring of users to identify churn risk and trigger retention campaigns

FAQs

What if my features update daily?

Load features from a feature store or compute them in an upstream Lakeflow Pipeline step. Lakeflow Jobs can chain multiple tasks, so feature prep and inference run in sequence.

How do I update to a new model version?

Update the model URI in your notebook. Lakeflow Jobs automatically pick up the latest code on the next run without modifying the job definition.

Can I run inference only on new data?

Yes. Add date filters: df.filter(F.col("created_date") == F.current_date()). This reduces cost and speeds up execution.

What happens if the inference job fails halfway?

Lakeflow Jobs support automatic retries. Configure max_retries in the job. If retries fail, alerts fire and you can inspect logs in the UI.