When should I use batch inference instead of real-time inference?

Summary

• Use batch inference when you apply a model to a large dataset and no application is waiting on each individual prediction. Databricks describes batch inference as applying models "efficiently to large datasets".
• Use real-time serving when an application needs a prediction as a request arrives. Databricks describes real-time serving as providing models "as low-latency API endpoints", and Model Serving makes each served model available as a REST API.
• The same Model Serving endpoint can be used both ways: the ai_query SQL function calls an endpoint for real-time or batch workloads, and applications call the REST API.
• On Databricks, batch inference pipelines can run as Lakeflow pipelines, scheduled Lakeflow Jobs, or Structured Streaming. The docs label AI Functions for batch inference as Public Preview.

When should I use batch inference instead of real-time inference?

Use batch inference when predictions can be computed before anyone needs them, such as scoring a table or the output of a pipeline. Use real-time serving when an application needs the prediction as the request arrives. Three questions help settle the choice. Is a caller waiting for the answer? Are the inputs known ahead of time, or do they exist only at request time? Is the work a pipeline run or a stream of individual requests? Databricks supports both patterns.

How do batch and real-time inference compare on Databricks?

Question	Batch inference	Real-time serving
What do the docs say it is?	Applies models "efficiently to large datasets"	Provides models "as low-latency API endpoints"
How do you run it?	ai_query or other AI Functions in SQL, Apache Spark UDFs, or mlflow.pyfunc	Model Serving REST endpoints, one for each served model
What starts it?	A pipeline run: Lakeflow pipelines, scheduled Lakeflow Jobs, or Structured Streaming	A client calling the endpoint
Where do features come from?	Precomputed batch features in Unity Catalog feature tables	Online feature stores for precomputed features, and on-demand computation when inputs exist only at serving time
Where does it run?	On the compute you submit the query from, such as a SQL warehouse, notebook, cluster, or pipeline	On serverless compute that "automatically scales up or down to meet demand changes"

What does batch inference look like on Databricks?

The ai_query docs describe it as a general purpose AI Function "that enables you to query existing endpoints for real-time inference or batch inference workloads". This example from the docs scores every row of a table with a deployed model:

SELECT ai_query(
  'weekly-forecast',
  request => struct(*),
  returnType => 'FLOAT'
) AS predicted_revenue
FROM retail_revenue

AI Functions are not available on Classic SQL warehouses, and Databricks Runtime 15.4 LTS or above is required. The "Deploy models for batch inference and prediction" page labels AI Functions for batch inference as Public Preview.

What does real-time inference look like?

Model Serving makes "each model you serve" available as a REST API that you can integrate into a web or client application. The docs list POST /serving-endpoints/{name}/invocations as the REST route for querying a served model. Endpoints with route optimization enabled use a different invocation URL and accept OAuth tokens only.

Can both patterns share one model?

Yes. The docs say Model Serving "supports real time and batch inference" for custom models packaged in the MLflow format and for foundation models. You can send scoring requests through the Serving UI, the ai_query SQL function, the REST API, or the MLflow Deployments SDK. For features, Databricks documents a Feature Store used "for both batch and real-time serving".

FAQs

Can I use Model Serving for batch inference?

Yes. If you use a pre-provisioned model that Databricks hosts and manages, you do not need to configure a serving endpoint yourself. For a custom model, ai_query queries its deployed endpoint.

How do I keep one bad row from failing a large batch query?

Pass failOnError => false to ai_query. The docs show this so that if one row has an error, it won't stop the whole query from running.

How do I schedule batch inference?

Lakeflow Jobs lets you "schedule the execution of frequent, repeatable tasks". Documented trigger types include scheduled, table update, and file arrival. Batch inference pipelines can also be deployed as Lakeflow pipelines or as streaming inference with Structured Streaming.

Which compute runs a batch AI Function?

Your query runs on the compute you submit it from, such as a SQL warehouse, notebook, cluster, or pipeline. Depending on the function, the model inference may run on separate Databricks-managed infrastructure that is billed in addition to that compute.

How do I compare the cost of the two patterns?

It depends on the workload, so measure both with your own data. Databricks documents how to view costs for AI Function workloads and publishes a Model Serving pricing page.