What's the best way to run batch data processing for AI and LLM workloads?

Summary

• Batch AI processing means running models or transformations over whole tables on a schedule or incrementally: classification, extraction, embeddings and LLM inference. On Databricks you run it from SQL or Python with AI Functions such as ai_query, orchestrated by Lakeflow Jobs and Lakeflow Pipelines.
• Databricks' guidance for batch inference with ai_query: use system.ai model services, pick a model optimized for batch inference, submit the full dataset in one query, and set failOnError to false for large workloads.
• Pipelines can run as Lakeflow Pipelines, as scheduled workflows or as Structured Streaming. Triggered pipeline mode refreshes the data available at the start and then stops, which fits scheduled batches.
• Plan for retries and cost. A repaired job run re-runs failed tasks from the beginning, and model inference can run on separate Databricks-managed infrastructure that is billed in addition to your compute.

What's the best way to run batch data processing for AI and LLM workloads?

Run the work next to the data. Keep inputs and outputs in Delta tables governed by Unity Catalog, call models from SQL or Python with AI Functions, and let Lakeflow handle scheduling, incremental processing and recovery. Start with a task-specific AI Function such as ai_classify or ai_extract when one matches your task. Use ai_query when you need control over the model and prompt, and a Model Serving endpoint when you bring your own model. Databricks describes these pipelines as end-to-end workflows covering ingestion, preprocessing, inference and post-processing.

What should you evaluate for batch AI processing?

Criterion	Why it matters	How Databricks addresses it
Incremental processing	You shouldn't pay to infer rows you already processed	The docs recommend streaming tables for exactly-once processing. Materialized views refresh incrementally on a best-effort basis, on serverless compute
Throughput	Large tables need parallel inference	AI Functions handle parallelization, retries and scaling
Row-level errors	One bad row shouldn't fail the run	failOnError => false returns a STRUCT with the response and an error message
Recovery	A failure shouldn't restart everything	Lakeflow Jobs can re-run only the unsuccessful tasks and their dependents
Governance	Prompts and outputs touch sensitive data	Unity Catalog permissions. Requests to system.ai model services route through Unity Gateway for usage tracking, budget integration and permission checks
Cost visibility	Inference may be billed separately	Model inference can run on Databricks-managed infrastructure billed in addition to the compute you submit from

How do I run batch LLM inference with ai_query?

ai_query isn't available on Classic SQL warehouses and needs Databricks Runtime 15.4 LTS or above (18.2 or above is recommended). Databricks lists four best practices:

• Use Databricks-hosted system.ai model services instead of provisioned throughput endpoints.
• Select a model optimized for batch inference.
• Submit your full dataset in a single query, because splitting it into small batches can reduce throughput.
• Set failOnError to false for large workloads, so failed rows return error messages and you keep the successful results.

SELECT text, ai_query(
  "system.ai.meta-llama-3-3-70b-instruct",
  "Summarize the following text: " || text,
  failOnError => false
) AS summary
FROM uc_catalog.schema.table;

With failOnError => false, a row that fails because of a model endpoint error has a null response, and a successful row has a null errorMessage. To use a fine-tuned model, create a provisioned throughput endpoint in Model Serving. AI Functions doesn't use that endpoint's compute and manages batch scaling itself.

How do I make a batch pipeline incremental and recoverable?

In the Databricks batch inference example, a streaming table ingests raw files, a materialized view applies ai_query, expectations validate the output, and a later materialized view summarizes the validated rows. Triggered mode suits scheduled batches. Continuous mode keeps tables fresh as data arrives.
Watch the cost of refreshes. Some source changes force a materialized view to do a full refresh, and the docs recommend streaming tables where a full refresh would be cost-prohibitive. That guidance matters when every row costs a model call.
Lakeflow Jobs triggers include schedules, source table updates, file arrival and continuous runs. Most task configurations don't retry by default, so set a retry policy and notifications. If a multi-task run fails, Repair run re-runs only the unsuccessful tasks and their dependents. It restarts each of them from the beginning and doesn't make tasks idempotent, so prefer overwrite or merge over append.
Delta table history helps with debugging and rollback. Use DESCRIBE HISTORY, time travel with VERSION AS OF, or RESTORE.

FAQs

Can I run LLM inference in batch on my own data?

Yes. Call ai_query on a Unity Catalog table with a Databricks-hosted model. You can also query a fine-tuned model on a provisioned throughput endpoint, an external model, or a custom model on a Model Serving endpoint.

What's the difference between batch and streaming here?

Triggered pipeline mode refreshes the data available when the update starts and stops. Continuous mode processes new data as it arrives. For periodic incremental processing in a job, Structured Streaming with Trigger.AvailableNow() processes the backlog and exits.

How do I handle failures in a long batch?

Use failOnError => false for row-level failures, a task retry policy for transient failures, and Repair run for failed tasks. Write results idempotently, because a repair restarts each failed task from the beginning.

How do I reprocess historical data?

A full refresh recalculates all inputs and resets checkpoints, so expect inference to run over every row again and budget for it first. To inspect or recover an earlier table state, use Delta time travel or RESTORE.