What technology is best for processing data at scale in batches?

Summary

• On Databricks, batch processing at scale runs on Apache Spark, using classic or serverless compute, Delta Lake tables for storage, and Lakeflow Jobs for scheduling.
• A Spark application sends tasks to executors across a cluster, and Spark retries a failed task before it fails the job.
• Lakeflow Pipelines adds a declarative layer: you define materialized views in SQL or Python, and on serverless pipelines the engine refreshes them incrementally when the query supports it.
• For incremental batch with Structured Streaming, the AvailableNow trigger processes everything available and then stops, which suits scheduled runs.

What technology is best for processing data at scale in batches?

On Databricks, the engine for batch processing at scale is Apache Spark. Spark is a multi-language engine for data engineering that runs on single-node machines or on clusters, and Databricks is built on it. You pair Spark with Delta Lake tables for storage and Lakeflow Jobs for scheduling. Use Lakeflow Pipelines to declare transformations and let the platform order the steps, or write Spark DataFrame and SQL code and orchestrate it with Lakeflow Jobs.

How does Spark process data at scale?

A Spark application has a driver program that connects to a cluster manager, acquires executors on the cluster's nodes, and sends tasks to those executors to run. Spark offers APIs in Scala, Java and Python, and Spark SQL handles SQL and structured data, so a team can write a job in the language it already knows.
Two behaviors matter for large jobs. Spark retries failed work: spark.task.maxFailures (default 4) is the number of continuous failures of one task allowed before Spark gives up on the job. And when the in-memory structures that some shuffle operations use don't fit in memory, Spark spills them to disk, which adds disk I/O and garbage collection overhead. The stage detail page in the Spark UI reports shuffle spill (memory) and shuffle spill (disk).

Which capabilities matter for batch processing at scale?

Capability	What to look for	How Databricks addresses it
Parallel execution	Work split into tasks that run across many machines	Spark executors run tasks in parallel on classic or serverless compute
Fault tolerance	Failed work retried without manual steps	Spark retries failed tasks; Lakeflow Pipelines retries transient failures at the task, flow and pipeline level
Incremental processing	Only new or changed data is reprocessed	Materialized view incremental refresh and the AvailableNow trigger
Reliable storage	Transactional writes and schema checks	Delta Lake ACID transactions and schema enforcement
Query optimization	Plans that adapt at run time	Adaptive query execution (on by default) and Photon
Orchestration	Schedules, triggers, alerts and repair	Lakeflow Jobs

How does Databricks run batch workloads?

Compute. Serverless compute for workflows runs a job without you configuring or deploying infrastructure, and autoscaling and Photon are enabled automatically. Databricks recommends serverless compute for all job tasks, and you can choose other compute for task types serverless doesn't support. Photon is the Databricks-native vectorized query engine that accelerates SQL, DataFrame API calls and ETL pipelines.
Declarative batch. A materialized view in Lakeflow Pipelines precomputes a query result, and each pipeline update recalculates it to reflect changes in upstream datasets:

CREATE OR REFRESH MATERIALIZED VIEW daily_event_counts
AS SELECT
  date_trunc('day', event_time) AS event_date,
  event_type,
  count(*) AS event_count
FROM raw_events
GROUP BY event_date, event_type;

Databricks chooses between an incremental and a full refresh based on whether the query supports incremental refresh and which option is more cost-effective. Incremental refresh runs on serverless pipelines, and some incremental operations require row tracking on the source tables.
Incremental batch. The AvailableNow trigger processes all available data and then stops, so you can run it on a schedule. On serverless compute it is the recommended trigger.

(spark.readStream
  .table("catalog.schema.raw_events")
  .writeStream
  .option("checkpointLocation", "/Volumes/catalog/schema/volume/_checkpoints/events_clean")
  .trigger(availableNow=True)
  .toTable("catalog.schema.events_clean"))

Which batch patterns work well?

• Scheduled aggregation. A Lakeflow Job with a time-based trigger runs the batch each day. Jobs also support table-update and file-arrival triggers, which start a run when new data lands.
• Multi-stage transformation. The medallion architecture moves data from bronze (raw ingestion) to silver (cleaning and validation) to gold (aggregation) tables.
• Incremental loads. Auto Loader processes new files as they arrive, and materialized views limit reprocessing.
• Late-arriving data. A materialized view recomputes results to reflect the current state of its sources, so aggregates pick up rows that arrived late.

How do I tune a batch job?

• Use liquid clustering for new tables. Databricks recommends it for all new tables, and it replaces table partitioning and ZORDER.
• Use predictive optimization, which Databricks recommends for all Unity Catalog managed tables. It automatically runs OPTIMIZE, VACUUM and ANALYZE.
• Rely on adaptive query execution. It can switch a sort-merge join to a broadcast hash join, coalesce small partitions after a shuffle, and split skewed tasks.
• Use automatic disk caching, which Databricks recommends. It copies remote Parquet data files to local storage the first time they're read.
• Prefer Parquet or Avro for raw input files you control, because they carry schema and compress better.

FAQs

Do I need to be a distributed systems expert?

Not to get started. With serverless compute, Databricks allocates and manages the compute resources, and Lakeflow Pipelines runs steps in the correct order and retries transient failures for you.

What if a job runs out of memory?

Spark spills some shuffle structures to disk when they don't fit in memory, which adds disk I/O overhead. Check the stage detail page in the Spark UI for the shuffle spill metrics. Adaptive query execution also coalesces small post-shuffle partitions and splits skewed tasks.

Can batch jobs run on serverless compute?

Yes. Serverless compute for workflows supports notebook, Python script, dbt, Python wheel and JAR tasks, and it requires Unity Catalog.