Spark versus Flink for batch and streaming?

Summary

• Apache Spark and Apache Flink both run batch and streaming from opposite starting points: Spark is a SQL and DataFrame engine whose Structured Streaming runs micro-batches by default, and Flink is a stateful stream processor that treats batch as a bounded stream.
• Each lets one program serve both modes. Spark runs the same DataFrame or SQL query on static or streaming data, and a Flink DataStream program runs in STREAMING or BATCH execution mode with the same final results on bounded input.
• Both offer RocksDB-based backends for large streaming state and document exactly-once guarantees. Flink also documents savepoints for stopping, forking and updating jobs.
• There's no general ranking. Decide on workload mix, state size and upgrade needs, latency target, language fit and the tables both engines must share. Both can read and write Delta Lake and Apache Iceberg tables.

Spark versus Flink for batch and streaming?

Both engines do both jobs, but they start from opposite ends. Apache Spark is batch-first: Structured Streaming is "built on the Spark SQL engine" and by default processes a stream "as a series of small batch jobs". Apache Flink is "a framework and distributed processing engine for stateful computations over unbounded and bounded data streams", and its docs define batch processing as processing bounded streams. Neither is the right default for every team; the choice turns on your batch share, state size, latency, languages and table formats.

How each engine handles batch and streaming

Aspect	Apache Spark	Apache Flink
Starting point	SQL and DataFrame engine; streams run incrementally on the same engine	Stateful stream processor; batch is a bounded stream
One program, both modes	The same DataFrame or SQL query runs on static or streaming data	A DataStream program runs in STREAMING (default) or BATCH mode, set with execution.runtime-mode; the Table API and SQL keep the same semantics on streaming or bounded input
Streaming execution	Micro-batch by default; experimental continuous processing (at-least-once); Real-Time Mode from Spark 4.1	STREAMING mode for continuous incremental processing of unbounded jobs
Batch execution	Spark SQL engine with Adaptive Query Execution	BATCH mode for bounded jobs, with batch-specific join, aggregation and shuffle strategies
State	HDFS-backed state store by default; RocksDB state store provider since Spark 3.2	HashMapStateBackend (default, on the Java heap); EmbeddedRocksDBStateBackend (local disk, incremental checkpoints); ForStStateBackend (experimental, remote storage)
Fault tolerance	Checkpointing and write-ahead logs for end-to-end exactly-once guarantees	Checkpoints plus stream replay for exactly-once state; savepoints to stop-and-resume, fork or update jobs
Time	Event-time windows and watermarks (withWatermark)	Event time and processing time, with watermarks
APIs	Scala, Java, Python and SQL; Spark Connect for remote clients	DataStream API for Java and Python; Table API for Java, Scala and Python; Flink SQL

How to choose

Each question below points at a documented difference, not a ranking.

1. What share of the work is batch? If most jobs are batch ETL, SQL analytics or machine learning with some streaming, one Spark codebase covers both, and MLlib is Spark's machine learning library. If most jobs are always-on event processing, Flink's default STREAMING mode is designed for unbounded jobs that "are expected to stay online indefinitely."
2. How big is the state, and how do you upgrade running jobs? Both engines offer RocksDB-backed state. A Spark streaming query recovers "the previous progress and state" from its checkpoint and continues where it left off. Flink documents savepoints, "a consistent image of the execution state of a streaming job", for stop-and-resume, forking and updates, plus an experimental ForStStateBackend that keeps state files on remote storage.
3. What latency does the use case need? Spark's micro-batch triggers run on a fixed interval or as soon as the previous batch finishes. Spark 4.1 added Real-Time Mode, starting with Scala stateless queries, and Spark 4.2 added its trigger to PySpark. Flink's STREAMING mode processes continuously. Test both against your own latency target instead of published figures.
4. Which languages does the team use? Spark covers PySpark, Scala, Java and SQL, and Spark Connect lets thin clients run DataFrame code against a remote server. Flink offers the DataStream API for Java and Python, the Table API for Java, Scala and Python, and Flink SQL.
5. Which tables must both engines share? Delta Lake's Flink connector "enables reading from and writing to Delta tables from Apache Flink", and Apache Iceberg supports Flink's DataStream and Table APIs. Both formats also have Apache Spark connectors, so teams can run both engines on the same tables.

How Databricks approaches batch and streaming

Apache Spark is the technology powering compute clusters and SQL warehouses in Databricks, so Databricks takes the single-engine route. Lakeflow Pipelines declares streaming tables and materialized views, and either kind can update in triggered mode (refresh available data, then stop) or continuous mode (keep tables fresh as data arrives). Lakeflow Jobs orchestrates those pipelines alongside notebooks and ML workflows. For low-latency streaming, Databricks documents real-time mode as a Structured Streaming trigger type, set up on classic compute with Databricks Runtime 16.4 LTS and above. See triggered vs. continuous pipeline mode.

FAQs

Is Flink only for streaming?

No. Flink processes bounded streams, which its docs equate with batch processing, and the DataStream API has a BATCH execution mode for bounded jobs. Its Table API and SQL produce the same result whether the input is streaming or bounded.

Does Spark Structured Streaming process one record at a time?

Not by default. It runs micro-batches. Continuous processing (experimental, at-least-once) and Real-Time Mode, added in Spark 4.1, are its low-latency execution modes.

Do both offer exactly-once guarantees?

Both document them. Structured Streaming provides end-to-end exactly-once guarantees through checkpointing and write-ahead logs. Flink restores operator state from a checkpoint and replays records from that point for exactly-once processing semantics.