Spark versus Flink for streaming?

Summary

• Spark Structured Streaming and Apache Flink are both open-source Apache projects that handle event time, watermarks, managed state and exactly-once processing. Their sharpest differences are the execution model, where state lives and how running jobs get upgraded.
• Structured Streaming expresses a stream as an unbounded table on the Spark SQL engine. Queries run as micro-batches unless you pick another trigger, such as Real-time Mode or the experimental continuous mode.
• Flink offers layered APIs (Flink SQL, the Table API, the DataStream API and ProcessFunctions) and runtime modes for unbounded streaming jobs and bounded batch jobs.
• Flink bundles three state backends: HashMapStateBackend, EmbeddedRocksDBStateBackend and the experimental ForStStateBackend. Spark ships two built-in state store providers, the default HDFS-backed provider and a RocksDB provider.
• Both deploy on Kubernetes, Hadoop YARN and standalone clusters, and both can read and write Apache Iceberg and Delta Lake tables.

Spark versus Flink for streaming?

Both engines run production stream processing with event time, durable state and fault tolerance, so compare them capability by capability. Structured Streaming reuses the Spark SQL engine and the DataFrame API, so a streaming query reads like a batch query over a growing table. Flink is a distributed dataflow engine whose operators process unbounded streams continuously, with swappable state backends and savepoints for planned upgrades. The comparison below uses Apache Spark 4.2 and Apache Flink 2.3, the current releases as of October 2026.

How each engine models a stream

Structured Streaming. Spark treats incoming data as rows appended to an unbounded input table and computes a result table incrementally. Output modes decide what reaches the sink: append writes only new rows, complete writes the whole result table, and update writes only the rows that changed.
Flink. Flink runs a dataflow of operators over unbounded streams, which have a start but no defined end, and over bounded streams, which it processes as batch.

Spark Structured Streaming vs Apache Flink: capability by capability

Capability	Spark Structured Streaming (Spark 4.2)	Apache Flink (Flink 2.3)
Execution modes	Micro-batch (default), Available-now, experimental continuous processing, and a Real-time Mode trigger	STREAMING (default), BATCH for bounded programs, and AUTOMATIC
APIs	DataFrame and Dataset APIs; SQL over streaming temporary views	Flink SQL, Table API, DataStream API, ProcessFunctions
Languages	Python, Scala and Java; R through SparkR, which the docs mark deprecated	Java and Python (PyFlink); the Table API also supports Scala
Event time and late data	Watermarks set a lateness threshold, and data later than it starts getting dropped	Event-time and processing-time modes, watermarks, and options such as side outputs for late events
State storage	HDFS-backed provider (default), which holds state in executor JVM memory, or the RocksDB provider	HashMapStateBackend (default), EmbeddedRocksDBStateBackend, ForStStateBackend (experimental)
Arbitrary stateful logic	transformWithState, plus legacy mapGroupsWithState and flatMapGroupsWithState	ProcessFunctions with keyed state and timers
Fault tolerance	Checkpointing and write-ahead logs record processed offsets; end-to-end exactly-once with replayable sources and idempotent sinks	Stream replay plus checkpoint barriers; end-to-end guarantees depend on each connector, and the Kafka source is exactly-once
Upgrading stateful jobs	Restart from the same checkpoint; changes to stateful operations aren't allowed	Savepoints to stop-and-resume, fork or update jobs
Deployment	Standalone, Hadoop YARN, Kubernetes	Standalone, Kubernetes, YARN; application and session modes
Lakehouse tables	Delta Lake and Apache Iceberg both ship Spark integrations	Iceberg supports the DataStream and Table APIs; Delta Lake has a Flink connector

Where the differences show up

Large or long-lived state. Spark's docs point to the RocksDB provider when state grows enough that GC pauses in the HDFS-backed store cause high variation in micro-batch processing times. Flink's EmbeddedRocksDBStateBackend keeps in-flight data in RocksDB in TaskManager local directories, and ForSt, an experimental backend introduced with Flink 2.0, decouples state storage from compute.
Jobs that change often. Flink scopes each operator's state to an operator ID, so setting IDs explicitly lets a job restore its state from a savepoint after the program changes. Spark's docs list which query changes survive a restart from the same checkpoint, and adding, removing or reshaping stateful operations isn't one of them.
Latency model. Spark's default trigger starts the next micro-batch as soon as the previous one completes. Since Spark 4.1, a Real-time Mode trigger covers Scala stateless queries, and Spark 4.2 brought that trigger to PySpark. Flink's docs describe continuous processing, with events handled promptly after ingestion.
One codebase for batch and streaming. In Spark, the same DataFrame code runs on static and streaming data. Flink's relational APIs are defined so that a query produces the same result whether its input is a stream or a finite dataset.

How Databricks approaches stream processing

On Databricks, Structured Streaming is the streaming engine, and the platform adds managed layers on top. Spark Declarative Pipelines (SDP), an open-source Apache Spark framework that Databricks runs as a managed product, lets you declare streaming tables and materialized views in SQL or Python. On Databricks they run in triggered or continuous mode. Auto Loader adds a cloudFiles source that picks up new files as they arrive in cloud storage. In Databricks Runtime 17.3 and above, the RocksDB state store is the default, and changelog checkpointing, which writes only records that changed since the last checkpoint, is on by default. For operational workloads that need an immediate response, such as fraud detection, real-time mode on classic compute adds a low-latency trigger to the same Structured Streaming APIs.

FAQs

Does Flink support SQL?

Yes. Flink SQL and the Table API share one underlying engine, and the Table API is available in Java, Scala and Python.

What happened to Flink's DataSet API?

Flink 2.0 removed it, along with the Scala DataStream and DataSet APIs. Batch programs now use the DataStream API or the Table API and SQL.

How do Spark and Flink handle late-arriving data?

Spark uses a watermark: data within the lateness threshold still updates state, and data beyond it starts getting dropped. Flink also reasons about event time with watermarks and can reroute late events through side outputs.