Structured Streaming versus Flink?

Summary

• Apache Spark Structured Streaming and Apache Flink are both open-source engines for stateful stream processing. Both document event-time processing, window aggregations, SQL, exactly-once guarantees, and RocksDB-based state.
• Spark runs queries as micro-batches by default. Continuous Processing (experimental) and, from Spark 4.1.0, Real-time Mode are its lower-latency options.
• The practical differences are the API model (DataFrames and SQL on Spark, versus Flink's layered SQL, Table, DataStream, and ProcessFunction APIs), the execution model, and the platform you already run.
• Choose by API fit, platform, and the latency and state size you measure on your own workload.

Structured Streaming versus Flink?

Apache Spark Structured Streaming and Apache Flink are both Apache Software Foundation projects for processing streams with state. They differ in API style, execution model, and how you operate them. This page compares what each project documents.

How do they differ in design?

Aspect	Spark Structured Streaming	Apache Flink
Programming model	Treats a live stream as a table that is continuously appended and runs your DataFrame or SQL query incrementally	Layered APIs: SQL, Table API, DataStream API, and ProcessFunction at the lowest level
Default execution	Micro-batch, a series of small batch jobs	A dataflow of tasks run on TaskManagers and coordinated by a JobManager
Latency focus	Micro-batch by default, with Continuous Processing and Real-time Mode as options	Lists low latency and high throughput among its performance features
Time and late data	Event-time windows, with withWatermark() setting the late-data threshold	Event time programs must specify how to generate watermarks
Windows	Tumbling, sliding, and session	Tumbling, sliding, session, and global, plus custom window assigners
State	HDFS-backed state store (default) or RocksDB provider (Spark 3.2 and later)	HashMapStateBackend (default, Java heap) or EmbeddedRocksDBStateBackend (local disk, incremental checkpoints)
Fault tolerance	Checkpointing and write-ahead logs, with end-to-end exactly-once when sources are replayable and sinks are idempotent	Checkpoints recover state and stream positions, and savepoints serve planned manual operations
Batch and SQL	The same DataFrame API and SQL engine as batch Spark	Batch queries on bounded data and continuous queries on unbounded streams, with SQL on both
Deployment	Standalone, YARN, or Kubernetes	YARN, Kubernetes, standalone, or as a library

What are Spark's lower-latency options?

Micro-batch is the default and provides exactly-once guarantees. Continuous Processing, introduced in Spark 2.3, is experimental. It gives at-least-once guarantees, supports only map-like operations (no aggregations), and has no automatic retries of failed tasks. Real-time Mode was added in Spark 4.1.0 (SPARK-53736) for continuous, sub-second processing. The 4.1 release covers Scala stateless queries, and Spark 4.2.0 adds the Real-time Mode trigger to PySpark (SPARK-54660).

How should I choose?

• API fit: Structured Streaming lets you write a streaming computation the same way as a batch computation on static data, in Scala, Java, Python, or R. For custom stateful logic, the Spark guide recommends the transformWithState operator (Spark 4.0 and later). Flink's ProcessFunction processes events from one or more streams with fault-tolerant state and event-time and processing-time callbacks.
• Platform: If you already run Spark for batch work, Structured Streaming uses the same engine. Flink runs as its own JobManager and TaskManager cluster, or embedded as a library.
• Latency: Both projects have low-latency features. Benchmark your own workload.
• State: Both offer RocksDB-based state. Check checkpoint cost and recovery behavior at your real state size.
• Connectors: Both document Kafka connectors, so either can read the same topic. Flink's connectors are not part of its binary distribution, so check that a connector release exists for your Flink version.

How does Databricks fit in?

Databricks documents Structured Streaming as a near real-time processing engine with end-to-end fault tolerance and exactly-once processing guarantees using familiar Spark APIs. Delta Lake tables work as streaming sources and sinks, and Kafka works as a source or sink.
Databricks does not support the continuous trigger mode. It documents real-time mode instead, a trigger type for Structured Streaming aimed at operational workloads such as fraud detection. It runs long-running batches, schedules all stages of the query at once, and passes data between stages with a streaming shuffle. Databricks says to use micro-batch mode for analytical processing such as ETL, and notes that real-time mode needs dedicated compute. It recommends benchmarking real-time mode against your target workload.
Lakeflow Pipelines provide a declarative framework for batch and streaming pipelines in SQL and Python.

FAQs

Does Flink support SQL?

Yes. The Flink documentation calls SQL its highest-level abstraction, and its homepage lists SQL on stream and batch data.

Can Structured Streaming use RocksDB for state?

Yes. Spark 3.2 added a built-in RocksDB state store provider, enabled through spark.sql.streaming.stateStore.providerClass. Spark also keeps its HDFS-backed provider as the default.

Is Continuous Processing ready for production use?

The Spark guide labels it experimental and lists its limits: map-like operations only, a short list of supported sources and sinks, at-least-once guarantees, and no automatic task retries. Databricks does not support the continuous trigger mode. Test those limits against your workload first.

Can I run both on the same Kafka topic?

Yes. Spark has a Kafka source, and the Flink documentation describes an Apache Kafka connector for reading from and writing to Kafka topics. Flink's streaming connectors are not part of its binary distribution.