What is Spark Structured Streaming?

Summary

• Structured Streaming is Apache Spark's stream processing engine, built on the Spark SQL engine. You write a streaming computation like a batch computation on static data, and Spark runs it incrementally as data arrives.
• It treats a live stream as a continuously appended table. The output mode (append, update or complete) decides which result rows reach the sink.
• By default, queries run as micro-batches. Triggers set the timing, and continuous processing is an experimental low-latency mode.
• Checkpointing and write-ahead logs record the offsets processed in each trigger. With replayable sources and idempotent sinks, you get end-to-end exactly-once semantics.

What is Spark Structured Streaming?

Structured Streaming is "a scalable and fault-tolerant stream processing engine built on the Spark SQL engine." You express a streaming computation the same way you would express a batch computation on static data. The Spark SQL engine runs it incrementally and continuously, updating the result as data arrives. The DataFrame and Dataset APIs, in Scala, Java, Python or R, cover streaming aggregations, event-time windows and stream-to-batch joins.
Spark Streaming (DStreams) is the previous generation, a legacy project that no longer receives updates.

How does the programming model work?

The key idea is to treat a live data stream as a table that is being continuously appended. Each arriving record is like a new row in an input table. A query on the input produces a result table, and every trigger appends new rows to the input and updates the result.

Concept	What it is	Why it matters
Input table	The stream, modeled as a table that grows with each record	You query it with ordinary DataFrame or SQL operations
Result table	The output of your query, updated every trigger	It holds what will be written to the sink
Append mode (default)	Writes only rows added since the last trigger	Only for queries whose result rows never change, such as select, filter and join
Update mode	Writes only rows updated since the last trigger	Without aggregations, it's equivalent to append mode
Complete mode	Writes the whole result table every trigger	Supported for aggregation queries

How does micro-batch processing work?

Internally, by default, Structured Streaming processes data streams as a series of small batch jobs. The trigger defines the timing:

• Unspecified (default): a new micro-batch starts as soon as the previous one completes.
• Fixed interval (trigger(processingTime='2 seconds')): micro-batches start at the interval you set. If a batch overruns it, the next one starts as soon as it completes.
• Available-now (trigger(availableNow=True)): processes all available data, possibly in several micro-batches, then stops. It replaces the deprecated one-time trigger.
• Continuous (experimental): a low-latency processing mode, added in Spark 2.3, with at-least-once guarantees. It supports only map-like operations, such as projections and selections.

The Spark 4.1.0 release added Real-Time Mode (RTM) for Structured Streaming with stateless Scala support, and Spark 4.2.0 added RTM trigger support to PySpark.

How does Structured Streaming recover from failures?

Every streaming source is assumed to have offsets that track the read position. The engine uses checkpointing and write-ahead logs to record the offset range processed in each trigger, and saves progress and running aggregates to a checkpoint location in an HDFS-compatible file system. Sinks are designed to be idempotent for reprocessing, so replayable sources and idempotent sinks give end-to-end exactly-once semantics. The guarantee depends on the sink: the file sink is exactly-once, while the Kafka and foreach sinks are at-least-once.

Which sources and sinks are built in?

Built-in sources are files (text, CSV, JSON, ORC and Parquet), Kafka, and socket and rate sources for testing. Built-in sinks are file, Kafka, foreach, foreachBatch, console and memory. Delta Lake tables work as streaming sources and sinks, and both Delta Lake and Apache Iceberg support append and complete output modes for streaming writes.
This example reads a Kafka topic (it needs the spark-sql-kafka-0-10_2.13 package) and writes the records to a table with toTable(), available since Spark 3.1:

df = spark \
    .readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
    .option("subscribe", "topic1") \
    .load()

df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)") \
    .writeStream \
    .option("checkpointLocation", "path/to/checkpoint/dir") \
    .toTable("myTable")

How does stateful processing work?

Aggregations, deduplication and stream-stream joins keep state across micro-batches in a state store, a versioned key-value store. The default HDFS-backed provider keeps state in executor JVM memory, backed by files. Since Spark 3.2 you can set spark.sql.streaming.stateStore.providerClass to org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider, which keeps state in native memory and on local disk to avoid large JVM garbage collection pauses. Since Spark 4.0, the guide encourages the transformWithState operator for complex stateful applications.
Watermarks bound that state. withWatermark() sets how late data is expected to be, so the engine can clean up old state:

from pyspark.sql.functions import window

windowedCounts = words \
    .withWatermark("timestamp", "10 minutes") \
    .groupBy(
        window(words.timestamp, "10 minutes", "5 minutes"),
        words.word) \
    .count()

How does Databricks run Structured Streaming?

On Databricks, Auto Loader incrementally processes new data files as they arrive in cloud storage. Databricks also documents real-time mode, a Structured Streaming trigger type for operational workloads that need an immediate response to streaming data. It's available in Lakeflow Pipelines too.

What else do people ask?

Can I query a stream with SQL?

Yes. Register a streaming DataFrame with df.createOrReplaceTempView("updates"), and spark.sql("select count(*) from updates") returns another streaming DataFrame. Check the query plan, because Spark can inject stateful operations when it interprets SQL against a stream.

Does checkpointing add latency?

Offset management does, because no data processing can occur until those operations complete. The asyncProgressTrackingEnabled option checkpoints progress in parallel with processing, but only for stateless queries that use the Kafka sink, and it doesn't support exactly-once end-to-end processing.