SDP versus plain Structured Streaming?

Summary

• Plain Structured Streaming and Spark Declarative Pipelines (SDP) use the same spark.readStream reads. The difference is who runs the query. In plain Structured Streaming you call writeStream and start() and manage each query. In SDP you define a streaming table and the pipeline runs its flows.
• SDP sets checkpoint storage once, in the pipeline spec, and orders dependent flows for you. Plain Structured Streaming sets a checkpoint location per query and exposes the Append, Complete and Update output modes, the built-in sinks and the trigger options.
• Neither sets watermarks for you. Databricks documents withWatermark() and the WATERMARK clause inside pipelines.
• SDP arrived in Spark 4.1. Streaming reads inside an SDP dataset are still spark.readStream calls.

SDP versus plain Structured Streaming?

Structured Streaming is "a scalable and fault-tolerant stream processing engine built on the Spark SQL engine". Its guide says the system "ensures end-to-end exactly-once fault-tolerance guarantees through checkpointing and Write-Ahead Logs." Spark Declarative Pipelines (SDP), added in Spark 4.1.0, works at the level of datasets instead of queries. The SDP guide says it "supports loading data from all the formats supported by Spark Structured Streaming (spark.readStream)", so the read is the same call in both.

What does the same Kafka ingestion look like both ways?

Plain Structured Streaming starts a query and owns its sink and checkpoint:

orders = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "orders")
    .load()
)

query = (
    orders.writeStream
    .format("parquet")
    .option("path", "/tmp/orders")
    .option("checkpointLocation", "/tmp/orders_checkpoint")
    .start()
)

If you set no trigger, Structured Streaming checks for new data "as soon as the previous processing has been completed". In SDP the function returns the streaming DataFrame and the pipeline does the rest:

from pyspark import pipelines as dp
from pyspark.sql import DataFrame

@dp.table
def ingestion_st() -> DataFrame:
    return (
        spark.readStream
        .format("kafka")
        .option("kafka.bootstrap.servers", "localhost:9092")
        .option("subscribe", "orders")
        .load()
    )

SDP also takes SQL, where the STREAM keyword marks the streaming source:

CREATE STREAMING TABLE basic_st
AS SELECT * FROM STREAM samples.nyctaxi.trips;

The guide lists start() and toTable() among the methods that should never appear in dataset code. The pipeline spec's storage field names "A directory where checkpoints can be stored for streaming tables within the pipeline."

How do they compare?

Aspect	Plain Structured Streaming	SDP streaming table
What you define	A query: source, transformations, sink, output mode, trigger	A dataset: a streaming table with one or more streaming flows
Starting it	start() on the writer, once per query	spark-pipelines run executes the pipeline
Several queries	Each runs concurrently in the session	The pipeline analyzes dependencies and orders the flows
Checkpoints	checkpointLocation on each query	storage in the pipeline spec
Output modes	Append (default), Complete, Update	Append flows in Apache Spark; Databricks also exposes Update flows; sinks are append-only
Sinks	File, Kafka, foreach, foreachBatch, console, memory	Streaming tables and materialized views; external targets through dp.create_sink, Python only
Watermarks	You call withWatermark()	You call withWatermark(), or use the WATERMARK clause in SQL on Databricks
Starting over	Some settings can't change after a run: discard the checkpoint and start a new query	--full-refresh clears all existing data and checkpoints for the datasets you list, then recomputes them

Does SDP set watermarks for you?

No. The Databricks docs say to "use watermarks when performing stateful stream processing in Lakeflow pipelines, including aggregations, joins, and deduplication." They also warn that an unbounded stream without a watermark "can cause a pipeline to fail due to memory pressure." What pipelines can manage is state store configuration: serverless pipelines do that automatically. Stream-stream joins are a Structured Streaming feature, available since Spark 2.3. Databricks shows one inside a pipeline flow, with a watermark on both sides and a time interval clause.

When should you use each?

Use SDP when several streaming and batch datasets depend on each other. In the guide's example, the streaming table orders is joined with a customers materialized view, another materialized view aggregates the result, and the pipeline runs them in dependency order. You set checkpoint storage once and can reset chosen datasets with --full-refresh.
Use plain Structured Streaming when you want Complete output mode, foreachBatch handling in your own code, a trigger chosen per query, or one standalone query inside an application. The SDP guide documents no trigger settings, and Databricks lists foreachBatch sinks, update flows and continuous mode as features that Lakeflow Pipelines add on top of SDP.

How do they fit on Databricks?

Lakeflow Pipelines extend SDP and run on Databricks Runtime. A pipeline runs in triggered or continuous mode, and pipeline mode is independent of the type of table being computed.

FAQs

Does SDP replace Structured Streaming?

They work together. A streaming table's query is a spark.readStream read, and the guide says SDP supports all the formats Structured Streaming supports.

Which Spark version has SDP?

Spark 4.1.0 introduced it.

Can I call writeStream inside a pipeline?

No. The guide says never to use methods that save or write to files or tables in SDP dataset code. To write to an external target such as Kafka, use dp.create_sink with @dp.append_flow.