What is a streaming table in SDP?

Summary

• A streaming table is one of three dataset types in Spark Declarative Pipelines (SDP), alongside materialized views and temporary views. The Spark guide defines it as "a definition of a table and one or more streaming flows written into it", so it processes only new data as it arrives.
• You create one with @dp.table and a streaming read in Python, or with CREATE STREAMING TABLE ... AS SELECT ... FROM STREAM ... in SQL.
• Any source spark.readStream supports can feed it, such as Kafka. The storage field in the pipeline spec names the directory where checkpoints for streaming tables can be stored.
• On Databricks, a Lakeflow Pipelines streaming table is a Delta table that handles each record once, assuming an append-only source.

What is a streaming table in SDP?
A streaming table is one of the dataset types in Spark Declarative Pipelines (SDP), the Apache Spark framework added in Spark 4.1.0. The Spark guide defines it as "a definition of a table and one or more streaming flows written into it", and says streaming tables "support incremental processing of data, allowing you to process only new data as it arrives."

How do you define a streaming table?

In Python, use the @dp.table decorator on a function that performs a streaming read:

from pyspark import pipelines as dp
from pyspark.sql import DataFrame

@dp.table
def basic_st() -> DataFrame:
    return spark.readStream.table("samples.nyctaxi.trips")

In SQL, the STREAM keyword marks the source as streaming:

CREATE STREAMING TABLE basic_st
AS SELECT * FROM STREAM samples.nyctaxi.trips;

For the equivalent pattern CREATE STREAMING TABLE target_table AS SELECT * FROM STREAM source_table, the guide says SDP creates the table "along with a flow that reads new data from source_table and writes it to target_table." Spark 4.2 injects spark into each pipeline file. On Spark 4.1, add spark = SparkSession.active() yourself.

How does a streaming table differ from the other dataset types?

Dataset	Guide definition	Python	SQL
Streaming table	A table plus one or more streaming flows, processing only new data	@dp.table with a streaming read	CREATE STREAMING TABLE
Materialized view	"a view that is precomputed into a table", with exactly one batch flow	@dp.materialized_view	CREATE MATERIALIZED VIEW
Temporary view	Scoped to one execution of the pipeline	@dp.temporary_view	CREATE TEMPORARY VIEW

What can feed a streaming table?

The guide says "SDP supports loading data from all the formats supported by Spark Structured Streaming (spark.readStream)." Its example reads a Kafka topic:

@dp.table
def orders() -> DataFrame:
    return (
        spark.readStream
        .format("kafka")
        .option("kafka.bootstrap.servers", "localhost:9092")
        .option("subscribe", "orders")
        .load()
    )

A streaming table can have several flows. Create the table, then point an append flow at it for each source:

dp.create_streaming_table("customers_us")

@dp.append_flow(target = "customers_us")
def append_customers_us_west() -> DataFrame:
    return spark.readStream.table("customers_us_west")

The SQL form is CREATE STREAMING TABLE customers_us; followed by CREATE FLOW ... AS INSERT INTO customers_us SELECT * FROM STREAM(...).

How are checkpoints and refreshes handled?

The pipeline spec's required storage field names "A directory where checkpoints can be stored for streaming tables within the pipeline." With no refresh options, spark-pipelines run performs a default incremental update. --refresh updates chosen datasets without clearing existing data. --full-refresh "clears all existing data and checkpoints for the specified datasets and recomputes them from scratch." Other datasets read a streaming table by name, for example a materialized view that calls spark.table("orders").

What does a Lakeflow Pipelines streaming table add?

Lakeflow Pipelines extend SDP, and the Databricks docs describe streaming tables there in more detail. A streaming table is "a Delta table with additional support for streaming or incremental data processing", and each record is processed once, assuming an append-only source. One pipeline owns and updates each streaming table. Streaming tables are always defined against streaming sources, and those streams should be naturally bounded or bounded with a watermark, because an unbounded stream without one "can cause a pipeline to fail due to memory pressure." If you change the query, existing rows aren't recomputed and only new rows follow the new logic, so reprocessing needs a full refresh. Joins in a streaming table also don't recompute when a dimension changes, and Databricks suggests a materialized view when the result must always be correct. For sources whose records are updated or deleted, Databricks points to AUTO CDC. Data quality expectations are a Lakeflow Pipelines feature that Apache Spark doesn't include.

FAQs

Do I manage checkpoints myself?

You name a directory with storage in the pipeline spec. The guide describes it as where checkpoints for streaming tables can be stored.

Can a streaming table have more than one flow?

Yes. The definition says "one or more streaming flows". Use dp.create_streaming_table with @dp.append_flow, or CREATE FLOW in SQL.

Can a materialized view read a streaming table?

Yes. The guide's pipeline example defines orders as a streaming table and joins it with spark.table("orders") inside a materialized view.

What is the difference between SDP and Lakeflow Pipelines?

Lakeflow Pipelines share SDP's authoring model and run on Databricks Runtime. Databricks lists additions such as AUTO CDC, data quality expectations, a queryable event log and continuous mode.