What is a stream-static join?

Summary

• A stream-static join joins a streaming DataFrame with a static (batch) DataFrame in Apache Spark Structured Streaming, typically to enrich each micro-batch of events with reference data such as customer or device attributes.
• Stream-static joins are not stateful: Spark buffers no past input for the join and needs no watermark, so the join adds no state store to size or clean up.
• With the stream on the left, Spark supports inner, left outer and left semi joins; right outer and full outer joins aren't supported in that direction.
• A streaming row is matched once, when its micro-batch runs. Reference data that arrives later doesn't update rows that were already processed.
• On Databricks, when the static side is a Delta table, each micro-batch joins against the latest valid version of that table, which should be slowly changing.

What is a stream-static join?

A stream-static join is a join in Spark Structured Streaming where one input is a streaming DataFrame and the other is a static, batch DataFrame. It's the standard way to enrich an event stream with reference data: each order gets its customer's region, each sensor reading gets its device model. Structured Streaming has supported it since Spark 2.0, and it's stateless, so there's no join state to size and no watermark to configure. The trade-off is that each match is decided once, when the streaming row is processed.

How a stream-static join works

You write it like any DataFrame join. As each micro-batch runs, its newly arrived rows are joined with the static DataFrame and the results go to the sink. The Structured Streaming guide pins down the semantics: in every supported join type, "the result of the join with a streaming Dataset/DataFrame will be exactly the same as if it was with a static Dataset/DataFrame containing the same data in the stream."
The guide also states that "stream-static joins are not stateful, so no state management is necessary." Two practical consequences follow:

• The join doesn't accumulate state as the stream grows. A stream-stream join, by contrast, buffers past input from both sides as streaming state.
• A streaming row gets one chance to match. If its reference row lands after the event was processed, an inner join has already dropped the event and a left outer join has already emitted it with nulls.

Supported join types

Join type	Stream on the left, static on the right	Static on the left, stream on the right
Inner	Supported, not stateful	Supported, not stateful
Left outer	Supported, not stateful	Not supported
Right outer	Not supported	Supported, not stateful
Full outer	Not supported	Not supported
Left semi	Supported, not stateful	Not supported

The pattern to remember: in an outer join, the stream must be the preserved side. Full outer stream-static joins aren't supported in either direction.

Example in PySpark

events = spark.readStream.table("raw_events")          # streaming input
devices = (spark.read.table("devices")                 # static reference data
           .select("device_id", "model", "site"))

enriched = events.join(devices, "device_id", "left_outer")

(enriched.writeStream
    .option("checkpointLocation", "/checkpoints/enriched_events")
    .toTable("enriched_events"))

DataStreamReader.table() and DataStreamWriter.toTable() have been available since Spark 3.1. The join itself needs no withWatermark() call. Selecting only the reference columns you need keeps the static side narrow.

Stream-static or stream-stream?

Aspect	Stream-static join	Stream-stream join
Inputs	One streaming and one static DataFrame	Two streaming DataFrames
Join state	None	Past input from both sides buffered as state
Watermarks	Not needed for the join	Optional for inner joins; required with time constraints for outer and semi joins
Late-arriving matches	Not re-matched	Can still match while the other side's input is in state
Typical fit	Slowly changing dimension or lookup data	Two event streams that must be correlated in time

How Databricks approaches stream-static joins

On Databricks, the documented behavior assumes the static side is a Delta table. Each micro-batch joins its records against the latest valid version of that Delta table, so later micro-batches pick up dimension changes. Because the join is stateless, you don't configure watermarking. The docs add two cautions: the static Delta table should be slowly changing, and output isn't deterministic while the static side changes, since reprocessing the same streaming data later can join against a different table version. See Work with joins on Databricks.
In Lakeflow Pipelines, a stream-static join is a streaming table definition:

CREATE OR REFRESH STREAMING TABLE customer_sales
AS SELECT * FROM STREAM(sales)
  LEFT JOIN customers USING (customer_id)

With each pipeline update, new stream records are joined with the most current snapshot of the static table. In triggered pipelines, the static table returns results as of the time the update started; in continuous pipelines, its most recent version is queried each time the table processes an update. Rows that were already processed aren't recalculated unless you run a full refresh. When results must always reflect current dimensions, Databricks points to materialized views, which recompute joins when dimensions change.

FAQs

Do stream-static joins need a watermark?

No. The join keeps no state, so there's nothing for a watermark to clean up. Other stateful operators in the same query, such as windowed aggregations, follow their own watermark rules.

Can I run a full outer stream-static join?

No. Full outer joins between a stream and a static DataFrame aren't supported in either direction. A stream-stream join supports full outer joins conditionally, with a watermark and time constraints.

Why are some events missing after the join?

With an inner join, events whose key has no match in the static side when their micro-batch runs are dropped, and they aren't re-joined later. Use a left outer join with the stream on the left to keep them, with nulls in the reference columns.

Does the static side pick up new data?

With a Delta table on Databricks, yes: each micro-batch uses the latest valid version. In open-source Spark, the static side is an ordinary batch DataFrame, so confirm how your source reflects updates before relying on it.