Does Spark Connect support Structured Streaming?

Summary

• Yes, for most of the API. The Scala section of the Spark Connect overview says "Majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener."
• PySpark marks support class by class. The API reference labels DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryManager "Changed in version 3.5.0: Supports Spark Connect."
• The client builds the query and the Spark server runs it. The client sends unresolved logical plans over gRPC and receives results as Apache Arrow batches.
• Two documented gaps matter for streaming code. RDD and SparkContext APIs are unsupported, and a foreachBatch function cannot read variables defined outside it.
• On Databricks, Databricks Connect is built on Spark Connect, and Lakeflow Pipelines declare streaming tables in SQL or Python.

Does Spark Connect support Structured Streaming?

Yes, for most of the API. The Spark Connect overview says "Spark Connect is available and supports PySpark and Scala applications", and it names DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener among the supported streaming APIs. Its wording is "majority", not "all", and it adds that "Support for more APIs is planned for upcoming Spark releases".

How does a streaming query run over Spark Connect?

Spark Connect separates the client from the Spark driver. The client translates DataFrame operations into unresolved logical query plans, encodes them with protocol buffers and sends them to the server over gRPC. The Spark Connect endpoint on the server turns them into Spark logical plan operators, and the standard Spark execution process takes over. Results stream back as Apache Arrow-encoded row batches. Because "the client does not run in the same process as the Spark driver", your application holds a StreamingQuery handle while the query itself runs on the server.

Which Structured Streaming APIs work over Spark Connect?

API	Over Spark Connect	Where the docs say so
DataStreamReader (spark.readStream)	Supported	PySpark reference: "Changed in version 3.5.0: Supports Spark Connect"
DataStreamWriter (df.writeStream)	Supported	Same label in the PySpark reference
StreamingQuery	Supported	Same label in the PySpark reference
StreamingQueryManager (spark.streams)	Supported	Same label in the PySpark reference
StreamingQueryListener	Supported	Named in the Spark Connect overview
foreachBatch	Supported, with different variable scoping	Same label, plus a note in the PySpark reference
RDD and SparkContext APIs	Unsupported	Spark Connect overview

What behaves differently over Spark Connect?

RDDs and SparkContext. The overview says "the Spark Connect protocol does not support all the execution APIs of Spark, most importantly RDDs", and that the client "does not have access to the static Spark configuration or the SparkContext". Streaming code that calls either will not run.
foreachBatch. The PySpark reference says this function "behaves differently in Spark Connect mode" and that "In Connect, the provided function doesn't have access to variables defined outside of it". In its example, a variable set inside the function stays at -1 over Spark Connect and becomes 100 in classic mode. Write each batch to its destination from inside the function instead of changing a variable outside it.
Release history. Spark 4.1.0 lists transformWithState under Spark Connect API coverage (SPARK-51827). Spark 4.2.0 lists a fix so that Spark Connect streaming listeners no longer silently mis-deserialize JSON-based offsets (SPARK-54390). If your listeners handle JSON-based offsets, use 4.2.0 or later.

How do you start a streaming query from a Spark Connect client?

Start a server with ./sbin/start-connect-server.sh from your Spark folder. The gRPC port is set by spark.connect.grpc.binding.port, which defaults to 15002. For a thin Python client, the docs use pip install pyspark-client==4.2.0. The Kafka source and sink ship in the separate spark-sql-kafka-0-10 artifact, so make it available to the Spark server that runs the query.

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

events = (
    spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
    .option("subscribe", "topic1")
    .load()
)

query = (
    events.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
    .writeStream.format("parquet")
    .option("path", "path/to/destination/dir")
    .option("checkpointLocation", "path/to/checkpoint/dir")
    .start()
)

Call query.awaitTermination() to block until the query stops, or inspect it:

print(query.id, query.runId)  # id persists across restarts, runId does not
print(query.status)
print(query.lastProgress)     # most recent progress update, or None
query.stop()

How does Databricks approach streaming over Spark Connect?

Databricks Connect "is built on open-source Spark Connect". Its Python limitations page lists streaming foreachBatch as not available on Databricks Connect for Databricks Runtime 13.3 LTS and below, and lists Spark Context and RDDs as not available. On standard access mode compute with Databricks Runtime 14.0 and above, files, modules and objects that a foreachBatch function references must be serializable and available on Spark, and the function cannot access dbutils.widgets.
For managed streaming without a hand-written query loop, use Lakeflow Pipelines. The docs say Lakeflow pipelines "extend and are interoperable with Spark Declarative Pipelines, while running on the performance-optimized Databricks Runtime". Streaming tables use Structured Streaming checkpoints with Delta Lake's transactional writes, so reprocessing after a failure "resumes from checkpoint". A pipeline runs in triggered mode, which stops after refreshing tables, or in continuous mode, which processes new data as it arrives.

CREATE OR REFRESH STREAMING TABLE ingestion_st
AS SELECT *
FROM STREAM read_files(
  "/databricks-datasets/retail-org/sales_orders",
  format => "json");

FAQs

Which client languages does Spark Connect support?

The overview says Spark Connect supports PySpark and Scala applications, and the Spark 4.0.0 release notes add "Full API compatibility for the Java client". Apache also hosts separate Go, Swift and Rust client repositories. Check each repository's README for what it covers before you rely on it for streaming.

Can I port a classic streaming job unchanged?

Only if it stays inside supported APIs. Remove RDD and SparkContext calls, and rewrite any foreachBatch function that depends on variables defined outside it. The overview says supported PySpark APIs are labeled "Supports Spark Connect", so check each API in the reference before you migrate.

Which Spark version do I need?

The PySpark streaming classes carry the 3.5.0 label, and the overview says Scala support for most APIs starts with Spark 3.5. The latest documentation is for Spark 4.2.0, released July 14, 2026.