What's the status of Spark Connect in Spark 4?

Summary

• Spark Connect is the decoupled client-server architecture introduced in Apache Spark 3.4, and the Spark 4.0, 4.1 and 4.2 release notes each list Spark Connect additions.
• Classic mode is still the default: spark.api.mode defaults to classic (since 4.0.0), so Spark Connect is opt-in.
• Spark Connect supports most PySpark APIs (since 3.4), most Scala APIs (since 3.5) and the majority of the Streaming API. RDDs and SparkContext are unsupported.
• Spark 4.0 added the pyspark-client package, Spark 4.1 made Spark ML on Connect GA for the Python client, and Spark 4.2 added the GetStatus API and a Spark Connect tab in the History Server.
• Go, Swift and Rust clients live in separate Apache repositories, and the Go README calls its client "highly experimental".

What's the status of Spark Connect in Spark 4?
As of October 2026, Spark Connect is part of Apache Spark and gains features in every release. The Spark 4.2 docs describe it as "a decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol". It was introduced in Spark 3.4, and the current docs are for Spark 4.2.0, released July 14, 2026. Classic mode is still the default. The spark.api.mode setting defaults to classic, so you opt in to Spark Connect.

What changed in each Spark release?

Release	Spark Connect changes in the release notes
3.4.0	A Python client for Spark Connect and a basic Scala client
3.5.0	The Scala client, Structured Streaming support in Python and Scala, and an initial Go client. The notes mention "more scenarios with general availability for Spark Connect"
4.0.0	pyspark-client (a lightweight Python client), a release tarball with Spark Connect enabled by default, full API compatibility for the Java client, the spark.api.mode setting, "Greatly expanded API coverage", ML on Spark Connect and a Swift client
4.1.0	"Spark ML on Connect is GA for the Python client", a JDBC driver for Spark Connect, and large-workload stability work such as zstd-compressed protobuf plans and chunked Arrow result streaming
4.2.0	The GetStatus API on server and client, a Spark Connect tab in the History Server, session release on process exit when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set, and Change Data Capture APIs. The notes say the release "continues to make PySpark faster and Spark Connect more capable"

What does Spark Connect support today?

The overview page states: "Support for more APIs is planned for upcoming Spark releases."

Area	What the Spark 4.2 docs say
PySpark	"Since Spark 3.4, Spark Connect supports most PySpark APIs, including DataFrame, Functions, and Column"
Scala	"Since Spark 3.5, Spark Connect supports most Scala APIs, including Dataset, functions, Column, Catalog and KeyValueGroupedDataset"
Structured Streaming	"Majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener"
UDFs	Supported "by default for the shell and in standalone applications with additional set-up requirements"
RDDs and SparkContext	"APIs such as SparkContext and RDD are unsupported in Spark Connect"

The Python API reference labels supported APIs, for example "Changed in version 3.4.0: Supports Spark Connect." on DataFrame.explain.

How do I try it?

Start a server from a Spark download, then connect with the pure-Python client:

./sbin/start-connect-server.sh
pip install pyspark-client==4.2.0

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
spark.range(1,10).show()

You can also set export SPARK_REMOTE="sc://localhost" and start ./bin/pyspark, which needs no code change. The default server port is 15002 (spark.connect.grpc.binding.port). An existing classic application can switch with SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate(), or with spark-submit --master "..." --conf spark.api.mode=connect.

What is not supported or behaves differently?

• No JVM access. The client "does not run in the same process as the Spark driver", and in PySpark it "does not use Py4J", so private fields such as df._jdf are unavailable.
• No static configuration or SparkContext. Connect "provides a session-based client", so the client has no access to them.
• Deferred analysis. Per the Spark docs, "Spark Connect defers analysis and name resolution to execution time". spark.sql("select 1 as a, 2 as b").filter("c > 1") raises no error until df.columns or df.show().
• No built-in authentication. Spark Connect "does not have built-in authentication", but its "gRPC HTTP/2 interface allows for the use of authenticating proxies".

Which other languages have Spark Connect clients?

Apache hosts separate repositories for Go, Swift and Rust, and each README states its own maturity. The Go README says the client "is highly experimental and should not be used in any production setting". The Swift README calls its library "a subproject of Apache Spark", and the 4.0.0 notes list "A new client implementation for Swift". The Rust repository is titled "Apache Spark Connect Client for Rust".

How does Databricks use Spark Connect?

The Databricks docs say Spark Connect is used in Scala notebooks on Databricks Runtime 13.3 and above and Python notebooks on Databricks Runtime 14.3 and above (both on standard compute), on serverless compute, in Databricks Connect, and in Lakeflow pipelines with environment versions configured. Databricks Connect "is built on open-source Spark Connect". On serverless compute, "Only Spark Connect APIs are supported. Spark RDD APIs are not supported." Databricks Connect "does not support connecting to open source Apache Spark servers", so use the Apache Spark client packages, such as pyspark-client, for those.

FAQs

Is Spark Connect production-ready?

Judge it per area, using the lifecycle wording in the release notes and READMEs. The 3.5.0 notes mention "more scenarios with general availability for Spark Connect", the 4.1.0 notes call Spark ML on Connect GA for the Python client, and the Go README warns against production use.

Do I need a separate Spark server?

Yes, for a remote session. For local testing, setting spark.remote to local[...] or local-cluster[...] starts a local Spark Connect server.

Can I use RDDs with Spark Connect?

No. The Spark docs say RDDs "can no longer be directly used" from Spark client applications, and that alternatives for missing RDD functionality are in the DataFrame API. The 4.2.0 notes list RDD API compatibility additions such as DataFrame.zipWithIndex in PySpark.