Is Spark Connect the default in Spark 4?

Summary

• No. Apache Spark 4.0 added the spark.api.mode configuration with a default of classic, so a Spark 4 session runs as Spark Classic unless you opt in to Spark Connect. The Spark 4.2.0 configuration reference, the current documentation as of October 2026, still lists classic as the default.
• Opt in with the SPARK_REMOTE environment variable, the --remote flag, SparkSession.builder.remote("sc://..."), or spark.api.mode=connect. A Spark Connect server binds to port 15002 by default.
• Spark 4 did add Connect-first packaging: a download "with Spark Connect enabled" (the 4.0 release notes call it an additional release tarball with Spark Connect enabled by default) and the pure-Python pyspark-client package, which only supports spark.remote with connection URIs.
• On Databricks, serverless compute, Databricks Connect, and notebooks on standard compute (Scala on Databricks Runtime 13.3 and above, Python on 14.3 and above) use Spark Connect.

Is Spark Connect the default in Spark 4?

No. In open-source Apache Spark 4.x, Spark Classic is still the default. Spark 4.0 introduced spark.api.mode, whose default value is classic; set it to connect and the application runs through Spark Connect instead. The Spark Connect overview states the rule directly: "If you do not use one of the mechanisms outlined here, your Spark session will work just like before, without leveraging Spark Connect." As of October 2026, the configuration reference for Spark 4.2.0 (released July 14, 2026) still lists classic as the default. Beyond the new setting, Spark 4 changed packaging: a download with Spark Connect enabled, and a Python client that only speaks Spark Connect.

What is Spark Connect?

Spark Connect, introduced in Apache Spark 3.4, is a decoupled client-server architecture "that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol." The client turns DataFrame operations into unresolved logical plans encoded as protocol buffers, sends them to the server over gRPC, and receives results as Apache Arrow-encoded row batches. Because applications run in their own processes, one that uses too much memory only affects its own environment, and the Spark driver can be upgraded independently of applications.

How does Spark Connect differ from Spark Classic?

Aspect	Spark Classic	Spark Connect
Default in Spark 4	Yes (spark.api.mode = classic)	No, opt-in
Where application code runs	On the Spark driver, with direct access to the driver JVM	In a separate client process that talks to the server over gRPC
SparkContext and RDD APIs	Available	Not supported
Python package	pyspark	pyspark[connect], or the pure-Python pyspark-client

How do I turn on Spark Connect in Spark 4?

Start a server, then point a client at it. ./sbin/start-connect-server.sh starts a Spark server with Spark Connect, which binds to port 15002 by default (spark.connect.grpc.binding.port).

./sbin/start-connect-server.sh

# Environment variable: no code change needed
export SPARK_REMOTE="sc://localhost"
./bin/pyspark

# The --remote flag
./bin/pyspark --remote "sc://localhost"

# A Classic application in Connect mode (Spark 4.0 and above)
spark-submit --master "..." --conf spark.api.mode=connect app.py

from pyspark.sql import SparkSession

# Explicit remote session
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

# Or switch the API mode in code (Spark 4.0 and above)
spark = SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate()

Setting spark.remote to local[...] starts a local Spark Connect server for testing. To run against a cluster URL such as spark://, use --conf spark.api.mode=connect with --master.

What changed for Spark Connect in Spark 4 releases?

Release	Spark Connect items in the release notes
4.0.0 (May 23, 2025)	New spark.api.mode configuration "to easily turn on/off Spark Connect"; an additional release tarball with Spark Connect enabled by default; the lightweight pyspark-client; ML on Spark Connect; a Swift client
4.1.0 (December 16, 2025)	SPIP: JDBC Driver for Spark Connect; Spark Connect usability and performance improvements
4.2.0 (July 14, 2026)	RDD API compatibility work, such as DataFrame.zipWithIndex in PySpark; a server-side GetStatus API for execution status monitoring

None of these releases changes the classic default.

How Databricks approaches Spark Connect

On Databricks, Spark Connect sits behind several compute types. Per the Databricks docs, it's used in Scala notebooks on Databricks Runtime 13.3 and above and Python notebooks on 14.3 and above (both on standard compute), on serverless compute, and by Databricks Connect. Databricks Connect is built on open-source Spark Connect and lets IDEs, notebooks and custom applications run code on Databricks compute. On serverless compute, only Spark Connect APIs are supported, not Spark RDD APIs. Spark Connect also defers analysis and name resolution until a schema is accessed or an action runs, so a bad column reference can surface later than it would in Spark Classic.

FAQs

Do I need to change my code to use Spark Connect?

DataFrame code usually runs as is; with SPARK_REMOTE, the docs say no code change is needed to start using Spark Connect. Code that relies on SparkContext or RDDs needs rework, because those APIs aren't supported.

Which client languages does Spark Connect support?

The Spark Connect overview documents PySpark (since Spark 3.4) and Scala (since Spark 3.5). Spark 4.0 added a Swift client implementation, developed in a separate Apache repository.

Does pyspark-client include Spark itself?

No. It's a pure Python library that doesn't rely on jars or a JRE, and it only supports spark.remote with connection URIs such as sc://localhost. For Spark Connect dependencies alongside the full PySpark package, install pyspark[connect].

What port does Spark Connect use?

Port 15002 by default, set by spark.connect.grpc.binding.port. The Scala REPL also attempts to connect to a local server on port 15002 by default.