Spark Connect versus the classic Spark driver?

Summary

• In Spark Classic, your application runs with the Spark driver, so it can use SparkContext, RDDs and, in PySpark, Py4J objects such as df._jdf. In Spark Connect, a thin client sends unresolved logical plans to a Spark server over gRPC and "does not run in the same process as the Spark driver."
• Classic is the default in Apache Spark: spark.api.mode defaults to classic. You opt in to Spark Connect with SPARK_REMOTE, --remote or spark.api.mode=connect.
• Behavior differs. Spark Connect defers analysis, looks up temporary views by name at execution time, and serializes Python UDFs at execution time.
• RDDs, SparkContext and df._jdf are not supported in Spark Connect. Most of the DataFrame, SQL and Structured Streaming APIs are.

Spark Connect versus the classic Spark driver?

Spark Connect versus the classic Spark driver?

Apache Spark has two API modes. In Spark Classic, your application runs with the Spark driver. In Spark Connect, a thin client talks to a remote Spark server. The spark.api.mode setting (classic or connect, default classic, since Spark 4.0.0) lets a Classic application switch to Spark Connect.

How do they differ?

Dimension	Spark Classic	Spark Connect
Process model	Application code runs with the driver; in client mode the driver launches inside the spark-submit process	The client does not run in the Spark driver process
Communication	PySpark reaches the JVM through Py4J	gRPC with protocol buffers; results return as Arrow batches
Version coupling	Client and driver must run identical software versions	The client and the Spark driver are decoupled
RDDs and SparkContext	Available	Not supported
Schema analysis	Eager	Lazy; df.columns and df.schema trigger an analysis RPC
Temporary views	Plan embedded in the DataFrame	Looked up by name at execution time
UDF serialization	At creation	At execution
Client dependencies	Java installed	pyspark-client is pure Python, with no jars or JRE

Query execution is lazy in both modes. Actions such as df.collect() and df.show(), and SQL commands such as spark.sql("insert ..."), run eagerly in both. The analysis difference is the one that surprises people:

df = spark.sql("select 1 as a, 2 as b").filter("c > 1")
df.columns

Spark Classic raises an error on the first line, because column c cannot be found. Spark Connect raises it only at df.columns (or df.show()), because the unresolved plan reaches the server then.

How do I switch an application to Spark Connect?

You have four documented options:

1. Set the SPARK_REMOTE environment variable, for example export SPARK_REMOTE="sc://localhost". The docs say no code change is needed.
2. Start a shell with ./bin/pyspark --remote "sc://localhost".
3. Set spark.api.mode to connect.
4. Install pyspark-connect to make Spark Connect the default for PySpark.

from pyspark.sql import SparkSession

SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate()

The same setting works at submit time: spark-submit --master "..." --conf spark.api.mode=connect. The docs note that spark.remote and --remote accept only local* values, while spark.api.mode=connect with --master also supports cluster URLs such as spark://.

Which migration gotchas matter?

• Temporary views: in Spark Connect a DataFrame stores only the view's name, so replacing the view changes the DataFrame. Use unique names, for example with a UUID.
• UDFs: in Spark Connect, Python UDFs read captured variables when they run, not when you define them.
• Errors: analysis errors surface at df.columns, df.schema or an action, not at the transformation.
• Schema access: calling df.columns on a large number of new DataFrames sends many analysis requests.
• RDD code: it must move to the DataFrame API. The Spark 4.2.0 release notes list some DataFrame additions under "RDD API compatibility" (SPARK-55227).

When does each one fit?

Spark Classic fits when:

• Your code uses RDDs, SparkContext, df._jdf or other private JVM methods.
• The driver can run close to the workers. Spark's cluster docs advise running the driver near the worker nodes, preferably on the same local area network.

Spark Connect fits when:

• Clients are remote applications, IDEs or notebooks that should not host a driver.
• You want to upgrade the Spark driver without updating clients, or keep client dependencies separate from the driver's.
• A client failure or memory spike should not affect the driver or other applications.

Which client languages exist?

The Spark docs say Spark Connect supports PySpark and Scala applications. Apache also hosts separate client repositories for Go, Swift and Rust. The Go README calls its client "highly experimental" and says it should not be used in production. The Swift README calls its project a subproject of Apache Spark.

How does Databricks use Spark Connect?

The Databricks docs list Spark Connect as used in Scala notebooks on Databricks Runtime 13.3 and above and Python notebooks on 14.3 and above (both on standard compute), serverless compute and Databricks Connect. Databricks Connect is built on open-source Spark Connect.

FAQs

Is Spark Connect the default?

Not in Apache Spark: spark.api.mode defaults to classic. Spark 4.0 added an extra release tarball with Spark Connect enabled by default, and pip install pyspark-connect makes it the PySpark default.

Can I run a Classic session and a Spark Connect session together?

The PySpark quickstart says to stop the existing regular Spark session first, because it "cannot coexist with the remote Spark Connect session".

Does Spark Connect support streaming?

Most of the Streaming API, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener.