When should I use Spark Connect versus the classic driver?

Summary

• Use Spark Connect when the client should run apart from the Spark driver: in an application, notebook or IDE, with its own dependencies, and with the server upgraded independently.
• Stay on Spark Classic, the default, when your code needs RDDs, SparkContext or driver JVM access, or depends on eager analysis errors.
• Both modes execute queries lazily, but Spark Connect also defers analysis and name resolution to execution time, which changes temporary view, UDF and error behavior.
• An existing application can try Spark Connect with spark.api.mode=connect.
• On Databricks, serverless compute and Databricks Connect use Spark Connect, and RDD APIs are not supported on serverless or standard compute.

When should I use Spark Connect versus the classic driver?

Use Spark Connect when the client should run apart from the Spark driver, for example inside an application, notebook or IDE. Use Spark Classic, which is the default, when your code needs RDDs, SparkContext or driver JVM access. In a classic application, client code runs on the Spark driver. With Spark Connect, the client "does not run in the same process as the Spark driver" and talks to a Spark server over gRPC.

How do Spark Classic and Spark Connect differ?

Aspect	Spark Classic	Spark Connect
Where the client runs	On the Spark driver	In its own process
Query execution	Lazy	Lazy
Commands such as spark.sql("insert …")	Eager	Eager
Schema analysis	Eager	Lazy
Schema access (df.columns, df.schema)	Local	Triggers an RPC and caches the schema on first access
Temporary views	Plan embedded	Name lookup
UDF serialization	At creation	At execution
RDDs and SparkContext	Available	Unsupported

Spark Connect supports most PySpark APIs (since Spark 3.4), most Scala APIs (since 3.5) and the majority of the Streaming API.

When should I choose Spark Connect?

The Spark docs list these benefits:

• Stability. "Applications that use too much memory will now only impact their own environment as they can run in their own processes."
• Your own dependencies. "Users can define their own dependencies on the client and don't need to worry about potential conflicts with the Spark driver."
• Independent upgrades. The Spark driver can be "upgraded independently of applications", and applications stay forward-compatible "as long as the server-side RPC definitions are designed to be backwards compatible".
• Remote use. The decoupled architecture allows "remote connectivity to Spark beyond SQL and JDBC", so "any application can now interactively use Spark" as a service. Spark Connect also "enables interactive debugging during development directly from your favorite IDE".

When should I stay on Spark Classic?

Stay on Spark Classic if your code uses RDDs, because the docs say they "can no longer be directly used from Spark Client applications", or if it touches driver JVM fields such as df._jdf. Code that relies on analysis exceptions at transformation time needs the explicit analysis triggers shown below. spark.api.mode defaults to classic.

How do I try Spark Connect on an existing application?

from pyspark.sql import SparkSession

SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate()

spark-submit --master "..." --conf spark.api.mode=connect

For local testing, setting spark.remote to local[...] or local-cluster[...] starts a local Spark Connect server. The docs note that spark.remote and --remote "are limited to local* values", while --conf spark.api.mode=connect with --master ... supports cluster URLs such as spark://.

What breaks when I migrate?

Symptom	Cause	Fix
An existing DataFrame changes after you replace a temporary view	The DataFrame stores only the view name	Create unique temporary view names, for example with a UUID
A UDF returns the latest value of an outer variable	Python UDF serialization is deferred until execution	Wrap UDF creation in a function factory that binds the value early
A wrong column name raises no error at the transformation	Analysis is deferred	Trigger analysis with df.columns, df.schema or df.collect()
Many new DataFrames each call df.columns	Each call causes a new Analyze request to the server	Build all column expressions, then use a single select

How does Databricks fit?

The Databricks docs say Spark Connect is used in Scala notebooks on Databricks Runtime 13.3 and above and Python notebooks on Databricks Runtime 14.3 and above (both on standard compute), on serverless compute, in Databricks Connect, and in Lakeflow pipelines with environment versions configured. On serverless compute, "Only Spark Connect APIs are supported. Spark RDD APIs are not supported." Standard compute also lists "RDD APIs are not supported" and says "If these features are required for your workload, use dedicated compute instead." Databricks documents the same lazy-analysis differences in "Compare Spark Connect to Spark Classic".

FAQs

Can I switch an application between the two modes?

Yes. The docs say that, depending on the value of spark.api.mode, "the application can run in either Spark Classic or Spark Connect mode".

What if my code uses RDDs?

Use Spark Classic, or move to the DataFrame API: "Alternatives for missing RDD functionality are provided as part of the higher-level DataFrame API". Spark 4.2.0 lists RDD API compatibility additions such as DataFrame.zipWithIndex in PySpark.

Is Spark Connect slower?

Measure your own workload. The docs do describe one pattern to avoid: calling df.columns on many new DataFrames sends many analysis requests to the server, and "Performance can be improved if users avoid large numbers of Analyze requests".

Can I monitor Spark Connect applications?

Spark 3.5.0 added a Spark UI page for Spark Connect, and Spark 4.2.0 makes the Spark Connect tab available in the History Server.