Should I build new apps on Spark Connect?

Summary

• Yes, if your new application is a client of a Spark cluster and uses the DataFrame, Dataset and SQL APIs. Spark Connect is a decoupled client-server architecture built for data applications, IDEs and notebooks.
• Stay on classic Spark if the code needs RDDs, SparkContext, the driver JVM or the static Spark configuration. Spark Connect does not support them.
• In Apache Spark, spark.api.mode defaults to classic, so you opt in to Spark Connect.
• Spark Connect defers schema analysis, temporary view lookup and UDF serialization until execution, so test error handling before you migrate.
• On Databricks, serverless compute, notebooks on standard compute and Databricks Connect use Spark Connect.

Should I build new apps on Spark Connect?

Yes, if your application is a client of a Spark cluster and uses the DataFrame, Dataset or SQL APIs. The Apache Spark docs describe Spark Connect as a decoupled client-server architecture that "can be embedded in modern data applications, in IDEs, Notebooks and programming languages." Choose classic Spark if your code depends on RDDs, SparkContext or the driver JVM.

How does Spark Connect work?

The Spark Connect client is a thin API. It translates DataFrame operations into unresolved logical query plans, encodes them with protocol buffers and sends them to the server over gRPC. The server translates the plans into Spark's logical plan operators, runs the standard Spark execution process and streams results back as Apache Arrow-encoded row batches. The client does not run in the same process as the Spark driver.

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").appName("SimpleApp").getOrCreate()
logData = spark.read.text("README.md")
print(logData.filter(logData.value.contains("a")).count())
spark.stop()

Start a local server with ./sbin/start-connect-server.sh. It binds port 15002 by default (spark.connect.grpc.binding.port). For a standalone Python app, install the thin client with pip install pyspark-client==4.2.0.

What do I gain by building on Spark Connect?

The Spark docs name three operational benefits:

• Stability. An application that uses too much memory only affects its own environment, and it defines its own dependencies without conflicts with the Spark driver.
• Upgradability. The Spark driver can be upgraded independently of applications, which stay forward-compatible as long as the server-side RPC definitions stay backwards compatible.
• Debuggability and observability. You can debug interactively from your IDE and monitor the application with its own metrics and logging libraries.

The Spark Connect API is code-compatible with earlier Spark versions, except for the usage of RDDs.

When should I stay on classic Spark?

The Spark docs say Spark Connect does not support the following, so keep using classic Spark if your code needs them:

• RDDs and SparkContext. The docs point to alternatives in the higher-level DataFrame API for missing RDD functionality.
• The driver JVM. In PySpark the client does not use Py4J, so private fields such as df._jdf are not available.
• The static Spark configuration. The session-based client has no access to it.

How do classic Spark and Spark Connect behave differently?

Area	Classic Spark	Spark Connect
RDD API and SparkContext	Available	Not supported
Driver JVM access	Direct	None; the client is fully separated from the server
Schema analysis	Eager, while the plan is built	Deferred; df.columns or df.schema triggers an RPC to the server
Temporary views	Plan embedded in the DataFrame	DataFrame stores the view name, so replacing the view changes the DataFrame
UDF serialization	At creation	At execution

Databricks documents fixes for these gaps. Give temporary views unique names (for example with a UUID), wrap UDF definitions in a helper function so they capture variable values when you define them, and call df.columns, df.schema or df.collect() when your code relies on catching analysis errors early.

How do I try Spark Connect on an existing app?

Set spark.api.mode to connect. The Spark docs show both forms:

from pyspark.sql import SparkSession

SparkSession.builder.config("spark.api.mode", "connect").master("...").getOrCreate()

spark-submit --master "..." --conf spark.api.mode=connect

For local testing, set spark.remote to local[...] or local-cluster[...] to start a local Spark Connect server. You can also set the SPARK_REMOTE environment variable, for example to sc://localhost, and the session becomes a Spark Connect session with no code change.

What should I check before I commit?

• Authentication. Spark Connect has no built-in authentication. Its gRPC HTTP/2 interface lets you put an authenticating proxy in front of it.
• Streaming. The docs say the majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener.
• Scala UDFs. In standalone Scala applications, register a ClassFinder and upload JAR dependencies with SparkSession#AddArtifact.

How does Databricks use Spark Connect?

The Databricks docs say Spark Connect is used in Scala notebooks (Databricks Runtime 13.3 and above) and Python notebooks (Databricks Runtime 14.3 and above) on standard compute, on serverless compute, in Databricks Connect, and in Lakeflow Pipelines with environment versions configured.
Databricks Connect connects IDEs, notebooks and custom applications to Databricks compute. It is built on open-source Spark Connect and, for Databricks Runtime 13.3 LTS and above, extends it to support Databricks compute modes and Unity Catalog.
Standard compute does not support RDD APIs, and Databricks says to use dedicated compute when a workload needs a feature that standard compute lacks.

FAQs

Is Spark Connect the default in Apache Spark?

No. spark.api.mode defaults to classic, and it was added in Spark 4.0.0. If you do not use one of the Spark Connect mechanisms, the Spark docs say your session works just like before, without Spark Connect. The 4.0.0 release notes also list an additional release tarball with Spark Connect enabled by default.

Which languages have Spark Connect clients?

The docs say Spark Connect supports PySpark since Spark 3.4 and Scala since Spark 3.5. The Spark 4.0.0 release notes list full API compatibility for the Java client. Apache also hosts separate client repositories for Go, Swift and Rust. The Go README calls its client "highly experimental" and says it "should not be used in any production setting."