What is dependency isolation in Spark Connect?

Summary

• Dependency isolation in Spark Connect means your application runs as a client in its own process, apart from the Spark driver. The Spark docs say users "can define their own dependencies on the client and don't need to worry about potential conflicts with the Spark driver."
• Spark Connect, introduced in Apache Spark 3.4, sends DataFrame operations to the server as unresolved logical plans over gRPC. Results come back as Apache Arrow-encoded row batches.
• The isolation covers your application code, not code that runs on the server. UDFs and their libraries run server-side, so ship them with addArtifacts in Python or AddArtifact for Scala JARs.
• Spark Connect does not support SparkContext or RDDs, and the client does not use Py4J.
• On Databricks, Databricks Connect is built on Spark Connect: application dependencies stay in your local environment, and UDFs run on Databricks compute.

What is dependency isolation in Spark Connect?

In Spark Connect, your application and the Spark driver run as separate processes. The Spark docs list this under stability: "Applications that use too much memory will now only impact their own environment as they can run in their own processes. Users can define their own dependencies on the client and don't need to worry about potential conflicts with the Spark driver." Spark Connect was introduced in Apache Spark 3.4 as "a decoupled client-server architecture."

How does Spark Connect separate the client from the server?

The client "is a thin API that can be embedded everywhere: in application servers, IDEs, notebooks, and programming languages." It translates DataFrame operations into unresolved logical plans, encodes them with protocol buffers and sends them over gRPC. The server turns them into Spark logical plans and runs normal Spark execution, then streams results back as Apache Arrow-encoded row batches.
"The client does not run in the same process as the Spark driver." In PySpark the client doesn't use Py4J, so JVM-level objects such as df._jdf are not accessible.

Piece	Where it runs	Where its dependencies live
Your application code and DataFrame calls	The client process	Your own environment
Query planning and execution	The Spark server	The server's own Spark environment
UDFs and the libraries they import	The server, when the function runs	Uploaded as artifacts or installed on the server

What does isolation not cover?

Situation	Why	What to do
A UDF needs a library the server doesn't have	UDF code runs on the server, not in your client environment	Python: spark.addArtifacts(..., pyfile=True). Scala: spark.addArtifact(<ABSOLUTE_PATH_JAR_DEP>)
Code calls SparkContext or RDD APIs	Spark Connect doesn't support them	Use the DataFrame API
Code reads df._jdf	The PySpark client has no Py4J connection to the driver JVM	Stay on public DataFrame APIs

The Scala docs add that UDFs, filters and maps also need a ClassFinder registered so the client can upload class files.

How do I set it up?

Start a server with ./sbin/start-connect-server.sh. The REPL connects to a local server on port 15002 by default. On the client, pip install pyspark-client installs "a pure Python library that does not rely on any non-Python dependencies such as jars and JRE in your environment." Then connect and ship a Python dependency:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

# Make a zipped Python package available to UDFs running on the server
spark.addArtifacts("helpers.zip", pyfile=True)

addArtifacts is new in Spark 3.5.0 and supports local files only. With pyfile=True the files are "directly inserted into the path when executing Python functions in executors." You can also set the SPARK_REMOTE environment variable, for example export SPARK_REMOTE="sc://localhost", which needs no code change.

How does Databricks approach Spark Connect?

Databricks Connect is "built on open-source Spark Connect" and extends it for Databricks compute modes and Unity Catalog. IDEs, notebooks and custom applications use it to run Spark code on Databricks compute. The docs spell out where things run: general Python and Scala code runs locally, DataFrame APIs run on Databricks compute, and UDFs defined locally are serialized and transmitted to the cluster. So you install application dependencies in your local environment and UDF dependencies on Databricks. Because UDFs are serialized, the client's Python version must match the Python version on the compute.
Declaring UDF dependencies from the client is a Public Preview feature. It needs Databricks Connect for Python 16.4 or above, a cluster on Databricks Runtime 16.4 or above, and the preview "Enhanced Python UDFs in Unity Catalog" enabled in the workspace, and the docs list it as not supported on serverless. You build a DatabricksEnv with withDependencies(...) and pass it to DatabricksSession.builder.withEnvironment(env). The dependencies are installed on Databricks compute and are available to every UDF that uses that session. They can come from PyPI or Unity Catalog volumes. Local packages, folders and Python files need version 18.1 or above of both Databricks Connect and the runtime.

FAQs

Which APIs does Spark Connect support?

Since Spark 3.4 it supports most PySpark APIs, and since Spark 3.5 most Scala APIs. SparkContext and RDD APIs are unsupported. The PySpark API reference labels supported APIs "Supports Spark Connect."

Can I try it locally?

Yes. Setting spark.remote to local[...] or local-cluster[...] starts a local Spark Connect server and gives you a Spark Connect session.

Can a misbehaving client take down the Spark driver?

The docs say client applications "no longer run on the Spark driver and, therefore don't cause and are not affected by any instability on the server."

Can I upgrade Spark without changing my application?

The docs say the Spark driver can be upgraded independently of applications. Applications stay forward-compatible as long as the server-side RPC definitions are designed to be backwards compatible.