How do I fix classpath or dependency issues with Spark Connect?

Summary

• Spark Connect separates the client from the Spark server. Code the server runs for you (Python and Scala UDFs, map and filter closures) needs its modules and classes on the server, not only on the client.
• Python: ship pure-Python modules with spark.addArtifacts(path, pyfile=True), and install third-party packages in the Python environment the executors use.
• Scala and Java: upload JARs with spark.addArtifact(...), register a ClassFinder such as REPLClassDirMonitor for compiled classes, and set server-wide JARs with --jars or --packages when you start the server.
• On Databricks, Databricks Connect is built on Spark Connect and documents its own UDF dependency APIs: addCompiledArtifacts() for Scala, and DatabricksEnv().withDependencies() (Public Preview) for Python.

How do I fix classpath or dependency issues with Spark Connect?

Spark Connect splits your application from the Spark server. The Apache Spark docs say the client "does not run in the same process as the Spark driver": it turns DataFrame operations into unresolved logical plans and sends them to the server over gRPC. Libraries you import in client code stay on the client. Anything the server has to run for you, such as a UDF, map or filter, needs its modules or classes on the server. Find out which side is missing the dependency, then apply the matching fix.

Which side is missing the dependency?

Symptom	Likely cause	Fix
ModuleNotFoundError inside a Python UDF, although the import works on the client	The module is not in the Python environment of the executors	Pure-Python code: spark.addArtifacts(path, pyfile=True). Third-party packages: install them in the executors' environment
ClassNotFoundException or NoClassDefFoundError when a Scala UDF, map or filter runs	The classes or JARs the closure uses never reached the server	registerClassFinder for compiled classes, addArtifact for JARs, or --jars / --packages at server start
NoSuchMethodError	A class on the server has a different definition from the one your code was built against	Build against the library version the server has
spark.sparkContext.addPyFile(...) is unavailable	Spark Connect clients have no SparkContext	Use addArtifacts

Read the full exception first. The server config spark.sql.connect.serverStacktrace.enabled defaults to true, which "sets the server-side stacktrace in the user-facing Spark exception".

How do I ship Python dependencies?

SparkSession.addArtifacts(*path, pyfile=False, archive=False, file=False) adds artifacts to the client session ("Currently only local files are supported"). With pyfile=True, ".py, .egg, .zip or .jar files" become Python dependencies that "are directly inserted into the path when executing Python functions in executors". archive=True unpacks archives on the executor side, and file=True downloads a file to every node. addArtifact has the same signature.

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()

# helpers.zip is a zipped Python package; the path is on the client
spark.addArtifacts("/path/to/helpers.zip", pyfile=True)

@udf
def clean(s):
    from helpers import normalize  # imported where the UDF runs
    return normalize(s)

df = spark.createDataFrame([(" a ",)], ("name",))
df.select(clean("name")).show()

addArtifacts is not pip install. Wheels are not among the documented file types, and the Python Package Management guide says of SparkContext.addPyFile() that it "does not allow to add packages built as Wheels and therefore does not allow to include dependencies with native code". For packages such as pandas or PyArrow, make them available on the executors with one of the approaches in that guide (Conda, Virtualenv, PEX or uv).

How do I ship JVM dependencies?

For Scala, the Spark Connect overview says operations "that reference User Defined Code such as UDFs, filter, map, etc require a ClassFinder to be registered to pickup and upload any required classfiles", and that "any JAR dependencies must be uploaded to the server using SparkSession#AddArtifact".

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.connect.client.REPLClassDirMonitor

val spark = SparkSession.builder().remote("sc://localhost").getOrCreate()

// Upload class files from the build output directory
spark.registerClassFinder(new REPLClassDirMonitor("/abs/path/to/classes"))

// Upload a JAR dependency from the client's file system
spark.addArtifact("/abs/path/to/dependency.jar")

The Scala API docs mark these methods experimental. addArtifact(path: String) takes local .jar and .class files, and addArtifact(uri: URI) and addArtifacts(uri: URI*) also accept Apache Ivy URIs.
To make JARs available to every session, set them when you start the server. ./sbin/start-connect-server.sh takes spark-submit options. --jars is a comma-separated list (directory expansion does not work), and --packages takes Maven coordinates and handles transitive dependencies. The client "does not have access to the static Spark configuration", and the configuration guide says server configurations are set "when you start the Spark Connect server". Put server-wide classpath settings in the launch command, not in client code.

How does Databricks handle Spark Connect dependencies?

Databricks Connect "is built on open-source Spark Connect", and "UDF code runs on Databricks compute". Install application dependencies on your local machine and UDF dependencies on Databricks.

• Python UDFs: DatabricksEnv().withDependencies("dice==3.1.0"), passed to DatabricksSession.builder.withEnvironment(env).getOrCreate(), installs the packages "on Databricks compute as part of the UDF's Python environment". This is a Public Preview feature. It needs Databricks Connect for Python 16.4 or above, Databricks Runtime 16.4 or above, and the Enhanced Python UDFs in Unity Catalog preview enabled.
• Scala UDFs: "the compiled classes and JARs must be uploaded to the cluster using the addCompiledArtifacts() API". Later uploads use spark.addArtifact(), including ivy:// coordinates for Maven libraries, and "all transitive dependency JARs must be included for upload".

FAQs

Which client libraries do I install?

For Python, pip install pyspark-client (the overview pins a version, such as pyspark-client==4.2.0). For Scala, add "org.apache.spark" %% "spark-connect-client-jvm" to build.sbt.

Does the client version have to match the server?

The overview says applications can be forward-compatible "as long as the server-side RPC definitions are designed to be backwards compatible". UDFs add constraints the Databricks docs spell out: the client's Python version must match the compute's Python version, and the client's Scala version must match the cluster's.

Does the artifact API work the same on Databricks?

The Databricks PySpark reference documents the same addArtifacts signature and notes that addArtifact is an alias. Databricks Connect for Scala also accepts spark.addArtifact().