How do I run UDFs over Spark Connect?

Summary

• Spark Connect supports UDFs. The Spark docs say they work by default in the shell, and in standalone applications with additional set-up.
• In a Scala application, register a ClassFinder (for example REPLClassDirMonitor) so the client uploads your compiled classfiles, and upload JAR dependencies with addArtifact.
• PySpark udf and pandas_udf support Spark Connect since Spark 3.4. The client serializes the function when the query runs, and the libraries it imports must be available on the executors.
• A variable captured by a Python UDF is read when you run the query, not when you define the UDF.

How do I run UDFs over Spark Connect?

Spark Connect supports user-defined functions. The Spark docs say UDFs are "supported, by default for the shell and in standalone applications with additional set-up requirements." The UDF code has to reach the server that runs it, and how that happens depends on the client.

Client	What you do	Why
Scala shell (spark-shell --remote)	Nothing extra	The Connect REPL registers its own ClassFinder
Scala standalone application	Register a ClassFinder, upload JARs with addArtifact	The client must find and upload your classfiles
PySpark	Define the UDF with udf or pandas_udf	The client serializes the function and sends it to the server

How do I register a ClassFinder in a Scala application?

The docs say operations that reference user-defined code, such as UDFs, filter and map, require a ClassFinder so the client can pick up and upload the classfiles. REPLClassDirMonitor is the provided implementation. It watches one directory, and its source requires an absolute path to an existing directory.

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.connect.client.REPLClassDirMonitor
import org.apache.spark.sql.functions.{col, udf}

val spark = SparkSession.builder().remote("sc://localhost").getOrCreate()

// Register a ClassFinder to monitor and upload the classfiles from the build output.
spark.registerClassFinder(new REPLClassDirMonitor("/absolute/path/to/build/output"))

// Upload JAR dependencies.
spark.addArtifact("/absolute/path/to/dependency.jar")

val squared = udf((x: Long) => x * x)
spark.range(3).withColumn("squared", squared(col("id"))).select("squared").show()

You can write your own class extending ClassFinder for customized search and monitoring. In the Spark source, registerClassFinder and addArtifact are annotated @Experimental.

How do I run Python UDFs?

The udf and pandas_udf docs both say "Supports Spark Connect" from version 3.4.0.

from pyspark.sql import SparkSession
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

@udf(returnType=IntegerType())
def add_one(x):
    if x is not None:
        return x + 1

df = spark.createDataFrame([(1, "John Doe", 21)], ("id", "name", "age"))
df.select(add_one("age")).show()

A Python UDF still needs its dependencies on the cluster. PySpark's packaging guide says your code and all libraries it uses must be available on the executors, and a Pandas UDF can raise ModuleNotFoundError: No module named 'pyarrow' if pyarrow is missing there. SparkSession.addArtifacts(*path, pyfile=False, archive=False, file=False) (new in 3.5.0) adds local files to the client session. With pyfile=True, .py, .egg, .zip or .jar files are inserted into the path when Python functions run on executors.

When does Spark Connect serialize the UDF?

Python UDFs are lazy in Spark Connect: serialization and registration are deferred until execution time. A UDF that reads a Python variable sees the value at the moment you run an action.

from pyspark.sql.functions import udf

x = 123

@udf("INT")
def foo():
    return x

df = spark.range(1).select(foo())
x = 456
df.show()  # Prints 456

Spark Classic captures the value when the UDF is created. To bind the value early, wrap the UDF definition in a function that takes the value as an argument.

Which mistakes come up most?

Mistake	What happens	Fix
Passing a relative path or a missing directory to REPLClassDirMonitor	The constructor's require checks fail	Pass an absolute path to an existing directory
Creating UDFs in a loop that reuse the loop variable	Every UDF sees the last value, because Python closures capture by reference and serialization is deferred	Create each UDF inside a helper such as make_udf(value)
Leaving a library out of the upload	A ClassNotFoundException is thrown on the server side during UDF execution	Upload the JAR with addArtifact
Relying on a package that is missing on the executors	Python can raise ModuleNotFoundError	Install it there or add it with addArtifacts

What does Spark Connect not support?

RDDs, SparkContext and df._jdf are not available, so code that depends on them has to move to the DataFrame API. The Spark 4.2.0 release notes list DataFrame additions under "RDD API compatibility" (SPARK-55227), such as DataFrame.zipWithIndex.

How does Databricks handle UDFs over Spark Connect?

Databricks Connect is built on open-source Spark Connect. Its docs say UDFs defined locally "are serialized and transmitted to the cluster where it runs." For Python, the client's Python version must match the Python version on the Databricks compute. For Scala, upload compiled classes and JARs with addCompiledArtifacts(), keep the client's Scala version equal to the cluster's, and include every transitive dependency JAR, because the upload APIs do not detect them automatically.

FAQs

Can my UDF use external libraries?

Yes. In Scala, spark.addArtifact uploads JARs, and Spark 4.0 added ivy URI support to addArtifact (SPARK-45506). In Python, install the packages on the executors or ship them with addArtifacts.

What happens if my UDF fails on the server?

The error reaches the client. Since Spark 4.0, spark.sql.connect.enrichError.enabled (default true) adds the full exception message and optionally the server-side stack trace. spark.sql.connect.serverStacktrace.enabled (default true) puts the server-side stack trace in the Spark exception you see.

Do Pandas UDFs work over Spark Connect?

Yes. The pandas_udf docs list "Supports Spark Connect" since 3.4.0.