How do I package an app that uses a Spark Connect client?

Summary

• Package the thin client, not a Spark distribution. For Python, the docs describe pyspark-client as a pure Python library that needs no jars or JRE. For Scala, the docs add spark-connect-client-jvm to the build.
• The app finds the server through a connection string, sc://host:port/;param=value, set with SparkSession.builder.remote(...) or the SPARK_REMOTE environment variable. The default port is 15002.
• Spark Connect has no built-in authentication. The connection string accepts a token (bearer token, which enables SSL) and use_ssl=true, and the docs point to authenticating proxies.
• Code that runs on the server needs its dependencies there: SparkSession#AddArtifact for JARs in Scala and addArtifacts for Python files.
• On Databricks, Databricks Connect is the client library, built on open-source Spark Connect.

How do I package an app that uses a Spark Connect client?

Package the thin client and the connection settings, not a Spark distribution. The Spark docs describe the Spark Connect client library as "a thin API that can be embedded everywhere: in application servers, IDEs, notebooks, and programming languages", and say client applications "can be submitted in the same way as any previous job". The Spark server runs elsewhere and the two talk over gRPC.

Which client package do I install?

Language	Package	What the docs say
Python	pyspark-client	A pure Python library with no jars or JRE. It supports only spark.remote with connection URIs
Python	pyspark[connect]	PySpark plus the packages Spark Connect needs, such as pyarrow and grpcio
Python	pyspark-connect	Makes Spark Connect the default and installs pyspark. It also supports local clusters such as local[*]
Scala	spark-connect-client-jvm	The sbt dependency the docs show for Spark Connect applications

PySpark 4.2 supports Python 3.10 and above. For a packaged Python application or library, the docs pin the client in setup.py:

install_requires=[
    'pyspark-client==4.2.0'
]

For Scala, the docs add this line to build.sbt:

libraryDependencies += "org.apache.spark" %% "spark-connect-client-jvm" % "4.2.0"

The Spark 4.0.0 release notes describe the Python client as lightweight, "at just 1.5 MB". Because the client and server are decoupled, the docs say the Spark driver can be upgraded independently of applications, and applications can be forward-compatible as long as the server-side RPC definitions are designed to be backwards compatible.

How does the app find the server?

Set the connection in the builder, or set SPARK_REMOTE on the machine that runs the app. The docs' example app uses the builder:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").appName("SimpleApp").getOrCreate()
logData = spark.read.text("YOUR_SPARK_HOME/README.md")
print(logData.filter(logData.value.contains('a')).count())
spark.stop()

How do I secure the connection?

The docs state that Spark Connect has no built-in authentication but works with existing authentication infrastructure, because its gRPC HTTP/2 interface allows authenticating proxies. The connection string carries the client-side options:

Part	Effect
sc://host:port/	The path must be empty. The default port is 15002
token=ABCDEFG	Bearer token authentication over gRPC. Setting it enables SSL
use_ssl=true	Connect with TLS. The certificates that verify the server must be available in the system. The default is false

For example, the docs show sc://myhost.com:443/;use_ssl=true;token=ABCDEFG. Keep that string in SPARK_REMOTE or your own configuration instead of hard-coding it in the package.

What does the server need from my app?

Scala operations that reference user-defined code, such as UDFs, filter and map, need a registered ClassFinder, and JAR dependencies are uploaded with SparkSession#AddArtifact:

import org.apache.spark.sql.connect.client.REPLClassDirMonitor
val classFinder = new REPLClassDirMonitor(<ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR>)
spark.registerClassFinder(classFinder)
spark.addArtifact(<ABSOLUTE_PATH_JAR_DEP>)

In Python, SparkSession.addArtifacts(*path, pyfile=False, archive=False, file=False) adds local files to the client session. With pyfile=True they are Python dependencies such as .py, .egg, .zip or .jar files.

What mistakes break a packaged client?

Mistake	What happens
Setting client options such as spark.connect.token	The configuration reference lists Spark Connect server settings. Client options go in the connection string
Setting a master and a Connect URL together	PySpark raises CANNOT_CONFIGURE_SPARK_CONNECT_MASTER: "Spark Connect server and Spark master cannot be configured together"
Using pyspark-client for local tests	It supports only connection URIs. Use pyspark-connect with spark.remote set to local[*]
Sending large requests	The server setting spark.connect.grpc.maxInboundMessageSize defaults to 134217728 bytes, and requests with a larger payload fail
Calling collect() on large results	The API docs say to use it only when the result is expected to be small

How does Databricks fit?

Databricks Connect is a client library for the Databricks Runtime, built on open-source Spark Connect. For Databricks Runtime 13.3 LTS and above, it is an extension of Spark Connect with additions for Databricks compute modes and Unity Catalog. Install application dependencies on your local machine and UDF dependencies on Databricks.

FAQs

Which languages have Spark Connect clients?

The Spark docs say Spark Connect supports PySpark and Scala applications. Separate Apache repositories host clients for Go, Swift and Rust. The Go README calls its client "highly experimental" and says it should not be used in any production setting, so read each README before you depend on one.