How do I use Spark Connect with Databricks?

Summary

• On Databricks you use Spark Connect through Databricks Connect, a client library built on open-source Spark Connect. DataFrame code runs on Databricks compute while the rest of your program runs on your machine.
• Install the databricks-connect package instead of PySpark, because the two conflict, and pin it to your compute's Databricks Runtime version, for example databricks-connect==17.3.*.
• Authenticate with the Databricks CLI (databricks auth login), then create a session with DatabricksSession.builder, using a profile that holds a cluster_id or .serverless() for serverless compute.
• The workspace must have Unity Catalog enabled, and the compute's Databricks Runtime version must be greater than or equal to the Databricks Connect package version.
• Databricks Connect doesn't connect to open-source Apache Spark servers. For those, use the open-source Spark Connect client and SparkSession.builder.remote("sc://...").

How do I use Spark Connect with Databricks?

Use Databricks Connect. Databricks documents it as built on open-source Spark Connect and, for Databricks Runtime 13.3 LTS and above, as "an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog." You install one package, authenticate once with the Databricks CLI, build a DatabricksSession, and write ordinary PySpark or Scala DataFrame code. It's the Spark Connect client Databricks documents for its compute.

How Spark Connect works in open source

Spark Connect arrived in Apache Spark 3.4 as a decoupled client-server architecture. The client turns DataFrame operations into unresolved logical plans, encodes them with protocol buffers and sends them over gRPC. The server translates them into Spark's logical plan operators, runs them with the normal optimizer, and streams results back as Apache Arrow-encoded row batches.
On a self-managed cluster you start the endpoint with ./sbin/start-connect-server.sh, which binds to port 15002 by default (spark.connect.grpc.binding.port). Clients install the thin pyspark-client package and call SparkSession.builder.remote("sc://localhost").getOrCreate(). Two limits apply in open source: SparkContext and RDD APIs aren't supported, and Spark Connect has no built-in authentication, so the open-source docs suggest authenticating proxies in front of its gRPC interface.
On Databricks, Databricks Connect supplies both the client and the authentication.

Set up Databricks Connect for Python

In a virtual environment, swap PySpark for Databricks Connect and log in:

pip3 uninstall pyspark
pip3 install --upgrade "databricks-connect==17.3.*"   # use X.Y.* matching your Databricks Runtime
databricks auth login --configure-cluster --host <workspace-url>
databricks-connect test

--configure-cluster writes the cluster's cluster_id into the configuration profile, and databricks-connect test fails with a non-zero exit code if it detects an incompatibility in the setup. Then connect:

from databricks.connect import DatabricksSession

# Classic compute: the profile written by `databricks auth login --configure-cluster`
spark = DatabricksSession.builder.profile("<profile-name>").getOrCreate()

# Serverless compute instead:
# spark = DatabricksSession.builder.serverless().profile("<profile-name>").getOrCreate()

df = spark.read.table("samples.nyctaxi.trips")
df.show(5)

Choose how the session finds its settings

Databricks Connect checks, in order, DatabricksSession.builder.remote(), a configuration profile, the DATABRICKS_CONFIG_PROFILE environment variable, per-property environment variables, and finally a profile named DEFAULT. It uses the first configuration it finds.

Option	When to use it	Trade-off
DatabricksSession.builder.profile("<name>")	Local development across several workspaces	The profile name lives in your code
DATABRICKS_CONFIG_PROFILE, or DATABRICKS_HOST plus DATABRICKS_CLUSTER_ID and auth variables	CI jobs and containers	Every environment must set them
A DEFAULT profile and a bare DatabricksSession.builder.getOrCreate()	Production code, which Databricks says should build the session with no parameters	Behavior depends on the environment's configuration
DATABRICKS_SERVERLESS_COMPUTE_ID=auto, or .serverless()	No cluster to manage	Serverless limitations apply, and Scala support for serverless is in Beta
DatabricksSession.builder.remote(host=..., token=..., cluster_id=...)	Quick tests with a personal access token	Credentials in code; Databricks recommends environment variables or config files

What runs where

Your Python or Scala code runs locally, so breakpoints and debuggers work as usual. DataFrame operations become Spark plans that run on Databricks compute, and results only come back when you call something like collect(), show() or toPandas(). UDFs, foreach, foreachBatch and transformWithState run on the compute too, which is why the local Python minor version must match the compute's when you use UDFs. Databricks Connect 17.3, for example, requires Python 3.12.

How Databricks extends Spark Connect

Databricks Connect adds what open-source Spark Connect leaves to you: workspace authentication through the Databricks CLI and SDK (OAuth user-to-machine, OAuth machine-to-machine, or personal access tokens), Unity Catalog support, and a choice of classic clusters or serverless compute (supported from Databricks Connect 15.1). Clients exist for Python, Scala and R, and the Databricks IDE extension uses Databricks Connect for debugging. Classic clusters must use the Standard or Dedicated access mode. See Databricks Connect.

FAQs

Should I install PySpark or Databricks Connect?

Databricks Connect. The databricks-connect package conflicts with PySpark, so uninstall PySpark first. Databricks Connect gives you the PySpark DataFrame API against Databricks compute.

Can Databricks Connect talk to an open-source Spark Connect server?

No. Databricks documents that Databricks Connect does not support connecting to open-source Apache Spark servers. Use the open-source Spark Connect client for those.

Which Databricks Connect version should I install?

One that matches your compute. The Databricks Runtime version must be greater than or equal to the package version, and Databricks recommends the latest package that matches your runtime, pinned with X.Y.*.