How do I use Spark Connect from a notebook or IDE?

Summary

• Spark Connect separates your client (a notebook, IDE or script) from the Spark server. The client sends unresolved logical plans over gRPC and gets Arrow-encoded results back.
• For open-source Spark, start a server with ./sbin/start-connect-server.sh, install pyspark-client in the notebook or IDE environment, and call SparkSession.builder.remote("sc://localhost:15002").getOrCreate().
• You can instead set the SPARK_REMOTE environment variable, so existing code needs no change.
• On Databricks, use Databricks Connect (databricks-connect and DatabricksSession). It does not connect to open-source Spark servers.
• RDDs and SparkContext are not available through Spark Connect.

How do I use Spark Connect from a notebook or IDE?

Run a Spark Connect server, then create a remote session from your notebook or IDE. The Spark docs call the client "a thin API that can be embedded everywhere: in application servers, IDEs, notebooks, and programming languages". It turns DataFrame operations into unresolved logical plans, sends them to the server over gRPC, and receives results as Apache Arrow-encoded row batches.

# On the server, from an extracted Spark download
./sbin/start-connect-server.sh

# In the notebook or IDE Python environment
pip install pyspark-client==4.2.0

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()

df = spark.createDataFrame([("Alice", 25), ("Bob", 30)], ["name", "age"])
df.show()

The server script listens on port 15002 by default (spark.connect.grpc.binding.port, since Spark 3.4.0).

Which Python package do I install?

Package	What the PySpark install docs say
pyspark-client	"a pure Python library that does not rely on any non-Python dependencies such as jars and JRE". It supports only spark.remote with connection URIs such as sc://localhost
pyspark[connect]	The Spark Connect extra of the full pyspark package
pyspark-connect	Makes Spark Connect the default and installs pyspark plus the Spark Connect dependencies

PySpark 4.2.0 requires Python 3.10 or above.

How do I connect from a script, Jupyter notebook or IDE?

Any Python process that can import the client and reach the server can open a session. The docs show three ways:

• Pass the URL in code, as above.
• Set export SPARK_REMOTE="sc://localhost" before starting Python or ./bin/pyspark. "With this approach, there is no code change needed to start using Spark Connect."
• Launch the shell with ./bin/pyspark --remote "sc://localhost".

Check the session type: if type(spark) includes .connect., "you are using Spark Connect". If a notebook already started a classic session, stop it first, because you must "stop the existing regular Spark session because it cannot coexist with the remote Spark Connect session". In an IDE, run the file with the regular Python interpreter. Spark Connect "enables interactive debugging during development directly from your favorite IDE".
Analysis is deferred, so a typo in a column name raises no error until the plan is analyzed. The Spark docs say you can trigger analysis with df.columns, df.schema or df.collect().

How do I secure the connection?

Spark Connect "does not have built-in authentication", and its "gRPC HTTP/2 interface allows for the use of authenticating proxies". The connection string format is sc://host:port/;param1=value;param2=value. The token parameter enables "standard bearer token authentication using GRPC" and "Setting this value enables SSL". The use_ssl flag connects with TLS and defaults to false. The overview's example is export SPARK_REMOTE="sc://myhost.com:443/;token=ABCDEFG".

Which APIs work through a remote session?

API	Through Spark Connect
DataFrame, Functions, Column	"most PySpark APIs" since Spark 3.4
Structured Streaming	"Majority of the Streaming API"
UDFs	Supported, with "additional set-up requirements" in standalone applications
Spark ML	ML on Spark Connect arrived in 4.0.0, and 4.1.0 made it GA for the Python client
pyspark shell	Works: ./bin/pyspark --remote "sc://localhost"
SparkContext and RDDs	Unsupported

How do I do this on Databricks?

Use Databricks Connect, which "is built on open-source Spark Connect". For Databricks Runtime 13.3 LTS and above it is "an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog". It lets you connect to Databricks compute "from IDEs such as Visual Studio Code, PyCharm, and IntelliJ IDEA, notebooks and any custom application", and it "does not support connecting to open source Apache Spark servers".

databricks auth login --host <workspace-url>
pip3 uninstall pyspark
pip3 install --upgrade "databricks-connect==17.3.*"

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()
df = spark.read.table("samples.nyctaxi.trips")
df.show(5)

Uninstall PySpark first, because the databricks-connect package "conflicts with PySpark". Use X.Y.* to match your cluster version. Pick the target compute with serverless_compute_id = auto in the DEFAULT profile, or the environment variables DATABRICKS_SERVERLESS_COMPUTE_ID=auto or DATABRICKS_CLUSTER_ID=<your_cluster_id>. The Databricks Runtime version of your compute must be greater than or equal to the Databricks Connect package version, and Unity Catalog must be enabled.
Python code runs locally and DataFrame operations run on Databricks compute, so results reach your client when you call collect(), show() or toPandas(). These are not available: the dataframe.display() API, Spark Context and RDDs. Inside a Databricks notebook the spark variable is already defined, and the docs say "Don't create a SparkSession, SparkContext, or SQLContext". Python notebooks on Databricks Runtime 14.3 and above on standard compute, and serverless compute, use Spark Connect.

FAQs

Do I need Java on the notebook machine?

Not for pyspark-client, which has no jar or JRE dependency.

Can I use Scala?

Yes. The overview shows the sbt dependency "org.apache.spark" %% "spark-connect-client-jvm" % "4.2.0" and SparkSession.builder().remote("sc://localhost").getOrCreate(). UDFs need a registered ClassFinder, and JAR dependencies go through addArtifact.

Does my existing PySpark code change?

Once created, the remote session "can be used the same way as a regular Spark session", except for the unsupported APIs.