How do I connect to Spark using Spark Connect from Python?

Summary

• Spark Connect splits your Python client from the Spark server. The client sends DataFrame operations over gRPC, and the server runs them and streams results back.
• Connect with SparkSession.builder.remote("sc://host:port"), available since Spark 3.4.0. The default port is 15002.
• Install pyspark-client for a pure Python client that needs no jars or JRE. pyspark[connect] and pyspark-connect add full PySpark.
• Pass options in the connection string, such as use_ssl=true and token=.... Spark Connect has no built-in authentication, so put an authenticating proxy in front of it.
• RDDs and SparkContext are not supported. Most PySpark DataFrame APIs and the majority of the Streaming API are.

How do I connect to Spark using Spark Connect from Python?
Spark Connect is the client-server architecture in Apache Spark. The Python client turns DataFrame operations into unresolved logical plans, encodes them with protocol buffers and sends them to a Spark Connect server over gRPC. The server runs them through the standard Spark execution process and streams results back as Apache Arrow-encoded row batches. To connect from Python you need a running server, a client package and one call to SparkSession.builder.remote(). This page follows the Apache Spark 4.2.0 docs.

Which Python package do I install?

The Spark docs list three PyPI options, and Python 3.10 or later is supported.

Package	Install command	What the Spark docs say
pyspark-client	pip install pyspark-client	A pure Python library with no jars or JRE; it only supports spark.remote with connection URIs such as sc://localhost
pyspark[connect]	pip install "pyspark[connect]"	Full PySpark plus the packages Spark Connect requires, including grpcio and pyarrow
pyspark-connect	pip install pyspark-connect	Makes Spark Connect the default; installs pyspark and the Spark Connect dependencies

Full PySpark needs Java 17 or later with JAVA_HOME set. The pure Python client does not.

How do I start a server to test against?

Run the script that ships in your Spark download:

./sbin/start-connect-server.sh

The Spark docs show this server running on localhost:15002, the default port.

How do I open a Spark Connect session?

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()

remote() takes a URL such as sc://host:port. With full PySpark installed, a regular Spark session in the same process "cannot coexist" with the remote one, so stop it first:

SparkSession.builder.master("local[*]").getOrCreate().stop()

You can also set the SPARK_REMOTE environment variable (export SPARK_REMOTE="sc://localhost"), and new sessions become Spark Connect sessions with no code change. To confirm, run type(spark). For Spark Connect the class path contains .connect., for example pyspark.sql.connect.session.SparkSession.

What does a complete first script look like?

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.remote("sc://localhost:15002")
    .appName("SimpleApp")
    .getOrCreate()
)

columns = ["id", "name"]
data = [(1, "Sarah"), (2, "Maria")]
df = spark.createDataFrame(data).toDF(*columns)
df.show()

spark.stop()

df.show() prints a two-row table with the columns id and name.

How is the connection string built and secured?

The format is sc://host:port/;param1=value;param2=value. The port defaults to 15002 and use_ssl defaults to false. The connection-string reference lists these parameters:

Parameter	What it does	Example
port	Port of the gRPC endpoint	15002, 443
token	Enables standard bearer token authentication	token=ABCDEFGH
use_ssl	Connects to the endpoint using TLS	use_ssl=true
user_id	User ID set in the Spark Connect UserContext message	user_id=Martin
session_id	Session ID used as a cache key; must be a valid UUID	session_id=550e8400-e29b-41d4-a716-446655440000

A TLS connection with a bearer token looks like this in Python, using the docs' example string:

spark = SparkSession.builder.remote(
    "sc://myhost.com:443/;use_ssl=true;token=ABCDEFG"
).getOrCreate()

Spark Connect has no built-in authentication. The docs say its gRPC HTTP/2 interface allows authenticating proxies, so you can secure the server without adding authentication logic to Spark itself.

How does a Spark Connect client differ from a classic Spark application?

Difference	What the Spark docs say
Separate process	The client does not run in the same process as the Spark driver and does not use Py4J, so private fields such as df._jdf are not accessible
No RDD API	The protocol describes work as logical plans, so it does not support all of Spark's execution APIs, most importantly RDDs
Session-based client	The client has no access to the static Spark configuration or the SparkContext

Supported APIs carry a "Supports Spark Connect" label in the PySpark API reference, so check it before you migrate existing code.

Why use Spark Connect?

The docs name three operational benefits:

• Stability: an application that uses too much memory only affects its own environment, and you define your own client dependencies.
• Upgradability: the Spark driver can be upgraded independently of applications.
• Debuggability: you can debug interactively from your IDE and monitor applications with their own metrics and logging libraries.

How does Databricks Connect relate to Spark Connect?

Databricks Connect is "a client library for the Databricks Runtime" that connects IDEs, notebooks and custom applications to Databricks compute. It is built on open-source Spark Connect, and for Databricks Runtime 13.3 LTS and above it extends Spark Connect to support Databricks compute modes and Unity Catalog. Install the databricks-connect package (databricks-connect==X.Y.* to match your cluster) after uninstalling PySpark, because the two conflict.

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()

This assumes default authentication was configured during Databricks Connect setup.

FAQs

Can I use Spark Connect from languages other than Python?

Yes. The Spark docs cover PySpark (most APIs since Spark 3.4) and Scala (most APIs since Spark 3.5). Apache also hosts separate client repositories for Go, Swift and Rust. The Go README calls its client "highly experimental" and says it should not be used in any production setting.

Does Spark Connect support Structured Streaming?

The docs say the majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener.

Can I use RDDs or SparkContext?

No. Both are listed as unsupported in Spark Connect.

Do I need Java on the client?

Not with pyspark-client, which is a pure Python library. Full PySpark requires Java 17 or later.