How do I run a Spark Connect server locally?

Summary

• Download a pre-built Apache Spark package, extract it, and run ./sbin/start-connect-server.sh from the Spark folder. ./sbin/stop-connect-server.sh stops the server again.
• Spark runs on Java 17, 21 or 25, so you need Java on your PATH or JAVA_HOME pointing to a Java installation.
• The server listens on port 15002 by default (spark.connect.grpc.binding.port). Clients reach it with the connection string sc://localhost.
• Connect a shell with ./bin/pyspark --remote "sc://localhost" or ./bin/spark-shell --remote "sc://localhost", or export SPARK_REMOTE="sc://localhost" once and start the shell as usual.
• Spark Connect has no built-in authentication. A local server is fine for development, but anything other people can reach needs an authenticating proxy in front of its gRPC endpoint.

How do I run a Spark Connect server locally?

Download Apache Spark, extract it, and run ./sbin/start-connect-server.sh from the extracted folder. The server listens on port 15002, and any Spark Connect client can reach it at sc://localhost. When you're done, ./sbin/stop-connect-server.sh stops it. A local server is the quickest way to try thin-client code before you point it at a shared cluster.

What do you need before you start?

Spark "runs on Java 17/21/25, Scala 2.13, Python 3.10+," and running it on one machine needs Java on your system PATH or a JAVA_HOME variable that points to a Java installation. On the downloads page, choose the latest release and a package type, typically "Pre-built for Apache Hadoop 3.5 and later". Then extract it and start the server:

tar -xvf spark-4.2.0-bin-hadoop3.tgz
# change into the extracted Spark folder, then:
./sbin/start-connect-server.sh

The script passes extra options through to spark-submit, so server settings go in conf/spark-defaults.conf or on the command line with --conf. Pass --wait as the first argument to keep the server in the foreground instead of running it as a background daemon.

How do you connect a client to the local server?

Option	When to use it	Example
SPARK_REMOTE environment variable	No code change: new sessions become Spark Connect sessions	export SPARK_REMOTE="sc://localhost", then ./bin/pyspark
--remote shell flag	A one-off PySpark or Scala shell	./bin/pyspark --remote "sc://localhost"
SparkSession.builder.remote()	Python application code	SparkSession.builder.remote("sc://localhost").getOrCreate()
SparkSession.builder().remote()	Scala code built with spark-connect-client-jvm	SparkSession.builder().remote("sc://localhost").getOrCreate()

The PySpark shell confirms the link in its welcome message, "Client connected to the Spark Connect server at localhost", and type(spark) shows pyspark.sql.connect.session.SparkSession. The Scala shell is an Ammonite-based REPL that tries a local server on port 15002 by default.
For a standalone Python app, install the client with pip install pyspark-client==4.2.0 and connect in code:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()
columns = ["id", "name"]
data = [(1, "Sarah"), (2, "Maria")]
df = spark.createDataFrame(data).toDF(*columns)
df.show()
spark.stop()

Which Python package should you install?

Package	What the PySpark docs say	Use it when
pyspark-client	"a pure Python library that does not rely on any non-Python dependencies such as jars and JRE"; supports only spark.remote with connection URIs	Your app only talks to a server over sc://
pyspark-connect	Installs pyspark plus the Spark Connect dependencies and makes Spark Connect the default	You want --master or --remote (including local[*]) to use Spark Connect
pyspark[connect]	Full PySpark with the extras Spark Connect requires, such as pandas, pyarrow and grpcio	You want the full PySpark package plus a Spark Connect client

How do you change the port or the connection settings?

On the server, spark.connect.grpc.binding.port sets the port (default 15002), spark.connect.grpc.binding.address sets the bind address (no default value), and spark.connect.grpc.port.maxRetries defaults to 0, which means "the server will fail fast in case of port conflicts." For example, ./sbin/start-connect-server.sh --conf spark.connect.grpc.binding.port=<port> moves the server to another port.
On the client, the full form is sc://host:port/;param1=value;param2=value. The scheme is fixed to sc://, "the path component must be empty," and parameter names are case sensitive. Setting token enables "standard bearer token authentication using GRPC," and "setting this value enables SSL." use_ssl defaults to false, and session_id takes a UUID string.

How Databricks approaches Spark Connect

On Databricks you don't start a server yourself. Databricks Connect "is built on open-source Spark Connect" and connects IDEs, notebooks and custom applications to Databricks compute. For Databricks Runtime 13.3 LTS and above, it's "an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog." Uninstall PySpark first, because "the databricks-connect package conflicts with PySpark," then run pip3 install --upgrade "databricks-connect==17.3.*", matching the version to your cluster. In Python, DatabricksSession.builder.serverless().getOrCreate() targets serverless compute and DatabricksSession.builder.profile("<profile-name>").getOrCreate() uses a configuration profile. databricks-connect test checks the setup.

FAQs

What if the port is already in use?

spark.connect.grpc.port.maxRetries defaults to 0, so the server fails fast on a port conflict rather than trying another port. Free the port, or start the server with --conf spark.connect.grpc.binding.port=<port> and use that port in your connection string.

Does Structured Streaming work over Spark Connect?

Mostly. SparkSession.readStream and DataFrame.writeStream have been labeled "Supports Spark Connect" since PySpark 3.5.0, and the overview lists DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener among the supported Scala APIs.

Can I run RDD code against the server?

No. The Spark Connect protocol "does not support all the execution APIs of Spark, most importantly RDDs," and the client has no access to the SparkContext.

Is a local Spark Connect server secure?

Spark Connect "does not have built-in authentication." Its "gRPC HTTP/2 interface allows for the use of authenticating proxies," so put one in front of any server that other people can reach.