Why do I get a gRPC connection error with Spark Connect?

Summary

• A gRPC connection error means the Spark Connect client couldn't open a working channel to a server. A common cause is that nothing is listening at the host and port in the sc:// connection string, whose port defaults to 15002.
• Read the gRPC status first: UNAVAILABLE points at reachability (no server, wrong address, dropped connection), RESOURCE_EXHAUSTED at a message over the size limit, UNAUTHENTICATED at rejected credentials, and UNIMPLEMENTED at a call the server doesn't have.
• Open-source Spark Connect has no built-in authentication. Adding token= enables SSL unless the host is localhost, so a token aimed at a plaintext server elsewhere breaks the connection before any query runs.
• Check three things in order: the server is running and bound where you expect (spark.connect.grpc.binding.port, spark.connect.grpc.binding.address), the client points there (SPARK_REMOTE or .remote()), and the client package matches the server's Spark version.
• On Databricks, Databricks Connect builds on Spark Connect. Its troubleshooting guide ties StatusCode.UNAVAILABLE to "Databricks Connect cannot reach your cluster" and says to uninstall pyspark before installing databricks-connect.

Why do I get a gRPC connection error with Spark Connect?

Because the client couldn't open a working gRPC channel to a Spark Connect server. Every DataFrame operation travels as a gRPC call, so the failure shows up as a gRPC error when the server isn't running, the sc:// connection string points at the wrong host or port, client and server disagree about TLS, or a message is too large. The status code in the error tells you which one, so read it before you change anything.

How the client reaches the server

Per the Spark docs, the client "translates DataFrame operations into unresolved logical query plans which are encoded using protocol buffers. These are sent to the server using the gRPC framework," and results stream back as Apache Arrow-encoded row batches. The client finds the server through a connection string shaped like sc://host:port/;param=value. The port defaults to 15002, and the Spark 4.2 parameters are token, use_ssl, user_id, user_agent, session_id and grpc_max_message_size. Since Spark 4.1, the Python client's gRPC exception also carries the gRPC status code (SPARK-51774).

Symptoms, causes and fixes

Symptom	Likely cause	Fix
UNAVAILABLE on the first query	Nothing listening at that host and port	Start the server with ./sbin/start-connect-server.sh; confirm spark.connect.grpc.binding.port (default 15002) matches the client's port
Works on the server host, fails from your laptop	Server bound to an address the client can't reach, or a firewall in between	Set spark.connect.grpc.binding.address, open the port, and test from the client machine
A restarted server won't come up	Port already in use; spark.connect.grpc.port.maxRetries defaults to 0, so the server fails fast on a port conflict	Stop the old server or pick another port on both sides
Breaks as soon as you add token=	Setting token enables SSL unless the host is localhost, and the open-source server has no built-in authentication	Drop the token for a plaintext server, or reach the server through an authenticating proxy that terminates TLS
UNAUTHENTICATED	The proxy or gateway rejected the credentials	Fix the token or the proxy's auth settings
RESOURCE_EXHAUSTED on big plans or local data	A message exceeded the configured size limit	Raise grpc_max_message_size (Scala client; PySpark sets gRPC channel options instead) or spark.connect.grpc.maxInboundMessageSize (default 134217728) on the server, or write large data to storage and read it server-side
UNIMPLEMENTED	The client called an RPC the server doesn't have	Match the client package to the server's Spark version
UNAVAILABLE mid-query after it worked	Connection dropped, server shut down, or a keepalive timed out	Check the server log and any proxy or load balancer between you

These mappings come from gRPC's own status table, which lists "Server shutting down" and "Keepalive watchdog times out" as UNAVAILABLE and "Sent or received message was larger than configured limit" as RESOURCE_EXHAUSTED.

A minimal working setup to test against

When in doubt, reproduce against a known-good local server, then change one variable at a time. Start the server from the Spark 4.2 distribution:

./sbin/start-connect-server.sh
export SPARK_REMOTE="sc://localhost"

Then connect from a client whose version matches the server, for example pip install pyspark-client==4.2.0, and force a round trip:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
print(spark.range(10).count())  # an action, so it must reach the server

If this works and your real setup doesn't, the difference is the address, the network path, TLS, or the version. pyspark-client is a pure Python client that "only supports spark.remote with connection URIs"; with the full PySpark package, pip install "pyspark[connect]" adds the gRPC dependencies, including grpcio and grpcio-status.
For remote servers that need security, put an authenticating proxy in front of the gRPC endpoint, as the docs suggest, and point the client at it with use_ssl=true or token=....

How Databricks approaches Spark Connect connectivity

Databricks Connect "is built on open-source Spark Connect," using unresolved logical plans and Apache Arrow on top of gRPC, with a Databricks cluster or serverless compute as the server. Its troubleshooting guide lists StatusCode.UNAVAILABLE, StatusCode.UNKNOWN, "DNS resolution failed" and "Received http2 header with status 500" under one cause: Databricks Connect cannot reach your cluster. Check the workspace instance name, the cluster ID and any environment variables, and confirm the cluster runs a version compatible with your Databricks Connect release. The guide also says to uninstall pyspark before installing databricks-connect, and warns that pointing Databricks Connect at an open-source Spark Connect server produces errors such as class-not-found or serialization failures.

FAQs

What port does Spark Connect use?

15002 by default on both sides. The connection string's port defaults to 15002, and the server binds spark.connect.grpc.binding.port, whose default is also 15002.

Does open-source Spark Connect encrypt or authenticate traffic?

Not by itself. The docs say it "does not have built-in authentication" and rely on authenticating proxies in front of the gRPC endpoint; the client's use_ssl and token parameters exist to reach such an endpoint.