How do I authenticate a Spark Connect session?

Summary

• Apache Spark ships Spark Connect without built-in authentication. The docs point to authenticating proxies, so a proxy in front of the server usually does the checking.
• On the client, credentials go in the connection string: sc://host:port/;token=<value> enables standard bearer token authentication using gRPC, and setting a token also enables SSL (except for localhost).
• use_ssl=true connects over TLS using certificates available on the system (the default is false). user_id sets the user in the session's UserContext for session management; it is not a credential.
• The same connection string works in SparkSession.builder.remote(), in the SPARK_REMOTE environment variable and with the shells' --remote option.
• On Databricks, Databricks Connect (built on Spark Connect) authenticates with OAuth user-to-machine, OAuth machine-to-machine or a personal access token, read from a configuration profile or environment variables.

How do I authenticate a Spark Connect session?

You authenticate a Spark Connect session in two places. The client sends a bearer token inside its connection string, for example sc://spark.example.com:443/;use_ssl=true;token=<token>. Something on the server side then has to verify that token, and open-source Apache Spark doesn't do it by default: the Spark Connect docs state that it doesn't have built-in authentication and that its gRPC HTTP/2 interface allows the use of authenticating proxies. Keep the Spark Connect port off shared networks, put a token-checking proxy in front of it, and connect through that proxy over TLS.

What gets authenticated

With Spark Connect, your application never hosts the Spark driver. It ships each DataFrame query to the server as an unresolved logical plan, serialized with protocol buffers over a gRPC channel, and the server's Spark Connect endpoint converts that plan into Spark's logical plan operators. Authentication therefore controls who may open the channel and submit plans.
The client side is configured entirely through the connection string. Spark's connection string reference defines the parameters below. They follow /; in the URI, use the HTTP URL path parameter style and are case sensitive.

Parameter	What it does	Default
token	Enables standard bearer token authentication using gRPC. Setting it also enables SSL, except for localhost.	Not set
use_ssl	Connects using TLS, assuming the certificates needed to verify the server are available on the system.	false
user_id	Sets the user ID in the Spark Connect UserContext message, used for session management.	Optional (Python client: SPARK_USER, else USER)
user_agent	Names the application acting on behalf of the user.	_SPARK_CONNECT_PYTHON in the Python client
session_id	Lets clients share a server-side session; the value must be a valid UUID string.	A random UUID

The port goes in the host:port part of the URI and defaults to 15002. Treat user_id as a label: the reference describes it for session management, and the client sets it freely, so it proves nothing about identity.

Step 1: send the token from the client

from pyspark.sql import SparkSession

# Load the secret at runtime; don't hard-code it.
with open("/run/secrets/spark_connect_token") as f:
    token = f.read().strip()

spark = (
    SparkSession.builder
    .remote(f"sc://spark.example.com:443/;use_ssl=true;token={token}")
    .getOrCreate()
)
print(spark.range(5).count())

Shells and existing applications can read the same string from the SPARK_REMOTE environment variable, and the Spark docs note that this approach needs no code change. The Scala client accepts the string in SparkSession.builder.remote(...).

export SPARK_REMOTE="sc://spark.example.com:443/;use_ssl=true;token=$(cat /run/secrets/spark_connect_token)"
./bin/pyspark

Step 2: enforce the token on the server side

Because setting token switches the client to TLS, any endpoint other than localhost must present a certificate the client can verify. Use the proxy or the interceptor to verify identity, and bind the server to a private address either way.

Approach	When to use it	Trade-off
Authenticating proxy in front of the server	Any server reachable by more than one person or host	You operate the proxy, and it has to forward gRPC over HTTP/2
Custom gRPC interceptor set in spark.connect.grpc.interceptor.classes	You want the check inside the Spark Connect server	Custom JVM code; each class must implement io.grpc.ServerInterceptor
Private bind address via spark.connect.grpc.binding.address	Local development or a locked-down network segment	Limits who can reach the port but identifies no one

Supply them as Spark configuration when you launch ./sbin/start-connect-server.sh. The listening port comes from spark.connect.grpc.binding.port, which defaults to 15002.

How Databricks approaches Spark Connect authentication

Databricks Connect sits on top of open-source Spark Connect, and from Databricks Runtime 13.3 LTS onward it adds changes so the protocol works with Databricks compute modes and Unity Catalog. It normally uses Databricks authentication, not a hand-built sc:// string (an advanced option for personal access tokens only). The credential types and their fields:

• OAuth user-to-machine (U2M): host
• OAuth machine-to-machine (M2M): host, client_id and client_secret
• Personal access token: host and token

Databricks Connect looks for settings in a fixed order and uses the first it finds: the remote() method, a configuration profile, the DATABRICKS_CONFIG_PROFILE environment variable, one environment variable per property (such as DATABRICKS_HOST), then a profile named DEFAULT.

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.serverless().getOrCreate()

Run databricks-connect test to check the environment, default credentials and compute connection. It exits with a non-zero code and an error message when it detects a problem.

FAQs

Does adding a token encrypt the connection?

Yes, except for localhost. The connection string reference says that setting token enables SSL, so the client otherwise connects over TLS. You can also set use_ssl=true without a token.

Does open-source Spark check the token?

Not by default. The docs state that Spark Connect has no built-in authentication and point to authenticating proxies. A custom gRPC interceptor registered through spark.connect.grpc.interceptor.classes is the in-process alternative.

Is user_id a form of authentication?

No. It fills in the UserContext that the server uses for session management, and any client can set it. Verify a token as well.

How do I keep the token out of my code?

Put the full connection string in SPARK_REMOTE at runtime, or read the token from a secret file as shown above. Treat the whole connection string as a secret, because it contains the token.