What protocol does Spark Connect use (gRPC)?

Summary

• Spark Connect uses gRPC. The Spark docs say the client translates DataFrame operations into unresolved logical query plans, encodes them with protocol buffers, and sends them to the server using the gRPC framework.
• Results come back over gRPC too. The server streams them to the client as Apache Arrow-encoded row batches.
• gRPC runs on HTTP/2. The Spark docs say this interface allows the use of authenticating proxies. Spark Connect has no built-in authentication of its own.
• Clients connect with an sc:// URL. The format is sc://host:port/;param=value, and the default port is 15002.
• Databricks Connect is built on Spark Connect. Its docs say it communicates with Databricks clusters via gRPC over HTTP/2.

What protocol does Spark Connect use (gRPC)?

Spark Connect uses gRPC. The Apache Spark project describes Spark Connect as a protocol that specifies how a client application communicates with a remote Spark server, and says its network communication uses the gRPC framework. gRPC uses an HTTP/2-based transport, and Spark Connect pairs it with two data formats.

Layer	Role in Spark Connect
gRPC	The RPC framework that carries requests and responses between client and server.
Protocol Buffers	The serialization format for the unresolved logical plan the client builds from your DataFrame operations.
Apache Arrow	The encoding for result data, which is streamed back to the client as row batches.

How does a query travel from client to server?

The Spark docs describe the path in four steps.

1. The client translates your DataFrame operations into an unresolved logical query plan and encodes it with protocol buffers.
2. The client sends the encoded plan to the server using gRPC.
3. The Spark Connect endpoint on the server translates the unresolved plan into Spark's logical plan operators, similar to parsing a SQL query. The standard Spark execution process then runs.
4. Results are streamed back to the client through gRPC as Apache Arrow-encoded row batches.

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()

# The client encodes these operations as an unresolved logical plan
# and sends it to the server over gRPC.
df = spark.range(10).filter("id > 5")
df.show()

The client does not run in the same process as the Spark driver. The docs note that an application that uses too much memory now only affects its own environment.

Which gRPC calls does the Spark Connect service define?

Spark defines the service as SparkConnectService in its protocol buffer file base.proto. gRPC separates unary calls from streaming calls, and the service uses both.

RPC	Call type	Purpose, from the file's comments
ExecutePlan	Server streaming	Executes a request that contains the query and returns a stream of responses.
ReattachExecute	Server streaming	Reattaches to an existing reattachable execution.
AddArtifacts	Client streaming	Adds artifacts to the session.
AnalyzePlan	Unary	Analyzes a query and returns metadata about it.
Config	Unary	Updates or fetches configuration.
Interrupt	Unary	Interrupts running executions.

The file defines additional RPCs beyond these six.

How do clients connect, and what does sc:// mean?

Spark Connect connection strings use the format sc://host:port/;param1=value;param2=value. The scheme is fixed as sc://, and the default port is 15002.

Parameter	What it does
token	Enables standard bearer token authentication using gRPC. Setting it enables SSL.
use_ssl	Connects to the endpoint using TLS. The default is false.
grpc_max_message_size	Maximum gRPC message size in bytes. The default is 128 * 1024 * 1024.

A valid example from Spark's specification is sc://myhost.com:443/;use_ssl=true;token=ABCDEFG.

Does Spark Connect handle authentication?

Not by itself. The Spark docs say that while Spark Connect does not have built-in authentication, it is designed to work with your existing authentication infrastructure. Its gRPC HTTP/2 interface allows the use of authenticating proxies, so you can secure it without implementing authentication logic in Spark.

How does this work on Databricks?

Databricks Connect is built on open-source Spark Connect. Its docs say the underlying protocol uses Spark unresolved logical plans and Apache Arrow on top of gRPC, and that Databricks Connect communicates with Databricks clusters via gRPC over HTTP/2. For Databricks Runtime 13.3 LTS and above, it is an extension of Spark Connect with changes to support Databricks compute modes and Unity Catalog.
You create the session with DatabricksSession:

from databricks.connect import DatabricksSession

# A configuration profile that holds the auth settings and cluster_id
spark = DatabricksSession.builder.profile("<profile-name>").getOrCreate()

# Or, on serverless compute
spark = DatabricksSession.builder.serverless().getOrCreate()

The advanced-usage docs also show the connection-string form, sc://<workspace-instance-name>:443/;token=<access-token-value>;x-databricks-cluster-id=<cluster-id>.

FAQs

Why does Spark Connect use gRPC instead of REST?

The Spark site says gRPC is performant and language agnostic, which makes Spark Connect portable, and the docs note that its HTTP/2 interface allows authenticating proxies. gRPC also supports server-streaming calls, and ExecutePlan is one: it returns a stream of responses.

Which languages have Spark Connect clients?

The Spark docs say Spark Connect supports PySpark and Scala applications. The Spark Connect page also lists Python, Go, Rust and Swift clients plus a third-party .NET client, and says you can write a client in any language. The Go client's README calls it highly experimental and says it should not be used in any production setting.

Can a Spark Connect client use RDDs or SparkContext?

No. The docs say the protocol does not support all of Spark's execution APIs, most importantly RDDs, and the client has no access to the static Spark configuration or the SparkContext. Because the client does not use Py4J, private JVM fields such as df._jdf are not reachable either.

Do client and server versions have to match?

Spark's stated goal is that the driver can be upgraded independently of applications, and that applications stay forward-compatible as long as the server-side RPC definitions are backwards compatible. Protocol buffers allow new fields to be added without breaking existing services. On Databricks, the runtime version of your compute must be greater than or equal to the Databricks Connect package version.