What is the Spark Connect client-server architecture?

Summary

• Spark Connect splits a Spark application into a thin client library and a Spark Connect endpoint embedded in the Spark server. They communicate over gRPC (HTTP/2) using protocol buffers.
• A request flows in five steps: the client builds an unresolved logical plan, the endpoint translates it into Spark's logical plan operators, Spark's standard execution runs it, and results return as Apache Arrow-encoded row batches.
• Server behavior is set with documented keys such as spark.connect.grpc.binding.port (default 15002) and spark.connect.grpc.maxInboundMessageSize (default 134217728).
• The protocol is extensible: Relation, Expression and Command messages carry extension fields, and server libraries register plugin classes through spark.connect.extensions.*.classes settings.
• Spark Connect has no built-in authentication. The docs point to authenticating proxies in front of its gRPC interface.

What is the Spark Connect client-server architecture?

Spark Connect separates a Spark application into two parts. A thin client library runs inside your application, IDE or notebook. A Spark Connect endpoint is embedded on the Spark server, on the driver side. The Spark docs describe the protocol between them as the DataFrame API with unresolved logical plans, carried over gRPC.

Which components make up the architecture?

Component	Role
Client library	Translates DataFrame operations into unresolved logical plans. The Spark docs cover PySpark and Scala; Go, Swift and Rust clients live in separate Apache repositories
Protocol buffers	Encode the plans in a language-neutral format, because client and server may use different languages or software versions
gRPC over HTTP/2	Carries plans to the server and results back
Spark Connect endpoint	Embedded on the Spark server; turns plans into Spark's logical plan operators
Spark execution	Spark's standard process, including its optimizations
Apache Arrow	Encodes the row batches streamed back to the client

How does a request flow?

1. The client converts DataFrame operations into unresolved logical plans.
2. The plans are encoded with protocol buffers and sent over gRPC.
3. The endpoint translates them into Spark's logical plan operators, much as a SQL query is parsed into an initial plan.
4. Spark's standard execution process runs. The optimizations happen on the server and are independent of the client application.
5. Results stream back in partial chunks of Arrow record batches, not all at once.

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").appName("SimpleApp").getOrCreate()
logData = spark.read.text("YOUR_SPARK_HOME/README.md").cache()
numAs = logData.filter(logData.value.contains('a')).count()

Start the server first with ./sbin/start-connect-server.sh. The PySpark quickstart shows it listening on localhost:15002.

Which server settings shape the architecture?

Setting	Default	What it does
spark.connect.grpc.binding.port	15002	Port the Spark Connect server binds
spark.connect.grpc.binding.address	(none)	Address the server binds (since 4.0.0)
spark.connect.grpc.maxInboundMessageSize	134217728	Largest inbound gRPC request; larger payloads fail
spark.connect.grpc.arrow.maxBatchSize	4m	Largest Arrow batch sent from server to client
spark.api.mode	classic	classic or connect: whether Classic applications automatically use Spark Connect by running a local server (since 4.0.0)
spark.sql.connect.enrichError.enabled	true	Adds full exception messages and optionally the server-side stack trace to errors on the client (since 4.0.0)

Server settings are passed when you start the server, for example in the config file or with --conf.

How do you extend the protocol?

Developers can extend the three main operation types in the protocol: Relation, Expression and Command. Each has a google.protobuf.Any extension field that carries a custom protobuf message. A Spark Server Library has four parts: the protocol extension, a Spark Connect plugin, the application logic and a client package. You package the code as a jar and set spark.jars plus a class list such as spark.connect.extensions.expression.classes (the relation and command equivalents are spark.connect.extensions.relation.classes and spark.connect.extensions.command.classes). The server also accepts classes implementing io.grpc.ServerInterceptor through spark.connect.grpc.interceptor.classes.

What does the client give up?

The client is session-based. It has no access to cluster properties that change the environment for all connected clients, which includes the static Spark configuration and the SparkContext. RDDs are not supported, and PySpark does not use Py4J, so df._jdf is not possible. The connection string can carry a session_id, because the server caches Spark sessions by user ID and session ID.

How is the architecture secured?

The docs say Spark Connect does not have built-in authentication but is designed to work with your existing authentication infrastructure. Its gRPC HTTP/2 interface allows authenticating proxies, so you can secure it without adding authentication logic to Spark.

How does Databricks fit in?

Databricks Connect is built on open-source Spark Connect and uses unresolved logical plans and Apache Arrow on top of gRPC. The Databricks Visual Studio Code extension uses Databricks Connect for built-in debugging of user code on Databricks. The Databricks docs list Spark Connect as used in Scala notebooks on Databricks Runtime 13.3 and above and Python notebooks on 14.3 and above (both on standard compute), serverless compute and Databricks Connect.

FAQs

Which Python and Java versions does Spark Connect need?

The Spark 4.2 docs say Spark runs on Java 17, 21 or 25, Scala 2.13 and Python 3.10 or above. The pyspark-client package is a pure Python library that does not rely on jars or a JRE.

Does the architecture add latency?

The Spark Connect overview gives no overhead figure. The Spark docs do document that operations needing a resolved plan, such as df.columns, df.schema and actions, send the plan to the server. Avoid calling schema accessors on large numbers of new DataFrames.

Can I reuse existing Spark code?

DataFrame code is code-compatible with earlier Spark versions, except for RDD usage. Setting the SPARK_REMOTE environment variable needs no code change.