What is Spark Connect?

Summary

• Spark Connect is a client-server architecture introduced in Apache Spark 3.4. A thin client library sends DataFrame operations to a remote Spark server, so your application no longer runs in the Spark driver process.
• The client encodes unresolved logical plans with protocol buffers and sends them over gRPC. The server runs them with Spark's standard execution and streams results back as Apache Arrow batches.
• It supports PySpark and Scala, including most DataFrame, Column, Catalog (Scala) and Structured Streaming APIs. RDDs, SparkContext and df._jdf are not supported.
• You connect with an sc://host:port connection string (default port 15002), the SPARK_REMOTE environment variable, or --remote on a shell.
• Spark Connect has no built-in authentication. The docs say it works with your existing authentication infrastructure, for example through authenticating proxies.

What is Spark Connect?

Spark Connect is a decoupled client-server architecture for Apache Spark, introduced in Spark 3.4. A thin client library, embedded in an application, IDE or notebook, sends DataFrame operations to a remote Spark server. The Spark project compares it to how applications connect to databases through a JDBC driver.

How does Spark Connect work?

1. The client converts a DataFrame query into an unresolved logical plan and encodes it with protocol buffers.
2. The plan is sent to the Spark server over gRPC.
3. The Spark Connect endpoint on the server translates it into Spark's logical plan operators.
4. Spark's standard execution process runs, so the query gets Spark's optimizations.
5. Results stream back through gRPC as Apache Arrow-encoded row batches.

How is Spark Connect different from Spark Classic?

Aspect	Spark Classic	Spark Connect
Client and driver	Your application runs with the driver	The client "does not run in the same process as the Spark driver"
JVM access from PySpark	Through Py4J (for example df._jdf)	No Py4J, so df._jdf is not possible
RDDs and SparkContext	Available	Not supported
Schema analysis	Eager	Lazy
Temporary views	Plan embedded in the DataFrame	Looked up by name at execution time
UDF serialization	At creation	At execution

How do I connect to a Spark Connect server?

Start a server with ./sbin/start-connect-server.sh, then create a session with a remote URL:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

The connection string has the form sc://host:port/;param=value, and the port defaults to 15002. If you set the SPARK_REMOTE environment variable, the docs say no code change is needed. A PySpark shell can also start with ./bin/pyspark --remote "sc://localhost". The Spark project says switching from a local session to a remote one is a matter of changing the connection string.

Which APIs does Spark Connect support?

• PySpark: since 3.4, most APIs, including DataFrame, Functions and Column.
• Scala: since 3.5, most APIs, including Dataset, functions, Column, Catalog and KeyValueGroupedDataset.
• Streaming: the docs say the "Majority of the Streaming API is supported", including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener.
• UDFs: by default in the shell, and with extra set-up in standalone applications.
• Spark ML: Spark 4.0 added ML on Spark Connect, and the 4.1 notes call it GA for the Python client.

Apache also hosts Spark Connect client repositories for Go, Swift and Rust. The Go README calls its client "highly experimental" and says it should not be used in production. The Swift README calls its project a subproject of Apache Spark. The Rust README describes a native Rust client.

How does lazy analysis change my code?

Spark Connect defers analysis. In Spark Connect, spark.sql("select 1 as a, 2 as b").filter("c > 1") raises no error until you call df.columns or df.show(), because the unresolved plan reaches the server only then. Calls such as df.columns and df.schema send an analysis request to the server, so avoid calling them on large numbers of new DataFrames, for example inside a loop.

What benefits does the Spark documentation list?

• Stability: an application that uses too much memory affects only its own process, and users define their own client dependencies without worrying about conflicts with the Spark driver.
• Upgradability: the Spark driver can be upgraded independently of applications.
• Debuggability and observability: you can debug interactively from your IDE and monitor applications with their own metrics and logging libraries.

How is Spark Connect secured?

The docs say Spark Connect does not have built-in authentication but is designed to work with your existing authentication infrastructure. Its gRPC HTTP/2 interface allows authenticating proxies. On the client side, the connection string accepts a token parameter for bearer token authentication (which enables SSL) and a use_ssl flag that defaults to false.

How does Databricks use Spark Connect?

Databricks Connect is built on open-source Spark Connect. The Databricks docs list Spark Connect as used in Scala notebooks on Databricks Runtime 13.3 and above and Python notebooks on 14.3 and above (both on standard compute), serverless compute and Databricks Connect.

FAQs

Do I still need a Spark server?

Yes. The client connects to a Spark server that runs a Spark Connect endpoint. For local testing, setting spark.remote to local[...] or local-cluster[...] starts a local Spark Connect server.

Can I move existing code to Spark Connect?

The Spark Connect API is code-compatible with earlier Spark versions except for RDD usage, where the DataFrame API provides alternatives. For a Spark Classic application, spark.api.mode (default classic, since 4.0.0) can be set to connect.

Does Spark Connect support Structured Streaming?

Most of it. The overview lists the supported streaming classes above, and Spark 4.1 added transformWithState to the Connect API coverage.