What's the best way to embed Spark in an application?

Summary

• Spark Connect, introduced in Apache Spark 3.4, lets an application embed a thin Spark client. The client sends DataFrame operations as unresolved logical plans over gRPC to a Spark server.
• Results stream back as Apache Arrow-encoded row batches, and the client runs outside the Spark driver process.
• PySpark is supported since Spark 3.4 and Scala since Spark 3.5. SparkContext and RDDs are not supported.
• Go, Swift and Rust clients live in separate Apache repositories. The Go README calls its client "highly experimental".
• On Databricks, Databricks Connect is an extension of Spark Connect for Databricks Runtime 13.3 LTS and above.

What's the best way to embed Spark in an application?

Use Spark Connect when the application needs DataFrame work done on Spark. The Spark docs describe the Spark Connect client library as "a thin API that can be embedded everywhere: in application servers, IDEs, notebooks, and programming languages." The application carries a small client, a Spark server does the work, and the two talk over gRPC. Without one of the Spark Connect settings, your Spark session works just like before, without Spark Connect.

How does Spark Connect work?

The client translates DataFrame operations into unresolved logical query plans. These are "encoded using protocol buffers" and sent to the server using the gRPC framework. A Spark Connect endpoint on the server translates the plans into Spark's logical plan operators, and from there "the standard Spark execution process kicks in", so Spark's optimizations still apply. Results are streamed back through gRPC as Apache Arrow-encoded row batches.
The docs name three operational benefits:

• Stability: an application that uses too much memory only affects its own process, and it can define its own dependencies without conflicts with the Spark driver.
• Upgradability: the Spark driver can be upgraded independently of applications.
• Debuggability and observability: you can debug interactively from your IDE and monitor the application with its own metrics and logging libraries.

How do I start a server and connect an application?

Start the server from a Spark distribution, and install the thin Python client:

./sbin/start-connect-server.sh
pip install pyspark-client==4.2.0

Then point a session at the server with remote:

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote("sc://localhost").appName("SimpleApp").getOrCreate()

columns = ["id", "name"]
data = [(1, "Sarah"), (2, "Maria")]
df = spark.createDataFrame(data).toDF(*columns)
df.show()

spark.stop()

You can also set the SPARK_REMOTE environment variable (for example export SPARK_REMOTE="sc://localhost") and create a session as usual. The docs say there is "no code change needed" with this approach. For a Scala project, the docs add "org.apache.spark" %% "spark-connect-client-jvm" % "4.2.0" to build.sbt and call SparkSession.builder().remote("sc://localhost").getOrCreate().

Which languages does Spark Connect support?

Client	What its own docs or README say
PySpark	Since Spark 3.4, "most PySpark APIs", including DataFrame, Functions and Column. SparkContext and RDD are not supported.
Scala	Since Spark 3.5, "most Scala APIs", including Dataset, functions, Column, Catalog and KeyValueGroupedDataset.
Go (spark-connect-go)	README: "highly experimental and should not be used in any production setting".
Swift (spark-connect-swift)	README: a subproject of Apache Spark. It lists Apache Spark 4.2.0 and Swift 6.4.0 as requirements.
Rust (spark-connect-rust)	README: "A fast, native Rust client for Apache Spark Connect". It lists Apache Spark 4.2.0 or later and states no maturity level.

The docs also say a "Majority of the Streaming API is supported", including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener. User-defined functions are supported, by default in the shell and in standalone applications with additional set-up.

What changes from a classic Spark application?

The docs list three differences:

1. The client does not run in the same process as the Spark driver, so it cannot reach the driver JVM. In PySpark the client does not use Py4J, so private fields such as df._jdf are not available.
2. The protocol describes operations as logical plans, so it does not support all of Spark's execution APIs, "most importantly RDDs". The docs say alternatives for missing RDD functionality are provided in the DataFrame API.
3. The client is session-based. It has no access to the static Spark configuration or the SparkContext.

In a standalone Scala application, operations that reference user code, such as UDFs, need a ClassFinder registered, and JAR dependencies must be uploaded with SparkSession#AddArtifact.

What other ways can an application run Spark?

Approach	What the docs say
spark.api.mode set to connect	Lets Spark Classic applications switch to Spark Connect, for example --conf spark.api.mode=connect with --master.
spark.remote set to local[...] or local-cluster[...]	Starts a local Spark Connect server, for local testing.
SparkLauncher	"Use this class to start Spark applications programmatically", launching the application as a child process.
A local master URL such as local[*]	Runs Spark locally with as many worker threads as logical cores on your machine.

How does Databricks fit?

On Databricks, use Databricks Connect. The docs call it a client library for connecting to Databricks compute from IDEs, notebooks and "any custom application". For Databricks Runtime 13.3 LTS and above, it is "an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog." In Python:

from databricks.connect import DatabricksSession

spark = DatabricksSession.builder.getOrCreate()

The documented requirements include Unity Catalog enabled on the workspace, a cluster in Assigned or Shared access mode (serverless is supported starting with Databricks Connect version 15.1), and a Databricks Runtime version greater than or equal to the Databricks Connect package version.

FAQs

Can I use RDDs or SparkContext over Spark Connect?

No. The docs list APIs such as SparkContext and RDD as unsupported. Use the DataFrame or Dataset APIs instead.

Does Spark Connect have built-in authentication?

No. The docs say it does not, and that its gRPC HTTP/2 interface allows authenticating proxies, so you can secure it without adding authentication logic to Spark.