Which language clients does Spark Connect support (Go, Rust, Swift)?

Summary

• The Apache Spark docs say Spark Connect "supports PySpark and Scala applications": PySpark since Spark 3.4 and Scala since Spark 3.5. Both clients ship as part of Apache Spark.
• Go, Swift and Rust clients live in separate repositories under the Apache Spark project: github.com/apache/spark-connect-go, github.com/apache/spark-connect-swift and github.com/apache/spark-connect-rust. All three use the Apache License 2.0.
• Maturity differs, so read each README. The Go README says the client "is highly experimental and should not be used in any production setting". The Swift and Rust READMEs carry no such warning: Swift lists Apache Spark 4.2.0 as a requirement and Rust targets Spark 4.2.0 and later.
• Every client speaks the same protocol: unresolved logical plans encoded as protocol buffers and sent over gRPC, with results streamed back as Apache Arrow-encoded row batches.
• On Databricks, Databricks Connect is built on open-source Spark Connect and is available for Python, R and Scala.

Which language clients does Spark Connect support (Go, Rust, Swift)?

Apache Spark documents two Spark Connect clients of its own: PySpark, supported since Spark 3.4, and Scala, supported since Spark 3.5. Go, Swift and Rust clients live in their own repositories under the Apache Spark project, each with its own README. Only the Go README warns against production use, so check the repository you plan to depend on before you build on it.

How can a Spark Connect client be written in any language?

Spark Connect turns the DataFrame API into a wire protocol. The client "translates DataFrame operations into unresolved logical query plans which are encoded using protocol buffers," sends them to the server over gRPC, and gets results back "as Apache Arrow-encoded row batches." The Spark docs call unresolved logical plans "a language-agnostic protocol between the client and the Spark driver," which is why a client doesn't have to run on the JVM.
Every client addresses the server with the same connection string, sc://host:port. The scheme is fixed to sc://, and the default port is 15002.

Which clients exist, and how mature is each one?

Language	Where it lives	What its own docs say
Python	Apache Spark (pip install pyspark-client)	Since Spark 3.4, "most PySpark APIs, including DataFrame, Functions, and Column"; SparkContext and RDD aren't supported
Scala	Apache Spark (spark-connect-client-jvm)	Since Spark 3.5, most Scala APIs, including Dataset, functions, Column, Catalog and KeyValueGroupedDataset
Go	github.com/apache/spark-connect-go	"highly experimental and should not be used in any production setting"
Swift	github.com/apache/spark-connect-swift	"a subproject of Apache Spark"; requires Apache Spark 4.2.0
Rust	github.com/apache/spark-connect-rust	A "native Rust client for Apache Spark Connect"; targets Spark 4.2.0 and later

The Python and Scala clients are covered in the Spark Connect overview. The pyspark-client package "is a pure Python library that does not rely on any non-Python dependencies such as jars and JRE." Scala projects add the client in build.sbt:

libraryDependencies += "org.apache.spark" %% "spark-connect-client-jvm" % "4.2.0"

What do the Go, Swift and Rust READMEs say?

Go. The repository "houses the experimental client for Spark Connect for Apache Spark written in Golang." Its status section adds that the Apache Spark PMC "reserves the right to withdraw and abandon the development of this project if it is not sustainable." The getting-started steps use a Spark 4.0.0 or later distribution, and the quick-start guide opens a session like this:

ctx := context.Background()
spark, err := sql.NewSessionBuilder().Remote("sc://localhost:15002").Build(ctx)
if err != nil {
	log.Fatalf("Failed: %s", err)
}
defer spark.Stop()

Swift. Apache Spark Connect for Swift "is a subproject of Apache Spark and aims to provide a modern Swift library" for Swift applications. Its requirements list Apache Spark 4.2.0, and the README says the project is "tracking the upstream changes of Apache Arrow project's Swift-support." Apps import the SparkConnect module, and the library reads the SPARK_REMOTE environment variable for the connection string.
Rust. The README describes a "native Rust client for Apache Spark Connect" that "builds spark.connect protobuf plans, manages the gRPC channel, and decodes Arrow results in Rust." It publishes the apache-spark-connect crate and a Python package, pyspark-client-rust, which the README calls a drop-in replacement for the pyspark-client package. Issues are tracked in ASF JIRA under the SPARK project, and GitHub Issues are disabled.

How Databricks approaches Spark Connect clients

Databricks Connect "is built on open-source Spark Connect" and is available for Python, R and Scala. For Databricks Runtime 13.3 LTS and above, it's "an extension of Spark Connect with additions and modifications to support working with Databricks compute modes and Unity Catalog." It looks for connection settings in the DatabricksSession class's remote() method, a Databricks configuration profile, or environment variables, and a profile can hold a personal access token or OAuth user-to-machine or machine-to-machine settings. In Python, DatabricksSession.builder.profile("<profile-name>").getOrCreate() opens the session, and Databricks Connect for Python can also connect to serverless compute. The Databricks Connect docs don't list Go, Swift or Rust.

FAQs

Is there an official Java client for Spark Connect?

The Spark Connect overview names PySpark and Scala applications only. The JVM client ships as the spark-connect-client-jvm artifact, which the docs add to an sbt build for Scala.

Is the Rust client community-maintained?

No. It's github.com/apache/spark-connect-rust, an Apache Software Foundation repository under the Apache License 2.0, and its issues are tracked in ASF JIRA.

Can I use the Go client in production?

Its README says no: the client "is highly experimental and should not be used in any production setting." Swift and Rust carry no such warning, so check the release you depend on and the Spark version it requires.

Does Spark Connect support Structured Streaming?

Mostly. The overview says the "majority of the Streaming API is supported" for Scala, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener. In PySpark, SparkSession.readStream and DataFrame.writeStream have been labeled "Supports Spark Connect" since version 3.5.0.