What API gaps does Spark Connect still have in 2026?

Summary

• The Spark docs say the Spark Connect protocol "does not support all the execution APIs of Spark, most importantly RDDs", and that the client has no access to the static Spark configuration or the SparkContext.
• In PySpark the client does not use Py4J, so private JVM fields such as df._jdf are not reachable.
• Most of the Structured Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener.
• Behavior differs even where an API exists: Spark Connect defers analysis, looks up temporary views by name and serializes Python UDFs at execution time.
• Spark 4.2.0 (July 14, 2026) added RDD API compatibility items and a GetStatus API. Databricks serverless compute supports only Spark Connect APIs.

What API gaps does Spark Connect still have in 2026?

Spark Connect splits the client from the Spark driver. The client translates DataFrame operations into unresolved logical plans, sends them to the server over gRPC, and receives results as Apache Arrow-encoded row batches. The Spark docs say it "supports PySpark and Scala applications". The gaps below come from the Spark 4.2 documentation and release notes.

from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost").getOrCreate()

Which APIs does Spark Connect not support?

Area	What the Spark docs say
RDDs	"The Spark Connect protocol does not support all the execution APIs of Spark, most importantly RDDs." SparkContext and RDD are unsupported in Scala and PySpark
SparkContext and static configuration	"The client does not have access to the static Spark configuration or the SparkContext."
Driver JVM	The client does not run in the same process as the driver, so it cannot directly access the driver JVM to manipulate the execution environment
Py4J internals	The PySpark client does not use Py4J, so accessing private fields that hold the JVM implementation of DataFrame, Column or SparkSession (for example df._jdf) is not possible
UDFs	Supported "by default for the shell and in standalone applications with additional set-up requirements"

Supported APIs are labeled "Supports Spark Connect" in the API reference, so check a method there before migrating code. Spark 4.0.0 added spark.api.mode (default classic, values classic or connect), which lets a Spark Classic application use Spark Connect by running a local Spark Connect server.

How do UDFs and dependencies work?

Operations that reference user-defined code, such as UDFs, filter and map, need a ClassFinder registered to pick up and upload class files, and JAR dependencies must be uploaded with SparkSession#AddArtifact. The docs' example is Scala:

import org.apache.spark.sql.connect.client.REPLClassDirMonitor
val classFinder = new REPLClassDirMonitor(<ABSOLUTE_PATH_TO_BUILD_OUTPUT_DIR>)
spark.registerClassFinder(classFinder)
spark.addArtifact(<ABSOLUTE_PATH_JAR_DEP>)

In PySpark, SparkSession.addArtifacts(*path, pyfile=False, archive=False, file=False) adds artifacts to the client session. With pyfile=True they become Python dependencies such as .py, .egg, .zip or .jar files. Currently only local files are supported.

How does behavior differ where an API exists?

The Spark docs' "Eager vs Lazy" guide compares the two modes.

Aspect	Spark Classic	Spark Connect
Query execution	Lazy	Lazy
Schema analysis	Eager	Lazy
Schema access (df.columns, df.schema)	Local	Triggers an analysis RPC and caches the schema
Temporary views	Plan embedded	Looked up by name
UDF serialization	At creation	At execution

Replacing a temporary view changes DataFrames that already reference it by name, so the guide suggests unique view names. A misspelled column surfaces at df.columns or an action, not at the transformation. A Python UDF reads its variables when it runs, so the guide's fix is a function factory:

from pyspark.sql.functions import udf

def make_udf(value):
  def foo():
    return value
  return udf(foo)

x = 123
foo_udf = make_udf(x)
x = 456
df = spark.range(1).select(foo_udf())
df.show()  # prints 123

The guide also warns that reading df.columns on many new DataFrames sends many analysis requests, and suggests building the columns in one select.

Which gaps have recent releases closed?

Release	Spark Connect items in the release notes
4.0.0	Lightweight Python client (pyspark-client), full API compatibility for the Java client, greatly expanded API coverage, ML on Spark Connect, a Swift client
4.1.0	JDBC driver for Spark Connect, ML Connect improvements, better usability and performance
4.2.0	RDD API compatibility items (zipWithIndex on Scala Dataset and PySpark DataFrame, DataFrame.toJSON in the Python client, spark.read.json, xml and csv accepting DataFrame input, emptyDataFrame), GetStatus API, Spark Connect tab in the History Server

Structured Streaming is mostly covered: the docs say the majority of the Streaming API is supported, including DataStreamReader, DataStreamWriter, StreamingQuery and StreamingQueryListener.
Separate Apache Spark repositories host Spark Connect clients for Go, Swift and Rust. The Go README calls its client "highly experimental" and says it "should not be used in any production setting". Check each README for current status.

What changes on Databricks?

Databricks Connect is "built on open-source Spark Connect" and adds support for Databricks compute modes and Unity Catalog. It supports Python, R and Scala, though serverless compute does not support R. Databricks Connect for Python does not support Spark Context, RDDs, libraries that use RDDs, Spark Context or the underlying Spark JVM, or distributed ML training.
On serverless compute, only Spark Connect APIs are supported, RDD APIs are not, and analysis and name resolution are deferred to execution time. DataFrame and SQL cache APIs are not supported and raise an exception. Streaming supports Trigger.AvailableNow() (recommended) and the deprecated Trigger.Once(), but not Trigger.Continuous(interval) or Trigger.ProcessingTime(interval).

FAQs

Can I use MLlib over Spark Connect?

The 4.0.0 notes list ML on Spark Connect and the 4.1.0 notes list ML Connect improvements. The RDD-based spark.mllib API is in maintenance mode and the DataFrame-based spark.ml API is the primary one. Check the API reference for "Supports Spark Connect" before relying on a specific algorithm.

Why do my RDD-based libraries fail?

The Spark Connect client has no SparkContext and the protocol does not support RDDs. Port that code to DataFrame or SQL APIs, or run it on Spark Classic.

Can I call JVM libraries?

In Scala, upload JAR dependencies with SparkSession#AddArtifact. The PySpark client cannot reach driver JVM objects such as df._jdf.