Why does my RDD code not work over Spark Connect?

Summary

• A Spark Connect client runs in a separate process from the Spark driver. The Spark docs say the protocol "does not support all the execution APIs of Spark, most importantly RDDs", and that the client has no access to the SparkContext.
• Use the DataFrame API or Spark SQL instead. The docs say alternatives for missing RDD functionality are provided as part of the higher-level DataFrame API, and PySpark reference pages marked "Supports Spark Connect" show what you can call.
• If RDD code that used to work now fails, check whether something switched the session to Connect: the SPARK_REMOTE variable, --remote, .remote("sc://...") or spark.api.mode.
• On Databricks, serverless compute supports only Spark Connect APIs, standard compute lists RDD APIs as unsupported, and Databricks Connect lists RDDs and Spark Context as not available.

Why does my RDD code not work over Spark Connect?

Spark Connect runs your client application in a separate process from the Spark driver. The client turns DataFrame operations into unresolved logical plans, encodes them with protocol buffers and sends them to the server over gRPC. The Spark docs spell out the consequence: because the protocol describes operations as logical plans, it "does not support all the execution APIs of Spark, most importantly RDDs". The client also cannot reach the driver JVM, the static Spark configuration or the SparkContext.

What can a Spark Connect client use?

Code you have	Over Spark Connect	Use instead
DataFrame API, Spark SQL	Supported: most PySpark APIs since Spark 3.4, most Scala APIs since 3.5	Keep it
sc.parallelize, sc.textFile	Not available, there is no SparkContext	spark.createDataFrame, spark.read.text
rdd.map, rdd.reduceByKey	RDD is unsupported	select, withColumn, groupBy(...).agg(...)
sc.broadcast, sc.accumulator	Created through SparkContext, which the client lacks	For broadcast joins, the broadcast() hint on a DataFrame
df._jdf and other driver JVM access	Not possible, the PySpark client does not use Py4J	DataFrame API
Structured Streaming	The majority of the Streaming API is supported	Keep it

How do I convert RDD code to the DataFrame API?

from pyspark.sql import functions as F

# RDD version: needs a SparkContext, so it fails over Spark Connect
rdd = sc.parallelize([1, 2, 3, 4, 5])
result = rdd.map(lambda x: x + 1).collect()

# DataFrame version: works over Spark Connect
df = spark.createDataFrame([(1,), (2,), (3,), (4,), (5,)], ["value"])
result = df.select((F.col("value") + 1).alias("plus_one")).collect()

DataFrame.collect() returns a list of Row objects, so read row.plus_one instead of a bare number. A key-based aggregation follows the same pattern:

# RDD version
pairs = sc.parallelize([("a", 1), ("b", 2), ("a", 3)])
counts = pairs.reduceByKey(lambda a, b: a + b).collect()

# DataFrame version
df = spark.createDataFrame([("a", 1), ("b", 2), ("a", 3)], ["key", "val"])
counts = df.groupBy("key").agg(F.sum("val").alias("total")).collect()

Check each call against the PySpark API reference before you migrate. The DataFrame.filter and DataFrame.coalesce pages carry a "Supports Spark Connect" note, and the RDD.reduceByKey and RDD.coalesce pages do not. Spark 4.2.0 also lists "RDD API compatibility" work under Spark Connect: DataFrame.zipWithIndex (similar to RDD.zipWithIndex()), DataFrame.toJSON, which returns a DataFrame in Connect mode and an RDD in Classic mode, and spark.read.json accepting a DataFrame.

How do I tell which mode my session is in?

type(spark)
# <class 'pyspark.sql.connect.session.SparkSession'>

If the class path includes .connect., the session uses Spark Connect. These settings make a session a Connect session:

Setting	Example
Environment variable	export SPARK_REMOTE="sc://localhost"
Shell option	./bin/pyspark --remote "sc://localhost"
Builder	SparkSession.builder.remote("sc://localhost").getOrCreate()
API mode	spark.api.mode set to connect (the default is classic)

The docs add that without one of these mechanisms, a Spark session "will work just like before". If RDD code fails after a configuration change, look for one of these settings.

What about Databricks?

• Serverless compute: "Only Spark Connect APIs are supported. Spark RDD APIs are not supported."
• Standard compute: the limitations page lists "RDD APIs are not supported" and says to use dedicated compute if you need a feature it lacks.
• Databricks Connect is built on open-source Spark Connect. Its Python limitations page lists Spark Context and RDDs as not available, along with libraries that use RDDs, Spark Context or the underlying Spark JVM.

FAQs

Can I still use spark.sparkContext in a Connect session?

No. The docs say APIs such as SparkContext and RDD are unsupported in Spark Connect, and the client has no access to the SparkContext.

Do DataFrames lose the optimizer over Spark Connect?

No. The server receives the unresolved plan, turns it into Spark's logical plan operators and runs the standard Spark execution process. The Spark SQL guide explains that the structure DataFrames expose gives Spark more information for optimizations, which RDD code does not provide.

Where is the list of supported APIs?

The PySpark API reference. The docs say supported APIs are labeled "Supports Spark Connect", so you can check an API before migrating existing code.