RDD versus DataFrame performance?

Summary

• Spark SQL gives Spark "more information about the structure of both the data and the computation being performed" and uses it "to perform extra optimizations". The functions you pass to RDD operations carry no such structure.
• SQL and the DataFrame API run on the same execution engine, so choosing between them is a style choice. The Spark quick start recommends moving from RDDs to Dataset, and Python has the DataFrame API instead.
• The differences come from mechanisms, not a fixed ratio: query optimization, adaptive query execution, data source pushdown such as Parquet filter pushdown, join strategy selection, and Encoders instead of Java serialization or Kryo.
• Keep RDDs for low-level control such as custom partitioners. RDD APIs aren't supported on Spark Connect, Databricks serverless compute or standard compute.

RDD versus DataFrame performance?

An RDD is "a fault-tolerant collection of elements that can be operated on in parallel", and you transform it with functions you write. A DataFrame is "a Dataset organized into named columns", conceptually equivalent to a table in a relational database, "but with richer optimizations under the hood". The Spark SQL guide contrasts its interfaces with the basic RDD API: they provide Spark "with more information about the structure of both the data and the computation being performed", which Spark SQL uses "to perform extra optimizations". The docs give no fixed speed ratio, so check your own workload with explain() and the Spark UI.

Aspect	RDD	DataFrame
What Spark knows	Elements and the functions you pass	Column names, types and the operations on them
Optimization	Lazy transformations, with no structure for Spark SQL to optimize	Spark SQL optimizations, and adaptive query execution (AQE) re-plans using runtime statistics
Reading files	Your code parses records, for example with textFile and map	Data sources with a schema, such as Parquet with filter pushdown (spark.sql.parquet.filterPushdown, default true)
Joins	You choose the strategy, for example with broadcast variables	Join hints, and automatic broadcast up to spark.sql.autoBroadcastJoinThreshold (10 MB)
Serialization	Java serialization by default, Kryo optional, Pickle for stored Python objects	Encoders, which are code generated and allow "filtering, sorting and hashing without deserializing" the bytes
Caching	cache() stores deserialized objects (MEMORY_ONLY)	cache() uses an in-memory columnar format that scans "only required columns"
Spark Connect	Not supported	Supported

What explains the gap in practice?

AQE is "an optimization technique in Spark SQL" that uses runtime statistics to choose a plan, and it has been on by default since Spark 3.2.0. It applies to Spark SQL queries, which includes DataFrames. Spark SQL also pushes filters into sources such as Parquet and chooses join strategies. With RDD code, you write those steps yourself. Because "the same execution engine is used, independent of which API/language you are using", a DataFrame expression and the equivalent SQL statement run on the same engine.
The Spark quick start says RDDs "are replaced by Dataset", which has "richer optimizations under the hood", and recommends it: "we highly recommend you to switch to use Dataset". Python has no Dataset API, but the SQL guide says that, because of Python's dynamic nature, many of its benefits are already available there.

When should I still use RDDs?

Use them when you want direct control over partitioning and per-partition processing. The RDD guide lists mapPartitions and repartitionAndSortWithinPartitions(partitioner), which takes a custom partitioner. In Scala and Java, a Dataset combines the strong typing and lambda functions of RDDs with the Spark SQL optimized execution engine. Check your platform first: the Spark Connect overview says "APIs such as SparkContext and RDD are unsupported".

from pyspark.sql import Row

sc = spark.sparkContext
people = sc.parallelize([Row(name="Ann", age=34), Row(name="Bo", age=19)])
df = spark.createDataFrame(people)  # RDD of Row -> DataFrame, types inferred
adults = df.filter("age >= 21").select("name")
adults.explain()  # prints the plan Spark SQL will run
names = adults.rdd.map(lambda r: "Name: " + r.name).collect()  # .rdd returns an RDD of Row

How Databricks approaches RDD and DataFrame workloads

Databricks documents the same split. The serverless compute limitations say "Only Spark Connect APIs are supported. Spark RDD APIs are not supported." The standard compute limitations also list "RDD APIs are not supported". Photon is "the Databricks-native vectorized query engine" for SQL workloads, DataFrame API calls, ETL pipelines and stateless streaming. The Apache Spark optimizer (Catalyst) still plans the query. The Photon docs also say it "doesn't support UDFs (User Defined Functions), RDD APIs, or Dataset APIs", and unsupported operations transparently switch to the Spark runtime with correct results.

FAQs

Does the DataFrame API perform differently from SQL?

Not because of which API you pick. The Spark SQL guide says the same execution engine is used whichever API or language you use, so developers can "switch back and forth between different APIs based on which provides the most natural way to express a given transformation".

How do I convert between RDDs and DataFrames?

For an RDD of Row objects, spark.createDataFrame(rdd) infers the types by sampling the dataset. In Scala, import spark.implicits._ allows .toDF() on an RDD of case classes. df.rdd returns an RDD of Row.

Do I need to tune RDD serialization?

It can help, and the tuning guide documents the options. It describes Java serialization as the default and Kryo as an option through spark.serializer set to org.apache.spark.serializer.KryoSerializer, with class registration for best results. For cached Scala and Java RDDs it suggests serialized storage levels such as MEMORY_ONLY_SER to reduce memory usage, at the cost of slower access.