What is the difference between an RDD, a DataFrame, and a Dataset?

Summary

• An RDD (resilient distributed dataset) is Spark's low-level abstraction: a fault-tolerant collection of elements partitioned across the cluster, transformed by functions that Spark runs as written.
• A DataFrame is a Dataset organized into named columns, conceptually a relational table. Because Spark SQL knows its schema and operations, its optimizer can plan the work, and the API exists in Python, Scala, Java and R.
• A Dataset is a strongly typed collection of domain objects in Scala and Java. It combines typed lambdas such as map and filter with Spark SQL's engine, using encoders to serialize objects into Spark's internal format.
• In Scala a DataFrame is literally Dataset[Row], and Python has no Dataset API, so PySpark code chooses only between DataFrames and RDDs.
• Use DataFrames by default. The Spark Quick Start recommends switching from RDDs to Datasets, and Spark Connect doesn't support RDDs at all.

What is the difference between an RDD, a DataFrame, and a Dataset?

They're three layers of the same engine, with less structure at the bottom and more at the top. An RDD is a distributed collection of arbitrary objects that Spark processes with your functions and can't inspect. A DataFrame adds a schema of named columns, so Spark SQL can plan and optimize its operations. A Dataset adds static types on top of that in Scala and Java, where a DataFrame is simply a Dataset of Row. Write new code against DataFrames, or typed Datasets in Scala and Java, and keep RDDs for the cases that truly need them.

What each abstraction is

Concept	What it is	Why it matters
RDD	A fault-tolerant collection of elements partitioned across the nodes of the cluster that can be operated on in parallel	Full control over objects and functions, but no schema for the optimizer, and no Spark Connect support
DataFrame	A Dataset organized into named columns, conceptually equivalent to a table in a relational database	Spark SQL plans and optimizes its operations; available in Python, Scala, Java and R
Dataset	A strongly typed collection of domain-specific objects that can be transformed with functional or relational operations	Typed transformations on Spark SQL's engine; Scala and Java only

How the three relate

An RDD records the transformations that built it. If a partition is lost, Spark recomputes it from those transformations, so RDDs recover from node failures automatically. The functions themselves stay opaque: Spark runs your lambda on each element but can't see what it reads or filters.
DataFrames and Datasets are both lazy representations of a logical plan. When an action runs, Spark's query optimizer optimizes that logical plan and generates a physical plan, and Spark uses the same execution engine whether you express the query in SQL, Python or Scala. The difference between the two is typing. DataFrame operations are what the Spark docs call untyped transformations over columns. Dataset operations are typed transformations over your own classes, backed by an Encoder for each type. The encoder isn't Java serialization or Kryo: it's code generated at runtime, and its binary format lets Spark filter, sort and hash data without deserializing it back into objects.

import org.apache.spark.sql.{DataFrame, Dataset}
import org.apache.spark.rdd.RDD
import spark.implicits._

case class Person(name: String, age: Long)

val ds: Dataset[Person] = spark.read.json("people.json").as[Person]
val df: DataFrame = ds.toDF()        // DataFrame is Dataset[Row]
val rdd: RDD[Person] = ds.rdd

ds.filter(p => p.age >= 18)   // typed: a misspelled field fails to compile
df.filter($"age" >= 18)       // untyped: a bad column fails at analysis
rdd.filter(_.age >= 18)       // RDD: an opaque function Spark can't optimize

Python and R have no Dataset API. The Spark SQL guide's explanation is Python's dynamic nature: you already access a row's fields by name, as in row.columnName. In PySpark the choice is between DataFrames and RDDs, and you can move between them:

df = spark.read.json("people.json")               # DataFrame
adults = df.filter(df.age >= 18).select("name")   # planned by Spark SQL

rows = df.rdd                                     # RDD of Row objects
names = rows.map(lambda r: r.name).collect()
df_again = spark.createDataFrame(rows)            # back to a DataFrame

When to use each

• DataFrame: the default for structured and semi-structured data, SQL-style transformations and any code that must run over Spark Connect.
• Dataset: Scala or Java code that works with domain objects and benefits from compile-time types, while still running on Spark SQL's engine.
• RDD: low-level control over unstructured records, or existing code you aren't ready to port. The Spark Quick Start says the RDD interface is still supported but recommends switching to Dataset.

How Databricks approaches RDDs, DataFrames and Datasets

Databricks runs the same three APIs, and the compute you choose decides which are available. Serverless compute supports only Spark Connect APIs, so RDD APIs aren't supported there, and standard access mode doesn't support RDD APIs either. Typed Scala Dataset operations such as map, flatMap and filter need Databricks Runtime 15.4 LTS or above on standard access mode. Photon, the Databricks-native vectorized query engine, runs SQL and DataFrame workloads while the Spark optimizer (Catalyst) still plans the query. Photon doesn't support UDFs, RDD APIs or Dataset APIs, and it falls back to the Spark runtime for unsupported operations, so DataFrame code is the form that gets the most from the platform.

FAQs

Is a DataFrame the same as a Dataset?

In Scala and Java, yes. DataFrame is a type alias of Dataset[Row] in Scala, and Java code uses Dataset<Row>. Python and R only have DataFrames.

Can I convert between them?

Yes. df.rdd returns an RDD of Row objects, spark.createDataFrame(rdd) builds a DataFrame from one, df.as[Person] produces a typed Dataset in Scala, and ds.toDF() converts back.

Do RDDs work with Spark Connect?

No. The Spark Connect protocol doesn't support all of Spark's execution APIs, most importantly RDDs, and Spark Connect clients have no access to the SparkContext.