Caching versus checkpointing?

Summary

• Caching (cache() or persist()) keeps computed partitions for reuse by later actions. It keeps the lineage, so Spark recomputes any lost partition.
• Checkpointing writes data to a checkpoint directory and cuts the dependency on the parent RDDs (for a DataFrame, it truncates the logical plan).
• Cache for reuse across actions. Checkpoint when the lineage or plan itself is the problem. For an RDD, the PySpark docs strongly recommend persisting it before you checkpoint it.
• On Databricks, RDD APIs are not supported on standard or serverless compute, and Databricks recommends a Delta table instead of DataFrame.checkpoint().

Caching versus checkpointing?

Caching keeps a dataset's computed partitions so later actions don't recompute them, and it keeps the lineage. Checkpointing writes the data to a checkpoint directory and cuts the dependency on everything upstream, so the lineage or plan can't keep growing. Caching is for reuse. Checkpointing is for truncation. Structured Streaming uses the word "checkpoint" for a third idea, covered below.

How does caching work?

cache() and persist() only mark an RDD or DataFrame. The first action that computes it keeps the partitions on the nodes, and later actions on that dataset, or on datasets derived from it, reuse them. Spark's cache is fault tolerant: a lost partition is recomputed from the transformations that created it. Spark drops old partitions in least-recently-used (LRU) order, and unpersist() removes them manually.
cache() uses the default storage level, which is MEMORY_ONLY for an RDD and MEMORY_AND_DISK_DESER for a PySpark DataFrame. persist() can set a level only when the dataset has none yet.

How does checkpointing work?

RDD.checkpoint() marks the RDD to be saved to a file in the directory set with SparkContext.setCheckpointDir(), and all references to its parent RDDs are removed. Call it before any job has run on that RDD. On a cluster the directory must be an HDFS path.
DataFrame.checkpoint(eager=True) truncates the logical plan, which the docs call especially useful in iterative algorithms where the plan may grow exponentially. It writes to that same directory or to the one set by the spark.checkpoint.dir configuration, available from Spark 4.0.0. The docs mark it experimental.
localCheckpoint() also truncates, but it stores the data on the executors through the caching layer. It sacrifices fault tolerance for performance: if an executor fails, the data may be gone and the job can fail irrecoverably. The docs also say it is not safe with dynamic allocation.

sc.setCheckpointDir("/tmp/checkpoints")   # an HDFS path on a cluster

rdd = sc.parallelize(range(1000)).map(lambda x: x * 2)
rdd.cache()        # keep partitions after the first action
rdd.checkpoint()   # mark it before any job runs on this RDD
rdd.count()        # first action; the checkpoint file exists afterwards
print(rdd.getCheckpointFile())   # a path now, None before the action

Should I cache an RDD before I checkpoint it?

Yes. The PySpark docs strongly recommend persisting the RDD in memory first, because otherwise saving it to a file requires recomputation. The cache then serves repeated reads and the checkpoint cuts the lineage.

What are the differences at a glance?

Aspect	Caching	Checkpointing
Purpose	Reuse computed data across actions	Truncate the RDD lineage or DataFrame logical plan
Lineage	Kept; lost partitions are recomputed from it	Parent references removed (RDD) or plan truncated (DataFrame)
Storage	Executors, in memory or on disk per the storage level	Files in the checkpoint directory
Trigger	Lazy, on the first action	RDD: file exists after an action runs. DataFrame: immediate by default (eager=True)
Cleanup	LRU eviction or unpersist()	Not cleaned when the reference goes out of scope unless spark.cleaner.referenceTracking.cleanCheckpoints is true (default false)

When should I use each?

• Cache when several actions read the same data. The RDD guide calls caching "a key tool for iterative algorithms and fast interactive use".
• Checkpoint when the lineage or plan is the problem, such as an iterative algorithm that keeps extending an RDD or DataFrame.
• Local checkpoint when you want truncation without writing to a reliable file system and can accept the failure risk.

Is a Structured Streaming checkpoint the same thing?

No. A streaming query's checkpointLocation holds what the query needs to restart. The Spark guide says the query saves "all the progress information (i.e. range of offsets processed in each trigger)" and the running aggregates there, and that the location "has to be a path in an HDFS compatible file system". Databricks lists offsets, commits, state and metadata, requires a different location for each query, and says that deleting the files or changing the location makes the next run begin fresh.

df.writeStream.option("checkpointLocation", "/Volumes/catalog/schema/volume/path").toTable("catalog.schema.table")

How do Databricks compute types handle caching and checkpointing?

• Disk cache: it copies remote Parquet data files to the nodes' local storage and differs from Spark caching. Databricks recommends automatic disk caching. A decommissioned worker takes its Spark cache with it, so with autoscaling enabled the cache can be unstable.
• RDD APIs: standard compute doesn't support them, and serverless compute supports only Spark Connect APIs. The RDD example above runs on neither.
• Serverless caching: DataFrame and SQL cache APIs aren't supported, including df.cache(), df.persist() and df.checkpoint().
• DataFrame.checkpoint(): Databricks recommends moving away from it and writing the DataFrame to a Delta table instead.
• Lakeflow Pipelines: checkpointing and retries are automatic, so a pipeline resumes from the last processed offset.

FAQs

What is the difference between cache() and persist()?

cache() is persist() with the default storage level. persist() accepts a StorageLevel, but it can assign a new level only if the dataset has none, so call unpersist() first to change one.

Does a cache survive an executor failure?

The cached partitions on that executor are lost, and Spark recomputes them from the transformations that created them, because a cache leaves the lineage intact.

Are checkpoint files deleted automatically?

Not by default. spark.cleaner.referenceTracking.cleanCheckpoints controls cleanup when the reference is out of scope, and it defaults to false.