When should I cache versus recompute?

Summary

• Cache a DataFrame when several actions reuse the same expensive result. Recompute when it is used once or is cheap to rebuild. The Spark docs warn: "Don't spill to disk unless the functions that computed your datasets are expensive, or they filter a large amount of the data."
• In PySpark 4.2, DataFrame.cache() uses MEMORY_AND_DISK_DESER. The RDD cache() uses MEMORY_ONLY.
• Cached data is shared across all Spark sessions on the cluster, so unpersist() affects every session.
• Databricks recommends its automatic disk cache. On serverless compute, df.cache(), df.persist() and CACHE TABLE raise an exception.

When should I cache versus recompute?

Cache a DataFrame when several actions reuse the same result and rebuilding it is expensive. Recompute when the result is used once, is cheap to rebuild, or would take memory that shuffles and joins need. The Spark docs call caching "a key tool for iterative algorithms and fast interactive use". They also say not to spill to disk unless the computation is expensive or filters a large amount of the data, because otherwise recomputing a partition "may be as fast as reading it from disk".

How do I decide?

Situation	Lean toward	Why
Several actions or iterations reuse an expensive result	Cache	Later actions reuse the stored partitions instead of rebuilding them
One action reads the result	Recompute	Nothing reuses the stored data
The computation is cheap and keeps most of the data	Recompute	Recomputing a partition "may be as fast as reading it from disk"
A filter removes most of a large input	Cache	The docs name heavy filtering, along with expensive functions, as the reasons to spill to disk
The result may not fit in memory	MEMORY_AND_DISK if it is expensive to rebuild, MEMORY_ONLY if it is cheap	MEMORY_ONLY recomputes partitions that do not fit; MEMORY_AND_DISK reads them back from disk
The source changes between uses	Recompute, or refresh	Cached entries stay until evicted or removed; REFRESH TABLE invalidates them

What does cache() do in PySpark?

In PySpark 4.2, DataFrame.cache() "persists the DataFrame with the default storage level (MEMORY_AND_DISK_DESER)". The RDD API differs: its cache() is shorthand for MEMORY_ONLY. Persisting is lazy, so data is stored the first time an action computes it, and the web UI's Storage tab does not list a persisted DataFrame until it is materialized.

from pyspark.storagelevel import StorageLevel

users = spark.read.parquet("/data/users")
enriched = spark.read.parquet("/data/events").join(users, "user_id")

enriched.cache()          # lazy: nothing is stored yet
enriched.count()          # first action materializes the cache
enriched.storageLevel     # shows the level that was set
enriched.write.parquet("/out/events_enriched")   # reuses the cached data
enriched.unpersist()      # removes the blocks for every session

users.persist(StorageLevel.DISK_ONLY)   # allowed only if users has no level yet

persist() can assign a new level only to a DataFrame that has none yet. unpersist() affects all sessions because cached data is shared across all Spark sessions on the cluster. Since 3.4.0, these methods also support Spark Connect.

Which storage level should I pick?

Level	What happens to data that does not fit	Note
MEMORY_ONLY	Partitions are not cached and are recomputed each time they are needed	RDD default
MEMORY_AND_DISK	Partitions are stored on disk and read from there	Default for CACHE TABLE
DISK_ONLY	All partitions are stored only on disk
MEMORY_AND_DISK_DESER	Uses memory and disk, with deserialized data	Default for PySpark DataFrames

The _2 variants replicate each partition on two cluster nodes. The serialized _SER levels exist for Java and Scala, and CACHE TABLE accepts them in SQL.

How do I size and monitor a cache?

Storage and execution share one memory region. The tuning guide says "Execution may evict storage if necessary, but only until total storage memory usage falls under a certain threshold (R)". The defaults are spark.memory.fraction 0.6 and spark.memory.storageFraction 0.5. To size a dataset, cache it and look at the Storage page of the web UI (default address http://<driver-host>:4040). Spark SQL caches tables in an in-memory columnar format, scans only the required columns, and tunes compression automatically. Two settings apply: spark.sql.inMemoryColumnarStorage.compressed defaults to true and spark.sql.inMemoryColumnarStorage.batchSize to 10000. Larger batches "can improve memory utilization and compression, but risk OOMs when caching data".

Does SQL caching behave the same way?

Not quite. CACHE TABLE caches immediately unless you write CACHE LAZY TABLE, while DataFrame.cache() is lazy. UNCACHE TABLE and CLEAR CACHE remove entries, and REFRESH TABLE invalidates them so they repopulate lazily on the next use.

CACHE TABLE testCache OPTIONS ('storageLevel' 'DISK_ONLY') SELECT * FROM testData;

How does Databricks approach caching?

Databricks recommends its automatic disk cache. It copies remote Parquet data files, including Delta Lake tables, to workers' local storage on first read, detects file changes, and invalidates stale entries; it is enabled by default on certain node types. The Spark cache, in contrast, is lost when its worker is decommissioned. On serverless compute, "Dataframe and SQL cache APIs are not supported": df.cache(), df.persist(), CACHE TABLE and the related commands raise an exception, so recompute there.

FAQs

What is the difference between cache() and persist()?

cache() persists with the default storage level. persist() accepts a StorageLevel, such as StorageLevel.DISK_ONLY.

When does the cache fill?

For a DataFrame, on the first action. For CACHE TABLE, immediately unless LAZY is given.

Does Spark ever cache for me?

Spark automatically persists some intermediate shuffle data, but the RDD guide still recommends calling persist on a result you plan to reuse.