RocksDB versus HDFS state store?

Summary

• Structured Streaming has two built-in state store providers. In Apache Spark the HDFS state store provider is the default, and the RocksDB state store provider has been available since Spark 3.2.
• The HDFS provider keeps state in an in-memory map inside each executor JVM and backs it with files in an HDFS-compatible file system. The RocksDB provider keeps state in native memory and on the executor's local disk.
• The Spark docs point to RocksDB when a query keeps millions of keys in state and large JVM garbage collection pauses make micro-batch times vary.
• Pick the provider before the first run. The provider class cannot change for an existing checkpoint, so switching means discarding the checkpoint and starting a new query.
• On Databricks, RocksDB is the default state store provider in Databricks Runtime 17.3 and above.

RocksDB versus HDFS state store?

Stateful operators keep data between micro-batches. In Structured Streaming that covers streaming aggregation, streaming dropDuplicates, stream-stream joins, mapGroupsWithState and flatMapGroupsWithState. Spark keeps this data in a state store, which its docs describe as "a versioned key-value store which provides both read and write operations." Two provider implementations are built in, and you can write your own by extending the StateStoreProvider interface.

How do the two providers differ?

Aspect	HDFS state store provider	RocksDB state store provider
Status in Apache Spark	Default	Built in since Spark 3.2, enabled by configuration
State while the query runs	In-memory map inside each executor JVM	Native memory and the executor's local disk
Durable copy	Files in an HDFS-compatible file system	Changes saved to the checkpoint location
Many keys in state	State objects add JVM memory pressure and can cause long GC pauses	State stays out of the JVM heap
Memory limit	Executor JVM heap	Optional bounded memory setting, a soft limit

Both providers write state changes to the checkpoint location you configure. For RocksDB the Spark docs say the changes "are automatically saved by Structured Streaming to the checkpoint location you have provided, thus providing full fault-tolerance guarantees (the same as default state management)."

Which one should I use?

The default provider is the baseline. The Spark docs position RocksDB for large state: if you "want to maintain millions of keys in the state, then you may face issues related to large JVM garbage collection (GC) pauses causing high variations in the micro-batch processing times." The docs give no hard threshold, so judge by your own batch durations and by GC time in the Spark UI.

How do I turn on RocksDB in Apache Spark?

Set the provider class before the query starts:

spark.conf.set(
    "spark.sql.streaming.stateStore.providerClass",
    "org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider",
)

Two optional features matter for production, a memory cap and changelog checkpointing:

spark.conf.set("spark.sql.streaming.stateStore.rocksdb.boundedMemoryUsage", "true")
spark.conf.set("spark.sql.streaming.stateStore.rocksdb.maxMemoryUsageMB", "<value>")
spark.conf.set("spark.sql.streaming.stateStore.rocksdb.changelogCheckpointing.enabled", "true")

• Bounded memory. Left unbounded, RocksDB memory use across instances "could grow indefinitely and potentially cause OOM (out-of-memory) issues." The first two settings cap it per node, but only as a soft limit that can be exceeded temporarily.
• Changelog checkpointing. It uploads only the changes since the last checkpoint instead of RocksDB snapshots. It is off by default in Apache Spark, and you restart the query for a change to apply. Spark supports moving between the two checkpointing modes in both directions without discarding the old state checkpoint.

Can I switch providers on an existing checkpoint?

No. The Spark docs list spark.sql.streaming.stateStore.providerClass among the configurations that cannot change after a query has run: "To read the previous state of the query properly, the class of state store provider should be unchanged." The remedy is to "discard the checkpoint and start a new query." A new checkpoint begins fresh, so plan how to rebuild state. The same list includes spark.sql.shuffle.partitions, because state is partitioned by hashing the key.

What should I watch after a change?

• Progress reports. Each stateOperators entry reports numRowsTotal, numRowsUpdated and memoryUsedBytes. With RocksDB, customMetrics adds rocksdb metrics such as rocksdbCommitFlushLatency.
• HDFS provider cache. loadedMapCacheHitCount and loadedMapCacheMissCount count cache hits and misses on state cached in the provider. Raising spark.locality.wait helps keep a provider on the same executor across batches.
• Spark UI. The Structured Streaming tab shows the aggregated number of total state rows and the aggregated state memory used in bytes.

How does Databricks handle the state store?

• RocksDB is the default state store provider in Databricks Runtime 17.3 and above. On earlier runtimes, set spark.sql.streaming.stateStore.providerClass to com.databricks.sql.streaming.state.RocksDBStateStoreProvider.
• Databricks recommends changelog checkpointing for all stateful Structured Streaming queries (Databricks Runtime 13.3 LTS and above). It is on by default from 17.3, and from 17.3 the runtime also caps RocksDB memory per node automatically.
• For Lakeflow Pipelines, Databricks recommends RocksDB for stateful processing, and serverless pipelines manage state store settings automatically.
• Photon does not support stateful streaming.

FAQs

Does RocksDB replace the checkpoint location?

No. Both providers need one. The checkpoint holds the offsets, the commits and the state store contents that the provider manages.

Which changes to a stateful query need a new checkpoint?

The docs say any addition, deletion or schema change to the stateful operations of a query, such as the grouping keys of an aggregation, is not allowed between restarts.