Why is my streaming checkpoint corrupted after a restart?

Summary

• Most "corrupted checkpoint" errors after a restart mean the checkpoint no longer matches the query or environment restarting it, not that the bytes are damaged. Read the error class first.
• The usual causes are two runs writing to one checkpoint, files that were deleted or never written durably, settings or Spark versions that changed between runs and, less often, real file damage.
• Prevent most of it by giving every query its own checkpoint on fault-tolerant storage, never editing files inside it, and leaving settings that Spark documents as fixed, such as spark.sql.shuffle.partitions, unchanged.
• If a checkpoint can't be saved, start a new one deliberately and choose the source's starting position, because the old offsets and state don't carry over.

Why is my streaming checkpoint corrupted after a restart?

Usually the checkpoint isn't damaged at all. Structured Streaming refuses to restart when what it finds in the checkpoint (the metadata file with the query ID, the offset log, the commit log and the state files) doesn't line up with the query, settings or Spark version that's starting. Four causes cover most cases: two runs writing to one checkpoint, files that were deleted or never durably written, configuration or version changes between runs, and genuine damage that newer Spark releases detect with checksums. The error class names the cause, so start there.

What a restart reads from the checkpoint

On restart, Spark reads the query ID from the metadata file, finds the last batch in the offset log, and checks the commit log to see whether that batch reached the sink. A batch that was logged but not committed runs again, which is normal: Structured Streaming expects micro-batches to be replayable, and exactly-once output depends on idempotent sinks. Stateful queries then load their state store from the files under the checkpoint. If any piece is missing, written by another run, or written under other settings, the query stops.

Match the error to the cause

Symptom (Spark error class)	Likely cause	Fix
CONCURRENT_STREAM_LOG_UPDATE or CONCURRENT_QUERY	Two runs share one checkpoint, for example an old run that never stopped	Stop the extra run. One query per checkpoint location, always
STREAMING_CHECKPOINT_MISSING_METADATA_FILE	The metadata file is gone but offset or commit logs remain	Restore the file or start a new checkpoint
STREAMING_STATEFUL_OPERATOR_MISSING_STATE_DIRECTORY, or CANNOT_LOAD_STATE_STORE with sub-condition CANNOT_READ_MISSING_SNAPSHOT_FILE	State files were deleted, or a stateful operator was added to a query that had none	Restore the files or start a new checkpoint
STATE_STORE_INVALID_CONFIG_AFTER_RESTART	A setting changed that must stay fixed for the checkpoint's life	Set it back to the old value, or use a new checkpoint
CANNOT_LOAD_STATE_STORE with sub-condition INVALID_CHANGE_LOG_READER_VERSION	The checkpoint was written by a newer Spark version	Restart on the same or a newer version
IllegalStateException: batch x doesn't exist after turning off asyncProgressTrackingEnabled	Async progress tracking doesn't checkpoint every batch	Re-enable it, set asyncProgressTrackingCheckpointIntervalMs to 0, run at least two micro-batches, then disable it
CHECKPOINT_FILE_CHECKSUM_VERIFICATION_FAILED, STATE_STORE_ROW_CHECKSUM_VERIFICATION_FAILED	A file or state row really is damaged	Restore from a copy if you have one; otherwise start a new checkpoint

Spark's guide names spark.sql.shuffle.partitions, spark.sql.streaming.stateStore.providerClass and spark.sql.streaming.multipleWatermarkPolicy as settings you can't modify after the query has run. State is hash-partitioned by key, so the partition count is part of the checkpoint. Changing stateful operators (grouping keys, deduplication columns, join type) breaks recovery the same way, but that's a code-change problem.

Storage rules that prevent most of it

• Keep the checkpoint in a directory on an HDFS-compatible, fault-tolerant file system, as the Spark guide requires. With the default checkpoint file manager, the cloud-integration guide adds that the store needs a fast, atomic rename(); otherwise checkpointing "may be slow and potentially unreliable".
• If you set spark.sql.streaming.checkpointFileManagerClass to the abortable stream-based checkpoint file manager, which avoids slow renames on object storage, isolate checkpoints strictly: the guide warns that reusing a location across parallel queries could corrupt checkpoint data.
• Always set checkpointLocation explicitly, and don't edit, move or prune files inside it.

Inspect the checkpoint before you delete anything

The State Data Source, experimental and read-only, lets a batch query read what the checkpoint holds:

# Operators, state stores and batch IDs recorded in the checkpoint (Spark 4.0+)
spark.read.format("state-metadata").load(checkpoint_path).show()

# The state key-value rows themselves; add .option("batchId", n) for a past batch
spark.read.format("statestore").load(checkpoint_path).show()

If the checkpoint can't be repaired, start a new one and set where the source resumes, since the old offsets are lost with it:

(spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", brokers)
    .option("subscribe", "events")
    .option("startingOffsets", "earliest")   # or a JSON map of per-partition offsets
    .load()
    .writeStream
    .option("checkpointLocation", new_checkpoint_path)
    .toTable("events_bronze"))

Starting too early reprocesses records the sink already holds; starting too late skips them. For a Delta Lake source, startingVersion plays the same role.

How Databricks handles checkpoint failures

In Lakeflow Pipelines, streaming-table checkpoints are internal to the pipeline. When one is corrupted or blocked by an incompatible change, Databricks documents three recovery paths: a full table refresh, a full refresh with backup and backfill, or a checkpoint reset for only the affected flows that keeps the table's existing data. See Recover a pipeline from streaming checkpoint failure.

FAQs

Is it safe to delete the checkpoint and restart?

Only if you choose the restart position. Deleting it makes the next run begin fresh, and for Kafka, startingOffsets applies only when a new query starts. Running aggregates and other state start over too.

Why did my query reprocess a batch after the restart?

That's expected behavior. A batch whose offsets were logged but not committed runs again so the output stays correct, which is why sinks need to tolerate replays.

Can two queries share a checkpoint location?

No. Each query needs its own checkpoint location; two runs on one location is exactly what CONCURRENT_STREAM_LOG_UPDATE reports.

Does Spark detect damaged state files now?

Increasingly. The Spark 4.1.0 release notes list file-level checksum verification for the RocksDB state store provider and automatic snapshot repair for the state store, and Spark 4.2.0 adds a state store row checksum for corruption detection.