What is a streaming checkpoint?

Summary

• A streaming checkpoint is the durable record a Structured Streaming query keeps of its progress and, for stateful queries, its state, so the query can restart where it left off after a failure or a planned shutdown.
• A checkpoint directory holds offsets (the source offsets processed in each micro-batch), commits (which micro-batches reached the sink), state, and query metadata.
• End-to-end exactly-once processing needs more than a checkpoint: Spark combines checkpointing and write-ahead logs with replayable sources and idempotent sinks.
• Each query needs its own checkpoint location. Deleting the files or switching to a new location makes the next run start fresh.
• On Databricks, you set checkpointLocation yourself in Structured Streaming code, while Auto Loader in Lakeflow Pipelines needs no checkpoint location because the pipeline manages it.

What is a streaming checkpoint?
A streaming checkpoint is the information a Structured Streaming query saves so it can recover after a failure or an intentional shutdown. Per the Apache Spark guide, the query saves its progress information (the range of offsets processed in each trigger) and its running aggregates to the checkpoint location. That location has to be a path in an HDFS-compatible file system, and the guide names HDFS, AWS S3, and Azure Blob storage as examples of fault-tolerant storage for state.

What is inside a checkpoint?

Component	What it records	Why it matters
Offsets	The source offsets processed in each micro-batch	Lets the query resume from where it left off without reprocessing data
Commits	Which micro-batches have been committed to the sink	Supports exactly-once semantics
State	For stateful queries, metadata about the operator, the state schema, and the checkpointed state store contents	Restores aggregations, stream-stream joins, deduplication, and custom stateful operators
Metadata	The unique query ID (configuration settings are stored as part of the offset log)	Identifies the query across restarts

A stateless query only tracks which rows have been processed from the source to the sink, so its checkpoint has no state to restore.

How do checkpoints support exactly-once processing?

Structured Streaming assumes every source has offsets, similar to Kafka offsets, that track the read position. The engine uses checkpointing and write-ahead logs to record the offset range of the data being processed in each trigger. Sinks are designed to be idempotent for handling reprocessing. Together, replayable sources and idempotent sinks let Structured Streaming provide end-to-end exactly-once semantics under any failure.
The guarantee depends on the sink:

• The file sink is exactly-once. The Kafka sink and the foreach sink are at-least-once.
• The foreachBatch sink depends on your implementation.
• The console and memory sinks are not fault-tolerant.
• Continuous Processing, an experimental trigger, gives at-least-once guarantees.

How do I set a checkpoint location?

Pass checkpointLocation as a writer option when you start the query. This example reads from Kafka and writes Parquet files:

df = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
    .option("subscribe", "topic1")
    .load())

query = (df.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
    .writeStream
    .format("parquet")
    .option("path", "path/to/destination/dir")
    .option("checkpointLocation", "path/to/checkpoint/dir")
    .start())

Spark also has a spark.sql.streaming.checkpointLocation setting for the default location of checkpoint data, which is unset by default.

What can I change between restarts?

Changes to the number or type of input sources are not allowed. Rate limits such as maxOffsetsPerTrigger can change, and filters can be added or removed. Changing the schema of a stateful operation, such as the grouping keys of an aggregation or the join type of a stream-stream join, is not allowed. Spark 4.2.0 adds streaming source naming (DataStreamReader.name()) so named sources can be added, removed, and reordered without breaking checkpoints.

How does Databricks handle checkpoints?

Databricks documents the same checkpoint contents (offsets, commits, state, and metadata). Its example points checkpointLocation at a Unity Catalog volume path:

(df.writeStream
    .option("checkpointLocation", "/Volumes/catalog/schema/volume/path")
    .toTable("catalog.schema.table")
)

Some sinks, such as the output of display() in notebooks and the memory sink, create a temporary checkpoint location if you omit the option. Those temporary locations do not ensure fault tolerance or data consistency and might not be cleaned up properly, so Databricks recommends always setting one.
Databricks Runtime 18.2 and above supports source evolution with named sources, which requires a fresh checkpoint. In Lakeflow Pipelines, you do not provide a checkpoint location for Auto Loader because the pipeline manages that setting. A full refresh of a streaming table clears the data and the flow checkpoints and reprocesses all records from the source.

FAQs

What happens if I delete the checkpoint?

The next run of the query begins fresh. For a Kafka source, startingOffsets then applies, because it is used when a new query starts: its default is "latest" for streaming queries.

Can I use the same checkpoint for two queries?

No. Each query must have a different checkpoint location, and multiple queries should never share one.

How often is progress written?

Progress is recorded for each trigger, and the Spark web UI reports the walCommit time, the time taken to write the offsets to the metadata log. For stateless queries that write to Kafka, asyncProgressTrackingEnabled (default false) lets a query checkpoint progress asynchronously, with asyncProgressTrackingCheckpointIntervalMs controlling the interval. It does not support exactly-once end-to-end processing.