Why is streaming lag building up (backpressure)?

Summary

• Lag builds when each micro-batch takes longer to process than new data takes to arrive. The unprocessed data waits in the source, so the backlog grows there.
• Confirm it with source backlog metrics such as Kafka maxOffsetsBehindLatest or Delta numFilesOutstanding, and with inputRowsPerSecond and processedRowsPerSecond.
• Typical causes: too little compute, Kafka reads capped by the partition count, oversized batches that spill, and too few shuffle partitions in a stateful query.
• Fixes: add compute, cap batch size with options such as maxOffsetsPerTrigger, raise Kafka read parallelism with minPartitions, and set shuffle partitions before the first run.

Why is streaming lag building up (backpressure)?

Lag builds when a Structured Streaming query needs longer to process each micro-batch than new data takes to arrive. Data the query has not processed yet waits in the source until a micro-batch reads it, so the backlog grows in the Kafka topic, the file set or the Delta table.
"Backpressure" has a narrower meaning in Spark's older Spark Streaming (DStream) API, which the Spark docs now call a legacy project. There, spark.streaming.backpressure.enabled lets Spark adjust the receiving rate automatically. In Structured Streaming, sources such as Kafka and Delta Lake have their own limits on how much one micro-batch reads.

How do I confirm the stream is falling behind?

In Python, query.lastProgress returns a dictionary describing the last trigger, and query.recentProgress returns the last few. Read inputRowsPerSecond (the rate of data arriving), processedRowsPerSecond (the rate at which Spark processes it) and batchDuration. The durationMs entry splits a trigger into steps. Databricks documents triggerExecution (plan and execute the micro-batch), addBatch (execute the micro-batch, excluding planning), latestOffset and commitBatch (commit the data written to the sink during addBatch), so the step that grows shows where to look. The Spark UI's Structured Streaming tab charts Input Rate, Process Rate and Batch Duration.
The most direct measure of lag is the backlog at the source. Databricks reports it in the sources metrics of the progress output:

Source	Backlog metrics
Apache Kafka	avgOffsetsBehindLatest, maxOffsetsBehindLatest, minOffsetsBehindLatest, estimatedTotalBytesBehindLatest
Delta Lake and Auto Loader	numFilesOutstanding, numBytesOutstanding

A backlog that keeps growing from batch to batch means the query is losing ground. On Databricks Runtime 17.0 and below, the latest Kafka offsets are fetched at micro-batch start time, so Kafka backlog metrics can return 0 when the query consistently consumes everything available at that moment. On 17.1 and above they are fetched after each micro-batch completes.

What usually causes lag?

Likely cause	Where to look	What to change
Not enough compute for each batch	The Spark UI Stages tab reports scheduler delay, GC time and shuffle spill	Add workers or use larger instances
Kafka reads limited by partition count	By default, one Spark partition per Kafka topic partition	Set minPartitions above the topic's partition count
Oversized batches after a backlog	numInputRows and batchDuration jump, and spill shows on the Stages tab	Cap each batch with a source option
Too few shuffle partitions in a stateful query	The partition count is below the cluster's core count	Set it before the first run

How do I fix it?

Add compute the stream can use

On Databricks, the Structured Streaming production guide says "Don't enable autoscaling for compute for Structured Streaming jobs" and recommends Lakeflow Pipelines with enhanced autoscaling for streaming workloads. Enhanced autoscaling scales on task slot utilization and task queue size.

Cap the batch size

Databricks says admission controls can help "maintain a consistent batch size and prevent large batches from causing spill and cascading micro-batch processing delays". The option depends on the source:

• Kafka: maxOffsetsPerTrigger is a rate limit on the maximum number of offsets processed per trigger interval, split proportionally across topic partitions.
• Delta Lake and Auto Loader: maxFilesPerTrigger (cloudFiles.maxFilesPerTrigger for Auto Loader) sets an upper bound on files per micro-batch, and the Delta default is 1000. maxBytesPerTrigger is a soft maximum with no default. With both set, the micro-batch stops at the lower limit of either.

A cap keeps batches a consistent size. It does not add throughput, so the backlog drains only when the query processes data at a higher rate than it arrives.

Raise Kafka read parallelism

By default, Spark has a 1-1 mapping of Kafka topic partitions to Spark partitions. If you set minPartitions to a value greater than the topic's partition count, Spark divides large Kafka partitions into smaller pieces. The Spark docs call it a hint: the number of tasks will be approximately minPartitions.

Set shuffle partitions before the first run

For stateful queries, Databricks states that the number of shuffle partitions is fixed at the time the checkpoint is created, and recommends 1-2 times the number of cores in the cluster. The Spark guide lists spark.sql.shuffle.partitions (default 200) among the configurations that cannot be modified after the query has run, because state is partitioned by a hash of the key, and says to discard the checkpoint and start a new query to change it.

Put it together

df = (spark.readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
      .option("subscribe", "topic1")
      .option("minPartitions", 48)              # example values:
      .option("maxOffsetsPerTrigger", 1000000)  # size them to your workload
      .load())

query = (df.writeStream
         .format("delta")
         .option("checkpointLocation", "/tmp/delta/events/_checkpoints/")
         .toTable("events"))

print(query.lastProgress)

FAQs

Will a longer trigger interval clear the backlog?

Not once batches already run back to back. The Spark guide says that if the previous micro-batch takes longer than the interval, "the next micro-batch will start as soon as the previous one completes". A longer interval only adds idle time when batches finish early.

Can a growing backlog lose data?

For Kafka, yes, if the stream falls behind retention. The topic setting retention.ms "represents an SLA on how soon consumers must read their data", because old log segments are discarded after that time under the "delete" policy. The Kafka source option failOnDataLoss (default true) fails the query when data may be lost, for example when offsets are out of range.