Real-Time Mode versus micro-batch: latency and cost?

Summary

• Real-Time Mode (RTM) is a trigger in Apache Spark Structured Streaming. It first shipped in Spark 4.1.0 for stateless Scala queries (SPARK-53736), Spark 4.2.0 added the PySpark trigger, and on Databricks it has been generally available since March 2026.
• RTM cuts latency with long-running batches, all stages scheduled at once, and a streaming shuffle that passes records on as soon as they're produced. Micro-batch runs one batch at a time, stage after stage.
• The cost is held capacity: RTM needs a task slot for every task of every stage for the whole batch, even while tasks sit idle waiting for data. Micro-batch can run on a schedule and stop.
• Batch duration is the main RTM dial. Longer batches checkpoint less often, which means longer replay after a failure and later metrics; shorter batches checkpoint more often, which can add latency.
• Choose RTM for per-event operational decisions such as fraud detection or personalization. Keep micro-batch, the default trigger, for ETL and analytics where a delay of seconds or minutes is acceptable.

Real-Time Mode versus micro-batch: latency and cost?

Real-Time Mode is Structured Streaming's low-latency trigger, and you pay for it with compute that stays reserved whether or not data is arriving. Micro-batch, the default trigger, runs one batch at a time with its stages in sequence, so its latency includes those waits, but its resources are free between batches. The rule: if the business acts on each event the moment it lands, use Real-Time Mode; if a pipeline feeds tables and dashboards, micro-batch is the cheaper, simpler choice.

What Real-Time Mode is, and who ships it

Real-Time Mode (RTM) is a Structured Streaming trigger, not a separate engine. Upstream, SPARK-53736 ("SPIP: Real-time Mode in Structured Streaming") shipped in Spark 4.1.0 with Scala stateless support, and Spark 4.2.0 added RTM trigger support to PySpark (SPARK-54660). The PySpark reference marks the trigger API "evolving," and the 4.2 programming guide's trigger table doesn't list the mode yet, so treat upstream support as early. In code, RTM is one trigger setting:

df.writeStream.trigger(processingTime="1 minute")  # micro-batch: a new batch every minute
df.writeStream.trigger(realTime="5 minutes")       # RTM (PySpark 4.2+): batch duration

In Scala and Java the trigger is Trigger.RealTime(), whose default batch duration is 5 minutes. On Databricks, RTM in Structured Streaming is generally available on Databricks Runtime 16.4 LTS and above.

How the two modes compare

Aspect	Micro-batch (default)	Real-Time Mode
Scheduling	One batch at a time; stages run in sequence	Long-running batches with all stages scheduled at once
Data between stages	A stage finishes before the next one reads its output	Streaming shuffle passes data on as soon as it's produced
What drives latency	Waiting for the next batch, then running its stages in order	Processing time per record, with no wait for a batch boundary
Checkpointing	After every batch	Between long-running batches; batch duration sets the frequency
Compute footprint	Used only while a batch has work; Available-now runs can process what's there and stop	Task slots for every task of every stage, held for the whole batch
Failure recovery	Replays from the last committed batch	Longer batches mean longer replay
Typical fit	ETL, medallion pipelines, analytics	Fraud detection, real-time personalization, operational triggers

Where the latency comes from

Databricks' documentation describes three execution changes: long-running batches "in which the system processes data as it becomes available in the source," all stages scheduled simultaneously, and data passed between stages "as soon as it is produced using a streaming shuffle." Together, those remove the batch-boundary wait and the stage-by-stage barrier that bound micro-batch latency.
Batch duration still matters, because checkpoints happen between batches. Longer batches mean "longer replays on failure and delayed metrics availability"; shorter ones checkpoint more often, which "may affect latency." Databricks recommends benchmarking your own workload to pick the interval.

Where the cost comes from

RTM's bill is driven by reserved capacity. Simultaneous scheduling "requires the number of available task slots to be equal to or greater than the number of tasks of all the stages in a batch," and a query won't start without them. Real-time tasks can also stay idle while waiting for data, so right-sizing decides whether you pay for work or for waiting. Databricks lists cost optimization as a reason to stay on micro-batch, because real-time mode "requires dedicated compute resources."
Micro-batch has more ways to save. An Available-now trigger processes everything that has arrived, in multiple batches, and then stops, so the cluster can shut down between runs.

How Databricks approaches Real-Time Mode

For Structured Streaming jobs on Databricks, the setup page requires classic compute with dedicated or standard access mode (standard supports Python only), autoscaling turned off, Photon turned off, spot instances turned off, and spark.databricks.streaming.realTimeMode.enabled set to true. Real-time triggers run in update output mode only. To keep cost in check, tune maxPartitions for Kafka sources and spark.sql.shuffle.partitions for shuffle stages, then watch busyTimeFraction: values near 1 mean tasks are fully used, lower values mean idle or blocked time. For latency, use the progress event's processingLatencyMs, sourceQueuingLatencyMs and e2eLatencyMs, since batch-duration metrics don't reflect end-to-end latency in this mode. Real-time mode inside Lakeflow pipelines is a separate Public Preview.

FAQs

Is Real-Time Mode part of open-source Apache Spark?

Yes. It shipped in Spark 4.1.0 for stateless Scala queries, and Spark 4.2.0 added the PySpark trigger. Check the release notes for what your version supports.

Is Real-Time Mode the same as continuous processing?

No. They're separate triggers. trigger(continuous=...) is the older continuous processing mode, which the Structured Streaming guide still labels experimental; trigger(realTime=...) is Real-Time Mode.

Can one query switch between the two modes?

On Databricks, yes: from Databricks Runtime 16.4 LTS, all real-time pipelines use checkpoint v2, which lets them switch between real-time and micro-batch modes.