What's the difference between micro-batch, continuous, and Real-Time Mode?

Summary

• Micro-batch, continuous processing and Real-Time Mode are three ways Apache Spark Structured Streaming can execute the same query. The trigger you set on writeStream picks the mode.
• Micro-batch is the default. Spark runs the stream as a series of small batch jobs, logs each batch's offsets before running it, and documents end-to-end exactly-once guarantees.
• Continuous processing has been experimental since Spark 2.3. It runs long-lived tasks with at-least-once guarantees, supports only map-like queries, and isn't supported on Databricks.
• Real-Time Mode runs long-running batches that schedule every stage at once and stream rows between stages. Spark 4.1.0 added it for stateless Scala queries, Spark 4.2.0 added the PySpark trigger, and it's generally available on Databricks.

What's the difference between micro-batch, continuous, and Real-Time Mode?

They're three execution modes behind one Structured Streaming API, and they differ in how Spark schedules work and records progress. Micro-batch runs a stream as a sequence of small batch jobs and is the right default for ETL and analytics. Continuous processing swaps batches for long-lived tasks, but it has stayed experimental and narrow. Real-Time Mode is the current option for operational, latency-sensitive streams: it keeps one batch open for minutes, runs all of its stages at the same time, and hands rows to the next stage as soon as they're produced.

How each mode runs a query

Micro-batch. With no trigger set, Spark starts the next micro-batch as soon as the previous one completes. Each batch writes the range of offsets it will read to the checkpoint's write-ahead log, runs as an ordinary Spark job, and is marked committed after the sink write. That bookkeeping is how Structured Streaming delivers exactly-once results with replayable sources and idempotent sinks. processingTime spaces batches at a fixed interval and skips a batch when no new data has arrived. availableNow drains everything present at start, in one or more batches, then stops. once still works but is deprecated in favor of availableNow.
Continuous processing. A continuous trigger launches long-running tasks that continuously read, process and write data, and it checkpoints progress asynchronously at the interval you pass. The guarantee drops to at-least-once. The Spark guide limits it to projections and selections (no aggregations, current_timestamp() or current_date()), Kafka and rate sources, and Kafka, memory and console sinks. Every task needs its own core, and a failed task stops the query instead of being retried, so you restart it from the checkpoint yourself.
Real-Time Mode. The Databricks documentation describes three execution changes. A batch runs for a long time (five minutes by default) and processes records as they arrive. Every stage of the query is scheduled at once, so you need at least as many task slots as there are tasks across all stages. A streaming shuffle passes data between stages immediately. Progress is checkpointed between batches, so a longer batch duration means fewer checkpoints and longer replays after a failure.

# Micro-batch: default, fixed interval, or drain-and-stop
df.writeStream.format("console").start()
df.writeStream.trigger(processingTime="2 seconds").format("console").start()
df.writeStream.trigger(availableNow=True).format("console").start()

# Continuous processing (experimental): async checkpoint every second
df.writeStream.trigger(continuous="1 second").format("console").start()

# Real-Time Mode: long-running batches of the given duration
(df.writeStream.format("kafka")
    .option("kafka.bootstrap.servers", brokers)
    .option("topic", "out")
    .option("checkpointLocation", checkpoint_path)
    .outputMode("update")
    .trigger(realTime="5 minutes")
    .start())

In Apache Spark's Scala and Java API the same choices are Trigger.ProcessingTime, Trigger.AvailableNow(), Trigger.Continuous and Trigger.RealTime; Databricks' setup guide shows RealTimeTrigger.apply() for Scala.

Side-by-side comparison

Aspect	Micro-batch	Continuous processing	Real-Time Mode
Trigger	default, processingTime, availableNow	continuous	realTime
Upstream status	Default engine	Experimental since Spark 2.3	Added in Spark 4.1.0 (stateless, Scala); PySpark trigger in 4.2.0
Unit of work	Short batch jobs, one after another	Long-running tasks, no batches	Long-running batches with all stages active at once
When progress is saved	Every micro-batch (offset log, then commit)	Asynchronously, at the checkpoint interval	Between batches
After a failure	Re-runs the uncommitted batch	Query stops; restart from the checkpoint	Replays from the last checkpoint
Query support	Full API, including stateful operators	Map-like operations only	Update output mode; on Databricks, aggregations, windows, deduplication and transformWithState
Sizing rule	Standard task scheduling	One core per long-running task	A task slot for every task in every stage
On Databricks	Supported	Not supported	Generally available on classic compute

How Databricks approaches low-latency streaming

Databricks announced general availability of Real-Time Mode in March 2026, and its trigger documentation says it doesn't support or recommend continuous processing: use Real-Time Mode instead. To run it, use classic compute on Databricks Runtime 16.4 LTS or above, turn off autoscaling, Photon and spot instances, and set spark.databricks.streaming.realTimeMode.enabled to true. Queries use update output mode. Apache Kafka works as a source and a sink, Delta Lake isn't a supported source or sink in this mode, and foreachBatch isn't available, so custom writers go through foreach. On serverless compute, Structured Streaming accepts only the availableNow and once triggers.

FAQs

Which mode should I use by default?

Micro-batch. It supports the whole Structured Streaming API and exactly-once delivery. Databricks positions Real-Time Mode for operational workloads that need an immediate response, such as fraud detection, and notes that it requires dedicated compute.

Can I switch an existing query between modes?

Between micro-batch triggers, yes: Databricks documents changing the trigger between runs on the same checkpoint, and the Spark guide says a supported query can move between micro-batch and continuous mode on restart. Check Real-Time Mode's operator and sink support before moving a query to it.

Is Real-Time Mode the same as Lakehouse//RT?

No. Real-Time Mode is a Structured Streaming trigger. Lakehouse//RT is a separate Databricks SQL warehouse type, in Beta, that serves read queries over Delta Lake and Apache Iceberg tables.

Is continuous processing the same as a continuous pipeline?

No. Lakeflow Pipelines has its own continuous pipeline mode, and the Databricks trigger documentation states that continuous execution in pipelines is unrelated to the continuous processing trigger.