When should I use Real-Time Mode?

Summary

• Use Real-Time Mode when a Structured Streaming query drives an immediate action, such as scoring a payment, raising an alert or feeding a live application, and the micro-batch cycle is what stands between you and the latency target. Keep micro-batch for ETL, medallion pipelines and analytics.
• Real-Time Mode is a trigger type in Apache Spark Structured Streaming. It shipped upstream in Apache Spark 4.1.0 for stateless Scala queries, Spark 4.2.0 added the PySpark trigger, and stateful support is still an open upstream epic.
• Each real-time batch is long-running, five minutes by default, and records are processed as they arrive inside it. The batch duration sets how often progress is checkpointed, not how long records wait.
• It asks for a narrower query shape and dedicated capacity: update output mode, Kafka or ForeachWriter sinks, and, per the Databricks docs, a task slot for every task across all stages.
• On Databricks, Real-Time Mode for Structured Streaming is generally available on classic compute with Databricks Runtime 16.4 LTS and above. It's a streaming trigger and has nothing to do with Lakehouse//RT.

When should I use Real-Time Mode?

Use Real-Time Mode when a streaming query feeds something that acts on each event right away, such as a fraud check, an alert, or a personalization or feature service, and the micro-batch cycle is the bottleneck in your end-to-end latency. If the output lands in tables for dashboards, reports or downstream ETL, keep the default micro-batch trigger. Real-Time Mode buys lower latency with dedicated compute and less frequent checkpoints, and that trade pays off only when something downstream reacts to individual records.

What is Real-Time Mode?

Real-Time Mode is a trigger type in Apache Spark Structured Streaming. A default micro-batch query starts each micro-batch as soon as the previous one completes. A real-time query instead runs long-lived batches, five minutes by default, and processes data as it becomes available within each batch. Between batches, Structured Streaming checkpoints progress and publishes metrics. So the trigger value is a batch duration, not a delay: a longer batch means fewer checkpoints, longer replays after a failure and later metrics, while records still flow as they arrive. Databricks' documentation describes two more mechanics on its platform: all stages of the query are scheduled at once, and data passes between stages through a streaming shuffle as soon as it's produced.
It's also a different trigger from continuous processing, the older mode set with trigger(continuous=...) that the Structured Streaming guide still marks experimental.

Where it ships, and at what stage

As of October 2026:

• Apache Spark 4.1.0 (December 2025) shipped "Real-time Mode in Structured Streaming (Scala stateless support)" (SPARK-53736), with Trigger.RealTime(...) and a Kafka source. The 4.1.0 source annotated those methods @Experimental; the 4.2.0 source no longer does.
• Apache Spark 4.2.0 (July 2026) added the PySpark trigger (SPARK-54660), trigger(realTime="...").
• Upstream stateful support, such as streaming aggregation and transformWithState, is tracked in SPARK-54699, an open epic with no fix version.
• Databricks announced general availability of Real-Time Mode for Structured Streaming on March 19, 2026. Real-time mode in Lakeflow Pipelines is a separate Public Preview.

Real-Time Mode or micro-batch?

Criterion	Real-Time Mode fits	Micro-batch fits
Who reads the output	A service or process that acts on each event	Tables, dashboards and downstream ETL
Latency requirement	Sub-second, end to end	Seconds or minutes
Sinks	Kafka, or a ForeachWriter you write	Delta Lake tables, files and the other standard sinks
Operators	The supported set, in update output mode	The full API, including foreachBatch, session windows and mapGroupsWithState
Compute	A dedicated, fixed-size cluster with a task slot for every task in every stage	Shared or autoscaling compute
Recovery	Can tolerate longer replays after a failure	Needs frequent checkpoints and quick recovery

Databricks' guidance lands in the same place. It recommends micro-batch for analytical processing such as ETL and medallion pipelines, for cost optimization when sub-second latency isn't required (because real-time mode needs dedicated compute), and for faster recovery.

Switching a query to Real-Time Mode

For a query that already fits the supported shape, the change is the trigger:

query = (
    scored.writeStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("topic", "payment-decisions")
    .option("checkpointLocation", "/checkpoints/payment-decisions")
    .outputMode("update")
    .trigger(realTime="5 minutes")
    .start()
)

In Scala on Apache Spark 4.1 and above, the same trigger is .trigger(Trigger.RealTime("5 minutes")), and Trigger.RealTime() with no argument uses the five-minute default. The Databricks docs show RealTimeTrigger.apply("5 minutes") for Scala. The Kafka sink requires a value column in the DataFrame.

How Databricks approaches Real-Time Mode

On Databricks, Real-Time Mode supports Scala, Java and Python. Setup is specific: classic compute in dedicated or standard access mode (standard supports Python only), Databricks Runtime 16.4 LTS and above, autoscaling, Photon and spot instances turned off, and spark.databricks.streaming.realTimeMode.enabled set to true. Serverless isn't supported. Beyond stateless queries, Databricks supports aggregations, tumbling and sliding windows, deduplication, some joins and transformWithState, in update output mode only. The GA announcement describes adoption as one cluster configuration plus the new trigger, with "no rewrites required" for existing Structured Streaming code.

FAQs

Is Real-Time Mode the same as continuous processing?

No. Continuous processing is an older, experimental trigger set with trigger(continuous=...). Real-Time Mode is its own trigger: trigger(realTime=...) in PySpark and Trigger.RealTime(...) in Scala.

Can I use Real-Time Mode with open-source Spark?

Yes. Apache Spark 4.1.0 supports it for stateless Scala queries and 4.2.0 adds it to PySpark. Stateful operators in Real-Time Mode are still in progress upstream.

Is Real-Time Mode the same as Lakehouse//RT?

No. Real-Time Mode is a Structured Streaming trigger that processes streams. Lakehouse//RT is a Databricks SQL warehouse type, in Beta, that serves low-latency read queries over Delta Lake or Apache Iceberg tables.