What is Real-Time Mode, and when did it ship?

Summary

• Real-Time Mode (RTM) is a trigger type in Apache Spark Structured Streaming for operational, low-latency streams. It replaces short micro-batches with long-running batches that process records as they arrive.
• It first shipped in open-source Apache Spark 4.1.0 on December 16, 2025, as "Real-time Mode in Structured Streaming (Scala stateless support)". Spark 4.2.0, released July 14, 2026, added the PySpark trigger.
• Databricks had it first: real-time mode in Structured Streaming entered Public Preview on Databricks on July 22, 2025, Databricks said in August 2025 that it was being contributed to Apache Spark, and Databricks announced general availability on March 19, 2026.
• You enable it per query with Trigger.RealTime(...) in Scala and Java or trigger(realTime=...) in PySpark, and the query must use update output mode.
• It is not the older experimental continuous processing trigger, and it is not Lakehouse//RT, which is a Databricks SQL warehouse type.

What is Real-Time Mode, and when did it ship?

Real-Time Mode is a Structured Streaming trigger for workloads that must act on events as they arrive, such as fraud detection or live personalization. In open-source Apache Spark it first shipped in 4.1.0 on December 16, 2025, with Scala support for stateless queries, and Spark 4.2.0 added a PySpark trigger on July 14, 2026. Databricks offered it earlier, as a Public Preview starting July 22, 2025, and announced general availability on March 19, 2026.

What is Real-Time Mode?

By default, Structured Streaming runs micro-batches: each batch starts after the previous one completes, and its stages run one after another. Real-Time Mode changes that execution model. Databricks' documentation describes three mechanisms:

• Long-running batches. A batch stays open, five minutes by default, and processes data as it becomes available in the source.
• All stages scheduled at once. Every stage of the query runs at the same time, so the cluster needs at least as many free task slots as there are tasks across all stages.
• Streaming shuffle. Data passes between stages as soon as it's produced.

Structured Streaming checkpoints progress and publishes metrics between batches, so the batch duration is a trade-off. Longer batches checkpoint less often, which means longer replays after a failure and later metrics; shorter batches checkpoint more often, which can affect latency.

Using Real-Time Mode in open-source Spark

In Scala and Java, the trigger is Trigger.RealTime. Called with no argument it uses a five-minute batch, and overloads accept a duration string such as "10 seconds", milliseconds, a length with a TimeUnit, or a Scala Duration. Spark 4.1.0 marks these methods experimental. The Spark 4.1.0 allowlist for real-time sinks covers Kafka, foreach writers and the console sink, and Kafka is a supported source:

import org.apache.spark.sql.streaming.Trigger

val query = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", brokers)
  .option("subscribe", "events")
  .load()
  .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
  .writeStream
  .format("kafka")
  .option("kafka.bootstrap.servers", brokers)
  .option("topic", "events-out")
  .option("checkpointLocation", checkpointPath)
  .outputMode("update")
  .trigger(Trigger.RealTime("5 minutes"))
  .start()

From Spark 4.2.0, PySpark takes the same setting as writeStream.trigger(realTime="5 minutes"), a batch duration string. Upstream rules show up as errors under STREAMING_REAL_TIME_MODE: only Update output mode is supported, asynchronous progress tracking must be off, and operators and sinks are checked against an allowlist controlled by spark.sql.streaming.realTimeMode.allowlistCheck. A trigger interval shorter than spark.sql.streaming.realTimeMode.minBatchDuration is rejected.

Release history

Date	Release or milestone	What changed
July 22, 2025	Databricks release notes	Real-time mode in Structured Streaming enters Public Preview
August 19, 2025	Databricks blog	Announces the mode is "being contributed to open source Apache Spark"
December 16, 2025	Apache Spark 4.1.0	SPARK-53736: Real-time Mode in Structured Streaming (Scala stateless support)
March 19, 2026	Databricks blog	Announces general availability of real-time mode in Structured Streaming on Databricks
May 27, 2026	Databricks release notes	Real-time mode in Lakeflow Pipelines enters Public Preview, with the update_flow API
July 14, 2026	Apache Spark 4.2.0	SPARK-54660: Real-time Mode trigger support in PySpark

How Databricks approaches Real-Time Mode

On Databricks, real-time mode for Structured Streaming runs on classic compute with Databricks Runtime 16.4 LTS and above, in dedicated or standard access mode (standard supports Python only). Serverless compute isn't supported. The setup docs ask you to turn off autoscaling, Photon and spot instances and to set spark.databricks.streaming.realTimeMode.enabled to true. Databricks documents Python, Scala and Java support, Apache Kafka as a source and sink, and arbitrary sinks through forEachWriter. Its examples write the trigger as .trigger(realTime="5 minutes") in Python and RealTimeTrigger.apply() in Scala. Delta isn't supported as a source or a sink in real-time mode.
In Lakeflow Pipelines, real-time mode is in Public Preview on Databricks Runtime 18.1.3 on the preview channel. The pipeline must run in continuous mode, and the output is a real-time update flow, defined with the @dp.update_flow decorator, that writes to a sink created with dp.create_sink().

FAQs

Is Real-Time Mode the same as continuous processing?

No. Continuous processing is an older trigger, Trigger.Continuous, added in Spark 2.3.0, and the Structured Streaming guide still labels it experimental. Real-Time Mode uses a separate trigger, Trigger.RealTime, introduced in Spark 4.1.0.

Is Real-Time Mode the same as Lakehouse//RT?

No. Lakehouse Real-Time (Lakehouse//RT) is a Beta Databricks SQL warehouse type for low-latency, high-concurrency read queries. Real-Time Mode is a Structured Streaming trigger that processes streams.

Which output modes does it support?

Update only. Upstream Spark rejects other modes with an error that points you to Update, and Databricks documents update mode as the only supported mode.

Does it support stateful queries?

The Spark 4.1.0 release covers Scala stateless queries. On Databricks, the reference lists supported aggregation functions, and stream-to-stream inner joins work with extra configuration; outer joins aren't supported.