How do I migrate a micro-batch job to Real-Time Mode?

Summary

• Real-Time Mode is a trigger type in Apache Spark Structured Streaming. Spark 4.1.0 added it with Scala support for stateless queries, and Spark 4.2.0 added the PySpark trigger.
• The main code change is the trigger: processingTime="10 seconds" becomes realTime="5 minutes" in PySpark, or Trigger.RealTime("5 minutes") in Scala and Java. The value is the batch duration, and it also sets how often progress is checkpointed.
• Real-time mode schedules every stage of a query at once, so the cluster needs at least as many free task slots as there are tasks across all stages.
• On Databricks, real-time mode has been generally available since March 2026. It runs on classic compute with Databricks Runtime 16.4 LTS or above, with autoscaling, Photon and spot instances turned off and spark.databricks.streaming.realTimeMode.enabled set to true.
• Real-Time Mode is not Lakehouse//RT, the Beta SQL warehouse type for low-latency reads, and it is not the experimental continuous processing trigger.

How do I migrate a micro-batch job to Real-Time Mode?

Move to a Spark release that includes the trigger, confirm that the query's sources, sinks and operators are supported in real-time mode, size the cluster so every stage can run at the same time, and then swap the micro-batch trigger for the real-time trigger. If the query only uses supported operators, the transformation code doesn't change. The trigger, the output mode and the compute do. Databricks recommends benchmarking real-time mode against your target workload to pick the trigger interval, so test on a copy of the job before you cut over.

What is Real-Time Mode?

By default, Structured Streaming runs micro-batches: each one starts as soon as the previous one finishes, or on a fixed processingTime interval. Real-time mode is a different trigger type. The Databricks docs describe how it works:

• It runs long-running batches, five minutes by default, that process data as it becomes available in the source.
• It schedules all stages of the query simultaneously.
• It passes data between stages as soon as it is produced, through a streaming shuffle.

Structured Streaming checkpoints progress and publishes metrics between batches. Longer batches checkpoint less often, which means longer replays after a failure. Shorter batches checkpoint more often.
Upstream, the Spark 4.1.0 release notes list "Real-time Mode in Structured Streaming (Scala stateless support)" (SPARK-53736), and the Spark 4.2.0 notes list "Add Real-time Mode (RTM) trigger support to PySpark" (SPARK-54660). The PySpark trigger reference notes that the API is evolving.
Real-time mode is separate from continuous processing, which Spark has labelled experimental since 2.3 and which gives at-least-once guarantees. It is also unrelated to Lakehouse//RT, a Beta Databricks SQL warehouse type for low-latency read queries.

How do you change the trigger?

In PySpark (Spark 4.2.0 or above), a Kafka-to-Kafka query changes like this:

query = (
    df.writeStream
    .format("kafka")
    .option("kafka.bootstrap.servers", broker_address)
    .option("topic", output_topic)
    .option("checkpointLocation", checkpoint_location)
    .outputMode("update")
    # was: .trigger(processingTime="10 seconds")
    .trigger(realTime="5 minutes")
    .start()
)

In Scala or Java (Spark 4.1.0 or above), use the Trigger factory: df.writeStream.trigger(Trigger.RealTime("5 minutes")), or Trigger.RealTime() for the five-minute default. The Databricks Scala examples use RealTimeTrigger.apply() from org.apache.spark.sql.execution.streaming instead.

What should you check before you switch?

Check	What real-time mode expects	Documented by
Spark version	4.1.0 or above for Scala and Java; 4.2.0 or above for the PySpark realTime trigger	Apache Spark release notes
Query shape	4.1.0 ships "Scala stateless support"; Databricks also supports aggregations, tumbling and sliding windows, dropDuplicates and transformWithState	Apache Spark, Databricks
Output mode	Update only; append and complete aren't supported	Databricks
Sources and sinks	Apache Kafka as source and sink; Delta isn't supported as either; forEach works, forEachBatch doesn't	Databricks
Task slots	Available slots must equal or exceed the tasks across all stages in a batch, or the job isn't scheduled	Databricks
Batch duration	Sets checkpoint frequency, so it trades replay time after a failure against per-batch overhead	Apache Spark, Databricks

How Databricks approaches Real-Time Mode

Databricks announced general availability of real-time mode for Structured Streaming in March 2026, after a Public Preview that started in August 2025. The prerequisites:

• Classic compute in dedicated or standard access mode (standard is Python only). Serverless compute isn't supported.
• Databricks Runtime 16.4 LTS or above.
• Autoscaling, Photon and spot instances turned off.
• spark.databricks.streaming.realTimeMode.enabled set to true.

From Databricks Runtime 16.4 LTS, real-time pipelines use checkpoint v2, which allows switches between real-time and micro-batch modes, so you can move a query back if real-time mode doesn't fit.
Real-time mode in Lakeflow Pipelines is in Public Preview on Databricks Runtime 18.1.3 on the preview channel. You set the pipeline to continuous mode, add spark.databricks.streaming.realTimeMode.enabled to its Spark configuration, and define an update flow with @dp.update_flow and pipelines.trigger set to "RealTime".
Databricks doesn't support or recommend continuous processing, and its docs point low-latency workloads to real-time mode instead.

FAQs

Is Real-Time Mode only available on Databricks?

No. It's part of Apache Spark Structured Streaming from 4.1.0 (Scala and Java, stateless queries) and 4.2.0 (the PySpark trigger). Databricks documents its own operator, source and sink support on top of that.

Can I keep my existing checkpoint?

On Databricks Runtime 16.4 LTS and above, checkpoint v2 allows switches between real-time and micro-batch modes. The upstream docs don't describe checkpoint compatibility for real-time mode, so test a restart against a copy of the checkpoint first.

Does real-time mode run on serverless compute?

No. On serverless compute, Databricks supports only the AvailableNow and Once triggers.

What happens to a Delta sink?

Delta isn't supported as a real-time mode source or sink on Databricks. Keep Delta stages on micro-batch triggers, or write to Kafka from the real-time query.

Is Real-Time Mode the same as Lakehouse//RT?

No. Real-time mode is a Structured Streaming trigger. Lakehouse//RT is a Beta SQL warehouse type that serves low-latency reads on Unity Catalog tables.