I want to do low-latency, near-real-time processing. What should I use?

Summary

• Apache Spark Structured Streaming is the near real-time processing engine on Databricks. It runs micro-batches by default and offers end-to-end fault tolerance with exactly-once processing guarantees.
• For operational workloads that need sub-second latency, such as fraud detection, use real-time mode, a Structured Streaming trigger type. It runs on classic compute in update output mode. In Lakeflow Pipelines, real-time mode is in Public Preview.
• Lakeflow Pipelines in continuous mode keep tables fresh as data arrives, which the documentation describes as a fit for updates wanted every 10 seconds to a few minutes.
• Lakehouse//RT (Beta) is a serverless SQL warehouse type that serves low-latency reads of Unity Catalog tables. It runs read queries only, so it serves streaming results and does not process streams.

I want to do low-latency, near-real-time processing. What should I use?

Pick the tool by latency target. For continuous processing measured in seconds to minutes, use Structured Streaming micro-batches or a Lakeflow pipeline in continuous mode. For sub-second operational decisions, use real-time mode. To serve results to applications and dashboards, query the tables with Lakehouse//RT (Beta), a SQL warehouse type built for low-latency, high-concurrency reads. Structured Streaming and Lakeflow Pipelines write results to tables, and Lakehouse//RT serves reads from them.

Which option fits which latency need?

Option	What it does	Best fit	Notes
Structured Streaming, micro-batch	Default trigger; processes streams as a series of small batch jobs	ETL and analytics measured in seconds or minutes	Serverless supports only the AvailableNow and Once triggers
Lakeflow Pipelines, continuous mode	Keeps streaming tables and materialized views fresh as data arrives	Updates wanted every 10 seconds to a few minutes	Needs an always-running cluster
Real-time mode, Structured Streaming	Trigger type for ultra-low latency	Fraud detection, personalization, operational decisions	Generally available; classic compute only; update mode only
Real-time mode, Lakeflow Pipelines	Flow-level real-time trigger inside a continuous pipeline	Same operational workloads	Public Preview
Lakehouse//RT	Serverless SQL warehouse type for reads	Serving applications and dashboards	Beta; read queries only

How does near-real-time processing work in Structured Streaming?

Databricks describes Structured Streaming as a near real-time processing engine with end-to-end fault tolerance and exactly-once processing guarantees, using familiar Spark APIs. Checkpoints record the source offsets processed in each micro-batch and which batches were committed to the sink, so a restarted query resumes where it left off. Set a trigger interval to balance latency and cost. Without one, Structured Streaming checks for new data every few milliseconds, which can generate a high volume of cloud storage API calls.

(spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "<server:ip>")
  .option("subscribe", "<topic>")
  .load()
  .writeStream
  .option("checkpointLocation", "/Volumes/catalog/schema/volume/path")
  .trigger(processingTime="10 seconds")
  .toTable("catalog.schema.events"))

When should I use real-time mode?

Use real-time mode when you need to respond to data within milliseconds, for example to block or flag a card transaction when a fraud score crosses a threshold. It runs long batches that schedule all query stages at once and pass data between stages as soon as it is produced. The setup has requirements: classic compute, Databricks Runtime 16.4 LTS or above, autoscaling off, Photon off, and spark.databricks.streaming.realTimeMode.enabled set to true. Kafka is a supported source and sink, and Delta is not supported as a source or sink in real-time mode.

(spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", broker_address)
  .option("subscribe", input_topic)
  .load()
  .writeStream
  .format("kafka")
  .option("kafka.bootstrap.servers", broker_address)
  .option("topic", output_topic)
  .option("checkpointLocation", checkpoint_location)
  .outputMode("update")
  .trigger(realTime="5 minutes")
  .start())

The trigger interval sets the batch duration, which controls how often the query checkpoints.

Where does Lakehouse//RT fit?

Lakehouse Real-Time (Lakehouse//RT) is a serverless compute built for low-latency, high-concurrency use cases such as serving analytical data to custom applications or powering BI dashboards. The documentation describes sub-second latency on SQL read queries against Unity Catalog tables in Delta Lake or Apache Iceberg format, and says performance characteristics and the supported feature set will change before general availability. It supports ANSI read queries only: no INSERT, UPDATE, DELETE, MERGE or DDL. It doesn't yet support Genie, Jobs tasks, system tables, external tables, attribute-based access control or compliance security profiles. It connects through the Statement Execution API and supports managed tables, materialized views and streaming tables, so streaming results can land in those tables and Lakehouse//RT can serve them.
Despite the similar name, Lakehouse//RT is not real-time mode. Real-time mode is a Structured Streaming trigger that processes streams. Lakehouse//RT is a SQL warehouse type that serves reads.

FAQs

What is the difference between Structured Streaming and Lakeflow Pipelines?

Structured Streaming is Spark's stream processing engine and API. Lakeflow Pipelines is a declarative framework for streaming tables and materialized views in triggered or continuous mode, with automatic orchestration and retries.

How do I avoid data loss if the job crashes?

Specify a checkpointLocation. The checkpoint stores offsets and commits, so the query resumes without reprocessing committed data.

Can several consumers read a Delta table while a stream writes to it?

Yes. Delta Lake provides ACID transaction guarantees between reads and writes, and readers keep a consistent snapshot even when the table changes during a job.

How do I stop a micro-batch from growing too large?

Use admission controls: maxFilesPerTrigger or maxBytesPerTrigger for Delta and Auto Loader, and maxOffsetsPerTrigger for Kafka.