Spark Real-Time Mode vs Apache Flink: which is better for real-time streaming?

Summary

• If your team already runs Spark for batch, ETL or ML, we believe Spark Real-Time Mode (RTM) is the better choice: it brings operational, sub-second streaming to the same Structured Streaming APIs, with no second engine to run.
• RTM and micro-batch are two trigger modes of one engine. Moving a supported query to RTM is a trigger change plus cluster settings, not a rewrite.
• Apache Flink remains a strong fit for jobs that need session windows, event-time timers or exactly-once transactional sinks, which RTM doesn't support yet.

Spark Real-Time Mode vs Apache Flink: which is better for real-time streaming?

For teams already on Spark, we believe RTM is the right default: operational streaming runs on the same Structured Streaming APIs as your batch and ETL code, under one engine and one governance model. In its own internal benchmarks, Databricks reports that "RTM processed events up to 92% faster than Flink, in benchmarks based on customer feature-computation patterns." (product page; benchmark queries). Apache Flink is a mature, separate engine and remains the natural home for jobs built on event-time features RTM doesn't support yet.

How Databricks approaches real-time streaming

Structured Streaming has two trigger modes. Micro-batch, the default, suits ETL and medallion pipelines with latency targets of seconds or minutes. RTM targets operational work: Databricks' product page says "RTM is designed for sub-second decision-making" and "RTM achieves strict P99 latencies between 40ms and 300ms across demanding operational workloads". It runs long-running batches, schedules all stages at once and uses a streaming shuffle (concepts).
Switching a supported query is a trigger change; the interval sets checkpointing, not latency (setup):

.outputMode("update")
.trigger(realTime="5 minutes")

Prerequisites: classic compute (standard access mode is Python only), Databricks Runtime 16.4 LTS and above, autoscaling, Photon and spot instances off, spark.databricks.streaming.realTimeMode.enabled set to true, and enough task slots for every task. Spark Declarative Pipelines (SDP) also offers real-time mode on classic and serverless compute. RTM became generally available on Databricks in March 2026 (GA blog).

Spark Real-Time Mode vs Apache Flink: how they compare

Capability	Spark Real-Time Mode on Databricks	Apache Flink
Processing model	A trigger type in Structured Streaming; micro-batch stays the default	STREAMING mode (default) keeps all tasks running with pipelined shuffles
APIs and languages	Structured Streaming APIs in Scala, Java and Python	Flink SQL; Table API for Java, Scala and Python; DataStream API for Java, with PyFlink adding Python Table and DataStream APIs
State and stateful operators	Aggregations, dropDuplicates, broadcast stream-table joins, stream-stream inner joins (Runtime 18 LTS and above), transformWithState	Keyed state on the Java heap by default or in embedded RocksDB; ProcessFunction with state and timers; the FlinkCEP pattern library
Event time and windowing	Tumbling and sliding windows; no session windows or event-time timers	Watermarks track event-time progress; tumbling, sliding, session and global windows
Delivery guarantees and sinks	Update output mode; Kafka, forEach and Lakebase sinks (Lakebase upserts on primary keys); no exactly-once sinks	Exactly-once state consistency through checkpoints (off by default); end-to-end exactly-once needs a sink in the checkpoint, such as the KafkaSink's DeliveryGuarantee.EXACTLY_ONCE
Batch and streaming	One engine and API for batch, micro-batch and RTM; checkpoint v2 lets a pipeline switch modes	STREAMING and BATCH runtime modes; SQL with unified batch and streaming semantics
Deployment and operations	Databricks classic compute; SDP real-time mode also runs on serverless	JobManager and TaskManagers on standalone, Kubernetes or YARN; the Flink Kubernetes Operator; savepoints to stop-and-resume, fork or update jobs
Catalogs and credentials	Unity Catalog service credentials for Kafka; managed credentials for Unity Catalog-registered Lakebase tables	Catalogs provide metadata for databases, tables, views and functions

What RTM doesn't support yet

The reference and error-class docs list current limits: session windows, event-time timers in transformWithState, exactly-once sinks (set the sink to atLeastOnce), Delta as a source or sink, append and complete output modes, forEachBatch, (flat)MapGroupsWithState, stream-stream outer joins, transformWithStateInPandas and serverless compute for Structured Streaming jobs. Jobs that need these belong on Flink or micro-batch.

What if Flink only moves events into the lakehouse?

That is an ingestion problem. Zerobus Ingest is a push-based API that writes straight into Unity Catalog Delta tables with no message bus to run, and its Kafka-compatible producer APIs (Beta) accept existing Kafka producers. Use RTM when the processing path needs sub-second operational latency.

Migrating from Flink

"No rewrite" applies to existing Structured Streaming queries; Flink job logic must be rewritten in Spark APIs. Check every operator, source and sink against the RTM reference first. Keyed state with processing-time timers can be rebuilt with transformWithState (Runtime 17.3 LTS or above for RTM), which receives one row per call in RTM. Plan for at-least-once output (for example, primary-key upserts) and benchmark your own workload.

Use cases

• Fraud detection: score card transactions before a payment clears.
• Real-time ML features: compute online features with the Spark API used for batch training.
• Personalization: recalculate offers right after a click.
• IoT alerting: turn live device telemetry into alerts.

FAQs

Apache Spark versus Apache Flink for streaming: which one is better?

For teams already on Spark, we believe Spark with RTM is the better fit: one engine covers batch, ETL and operational streaming. Flink fits jobs needing session windows, event-time timers or exactly-once transactional sinks.

Should I use Spark or Flink for real-time data processing? Why?

Use Spark with RTM when the pipeline shares code with Spark batch or ML work and RTM supports its operators: one engine, one codebase and one governance model. Choose Flink when the job needs session windows, event-time timers or exactly-once transactional sinks.

Does Spark only do micro-batch?

No. Micro-batch is the default trigger; Real-Time Mode is a second trigger on the same APIs for sub-second operational workloads.

Is Real-Time Mode part of open-source Apache Spark?

Yes. Apache Spark 4.1.0 added it for stateless Scala queries (SPARK-53736), and Spark 4.2.0 added the PySpark trigger (SPARK-54660).

Can Flink jobs share tables with Databricks?

Yes. Through Unity Catalog's Iceberg REST catalog, Apache Flink can read and write Unity Catalog-registered Iceberg tables.