What are the best open source projects that I can use for low-latency data processing?

Summary

• Low-latency pipelines can be assembled from several open-source projects: a broker such as Apache Kafka or Apache Pulsar, a stream processor such as Kafka Streams, Apache Flink or Apache Spark Structured Streaming, and a store your application can query.
• Pick the processor by where its logic should live: Kafka Streams is a client library with no separate processing cluster, Flink is a distributed engine for stateful event-time processing, and Structured Streaming reuses batch-style DataFrame and SQL code.
• Structured Streaming runs micro-batches by default. Real-Time Mode, added in Apache Spark 4.1.0 for Scala stateless queries, is the upstream option for continuous, sub-second processing.
• Project documentation describes capabilities, not the latency you will see, so measure end-to-end latency on your own data.

What are the best open source projects that I can use for low-latency data processing?

No single project is best, because a low-latency pipeline has three jobs: moving events, processing them, and serving results. Apache Kafka or Apache Pulsar moves the events, a stream processor computes on them as they arrive, and a store your application can query holds the results. Choose the processor first, since it shapes how you deploy and operate the rest.

Which open source projects cover low-latency data processing?

Project	Role	What its own documentation says
Apache Kafka	Event streaming platform	Publishes and subscribes to event streams, stores them durably, and processes them as they occur or retrospectively
Apache Pulsar	Messaging and streaming platform	An open-source, distributed platform with tenants for multi-tenancy and replication across geographic regions
Kafka Streams	Stream processing library	Builds applications and microservices whose input and output data are stored in Kafka clusters, with no separate processing cluster
Apache Flink	Stream processing engine	Stateful computations over unbounded and bounded data streams, with event-time processing and late data handling
Apache Spark Structured Streaming	Stream processing engine	Express a streaming computation the way you would express a batch computation, run by the Spark SQL engine
Delta Lake	Open table storage	Extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling; tables work as streaming sources and sinks

How do the stream processors differ?

Kafka Streams is a client library, so it runs inside standard Java and Scala applications. Its documentation lists one-record-at-a-time processing, event-time windowing with out-of-order arrival of records, fault-tolerant local state and exactly-once processing. It fits applications whose input and output already live in Kafka.
Apache Flink is a distributed engine. Its documentation highlights exactly-once state consistency, event-time processing, late data handling, layered APIs (SQL, DataStream API and ProcessFunction), and support for very large state with incremental checkpoints.
Structured Streaming shares its API and engine with batch Spark, so one codebase can often serve both. It processes micro-batches by default.

Can Spark Structured Streaming do low-latency processing?

Yes, in Real-Time Mode (RTM). The Apache Spark 4.1.0 release notes describe RTM as the first official support for Structured Streaming queries running in real-time mode for continuous, sub-second latency processing, with Scala stateless support. The 4.2.0 notes add RTM trigger support to PySpark. On Databricks, real-time mode is a Structured Streaming trigger type for operational workloads such as fraud detection, and it supports Apache Kafka as a source and sink.

What does a low-latency architecture look like?

1. Source: applications, devices or sensors emit events.
2. Transport: Kafka or Pulsar accepts the events and delivers them to consumers.
3. Processing: Kafka Streams, Flink or Structured Streaming filters, enriches and aggregates them.
4. Serving: results land in a store that your application queries within its own latency budget.
5. Measurement: time each hop, then the whole path from event entry to result.

How do these projects work with Databricks?

You can combine these open-source projects with Databricks like this:

• Ingest: Structured Streaming reads Kafka with the kafka format (kafka.bootstrap.servers and subscribe options). Lakeflow Connect also lists streaming connectors for message buses such as Apache Kafka.
• Store: write results to Delta tables, where batch and streaming workloads share a single copy of the data.
• Govern: Unity Catalog captures lineage automatically for queries run on Databricks, including streaming between Delta tables.

FAQs

Can I use Spark Structured Streaming for sub-second latency?

Yes, with Real-Time Mode. Databricks documents micro-batch, the default trigger, for analytical work whose latency requirements are measured in seconds or minutes, and real-time mode for sub-second operational decisions.

Do I need a separate cluster to run Kafka Streams?

No. The documentation lists "no separate processing cluster required": you deploy it as part of a standard Java or Scala application.

Which projects handle late and out-of-order events?

All three processors document it. Kafka Streams supports event-time windows with out-of-order arrival and a grace period that controls how long it waits for out-of-order records. Flink lists late data handling. Structured Streaming uses watermarks.

Should I choose Kafka or Pulsar?

Both are distributed platforms for publishing and consuming event streams. Kafka documents replicating topics across geo-regions or datacenters, and Pulsar documents geo-replication and tenants. Choose the one your team can operate; Kafka's documentation notes you can self-manage it or use a managed service.