Structured Streaming versus Kafka Streams?

Summary

• Structured Streaming is a stream processing engine built on the Spark SQL engine that runs on a Spark cluster. Kafka Streams is a client library you embed in a Java or Scala application, with input and output data stored in Kafka clusters.
• Structured Streaming processes micro-batches by default, with Continuous Processing and Real-time Mode as other options. Kafka Streams uses one-record-at-a-time processing.
• Both keep fault-tolerant state: Spark checkpoints state to an HDFS-compatible file system, and Kafka Streams keeps local state stores backed by changelog topics in Kafka.
• Exactly-once in Spark relies on replayable sources and idempotent sinks. Kafka Streams enables it with processing.guarantee set to exactly_once_v2; the default is at_least_once.
• Choose by where your data lives and how you want to run code: DataFrame and SQL jobs on a cluster, or a library embedded in your Kafka-based services.

Structured Streaming versus Kafka Streams?

Apache Spark Structured Streaming and Kafka Streams both process continuous data, but they are packaged differently. Structured Streaming is an engine you run on a cluster. Kafka Streams is a library inside your application. This page compares what each project documents.

How do they differ in architecture and execution?

Aspect	Structured Streaming	Kafka Streams
What it is	A scalable, fault-tolerant stream processing engine built on the Spark SQL engine	A client library for applications and microservices whose input and output data are stored in Kafka clusters
Where it runs	A Spark cluster managed by Standalone, YARN, or Kubernetes	Inside your own application, with no separate processing cluster required
Languages and APIs	DataFrame and Dataset API in Scala, Java, Python, or R, plus SQL	Standard Java and Scala applications, using the Streams DSL or the lower-level Processor API
Processing model	Micro-batch by default, with Continuous Processing (experimental) and Real-time Mode (Spark 4.1.0)	One-record-at-a-time processing
Parallelism	Work is split into tasks that run in executors on worker nodes	A fixed number of tasks set by the input topic partitions, so maximum parallelism is bounded by the partition count
State	State store providers (HDFS-backed by default, RocksDB since Spark 3.2), checkpointed to an HDFS-compatible file system	Local state stores (persistent stores use RocksDB by default, or in-memory) with changelog topics in Kafka
Exactly-once	End-to-end with replayable sources and idempotent sinks, and at-least-once in Continuous Processing	processing.guarantee set to exactly_once_v2, with at_least_once as the default
Sources and sinks	Sources include files and Kafka; sinks include files, Kafka, foreach, and foreachBatch	Reads Kafka topics and writes Kafka topics, and processors can also reach remote systems

What does each model mean in practice?

Structured Streaming treats a live stream as a table that is continuously appended and runs your query incrementally. In micro-batch mode, which is the default, the query runs as a series of small batch jobs on executors that the cluster manager provides.
Kafka Streams builds a processor topology, a graph of stream processors connected by streams. Each processor receives one input record at a time. To scale up, you run more instances of your application on more machines. Kafka Streams uses Kafka's parallelism model to balance the load across them. If an instance fails, its tasks restart on other instances. Instances beyond the number of input partitions stay idle until needed.
Both support event-time windowing. In Kafka Streams, a window's grace period controls how long to wait for out-of-order records, and later records are discarded. In Spark, withWatermark() sets the late-data threshold.

How do they recover state?

Spark saves progress and state to the checkpoint location, and restores it after a restart. Kafka Streams keeps a replicated changelog topic for each state store. After a failure, it restores the store by replaying that topic. You can configure standby replicas with num.standby.replicas to shorten restoration.

Which one fits my workload?

• Data lives in Kafka and logic is a service: Kafka Streams fits an application or microservice that reads and writes Kafka topics and needs no processing cluster.
• DataFrame and SQL logic: Structured Streaming fits teams that write DataFrame or SQL code, join streams with static data (stream-static joins are supported), or read from files as well as Kafka.
• Latency: Each project documents its own latency features. Benchmark your own workload.

How does Databricks fit in?

Databricks documents Kafka as a source or a sink for Structured Streaming workloads on Databricks, using the kafka format with options such as kafka.bootstrap.servers and subscribe. In Lakeflow Pipelines and in Databricks SQL streaming tables, you can read Kafka with the read_kafka SQL function. To monitor lag behind Kafka, Databricks points to the avgOffsetsBehindLatest, maxOffsetsBehindLatest, and minOffsetsBehindLatest metrics.
Databricks does not support the continuous trigger mode, and documents real-time mode as its low-latency trigger type for Structured Streaming. Lakeflow Pipelines provide a declarative framework for batch and streaming pipelines in SQL and Python.

FAQs

Can I migrate between them?

You would rewrite the logic. Kafka Streams applications use the Streams DSL or Processor API in Java or Scala, while Structured Streaming queries use DataFrame or SQL operations.

Can I use both in one system?

Yes. Kafka Streams can write results to Kafka topics, and Structured Streaming can read Kafka topics through its Kafka source.

Is Kafka Streams limited to Kafka?

Its input and output data are stored in Kafka clusters. The Kafka documentation also notes that processors can access other remote systems while processing a record, so results can be written to an external system.

Does Structured Streaming need Kafka?

No. Its built-in sources include a file source and a Kafka source, plus socket and rate sources meant for testing.