Spark versus Kafka Streams?

Summary

• Apache Spark Structured Streaming is a stream processing engine built on the Spark SQL engine that runs as a Spark application; Kafka Streams is a client library embedded in your own Java or Scala application, with input and output data stored in Kafka.
• Kafka Streams fits when both ends of a pipeline are Kafka topics and you want processing inside an application or microservice, with no separate processing cluster.
• Structured Streaming fits when a pipeline also reads or writes files and tables, joins streams with batch data, or needs Python or SQL alongside Scala and Java.
• Both document exactly-once processing: Kafka Streams through the exactly_once_v2 processing guarantee, Structured Streaming through replayable sources, checkpointing and idempotent sinks.
• On Databricks, Structured Streaming reads and writes Kafka with the kafka format, and the read_kafka SQL function feeds Kafka data into Lakeflow Pipelines streaming tables.

Spark versus Kafka Streams?

Use Kafka Streams when your input and output both live in Kafka and you want the processing to run inside your own Java or Scala application, with no separate processing cluster. Use Apache Spark Structured Streaming when the pipeline also touches files, tables or batch data, needs Python or SQL, or should share code with batch Spark jobs. The core difference is architectural: Kafka Streams is a library you embed, and Structured Streaming is an engine that runs as a Spark application on a cluster.

What are Structured Streaming and Kafka Streams?

Structured Streaming is a scalable, fault-tolerant stream processing engine built on the Spark SQL engine. By default it processes a stream as a series of small batch jobs (micro-batches). It also offers a Continuous Processing mode with at-least-once guarantees, and Apache Spark 4.1.0 added a Real-time Mode with support for stateless Scala queries.
Kafka Streams is a client library for building applications and microservices whose input and output data are stored in Kafka clusters. It has no external dependencies on systems other than Apache Kafka, processes one record at a time, and offers a high-level Streams DSL plus a low-level Processor API.

Spark Structured Streaming vs Kafka Streams: how they compare

Capability	Spark Structured Streaming	Kafka Streams
Where it runs	As a Spark application on Standalone, Hadoop YARN or Kubernetes, with executors on cluster nodes	Inside your application, on containers, VMs, bare metal or cloud; no separate processing cluster
Languages and APIs	DataFrame and Dataset APIs in Python, Scala and Java, plus SQL on streaming DataFrames	Java and Scala, through the Streams DSL or the Processor API
Sources and sinks	Built-in file and Kafka sources; file, Kafka, foreach and foreachBatch sinks	Kafka topics for input and output
Processing model	Micro-batch by default; Continuous Processing and Real-time Mode as alternatives	One record at a time
State	State store with an HDFS-backed default provider and a RocksDB provider	Local state stores, backed by replicated changelog topics, with optional standby replicas
Guarantees	End-to-end exactly-once with replayable sources and idempotent sinks; checkpointing and write-ahead logs track offsets	processing.guarantee of at_least_once (default) or exactly_once_v2
Windowing	Tumbling, sliding and session windows, with watermarks to bound late data	Tumbling, hopping, sliding and session windows, with a grace period to bound late records
Scaling	Tasks run in parallel on executors across the cluster	More threads or instances; parallelism is bounded by the input topics' partition count

How to choose between them

• Where the data lives. If every source and sink is a Kafka topic, Kafka Streams keeps the pipeline inside Kafka plus your application. If you land data in files or tables, Structured Streaming's file sink and foreachBatch cover it.
• Deployment model. Kafka Streams deploys like any Java application, using your existing packaging and operational tools. Structured Streaming runs as a Spark application, on your own cluster or a managed Spark service.
• Language and team. Kafka Streams targets Java and Scala developers. Structured Streaming adds Python and SQL.
• Batch and stream together. Structured Streaming uses the same DataFrame API as batch Spark and supports stream-static joins, so one codebase can serve backfills and live data.
• Serving state from the app. Kafka Streams interactive queries let your application expose its own state stores, which suits services that answer lookups directly from stream state.

In Kafka Streams, exactly-once is a single setting:

Properties settings = new Properties();
settings.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG, StreamsConfig.EXACTLY_ONCE_V2);

In Structured Streaming, recovery depends on the checkpoint location you set on each query with .option("checkpointLocation", ...).

How Databricks approaches Kafka and streaming

On Databricks, Structured Streaming connects to Apache Kafka with the kafka format for streaming reads and writes, and it also supports batch reads of a topic. In SQL, the read_kafka table-valued function reads Kafka data into Lakeflow Pipelines or streaming tables in Databricks SQL, so a Kafka topic can feed streaming tables directly from SQL. For lower-latency pipelines, Databricks documents a real-time mode trigger for Structured Streaming, and its reference lists Apache Kafka as a supported source and sink.

FAQs

Can Spark read from and write to Kafka?

Yes. Structured Streaming has a built-in Kafka source and Kafka sink, so one Spark job can consume topics and write results back to Kafka.

Does Kafka Streams need a cluster?

It needs a Kafka cluster for its input and output topics, but no separate processing cluster. The library runs inside your application, and you scale by starting more instances or threads.

Do both support exactly-once processing?

Yes. Kafka Streams provides exactly-once semantics when processing.guarantee is exactly_once_v2, which requires brokers on version 2.5 or newer. Structured Streaming provides end-to-end exactly-once semantics with replayable sources and idempotent sinks.

Can I use both in one architecture?

Yes. Because Kafka Streams reads and writes Kafka topics and Structured Streaming has a Kafka source, a Kafka Streams service can process events in Kafka that a Spark job then reads into tables.