When should I use Kafka, Kinesis, Flink, or direct lakehouse ingestion?

Summary

• Use Kafka when you need sub-second latency, partition-level ordering, long event retention, replay capabilities, or support for many independent consumers consuming the same stream.
• Use Kinesis (AWS) or Event Hubs (Azure) for fully managed, cloud-integrated streaming with operational simplicity when you're on that cloud platform.
• Use Flink or Spark Structured Streaming when you need complex stateful stream processing (windowing, joins, complex aggregations) that goes beyond simple ingestion.
• Use direct lakehouse ingestion (Auto Loader, Structured Streaming to Delta) when your sources can write files to cloud storage, you can tolerate second-to-minute latency, and you want the simplest solution with built-in schema evolution.

When should I use Kafka, Kinesis, Flink, or direct lakehouse ingestion?

The choice depends on latency requirements, architectural complexity, operational overhead, and whether you need multiple independent consumers to replay events. Each pattern suits different scenarios.

Key capabilities to evaluate

Capability	Kafka	Kinesis	Direct Lake	Flink
Sub-second latency	Yes	Yes	No (seconds to minutes)	Yes
Partition ordering	Yes	Limited	N/A	Depends
Event replay	Yes	Yes	No	Depends
Multiple independent consumers	Yes	Yes	No	Depends
Operational complexity	High if self-managed	Low	Low	Very high
Cost at scale	Varies	Varies	Often lower	Very high
Schema evolution	No	No	Yes (Auto Loader)	No

Message buses: Kafka and Kinesis

Kafka (or managed Kafka like AWS MSK) retains events durably with partition-based ordering, allowing many independent applications to consume the same stream at their own pace and replay events. Use Kafka when you need strict ordering, long retention, or complex event-sourcing patterns where multiple services depend on the same stream.
Kinesis provides AWS-managed streaming with tighter cloud integration (IAM, CloudWatch) and lower operational overhead than self-managed Kafka. Use Kinesis when you're on AWS, want managed simplicity, and don't need Kafka's advanced features. The tradeoff: Kinesis throughput is bounded by shard capacity and requires explicit scaling.
Both incur per-message costs that scale with volume; file-based ingestion often has lower operational cost at very large scales.

Direct lakehouse ingestion

When sources can write files to S3, ADLS, or GCS, direct ingestion into Delta Lake is often ideal. Databricks Auto Loader automates discovery and ingestion of new files, handles schema evolution automatically, and provides ACID semantics through Delta transactions. Latency is typically seconds to minutes (time for source to write a file plus Auto Loader detection), but operational footprint is minimal.
Use direct ingestion when sources naturally produce files (cloud APIs, batch exporters, data platforms), you can tolerate sub-minute latency, and you don't need multiple independent consumers competing for the stream.

Stream processors: Flink and Spark

Apache Flink and Spark Structured Streaming handle stateful stream processing (windowing, stream joins, complex aggregations). They sit downstream of your ingestion layer: ingest data first via Kafka, Kinesis, or direct lake, then feed it to the processor. Use these only for complex logic simpler approaches can't handle; for filtering, type conversions, and basic transformations, Lakeflow Pipelines suffice and are easier to operate.

Quick decision guide

• Latency sub-second and need replay or ordering? Use Kafka or Kinesis.
• Seconds to minutes acceptable and sources write files? Use direct lake (Auto Loader).
• Multiple independent consumers needed? Use Kafka or Kinesis.
• Complex stateful processing required? Use Spark Structured Streaming or Flink.

How Databricks approaches streaming ingestion

Databricks recommends file-based ingestion (Auto Loader) as the default for most use cases. Pipelines are defined once in SQL or Python via Lakeflow Pipelines and run on a managed schedule with automatic schema evolution.
For message bus scenarios, Databricks provides connectors and Lakeflow integration to ingest from Kafka and Kinesis into Delta tables, abstracting away consumer group management and exactly-once semantics. For complex processing, Spark Structured Streaming integrates natively with Delta and Unity Catalog.

FAQs

Can I bridge Kafka and direct lake ingestion?

Yes. Write a Kafka consumer that batches events into files and lands them on cloud storage every few seconds, then use Auto Loader to ingest. This trades Kafka's low latency for Auto Loader's simplicity and schema evolution.

Is Flink required for windowing or aggregations?

No. Spark Structured Streaming supports windowing, aggregations, and stateful operations and is simpler to operate on Databricks. Use Flink only if Spark's capabilities don't cover your use case.

How do I choose between managed Kafka (MSK) and Kinesis on AWS?

MSK if you need Kafka semantics (strong retention, partition ordering). Kinesis if you want minimal operations and AWS-native integration. For many use cases, direct lake ingestion offers simpler operations.

Does direct lake ingestion support exactly-once semantics?

Yes. Auto Loader uses file-based checkpointing and Delta's ACID transactions to ensure exactly-once ingestion without managing consumer state.