Zerobus versus Kafka for streaming ingestion

Summary

• Zerobus Ingest, part of Lakeflow Connect, is a push-based streaming API that writes records straight into Unity Catalog Delta tables with no message bus to run. Apache Kafka is an open-source distributed event streaming platform for many producers and consumers.
• Choose Zerobus Ingest when the lakehouse is the destination, and Kafka when the same events must reach several independent consumers.
• Zerobus Ingest enforces the target table's schema, never evolves it automatically, and guarantees ordering within a single stream. Kafka guarantees ordering within a topic partition.
• They work together: Databricks reads Kafka topics with Structured Streaming, and Kafka producers can write to Zerobus Ingest through its Kafka-compatible APIs (Beta).

Zerobus versus Kafka for streaming ingestion

Use Zerobus Ingest when the only place your events need to land is a Delta table in Databricks, and use Apache Kafka when the same events have to reach several independent systems. Zerobus Ingest is a direct path into the lakehouse: producers push records to a serverless endpoint, and the records land in a Unity Catalog table with no broker in between. Kafka is a general-purpose event backbone that any number of consumers can read.

What is Zerobus Ingest?

Zerobus Ingest is a push-based streaming API in Lakeflow Connect that, in the docs' words, "writes data straight into Unity Catalog Delta tables at high scale, with no message bus to run." It's generally available. You create a target table, then push records to it. The service validates each record against the table schema, makes accepted records durable and acknowledges them, then materializes them into the table, which the docs say typically happens within seconds. An acknowledgment confirms durability, not that the record is queryable yet.
Producers connect through SDKs over gRPC (Python, Java, Rust, Go and TypeScript), a REST API or an OpenTelemetry (OTLP) endpoint, all generally available. MQTT and the Kafka-compatible APIs are in Beta. The service is serverless and on by default, and you scale by opening more streams.

What is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform. Producers publish events to topics, and consumers subscribe to them. The Kafka docs describe producers and consumers as fully decoupled, and topics as multi-producer and multi-subscriber. Events aren't deleted after they're read; a per-topic setting controls retention. Topics are partitioned across brokers and can be replicated, even across regions. Kafka Connect handles import and export, and Kafka Streams handles stream processing.

Zerobus Ingest vs Apache Kafka: how do they compare?

Capability	Zerobus Ingest	Apache Kafka
What it is	Serverless push API that writes to Unity Catalog Delta tables	Open-source distributed event streaming platform
Data path	Producer to Zerobus Ingest to a managed Delta table or streaming table	Producers to topics on brokers, read by any number of consumers
Who reads the data	Anything that queries the Delta table	Independent consumers of each topic, within its retention period
Ordering	Guaranteed within a stream; none across streams	Guaranteed within a topic partition
Delivery	At-least-once when the interface acknowledges durability and the client retries	Various guarantees, including exactly-once processing
Schema	The table schema is the contract and is never auto-evolved	Events carry a key, value, timestamp and optional headers
Operations	No brokers or partitions to manage; you create the table and a service principal	Self-managed cluster or a vendor-managed service
Processing	Downstream, on the Delta table	Kafka Streams, plus Kafka Connect for import and export

How does Databricks approach streaming ingestion?

Databricks treats the governed Delta table as the destination. The documented Zerobus Ingest setup is: confirm your region is supported, get the endpoint, create the target table, create a service principal, and grant it USE CATALOG, USE SCHEMA, MODIFY and SELECT. Because every write is governed by Unity Catalog, lineage tracking and fine-grained access control apply as soon as data lands.
Two rules shape producer design. First, Zerobus Ingest never creates tables or changes their schema. You evolve the table first (for example, adding a column with ALTER TABLE), then update producers. By default it rejects records with fields that don't match the schema, and records that no longer fit after a breaking change go to a durable fallback location instead of being dropped. Second, ordering is per stream, so if you need one total order across many producers, carry a timestamp or sequence number in the data.
When Kafka already serves other consumers, Databricks reads it directly. Structured Streaming has a native Kafka source, the read_kafka SQL function works in Lakeflow Pipelines and Databricks SQL streaming tables, and Lakeflow Connect has a managed Kafka connector (Beta) that writes topics into streaming tables.

FAQs

Can I use Kafka and Zerobus Ingest together?

Yes. If Kafka also feeds systems outside the lakehouse, the Zerobus Ingest docs suggest keeping it and copying the data into Delta with a managed streaming connector. If a Kafka producer's only destination is the lakehouse, the Kafka-compatible APIs (Beta) let it write to Zerobus Ingest without a Databricks SDK. Those APIs are write-only.

Is Zerobus Ingest a replacement for Kafka?

Only when the lakehouse is the single destination. Databricks describes Zerobus Ingest as a single-sink architecture that pushes data directly into Delta tables, so it doesn't fan events out to other systems. If many independent applications need the same events, Kafka topics are designed for that.

Does Zerobus Ingest support schema evolution?

Not automatically. You add columns with ALTER TABLE, then update producers. A rescue column (Beta) can capture unmatched fields instead of rejecting the record.

How much throughput can Zerobus Ingest handle?

The quotas page lists default throughput quotas per stream and per target table, and your Databricks account team can raise them. Concurrent streams per workspace are unlimited, and opening more streams is the intended way to scale out.