Skip to main content

Zerobus versus Kafka for high-volume event ingestion into a lakehouse

Summary

  • Zerobus Ingest is a fully managed, serverless service in Lakeflow Connect that streams data directly into governed Delta tables.
  • Apache Kafka is an open-source distributed event streaming platform that runs as a cluster of servers and stores events in partitioned topics.
  • When the lakehouse is the only destination, Zerobus Ingest writes high-volume events into Unity Catalog Delta tables without a broker in the path. When several systems must read the same events, Kafka's decoupled producer and consumer model applies.
  • The two can work together: Kafka producers can target Zerobus Ingest through its Kafka-compatible APIs (Beta, write-only), and Databricks reads existing Kafka topics with Structured Streaming.
  • Zerobus Ingest never auto-evolves the target table schema, and a confirmed offset means a record is durable, not yet queryable.

Zerobus versus Kafka for high-volume event ingestion into a lakehouse

Zerobus Ingest and Apache Kafka cover different parts of an ingestion design. Zerobus Ingest is a write path: producers send records to a Databricks endpoint, and the records land in Unity Catalog Delta tables. Kafka is an event streaming platform: brokers store partitioned topics, and consumers read them independently of producers. If the lakehouse is the only destination for your high-volume events, Zerobus Ingest removes the broker from the path. If several systems must read the same events, a Kafka topic decouples producers from consumers, and Databricks can read the topic with Structured Streaming.

What is Apache Kafka?

Apache Kafka is an open-source distributed event streaming platform. It runs as a cluster of one or more servers, and some of those servers form the storage layer, called brokers. Producers write events to topics, and each topic is partitioned across brokers. Kafka guarantees that a consumer of a topic-partition reads that partition's events in the order they were written. Topics can be replicated so that multiple brokers hold a copy of the data. Producers and consumers are fully decoupled from each other. Kafka Connect supplies import and export connectors for external systems, and the Kafka Streams API supports stream processing applications.

Zerobus Ingest vs Apache Kafka: how they compare

Capability Databricks Zerobus Ingest Apache Kafka
What it is Fully managed, serverless service in Lakeflow Connect that streams data directly into governed Delta tables Open-source distributed event streaming platform
How it runs Serverless endpoint, on by default in supported regions, with nothing to start, provision, or scale A cluster of one or more servers, some of which form the broker storage layer
Scaling Capacity scales automatically as you open more streams, within documented quotas A topic is spread across partitions on different brokers
Where events land Unity Catalog managed Delta tables or streaming tables Topics on brokers; Kafka Connect moves data to and from external systems
Who reads the data Readers query the Delta table; every write is governed by Unity Catalog Consumers subscribe to topics and are fully decoupled from producers
Ordering The gRPC interface preserves per-stream ordering Order is guaranteed within a topic-partition
Durability A confirmed offset means a record is durable, not yet queryable; delivery is at-least-once A topic can be replicated so multiple brokers hold a copy
Producer interfaces gRPC SDKs, REST and OpenTelemetry (GA); MQTT and Kafka-compatible APIs (Beta) Kafka clients that talk to brokers over a TCP network protocol

How does Databricks approach high-volume ingestion into the lakehouse?

Zerobus Ingest is part of Lakeflow Connect, and Databricks announced it as generally available on February 23, 2026. The announcement describes message buses such as Kafka as multi-sink architectures and presents Zerobus Ingest as a direct path from source systems to the lakehouse.
For high-volume streaming producers, the documentation points to the SDKs over gRPC, which it describes as offering the highest sustained throughput. REST suits lightweight clients, and OpenTelemetry suits existing observability pipelines. Zerobus Ingest enforces documented quotas, including per-stream and per-table throughput for gRPC, so review them before sizing.
Existing Kafka producers can write to Zerobus Ingest without a Databricks SDK through the Kafka-compatible APIs, which are in Beta. The topic name is the full catalog.schema.table name. The APIs are write-only, accept JSON only, and ingest record values only: keys, headers, partitions and timestamps are not persisted. To land an existing Kafka topic in Delta, Databricks documents a Structured Streaming read with the kafka format followed by a write to a Delta table.
Plan for schema behavior. Zerobus Ingest validates every record against the target table and does not auto-evolve it, so alter the table first and then send records against the new schema. A VARIANT rescue column (Beta) can capture fields that do not fit the schema instead of rejecting the record.

FAQs

Can existing Kafka producers write to Zerobus Ingest?

Yes, through the Kafka-compatible APIs (Beta). Point an Apache Kafka producer client at the Zerobus endpoint and use the three-level table name as the topic. The APIs implement the producer-side subset of the Kafka protocol: Produce, Metadata, ApiVersions, and the SASL handshake APIs. Delivery is at-least-once.

Can Zerobus Ingest read from a Kafka topic?

Not through the Kafka-compatible APIs, which are write-only: consumer, admin, and transactional APIs are not supported. Databricks documents reading a topic into Delta with the Structured Streaming Kafka source.

Is a record queryable as soon as it is acknowledged?

Not necessarily. The documentation states that confirming an offset means the record is durable, not that it is queryable in the Delta table yet.

Do I need Kafka if the lakehouse is my only destination?

Databricks presents Zerobus Ingest as a direct path from source systems to the lakehouse. If independent systems must each consume the same event stream, Kafka topics decouple producers from consumers, so each consumer reads on its own.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.