Skip to main content

Ditch the Message Bus with Zerobus Ingest

Summary

  • ZeroBus Ingest allows data producers to push data directly to Delta tables in the Databricks lakehouse via REST, Kafka-compatible, and OpenTelemetry APIs, eliminating the intermediate message bus and the infrastructure overhead of broker management and capacity planning.
  • ZeroBus provides ordering and high-durability guarantees, petabyte-scale throughput demonstrated at one petabyte per day to a single Delta table, and support for liquid clustering and the JSON Variant type.
  • Global companies including Joby Aviation and Toyota Motor Corporation have adopted ZeroBus to simplify real-time architectures, and the roadmap includes syslog, Prometheus, and OTLP protocol support.

Ditch the Message Bus with Zerobus Ingest

Watch: Ditch the Message Bus with Zerobus Ingest
Streaming is evolving—quickly. Modern architectures are eliminating infrastructure sprawl in favor of a direct-to-lakehouse model, eliminating the need for intermediate message buses and the expense of maintaining them. Zerobus Ingest allows you to push data directly to the lakehouse with near real-time writes, high throughput, and support for thousands of concurrent clients. In this demo-packed session, we’ll cover: How Zerobus works and how to build production-ready pipelines—without the heavy ownership costs of DIY infrastructure. The latest supported interfaces (e.g., REST, OTel, and Kafka-compatible APIs), as well as new features designed for massive scale (e.g., liquid clustering, JSON variant type support, etc.). See how global innovators like Joby Aviation and Toyota Motor Corporation have embraced this shift to streamline their real-time architectures. Join us and discover a direct ingestion framework to simplify your stack and accelerate your impact.
Talk By: Aleksandar Tomic, Senior Staff Software Engineer, Databricks ; Victoria Bukta, Member of Product Staff, Databricks ;

Chapters

FAQs

What is ZeroBus Ingest and how is it different from Kafka?

ZeroBus Ingest is a Databricks feature that allows data producers to push data directly to Delta tables in the lakehouse via REST, Kafka-compatible, and OpenTelemetry APIs, without routing through an external message broker like Kafka. Unlike Kafka, ZeroBus requires no broker management, capacity planning, or partition forecasting — Databricks manages durability and ordering guarantees natively.

What ordering and delivery guarantees does ZeroBus provide?

ZeroBus provides ordering and high-durability guarantees for ingested data, ensuring that events are stored reliably and can be consumed in order by downstream subscribers and processing jobs. This makes it suitable for production use cases like IoT telemetry, healthcare event streams, and retail transaction data where data integrity is critical.

What is the performance capacity of ZeroBus Ingest?

ZeroBus has been demonstrated ingesting one petabyte of data in a single day to a single Delta table, showcasing its capability for massive-scale production workloads. It also supports liquid clustering and the JSON Variant type to optimize storage and query performance for high-cardinality streaming data.

Which companies are using ZeroBus Ingest in production?

Global companies including Joby Aviation and Toyota Motor Corporation have adopted ZeroBus Ingest to simplify their real-time data architectures by eliminating intermediate message buses. This video covers their use cases as examples of how direct-to-lakehouse ingestion reduces operational complexity in IoT, manufacturing, and transportation domains.

Full transcript

[00:08] Hello everyone. Thank you for coming to this session, ditch the message bus with zero bus ingest. I'm Victoria Bukta. I'm a product manager here at Databricks. I work on all things to do with streaming ingestion and I'm joined by Alexander.
[00:21] >> Yeah, I'm Alexander, Alexander Tomic. I am technically the architect of zero bus. >> So, you may have a architecture today that looks something like this. We have
[00:34] a variety of data producers on the left here, cloud applications, edge devices, IoT devices. All of these applications, these data producers want to offload
[00:46] data quickly. So, we want low latency offloading of data, but high durability guarantees that the data is stored. Typically, we add a message bus in between, think Kafka, Kinesis, Pub/Sub,
[00:59] Event Hub. We offload the data to these types of message buses. They give us this low latency, high durability guarantee of the data. Once the data is in the message bus, we have a variety of different subscribers
[01:12] that could be reading, processing, acting on this data. Think agents, microservices, ingestion jobs, maybe some real-time analytics. But, having this end-to-end architecture also
[01:24] introduces a variety of problems. So, let's just talk about scaling for a minute. When we think about scaling the message bus, this is infrastructure that lives outside of Databricks. There are
[01:36] two ways that you can scale a Kafka cluster or scale your operations. First, there's physical scaling with brokers. You may have to add more disks if there's not enough physical space on the
[01:48] cluster itself to handle your load. You have to think about when you're going to add more brokers, when you're going to add more machines because this causes a rebalance event on the cluster itself,
[02:00] maxing out CPU and network bandwidth. Then, there's the consumer side. Can my consumers keep up with the rate of the events coming into these Kafka topics, to these uh message bus topics? This is
[02:14] where partitions come into play. We have logical scaling. A single topic can be divided across a number of partitions. But, I need to then forecast for consumer
[02:26] uh scalability. So, for each of these, if you're going to host a message bus, you're thinking about capacity planning. You're thinking about forecasting for your consumers. What's the rate of ingestion that they can keep up with?
[02:39] The rate of processing they can keep up with. So, all of this is mental load for you to have to scale your operations always for peak capacity. But, do you actually need to run all this
[02:51] infrastructure all the time? So, this was like the the the the core problem that we wanted to solve with Zero Bus. When you have all of these different data producers that are sending data to the message bus, then
[03:04] being like ingested in directly into the data lake, when the message bus is only being used as a pipe, can we simplify this? And we did with Zero Bus ingest. Zero Bus ingest is a direct write API to
[03:19] Unity Catalog tables. The idea here is that you can take a data producer, that cloud app, edge application, IoT device, it's already doing this push motion, pushing events. Now, they can push data
[03:32] directly to Zero Bus, completely bypassing the the message bus, Kafka clusters, Kafka topics, when the sole destination is the data lake. Zero Bus supports high-throughput workloads up to
[03:45] 10 GB a second throughput to a given table. We offer low latency. Offloaded the data within 200 milliseconds. The data is queryable in the data lake within 5 seconds and this
[03:57] is a fully managed product. All you have to do is create a table and you can start ingesting data. There's no cluster to spin up or size. We're an always-on API that's ready to accept data.
[04:11] But what is Zero Bus Ingest? >> So let me tell the story what Zero Bus is all about. It is a dedicated service that does one thing. It pushes the
[04:24] streams of data into your Unity Catalog tables. Uh it's one by default endpoint which means that I could assume that this audience already uh has their Databricks workspace which
[04:38] means that you also have a Zero Bus workspace. Uh Zero Bus provides record record push through various protocols and formats. Data can be pushed through GRPC bidirectional streaming, REST, hotel,
[04:52] and MQTT and Kafka wire compatibility are just around the corner. Uh we also provide SDK support for various languages, Python, Rust, or Java, Go, and TypeScript. And for the
[05:05] next couple of minutes, I'll walk you through uh an example how to use Zero Bus and hopefully make all of this a bit more tangible.
[05:18] Uh let's first go back to the title of this presentation. So when to ditch the bus? Uh it is not a trivial choice and Zero Bus doesn't claim that you should always replace a bus uh like Kafka.
[05:31] It perfectly makes sense to use uh the bus when you need microservice communication for example. That is a bunch of microservices that are exchanging the messages through the bus or doing fan outs to multiple
[05:44] consumers. But uh if you keep uh message buff, if you keep the bus just as a buffer between your producer and your destination table, we think that Zero Bus is much, much more efficient
[05:57] solution to solve this problem. Uh we noticed that many, many teams keep these large infrastructures between their producer and destination table with main goal to amortize the load.
[06:11] There are various patterns like uh adding Kafka or actually adding multiple various services, OTP databases, buses, just so you amortize this river of data
[06:24] that's going through. Uh and if you just think from the first principles and like amount of hops and CPU cycles that are wasted for this load, it really doesn't make sense. Zero Bus is built to solve solve this problem and it does it in
[06:37] much more efficient way with in the end much better throughput and much lower latency and time to query.
[06:51] Now, uh I would like to cover two core principles of Zero Bus. Scale and simplicity. I will start start a bit backwards by first talking about scale. So, uh last week we published like you probably wonder like what is this
[07:04] animation about? Like I'm kind of like also hypnotized by it by it and I kind of enjoyed making it. Uh so, last week we published a blog post that tells the story uh of how Zero Bus was able to
[07:17] ingest more than a petabyte of data in a single day into a single table. Uh as a test, we used NASA's uh telescope survey of every visible object in our galaxy, our Milky Way galaxy, uh
[07:31] in span of 13 years. So, like just imagine like every planet, star, asteroid that's that's basically the the data set. And this visualization is how we
[07:44] build like the visible part of the universe around us by being adjusted through Zero Bus. So, just to illustrate what we did, like we had to ingest this data set of 13 years of every object in Milky Way galaxy 13
[07:59] times in a the 13 years of data 13 times just to get to this petabyte scale and Zero Bus was able to end up with this final table uh with all of this data in less than 24
[08:13] hours. Uh please uh do scan the QR code that is on the screen. We also go through uh benchmarks. Uh we try to make everything reproducible so you can try it out.
[08:27] And it should be fun read especially because we also go through some internal deep dives, architectural decisions, how we manage to make this work at this this scale.
[08:42] Everyone good? Please. Okay, I'll give like uh five more seconds for everyone to scan it.
[08:54] Thank you. And yeah, like I spent embarrassing amount of time reading about astronomy and ways to map this this like our our galaxy to to this space. So, yeah, like that's uh
[09:07] why some of the Zero Bus features are coming late. Uh >>
[09:19] scale, but let's see how Zero Bus looks from developer point of view. We tried really hard to make it in a way in which both uh in which hello world injection and
[09:32] petabyte level injection are awfully the same code just executed on more machines, more producers. Uh the steps to get started are following. So, you create a table. This is just regular Databricks table, so you
[09:45] can use DBR, Spark, whatever to create a table. Um then you create service principle to give permissions to the table, and you are ready to start coding.
[09:57] The code should be as simple as this. Uh the nice part, as Victoria explained, is that you don't create any pipelines, you don't create any clusters. You won't experience any additional latency
[10:11] on the first message that you send. You can just start pushing if you have uh Databricks workspace. Uh so, what you do is just create a stream by specifying authentication and table
[10:24] name, and you call ingest record, and that's it. Your data are is going to show up in your destination table soon, as you will see in the demo I am about to show.
[10:36] Before the demo, let me cover what is this concept of a stream. So, stream is a direct connection from the client to the robust to the robust service. Uh communication is asynchronous in a way
[10:48] that under the hood uh stream streams are these bidirectional streams. Clients send the data, and ZeroBus responds with offsets that send the signal that the
[11:00] data is durable. So, the idea is that clients just keep sending, receiving offsets without blocking, so you can achieve this high throughput um and very low latency.
[11:12] Uh streams are ordered, so you can expect that uh data in your destination table arrives in the same order as you sent it. Uh SDK SDKs provide logic for handling
[11:24] streams, managing offsets, and also doing recovery in case of network glitches or client or server failovers. So, a lot of these complexities hidden by just using SDKs, so you can expect to
[11:36] just to send data and await on the record when it's durable, and you can do that in parallel by sending millions of records per second.
[11:50] And let's let's jump to demo. Demo is going to be live, so please be gentle if something something goes wrong. Again, what we're going to do is we're going to drive you pretty much through all the steps you need to take to to get Zero Bounce running. So, we're going to
[12:03] create a table in UC. We're going to set up SDK locally. We're going to ingest the live camera feed, so like I'm going to use my Mac OS my MacBook camera and ingest this data to
[12:18] Zero Bounce table. Uh So, let me see what's there.
[12:41] So, we start by simply creating the table. We will call it camera frames, and schema is super simple. So, you do timestamp, which is current timestamp of when photo was taken, frame number, and
[12:53] binary blob that's representing the image that we're sending. Let me execute that.
[13:06] We grant permissions to that table for principal token that I previously created, and we can see that table should be empty. Yeah. Let me Okay, it's there, surprising.
[13:20] So, let's see how the code looks like.
[13:32] >> So as you see the code should be pretty simple. So in total 40 lines of code outside of this camera handling the core 03 bus is logic is super simple. What we do is that we created a stream. We go through a sequence of frames as they're
[13:45] arriving from camera and we call ingest on each of these streams to send it to 03 bus.
[13:57] Um let's see what create stream is. So create stream just reads the 03 bus endpoint. Again, you already have this. It reads your workspace URL. You already have that. It reads 03 bus table. That's the thing that we just created and
[14:11] it reads the secrets. And that's pretty much it. We now have a stream and each ingest call is basically just creating this proto encoded object. As I said, JSON might be simpler here but like we
[14:23] chose proto for this demo. Uh and we send JPEG that's arriving through camera. And we call stream ingest record and that's pretty much it. Now let's run it.
[14:38] And like I should go down a bit to make sure that camera is going to capture me. So frames are coming. Let's go back.
[15:00] Okay, they're coming. And let's try to plot them. So again like very simple PySpark script. And usually my face is going to look kind of ridiculous when I execute this for the first time. So let's see.
[15:13] So these are Here it is. Okay, here it is. Uh uh >>
[15:25] photo shoot? >>
[15:39] here. Okay, here we are. >> There we go. >>
[15:57] Uh let me just Let's go. Okay, and I don't know how to use this thing. Like how to do everything. Slide show.
[16:09] Sure. >> Okay. Okay, let me take over. Okay, so you got to see the end-to-end demo of how to use ZeroBuzz, but there's a variety of different ways that you can actually utilize us. So, we'll go through a couple of best practices. The
[16:22] first thing that I want to cover is our breadth of APIs. So, we showed you how to use an SDK. Um the SDKs that we produce are actually using GRPC communication under the hood. That is
[16:35] that client-to-server communication that Alexander walked you through. That is the go-to choice if you want to build a high-throughput uh streaming client. So, think like CDC use cases, clickstream, maybe some sort
[16:49] of event forwarder. You want to forward data from an on-prem uh broker, on-prem data center to the cloud. This is like the go-to choice. We also have a REST API.
[17:02] The REST API, the way that I like to talk about it is it's the bare-metal approach. When you can't install a library, when you can't install extra dependencies, this is a a good way that you can use ZeroBuzz. You can just make
[17:16] a post request, a rest post request, give us a JSON blob, and we will ingest it directly to the lakehouse. This is the go-to way if you want to have, for example, a bunch of chatty clients. So,
[17:29] imagine you have a smart device. Maybe you have your product is a smart refrigerator that's sending a signal maybe every 5 minutes. So, this would be like the chatty client use case.
[17:44] Next, we have open telemetry. We support open telemetry APIs, so you can take anything that is compatible with the OTEL spec to send logs, metrics, traces, and forward those logs, metrics, traces
[17:57] to Zerobus. So, for example, you can take syslog-ng, configure it with Zerobus, just config change, there's no like code change that you have to do, and it will send data directly to us,
[18:09] and we archive that data in the data lake. Actually, for syslog-ng specifically, there's a Databricks community blog that will walk you through that. Next is message encoding. So, we talked
[18:21] about Protobuf, that was what was in the demo, but we support Arrow and JSON as well. Protobuf should be the go-to choice for generic record-by-record ingestion when you know the schema of the data.
[18:34] Arrow, we can use for batch ingestion. So, say you are an aggregating service. Maybe you have you know that you want to build a high-throughput client, it's gathering batches of messages, you can
[18:46] take the time to actually orient it in a columnar format, and send it to Zerobus. This is the best way if you want the maximum throughput from your client. Then there's JSON. JSON is flexible,
[18:59] it's easy to get started, you can basically just give us a blob and and we ingest it into the table, but it it is going to be slower compared to the other formats. Just the sheer byte size, the
[19:11] compression ratio is not going to be as good with with JSON. Next is schema management. Schema management is a really big problem when we talk about streaming data. One thing
[19:23] that I want you to take away from this session is at no time does ZeroBus modify tables. ZeroBus never creates tables. In the demo, you saw that we had to first create that table. ZeroBus will
[19:36] also never modify, alter the table schema. For us, the table schema is the authority. It's the gatekeeper of what shape of data is allowed to go into the table.
[19:48] So, ZeroBus is doing that schema validation. As long as you send it a subset of record, or sorry, subset of fields, the data will be accepted in. If you send it extra columns, columns
[20:01] that do not exist in the target table, or maybe you're missing a required column, ZeroBus will reject this record. It will not silently drop. That's like a clear distinction I want to make. It
[20:14] will tell the data producer that you did something wrong, that the schema is incorrect, and this is how the schema is incorrect. Now, the next evolution of this is we're going to be introducing the concept of a
[20:26] rescue column. This will be coming in the next few months. The rescue column is your way to tell us that you want anything that does not fit into the table to be ingested there. So,
[20:39] it's it becomes persisted. You could If you're a familiar with dead letter queues in the eventing universe, you can think of this column as the dead letter column for for this data. What this
[20:51] gives us at the end of the day is this choose-your-own-adventure approach to what data you want to come into the table. How strict do you want to be? Accept everything, or accept only a subset of data. That's up
[21:05] to you. The next thing I want to talk about is our Kafka producer API support. This is going to be released in July, and I'm really excited about it.
[21:17] What I like about Kafka and what the Kafka ecosystem has done really well is it's everywhere in the entire ecosystem of eventing. They've created this standardized API where there's a huge ecosystem of tools, not just client
[21:30] libraries, but Kafka Connect and log forwarders such as Logstash, Fluentd. They all know how to send data to Kafka. When Zero Bus supports Kafka compatible producer APIs, you'll be able to just
[21:44] make a config change to these applications to send data to Zero Bus as if we were a Kafka broker. So, no code change required, just a config change. And Zero Bus writes all that data
[21:57] directly to Delta tables in your data lake. So, I'm going to do a little bit of a demo now of what this would look like to do Kafka compatible to use our Kafka compatible APIs. Who here like uses or
[22:12] has um familiar with Debezium? Like, hands up. Yeah, a lot of people here use Debezium. All of Debezium for context is used for CDC capture from relational databases. So,
[22:25] in this demo, what we'll do is locally set up a MySQL database, a local Kafka cluster, a local Kafka Connect cluster where we're going to run the Debezium connector. We're going to pipe this
[22:38] together in the normal way to get all the change feeds from the MySQL database to Kafka, and then we're going to make a config change and I'll point that connector to Zero Bus and read those
[22:50] events in the data lake. So, let's get started. So, the first thing that we're going to do is just set everything up. So, we are using Docker Compose, we're setting up
[23:03] our local MySQL, my uh local Kafka cluster, local local Kafka Connect. We're waiting for the MySQL database to now spin up.
[23:16] And we do this select star from the table. So, we have this customers table. It has three rows in it. We have Alice, Bob, and Carol.
[23:30] So, the next thing now that we have our MySQL database is we want to launch our Kafka connector. So, this is just like vanilla uh Debezium. We see that we're uh launching this Debezium MySQL connector.
[23:43] I am including the inventory customers table. And now, let's run this. And on the right side,
[23:55] what we should see is the original snapshot events. So, when we originally launch the connector, we see all these read events. We're getting a snapshot of what's already in the table.
[24:07] So, next what we want to do is actually make some changes. So, we're inserting a record, updating a record, and deleting a record. So, we've confirmed that everything is connected. Debezium is
[24:19] running, we can connect to the MySQL database, I can output to Kafka. Next, let's switch everything over to Zero Bus.
[24:34] So, the first thing to do when you get started with Zero Bus is create that table. So, we have this table that we have set up with just operation, before and after fields. We see that there's no data there. And now, we're going to change the
[24:46] config. What we're changing here is adding the producer bootstrap server. So, what this is is just the URL to connect to Zero Bus. So, workspace URL, ZeroBus region,
[24:59] all that stuff. There's a couple parameters here for OAuth authentication with ZeroBus. And at the bottom here, these transformers are just to route that customer data to our ZeroBus default
[25:14] customer CDC table. It's what you see here on the right. So, now we launch that connector. It's going to do a select to do that
[25:26] initial uh snapshot. So, here we have those read events. Exactly how we had it on the Kafka side, we have those read events coming into
[25:38] the table. The next thing that Debezium is going to do, we're going to First, we're going to do those updates. So, we did a insert, update, and delete. Now, Debezium will
[25:52] see those events and forward the data to ZeroBus through our Kafka compatible APIs. And then we can query the table and we see that those events made it directly to the table. So, we have all the data
[26:06] there. So, now with this, you can completely bypass Kafka for this CDC type use case. We're able to take an existing compatible Kafka client, direct it to ZeroBus. I didn't have to install
[26:18] anything new, and the data is archived in in the lakehouse.
[26:32] >> So, we have lots of customers that are using this type of integration for these use cases of on-prem to cloud connectivity. One customer that we're really proud to be working with is Meta. They're able to cut their end-to-end pipeline latency to under a minute with
[26:46] the combination of ZeroBus ingest and declarative pipelines for transformations. What they're What Meta is doing is actually their operations are split between their own on-prem data centers that they run and own and cloud
[26:59] operations. And Zero Bus is being used as that bridge. We see this all the time in a variety of industries, not just these cloud-native companies or these massive tech companies, but think manufacturing,
[27:12] think health care. There's lots of use cases, energy utilities. There's so many use cases where we need to bridge the the gap from edge devices, edge data centers to the cloud. And this is
[27:26] what Zero Bus is being used for for a variety of our customers at scale. So, what comes next? Like we talked about what's here, a little bit about what's coming, but what's the next evolution of Zero Bus and real-time
[27:40] eventing at Databricks? So, you heard a little bit about this earlier this week in the keynote. What I'm happy to talk about is our real-time analytic stack. We have Zero Bus ingest for streaming
[27:53] ingestion. We have Spark real-time mode for these streaming real-time operations calculations. And then Raiden, Lakehouse RT, for serving that data out.
[28:06] Right now, these are separate products, but the vision is we're going to give to you a fully integrated real-time stack. Lakehouse RT, Raiden, reading the in-flight data in Zero Bus. You no
[28:19] longer have to wait 5 seconds to query that data. You'll now be able to query Zero Bus's in-flight data. 200 ms fresh data powering your queries.
[28:31] Similar with real-time mode Spark. If you're using real-time mode right now, you're familiar with I ingest data from Kafka, I do a transformation, I then have to output that result in some sort
[28:43] of real-time store. Kafka, Kinesis, relational database. But now that could be Zero Bus. Zero Bus gives you that archive of the events in the lakehouse, and you could do a dual write to lake base.
[28:57] This end-to-end stack, these three pieces of technology working together, is going to be your real-time analytics toolkit. And then Zero Bus itself specifically,
[29:10] we're going to be expanding the Zero Bus family of protocols. So today we have REST API, gRPC in GA. End of month, our Open Telemetry APIs will also be GA.
[29:22] July, the Kafka compatible APIs will be in beta, and then in August, MQTT compatibility. There's also a variety of other features that are coming very soon, such as Avro
[29:34] support, scopes uh scope service credentials, long-lived token support, private link to storage, metadata column, and that rescue column that I talked about earlier uh around schema management.
[29:50] So, we talked about ditching the bus. Are you ready to ditch your message bus? Well, if you have this real near real-time offloading use case, you have these lakehouse consumers of the data,
[30:02] you're all right you're all right with this lakehouse actually being the source of truth of your data, Zero Bus is the way to go. We have a variety of different case studies that you can read more about how major customers are utilizing Zero Bus.
[30:16] For example, Joby Aviation, this is a manufacturing use case. They build these vertical takeoff aircrafts. They were at Summit last year. They connect their on-prem factories to Zero Bus to get the
[30:29] data to the cloud. Similarly, we work with Toyota Motors. They're streamlining IoT data from their factory flo- floors, actually connecting Raspberry Pis running Zero Bus SDK to send data to the
[30:42] cloud. So, it can be used in so many different use cases. Um we're happy to take questions. We'll be down here in the front. I want to hear about your potential ZeroBus use case, and what do you want to see come
[30:54] next? Thank you very much. >> Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.