Zerobus Ingest, Lakebase and Databricks Apps in Action: Data Streaming with Databricks

Summary

• IoT streaming at scale: Ingests thousands of concurrent IoT events from devices worldwide in real-time using Zerobus Ingest (no Kafka), Lakebase for low-latency serving, and Databricks Apps for visualization.
• Mobile sensor collection: Deployed globally with colleagues submitting IoT events from mobile phone sensors (pitch, roll, acceleration data) at configurable frequencies, demonstrating true distributed data collection.
• Single-digit latency: Achieves single-digit millisecond access to historical data traces using Postgres OLTP in Lakebase, supporting real-time fraud detection, fleet telemetry, and other sub-second use cases.

Zerobus Ingest, Lakebase and Databricks Apps in Action: Data Streaming with Databricks

Join Frank Munz, Principal Tech Marketing Engineer at Databricks, for an exciting product demo of a global real-time IoT data streaming solution!
In this video, Frank demonstrates how we ingested and processed thousands of concurrent IoT events from dozens of sources worldwide. See the power of Zerobus Ingest for ingestion and Lakebase for low-latency serving, all without the need for complex Kafka clusters or Kinesis.
Key Highlights:
Real-time Visualization: Watch live sensor data from mobile phones worldwide as it's displayed on an interactive map.
Simplified Architecture: Learn how Zerobus Ingest and Lakebase eliminate the need for complicated streaming setups like Kafka.
Low-Latency Performance: Experience single-digit millisecond access to historical data traces with Postgres OLTP in Lakebase.
Genie Integration: See how you can use Databricks' AI assistant to quickly generate insights and visualizations from your streaming data.
From predictive maintenance and fleet telemetry to sub-second fraud detection, Databricks provides a unified, serverless architecture that handles your most demanding, high-concurrency event streams.

Chapters

• 0:00 The challenges of IoT data
• 0:40 Introduction to the solution
• 1:37 How the application works
• 2:50 Real-time demonstration
• 4:49 Implementation and setup
• 6:12 Analyzing data with Genie
• 7:00 Conclusion and resources

FAQs

What problem does this solve?

Traditional IoT requires Kafka and complex pipelines. This architecture eliminates that complexity with Zerobus Ingest and Lakebase—no Kafka, no operational overhead.

How does mobile collection work?

The mobile app collects sensor data (pitch, roll, acceleration) at configurable frequencies and continuously sends events to Zerobus. Zerobus handles massive concurrent ingestion.

What is the end-to-end flow?

Mobile app sends to FastAPI front end, which forwards to Zerobus Ingest. Results persist to Lakebase for serving, and Databricks Apps renders interactive visualizations.

How can you analyze streaming data?

After data persists to Lakebase, use Genie to generate insights from historical data through natural-language questions about patterns and trends.

Full transcript

0:00 The challenges of IoT data

What happens when thousands of devices start streaming data at the same time? What happens when the data is messy, unpredictable, and constantly flowing in? And what happens when your business can't wait for insights? Not until tomorrow, not until the next hour, and maybe not even until the end of this sentence. That's the reality of IoT and streaming data. And for years, the answer has looked the same. Stand up Kafka, stitch together pipelines, and wire up services, and then hope it all holds together when the traffic spikes.
Not anymore. Hey, I'm, tech marketing at Databricks, and today I'm going to show you a scenario where we ingested and processed thousands of concurrent IoT events streaming in from devices all over the world in real time. And to show you this at scale, I didn't simulate it, but I deployed it internally at Databricks, and colleagues from across the globe created IoT events with sensors from their own mobile phones. The application is built with Zero bus for ingestion, Lake base for low latency serving, and Databricks apps for the front end and the back end.

0:40 Introduction to the solution

No Kafka, no duct tape, and no sleepless nights. In fact, everything you've been seeing on on this map since I started talking is the application. It's been running the entire time. And now, let me show you how this works. This is how it looks like. This is the back end. It's served as a Databricks app. You scan a QR code, it takes you to this mobile application, but you can also get to this dashboard, which is like visualizing then the data.

1:37 How the application works

Once you are on the app with your mobile phone, you allow it to access the sensor data, and then it's continuously collecting that sensor data and sending it to Zero bus. And you see there is like pitch data, roll data, acceleration, and every single event is of course recorded. Uh it's counting on the mobile application. You can define the frequency. Do you want to send one per second or every 10 seconds an event. So, it keeps track of the data rate and the number of events um that it has sent to Zero bus.
So, it's a pretty cool um back end. It's running all over the world. So, that's the architecture. The mobile application is sending this to this front end, which is using fast API, which is then sending it to Zero bus, and then in a second I can show you that Zero bus view, which is visualizing the Zero bus data. Um the cool thing about this architecture, there's no Kinesis, there's no Kafka in place here. So, we don't need to configure a cluster and set up Kafka and define, you know, topics and replication.

2:50 Real-time demonstration

Uh instead, what we have here is we have this extra Lake base, which is giving us fast access to those traces. And if you look at the top right, that took us like 12 milliseconds, and that's a trace I did the other week in my lunch break. So, I went out, I made an appointment for my hairdresser, and then I bought a sandwich, and here I was trying to measure the speed and see if the speed measurements make any sense. It's 4.
5 m/s. And so, it's not really fast, but I was carrying my lunch, and then I was slowing down. I walked back here. Every purple dot here is an IoT event that is like generated on the phone and then sent to Zero bus. And you can guess that's Manhattan. Thank you, for recording that. That's Brazil. All of these, you know, clients sending data at the same time. This is a beautiful trace. Look at the Look at the IoT events.
Aren't they fantastic? It's actually someone moving in a really, really fast pace. If you look at this, 17. 5 m/s. So, he's either world-class runner or he's like a good motorbike or decent car. Uh same thing in Singapore. All these clients sending data at the same time. Um again, that's like 2 milliseconds. It's like single-digit milliseconds very often. Can you guess which city that is? A lot of high buildings, so the GPS coordinates aren't that great.
Let me help you. If I zoom here, that's a cruise ship. That's actually Circular Quay, and if you look here, that's the Opera House. So, this is Sydney. So, we got experts for Zero bus and Lake base all over the world, and they can help you with these implementations. And this is this three-level namespace. A simple table. Remember, we don't have to configure Kinesis or Kafka. Just define a table, define the columns of this table, and then here everyone can access this table.

4:49 Implementation and setup

That's for read. To write, we need a service principal. I define that service principal here with the CLI. Sometimes I actually do it just with a UI, and then that's the magic piece here where we say grant modify. So, to write to that table, you have to be in the role of this service principal, which is the client for Zero bus. So, the client, which is running on that mobile application, is using that um service principal to write to that table.
And this is how the magic happens, you know, without Kafka, without a cluster, uh without topics. And looking into Zero bus, um that's part of the back end application um that is deployed. Uh you see the data streaming in. So, that's literally live data streaming in now from somewhere in the world. You can guess from latitude and longitude. You see also it's two different clients. One is pointing straight north with a zero um heading, and the other one is having some heading, and apparently um someone is moving.
And that's all data streaming in right now. So, it's live streaming data being um persisted in this UC table. And remember, the trace is being served by Lake base. Lake base is the low latency, high throughput um high transaction rate OLTP database that is serving these traces. And now some people say, "Well, it kind of looks like a fitness application, but a fitness application would give us so much more." That's true.

6:12 Analyzing data with Genie

This is why I'm spinning up here a new uh Genie space. Uh Genie lets you uh access, you know, um columnar table. Uh and that's what I do right here. So, I'm I I ask Genie, I say, "Look, give me a a diagram. I want to see the speed over time." And that's the trace where I was running, uh well, running in quotation marks. Running with 4. 5 m/s, and you see that peak. We've put this all together in a blog posting. If you face something similar like this, and remember, it doesn't have to be, you know, plotting fitness events on a map.

7:00 Conclusion and resources

It could be security events, it could be log events, like anything related to event ingestion with Zero bus and low latency serving with OLTP databases, we got you covered. These are the implementation details. You see the authors of the blog as well at the top. There's different options, you know, how to replicate, how to get this real-time data into Lake base. They're all described here. So, if you're looking into some kind of these implementations, talk to our experts.
You're super welcome. this is all I had for today. So, thank you for joining, and let's talk to you soon. Bye-bye.

Learn more about the Databricks Data and AI platform.