Lakeflow Connect: eliminating hops in your streaming architecture
Summary
- Shift-left ingestion architecture: Lakeflow Connect simplifies real-time data ingestion from IoT, clickstreams and telemetry sources by embedding data flows closer to the source, reducing unnecessary hops.
- Three-pillar solution: The platform comprises connect (ingestion), declarative pipelines (ETL/transformations) and jobs (orchestration), all built on Databricks' governance and optimized storage layers.
- Zero Bus direct write API: A modern approach enables analytics and AI integration directly into operational systems, simplifying architectures and scaling more efficiently than traditional multi-hop patterns.
Lakeflow Connect: eliminating hops in your streaming architecture

Talk by Nikola Obradovic, Sr. Staff Software Engineer, Databricks ; Victoria Bukta, Member of Technical Staff, Databricks
In this video, we’ll explore new patterns that simplify real-time data ingestion into your lakehouse — especially for sources like IoT, clickstreams, telemetry and more. We’ll begin with a quick look at the evolution of the ingestion landscape and the growing need to embed data flows closer to the source. Then, we’ll introduce a modern approach that helps you “shift left” on ingestion — making it easier to integrate analytics and AI directly into operational systems, rather than treating them as afterthoughts. This shift can lead to simpler architectures that reduce unnecessary hops and scale with your operations. To close, we’ll share a practical decision framework to help you determine which ingestion options best fit your use case, followed by a live Q&A to help you get started quickly.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00Introduction to LakeFlow1:25LakeFlow Connect pillars2:23Business and technical needs3:38Joby case study5:31Architectural challenges7:27IoT and clickstream flows8:54Zero Bus direct write API10:39Soracom use case11:56Joby solutions13:51Demo: Building a data agent23:28Roadmap and features25:27Closing thoughts and Q&A
FAQs
What is Lakeflow Connect?
Lakeflow Connect is a unified intelligent data engineering solution built on Databricks, comprising connect (ingestion), declarative pipelines and jobs for orchestration.
What does shift-left ingestion mean?
This approach embeds data flows closer to the source systems, making it easier to integrate analytics and AI directly into operational systems rather than treating them as afterthoughts.
What is Zero Bus?
Zero Bus is a direct write API mentioned in this video that helps eliminate hops in streaming architectures by enabling data to flow directly from sources to Databricks.
Which use cases does this address?
This video demonstrates Lakeflow Connect's effectiveness for IoT, clickstream and telemetry data sources, with case studies showing simplified architectures.
Full transcript
0:00 Introduction to LakeFlow
Hello everyone. Thank you for coming to uh LakeFlow Connect where we're going to talk about how we can help you eliminate hops in your streaming architecture with zero bus. So I'm a product manager member of product staff here at data bricks and I'm a PM on LakeFlow connect. So all the things to do with ingestion before data bricks I was at tabular so we were part of the tabular acquisition um all things Apache iceberg and I was a PM there and in my previous life I was an engineering manager at Shopify focusing on data ingestion hi folks I used to be a SQL server engineer for a long time and now I'm here software engineering data bricks so what we're going to do today is just introduce the concept of LakeFlow Connect and what that is.
Then we're going to talk a little bit about the business tradeoffs and technical trade-offs that you may be asking yourselves when building one of these ingestion pipelines for streaming. And then deep dive into some architecture designs that we see every day in the IoT clickstream telemetry use cases. We're going to show you zero bus. How does this augment your your architecture? We're going to do a demo in action and then finish up with some closing thoughts.
1:25 LakeFlow Connect pillars
So what is LakeFlow Connect? LakeFlow Connect is a unified intelligent data engineering solution built on data bricks. So all of this is built on top of our unified governance layer of unity catalog with optimized storage with delta lake parquet and iceberg. So those are our open standard of table and file formats. So lakeflow has three pillars. There is connect which is everything to do with ingestion. There is declarative pipelines.
So this is everything you need to do um for your ETL. So this could be transformations. And then we have jobs for orchestrating those workloads whether they are analytic workloads or AI workloads. So connect the goal of connect is to provide ingestion solutions that are simple and lowmaintenance and are unified under our lakehouse architecture. And finally it's efficient end to end to bring that in in that data in so let's talk a little bit when we want to build one of these ingestion pipelines the business and technical trade-offs that we encounter.
2:23 Business and technical needs
So I'll talk a little bit about two like maybe common use cases. Maybe you have these in your business. So maybe I want to have an ML recommendation system. I have something deployed, an ML recommendation system. It's making recommendations to my customers. I want to archive those recommendations for historical analysis to understand how well we were making those recommendations and maybe use them for training later down the road.
So I want to collect that historical data. But because this is a serving application, I want to get this data off of that serving app as soon as possible. Similarly, in the clickstream use case, maybe I want to run an AB test. So I have edge devices like a mobile phone running an app or maybe a web browser and I want to build a business dashboard so we can see the results of that AB test. Now the business like the person looking at this business dashboard may be only checking it once an hour. it they're not looking at it all all times during the day.
But from the front end, we want to offload those events quickly. So let's deep dive into a customer use case that we that we've been working on with. So Joby is a completely vertical vertically integrated business. They build their own aircrafts and with these aircrafts along with the manufacturing pipeline and their flight analytics, they're using all that data to improve the technology. They want to improve the device.
3:38 Joby case study
They want to improve the aircraft. They want to improve the speed. And they want to be able to trace that data all throughout the path from the manufacturing to flight. So let's start with their architecture, but let's start at the business goal. So in the upper right hand corner here, we have a business goal. I want some sort of dashboard. I want to build some sort of AI agent. Maybe some transformation transformations on my manufacturing data.
Now to get that data in today, has several factories. Factories that are manufacturing different parts of the aircraft. Now these factories operate independently. They have some sort of forwarding agent. This forwarding agent today pushes it pushes messages to a message bus. Then now in the message bus, these messages are durable and we can ingest these messages using Spark into our data lake. Now there's a number of different challenges that has here.
So one is that every factory is different. They have a different purpose. They have different equipment. The machinery at each of these factories are different. They have different standards. The individual devices are coming from different manufacturers and they may change over time. And finally, each independent factory site has their own limitations and requirements. They have different dependencies. This could be the internet reliability of the site.
They can differ from place to place. So we want to be able to offload that data as quickly as possible. So we understand a little bit more of the use case, but who is actually operating on this data? So I want to ask this question of like who who is the data operator here and what is the time to action that's actually required for the business. So who is looking at the dashboard? So we want to build some sort of dashboard and when are they looking at it?
5:31 Architectural challenges
So I mentioned a business user is looking at the dashboard. When are they looking at it? Maybe a couple times a day, maybe we don't need some like real-time ingestion. Who is reacting to the pager? Maybe we have these events coming in. We want to page an engineer. Pager goes off, but it may take a minute or two for that human to like react and wake up, get their laptop and uh and fix the system. So, we have a human looking at the dashboard.
We have machines that are waking up humans and machines that are maybe performing nonrealtime analytics to understand the performance of the overall system. So the point that I want to make here is that we actually don't have any real time operators on the data. So just to go over some of the telemetry and telemetry business and technical constraints here we have leadership who wants a dashboard. We have developers who want historical data.
But on the technical constraint side, we have buffering constraints. So there's no real-time business need. But on the technical side, we want to offload this data as soon as possible. This could be due to, you know, reliability of the machines. These connections could be ephemeral. We could be losing missing this data if we don't offload this data quickly. So how can we offload this data quickly from the source? So let's dive into how people are doing this today with like IoT, clickstream and telemetry architectures.
So typically you add a message bus, you add some sort of staging area to be to make that data durable. So here's an example of a telemetry use case. So you may have a series of cloud applications that are pushing data to a message bus. You're ingesting that data with Spark into your data lake. And the clickstream use case looks awfully similar. You have these different kinds of edge devices again pushing data to the message bus making it durable.
7:27 IoT and clickstream flows
You're using Spark or some sort of other ingestion tool to persisted into the data lake. And then finally the IoT case that we went over with. Now we have some sort of forwarder again pushing data to the message bus and then ingesting it. So we have this two hop architecture across all of these different use cases. What problems does this create for us? So, first of all, there's the operational complexity of now I have this middleware.
I have this middle piece that I'm going to have to manage. But also, the data is duplicated. We have data in the message bus as well as in the lakehouse that we need to govern. Things are getting really complicated. In addition, because we have this new component, this new piece of infrastructure, this is a new tool that your engineers are going to have to learn how to work with. So this creates additional complexity and additional skills required to do the job.
So now we have more problems to solve. How can we reduce the data duplication, reduce architecture complexity and offload this data quickly from the source. So I'm happy to announce, you probably heard this in the keynote, zero bus with our direct write API. So the goal of this zerobus architecture is that we streamline your ingestion at scale as part of a unified platform with data bricks. So you'll be able to have these producers, they could be these edge producers, they can be cloud applications and instead of talking to a message bus now you can talk directly to our service or zerobus service and it will take care of acknowledging the message making that message durable and persisting it in your data lake.
8:54 Zero Bus direct write API
There is no setup required. You just create a table and start pushing data. So just some overall key benefits and some some stats here. So today zero bus our direct write API that we're offering um does not do real time but we offer five minute lat uh 5-second latency to get that data in. In terms of throughput, for every connection, we offer roughly about 100 megabytes a second of throughput, but we can support thousands of connections all writing to the same table simultaneously.
This completely streamlines your architecture. We've reduced operational burden. We've eliminated an additional hop that the data had to go through from source to sync. And because of we eliminated this extra hop, there is no more data duplication. There's only one source for your data and that is your Delta table within data bricks. This is all part of the unified datab bricks platform. And because we've built this on top of data bicks primitives like delta, it works seamlessly with the rest of the data bricks platform.
So I want to talk about sorcom. Sorcom has been a great customer that we've been working with. Sorcom is a native IoT platform that helps accelerate innovation by handling the heavy lifting that's typically used that comes with connecting devices to the cloud. So this is a quote um from Sorcom. We can now push highfidelity data from devices around the world to the data bricks intelligence data intelligence platform using our cellular satellite and low power wide area network services to accelerate time to insights for gamechanging results.
10:39 Soracom use case
So this greatly simplifies now for Sorcom and Sorcom's customers how they streamline data from any device around the world now to data bricks to get real insights. So a little bit of what the architecture looks like before. This was our multihop architecture. We had these producers both cloud applications, edge devices, factory site forwarders pushing data to a message bus. Now we've eliminated the message bus. We've eliminated those ingestion jobs that you had to man manage before.
Now you just push directly to zero bus. No setup. The data is materialized in your lakehouse. So let's jump back into the case. We learned all about and problems. So from fac from factory to flight um let's show let's see how zero bus helps. is able to use our manufacturing agents with LakeFlow Connect zero bus to push gigabytes of data of telemetry directly to our lakehouse accelerating time to insights all with data bricks lakeflow and the data intelligence platform.
11:56 Joby solutions
So this is a piece of technology that we're helping customers like and Sorcom scale. So zero bus is helping here to streamline the ingestion for this telemetry data from their ma manufacturing sites to data bricks. It it's eliminating that complexity. But what's really special here is we've given the joby the ability to build these onrem agents. So Joby has these unique needs that we covered before that on their factories.
Every factory is different every device is different they have different needs. We've provided them with our zerobus SDK where they've been able to build a and support their own agents to maximize logo stability and network independence. So together with zerobus and these agents that they were able to build and are building this eases their implementation in a plug-and-play manner. Implementing the on-prem agent can be done in a plug-and-play format on their existing pipelines.
So very little needs to change. So this was their architecture before and now they can run their own customized agent for their needs using our SDK to push data to zero bus as part of LakeFlow connect to materialize this data in near real time into the lakehouse. So I talked a lot about zero bus but now let's see it in action and let's actually build something. Yeah, thank you. So let's do a brief overview. What are we going to do as part of the demo?
13:51 Demo: Building a data agent
So we are going to build the performance metrics collecting and forwarding agent uh similar to the example uh job had that which showcased and once uh data is in delta bricks we're going to use declarative pipelines to postprocess the data and then we're going to build some uh dashboard and alerting on top of this data. So uh let's dive in. switch to my laptop. Hopefully, you see something. Um, uh, we're going to start with a simple Python application which does, uh, basic performance metric collecting from my computer.
Uh, so let's run it, see how it looks. So, you can see it uh, in real time collects uh, CPU usage, memory usage, disk stats, and some network statistics as well. And what we are going to do, we're going to upgrade this application to become this forwarding agent uh so that we can uh forward this data to datab bricks and scale this app across fleet of devices and build some monitoring of this fleet. So let's stop the app for a second and before we start changing the app uh we need a table in datab bricks workspace.
So I what I did in advance, I pre-created uh the table with appropriate schema matching the data I'm collecting. Uh and I'm just going to run a a sanity check query just to confirm that table exists and that we have some rows there. We have about like 3k rows. So now we can go back uh to our app. And before we start changing any code uh what we need is to uh generate proto schema uh which uh matches our uh datab bricks delta table schema and it's simple to use this uh by using zero bus SDK u it's one command to generate this protoile and now that we have protoile we need to generate uh uh protobuff code so it's also a single command and after running these commands here we can see a protoile generated it roughly m I mean it exactly matches the schema of the datab bricks table I created uh so let's now go to our app what we need to do is uh first import uh the zero bus python SDK and create SDK handle sockk handle is created by uh providing adding server endpoint datab bricks workspace URL and authorization token.
And now we can go to the code of our app. And here we have the matrix collection loop. Uh and below it we have uh the looping of matrix collection. So uh before going into the loop we need to create uh table properties object by using table name and descriptor of generated protobuff. And then we can create stream by using uh SDK handle and invoking create stream on uh table properties. Now we can dive into the loop. Here we have a matrix collection.
Uh here I have the structure I used uh for my simple terminal applications. I'm going to switch this to protobuff generated structure and I go below all the metrics and we don't need anymore to dump this to a terminal. We are going to send it to our delta delta table. And how are we going to do it is by creating a record future object uh by awaiting on stream in just record command. Now what will this will do is it will uh without blocking just send the record to delta table but it would be also good to uh periodically for this kind of app uh await on the record future and get the act back that uh which means that the data is persisted in datab bricks table.
So let's use loop count that I have added and await on record future. So uh this will confirm like that the act has been received and I'm going to since I'm taking about like one second to collect the metrics I'll do this once a minute and also for purpose of um uh exception handling I'll gracefully close the stream and finally close. So now that we are code complete, uh we can go ahead and uh rebuild our app. Let's reinstall the new wheel and rerun.
So now the app is starting. As you can see, stream is created and record starts flowing. So with each loop, one record is sent to our table. So let's go back where we created a table to our workspace. You can see the count is 311. Let's refresh. It's now 3115. So it's continuously increasing. uh here we I I have pre-created a continuous declarative pipeline which processes the data does some filtering categorization prepares the data so I can build a dashboard like this so let's refresh the dashboard here on desk dashboard I can see my whole fleet of devices uh besides my laptop I added my dev machine here um so I can CPU usage over time we have disk usage over time we have memory usage We have uh network data will be populated and we have like uh OS distribution of the device pushing data and also in my last tab I have some alerts also that I created uh alerts like uh prolonged high resource usage network errors disk billing full or frequent restart of devices.
So now that we have all this let's come back one more time to our app. It's still sending rows as you can see, but let's wait for a minute and let's change code a bit. So, let's hardcode here that disk usage percentage for this device is 95. So, it's like near full disk. So, we're going to mock that my laptop has this problems and we are going to rebuild the app and redeploy same as previously. Again, app is starting, stream is created, and we are good to go.
If you go back, let's go back to our dashboard. Let's go back to disk usage over time usage. Let's refresh. As you can see, uh dashboard immediately spiked and we immediately detected that one of our devices has the disk issue. So if you go back, I'll go just for a second one more time to the app. Uh we have the loop is still going on. And here we see like how fast rows are are actually going. Like my logic of my application takes 1 second and it's like usually 1.
01 or or nothing to send a row and the only actual like uh longer processing that we have is when we wait for act and here it's 2 like 16 seconds but it can go like up to 5 seconds as uh mentioned earlier in the slides. So here you go. This is the acro here. Um so we can go back again if you also one more time refresh the dashboard. We'll see like this data continues like issue is not uh resolved yet. And um yeah we can also compare the number of records we have ingested so far.
And yeah, we can see like that for all this time we ingested like 200 new records. And um that's for the demo. So you got to see zero bust in action but now you're probably asking yourself like when can I use it? What's available today? What's coming tomorrow? So today in the month of June we are in private preview. We have Azure and AWS support. We support at least once delivery of data and we have customerf facing SDKs that you can use in both Python, Scala, Java.
23:28 Roadmap and features
So you can start building one of those forwarders that we showcase today. Now the message encoding over the wire is protobuff. So we're looking at maybe expanding that to other other encodings in the future based on customer demand. So come talk to us, let us know. And like I said that we have we support 100 megabyte per second throughput per one of these connections. So per stream that you open up through the SDK or from a network of a collection of devices and in terms of message sizes our our limit is 4 megabyte size messages that you can send down the wire.
So what's coming in the future is expanded cloud support for for GCP an API for exactly once delivery of data and also a complementaryary batch file API for larger bulk use cases and lower latency acknowledgement of the data. Now that's all the ser service side but in terms of table features that we support today versus tomorrow to today we do not support schema changes as part of the service but as a fast follow we're working on supporting schema changes such as type widening um default value support and this new concept of a quarantine table so that when we have bad records we can still make those records durable but instead we put them into this alternative table so you can rest assured that your data is durable. but your target table remains clean.
25:27 Closing thoughts and Q&A
So, just some final closing thoughts before I let everyone go. That was a lot of information and you're probably thinking this is just another ingestion tool. Data bricks has a lot of different ingestion tools. We have Spark, we have declarative pipelines, now we have zero bus. When should I use one of these? Like what is the right use case? And I don't think there's any perfect answer, but some guiding questions to help you with the thought process is what I want to leave you with today.
So we have the question of what is your data source? And when you think about your data source, what makes more sense? Would you rather push data from that data source? If you're pushing data, zerobust makes a lot of sense. If you need to pull data from a data source, maybe an alternative solution like Spark is the right choice. Do you need who needs to consume this data? is another one. So, do we have operational consumers consumers as well as analytical?
A lot of the examples that I gave today of architectures only had analytical consumers of the data and more specifically the only consumer of the data was for ingestion. So, if you only are using this message bus for ingestion, zerobus could be a really good alternative architecture. Also, how many consumers of the data do we have? a single or multiple. If you have multiple consumers of the topic on the message bus and some of those are operational applications then maybe it makes sense to keep that message bus around and having this replication pipeline through declarative pipelines or spark.
And the other question is who is the data operator? Do you need to have real time is near real time support enough? Like I mentioned today, as part of zero bus and its direct write API, we support 5-second latency from sending the message to then being able to query the data. But if you need lower latency than that, maybe this is not the right choice, at least for now. And then also the other the other point I want to make here is about customization requirements.
What kind of customization do you need to do? Zerobus today does not do any transformations of the data and that is by design. What we've built is purpose-built compute specifically to streamline ingestion from source to sync and by we this is why we don't add transformations in the pipeline. So thank you all for coming today. That's the end of the talk. This is the last talk on LakeFlow connect um that was part of the schedule.
I want to urge everyone to go on the app and fill out the survey. let us know that we did really well. And if you have any questions, come meet us at the front um by the stage and we're we're happy to talk more. Thanks everyone.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.