ZeroETL Was Just the Start: Operational Databases belong in the Medallion Architecture

Summary

Lakebase eliminates the gap between operational and analytical systems by acting as both the Bronze layer—natively feeding downstream analytics without data copies—and the Gold serving layer—delivering refined datasets to production applications at low latency.
Sync tables demonstrate the scale of this integration, with 16 billion rows residing in the lakehouse backed by zero-latency writes with asynchronous propagation to the analytical layer.
Lakebase supports AI agent backends, pgVector for semantic memory, and database branching for safe schema changes, extending the medallion architecture to cover the full operational and agentic data lifecycle.

ZeroETL Was Just the Start: Operational Databases belong in the Medallion Architecture

The medallion architecture dictates how we typically think about data, but that framework has historically been limited to the data lifecycle strictly within the Lakehouse. Lakebase takes this a step further by expanding the architectural blueprint to redefine how data enters your Bronze layer and how it is consumed from Gold. Rather than extracting and moving data across system boundaries, your operational database natively acts as the Bronze foundation to immediately feed downstream analytics. On the other end, Lakebase closes the loop by directly serving those refined Gold datasets to production applications with operational low latency, eliminating the need to export data out of your ecosystem.
Talk By: Pranav Aurora, Senior Product Manager, Lakebase, Databricks ;

Chapters

FAQs

How does Lakebase fit into the medallion architecture?

Lakebase can serve as the Bronze layer by acting as the operational data source that natively feeds downstream analytics in the lakehouse, removing the need to extract and copy data. It also serves as the Gold layer, reading refined analytical data back into applications at operational low latency.

What is zero-latency writing in Lakebase?

Lakebase supports synchronous writes at operational speed with asynchronous propagation of that data to the analytical layer. This means applications writing to Lakebase see immediate write confirmation while analytics on the same data are kept current without a separate ETL job.

How does Lakebase support AI agent workloads?

Lakebase can serve as both the operational backend and persistent memory for AI agents, with pgVector enabling semantic memory for retrieval and reasoning. Database branching lets agents experiment with schema changes safely without risking production data.

What was the problem with traditional OLTP and OLAP separation?

Traditional architectures required two completely separate systems—one for transactional workloads and one for analytics—with a data movement pipeline in between. Lakebase addresses this by unifying both ends on a single platform, eliminating copy, transform, and governance overhead.

Full transcript

[00:08] Hello everybody. Good evening. I know this is extremely late in the day. And yeah, zero ETL is just kind of like tough topic to cover at this time. But I will try my best. Um
[00:20] Love to meet you all. My name is Pranav. I'm on the Lake Base team. I believe Lake operational databases belong in the medallion architecture. I believe this so much that this is my third time trying to solve this problem.
[00:32] I started at a company called SingleStore um where we built an HTAP system where we thought we could do OLTP and OLAP in the same place. I started another company after that called Mooncake, which was a way to bridge
[00:45] Postgres and Iceberg. And that was kind of painful, too. Like didn't solve the problem the right way. And I think we're going to do this the right way with Lake Base this time around.
[00:58] Cool. Some of this is stuff you saw this morning, so I can kind of breeze through it. But yeah, there are two ends of the data stack. The OLTP end with low latency reads and writes. Uh and then the analytics end based on
[01:11] large aggregates and scans. It's pretty well known, pretty well defined. Store data in different formats, different engines just with different properties designed for these workloads. Indexes are different across the stack.
[01:23] And over the last couple of years, it finally feels like there's consensus. Like both ends are kind of solved. Like you should just go for OLTP to a disaggregated storage and compute engine
[01:35] like AWS Aurora. That solves some real problems in databases and and and you know, get that. And on the OLAP side, just get a lake house, right? Like store data in cheap object stores in open formats. Bring your own engine, whether
[01:47] that's Spark or that's DuckDB. Um whatever you need. I guess both ends of the problem or both ends of the stack have been pretty well innovated on and pretty well solved.
[02:02] For each one like that I go back to like you know, you look at innovation in the last couple of years, it's pretty good. Like OLTP now has provision flexible provisioning. You can get really really high levels at scale. Most OLTP systems have like four levels of caching. You don't need to keep all data in in in
[02:15] memory. Uh and you get good economics on the OLTP side with you know, cold data on object store. You pay for what you need. Computer's largely serverless. And the OLAP side you get one copy of data for all analytics. It's unified governance
[02:28] with UC. All of the ML AI workloads on the same side. It's cost-efficient, right? So we've solved individually a lot of pains. But still most enterprises look like
[02:40] this. Um on one side you have your application that's writing data. That's you know, whether that's chat history today, whether it's a banking application. To do anything with this data, whether it's
[02:52] reason on it, analyze it. You kind of see a typical stack of I set up a Debezium into a Kafka into a Flink. I get this into the lakehouse as my bronze table. And then finally that's can be made useful.
[03:04] Um and often times you have two stacks. You have a real-time stack with Flink and a batch stack with Spark. And you're still negotiating 15-minute hour-long delays across teams, right? This is on one end of what we see with OLTP and
[03:17] making OLTP data reasonable into the lakehouse and you know, run analytics on it. On the other side, once you're done with all your processing, you've built your features, you've built your you know, you cleaned the data, you've created a gold table. I want to serve
[03:29] it. Great. Now I go get Airflow and then I write my own JDBC sync, streaming sync for real-time into another OLTP, the reverse ETL process. And I do the one for the batch mode. Like we see this all
[03:41] over. And you're dealing with things like logical replication on the OLTP side. You're dealing with things like bulk uploads on the other side of the OLTP. Things that OLTP systems just aren't designed to do well. You just want it to run low latency
[03:55] queries consistently. You don't want it to manage its own replication slots to handle a massive bulk upload in between, go in and, you know, deal with locks and managing these tables yourself.
[04:11] I kind of look at it as like there's two separate stacks with some duct tape in between with man tree data duplication, two assets to govern, more often more, where you have data in Debezium and Kafka and Flink. Like, how are you doing
[04:24] any governance on that? And then, the last one I really care about is it like it actually takes down OLTP workload, right? Like, when you're building on an OLTP system, you kind of want it to be just mission critical,
[04:36] reliable, just on boom, millisecond level things, and logical replication bulk loads just not designed for these workloads. And a lot of attempts have been, you
[04:48] know, made to unify all these things. And I think most recently, the kind of things we agree on is like on one end, we have zero ETL, which is like I can get data from my Aurora or any OLTP into an OLAP system. It's zero
[05:02] ETL. It's kind of lies. And then, on the other hand, there's like a lot of reverse ETL tools, right? Like, oh, it's you get your gold data, now you get in a reverse ETL vendor uh to get this data back into a system ready to serve.
[05:16] Again, kind of lies. Yeah, instead of building better pipes, let's build a better database, right? That that is like the approach we have with Lake Base and what we can do with
[05:29] building an OLTP into the Databricks platform. Some of you might have seen this before. We We talked about what a Lake Base is. What makes a Lake Base a Lake Base roughly a year ago. We've seen the shift of, you know,
[05:42] across the last maybe 20 years, where you start databases have been this monolith system you deploy with storage and computers tightly coupled. Over the last couple years, maybe the last 10 years with systems like Aurora, you've seen sort of separation of these
[05:54] two things where the database engine is is is separated from storage. You can scale them independently. Great performance, you know, better better economics. Cool, the right direction. What Lake Base is is building an open
[06:08] OLTP system. You saw some of this today with Altap. We believe all data should be stored in these open formats in cheap cloud data lakes. And you can throw the engine at this data for the right task.
[06:21] For read queries that need, you know, single digit, double digit millisecond level things, use Postgres on this data, get fast performance. Things like bulk uploads, you can use Spark to write this data, bypass Postgres compute altogether. Postgres sucks at bulk
[06:34] uploads. Right? It's it's it's it's it's going to going to eat up all the wall slots. Um so, you Spark parallelize, write data directly to storage. Um this is the era of the Lake Base.
[06:51] And we do have a Lake Base at Databricks with Databricks Lake Base. It's just Postgres. Um it's a full Postgres built on a fully sort of serverless auto scaling compute. Um the key slot here on the bottom is like
[07:04] we don't believe there should be silos between the operational and analytical stores. And we can give you central governance with UC with this. So, how we built this is we first separated storage and compute for OLTP.
[07:17] A lot of this comes from the Neon architecture. Um you know, Neon storage engine that we we have, you know, built into Lake Base. What we did a step further, with stuff you saw today with Lake Base with with
[07:30] Altap, sorry, is being able to use open formats, Iceberg, Delta, as durable cloud storage. Right? So, that's This is Gen 2. This is Gen 3.
[07:42] And now they have the foundations as the Lakehouse, right? So, Lakebase and Lakehouse share the same storage foundation. That is Lakebase. It's serverless post first on the lake.
[07:54] A cool key property of Lakebase is you can access Lakebase storage directly. One way to think about what we showed with ELT app this morning and the demo
[08:07] was I have an app that's writing these tables these transactional rights and I'm also serving these transactional reads out of Lakebase. But that doesn't mean I can not use DB SQL to get fast analytics on the same
[08:20] data with transactional consistency. That doesn't mean I can I cannot offload certain operations to Spark, whether that's bulk rights, index builds, data is just stored in these open
[08:32] formats that engines already know how to read and write. And this is very very fundamental to what makes Lakebase a Lakebase.
[08:46] So, let's take a common workload, right? I have a reverse ETL pipeline. After I do a bunch of processing, I create my gold table into Lakebase. And now I want to serve this. I want to get millisecond level reads to my applications on these gold tables.
[08:58] What you would normally do in a legacy OLTP system is point this massive data set into compute and say go deal with it. Go write this data, right? While it's serving the application workload. You can say, "Oh, I've got read replica
[09:11] Why did that happen? Um anyways, yeah, it's like you're throwing you're you're doing these rights directly into the computer of the OLTP system.
[09:27] What if you did this? What if we wrote to storage directly and not talk to compute? Let compute for Lakebase just focus on serving that read workload.
[09:41] That is some of the stuff we're doing with Lake Base. You can run bulk uploads and bulk operations without touching the OLTP workload. For legacy OLTP systems, this means writing directly to computer that floods wall, you're managing cache, you're managing out of memory issues.
[09:54] You know, your your serving latency tanks at the time of these operations. You're managing connections, long-running connections because bulk uploads take these long-running connections. With Lake Base, we can just write this data directly into object store. We can bypass Postgres and the the OLTP traffic
[10:09] and massively benefit from parallelism, right? I can use Spark, distribute this across hundreds of nodes, write Postgres pages in place, and then just tell Postgres, "Hey, I wrote you a bunch of data. Don't forget to that to read them."
[10:21] Um and that's what we do today for data loads. So, sync tables is like a mechanism that it might have Some of you might have played with. It's a way to get data from the lakehouse into Lake Base. This is now powered by this mechanism. It's called direct writes underneath the hood.
[10:34] And very soon we're bringing this to index builds. I don't know if people have, you know, built indexes on large tables in an OLTP system. You know this is kind of scary, running a very, very long-running transaction while building your app on it and just kind of praying
[10:46] everything works. We can delegate that to Spark, too. Like, why not just build these indexes in parallel? A great example is our new vector search index, which is highly parallelizable because of the, you know, the the architecture we have with
[10:58] K-means clustering. You can also do this for large DDL. So, this is some of the things that we can do with ELT app, with our Lake Base architecture of having compute and storage be so separated and storage
[11:11] being on these open foundations that anyone can access, whether that's Spark, whether that's analytic computes. Right? So, this is, you know, a key differentiating feature. Um so, this is available today. Uh this
[11:24] powers sync tables. A common mechanism we see is people do all their processing on the lakehouse, create their final gold data set, and say, "Now I want to serve this." With like, you know, two buttons and two clicks, you can get this into Lake Base superficially. This replaces your
[11:37] reverse CTL into a external OLTP system. All of this is now on the Databricks platform managed by UC, and it's fast. How we do this is, you know, Spark workers parallelize page and index builds directly to object store, and
[11:50] then we just tell Postgres, "Hey, this is a pointer to your data." The way to think about this now is like, you can get data available in Lake Base, and you can use Lake Base as a cache for your serving workloads extremely, extremely
[12:03] efficiently. >> Yeah. I guess like this is numbers from a customer I shall not name, but they were running their own custom JDBC sync
[12:15] last year, this time last year. Um, and this was, you know, hours and hours long. This was running daily. They were uploading a bunch of data from their gold table into uh into their external OLTP system. Last
[12:29] year, we got them on sync tables. That became 4 hours, and that was a win. Like, that was we had to convince them that we're not doing anything wrong, and that's actually 4 hours. Today, that's down to 2 minutes.
[12:42] And that is direct writes for you. Um, and okay, you might say 500 GB is tiny. Come on, like, we're analytic people, we're lake house people. Like, talk to me in like real numbers.
[12:55] Let's try Let's try some real numbers. Sorry, this chart may not be the best way to show this, but uh Let's try 10 TB. Right? It's still sub 20 minutes. You can You can get 10 TB of lake house
[13:08] data, which explodes, by the way, a lot more and when you when you get rid of the columnar compression, into Lake Base, um, and you get them ready to serve into an OLTP system in less than 20 minutes. And this is
[13:20] still work we have to do. How we do this is just throw more compute at the problem, right? This is what Spark gives us. So, have to go think about these things. I just tell Spark, "Go do this." And it'll figure out how many computer how many executors to throw at the
[13:32] problem and and get it done efficiently. So, we completely get rid of the OLTP bottleneck for these kind of bulk operations, for these bulk loads. Uh and it scales really well. It's It actually It's really cool to see some of this work.
[13:48] This is you know, mechanism that some customers have been using in beta and I've seen some really exciting numbers. WMG being one uh and they've been so kind to, you know, allow us to share some of their uh their performance gains. They were using
[14:00] sync tables on 16 billion rows of lakehouse tables to get data to serve. That was a very common workload. So, they're doing all these analytics on on artists and songs and, you know, what what artists are performing in which country really well, what song is doing
[14:14] well in which region. That was all done on the lakehouse uh with our analytic engines. And eventually, they were building a user-facing app like a It's a It's an app that artists can use and log in and see how well they're doing.
[14:26] Just to get the data every day updated into Lake Base was, you know, hours and hours before this. With sync tables, we got it down to 240 minutes. And that was a win. Today, it's at 10 minutes.
[14:38] Uh and this is while we're still in beta. And this is, you know, cost and performance savings for you, right? You're not provisioning massive OLTP systems to accommodate for the right because the right workload is completely separate. So, you can have your OLTPB
[14:51] extremely small footprint during the sync. And the pipeline runs a lot faster. What we want to do next is index builds. Uh a current benchmark to think about is like 16 billion rows takes 30 minutes
[15:03] today. And we can apply the same kind of parallelism and scaling laws, and you can kind of imagine where we're going. So, in the not-too-distant future, you'll be able to to load this data into the row-oriented format and index it in
[15:16] minutes. And this is large amounts of data. And this is separate from the serving workload. So, I want you to think about it this way. I'm building my medallion
[15:28] architecture. I eventually create my gold table. That's a win. This gold table is, you know, what I want to deliver to my applications, and I'm doing maybe point lookups on this gold table.
[15:40] Lake base can now be a good serving engine for these workloads, especially when you're incorporating this with the rest of your application. A very common one I see is actually to do with agent memory, where the lake house has a lot of cool functions on
[15:53] parse documents and creating embeddings, which are extremely parallelizable. So, you can create, you know, hundreds of billion hundreds of millions of rows of embeddings with a lake house. And then I want to serve this. I want to build a an agent on top
[16:05] of this this rich, you know, context, this rich history, and I want the agent to be able to reason with the data that exists the the the data that I've processed and and and and indexed.
[16:18] Um direct writes is a great mechanism with sync tables. So, you process all of this with the lake house, and then I want to serve this to an agent, create a sync table. This will get data into lake base very efficiently, and agents love writing post grass for some reason. So,
[16:30] like agents will know how to read this data and query this data with PG vector um and and and and post grass really efficiently. So, I want you to think about it as like I'm building an application. Some part of the application has data
[16:43] requirements from the lake house. Use lake base, use sync tables with direct writes, um and hopefully we can get some of, you know, some performance wins here.
[16:55] Any questions in this because I'm going to like change tone pretty quick. Feel free to Cool. Okay, I'll take some at the end as well. Uh cool. Great. What about getting data out of
[17:08] lake base? This is another challenge I've seen a lot of OLTP operational people thinking about like, how do I get data out of my OLTP system? And then you go look at each other saying, logical replication, I'm going to manage these slots. I'm going to like
[17:20] figure out how to take these snapshots that run daily. And then if schema changes, uh figure it out later."
[17:33] This is what we showed today, right? Like Lakehouse writes data into these open formats with directly with ELT. So, you can just, you know, get other engines to reason on this data can completely keep that separate from the
[17:45] OLTP workload all together. Analytics being one. This was the demo at uh the keynote today that you saw. And the reason why we can do this is because the data's transposed with lossless compression into these open
[17:57] formats. People know how to read these open formats. So, run analytics directly on these tables. The other thing that we launched in public preview and and we're heading towards GA is change data feed. So, I'm
[18:11] I'm I'm sure the Lakehouse folks are familiar with this concept of Delta change data feed. You can go build pipelines with it, only reason about incrementally written data. Lakehouse has that too. So, like part of this ELT story is
[18:23] you can kind of do what you wanted to do with a a Delta table with Lakehouse, right? It's it's it's it's integratable into your OLTP in into your medallion architecture. Um and this is a very common workload I
[18:35] see, which is people have rights into Lakehouse often for, you know, chat history. I'm building an agent. Uh I want to store what my my users are asking. I want to see what each uh loop in the agent is writing. And based on
[18:49] that, I want to go build downstream eval pipelines. I want to go build downstream analytics, you know, what's the ratio of good thumbs up to thumbs down, what documents are doing well, what questions are being answered really well, what users are really happy, kind of
[19:02] downstream processing in like eval frameworks. Lakehouse works really well for this workload, right? As you write in directly into Lakehouse, it has a CDF mechanism. So, you can point Spark declarative pipelines or DLT or any
[19:15] engine to read the CDF, this change data feed, only read the incrementally changed data, process it, right? And go build downstream pipelines. So, customers are using this all the time,
[19:28] especially for agent use cases, um and up-leveling their, you know, lake-based data with the medallion architecture, and CDF enables this. And again, this is completely isolated
[19:40] from the OLTP workload. And yeah, this is something that, you know, we should we should think about in terms of getting data out of lake-based. You can also point the CDF to, you know, stream processing, um to Kafka, to other Postgresses. CDF
[19:54] is a mechanism for you to, you know, read incrementally changed data and write it to another system. So, quick demo, like, kind of not very happy
[20:07] with the demo, but I I'll try my best to make this entertaining. Uh here I have like a agent use case, which is like I am collecting a bunch of data. I'm I'm I'm building an agent, and then like I have agent runs. So, like,
[20:20] what is the agent being used here? So, in this case, I have like a data analyst agent. This is the session. This is the task it was given, whether it's completed or failed. And then, for each agent run, I have multiple steps. So, I
[20:32] can see like, okay, for this run, it did this tool call, it reasoned, it did this retrieval, it did this response, it wrote this file, um tokens it used. So, I can get This is typically how people set up lake-based
[20:46] Postgress for some of the agent back-end workloads. I also have a table for user feedback, right? Like, what is good, what is bad, um thumbs up, thumbs down. You can think about it that way. All this is all very rich data, right? If I'm getting all
[20:58] this information from how customers are using my agent, I want to know how I can make my agent better over time. Um and CDF gives you a good functionality to do that. So I can go set this up once at the schema level.
[21:11] This will write the change data feed into you see as a open table that I can reason with and then once I set it up all existing and future lake base tables will have a CDF enabled with it
[21:23] and you can run schema changes on it and the CDF will you know represent the schema change on the destination side. It's a table in you see with history right so it's just like a a
[21:36] table in you see that's an open table with with history. When I query this table so let's say I query this table in in you see what I see is a couple rows. I see inserts and then for updates I can
[21:48] see a pre image and post image and then delete. So we keep the full history right so one great use case for this is like audit logs. I want to see the full change history of everything that's happening in my old TP who's working on
[22:01] what when these changes happened you know when data was deleted and be able to track that to a particular post cursor lesson. So you can do that. The other thing you can do is go build
[22:13] an STP pipeline with it. Go build a true real time pipeline a streaming pipeline. I don't know how to do any of this like I actually I'm really not that intelligent on lake house stuff. So I asked Genie to do this for me which is
[22:26] go read these tables and just like help me build a pipeline of like what I should do with this data like how I can up level this and make useful information for my business right like I have all this rich context coming in.
[22:39] What should I do? And it kind of just does it like it it it built me a fine tuning data set built on sort of real time data that was coming in directly from my application. I up
[22:51] leveled it oops sorry. Sorry give me a second I need to go back.
[23:05] Um I up-leveled this and eventually I got a couple tables. I got a silver table, which was just cleaned up, and then I got two sets of gold tables for me. What do I really care about? I care about uh
[23:18] feedback that was really bad to understand what was bad about it, and that can be my negative like fine-tuning data set, like data set I used to understand these were the bad operations. And then I have my good data set, which is like things that customers said well done on.
[23:32] And for each of these data sets, I can use that to go back into my agent, right? And like understand these are the these are the tool calls that are problematic. These are the retrieval methods that suck. This is what I should go in and iterate with. And you can do this all in real time without
[23:45] configuring pipelines, without configuring, you know, separate sync modes, and and understanding how do I get my data to the lakehouse. With Genie, with like two buttons, kind of get it going.
[23:57] So, that's kind of what we see a lot of customers do with Lakebase CDF. Um and you can now build pipelines with Lakebase as the source of your medallion architecture. The way I would like to think about it
[24:09] is like now your medallion stack, your medallion architecture, has an OLTP database built in. On one end, as your application is writing data, that is your not bronze table. Like you don't have to think about how do I create a bronze table from OLTP data, which is not that easy.
[24:24] Especially with the high velocity of writes from Lakebase or from from an OLTP system, especially getting data out of an OLTP system. And finally, once you do create your gold table, how do I bring this to applications?
[24:37] Think of Lakebase then, right? So, that entire medallion medallion stack can have an OLTP system. On the bronze end, it's your it's it's actually Lakebase with CDF. It's it's just your your your operational data,
[24:50] and on the gold end, it can be served via Lakebase to applications. And that's my talk, I guess. Yeah, these are core capabilities of ELTP. So, when you saw that today, this is
[25:02] what ELTP enables, which is you know, bulk writes on Postgres without impacting OLTP compute, run analytics on post on this operational data without any ETL pipelines, and use
[25:15] operational data to go and build, you know, fine-tuning data sets, real-time pipelines, all without pipelines, all without thinking about data sync, you know, how do I get this data into the right format? How do I get my data into
[25:27] into the lakehouse on the lake base? So, that is it. Yeah, I guess I'm like 14 minutes early, so I can put time for questions, but hope that made some sense.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.