OLTP on the Object Store: Reimagining Postgres with the Lakebase Architecture

Summary

Lakebase decouples Postgres storage from compute by running natively on object storage, enabling compute to scale to zero when idle, instant provisioning, and analytical queries on operational data without read replicas.
The architecture includes a multi-tenant page store, WAL dispatcher, and buffer pool with a write-ahead log optimization to address the latency challenges of object store I/O.
Agents are already creating and dropping millions of Lakebase databases, and copy-on-write branching enables safe AI-agent experimentation without touching production—a key design requirement for the 2026 agentic development landscape.

OLTP on the Object Store: Reimagining Postgres with the Lakebase Architecture

Traditional databases force architects into a familiar set of tradeoffs: scale compute and you pay for storage you don't need, add read replicas for analytical queries and you introduce replication lag, separate your OLTP and OLAP systems and you're managing data movement and consistency across boundaries. These aren't bugs in the design; they're consequences of tightly coupled storage and compute. Lakebase decouples storage from compute and runs Postgres natively on object storage. In this video, you'll learn what that architecture looks like, why it changes the tradeoff calculus, and what it makes possible: analytical queries running directly on operational data without read replicas, compute that scales to zero when idle, and transactional workloads sitting alongside lakehouse analytics on the same underlying data. We'll map the architecture to concrete use cases, including real-time applications and AI workloads that need fresh operational data, and hybrid workloads that previously required two separate systems. You'll leave with a clear mental model of how Lakebase works and where it fits in your stack.
Talk By: Hans Norheim, Principal Software Engineer, Databricks ;

Chapters

FAQs

How does Lakebase store data differently from traditional Postgres?

Traditional Postgres was designed for spinning-disk storage, coupling compute and storage tightly together. Lakebase uses a new storage engine that runs Postgres natively on object storage, separating compute from storage to enable elastic scaling, scale-to-zero, and lower costs.

What is the WAL dispatcher in Lakebase?

The write-ahead log (WAL) dispatcher is part of Lakebase's architecture that optimizes how database changes are written to and read from object storage. It addresses the higher latency of object store I/O compared to local disk, keeping transactional performance at a level suitable for production workloads.

How does copy-on-write branching work in Lakebase?

Copy-on-write branching lets users and AI agents instantly create an isolated copy of a database that shares the same underlying storage pages as the source. Changes in the branch do not affect the original database, enabling safe experimentation, schema migrations, and agent-driven development without production risk.

Why are AI agents a key design consideration for Lakebase?

In 2026, agents are already creating and dropping millions of Lakebase databases autonomously, far exceeding human-initiated provisioning rates. This requires instant provisioning, autoscaling, scale-to-zero, and branching—capabilities that traditional Postgres deployments cannot deliver at that speed and volume.

Full transcript

[00:08] All right, let's get going. So, my name is Hans. I'm an engineer from the Lakebase team. I work on the high availability in Lakebase. Um, I'm going to talk to you today about how we built OLTP on the object store and what kind
[00:20] of stuff we did under the covers in Postgress to make it ready for the cloud and AI. So I'm going to take you under the hood in Lakebase and uh tell you how things work uh when you create databases in Lakebase.
[00:33] So why are we talking about this right? Uh in 2026 we expect more more code to be written by agents than by humans and that is changing how applications are written and it is changing how those
[00:47] applications use databases. Right? In the old days, you might deploy a database by putting a server in a rack and putting a database on it and let it let it run there for 15 years. Maybe you patch it a little bit, right? But that was sort of how you used databases in
[00:59] the old days. Now you have data uh agents that create and drop and modify millions of databases all the time. And that changes the requirements from what we want out of a database system. And if
[01:13] we sort of look at what that translates into, right? Uh actually yeah I should also mention one thing we track how many databases are created in lakebase by humans and by agents respectively and just in the last two years we have seen
[01:26] that number grow by a lot. So it's it's it's real that uh that agents are going crazy with databases. Um in terms of what that what that means that we want out of a database right uh
[01:38] we want the database to be familiar and capable. Uh we want it to be Postgress uh because humans and agents alike love Postgress for its rich functionality. We want to do stuff like what we touched on
[01:50] in the keynote on Tuesday uh which is LTAP uh analytics, right? So you have doing analytics directly on operational data uh lake um lake transactional analytical processing. We want the
[02:04] database to be nimble uh because agents love to experiment and do things quickly, right? We want instant provisioning. We want branching so we can these agents can experiment without breaking production. Uh we want autoscaling and scale to zero such that
[02:17] agents can can scale these databases up and down and shut them down when you don't need them. And then of course you never know if your app might turn real. So you want this thing to be mission critical as well, right? You want it to be scalable, fast, and and preferably
[02:30] highly available. So that's sort of the the things that we're going to look for and and build for in in Lakebase. And now you might ask why do we focus so much on Postgress, right? And that is
[02:42] because Postgress is the world's most popular database uh and loved by humans and and agents alike. This is from a stack overflow survey. Uh it is very popular and is growing very fast and it's for good reason because Postgress
[02:56] is a very good database. It's open source, functionality rich, has extensions to do pretty much everything you want to do. Um and um and yeah uh it's it's a good starting point for a database system for for AI and uh and
[03:09] agents but that means that we have a bit of a challenge because Postgress started in the 80s as a research project up the road from here at Berkeley and a lot of the code in Postgress was written in the
[03:22] '90s and a lot of the architectural decisions was taken in the '90s. Uh there's actually live code in Postgress today that was written in 91 because it works so why change it right? uh but that means that a lot of the
[03:34] architectural decisions in Postgress were made before the cloud existed and definitely because before the the agentic workloads of today existed. Uh so how do we then take uh that keep it
[03:46] Postgress without making it kind of Postgress because we want it to be really Postgress and fully compatible and make that ready for cloud and AI. So what I'm going to do today is to take you on a journey on how we took
[03:59] Postgress changed it under the hood but kept it Postgress and built all these uh capabilities that we are shooting for uh shooting for here. So
[04:12] this is where we start right if you just take Postgress out of the box and don't do anything with it and put it on a machine you kind of get this you have a node you run Postgress on it stores data and transaction logs on local disk right
[04:24] and sort of that's the vanilla Postgress setup and of course it's Postgress there's no doubt about that but you can't do things like Laptop analytics on it because you can't have external systems come and access this data because it's very much tightly coupled
[04:38] and only accessible to as Postgress right uh you probably cannot do instant provisioning you can't do branching you can't do autoscaling right because this is all a statically provisioned machine so you don't get these nice properties
[04:50] that you want for for agentic workloads and postgquest is reasonably scalable and reasonably fast so it's not that it's particularly bad but there's room to do better in terms of both scalability and uh and performance and
[05:02] it's not highly available because it's just a single machine right now there are ways to turn some of these things screen with sort of without modifying Postgress a lot. Like you can set up replication and readable secondaries and
[05:15] do lots of clever things. But what you'll quickly find out is that you can turn some of these green, but at some point you start running into evolutionary limits of the architecture because you turn one green and then you turn another one red. So sort of you
[05:27] you're going to run into trade-offs and limits without pulling some sort of ninja move uh to to to really turn this whole list green. And as you might have guessed by now, the ninja move that I'm
[05:40] talking about is separation of compute and storage, right? Because the trick that we do is taking this sort of local hard drive here and we extract it out into a separate system. So we can scale
[05:52] the storage independently and then you practically get infinite storage from the from the view of the single Postgress that turns Postgress stateless. And when we design systems, right, we always love to make services
[06:05] stateless because they're easier to scale, easier to deal with. Uh and that means that you can now scale Postgress and compute independently as well up and down as the workload goes up and down, right? So it means you can have a small database with lots of traffic or a huge
[06:18] database with little traffic and you can scale these dimension uh dimensions independently, right? And putting this data then in the lake that means that you can now do analytics directly on the data with uh with this Raiden thing that we talked about in in the keynote.
[06:31] So, so sort of decoupling comput and storage is the trick, right? But that's sort of easier said than done. What does it really mean to decouple compute and storage for a Postgress database, right?
[06:43] And to understand that, we need to take a look at how Postgress executes transactions when you when you talk to it. So on top here we have Postgress. On the bottom we have the local hard drive uh that Postgress is running on. This is
[06:56] kind of the vanilla Postgress setup, right? And over on the left here is us. And we send a query to Postgress. We say run us run this query for us. Right? It's going to go into Postgress going to hit the query executor. Post is going to
[07:08] try to figure out what we're trying to do. It's probably going to start reading some data from tables and indexes to satisfy our query. And if we're just reading, that's kind of it. It'll send the results back to us and then we're done, right? However, if we are doing an
[07:21] update or a delete or an insert, we're modifying data, then it'll start modifying this data, right? But crucially, it'll also journal all the changes that we're making to the database into this thing called the write ahead log, the transaction log.
[07:34] And pretty much all databases do this to maintain consistency of your database because we need to have a journal of everything that happened in the database in in case something goes wrong. Say that you rolled back the transaction or
[07:47] the system crashes and we need to figure out what happened, right? This is how pretty much every database system out there satisfied this thing you might have called of which is called acid, right? The A in acid stands for atomicity which means that you either
[07:59] finish the transaction completely or you undo it completely right and the journal the transaction log is pretty much how all databases satisfy that property and then finally when you commit the transaction then Postgress will write
[08:11] that transaction down to disk to have a persistent record of the fact that transaction was committed and this is the D in in acid which stands for durability which means that you can't lose the transaction if it was committed right so there's a lot of machinery
[08:25] going on here uh to maintain the safety of your database and the consistency of your database. It's not just like scribbling a bunch of data, right? It's doing this very meticulously and that has some overhead. It's writing all the
[08:37] data twice for example, right? Both on the left side on the right side. Uh and and that means that this storage at the bottom uh is very sort of tightly coupled to Postgress because you can't just like look at this data and make sense of it because it's very much owned
[08:50] and and sort of modified by Postgress all the time. you can from an external system come and make sense of it without sort of involving Postgress in the process. So that's what we mean by tight coupling uh between Postgress and
[09:02] storage. So what we have done in lakebase is that we have effectively taken that hard drive that is local to Postgress and we have extracted it out and made it into a distributed system in itself and made it
[09:15] basically into a service that Postgress can use. Uh and that service has many machines in it, many nodes. So it is practically infinitely scalable to a single postgress because in fact it's
[09:27] multi-tenant. We run many Postgresses on top of this storage service to get economies of scale as well. And then we have uplevelled the interface because a hard drive is basically just read and write bits and right it's a very
[09:40] low-level interface. Here we have uplevelled that interface to say read units of data in Postgress that's called pages because Postgress stores pretty much all the data in your database in 8 kilobyte pages. So the storage system
[09:53] exposes an interface to read a page of data and then exposes another operation to write the journal to write the write ahead log down into storage. And notably you might see there is no write
[10:05] operation on the left here because in this architecture Postgress doesn't actually need to maintain the table data directly. All the modifications to the database is captured through the write ahead log because the write ahead log is basically a full description of
[10:17] everything that happened to the database. Right? So you can make out uh any the the history of the database at any point from this wall. Uh and the way we tie this into Postgress is to hook it in at the IO interface level because
[10:30] Postgress has extensions for everything. You can extend pretty much everything in Postgress. So we really didn't have to modify very much in Postgress to do this. We just hooked it in and replaced the place in Postgress where it is doing IO to the hard drive. And that means
[10:44] that pretty much everything in Postgress just works extensions and everything because they all go through that same interface to access the hard drive. So we could move the hard drive out of the machine and to a distributed service and everything just kept working. Uh and
[10:57] that's what we mean when we say fully compatible Postgress. We didn't break anything in doing this thing. So the way uh modifications are now done is that up in Postgress the query comes in right
[11:10] it'll need some data to execute the query. It'll read that out of storage and then it'll start modifying and then it'll send the modifications down into storage and those modifications go to uh a piece of machinery inside our storage tier that we call safekeepers. Uh and
[11:23] safekeepers they receive this uh this this log or journal of changes and safekeep. That's why it's called a safekeeper. And in reality this happens uses a paxus algorithm replicated across
[11:36] multiple availability zones to make sure it's super available. uh and then it'll persist that to a local SSD drive on the machine across three availability zones to get give you super low latency on commit because at the bottom here there
[11:49] is the object store but the object store of S3 Azure storage and such has much higher latency on operations. A typical S3 operation takes like 50 milliseconds to read or write and that is too high to
[12:01] put on the commit path for a transaction in operational stores. So we write it to a local SSD and then we asynchronously offload it to the object store after to give you super low commit latency.
[12:13] Now when the safekeepers have received the log, they'll send it over to the page servers and the page servers is the machinery that gives you low latency for reads in this architecture because just
[12:25] like you can't go to the object store directly on the right path, you you can't do it on the read path either if you want good performance. So these page servers they effectively cache all the data in the database on local SSDs to
[12:37] give you a read latency of 1 millisecond or better uh and give you good performance up in Postgress and at the same time they will then write all this data back into object store to make uh the object store and the lake the
[12:49] authoritative store for all this data so that other systems can come and read it as well. But what you can sort of think of the storage layer in lakebase as is an accelerator on top of the object store that gives you good performance for all TP workloads while still putting
[13:03] all the data in the lake such that you can come in from the side and do analytics with Raiden on it for example in in reality both the safekeepers and page servers are spread across availability zones so it's super
[13:16] available and if your database is very large it's in fact it is charted and spread across many page servers So you get good performance by using many machines resources to to accelerate the reads. Usually we split it in like
[13:29] chunks of 100 gigabytes or so. Uh so you get excellent read performance by effectively doing scale out. And this happens under the covers without you having to think about it. So that's sort of the trick to storage
[13:43] in uh in lakebase uh how we sort of separated storage out of Postgress and now made Postgress effectively stateless. There is no storage up in Postgress which means that we can scale it and manage it much more easily and
[13:56] the hard part is now now down in the storage system that that we have to worry about and and not Postgress and that is also how we do branching because you might have heard that we do branching uh and that allows agents to
[14:09] quickly do a copy and write clone of data do experimentation isolated without impact on production data without impacting production. Right? All of this magic is happening down in the storage layer and it's effectively metadata only
[14:22] operations. So when you create a branch, we just do some metadata operations inside storage and start tracking the modifications and data independently for that branch. So that we only store the little deltas that you have in the
[14:34] branch relative to the main database. And when then Postgress comes and reads, we just oh you need to read from this branch. We'll see if there was a modification. If not, we'll go back to the main branch. And Postgress in this architecture has actually has no idea
[14:48] that it's operating on a branch. All of that magic is in storage. It just like sees it as another volume of data essentially. And all the magic of branching happens down in storage with really no modifications to Postgress at
[15:00] all. So separating out storage allows us to do tricks like this as well. And of course as I hinted to a couple of times uh putting this data down in the
[15:12] lake and separating it out from Postgress now allows you to give access to other systems right in the classic way if you want to if you use lakehouse federation for example you you would you would enter through the front door through Postgress and query data but
[15:26] that wouldn't be very fast and it would probably also affect the operational workload uh and so that's not generally a very good idea for a scalable system but now you can take then Raiden and lakehouse real time uh and query the
[15:40] storage directly because separating out storage means that we now have a consistently uh sort of a a transactionally consistent view of the data outside Postgress and we don't need to go through Postgress to read this
[15:52] data we can read it directly instead and of course you might say well isn't this in row format right is that going to be very good for analytics but turns out we can then use our storage tiers to
[16:05] do smarts there as because we can in the storage tier as Postgress is writing data we can transcode this data to columner format uh and accelerate analytics queries on it without really
[16:17] Postgress even knowing that this is happening right it doesn't put additional load on Postgress it doesn't uh even know that it is happening but it can then analytics can come in from the side and read this data in columner
[16:29] format and do fast analytics on it so basically a lot of the tricks in this uh this uh separation of comput and storage is that we offload a lot of work out of Postgress. So Postgress has less things
[16:42] to do uh and can give you better performance uh by using its CPU cycles for useful transactional workload while we do a lot of the magic further down in the stack unbeknownst to Postgress
[16:54] really. So that's sort of the the the really the trick of uh of separation of compute and storage and that means that we have start started turning some things green here. We have now done a uh enabled us
[17:07] to do LTAP analytics by directly accessing the data. We have introduced branching which means that you can now experiment with agents without really modifying Postgress. And we have made it more scalable because the storage tier
[17:20] is basically infinitely large in terms of how much data you can store. You can store terabytes and terabytes, right? And we have made it faster because we have offloaded work from Postgress and just moved it into the storage tier. So
[17:32] now we've started started turning some boxes green here without really regressing anything else. Right? So that was a lot about storage. Um now let's start talking about how we do some
[17:45] of these other things that are more compute related. Uh instant provisioning, autoscaling and scale to zero. So how do we do that right? Well, the way we do it is that we run large
[17:58] compute clusters where we put many Postgresses on fairly large nodes. So, we have compute clusters that have many big machines. They're like 48 core CPU machines and larger. And then we pack
[18:11] many Postgress instances onto those machines. And as you can see, some of Postgresses are large, some are some are small, right? Because and some are actually changing size up and down uh live, right? as the system is
[18:23] autoscaling and your workload is driving less and and more work uh on on your postgress. So we do sort of pack the cluster full of all these postgresses while allowing some room for these to
[18:36] expand and shrink and allow some buffers for new ones to come in, right? Because the way we do instant provisioning is that we always ensures that there is a hole somewhere in the cluster for your Postgress to start if you need it. And
[18:50] then we use economies of scale from this large cluster to amvertise the cost of keeping those holes available across a lot of customers so that we effectively make sure that there's a hole available for your postcrist to start without
[19:02] charging you for that hole. Right? And we advertise the cost of that across a lot of customers and pass the benefits of that on to you. And uh and that allows us to essentially when you create a Postgress we can place it start it and
[19:14] give you access to it in less than a second most of the time sometimes hundreds of milliseconds. So that's how we do instant provisioning. Now let's build on that to how we do uh
[19:29] scale to zero because scale to zero is this feature that we have where you can say turn off my database when I'm not using it because there are sort of two things that you pay for in lakebase. You
[19:41] pay for compute which is a running postgrad and then you pay for storage and comparatively speaking storage is much cheaper than the compute. So to have some data sitting around not being used is not that expensive. To have a Postgress sitting there running doing
[19:53] nothing is relatively expensive. So what we offer you is uh a scale to zero option meaning you can say if I don't use my database for 5 minutes just turn it off. Uh and if I try to connect to it again turn it on very quickly. Uh and
[20:07] then we use the instant provisioning to do that in less than a second typically. So if you don't use your database, we can turn it off and then if you if you try to connect to it, we'll start it up again in less than a second while you wait. And all you see is like a little latency stall as you as you try to
[20:20] connect. And the way we do that is that we have put a proxy in front of our compute cluster. So when you connect to your database, you really connect to a proxy that actually natively speaks the
[20:33] Postgress protocol. So your client thinks that it is talking to a Postgress but it is really just talking to a proxy that is impersonating a Postgress and then it negotiates with your client to figure out which database is it that you're trying to talk to. Uh and then it
[20:47] goes to see is that database available right because in the steady state here if your database has been shut down there's actually no Postgress running for your database at all. Of course storage is there and highly available so all your data is readily available but
[21:00] there's no Postgress running. So, it'll receive your connection, uh, figure out what database you're trying to connect to. It'll authorize you, see that you're coming from an IP address that is or a private link that is that is correct for
[21:12] this database, that you have a valid username and password so that you can't like start somebody else's database and run up their bill. That will be not very nice, right? So, so we check that you are you before we before we proceed.
[21:24] Then proxy will talk to our control plane and say, "Hey, uh, somebody's trying to connect to database ABC. Uh, where is it?" Right? can you give me the IP address for the thing and then the control plane will say oh I don't know it's it's not running currently and then
[21:36] oh I should start it right and then it'll call down to the compute cluster and start postgress for that database uh and then it'll return the IP address of that Postgress that was started to the proxy and the proxy will forward the
[21:49] connection and this whole rigomearroll happens in less than a second while you're waiting so all you're seeing is like a latency stall on the connect and then it goes through and you go into the database and then the next time you connect. If you connect again like a
[22:02] couple seconds later, then it'll go straight through because now your database is running. So there's no latency uh delay at all to to to get in and at that point you can start running queries and and party as if the database was was always on, right? And then at
[22:16] some point you finish the workload. It's uh it's 5 p.m. and you go home or the agent has gone to sleep or or or whatever, right? Uh so the database goes idle. So our control plane for all the
[22:28] databases that have scale to zero enabled is continuously monitoring the workload on all of these Postgresses to see if they're idle. And as soon as it detects a Postgress that has been idle for the configured timeout because you can configure how long it should wait to
[22:41] shut it down. Default is like 5 minutes. But once it detects a Postgress that is idle, nobody's connected, nobody's writing queries, and it's been five minutes, it'll shut it down and stop billing you for it and free up the space in the cluster for somebody else to come
[22:53] grab. uh and at that point you again only pay for the storage that you store uh with with no compute bill. So that is how we do scale to zero and it builds on how we're able to do
[23:06] instant provisioning uh and we leverage that to effectively just spin up a database when you try to connect to it but but not before and that allows you to obviously just pay for the thing when you use it as opposed to pay for
[23:18] continuously running compute. Now we also do autoscaling right and the way this works we can sort of zoom into a single node in the cluster on this
[23:30] cluster on this node there's a few running postcresses and uh it is sort of allowing these postgresses to grow and shrink as the workload comes and goes right or goes up and down and specifically we have inside or next to
[23:44] each postgress we have a little thing we call the autoscaling agent which is continuously monitoring the resource consumption in postgress it's ask Postgress like hey do you have as much memory as you need hey do you need more CPU uh what is your working set do I
[23:57] need to increase the memory to get uh your caches to cover all the data accessed right so it's continuously checking with postgress hey do you need more resources right or or do you need less resources are you are you wasting resources right now and then if it finds
[24:09] that a change is needed then it calls down to what we call the autoscaling demon which is running on the host machine that sort of covers the entire machine and says hey I need more resources this postgress is about to run out of memory or something
[24:21] And then that autoscaling demon will change the size of that VM and effectively hot add memory or hot add CPU to the VM that is running Postgress and then Postgress will expand and sort of start consuming those resources and
[24:34] that's how we can within seconds scale up your Postgress as your workload is increasing uh so that you effectively only pay for the size of box that you're currently using and not a statically
[24:46] provisioned sort of provision for peak uh Postgress Okay. Now the observant of you might have seen that I previously said container and now I said VM, right? And this uh diagram
[24:58] says microVM. So these boxes that are around these Postgresses, they're actually virtual machines. They're not just Kubernetes containers or something like that. And there's a reason for that because Postgress has a very large
[25:11] surface area. There's lots of extensions, lots of thirdparty code. You can run custom code in there. It's it's kind of there is a nonzero probability that somebody finds some sort of security exploit in there and we don't
[25:24] want them to able to be able to break out of of this box and sort of sort of start accessing other Postgresses on the node or to sort of break the system right so we actually put a full virtual
[25:37] machine isolation boundary around each Postgress such that if somebody managed to find a security exploit in Postgress or any of the extensions they cannot go outside the box and that is how we ensure what's called hostile
[25:50] multi-tendency. We basically make sure that nobody from breaking a single thing can get out of the box and attack the system. So so in this uh you it's not enough to find an exploit in Postgress and the
[26:02] kernel and the Linux kernel. You also need to break a virtual machine boundary which is nearly impossible. So, and we call these microVMs because they're tiny little VMs that basically only run
[26:14] Postgress. So, they have very little overhead. It's not like a big clunky virtual machine that you'd normally think of as as virtual machines. So, that's uh how we do autoscaling.
[26:26] So with that we have covered how we do instant provisioning. How we on top of instant provisioning build scale to zero and then how on top of that build autoscaling so you can scale the
[26:39] workload and scale postgress and pay just for what you use and not for statically provisioned compute. So now it's starting to look pretty good. We have a lot of green check marks here. Now the final thing we should talk about
[26:51] is high availability, right? How do we make this thing highly available? Because well, if your workload takes off, then you'd want it to be highly available and mission critical, right? The first kind of thing to realize is
[27:05] that without doing anything, this actually gives you much higher availability than just running Postgress on a single node because the storage tier and the lake is already highly available, right? Obviously, something
[27:17] like S3 is super highly available, right? It's spread across availability zones. Our storage tier that we talked about has these page servers and safekeepers spread across multiple availability zones and in itself is super highly available. So already
[27:31] without doing anything your data is highly available and that is a very good starting point because it's now much easier to just have a stateless postgress on top that accesses your data that is already highly available. So, and that's what we effectively do. If
[27:44] you don't enable any high availability in your lakebased configuration because you can choose how high availability you want to pay for, right? If you just choose the basic one, you basically just get a Postgress running uh that is on
[27:56] top of highly available data and if that Postgress were to fail, then we can just start it somewhere else because it's stateless, right? So, we don't need to move your data over and stuff. We just start it somewhere else. Of course, it's not going to be uh it's going to be like
[28:09] a second or so because we have this instant provisioning, right? Uh but and it the caches are going to be cold. You might see a little bit of performance impact and stuff, but it is highly available. Uh and then we put a this
[28:21] proxy of ours in front, right? So if your Postgress blows up, then you just reconnect and the proxy will send you over to the new location. So you would just reconnect and then the system figures out the rest, right? You don't have to reason about, oh, where's the
[28:33] next thing and what's the IP address of that and stuff. So basically without doing very much at all you get quite a bit higher availability than if you just ran a single Postgress. Now of course some systems are super
[28:45] mission critical and you need the like enterprise grade availability right so that then we also enable uh you in the UI to say I want really high availability for this database right and then what we do under the covers is we
[28:58] start multiple postgresses for your database such that if one of them fails there is one already there ready to take over with warm caches and stuff like that so that you have the best possible availability.
[29:11] This is this is how uh how you get sort of the the 49s plus u availability. So that enables you to get super high availability but you can also do other
[29:23] tricks with it. You can do read scaling because if you provision multiple Postgress nodes uh you don't just need to use them as hot standbys. You can use them to scale your reads because this storage tier of ours basically makes the
[29:36] data transactionally consistent available to anybody that asks. Right? So what you can do is you can provision more Postgress instances as readers and they don't need a full copy of the data. They can just read the data that they
[29:48] need when they need it from the storage layer. Uh and then start executing readonly query the read only queries. They can only be one writer because Postgress is a single writer system but a lot of the time your workload is writing a little bit and reading a lot
[30:01] right and and you can then scale out the reads and get more throughput by just adding more Postgress instances. And then we give you a feature in the proxy such that we actually give you two DNS names to connect to. You connect to one
[30:14] DNS name uh if you want to write and then you connect to another DNS name that basically just has a - RO readonly at the end. Uh that will load balance you across these readonly replicas uh so that you get good throughput and low
[30:27] balance throughput uh without really doing much other than than connecting to a different DNS name. um and and that enables you to to scale reads as well uh and get and and multiply your throughput.
[30:39] So that is high availability and how you can scale reads. And to close it off uh I wanted to show you a demo of how we leverage this architecture to do something cool. uh
[30:53] and this is going to be uh sort of what we do under the covers that you don't have to really think that much about but sort of how we leverage this architecture to do things for you that sort of improve your your experience. So
[31:06] we talked about how we can spin up Postgress databases quickly in this architecture right or Postgresses quickly. We leverage that to do something we call zero downtime patching because from time to time you need to update Postgress right because uh
[31:19] Postgress might release a new minor release or there might be a security patch or something for Postgress and that means that Postgress needs to restart to pick up the new bits right uh and we then use this ability to quickly
[31:31] spin up Postgresses to improve the experience. Of course, you configure a maintenance windows. You can choose exactly when this happens if you want would like it to happen on a Saturday night or something, but it's good to make it as smooth as possible even if
[31:44] you can control when it happens, right? So uh we under the covers when we need to patch Postgress instead of just like restarting it in place and sort of interrupting the workload a lot we
[31:56] actually spin up a new one in the background that you don't see a hidden Postgress in the background and then we warm that up warm up the caches and get it all good and ready to go and then once it's ready we quickly switch to it
[32:08] so that your workload sees way less impact. Uh so that sort of enables you to get the benefit upgrade benefit of having a hot standby without paying for that hot standby because we kind of
[32:20] throw it in for free when we need to do the upgrade. Uh we actually wrote a blog post on it. So if you scan this QR code uh you you'll get to the blog post. But what I'll demo now is basically what we go into into great detail on in that
[32:32] post in that blog post. So this is kind of how it looks, right? You you you start with uh a running Postgress. We call it compute. Here at the bottom there's storage. It has all the data, right? Then you have your running uh
[32:45] Postgress that has a bunch of this data cached. It's nice and good, nice and warm, running with good throughput, but it needs to restart, right? So then what we do is we start another Postgress side by side that you don't see in the
[32:58] background. We warm it up and as soon as it is good and warm, we switch the workload over to it. And the idea with this red line at the top is that your workload sees let maybe a little blip, but it's a very small blip and not a very big blip, right? Because we quickly
[33:11] switched you over to a postcourse that was ready. So I have a a demo video here where I'm doing basically this. So I'm running pgbench which is like a standard benchmarking tool on a Postgress and
[33:24] it's cruising at like some 30,000 transactions a second. Uh this is just a readonly workload. Uh, and now I'm going to restart this Postgress from the admin console and simulate uh, essentially
[33:38] doing a upgrade of that Postgress because I'm basically restarting it manually. So there the the workload restarted and now we're down to like 13,000 transactions a second and slowly crawling our way up. Like if this was
[33:50] your production workload, that would not be so good because you would see impact and and throughput fall, right? And it's still working on getting its way back up to 30,000. And if we sort of see what happened in the background in the console, we
[34:03] actually give you a system log of all the things that happened. We basically just suspended one compute and started another one. It's basically just like a kill and restart. Right? So now instead I have changed configuration on the back end to enable this pre-warming uh
[34:17] behavior and I'm going to restart this database again now with pre-warming enabled. And now we can actually see in the system log that there's a a lot more stuff going on. Now there is a pre-warming in progress, right? And then
[34:30] there's a a promote and a switchover that is pending. And now it's actually pre-warming in the background while my workload is still running unaffected in the foreground. And boom, there the pre-warming was done. It switched and
[34:42] now we're immediately up to 25,000 transactions and there were already over 30,000. So we basically pre-warmed it and gave you a lot less impact on the running workload as compared to just
[34:55] like a hard restart and there we're like now cruising well above 30,000. Uh so this is sort of how we used the ability to quickly provision Postgress uh and gave you a much better upgrade
[35:07] experience without really charging you for that. And this is basically what happened right uh a restart with pre-warm sees a little dip. A restart without pre-worms is a big dip in the workload. That's effectively just a
[35:21] graphical representation of what what I just showed. So with that we have basically built us towards all green check marks on the system. Uh we can do it's still Postgress. It's fully compatible
[35:34] Postgress. It's you can basically do all the Postgress things you know right? Most of the extensions just work. You can do LTAP analytics. You can do instant provisioning. You can branch. you can autoscale and it's more
[35:47] scalable, it's faster and it's highly available. All right, so that's how lakebase works under the covers. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.