Enterprise Resiliency Strategies with Lakebase

Summary

This video covers the full range of enterprise-grade resiliency capabilities in Lakebase, including customer-managed encryption keys, high availability through synchronous replication without write amplification, instant point-in-time recovery, and strategies for cross-region and cross-cloud disaster recovery.
The resiliency onion framework organizes these capabilities in layers—from infrastructure integrity and data integrity at the core through high availability to disaster recovery—and the video explains how to match each layer's strategies to specific RPO and RTO requirements.
A live demo shows cross-region DR failover and recovery branch workflows, and the video addresses cross-cloud disaster recovery scenarios such as Azure-to-AWS failover for organizations with data sovereignty requirements.

Enterprise Resiliency Strategies with Lakebase

Building enterprise applications on Lakebase means meeting the security, compliance, and availability requirements your organization demands. this video covers the full range of enterprise-grade resiliency capabilities in Lakebase: customer-managed encryption keys, high availability, point-in-time recovery, and strategies for cross-region and cross-cloud disaster recovery. You'll learn how to architect for resilience from the start, what the tradeoffs look like between different recovery approaches, and how these capabilities integrate with Databricks' broader compliance and network security controls. Whether your requirements are driven by regulatory mandates, business continuity planning, or both, you'll leave with a clear framework for running Lakebase in production with the governance, auditability, and data protection your enterprise requires.
Talk By: Jonathan Katz, Sr. Staff Product Manager, Databricks ;

Chapters

FAQs

What is Lakebase and why does it need enterprise resiliency features?

Lakebase is Databricks' managed PostgreSQL offering that runs directly on the lakehouse, designed to support operational and transactional workloads alongside analytics. Like any production database, it requires resiliency capabilities to ensure data availability and integrity when disruptions occur. This video covers the full spectrum of resiliency features available in Lakebase, from infrastructure-level protections through to cross-cloud disaster recovery.

What is the resiliency onion framework described in this video?

The resiliency onion is a layered framework that organizes Lakebase resiliency capabilities from foundational infrastructure protections at the core through data integrity mechanisms, high availability, and finally disaster recovery at the outer layers. Each layer addresses a different class of failure, and the appropriate combination depends on the RPO and RTO requirements of the specific workload. This video explains how to match resiliency strategies to business risk tolerance and compliance requirements.

How does Lakebase achieve high availability without write amplification?

Lakebase implements high availability through synchronous replication using a write-ahead log and safekeeper quorum architecture, which ensures data consistency without requiring writes to be acknowledged by all nodes before completing a transaction. This approach avoids the write amplification penalty common in traditional HA configurations. The result is a high-availability setup that maintains low latency for write-heavy operational workloads.

What cross-region and cross-cloud disaster recovery options does Lakebase support?

Lakebase supports disaster recovery configurations that span cloud regions, allowing organizations to fail over to a secondary region in the event of a regional outage. Cross-cloud DR—for example, from Azure to AWS—is also supported for organizations with data sovereignty requirements or multi-cloud strategies. This video includes a live demo of cross-region DR failover and shows how recovery branches can be used to validate data before fully switching over production traffic.

Full transcript

[00:08] First and foremost, even before you know what talk this is, is anyone here a lawyer? All right, so forward-looking statements yada yada yada. All right, welcome to enterprise resiliency strategies for Lakebase. Uh, my name is Jonathan Katz. Uh, I help lead product management for
[00:21] Lakebase. I'm also on the core team of the open source Postgress project. And this is arguably, you know, if you're running lakebased, this is arguably the most important talk you're going to go to today. We're not being recorded, so I feel like I can say that. But really one
[00:35] of the main, like there's a lot of fun things about databases. I've been around databases pretty much my, you know, entire professional career. I've used, you know, I started with Postgress as a user, you know, running and building applications, eventually became an operator more so by accident and, you
[00:49] know, and ultimately, you know, becoming a vendor of database solutions. But when I started making that transition from app developer playing with all the fun features and getting into you know running you know real applications
[01:01] managing applications this became like my reason to etcher like my reason for like you know wanting to work with Postgress and making sure that Postgress could support you know production level mission critical workloads because it
[01:14] really comes down to this at the end of the day is that every database needs resiliency. We need to make sure that our operational data is available and works. But you don't always need the
[01:26] same level of resiliency for your workloads. And that's the plan today is to step through the different uh resiliency strategies available, you know, today for lakebase and when they apply to your workload. Here's where
[01:39] we're going to start. First, some common definitions. So given like you know a timeline of all changes coming into a database we need to understand the first concept RPO or a recovery point
[01:51] objective. The reason why this is important is that this is your maximum allowable window of having inconsistent data changes in your database if there's an event such as a disruption or a failover or a power
[02:03] loss. And the keyword is maximum. I'm going to use the word data inconsistency. A lot of people view it as data loss. Data loss is the worst case but we're going to see how we can avoid data loss. Then there's RTO which
[02:17] is your maximum tolerable uh not having your data available or downtime. The key again keyword is maximum. A lot of times you see people refer to RTO as oh this is you know this is my target. The
[02:30] reality is usually your RPO and RTO windows tend to be a lot smaller even even in normal circumstances. So these are the maximum bounds by which you should define all your workload and this matters as we look through the different
[02:42] strategies available. Now to build up resiliency we kind of need to have an onion of strategies and at the core of the onions your infrastructure where you're running your workload and really it's the fundamental piece like where
[02:55] and how you run your workload is going to determine how you you apply your your resiliency strategy. Sorry I'm having uh I'm having interference while trying to click. There we go. Then the next is what your database engine provides you know
[03:07] whether it's lakebased or something else and this is like the data integrity layer like how you write your data and in some ways how you read your data impact you know your data's availability then we get to high availability um this
[03:21] was like a hot topic 10 years ago like in some ways high availability is a commodity but like when do you really need high availability making sure that you always have you know write and read access to your data and finally disaster recovery which is going to you know be
[03:33] the outermost layer of the onion. You know, disa the joke is like disaster recovery like really encompasses all those things like that is the full topic. But we tend to refer to disaster recovery when we want to go outside our
[03:45] normal regional deployment. So um if you find me boring um if you don't want to hear the full talk, this is your cheat sheet. Like I'm putting it up front so you can take a photo and leave. But yeah, I'm fair. I'm a New
[03:59] Yorker. I'm fair. Now, the more resiliency you add, the more it's going to cost. That's just the reality. That's the laws of physics. We can do things to like minimize the cost. And like that's what we're going to do as we step
[04:11] through this. But this is why, you know, this goes back to that first statement. Cost doesn't necessarily mean you don't want to pay that cost. Cost is about risk management. For like your most critical production workloads, your your
[04:24] mission critical workloads, like you're going to spend a lot of money to make sure those stay up. your data, you know, you you can recover from things like data corruption or, you know, full regional outages, but for your development workloads, like none of that
[04:36] might matter. Like, you know, you're just deploying on infrastructure and off to the races. For production, you're a good start in the middle. Like you you're going to need high availability, but we're going to see how we build that up today, you know, with Lake Base, which leads us into the first part. I
[04:49] should have first asked, oh, this is the DR talk. There you go. All the lights are out. By the way, we actually joked in the keynote today like could we plan like a power outage to uh
[05:02] but so first off what's bas is everyone familiar with it because I'm still going to define it anyway. Um, lake base is our managed Postgress offering. You know, really designed from the ground up where we take like the core Postgress engine and all the goodness that's built
[05:15] been built over the past 40 years, but bring it to the lake and have like and basically be able to leverage like fully elastic uh uh storage and compute and we often think about this in terms of scale and you know all these wonderful things
[05:28] we can do with agentic workloads coming to Lakebase was actually a core part of our resiliency strategy. So we get to our first bit infrastructure resiliency. You know why does this lakebased architecture well what does lakebased
[05:41] architecture inherit from our infrastructure. So we keep talking about lakebased separation of storage and comput separation of storage compute but really what that means is we're using elastic compute and blob storage. And we
[05:56] tend to think of these things as elastic. You only use as much as you need. They scale up infinitely scale down infinitely. but they have some magical resiliency properties. Let's start with storage. So
[06:08] most blob storage systems like have a different durability model than traditional block level storage and they'll advertise 11 9 of durability which roughly translates to like you know one out of every like trillion
[06:20] operations something bad may happen but like that's pretty good and like it's often better than you know just individual discs in terms of that. But again you get that from just using blob storage. It's also very available like
[06:33] these blob storage systems, you know, can accept reads and writes across all of your availability zones. So, think of an availability zone in the cloud as like its own data center. Like there's physical separation between them usually and as soon as you write to blob
[06:46] storage, it becomes available. And then finally, scale. And normally you think about this scale as like okay, I can store unlimited data here. But it's a resiliency property because if
[06:58] you keep adding data to your database, you don't need to think about like how much storage I need to provision like it just keeps being added in. And that's important that avoids one kind of outage. Then we have compute.
[07:11] Compute again you also have scale. You need more compute, you go grab more compute. I mean if you if you try to grab GPUs like you know there are certainly some capacity challenges there but usually standard CPUs are available
[07:23] so you can keep scaling your resources and avoid avail you know avoidability issues there but also using elastic compute makes it easier to be able to have you know zonal redundant topologies. So like basically how you're able to deploy your computes in ways
[07:36] where your data is always accessible. And finally, life cycle management. Instead of having to take disruptions to like buy new hardware, you know, because your old, you know, hardware caught on fire or you want the latest processors,
[07:49] you swap it out. You get new compute. And again, from a resiliency standpoint, that means you're staying up to date with the latest hardware. You know, often, you know, the latest software packages, etc., based upon how that image is built. Why am I telling you
[08:02] this? Because you don't need to worry about this. You don't need to manage, you know, the servers and the images and the operating system layers. Like Latebase takes care of that. We have people who just stare at this full time and I'm glad I get to work with them
[08:14] because I hated dealing with this. But you get this for free as part of paying for lakebase.
[08:26] integrity, which is what the engine provides. Now lakebase gets a lot from Postgress like Postgress again very battle tested. It's been open source for 30 years. In fact, we just had like a 30th anniversary celebration for the
[08:38] open source Postgress project last month. But in order to be able to use like this infrastructure correctly, particularly this, you know, cloud/lake infrastructure, you know, Lakebase has to do its part to
[08:51] be able to leverage those features, but you also have to do your part, too. And here's the shared responsibility with your database engine. your you know basically it has to protect against infrastructure glitches you know bugs
[09:05] issues etc data compromises and like self-inflicted wounds for example I once deleted a production application that I was responsible for like at 3 in the morning by not thinking needed recovery
[09:17] from that so lakebase is going to provide like the engine level things to be able to handle with that um be able to do transactionally consistent rights make sure they're durable and provide the mechanisms to get out of issues such
[09:29] is Jonathan deleting all the payment transactions. You know, we'll be able to do a point in time recovery and get them back. But you're going to have to be able to figure out like what your backup schedule is. You know what, you know, you want to be able to test your restores, make sure you're able to get
[09:41] all the information back. And, you know, invest in your standard operating procedures for this. When you drop that table at 3 in the morning, you don't want to think about having to restore it. You want to be able to read a script and get it done or write a script so
[09:53] your agent can get it done. So we're going to do a little bit of a deeper dive here and we're going to see like you know what lakebase has done to make sure that your data is safe you know from a foundational layer.
[10:06] So the way postgress supports durability is in this overly simplified diagram like let's say like a write transaction goes in a right goes to Postgress. Postgress first like tries you know as the transaction come in comes in it says
[10:18] like write this to my write ahead log. A write ahead log is basically all the changes that occur in your database system. So kind of a fun fact, Postgress is an appendon system. You you might not think about that because you do updates
[10:30] and deletes, but behind the scenes is actually just like appending everything in and that's effectively what the write ahead log is. But that's still not a durable commit. Like basically, you know, all these changes are coming in, but they're not officially, you know,
[10:42] flush to disk until this commit comes in. So that's part one. But then that data is still not in its final format. Postgress has to do this thing called a checkpoint to basically put it into a way that's like very easy for it to
[10:55] read. This clicker is driving me crazy. Um, so here's the thing about this. So first there's like a bunch of steps to be able to get this done, but you're writing to
[11:08] a disk that's only a single a in traditional Postgress and that could be problematic, right? What if that a is unavailable? Well, you can't get your data. So traditionally, I'm going use the word traditionally because this is
[11:21] the older model. It's going to, you know, we're going to update it. You would basically want to do a double write where you do those that series of writes into that first a and then copy them into the second a. But suddenly like we're doing a lot of writes in
[11:33] order to make this right durable. We're spending a lot of time and by the way that transaction doesn't commit until um all those rights are completed in the second a. So you're basically having your performance by going through that.
[11:46] And again like this is the most important bit. I said postgress is an appendon system secretly. So basically we have to do like this double write of our write ahead log to these two different a but that's a key point there
[11:58] and like this is what we did in lakebased to be able to make sure your rights are safe and durable. So lakebase has this concept of a writer you know it's just your postgress instance you know living in the compute layer and then it has its you know this this you
[12:10] know customized storage layer. The first part of this is these things called safekeepers. So safekeepers are these redundant you know pieces you know these redundant processes where all your write ahead log transactions go to and then it
[12:24] has the page server which is similar to it it which the page server serves like the postgress compatible data for being able to read everything effectively it's like your your cache. Now the key bit is that we make it zonally redundant which
[12:38] means that when you do a write every right that comes in to lake base is going to go to multiple zones and it's going to go to each of these you know safekeepers and basically you know we consider I'll go back a second we consider uh everything written once it's
[12:51] written to two of the three safekeepers and finally you probably heard that lakebase writes everything to the lake so eventually what happens is all these transactions get flushed so here's how the flow works a right goes to lakebase
[13:04] writer lakebased attempts to write it to two of the three uh safekeepers and as soon as an act and it commits we're good but notice like that's half the work that we were doing in the traditional Postgress model. So we're getting that
[13:16] multiszonal redundancy but we're doing half the work. So think about we'll hold on to that thought later because basically once we get to high availability we have a way to do a much more efficient write get the performance
[13:28] that we need and still be highly available. Now we ultimately you know as we read the data we want to be able to read it efficiently like reading like every single change that came out is not efficient. So the page server accesses
[13:40] that cache and reassembles all the changes into something that resembles the standard Postgress page which is the unit of data that you write and eventually what happens is that you know similar to that checkpoint that we saw in the Postgress process we do a
[13:52] checkpoint and flush all that information to storage and it's in object storage. Remember, object storage is zonally redundant. Like your data is available all throughout that region that you're in. Pretty cool, right? So,
[14:04] we basically got like more storage redundancy with less work with the lakebased architecture. And that's really cool. Now, what happens if um we have this slide? What happens if say a safe goes
[14:18] out? Well, it's okay because remember if we write to two or three, we're good. And then we're able to ultimately bring a safekeeper back online to make sure that you know we can continue having you
[14:30] know a nice quorum of being able to write everything too. And when a page server goes out well okay we you know we can fail over into a different re into a different zone and we're able to still have that discache available. So again,
[14:44] why am I telling you all this? Because you don't need to worry about this. Like we're taking care of this for you. But as you build up your resiliency strategy, you need to understand like, okay, like all my rights are zonally redundant and I'm able to always, you
[14:57] know, always have access to my storage. Now this covers the right aspect of it, but we also need backups. And Lakebase has a unique way of doing backups. So
[15:09] imagine like we're writing data in like this you know let's say this resembles like a safekeeper this is actually you know further view of the lakebased data when you take a backup with lakebase uh we effectively do this thing called a
[15:21] snapshot because we know all the changes that have come in every time we do a backup you know we basically insert something that says this is a snapshot keep this data immutable make sure we can always restore from this and you
[15:33] know we you know behind the scenes we keep allowing uh data to be written while maintaining that copy of your data such that you know it's immutable it can't be modified and the cool thing is if you want to restore from a snapshot
[15:46] it's instant because with you know with with lakebased you know we have this concept of branching where you can have like this isolated view of your data and we can spin it up very quickly so you can basically uh start making new
[15:59] changes to your data without impacting the previous backup and again we have like the full timeline of everything that happened so like this is very quick and instant And point in time recovery works the same way. Who's ever had to
[16:11] perform a point in time recovery in production? Yeah. How much how many of you enjoyed it? Because like typically one. Yeah. You're on fire. You enjoyed it. Wow. I we got talk. So my my my point in time
[16:25] uh war story again it was something you know I was at fault. I had a logical bug in the software. It had been going on for months. We found it. Unfortunately we had like you know a huge retention of our wall archive logs because of another logical bug. bug I had but I was able to
[16:38] get out of it by doing a point in time recovery but it took days days with lakebased the point in time recovery is instant because we can just literally point back in time where that change
[16:50] occurred you get this copy on right you attach your compute to it and you can continue to go through uh go through it so what are so ultimately what are these advantages for backups with lakebase
[17:03] well as I'm emphasizing the point is that like there like restores are instantaneous like this saves you a lot of pain and trouble especially for those who have you know gone through that process but also like if you're trying to do things like you know restore to
[17:16] like a new instance or restore you know I want to be able to like test some changes before deploying them to production um you were able to do this very quickly. The nice thing too is that if you use the scheduled backup mode, which you should always use scheduled
[17:29] backups for anything in production, um they're all incremental, so you're not having to do like a full snapshot every single time where you can incrementally build on them. And ultimately, point in time recovery becomes safer because when you do that, you get that isolated view
[17:42] and you can safely test changes before applying them back uh to to production. Just a fun fact, um, if you're using agentic development tools, they love snapshots. They love point in time recovery. And it's great because you
[17:54] think about it, an agent goes in, they make a destructive schema change, well, it turns out like it was wrong. Well, they just go back to the snapshot and try again. You don't have to like rehydrate your entire database uh to be able to get that done.
[18:09] So, I talked about shared responsibility. So, what do you need to do for your lakebased data integrity plan? Always take backups, please. Um when uh I used to help run or maybe I
[18:21] kind of still help run a Postgress conference in New York and I'd always put like a talk on backups there and like 10 people would go to it and it and the thing is like it's such an important topic and you really need to understand how to manage them because backups get
[18:33] you out of trouble. It's one of those things that if you don't have it and you need it like you heavily regret it. We make it as simple as possible. just like flip a switch, get a daily backup at, you know, at a minimum. Um, you can
[18:45] decide what schedule makes the most sense for you. But I'm just gonna like spend like most of my time on that point because you should take backups. Um, the rest I hope you've read. Um, we try to make it as simple as possible. Just also like a PSA, you know, public service
[18:58] announcement. P who's used PG dump before? Yes, it's Postgress utility. It lets you get data out of it. It is not a backup utility. Like do not use PG dump for backups. You can use it for transferring
[19:10] data. Use it you know you can use it to keep a copy of your schema. It is not for backing up data PSA. And finally in development um just so you know if you know if you turn agents
[19:23] loose on lakebase um they will be using the snapshotting feature. They love it. Um and also in that case you know you typically in production you want to keep a data retention window that's long enough for you to react to changes. So,
[19:35] for example, if no one's watching your database over the weekend, you probably want to have at least a 72-hour window in case like something really bad goes wrong in development. Keep that window small. Like, you don't want to have to pay for like retaining all of those changes, you know, throughout the
[19:47] database. Okay, we good with good with data integrity. Do we have some strategies now? All right, high availability, the feature that everybody wants in production.
[19:59] Here's how high availability used to work in Postgress. So before postress supported replication um basically you would keep a copy of your data in you know a different a and if like your
[20:11] compute went down or you know something of that nature you would fail over and you read from that and you'd reverse the you know reverse the storage it actually is a little bit harder than this but believe it or not there are still systems that run out here like this
[20:24] because it's you know durable and well proven now postress added streaming replication and to do a high availability by the way it needs to be RPO equals Z because if you do an automatic failover, you cannot lose
[20:36] data. So RPO must be zero. The challenge with RPO0 is that you need to do synchronous replication. And when you do that, if your secondary A becomes unavailable, your primary A
[20:50] becomes unavailable because you can't accept writes to it because you can't finish writing it the second time to the other A. That's problematic, right? You don't want your secondary to take down your primary. Now, Postgress evolved to
[21:04] be able to support quorum commits. This actually came out in Postgress 10, which is almost a decade ago. And the nice thing is that it allows you to do that quorum like that quorum write that we showed uh earlier with uh you know,
[21:16] lakebased storage. Now, there's a couple problems here. Um and by the way, I say this as someone who's like built and deployed systems like this and even offered them you know in uh vendored products. Look how many writes you have
[21:29] to do to one be able to get the core in but also look how many how much infrastructure you need to run and again if resiliency matters do this like you should 100% do this but it's expensive see we keep going up the cost chain
[21:43] so with lakebased no matter what you know if you're running you know single a multi-AZ your storage is always highly available and again like we've made that right path very efficient so that uh
[21:55] you're not you don't have to do like four writes, six rights etc to get all your data done. But we also offer compute HA and again RPO equals zero. All your transactions are available and
[22:08] we you know we and we do it very efficiently where you only need to have like one other compute available to be able to get compute HA such that you know your your writer in a1 goes down your reader in a2 becomes available.
[22:21] If you want additional resiliency, you can add, you know, another reader. Again, depends on the nature of your workload. If you're skewing towards mission critical, you probably want to do that. You don't have to, but it gives you another layer of resiliency.
[22:34] And um Oh, I thought I had another slide there. My bad. See, of course, because you can see my slides, you can see what's coming. Um, so here's the thing, like this is actually really nice because no matter what, when you get your rights, they're strongly
[22:46] consistent. RPO equals zero. Pick the buzzword to describe it. But you also have a single endpoint. So you have a writer, you know, a readwrite endpoint. The readwrite endpoint is always going to point to, you know, the the active writer. So you shouldn't have to do
[22:59] anything on your application side except you should always have retry logic in your application in case like a right, you know, uh doesn't go through because I always did that in my applications. Wink wink. I did not. Um but I wish I
[23:13] did. I want to go back to like Jonathan 10 years ago and tell him to do that. But the other big thing is that you don't have a query performance trade-off between HA and nonHA. I've definitely worked with the customers where there's like they said like look our query
[23:25] performance it's like it matters so much like we can't take that hit with like you know the right amplification traditional Postgress with Lakebase you don't get that there's no right amplification the rights occur up front and we do everything to make the rights as fast as possible so you have no
[23:38] excuse for a production workload to not have HA and of course the readers can help you know scale out workloads further. So like really you have a production workload, you care about it, enable HA. So we get to finally the final layer of
[23:51] the onion, disaster recovery, which again is everything we've heard to date. But most people when they refer to disaster recovery, you have like this well architected system and the whole
[24:03] region goes out. Um at a previous employer I was at, we'd say, well regions never fully go out. They can get impaired, but like we've seen like regions can fully go out. um for all sorts of reasons.
[24:16] So the natural thing to do is okay well let's just like make the data available in another region you know and make a full copy of it. And the thing is that it's actually harder than it looks like
[24:28] even if you've like stitched together like cross region systems it can go very very wrong if you're not careful. Now the nice thing is that um you know the failure mode we're protecting against is
[24:41] that region A goes out you have to reroute you know traffic all the way to region B. But there in lies the challenge because you have to make sure that your applications are able to receive traffic in the additional region and everything consuming those
[24:53] applications can and you effectively have a turtles all the way down problem that even if your database is resilient to a regional outage you still have to make sure everything else works. So let's start building up our strategy
[25:06] here. Like your database is just like one part of the DR plan. You have to make sure everything that consumes it can work with it. The other thing too is that um
[25:18] typically when you do cross regional DR like you're not doing synchronous commits. you're not doing, you know, what we showed before because if we had to wait for, you know, the the other regions to to acknowledge those commits,
[25:30] that's going to be like very low latency or sorry, very high latency and you're going to kill the performance of your application. But the challenge is that in most most I don't want to say all, but like in most
[25:42] cross region DR systems, there's this notion of like failover and fail back. But remember, you might not have necessarily replicated all your transactions over from region A to region B before the failover. That's why RPO is greater than zero. And this leads
[25:55] to like a challenge of silent data loss. And many systems you don't even realize that that happened, including in managed systems. And it's actually very hard to get those transactions back. So we're going to show like we're ultimately
[26:07] going to show how lakebase solves that. But a key part of your cross region DR strategy is like who owns the decision to make that failover because at the best case you're going to have data inconsistency. Like some track actions won't be there and someone's going to be
[26:20] responsible for repairing those. So think very carefully about that when you design this. And then like do you truly need like mission critical DR, RTO close to zero, RPO close to zero or is it okay just to keep like a redundant copy of
[26:32] your data in the other region like we call this like pilot light DR or checkbox DR and most production applications it tends to be checkbox DR certainly there's very good reasons to be able to you know run additional
[26:44] compute in in other regions you know it's great for data serving and it's great for those like truly mission critical workloads but you have to weigh the cost versus the risk like what failure are you are you protecting against And again, would you actually do
[26:56] a failover if you have a regional outage? Like what's the threshold where you're willing to take like RPO or you know data inconsistency?
[27:09] transactions consistent or cost? Like how much you know how much is this workload uh you know worth to you? And again I don't necessarily know the answer. You do. But the idea is to be able to give you that strategy.
[27:21] So here's what people typically do for cross region DR. So the most well the cheapest I don't want to say it's the most common but certainly the cheapest is that you take a snapshot you export to another region
[27:33] you do it like once a day. Hey you have car region DR there. It's like it's very cheap. It's very effective. It has very high RTO though. Um especially if you're not using lakebased but you know if you have like a multi- terabyte database
[27:45] it's going to take time to rehydrate that and get your workload you know up to speed. This is kind of like you should you might want to do this anyway. Um this becomes your backup of last resort if there's some kind of compromise to your system. Um you keep your backups you know separate and safe.
[27:58] Uh but this is not like the core part of a cross region DR strategy. The next bit is having managed periodic uh disaster recovery. So you know occasionally replicating transactions or data uh to another region. The nice
[28:11] thing is that when you have this you have like a cost-effective controlled failover which is what we're going to see in a little bit with lakebase. It's not for your mission critical use cases like this is for your production use cases and again it keeps your data you
[28:23] know safe in another region. It protects against you know several failure modes regional outage perhaps you know data compromise etc. Now what I've seen and again it might be viable bas based upon what you're doing
[28:35] is using like kind of some kind of self-service uh replication across regions or it could be logical replication across regions. And what's cool is that you could actually end up streaming data into sources. is they
[28:47] could be read only or they could be writable especially with logical replication. This is actually good for data availability where you want to be able to do read serving from other regions. Um but you really shouldn't use this for DR because now you're managing
[29:01] the DR. Um as we mentioned there's a lot that goes into like just doing a failover and making sure that you don't make your data inconsistency problem worse. But certainly like you can use like logical replication to stream data
[29:13] into other regions and server like that that is a very good use case. Anyway, this be got like the managed streaming DR solutions which are the ones that get you to like low RPO and low RTO windows and again great for mission critical workloads. And then
[29:26] finally my least favorite active active. Um people think like oh I make rights available everywhere and you know I have rpo zero or really it's more like rto0
[29:39] rpo close to zero. These are very hard to manage because just naturally accepting having multi writer means that you have data inconsistency from the get-go. Um I was responsible for active
[29:51] active products before and I usually would try to convince people not to use them. Now there's workloads that require them. you're running something like that's like so critical that you always need to be available like use it. Um but
[30:04] use this very carefully and very sparingly and as you can imagine you know the cost increases significantly and the complexity increases you know as you go down the stack. So you may have heard today lakebase has
[30:17] uh has uh disaster recovery available and uh have a little bit of a demo for how it works. So obviously I'm doing this live right. Uh so here we are. We're in lakebase today and we are
[30:29] setting up uh it's called cross workspace replication but workspaces exist in different regions. So one region on one side, one region in the other. So you pick the region that you want and the region selection you can
[30:42] actually gate as a workspace administrator. So you can't you don't have people uh replicating data from like a production region to a nonpr region. we see that uh the region set up on the other side or the project set up and it is the you know it is a copy of
[30:56] the data from uh region A. So let's inspect the data in region A like this was a real uh lakebased project. I I can't say is because obviously I'm not demoing it live but um and we can
[31:08] connect to it. We can write data to it. Uh we can create new branches from it. So here's an example of uh one authenticating into lakebase but also inserting data. So as you can imagine
[31:21] we're ultimately going to do a failover. Um this is us inserting data before the failover. And we can see that like every lakebased feature that you expect works. You can create a branch. And keep in mind like a great feature about
[31:33] lakebased is you branch instantly. Uh so you have like a fully you know a fully isolated view of a different set of your data available. and the branch gets uh replicated over into the other region.
[31:45] We're copying everything over. So the next thing we're going to do is we are going to go into the other region and we are going to do a failover. So
[31:58] from the dashboard at least uh in our private preview uh you basically say like hey click this I acknowledge a failover and we fail over. So we can see that the the second region becomes our
[32:12] primary and our first region um in this case you know the region was you know healthy but let's say like it was unhealthy we refresh it it comes back online we can see that now it's in readon mode and there's going to be some
[32:24] stuff we're going to come back to um there's some interesting artifacts there that u may make our life much easier for DR so we're back into like our new you know our new primary region let's connect
[32:36] like let's see what data is available ailable there and we're able to connect and we you know we leverage the data bricks identity management system so you have to copy your ooth token in um and
[32:49] we're in our database so like let's see what data made it in there so it looks like we have some data but I don't think that was like all the data that we had added into that region so some some data may be stuck in the old region that's
[33:01] okay um let's see you know what we're going to do is like we're going to try to add some more data into our our our new primary region because you know it is our primary region. Our applications connected to it.
[33:13] So we're going to go over um and we are now going to go back into our old you know the old old primary region and we do a fail back because we
[33:26] ultimately want to be back in that region. That's where most of our traffic is. So we're doing a we're doing a fail back. Uh I will note uh in the private preview it's you know it's a failover. So you see all we saw that we did a double failover but uh ultimately we're
[33:40] going to support switchover which is an RPO equals zero failover. So you can see that all of our data is there and there's these things there's these special branches there are called recovery branches and the nice thing about a recovery branch is that we're
[33:54] able to uh preserve all the unreplicated transactions from that original region. This is key because the pro, you know, the problem with a lot of DR systems out there is that those transactions just
[34:06] get lost forever, but we have them there and you can connect to them. You can inspect them and you can make sure that all of your data is like they're unhealthy and then ultimately you can merge that data back in with your primary branch. Uh why don't we do that
[34:20] automatically? Well, we don't know how your data diverged. We you know you know how your data diverged and you're able to uh you know repair all of those. So this is how we're able to this one this is how lakebased DR works and uh you
[34:33] know as mentioned it's cross region and you know it works great. So with that um just being cognizant of time I just want to tell you that that actually wasn't cross region DR that was a big lie. It
[34:46] was cross region DR because those were two different regions but they were in two different clouds. So we took it a step further. If you know if you were staring closely you were seen at the top uh the first region was in
[34:59] Azure the second region was in AWS and the reason why we went for crosscloud well there's multiple reasons why I can use the the fancy market slide
[35:12] um the first thing you know we were hearing from customers that from a data availability standpoint they always want to make sure that their data was like closest to their application or their agents or the models that they're using
[35:24] But we want to add additional resiliency like when cloud outages happen uh that you know they can be painful. I think that's what Nikita said earlier. But even you know more practically it comes down to data sovereignty like when
[35:36] you're designing your DR strategies for countries that have you know fewer regions from the from the major cloud providers there you know you might have an AWS region Azure region but you don't have two of them. But with lakebased DR
[35:49] you can have a DR strategy that you know respects your data sovereignty rules. And again with cross cloud we also get cross region. So if you're operating within a single cloud provider you're able to have you know a full cross uh
[36:03] cross region DR strategy. And it actually uh comes down to uh almost like an overly simplified diagram. So your typical cross region DR looks like this. You know we we just do the networking
[36:16] across you know the the cloud boundaries. The trick though isn't just about data replication. It's about identity management. And if you're using, you know, Daybreaks identities as part of
[36:28] your lakebased authentication and authorization strategy, which you should be, um, because it helps with your universal governance, we propagate the daticks identities across the clouds. So that way things connect, things just
[36:41] work. That was probably a key point I missed in the demo was, hey, you were just using the same identity to to connect to everything. you know, all you had to do is refresh your authentication token. We'll click on recovery branches. So
[36:54] recovery branches are something that, you know, we think are novel. You can tell us if these are novel where your unreplicated data is preserved in their original region. So imagine, you know, we're in region A, blue is everything that's replicated over. Gold is the
[37:06] things that are not. We do a failover and we start writing new transactions within region B. And when we do the fail back, we're able to start writing new transactions, but we're able to go back and see those old transactions and
[37:19] reconcile them, you know, within, you know, within our uh our database. So, do you think that is that useful? Cool. That's the goal. Um, where we want feedback, I'm going to put my PM hat on
[37:31] for a second. Um, the immediate thing, you know, customers usually ask me uh when I've talked about this is like, oh, cool. Are you going to provide a tool to be able to do the uh, you know, to be able to do the diff? If you want that, like, let's talk. I'm just going to tell you it's hard. You know your data better
[37:44] than we know your data because it's your data. You own it. All right. So, we're about at time. This is this is perfectly planned to be right at time right now. So, what are the takeaways? So, the first is resiliency
[37:58] is layered. Like I said, it's an onion. When you talk about disaster recovery, you're really talking about everything inside the onion as well. From your infrastructure to your choice of engine, which we hope is lake based. Like that's what we're trying to do here. um you
[38:10] know you know coming to you know high availability where you're you know creating these layered deployments to be able to give you more redundancy and finally to your cross region strategy and you know we built the foundation to both make this safe and perform like
[38:23] safety is our number one concern uh I see some of the lakebased engineers in the room and when we talk about roadmap priority the first thing I always put at the top is resiliency we want your we want your data to be resilient we want to be available and we want to give you
[38:35] the tools to make those decisions and then we get to the fun stuff like features and performance performance, but like first and foremost, an operational database needs to be up and running. Um, the TLDDR is, you know, for production, you know, aside from
[38:48] scheduling backups, turn on high availability. It just gives you that much more, you know, redundancy. And then, like, if an a goes down, you don't have to be upset or get paged in the middle of the night. Things should just work. And then, you know, we do have,
[39:00] you know, cross region and, you know, crosscloud DR and private preview. If you're interested, uh, please reach out to your account team to have them, uh, hassle me to make sure that we can get it enabled and then I will hassle the
[39:12] engineers in the room to turn it on. Um, so they know it's coming. I see you, Jordan. By the way, Jordan was the person who did that demo. I just wanted to say hello back there. Jordan, wave. Wave. Cool. So, we have 20 seconds left. Um,
[39:26] from our UX research team, they want your feedback about uh, some of the upcoming studies. I promised them I put this slide in. And please complete your surveys. Uh I love feedback, good, bad, indifferent. Um please do that. Please
[39:38] do that for all the speakers you go to. And thank you for your time. We are at time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.