Introducing Databricks Managed Disaster Recovery

Summary

Databricks Managed Disaster Recovery delivers turnkey cross-region resilience for lakehouse workloads through a unified API that coordinates Unity Catalog replication, data replication, and workspace asset replication with 15-minute RPO and RTO targets.
A live demo covers creating a failover group, monitoring replication lag, triggering a forced failover, and verifying that applications automatically reconnect via Stable URLs — all in real time.
Managed DR is priced at a 30% compute uplift with no extra charge for Unity Catalog replication, enabling organizations to move from unprotected to fully resilient in minutes rather than months of DIY pipeline work.

Introducing Databricks Managed Disaster Recovery

When a region goes down, every minute counts. Until now, achieving disaster recovery on a lakehouse meant stitching together brittle, DIY replication pipelines metadata, data, and workspace assets. Today, we're changing that. Managed Disaster Recovery brings turnkey, cross-region resilience to the Databricks Lakehouse. In this video, we'll introduce for the first time Databricks' Managed Disaster Recovery offering. We'll cover: - The architecture: how UC replication, data replication, and workspace asset replication work together under one unified API to deliver 15-minute RPO and RTO targets - What gets replicated — and what we learned building DR for managed tables, Managed Iceberg, and workspace assets at scale - Live demo: we'll create a failover group, watch replication in action, trigger a forced failover, and show applications seamlessly reconnecting through Stable URLs — all in real time Whether you're in financial services with strict RTO mandates, running mission-critical ML pipelines, or simply tired of the "what's our DR plan?" question having no good answer, this video will show you how to go from unprotected to fully resilient in minutes — not months.
Talk By: Bart Samwel, Principal Software Engineer, Databricks ; Sirui Sun, Sr. Manager, Product Management, Databricks ;

Chapters

FAQs

What is Databricks Managed Disaster Recovery?

Managed Disaster Recovery is Databricks' turnkey cross-region resilience product that replicates Unity Catalog metadata, managed table data, and workspace assets under a single unified API. It targets 15-minute RPO and RTO, making it suitable for mission-critical lakehouse workloads in financial services and other regulated industries that cannot tolerate extended outages.

What is the difference between RPO and RTO in disaster recovery?

RPO (Recovery Point Objective) defines the maximum tolerable data loss measured as time — essentially how far back you would roll back after a failure. RTO (Recovery Time Objective) measures how long it takes to restore service after the failure occurs. Managed DR targets 15-minute windows for both metrics, which is suitable for most mission-critical lakehouse use cases.

How much does Databricks Managed Disaster Recovery cost?

Managed DR is priced at a 30% compute uplift on protected workloads, with no extra charge for Unity Catalog replication. This predictable pricing model allows teams to budget for resilience without worrying about separate data transfer or replication costs layered on top.

What happens to my Delta and Iceberg table history during a failover?

Managed DR preserves the full Delta and Iceberg table version history during replication, so the replica region maintains a consistent and queryable copy of all table data including historical versions. Applications reconnect seamlessly through Stable URLs after failover without requiring endpoint reconfiguration.

Full transcript

[00:08] All right, everyone. Welcome uh let's get started. Welcome to introducing data bricks manage disaster recovery. So in this talk we are going to introduce and launch a new product uh for data bricks which we're super excited about uh where
[00:20] we um are able to provide resilience for lighthouse workloads. But first uh let's introduce ourselves. So I am Siri. I am the product manager leading this efforts. Uh I also work on manage tables
[00:32] and everything storage related. >> Yeah. Hi, I'm Bart. Uh I'm a software engineer at data bricks and I am in charge of managed disaster recovery. And in the past I've worked on things
[00:45] related to Delta Lake. You may know me from things like deletion vectors and liquid clustering. Um, I have a history working on databases and uh and that sort of thing. And now
[00:57] >> this is the the pet project. So uh Siri, take it away. I'll I'll get off stage for a moment. >> All right, we got to be prepared for them. So anyways, so anyone who's been
[01:09] working in the cloud uh long enough will know this as a fundamental truth, which is that cloud outages are a matter of if but not when. Uh and so how many of you have actually looked at for example the EC2 or the Azure VM SLAs's? True of
[01:22] hands. Anybody? Okay. Actually a good number of good number of you. The thing that a lot of people are surprised to learn is that their SLAs's are uh are really interesting if you look at them. What they say this one is at the entire
[01:36] region level for your EC2 VMs you can expect to get between 99% and 90 and four 9ines between two nines and four nines of reliability. That's like pretty good. That's good enough that if you go
[01:50] and complain, they may give you 10% of your money back. But so clearly they expect this to be so uh so common place that is not a really big deal for them, right? And so I bring this up to
[02:02] illustrate that cloud outages are extremely common and you really need to think about what do you do in the face of them? And so the question that we want to pose that we ask you to pose is what is the right level of resilience
[02:15] for my workloads. Uh really the first best practice we want to share is thinking uh deeply through all of the workloads that you're running not only in the lakehouse but everywhere else and thinking about what is the right level
[02:27] of resilience that you want to have and how much is that going to cost. And so to do that we I want to motivate some language to to talk about resilience. And I'm going to introduce two terms that we'll keep referencing throughout
[02:39] the rest of the talk. The first one is recovery point objectives. So these all relate to the fact that if you want resilience, you want to have a plan when your primary goes down. And so you want to have a secondary region in which you
[02:52] are uh which is your backup plan essentially. And the recovery point objective is how far behind is your secondary in terms of replication from your primary. So you're probably going to be doing some sort of replication.
[03:05] and that replication may be behind by some amount of time. The RPO is how far behind that is and it really governs when you do a failover and you switch over to the secondary, how much data you might be losing relative to the primary.
[03:18] And the second number there or the second uh data point there is the recovery time objective which is how long it takes for you to recover your workloads from the primary to the secondary. Right? If an outage happens in the primary, how long does it take?
[03:30] Is it minutes? Is it hours to go to the secondary? And so using this language, we can think of our options as lying on a spectrum which run from the highest cost but the best RPO and RTO to the
[03:44] lowest cost but the worst RTO and RPO. And the exercise here is for each of your workloads really thinking deeply about well where on this spectrum is the right place for that workload to be. So on one end of the extreme we have
[03:56] workloads that are you know where you really really need zero RPO and zero RTO and there the advice or there the common practice is what we call an active active workload where everything is running in the primary and everything is
[04:09] running in the secondary including compute and there of course you can guess the the cost is going to be very expensive. You're essentially doubling all of your compute and storage costs. On the other end of the spectrum is
[04:22] basically just doing nothing. And here the idea is if you're okay with a recovery time with an RTO of hours or days, which for a lot of workloads I actually think is pretty common. There's a lot of customers I talk to where I ask
[04:35] them what the RTO is and they say, "Oh, you know, maybe it's three or four or seven days." Sometimes the best advice is just to do nothing. Because if you look at cloud outages and how often they tend to last, uh, you know, it's very
[04:47] rare that they'll last more than a few days. And so if you're able to accept accept that level of recovery time, it can be best just to do nothing. And this is obviously free. But in the middle is where I think is a really good sweet spot, especially for
[05:00] mission critical workloads in the lakehouse where you employ an active passive solution. So the idea here is to take a your primary workloads and in the secondary to be replicating the data all
[05:13] the time and to be replicating the compute definitions and the things that are needed to run your jobs and to run your notebooks but to not have those workloads actually running all the time and in doing so you can get to an RPO
[05:26] and an RTO of you know order of 10 or 15 minutes but still not break the bank because you're not doubling your compute costs. So, for the rest of this talk, we're going to actually zoom into this sweet spot because this is where we've
[05:38] designed the manage disaster recovery offering to operate, but just know that this may not be actually the best place to be for all of your workloads. And again, our advice is look at your workloads if you haven't already and think about well, where on this spectrum
[05:51] do you want to be? But okay, if we want to zoom into this active passive workload, two challenges today, broad categories of challenges. The first one is well you want to do the replication
[06:03] of data and metadata and everything that makes your lakehouse tick. How do you do that effectively? How do you do that without doing a bunch of DIY coding? That's pretty challenging today. The second challenge is well even once you
[06:15] assume you replicated from a primary to a secondary, how do you make sure you very quickly are able to fail over to that secondary when a disaster strikes without having to do a bunch of reconfiguration to point everyone to the
[06:27] secondary? And so that's those two challenges are specifically what our managed offering uh does. So we are launching on stage today a gated public preview of data bricks managed disaster
[06:39] recovery at a very high level. It does two things. The first thing is managed replication of the lakehouse assets that you need to run your active passive workloads and it includes workspace
[06:52] assets like notebooks uh dashboards reports. It includes the catalog and the metadata there and it also includes your managed data like the data underlying your manage tables. And secondly, it
[07:04] includes a stable connection string that you can use which always points to your primary. So this solves the second problem that I just raised which is well when my uh primary region goes down, how
[07:16] do I quickly go and fail over to the secondary? Okay, so uh this is how it works at a high level but a demo is worth a thousand words. So I actually wanted to show you guys this running live. And so this is production code. This is a live
[07:29] thing. This is not a recorded video. Um and so the first thing I'll show you is the prerequisite. So to set this up, the first thing we ask you to do is just simply provide us a primary and a
[07:41] secondary workspace and a secondary meta store into which we do the replication. So these should you know the workspace especially should pre preferably be empty. The metas store can just exist. But here in my accounts console you can see I have a primary and secondary pair
[07:54] well two primary and secondary pairs of workspaces. And then similarly if I go to the catalog I can see the same thing. Okay. Now from there the question is well what do we actually replicate because you may not want to replicate
[08:06] your entire uh data bricks state. And so there we have this new resource that we built called the failover group. And this is the thing that's in charge of doing replication and failover. And this is how you configure what to replicate.
[08:18] So, this is a new uh piece of UI in the accounts console. And you can see it uh I have a few failover groups already, but let me just go and uh show you what it looks like to create one. So, when I go create one, the configuration is
[08:30] pretty simple. It'll ask me what uh what to give to give it a name. I'll then be able to select a primary and secondary workspace. So, you know, just for kicks, I'll say primary is here, secondary is here. Um, we'll talk about stable URLs
[08:44] in a second, but the other big piece of configuration is simply, well, what cataloges should we go and replicate? So, you don't have to replicate all of the data in UC, you can select specifically the cataloges that make up your mission critical workloads. So, I
[08:57] can say something like, you know, my DR catalog and then I can also optionally say, hey, also for the workspaces that I configured up here, replicate their workspace assets, notebooks, jobs, things like that. The last bit of advanced configuration
[09:10] which is optional is storage location mappings. So we understand you may have external locations in region A and you may want to, you know, make sure they're mapped the same way to storage locations in this the secondary region. And so
[09:24] this is how you just tell us how those mappings work. And from there, that's it. You can just hit create. And I won't do it now because I've already created a few. But once you've created them, we will go ahead in the background do replication for you uh for all the
[09:36] assets that you just uh that you just had. So I will I'll show that to you live as well. And so here on the left tab on the left uh window I have my primary workspace uh and that happens to
[09:48] be in Dublin uh EU West one. And then uh on the right tab I have the secondary workspace which happens to be in uh Frankfurt. And I've already set up a failover group that's doing replication between them. So this is uh to me both
[10:01] exciting and boring by design because they're just the same workspace and they're being replicated in real time. So the first thing I can show you um is the fact that if I go into the catalog
[10:13] of cataloges that I've created that are being replicated. And if I click into them very simple I can see that a I have this schema uh called DR test one with five tables in the primary and then over
[10:25] here the same schema exists in the secondary. the same tables but not only that but the same UC permissions the same ales that you set as well as the version history of the tables is preserved as well so it's a very high
[10:37] fidelity replica moreover we also for this failover group did say to include the workspace assets and so I'm going to go over and I can actually show that live because those are a bit easier to create so in my primary and secondary I
[10:50] already have a folder called super important notebooks uh I'm going to show that both are empty to start uh but over here in the primary what I can do is I'll just go ahead and create a notebook. So, you know, I'll call it uh
[11:04] Dr. Dice presentation and I'll just say, you know, print hello. Okay. And then from there, I'll save it. Refresh just to make sure it's there.
[11:18] And then over here, you already saw it show up, right? So on the secondary here over in the Frankfurt region, the the the notebook has already replicated and shows up there. So that's replication notebooks and uh
[11:30] data uh moving from uh being replicated very quickly from one region to another. And so if I go back uh if I go back to this failover group, I can see it running. This is one we were just using.
[11:42] And the one thing additionally that we do is make sure there's great observability around how this uh how replication is performing right now. And so we do show you the RPO, which as you recall is how far behind is the
[11:55] secondary running. And in this case, what we're telling you is everything in the secondary is no more than 8 minutes behind the primary. Now, some things might be more up to date than 8 minutes, as you just saw that notebook replicated very quickly,
[12:07] but this is an invariant you can use when you go and fail over. Okay. From there, we have two workspaces which are the same, but you'll notice that they actually have two different URLs. And so what we can do what we do is provide you
[12:21] with a stable URL which does not change so that you can make sure your downstream clients have as little impact as possible during outages. So here you can actually see it. Uh it's a it's just a link that we provide you for every
[12:33] failover group that you can attach. When I click on it, it does something very boring which is just direct you to the primary workspace at any given point in time. You can use this for web browsing. We can use this for uh APIs for ODBC and
[12:47] JDBC connections. But critically, let's say our primary workspace goes down and I want to do a failover. So if I click into the failover uh button over here, it'll just have me confirm that I want
[13:00] to do a failover, which I can do. And now failover is very quick. All it does is one, it replica uh it changes the direction of replication. So the primary is now what used to be my
[13:13] secondary and it's replicating into what used to be my primary which is now the secondary. And two is it points the stable URL to the new primary. So if I go back to that same stable URL, what I'll see is that now it points uh
[13:27] to the secondary workspace. Do one little refresh here. Just like that. So now uh I'm serving from this stable URL out of my secondary uh in the Frankfurt
[13:39] region. But critically for my end users, they may not even have to know, right? They don't know. Uh there might be an outage, but everything from data bricks just continues working just as is. Awesome. Okay, so that was a quick
[13:52] overview demo of managed DR. You might be wondering, uh I might be wondering, not sure what happened there. All right.
[14:07] Okay. One more announcement. You might be wondering how much this costs. Uh very quickly, the managed DR offering has two pricing components. The first one is we are introducing a new missionritical add-on. So this is an a add-on that you apply to your workspace
[14:20] which provides you one the ability to run managed DR the way I just showed you. And two, it also includes all of the enhanced security and compliance capabilities that we already provide in our enhanced security and compliance add-on. And so both of with both of
[14:34] these uh the price for this is going to be a 30% uplift on the compute for the participating workspaces. But for the first year it's going to be a 15% effective uplift due to a promotion. And that's just to allow uh you guys to try
[14:47] it out and effectively you can think of it as you know no additional uplift for DR because the enhanced security and compliance uh capabilities are all 15% anyway. The second thing to add is that
[14:59] well for UC replication and workspace asset replication that is no additional charge but for replicating the managed data we do have a uh very straightforward 2 cents per gigabyte charge there as well.
[15:14] Okay, so with that now that you have a sense of how the managed DR offering works I want to bring up Bart to go into the details of the nuances of how it works and also provide some best practices. Thanks Siri.
[15:31] All right, so you've configured your managed DR. So you've set up your replication and you can fail over. Job done, right? You
[15:43] don't have to do anything anymore. It's magic. It just works. I mean, the demo that you just saw was fantastic and it showed that this was all very smooth, etc. But your job is not entirely done
[15:55] because when you fail over there is a little bit of data that you lose because of this RPO concept and that introduces some complexities
[16:07] and you will still need to be aware of what happens there. So, what I'm going to do is I'm going to walk you through the life of a failover group and I'm going to show you exactly the things
[16:19] that you need to know that you need to take care of for your failovers to actually work. And I do encourage you to also test your failovers to run them regularly so that you actually know that
[16:32] the things you've done actually make sure your your workloads are safe. So, let's get started. So like Siri said
[16:44] you have a a workspace and a meta store with some cataloges in it in your primary region which will be we will call region A and in region B we expect
[16:57] you to create a workspace um and a meta store might already be there or you create one and then you set up this failover group uh and you in this case we tell it to replicate
[17:10] cataloges A and C. So what happens when you do this? Well, first of all, we create the cataloges for you in the secondary region. Identical identical permissions, identical everything.
[17:24] And then we do something that is called an initial sync. So that means that all of the resources that are there in region A get replicated over to region B the first time.
[17:37] So that's like the starting point, right? While this is happening, your replication isn't up and running yet. We're doing the the initial synchronization between the regions. And
[17:50] one of the things that you need to think about here is that initial synchronization takes time. So there's actually a lot of work to be done. We have to figure out all of the objects
[18:02] that exist. And I have heard of customers that have millions of workspace objects and there can be many many tables and there's also a lot of data involved. Data tables can be well
[18:15] gigabytes is small data. Uh terabytes, pabytes even it's it can get very large and all of that needs to be transferred one from one region to the other. And to give you some sense of perspective here,
[18:28] a cloud provider network gateway typically does not offer more than 100 gigabits of network bandwidth. And I've put in a chart here how long it takes to replicate some amounts of data at that
[18:41] uh bandwidth if you had that full bandwidth available to you just so that you get some sort of sense. And if you have 10 terabytes of data, then it will
[18:53] take at least 15 minutes to transfer. And usually you might have less bandwidth available. So it might even take longer than that. If you have a 100 terabytes, it can take two hours up to a
[19:05] day to transfer that. Um so this is the kind of challenge that you deal with and you should expect the initial sync to take a while. Once it's up and running though,
[19:17] then we are in operation. And what the system does then is we do incremental sync. So that means we only replicate
[19:29] what changed. So in this case I've highlighted a number of elements that have changed in region A and we run this change detector which just checks which
[19:41] things change in a very efficient way and then it simply copies over those things and nothing else. This might still take time. If you have
[19:53] a large change to your tables, then that data still needs to transfer through those network pipes. But if the changes are small, then it's also going to transfer very quickly.
[20:06] So I mentioned tables. So how are tables copied? Um, one of the things that you should realize probably many of you know that these
[20:19] delta and iceberg tables are multi- version. So in this example, I've listed four versions. Every time you you add a change to a table, you modify it, it creates a new version of the table and
[20:32] the history is preserved. And most users will just see this latest version. But then maybe somebody might time travel back to version three for auditing purposes. They just want to know what was in the table like an hour ago. Maybe
[20:46] you have a streaming pipeline that got up to version two and needs to know exactly where it left off so that it can continue from there when it catches up. So all of these versions are important.
[20:59] Now, maybe some of you have built disaster recovery for yourself using custom methods, DIY. Uh, can I see a show of hands who of who's done this? Okay, that
[21:12] that's quite a lot of people. So, if you if you did this, then you might have used this deep clone command that we have available in data bricks. And
[21:24] the the problem with this is that it doesn't preserve those versions. If you would clone this table, it would create one version in your replica corresponding to the latest state. Now,
[21:36] let's say that you have two new versions in your primary uh in your primary that get added and then your replication job runs. Then the deep clone would create
[21:48] another version, but those versions totally don't line up. So any kind of reasoning about how that matches up is very hard to do. So, I'm happy to tell you that in
[22:00] managed disaster recovery, we've fixed this. So, manage disaster recovery will actually create an identical version of your table with all of the history intact, all of the versions intact, and
[22:12] it will also copy all of the related metadata such as change data feed and iceberg metadata. So, managed iceberg tables will work. Uh, everything is
[22:24] identical. So with that let's move on to the next topic. You have this thing running. It's replicating happily. And uh Siri already
[22:37] showed you this concept of RPO the recovery point objective. So I want to distinguish between two different things here because you'll you'll want to know this difference
[22:51] actually when you're talking about targets. So you want your data to be uh at most this much behind. Then you're talking about a relative RPO. So you're talking about I want my data to be at
[23:04] most 15 minutes behind for instance. That's that's a target that you can verify you can check that the system maintains that sort of um SLO maybe. Um,
[23:17] but when you're doing failover, actually what you care about is not the relative number. You care about the absolute number. You care about up until when is my data replicated? What is my RPO high
[23:30] watermark, so to speak? And that's really just a time stamp. It says, what data am I guaranteed to have in the secondary when I fail over?
[23:43] So now we're going to go deep. How does the system actually determine this RPO high watermark and when does it advance? So I've painted a timeline here going
[23:56] from 810 to 8:22 and we're going to run a scenario here. So let's say at 810 our change detector runs and it
[24:08] checks what changes exist and then it takes a couple of minutes to copy those changes over to the to the secondary region. Then after it's done, the change
[24:20] detector runs again and we copy changes once more. Right? So far so good. So what does this do to your RPO? I'm going to try and
[24:36] point some things out here. So the absolute RPO here uh in the beginning, let's say that that's 803. So the data up until 803 is guaranteed to be replicated.
[24:49] The relative RPO at this point is 7 minutes. That's the difference between now and the absolute RPO is seven minutes. So as we copy the absolute RPO stays at 803 but our relative RPO keeps
[25:03] increasing until we get to 11 and then at this point the copy is done and we are up to date up until 810. So that's the point where the absolute RPO increases and the relative RPO goes down
[25:17] to 5 minutes. But then we go again and we check for changes and we start copying and the relative RPO goes up again until the next cycle of copying is done and then the RPO is updated, the
[25:31] high watermark is updated and the relative RPO drops again. So what does this mean for a change that you're making in your primary region? So let's say you are
[25:44] making a change at 811. Note that this is exactly after we check for changes. So the change detector just ran. So when do we see this? Well, we
[25:56] actually only see it when the change detector runs next. So that's at 8:16 in this example. And then it feeds into the copy cycle. So it gets copied with everything else. And when are you sure that this data
[26:10] hits the secondary region? Well, it's at 8:22. when that next copy cycle is done. So that's basically your lifetime of a change and how it progresses through this whole replication cycle. And this
[26:25] may help you understand a little bit how this RPO progresses and how to think about it. But let's do a little bit of a quiz to make sure that people understand what happens here. So I want to see a show of hands in a moment. Uh so think
[26:39] about your answer carefully. So we have this scenario and these are different times. At 9:20 we're happily replicating and the RPO is at 910 absolute or 10
[26:52] minutes relative. So at 9:21 disaster strikes oh nos your pager goes off. So you get out of bed. Well is 9:21 you're you don't get out of
[27:04] bed. You you go to your computer and you figure out is this a temporary thing? Should I just wait this out or is this a reason for failover? And you decide this is bad. I'm going to trigger a failover.
[27:16] 9:33 you trigger that failover and B is now the primary at 9:35. So now the question is what data at this point do you have in region B?
[27:31] Answer A. Is that data as of 911 10 minutes before the disaster happened? Anybody think it's A? Okay. B. Data as of 9:23 10 minutes
[27:47] before the failover. Show of hands. No. Um maybe you have C all data before 910 and nothing after 910. Who expects
[28:01] this? Ah quite a lot of hands here. and D all data before 910 and some data from after 910. I I would say so
[28:15] a lot of hands for C and some hands for D but actually the answer is D. So as we're copying data actually does
[28:27] appear on the secondary but um only when we're done with the full replication cycle when we're done copying then everything is there so we can up update the high watermark but
[28:40] data does get copied as we're going so that explains why it's D and not C. So um
[28:52] now that you understand this, let's talk about what happens to table histories when you fail over because this is this can be somewhat subtle. So let's say that you have a table in region A and
[29:05] that table has seven versions but at the point where you have a disaster happening, region B only has four of those versions. So um data has been replicating but the replication
[29:18] fails now because the primary is down and you fail over. So bam we're failed over region B is now the primary. So what's happening here? So region B
[29:30] starts up compute starts running your jobs start running and they create new versions there. So you have version five there and version six. But note
[29:42] that region B has versions five and six, but region A also has versions five and six, and it even has version seven. Those are not the same versions. They're
[29:54] different data. And after you fail over, you have to realize that region B is now your source of truth. It's where your primary is. You've you've restarted history from a
[30:07] little bit farther back in time. And that means that what happened in region A is now a divergent history. It's an alternate history that didn't really happen as far as your source of truth is
[30:19] concerned. So what happens when we then start reverse replicating is we want to make region A faithful copy of region A of region B again because region B is now the
[30:32] primary. Region A should become the secondary the copy. So what happens? we actually have to get rid of that divergent history and
[30:44] copy over those versions from region B. So um this is this is subtle but I think it helps you understand a little bit more
[30:56] what kind of thing you have to deal with when a failover happens. So there's one final topic that I want to cover in this deep dive. We have a couple of minutes left here and uh I
[31:11] wanted to talk to you about ETL. So there's two things that are interesting at failover. One of them is ingestion and the other one is jobs and pipelines.
[31:24] Um closely related. So let's start with ingestion. Let's say that you have in the primary a an ingestion job running from some source. It doesn't really matter what
[31:37] source. Let's say that this source tracks progress by a time stamp. So we have data in this state. We've processed data up until 8:21.
[31:49] And every version of this table here actually contains data from the source up until a particular point in the sourc's data stream. So version one has
[32:01] all the data up until 750. version two goes up until until 757, etc. And we've replicated this to region B and right now we're at version four here. So these
[32:14] last two versions haven't been replicated yet. Okay, so disaster strikes and we fail over
[32:26] at this point. What would happen if we would take that same ingestion job, start it on the new primary on region B and just let it continue where it left off at 821.
[32:39] Well, in that case, we would create a version five which has data starting from 8.21 until the point where we actually recovered at 8:35.
[32:51] So, at this point, what's wrong? Well, where's that data from 809 until 820? It's not there because it only happened, it was only ingested in this alternate
[33:04] history on the old primary. So that data is no longer there. It's not the source of truth. Region B is the source of truth and it doesn't have it. So what do you need to do? Well, when you fail over,
[33:18] your ingestion pipelines have to rewind and ingest a little bit more data. and you have to figure out what that boundary is and ingest from that. If you have battery statement pipelines that
[33:30] work by day for instance, then it's easier. You just reingest a whole day. That's that's simple. If you have streaming pipelines, you might have to overlap a little bit with the previous one and do dup dduplication so that you don't get duplicate ingestion. Um it
[33:44] really depends on your situation, but you have to figure this out for your pipelines. And if your pipelines aren't ready to do this, then you are not ready to fail over. So this is a job that you still have to do.
[34:00] So the other part is pipelines. So let's say that you have a pipeline with table A, B and C feeding into each other and they run every minute or every two minutes and they produce data. So
[34:13] basically table B is derived from table A could be bronze, silver, gold layer style pipelines and I've listed here the versions. So table A has four versions
[34:26] and table B has two versions. It runs every two minutes the pipeline that or the the job that produces that. And I've listed dependencies. So for instance this version is produced from this version. Now, normally you would only
[34:40] see the latest versions and not care. At failover, let's say that you have a failover that happened at RPO 810. Then you might lose some of these versions. And some of this
[34:54] is okay, right? This one is gone, but that's kind of okay. This one, however, this version is there, but it was derived from data that didn't make it.
[35:07] So now you've lost your lineage of your downstream data. So really you don't want to have this situation because then well what is that data produced from? Why is it there? You you can't actually
[35:20] prove why that data was there. So you might actually need to go back and set things up differently, reingest some data, reprocess some data to get things back in order again. And it really
[35:33] depends on your workload how you do this. So there are three ways to handle this. Like I said, you can do this in a custom way. Option two is you actually use u spark
[35:48] declarative pipelines. Uh with this product, this is actually a managed product. So we know exactly what your state is of your pipelines and of your tables and we can correlate this
[36:00] and we can actually correct for this and make it right. And your final option is to wait until the first half of next year uh when we will uh when we expect to be able to
[36:14] provide cross table consistency uh for your tables that you replicate. So uh with that I want to conclude my deep dive and I'm
[36:26] going to hand it over to Siri who's going to finish with some final words. >> Let's do it. Is my work? Is my mic working? Yeah. All right. Second time's a charm. All right. So, in closing, I just want to recap everything that we
[36:39] covered today. I'm going to start with where we started. Outages are a matter of if and not when. And so, it's very important if you haven't done so already to think through the workloads in your lakehouse and determine the right
[36:51] resilience posture, right? Where on that spectrum you want to be and what do you want the target RPO and RTO to be. From there, you saw how you're able to lean on our managed DR product, which we're launching to public preview today for
[37:04] the hardest challenges, right? Replication at scale failover using connection, stable connection strings. And finally though, you saw how it is still important to exercise these resilience muscles. So, one thing I
[37:17] always advise everyone is don't just set this thing up and hope that it works in case of a disaster. What it's really important to do is set it up, but have regular failover drills to be able to exercise those muscles and to make sure
[37:29] that when the disaster happens, it's not the first time you're exercising those muscles. And as part of that, hopefully you can catch some of the nuances that Bart went through. So, for example, making sure that you have pipelines that
[37:41] work well after failover. And so, with that, that is our talk. That is manage disaster recovery. And uh we look forward to having you guys try it out. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.