Streaming resilience: building recovery into data pipelines with Databricks

Summary

National Australia Bank operates 3,800 streaming pipelines with a 99.6% success rate after migrating to Spark Declarative Pipelines, reducing the operational overhead that previously required manual intervention from senior data engineers for recovery.
Databricks' new streaming rewind and replay capability enables a two-step recovery process—selecting a rewind point and replaying data—that reduces recovery time from hours to minutes while maintaining data consistency across complex multi-table pipelines.
NAB replaced 650 custom lines of pipeline recovery code with a single API call, demonstrating how declarative semantics and automatic state management transform streaming reliability at enterprise scale.

Streaming resilience: building recovery into data pipelines with Databricks

Streaming pipelines are business-critical infrastructure, but recovering from failures has been manual and complex. When a transformation error or bad data corrupts a streaming pipeline, recovery traditionally requires manual coordination across checkpoints, Kafka offsets, and downstream impacts.
This talk explores how National Australia Bank transformed streaming reliability using Spark Declarative Pipelines and the new streaming rewind and replay capability. Learn how automatic state management, declarative semantics, and time-travel recovery reduce operational overhead from hours to minutes, enabling consistent recovery across complex multi-table pipelines with guaranteed data consistency and auditability.
🤝

Chapters

FAQs

What is streaming rewind and replay in Databricks?

Streaming rewind and replay is a Databricks capability that allows data engineers to recover streaming pipelines by selecting a point-in-time rewind checkpoint and replaying data from that point forward. It automates the previously manual process of coordinating checkpoint resets, source offsets, and downstream impact assessment.

Why is recovering streaming pipelines so difficult?

Streaming pipeline recovery is complex because it requires coordinating multiple components simultaneously: checkpoints, source offsets, pipeline state, and downstream table consistency. In traditional architectures, recovery required a runbook and a senior data engineer to manually fix code, coordinate state, and validate data quality.

How did National Australia Bank migrate to Spark Declarative Pipelines?

NAB migrated using a phased approach that standardized their bronze layer first, then uplifted silver and gold layers, with a streaming-first service level objective across their ADA platform. After migration, NAB runs 3,800 pipelines with a 99.6% success rate, as described in this video.

What was NAB's proof-of-concept for the streaming rewind and replay feature?

NAB conducted a proof-of-concept using a finance system reconciliation use case that required strict consistency guarantees, validating how streaming rewind and replay handled recovery scenarios. The POC uncovered both capabilities and gaps, and the findings are shared in this video alongside a live demo of the recovery process.

Full transcript

[00:08] Hello everyone, and thanks for joining us for this session. Um, I hope you all are having some insightful summit so far, and um it's the um I think it would be the it would be your last session for the day. It's a long day, but if you are using
[00:23] streaming pipeline in your production workload or planning to use, stay tuned. We have surprise for you. This is our forward-looking statement. Everything stated today, um it applies to that. I'll just get through.
[00:39] Okay, um before we dive in, let me get a quick sense of the room. How many of you are data engineers here? Okay. Quite a few. And how many are from platform or operations team?
[00:54] Okay. There's a mixed audience. How many of you are already using Delta Live Table or Spark declarative pipeline in your production workload? Okay. That helps to place this at the right label. Um so, let me set the scene with
[01:09] something you most have probably faced in your workload, right? You have a streaming pipeline in your production workload, a new update goes in, and a subtle transformation error corrupts your data. Your dashboards are wrong, and your ML
[01:26] features are either stale or worse, corrupted. And before monitoring catches, it has been few hours since your bad data has been flowing through to your target table. And now you have one of two options. One, either you have a sequence of
[01:42] well-coordinated steps to recover your streaming pipeline, or you may not have a run runbook, and you have to pull in one of your senior data engineers to manually fix the code. This is the problem we are here to talk about.
[01:59] And see how we can transform and modify this in a seamless fashion. My name is Parthasarathy Sahu. I'm a senior delivery solution architect at Databricks. And I'm joined by Dheeraj
[02:14] Fully, head of data reliability engineering at National Australian Bank. What we are going to cover today? First, obviously, what's the challenges with data pipeline and specifically
[02:30] recovering streaming pipeline, why it's hard. Then Dheeraj will share us NAB's journey, why and how they move to Spark declarative pipeline, and their findings from the POC they did with the new feature that we're going to talk about.
[02:46] And we'll end the session with key takeaways and next steps. So, the problem statement Streaming pipeline have moved from nice to have a business critical infrastructure for most of you. What we
[03:02] most frequently talk about is what streaming make possible, real-time analytics, personalizations, or fraud detection. What we talk far less is what happens when streaming pipeline breaks. And the truth is recovering streaming pipeline
[03:19] is hard. You have to run through manually manual coordinated stuff, which takes a lot of time and effort. So, why it is hard? Now, when a batch job fails, usually it's straightforward. You fix your code,
[03:36] rerun the job, and you have the answer. Usually your batch job state resides in your source system, not in the pipeline itself. Whereas, streaming pipeline is fundamentally different. Streaming pipeline is not just a job,
[03:52] it's stateful. And its states resides in four distinct places. Your Delta table version, your streaming checkpoint, your source or Kafka offset, and your aggregations logic such as window functions or
[04:08] watermark. Now, all four of these state has to stay consistent. For example, if you just recover, let's say Delta to Delta table version, and you forgot to reset your checkpoint, then you are
[04:24] either missing your data or the data is corrupted. And here's the root of it. Structure streaming was initially designed to move forward with forward only progress, right? There is no native undo function.
[04:42] And recovering one streaming table is hard, but manageable. But let's say you have a chain of streaming table and materialize view in a pipeline. If you are fixing the root, you have to redeploy each downstream pipeline in a
[04:57] consistent fashion. Now, we'll see how NAB fixed it at scale. Before that, let me just do a quick refresher. I know most of you are here familiar with Spark declarative pipeline, but how it helps and how it
[05:15] enables the streaming pipeline recovery, which we'll talk about in a little while. Ideally, your data engineer need to focus on the transformation logic that is unique to your business. Whereas, when you are moving a pipeline to production, there is a lot of manual
[05:32] framework that you have to build or stitch together, right? Such as your table dependency management, your backfill, checkpointing and retry, and monitoring logic. Instead, you should be focusing on that is what is unique to your business. And
[05:49] that is why Spark declarative pipeline helps. You focus on what your true business requirement are, and you define that with either a streaming table and materialize view, and how they are related. Rest is taken care by the platform. Be it the dependency graph,
[06:06] retry and recovery, checkpoint, quality checks, and all of those. Now, with the context being set for why streaming pipeline is hard and what is Spark declarative pipeline, I would like to hand over to Dheeraj, who will
[06:22] take us through NAB's journey on how and why they migrated to Spark declarative pipeline. Dheeraj. Hope everyone can hear me okay. Yep. Okay. Thank you.
[06:43] Um thanks, brother. Hello, everyone. Uh I'm Dheeraj. I work for the National Australia Bank. Um came from uh Sydney, one of the best harbor cities in the world. Let me uh share the NAB journey from a custom Spark framework to the
[06:59] uh SDP, Spark declarative uh pipelines, and then what we unlocked operationally. So, if I use a term DLT, that means it's SDP, because we started this journey as a Delta Live Tables, and then it went through a few rebranding things.
[07:15] Uh and I'm also aware I'm standing between you guys and then the food party and then drinks. So, we'll make it quick and then more informative. So,
[07:32] Okay, something uh about NAB. NAB is uh one of the big four banks in in Australia, and um the largest business bank in the country with 170 years of history, 38,000 uh colleagues, and then 10 million customers using internet and mobile banks. And then we have a 700
[07:48] branches and locations. So, in a highly regulated environment, heavy scrutiny how the data is stored, lineage, and then controlled. Everything we do in the platform has to clear that bar. So, which means like you know, that's kind of a shape our architectural choice
[08:05] including the move to declarative view pipelines. Okay, so let me introduce Ada. This is an apps lakehouse which is our unified data
[08:21] platform for the real-time analytics and data science, AI, natural language, Genie, everything. So, the best way to read this slide is from left to right. So, on on ingestion side, we have this
[08:39] five trend connectors which kind of gives us a SAS connectivity. And then the HVR sources like give us the high volume database replication in a in a real-time without moving the data like in a keeping it in a private secure place.
[08:55] And auto loader, DLT, which is for the file-based ingestion. We can read incrementally like you know, incremental changes. So, once the data in on the platform, we have full capability in
[09:11] GenAI ETL like you know, unified governance, data science, and ML use cases. And then also we have a Genie space, Power BI, and then AI BI dashboard for end users. All will help our business use case, FinCrime, customer journey personalization, and international
[09:27] operations. Okay, so we when we started we built data on
[09:44] three key core principles. One is a zero copy, which means no more reprocessing the data every day. It's only like you know, the changes only. Second thing is to less to no custom coding. Wherever the out-of-the-box functionality available,
[10:00] start using and then custom coding is your like you know, last option. And then third one is platform should be able to support near real-time use cases. So, we have hundreds of engineers building using this platform to build
[10:17] those injection like in data injection pipelines. And then those with the custom Spark framework, they do things differently to achieve the same outcome. So, we wanted to standardize the things. So, that's when we started
[10:32] this SDP DLT journey. So, we started with the two stream approach. One is the new sources, which we call it as a front book. They will start ingesting using the SDP
[10:48] pipelines. And then the back book for the existing sources, which currently running on custom Spark, we will migrate them like you know, to the SDP.
[11:06] So, when we started this Sorry. Yeah.
[11:24] Yep. Okay, so when we started Bronze was actually like you know, using SDP from day one with standardized connectors, uh streaming table with your um expectations customer like in technical and the business expectations with the quarantine rule in in place.
[11:39] The opportunity was in uh, silver and gold where we using custom uh, Spark framework with a legacy SQL uh, and then bespoke margin and SCD1 and 2 logic. So, we I mean, that's probably one of the
[11:55] extreme uh, scenario where we have a uh, like you know, use case with uh, 64 union hours to meet our business like you know, one of the complex business requirements. So, the the the goal become shift these uh, challenges into the opportunities and then how we can manage them like you
[12:11] know, incrementally uh, uh, consistently load the data. Okay. So, we made a deliberate choice of
[12:26] expressing as a part of this like you know, migration journey, expressing uh, SLO uh, which is like you know, latency shouldn't exceed its its a processing time. What that means is um, streaming first thinking. So, even based uh,
[12:41] watermarks, incremental processing uh, and then real-time data flow. Okay. So, the um we wanted to have a consistent way of doing end-to-end architecture like you
[12:56] know, bronze, silver and gold using like you know, uh, the the uh, STP and then we also want to have pipelines with the SLO and then also auto scaling uh, and then for uh, fast recovery as well.
[13:15] So, and then the final one is platform should be able to support near real-time uh, like you know, use cases and then currently we do have a couple of use cases under 15 minutes end-to-end. Uh, we platform we're to support that one.
[13:37] Okay, this is a pretty standard medallion architecture. So, the source loading into the bronze, which is streaming tables. And then like which is a true to source without any business transformation rules. And then the silver is in streaming table as well with with the
[13:54] rich business rules and then the conformance layer. The currently the gold is in we use the delta tables, but in the future we planning to move it to a streaming table and materialize views as well. So, the platform manages the incremental
[14:12] refresh and then consist consistent way of like you know doing things including the governance and then lineage thanks to Unity catalog.
[14:29] Okay, this slide summarizes the NAB benefits. Like you know, so far when we completely stopped the old custom framework pipelines. So, what we've seen so far is like you know, reliable reliability currently we have a 3,800 SDP pipelines running on
[14:45] the in in silver with a 99.6 success rate, improved data quality, which means we already got a stronger like you know, baseline there. And then the second benefit is simplicity and
[15:00] latency. So, one of the key things is moving away from bespoke uh frameworks to the standardized approach which SDP. Uh we can handle those uh patterns slowly changing dimension one dimension two patterns like you know, much more seamlessly in in SDP than than
[15:18] than the custom way. And then one of my favorite one is uh late arrival data. So, in in SQL or custom Spark world, if you find any issue after 15 days, you need to roll back the state to like a 15 days
[15:33] old and reprocess the entire data. Whereas, in SDP, if you simply plug that like, you know, late arrival data and it will process the entire thing and then set you are all the start dates and end dates.
[15:51] Um so, we expect 15% lower run cost when we turn off the all the Spark jobs and then running only on SDP. Currently, we we run both in parallel, like, you know, our SDP and then custom custom jobs.
[16:08] So, as Silver is Sorry, Bronze is 100% declarative, as I mentioned, all the new Silver is going on SDP. We We completed the majority of the uplift of the existing custom Spark Spark SQL to the SDP, which is running in production. We call it as a technical
[16:25] migration. Now, we are in the middle of testing those jobs for the safe cutover between, like, you know, custom Spark to the SDP. So, once that happens, we can claim these these benefits.
[16:44] So, part of this journey, one question came out when things go I mean, things do go wrong and then how do we recover more quickly? So, let me hand over to Partha to talk about the new feature.
[17:01] Thanks, Niraj. So, earlier we talked about why streaming pipeline is hard and how Spark declarative pipeline is making building and operating production pipeline um seamless.
[17:17] And we have been adding lot of new capabilities on top of Spark declarative pipeline. Um for example, the custom pattern that uh earlier engineer used to custom code, we are taking those pattern and building
[17:32] a high-level API that you can use. So, with that, I'm excited to announce we have added a new capability called rewind and replay or streaming re- uh streaming recovery and um time
[17:48] travel. So, what is streaming time travel? Streaming time travel lets you recover your pipeline to a previous consistent state. So, if you have a pipeline which has failed due to uh a new update that has a
[18:04] transformation logic issue or a bad data sifted, using streaming time travel, you can pick a previous consistent time travel uh sorry, rewind point, and it will recover the pipeline for you. No more hand stitching and
[18:19] uh reset of checkpoint or no need to do a full refresh of your streaming table. So, there are essentially two main step that um uh you have to perform. First is rewind. You pick a rewind point, which uh I'll
[18:36] show you quickly um how it looks like in the UI, but essentially you pick a rewind point, and then you replay, which is you run your streaming pipeline either manually or it runs on your schedule trigger, and it will replay all the data only for the required timeline
[18:54] from that failure uh point to now. And it it is maintained um and auditable. So, if in case there are edge cases where you still need to go and do some manual validation, you can still do
[19:10] that. Now, it has streaming pipeline is scoped at a pipeline level. So, which means you can rewind the entire pipeline with cascade option. When I say uh pipeline, you can have a stream
[19:26] single streaming table as part of your pipeline, or it may be a multiple chain of flows, materialized view, and streaming table. So, with this capability, you can pick and choose your scope or blast areas, I should say. Now, let me show how it looks in the UI.
[19:43] So, streaming rewind replay, it exposes through the workspace UI. If you go to your workspace, select your pipeline, and go to pipeline action, you will see this rewind pipeline option. Once you hit this option, it will expose
[20:00] you all the consistent uh recovered or rewind points based on your Delta table version. So, you don't have to guess what's the exact timestamp you need to recover it to. So, once you have then selected the timestamp, next, it shows you the
[20:16] dependency graph. With help of lineage, you can find out let's say if you want to recover the root and rewind it, then what's the impact down size. So, you can pick only the one you want, or you can select all the downstream
[20:31] data sets to be rewound. Once you have selected, if you see on the right-hand corner, there is a reset all checkpoint option. Once you select that, it will show you all the impacted data set that's going to be rewound, and the
[20:47] checkpoint will reset. When you hit rewind, that's it. Next, when your pipeline runs or scheduled based on your trigger, it will rewind from that rewind point to the now, and it will continuously
[21:02] uh do the rest of incremental processing from there. Now, let me hand over to Dheeraj who were one of the key design partner in this feature and share their POC findings.
[21:22] Uh okay, so let me give some context why we like you know put our hand up for like you know part of the private preview for this particular feature. So, few months ago we have a requirement like business requirement. Like you know we
[21:38] their data like you know it's one of the finance systems and then they don't want to load the ranges the data unless it's fully reconciled. So, that means for any reason if this data is not reconciled they want us to roll it back to the previous state where the data is reconciled. So, that's one of the
[21:55] requirements. As a one of the data principles um let's not use any custom code, look for the out of the box functionality as a first option. Custom code is the last resort. Then we reached out to Databricks like you know teams to see if they have any out of the
[22:10] box functionality to handle the scenario. So, at that time we got to know about this even and replay functionality. I know it was not ready that time so now built a tiny module custom code to meet our like the timelines and then the
[22:26] business requirements. But that's kind of one of the reasons we put our hand up for the like you know private preview testing as well. Um So, in this one like you know what follows is what we tested and then I'm going to explain what worked, what are
[22:43] the gaps we identified in terms of like you know the private preview testing as well part of the POC. So, imagine it's a like you know 3:00 a.m. source breached one of their interface
[22:59] spec contract and then a bad data came through to the um bronze and then the silver. Then operations team going to like, you know, decide between should I do the full refresh or the restore per table. And then if you need to go with the
[23:14] restore per table option, you need to coordinate, look at your like checkpoints and then what's your downstream impact, what's the lineage and then all those kind of things. Uh this probably going to take hours. So with the rewind and replay, what we
[23:30] noticed is like, you know, when you give the you known like, you know, good data timestamp to the SDP pipeline and then trigger that one, it will rewind and replay and then reprocess your data and then bring it up to the current current state.
[23:51] So this is one of the we created this dummy table customer dim just for this demo. So this API call, we know the rewind timestamp and you know, the known best correct data. And then in identify as Spartan mentioned, you can mention one table name or if you don't mention
[24:07] anything, it's going to like, you know, rewind and replay for all the tables in the in in a particular pipeline. Uh if you mention cascade true, it's going to look at your downstream and then like, you know, try to fix the data as well. Um
[24:31] Okay, so as a part of the POC, this is we simulated this scenario. We triggered the SDP pipeline for a day one and day two data. And then on day three, we introduced a bug like, you know, on a day three sorry, on on a day three,
[24:46] we introduced a bug in the day two data. So and then we triggered the SDP pipeline rewind and replay as of a day one. And so this one restored the time it back to like you know day one and then reprocessed the data.
[25:02] So if you look at the results your table version number 38 row number four that's that's your like you know day one version which is a known correct data and then the day two is your version 39 which has a bug like you know
[25:19] issues and then when we run the restore function it brought it back to the version back to 38 which is a known like you know good version.
[25:40] Okay, so as a part of this POC we got a couple of findings. So it works all the scenarios where you have a people reading like a consumers reading from the tables and then also materialize views and then also if you're reading from
[25:57] SDP silver to the SDP gold or non SDP gold which is like you know nothing but your target table it it works for the materialize views enzyme will taking care of that link incremental refresh you don't need to worry about
[26:13] it. So the CDF couple flows where you are reading from the CDF in our case which is a from bronze to the silver because we completely like both are on SDP we read from the CDF that's where we identify a
[26:30] gap. So So we provided the feedback to the data bricks to work on that one so that we can start using this as end to end
[26:46] like you know functionality once once this gap is addressed.
[27:01] Okay, so what's uh uh next? So, we want to like, you know, extend the declarative patterns fully to the gold as well, so that we can run end-to-end on on SDP via bronze, silver, and uh gold. And that kind of it gives 100% adoption, like, you know, uh for our business and provide some self-service uh capability.
[27:18] And then also, once this adopt skip offset feature is available in the rewind and replay, we can probably uh start testing and then, like, you know, working closely with the operations team to see how we can standardize this whole um recovery option.
[27:34] Uh so, So, as uh you guys see in, like, you know, this uh space is evolving. Uh so, this morning in the keynote, there's a
[27:49] Genie uh like, you know, zero ops and all those things. We're going to continue to, like, you know, uh keep ourself, like, you know, being curious and then open-minded, so that if there any features are coming, we, like, you know, uh see if that fits for the NAP, we can start uh like, you know, using
[28:05] them. Okay. So, the when when we look at the, like, you know, the challenges, like, you know, one of the the the operational reality is,
[28:21] like, you know, uh the pipeline of uh sorry, operational reality of the pipeline hasn't gone away. So, what's changed is what we can do about it. So, for the NAP specifically, the key difference is your recovery process every, like, you know, uh
[28:37] custom script for incident to a standard operating procedure using this uh key functionality. And as I mentioned, the 650 lines of code, like, you know, which we as a custom code we built uh in the absence of this functionality. Hopefully, if it's a GA after all
[28:52] tested, we can retire that code. Okay, we got a few couple of more NAB breakout sessions. If you are keen to attend, that's happening tomorrow. But
[29:09] thank you all for your time today, and then let me hand over to Partha for the final remarks. That's exciting, right? 650 lines of code with just one API. Now you can use streaming rewind streaming replay API to
[29:24] recover your streaming pipeline seamlessly. Um And one note, obviously or a key takeaway, the streaming replay or rewind is in a private preview. So, if you'd like to try it out
[29:40] and give feedback, reach out to your account team or let us know and we can help you get that set it up. And yeah, please do provide feedback. Um this helps us on what we bring every year at the dice.
[29:56] Thank you, everyone. Thank you for your time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.