Skip to main content

Modernizing Retail CDC: From Batch to Real-Time with Lakeflow

Summary

  • Asda, a top-three UK grocer, replaced a batch CDC pipeline that delivered only the previous day's depot data after a 7-hour midnight load with Lakeflow Connect, ingesting 450-plus tables from the Manhattan warehouse management system in just four days following a three-day proof of concept.
  • Lakeflow Connect eliminated the need for separate VM-hosted CDC infrastructure, with Spark Declarative Pipelines' Auto CDC resolving changes in the silver layer and Unity Catalog providing governed access to real-time data.
  • The video covers the evaluation that compared vendor tools, an in-house build, and Lakeflow Connect, the lessons learned from a Debezium prototype around VM checkpointing, and how Lakeflow replaced an estimated three months of custom development work.

Modernizing Retail CDC: From Batch to Real-Time with Lakeflow

Watch: Modernizing Retail CDC: From Batch to Real-Time with Lakeflow
Real-time data is critical for retail operations, but legacy batch CDC pipelines delay decision-making. Asda, UK's top-3 grocer, inherited a complex batch pipeline pulling Manhattan depot data, with analysts seeing only yesterday's data after a 7-hour midnight load. When Manhattan moved to replicated MySQL, Asda evaluated options: expensive vendors or risky in-house builds. After a three-day Lakeflow POC, they pivoted and ingested 450+ tables, including 150GB snapshots, in just four days.
Cassidy Louch and Marius Panga explain why Debezium taught CDC lessons but Lakeflow's managed service eliminated operational overhead. Learn how Lakeflow's gateway eliminated VMs, Spark Declarative Pipelines' Auto CDC resolved changes, real-time ingestion feeds Unity Catalog with proper naming, and how Lakeflow replaced three months of custom work while handling 100GB daily volumes.
🤝

Chapters

FAQs

What is Lakeflow Connect and what does it do?

Lakeflow Connect is a managed data ingestion service within the Databricks Data and AI platform that enables real-time Change Data Capture from source databases without requiring teams to build and maintain their own CDC infrastructure. It provides a managed gateway that connects to source systems directly, eliminating the need for VM-hosted tools like Debezium.

Why did Asda choose Lakeflow Connect over other CDC options?

Asda evaluated specialized vendor tools, an in-house build, and Lakeflow Connect after their Debezium prototype encountered VM checkpointing issues requiring manual intervention. A three-day Lakeflow proof of concept successfully ingested their Manhattan system tables, demonstrating it could eliminate CDC infrastructure overhead while providing a scalable managed solution.

How quickly did Asda migrate their Manhattan warehouse data using Lakeflow Connect?

After a three-day proof of concept, Asda ingested 450-plus tables from their Manhattan warehouse management system — including 150GB snapshots — in just four production days. This was a significant acceleration compared to an estimated three months that an in-house CDC build would have required.

How does Lakeflow Connect work with Spark Declarative Pipelines for end-to-end CDC?

Lakeflow Connect handles the ingestion layer by capturing raw change events from the source database into Databricks. Spark Declarative Pipelines with Auto CDC then process these events in the silver layer, applying inserts, updates, and deletes to maintain accurate up-to-date tables that analysts and downstream applications query through Unity Catalog.

Full transcript

[00:08] Hello everyone. Thank you for joining us. Uh I am Marius. I'm a solutions architect with Databricks from the UK. And I'm joined by Cassidy, lead data engineer at Asda, one of our largest UK customers. Now, it's always hard to go immediately after the keynote, especially at Databricks, and especially a keynote
[00:24] that had Matei in it. So, please bear with us. We will do our best. And now, the chat today the talk is around Lake Flow and specifically around Lake Flow Connect. And the way we've structured it is I'm just going to give you an overview reminder of what Lake
[00:40] Flow Connect is, why should you care, and where are we with it right now. And after that, Cassie is going to take the stage and he's going to give you an idea of the journey of a large customer, specifically Asda, top three retailer grocer in the UK, how they managed to
[00:57] deploy Lake Flow Connect on top of one of their main data sources, Manhattan. So, we thought it would be quite valuable for you to just see that through a customer's eyes, see some of the challenges, frustrations, decision points, and wins with with uh
[01:12] with Lake Flow Connect. Now, just one bit of housekeeping. After every session at Summit, you will get a survey notification. Uh we would be very grateful if you managed to take a a minute and just uh fill it in cuz it helps us to just
[01:27] prioritize the the quality content going forward. Okay. So, what is Lake Flow Connect? Or maybe more important, why? Why Lake Flow Connect? So, over the last two days, especially during the the keynotes, you've heard about a lot of innovation coming to Databricks, things like Genie
[01:44] One, enhanced semantics and metric views, the really coolly named Raiden engine that's coming. But, one thing that all these capabilities have in common is you need to have the data. So, the data needs to be in Databricks, otherwise they're not extremely useful
[01:59] to your organization. And before Lakeflow Connect, Databricks didn't really provide you an easy service to get data in. So, what customers normally did is you either take a specialized tool off the shelf and just use that to get data in the platform or just build a
[02:14] custom solution. And that's fine. That works, but some of the challenges are this can be quite inefficient, especially as the data variety and volumes keep growing. You might end up with multiple different tools for the various sources you have and teams that need to maintain those
[02:30] tools and enhance them and support them, sometimes inside your org, sometimes external. And also the governance gets quite uh fragmented. Stuff doesn't just flow through Unity Catalog in a in a seamless way. So, the hero of this this narrative and
[02:46] the story is Lakeflow. So, what is Lakeflow? Lakeflow is our solution for data engineering and wrangling, and it has three main pillars. So, we have Lakeflow Connect, which is the way to actually ingest data from a variety of data sources.
[03:02] And this this can either be code-based or it can be UI-driven, where you just select everything you need, configure it, and just run it. Then we have Spark Declarative Pipelines, SDP, previously known as DLT, which is the data wrangling, the ETL
[03:18] engine. And there you can either write your code in a declarative fashion, either in Python or SQL, or again, if you prefer a more like Alteryx-like interface, where you drag and drop stuff, we have Lakeflow Designer that allows you to just drag and drop your transformations, but everything still
[03:34] gets converted to Spark Declarative Pipelines. And the last, the third uh pillar there is jobs. That's where your orchestration lies, where you set up the sequence of the various activities you want to do. And again, just like before, you can do that in a code fashion or you can do
[03:51] that using just pure drag and drop. Now, why Lakehouse Connect? Why does this matter? Well, it's really simple. It's a managed service. If you go via the UI, all you need to do is specify your data source, specify the
[04:07] credentials, the schema or database, the tables you want to ingest, and then on the destination on the Unity Catalog side, you specify a catalog, a schema, all the tables get created, and then they keep they get kept in sync in a in a incremental fashion. So, really easy
[04:23] to set up. Because it's already a Databricks product, it will be unified. So, it's already under Unity Catalog. You will get the access control, the lineage, the auditing, all that is there. And it is quite efficient cuz it's powered by Spark Serverless. So, it will scale as
[04:39] your data scales and gets bigger. So, if we go back to that original architecture, that's where Lakehouse Connect sits. It just simply hides the whole thing. It unifies it, and it allows you to get this data quicker into your platform.
[04:55] Now, with Lakehouse Connect, we've been on a bit of a journey as well, where initially about uh 2 years ago, we launched it, but we only had a couple of data sources, very popular ones, but just a few. So, for customers that
[05:10] really like the approach and the vision, but had a data source they really wanted to ingest and wasn't supported, that was quite frustrating. And I'm sure Cass can can tell you more about that. Now, over the last 2 years, that has changed a lot. I feel like we hit critical mass
[05:25] both in terms of what we support and the development velocity. I put this slide in before the summit. I knew it was going to be out of date quickly. Yesterday at the keynote, there was an updated version of this, and you had a lot more data sources supported.
[05:42] But I'm going to highlight there's quite a few pillars there. So, we support SaaS applications like Workday or Salesforce, file sources, like SFTP or ADLS, databases, most of them SQL Server, Oracle, whatnot, and then also all the main warehouses as well.
[05:58] Now, if you have a data source you're considering ingesting, it's very likely it's already supported. If not, we're working on quite a few in parallel, so we might already be working on them, or they might be in the pipeline. And if that's still not the case, we provide
[06:13] tools for the community to build those sources, as well as you can build it yourself. So, it's very likely that that Lake Formation can can have you covered. And with that out of the way, I'm just going to hand it over to Cass for the the main part of the presentation. Perfect. Thank you so much, Marius.
[06:31] So, what is Asda? So, we are top three retailer in the UK. I'll chuck some details up here. We hire and employ 150,000 employees in the UK across across the whole country. We
[06:46] we have 18 million customers that shop with us each week, so we've got lots of data coming in and out. We provide a whole host of life services, such as insurances, credit cards, and a whole host of other things as well.
[07:01] You can see in cases and depots, so we have 27 depots that we use to ship stock from depots to store. Then we have some trunking depots as well, which are used to to move stock between depots to make sure that we're fully stocked. And mainly for this talk today, we'll be
[07:18] focusing on the depots. So, let me move forward. So, what we're covering today is our journey with ingesting Manhattan data. So, we use Manhattan for our depot management services. So, they manage all the
[07:34] information, all the IoT, colleague movement, stock movement, all within the depots. So, that's everything to do with stock in and out, um logistics as well, um incoming and outgoing um
[07:50] logistics. Um and we've got geofencing as well. So, um the thing with Manhattan is because we're we're capturing such a large vast of processes, we we have tables which are really high cardinality, really high volume such as task tables. So, they by
[08:07] by design capture everything that's happening in the depot. They're millions upon millions upon millions of of rows. And then we have some smaller tables as well which are for an example, um how many depots do we currently have in the organization? Which would be 27,
[08:23] very rarely updates. And and we um have a whole host of data um of tables that that sit between them two at extremes. So. I inherited an existing solution for Manhattan. So, we previously were
[08:38] bringing in um Manhattan before I joined the squad, um we were using um a Google Pub/Sub endpoint that Manhattan provide. Um and then we were pulling data into the Databricks um via that medium. So, again, this is the um not so pretty
[08:55] um project that I inherited. So, there was a bit of tech debt here. Um I was uh tasked to maintain it and also come up with a way to improve it. So, um from left to right, we are pulling the data in initially um from the streaming endpoint in 70-minute batches. Um we're
[09:11] saving those 70-minute chunks um throughout the day. Um we have a a follow-on job which then would split our streaming messages um by source table change. Um we then have a midnight load which then attempts to load all of the tables and then
[09:28] ultimately we sync it into Unity Catalog. Now, this is quite a behemoth within Asda. Um there's a lot to manage here. It looks quite simple here, but when we start digging into the um the code, there's a lot to manage. So, there's a couple of issues with this when I came into the squad that we initially had to
[09:44] highlight to kind of figure out what can we do with this. So, we're essentially losing all the advantages of a streaming process because we're turning it into a batch process by reading it in in 70-minute chunks. We have loads of code code bloat. This is a custom framework within
[09:59] side Astra, and it's very hard to manage. Very little people know how to use it. They you could probably say there's some outdated practices. This solution was made 3 years ago. And again was had some I probably could say questionable approaches to handling some problems.
[10:17] The midnight load, we're collecting a whole day's worth of depot data, which like I mentioned, there's a lot of it. That midnight load is taking about 7 hours. So, we essentially never have fresh data in the platform. The midnight load
[10:33] orchestrates at midnight. It finishes around 7:00 8:00 o'clock in the morning, and that's only loading the previous day's data. So, our analysts are always screaming saying, "We need in-day data." Especially when it concerns logistics, current stock in depots and warehouses, what's our
[10:48] availability like for sending stock to stores. And in general, it it it provided data, but we were losing out on so many so many benefits which we were wanting to make use of with the Manhattan data.
[11:03] The The other frustrating part of this process is that we also don't make use of unit catalogs three-part naming convention. So, everything is sunk into a single catalog into a single schema. Again, very unusable for us internally
[11:19] in engineering, and then also the analysts are having to deal with horrible naming conventions. So, the naming convention currently for the for the legacy solution is source system schema {{}underscore} table name, which is a nightmare to have to deal with. So, let me
[11:35] move forward. So, we've got the old solution and on top of that it's not very healthy, needs some maintenance, and we're probably looking at replacing it sometime soon. The way that we integrate with Manhattan going forward is is also going to change. So, they put a polite request in quite a while ago that they
[11:52] no longer want to use Pub/Sub to provide us the data that we need. They want to provide us with a replicated MySQL database so that we can extract the data however we want. This already as a project is start to grow legs. We've got to maintain the old thing,
[12:08] decide if we want to replace it, and then also it's essentially a fresh greenfield project as well because the way that we get the data is fundamentally changing. To top this off, we don't have any large-scale CDC pipelines in Azure either. So, this is all uncharted territory for the
[12:23] engineering squad. I say squad, this is just me on this at the time. So, we've been cracking through the old solution. We've kind of decided as a squad and as a business that the the legacy solution is is not fit for purpose anymore. We want to take
[12:39] advantage of in-day updates and be able to give the analysts the data that they need to be able to provide the reports to the to the business. So, we started by evaluating what our solutions are internally. So,
[12:55] like like most solutions, there's a vendor option. There's there's pros and cons to vendors. They obviously sometimes cost a little bit more. There's little to no engineering in terms of what we have to do. And also we get a really nice service
[13:10] wrapper often with with vendors. And it would be for all intensive purposes like a hands-off. We pay you guys to sort out all of our Manhattan ingestions. This was somewhat unfeasible given our tight deadlines. Forgot to mention we
[13:25] had a very tight deadline for this. We wanted around a 4-month turnaround, um and we'd already spent a lot of that deciding if we needed to replace in the first place. So, the to onboard a new vendor for us was going to essentially take us over the deadline to be able to actually get them
[13:40] integrated. Um so, uh the vendor option was shut down quite early on. We then thought we could make something in-house and leverage some open-source tooling or make something that's very similar to
[13:55] the to legacy solution, but make it fit for purpose. So, um the tooling that we decided we were going to use if we were going to go for an in-house um approach was using something called Debezium. So, um if anyone doesn't know what Debezium is, it essentially is a piece of software that um you have a
[14:12] source and a sink. Um it will um interpret the bin log of a database and then just sync the messages um and the data to wherever you want to sync it to. So, Debezium would be a little bit more work Well, a lot more
[14:28] work than a vendor. Um it would require us to essentially be um creating any features that we wanted. We would be a very It'd be a very lean product, especially with the deadlines that we had. And um yeah, in general, we'd be starting starting afresh. Um
[14:44] the other option at this time was Lake Flow Connect. So, quite kindly, Marius had been talking to us um about Lake Flow, this fantastic new tool that's coming around that might fit our purposes um for our CDC pipeline. Now, Lake Flow looked new and shiny, and it was definitely very new at the time.
[15:00] Um the problem we had with Lake Flow at the time was we needed the MySQL connector, which at the time was in private preview, and due to our internal policy, we weren't allowed to push anything to bridge production that depended on private preview.
[15:16] Um so, we were kind of waiting for the guys at Databricks to release the connector. Now, quite rightly, they didn't have too much of a time scale for us, but um that didn't really it wasn't going to give us what we wanted. We we essentially LakeFlow was there and it was not fit um fit for
[15:34] um what we needed at the time based on the connectors. So, ultimately, we had to go with Debezium or an in in in-house um engineering solution. So, that's ultimately what we decided to go with. So, um some of you guys might laugh. We just
[15:51] needed to create a diagram that just explained the flow. Um this is nothing fancy. We literally just threw this up in draw.io really quickly just to get an idea of what the flow would look like for our new solution using Debezium. So, from left to right, we have the Manhattan source. We would
[16:07] sit Debezium server on a VM. Debezium would look at the bin log um and it's always on, so it's always looking at the bin log collecting data, and then we would sync that into Event Hubs. Once the data's in Event Hubs, we would then uh consume it um in Databricks
[16:24] using the um they have a Kafka endpoint in Event Hubs, and Databricks can natively connect to a Kafka. So, um and again, the structure within Databricks is we would sync all of our um messages, all of the changes for all the tables that we're receiving into a
[16:41] one big table. Internally, we call this the OBT. And then we would do a per table split and then um load um on the far right there. So, this is a very crude diagram. We essentially start cracking on with
[16:56] the Debezium solution um and a few weeks fly by. We had some headaches with a few different things, um and we started to realize how much code we're having to write to create this solution from scratch. But ultimately, a big part of it was on me as well. I'd never created a CDC pipeline before. I was learning about
[17:13] CDC for the first time. Um I was having a lot of fun doing it, but until this point I was scratching my head a lot. So, this piece of code here is what you what we use to essentially ingest the data into Databricks. So, this is the the kind of eureka moment for the Debezium solution. Um so, this essentially is
[17:30] just using the the Kafka connector and we're connecting to the um the endpoint in Event Hubs. Um this is a really cool piece of code. I really enjoyed it. As an engineer, when you first get data into the platform um and everything starts working as you plan it, um it's a yeah, it's a big weight off the shoulders. Um data is
[17:46] very raw, so we're seeing lots of insert, update, and deletes. Um very very user-friendly data to use, but nonetheless, we had pure bronze data into the platform at this point. So, we've been working with Debezium for a
[18:02] little bit. We we we progressed a few more weeks. We were validating the bronze layers, and we were starting to get a feeling for how the solution was going through. And we had one issue that was that that that kept coming up. So, this was um because Debezium is hosted on a VM,
[18:18] we didn't have capacity to be able to spin up infrastructure to make the checkpointing for the VM resilient to a critical failure. So, in case where the VM would drop, um we would lose the checkpointing, and then we would lose we'd we'd we'd lead to data loss. So, um nonetheless, we've been working with
[18:34] the Debezium solution. Everything else is working as we intended, um but the VM issue kept coming up. So, essentially, as a business, we decided we were going to accept the risk risk due to the deadlines, and then we'd re-evaluate in 6 to 12 months to see if we could pivot to Lake Flow or whatever tooling was available at the time.
[18:56] So, again, we have the issue in the back um with the VMs and and a couple of other headaches as well, which I won't mention. Um but, we start progressing through to the silver layer. So, um data is in the platform. We've got over 246 tables in the bronze layer, and then we're starting to build out the CDC
[19:11] resolution. So, we're applying the updates, inserts, and deletes over a given sequence to make sure that the data behaves as it's supposed to and it and it's identical to the source system. So, we used part of Spark declarative pipelines a
[19:27] function called Auto CDC. Now, Auto CDC is really cool. I like things that abstract complex logic and essentially allow us to make fast progress. When I found out about Auto Auto CDC, I kind of didn't believe that it would
[19:42] do what I wanted to do quite quickly, but essentially you pass in the data that you want, you chuck in what it defines as a delete from the event, and you also define a given sequence number. So, you could this could be an event date time in the database, it could be the bin log sequence or the true
[19:58] sequence, and then you can also specify the SCD type as well. For us, this this is a very quick turnaround on getting data into the silver layer and then also having something that's usable. Yeah, I must got very excited at this point. The data is essentially behaving
[20:13] as the source system despite the VM issues. So, we carry on progressing and then annoyingly, Marius has an update on Lake Flow. So, we progress for a few months making good headway and the guys at
[20:29] Databricks say, "So, that MySQL connector that you guys are really interested in that you essentially written off the board is available and you guys can use it if you'd like to." And at this point, it was like we were we were bummed out as a squad because
[20:44] we'd committed so much time to the existing solution and then these guys have an update. So, nonetheless, we entertained a meeting with them and they walked us through a Databricks asset bundle which contained everything needed to do a bit of a small test for a
[21:00] Lake Flow Connect pipeline that was using the MySQL connector. So, again, as an engineer, I was really kind of disappointed that we couldn't use Lake Flow. The steer from the business at the time is um when we had the meeting with Databricks was, "I know that Lake Flow
[21:17] looks shiny, and I know you really want to use it, but ultimately we've sunk so much time into Debezium that it would be too much of a risk at this point if it falls over to um bin what we already have um and pivot to Lake Flow." So, again, very disappointed.
[21:32] So, I, like I said, very passionate about at least having a go with Lake Flow. And um we were waiting for some modeling to come back for the silver layer for the for the Debezium solution. Um so, I kind of thought, "Can I squeeze
[21:48] in a quick POC of Lake Flow to see how it goes?" Um and this is what we got. So, this is a this is a small snippet of what Lake Flow looks like in terms of config. So, essentially, you specify pipeline name.
[22:03] You specify a gateway. So, there's there's two parts to Lake Flow. You have a gateway that's connected to the source database, which is pulling data for you. And then you have the pipeline, which actually does all the heavy lifting. So, um you specify the source type, which is the connector. And then in the objects um
[22:20] uh key that you can see there, you can specify what you want to pull. So, there I in this example, I've got a single table I'm pulling through. So, you specify where the table is, um what schema it lives in, and then also what table you want to pull back. Um and then also you pass in some details about where you want it to sync
[22:36] to in Unity Catalog as well. Um and essentially, you can just deploy the pipeline under DABS. Um and all being well, it should work. So, we've got the code from Marius and team. We've got the okay to enable it in the workspace. Um but then we have this
[22:53] quite progressed solution which the business is saying, "Please just deliver what you said you wanted to deliver." So, I decided with 3 days at the end of a sprint that was quite quiet because we were waiting for architecture to come back with a result for something that I was going to have a go to see how much
[23:09] headway I could make with LakeFlow. So, um after three short days of working with LakeFlow this is what we got. So, we had managed to ingest in three days all 246 tables using LakeFlow.
[23:27] In that three days we'd completely surpassed where I'd got to using Debezium. So, in the same step we're ingesting the data, we're resolving the CDC, and then we're outputting the data to a well-structured Unity Catalog. Um if I'm honest, I was blown away with the
[23:44] amount of work that we managed to get done in LakeFlow. As you can see from the GIF, um there's a lot of tables in there. Um all the data comes in via the first staging table from the gateway, and then it populates out based on the table object um that you define in the LakeFlow config. So,
[24:01] on top of this, when we were developing the Debezium solution, we got an additional request from the business. They said, "The 246 tables that you're ingesting, we do need that, but can you also ingest another critical 200 tables that you had no idea about, and the deadline for that is actually before the
[24:17] original ask?" Um um and we were like, "No, we can't do that." Um so, we had some very angry stakeholders um trying to squeeze in a request. One day extra worth of work, I managed to also fulfill our additional request.
[24:33] So, in total, using LakeFlow in four days, we'd managed to get in um just over 450 tables. They're not small tables. We have some tables which are over 150 GB on snapshot. Um we've got um I think around 70 of them receive
[24:49] millions of updates every single day, and the rest of them um are lower volumes. But nonetheless, we receive about 100 GB of data daily through Manhattan. So, to be able to churn out this much work in such a short amount of time and also surpass what we'd already built, we were really impressed with. Um
[25:07] So, at that point, we'd surpassed where we got to with the Debezium. We're pivoting to Lake Flow. I taken my liberties and kind of gone off piece a little bit and decided I was going to run with it. And then, as a squad, when I pitched Lake Flow back, we decided we're going to commit to Lake Flow finally, even after sinking 2 to 3
[25:24] months worth of work using the Debezium. So, the main thing that that really helped Lake Flow for us at the moment is that we didn't have any VM overhead to have to handle. That's all handled by Databricks. Um the gateway um essentially stands as the Debezium that for the same section that we were going
[25:39] to use. Um it removes Event Hubs out of the equation. And then also, we don't have an open source source tooling to have to maintain um and update and and and and keep track of. Um date Lake Flow is also really feature-rich um as it as it came out.
[25:55] So, we have per table re-replayability. We've got um classic like um schema migration, not schema migration. Um Oh, what's the word? Um so, we've got um per table um
[26:10] CDC CDC SCD type 2. Um and then, it's also real-time ready out of the box as well. So, um we'd managed to squish all of the main issues with the legacy um legacy solution. And then, we'd also managed to add a load of nice-to-haves
[26:26] that would have taken a long time for us to iterate over for Debezium. On top of that, it's also Dabs native, so I didn't really have to write much Python at all. Um it's all mainly YAML config, and it and it kind of fit with our current CI/CD stack as well. So, we enabled Lake Flow in the work in the workspace, and there was basically no
[26:42] more overhead to be able to get it working. We didn't have to do anything to Unity. We didn't have to kind of change anything. It was ready to go out of the box um in our case. So, that being said, this is our Lake Flow structure. So, I've got an example here. So, I've got a
[26:59] schema that we ingest called default pick pack. This is one of the biggest tables from Manhattan. Tables task task or TSK task. Um and that essentially tracks, like I said before, everything that happens in the depot. It's a massive table. Um we are continuously polling the uh
[27:16] database for changes. We're ingesting things in real time. We're resolving the CDC and then we're outputting it into a really nice structured Unity Catalog. So, we decided to name um the catalog Manhattan. We name the schema that the table comes from the
[27:31] schema in Unity Catalog and then we also keep we retain the source table name as well. So, what this meant for our analysts um is that when they look in the catalog, the structure in Unity is the same as the source system. So, they have a source system they have access to which
[27:47] they can't really use too hard because it is the production system for Manhattan. Everything looks and behaves similar. So, any of their kind of checks that they want to do against a source system, they essentially can drag and drop the same query against our Unity Catalog. Um this was massive for the guys.
[28:03] Um when I was when I was presenting this back to them um to say that it would be in day data, that it would essentially behave as a source system. This was yeah, they were chuffed. It is the only way I can put it really. Um and yeah, that's essentially our implementation of
[28:19] of uh Lake Flow Connect. The um going forward, we're looking at um adding more um kind of modeling to the to the end state. But as it is, we're dropping the data as it is to the analyst because that's how they've used it in the past to to to save any disruption. Um and
[28:34] this is in production. This uh this I delivered this um and it's working as expected. Um we've got um we're currently on a non-real-time um update cycle at the moment just to see how costs are going. So, everything, all 400 450 plus tables are updating every 15
[28:52] minutes. Um and yeah, it's a joy to see all them tables in the catalog. I'm sure you guys share the same passion that when you see everything in there working as expected, all the green ticks on the pipeline, it's really satisfying. So, all the time that I spent on the Debezium solution, you'll probably say
[29:09] is wasted time. But, for someone like me who'd never created a CDC pipeline before, the learnings and and and understanding how something worked from the inside, having to define my sequence columns, having to define um the events, understanding the different database events that are logged into the bin log,
[29:25] allowed me to to be able to confidently talk with Marius and squad to be able to ask the right questions to allow us to pivot to Lake Flow con- so confidently. If I hadn't have learned and and struggled with the Debezium solution initially to figure out how the best practices for CDC work, um we wouldn't
[29:41] have been able to in 4 days say we're going to go with Lake Flow. So, um yeah, that's me.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.