Skip to main content

What is disruptive about Lakeflow Declarative Pipelines? - Data Engineering with Databricks

Summary

  • Declarative programming approach: Simplify ETL by declaring what data transformations you want rather than writing procedural code to handle edge cases.
  • Change data capture simplified: Declarative CDC automatically handles common pitfalls like out-of-order updates and duplicate keys without complex logic.
  • Built-in reliability and quality: Lakeflow provides automated reliability patterns and data quality checks, eliminating common procedural errors.

What is disruptive about Lakeflow Declarative Pipelines? - Data Engineering with Databricks

Watch: What is disruptive about Lakeflow Declarative Pipelines? - Data Engineering with Databricks
Lakeflow Declarative Pipelines helps you simplify batch and streaming ETL with automated reliability and built-in data quality.
Watch this video to learn more about Lakeflow Declarative Pipelines for efficient ingestion, intelligent data transformation, and automated operations.
Additional resources:

Chapters

FAQs

What is declarative programming in Lakeflow?

Declarative programming means you specify what transformations you want (clean, enrich, filter data) rather than writing the procedural code to handle all edge cases.

Why are procedural change data capture (CDC) solutions complex?

Procedural CDC solutions must handle pitfalls like multiple updates to the same key arriving out of order, requiring complex logic and aggregations to ensure correct final values.

How do declarative pipelines like Lakeflow simplify data engineering and reduce maintenance?

Declarative pipeline frameworks like Lakeflow enable data engineers to describe the desired data transformations using SQL and configuration rather than writing imperative orchestration code. The platform automatically manages scheduling, dependency resolution, error handling, and optimization, eliminating thousands of lines of infrastructure code. This approach dramatically reduces development time for new pipelines, makes pipelines easier to understand and maintain, and allows data engineers to focus on business logic rather than orchestration plumbing.

How does Lakeflow solve CDC challenges?

Lakeflow uses declarative programming to automatically handle CDC logic, including managing out-of-order updates, duplicate keys, and selecting the latest value for each key.

Full transcript

0:00 Introduction to LakeFlow

Pipelines. And the reason it does this is it simplifies the process of cleaning, enriching, filtering, and transforming data so that it's ready for consumption downstream. Behind this though is this key idea of declarative programming. And I want to talk about what makes this idea so important. And in order to do that, we're going to look at an example use case that's very common in data engineering called change data capture.

0:30 Change data capture overview

The goal sounds really simple. We're going to update a delta. We're going to take a change feed uh from upstream that says how data is changing and we want to update a delta table with the latest value for every key. It sounds simple, but there's a bunch of common pitfalls when there's multiple updates to the same key and they might arrive out of order. And so, while it seems easy at first, in reality, it can be difficult to write a program that's going to handle all of these cases.
Let's take a look at it. So, to recap, basically we're going to start here with this change feed that has a bunch of different rows that are coming in. There's a primary key of the ID. There's a time stamp that says when this change happens, and then there's a bunch of columns that are the actual data that we're wanting to process. In this case, you can see we're going to take the two IDs, one, one, and two, and create the final table that has the latest value for every key.
So, if you're an expert in structured streaming, you'll say, "This sounds very easy. I'm going to use the merge command, which allows us to take those updates and apply them to an existing delta table. I'm going to wrap it up in a function and I'm going to use structured streaming to call that function every time new data arrives. Here's a really simple example that you can find on the internet and it looks pretty easy at 14 lines of code.

1:30 Procedural CDC pitfalls

But there's actually a bunch of problems with this under the covers. One problem is this code doesn't handle the case where multiple updates come to the same key in a single micro batch. The system when it sees these doesn't know which one to pick. So we're going to need to fix that. So the solution is pretty simple. We're going to instead of just taking all of the data, we're going to run an aggregation that uses max by to pick the largest time stamp for every key.
So we'll group by that key, take all of the columns, pick the maximum one for the for any given time stamp, and then explode back out that results that we end up with the final row that we're looking for. So we add this into the code. Now, it's a little bit more complicated. We're up to 18 lines of code. But unfortunately, there's still some problems with it. Another problem happens if rows arrive out of order, but they're in different microbatches.
So, it arrives so late that it's being processed in the next execution of the stream. So, as you can see, we've got this this problem happening here. And in order to handle this, we're going to have to do something a little bit more complicated. The solution is we're actually going to take the timestamp from the change and store it in the table so we can remember when each individual row is updated. And so now when we're doing that merge operation, for every new entry in the change feed, we'll have to check to see if the timestamp is greater than the one that's stored in the table.
If it's greater, we'll use that new data from the source. If it's not, we'll keep the existing data in the target. So we'll modify our program again. We can see it's getting a little bit more complicated. And now we're having to write logic for every individual column to make sure that it's handled correctly. We're now up to over 22 lines of code. Unfortunately, it's still not complete. Another really common thing that you need to handle is deletes that happen upstream.
And so in this case, we'll add another column to the change feed that indicates whether or not it's indicating a deletion is happening upstream. We'll add that in. And so pretty simple. Now we're at 23 lines of code. But now we run into one of the most subtle problems. What happens if an update arrives late after a delete? We no longer have that time stamp that we stored because we actually deleted the row from the table.
And the result is a zombie row, meaning a row still lives in the table even though it's supposed to have been removed. And the solution starts to get pretty complicated. We're now going to actually have to keep tombstones in the table that remind us that this row was deleted recently. So instead of deleting a row now, we'll softdee it and we'll store a timestamp in the table of when that row was deleted. We don't want our users to see these tombstones.
We'll have to redirect all of our downstream consumers to read that data out of a view that filters out these tombstones. And we don't want these tombstones to accumulate forever. So, we'll build a cleanup process that removes them over time. So, a couple more lines of code, a view that we have to redirect all of our readers to, and this job that will clean up these tombstones periodically. So, now we're at 24 plus lines of code, a view, and a cleanup job that we have to manage just to handle this simple use case.
And so this leads us to ask, there's got to be an easier way to accomplish that. And this is where declarative programming comes in. In declarative pipelines, we have declarative change data capture. With one single SQL command, we handle all of the different cases that can happen. And you'll notice here, you're just telling the system what needs to happen instead of how to handle all of those individual cases. We take a change feed from source data.

4:44 Declarative CDC solution

We apply it into CDC data. We tell it what the primary key is and how to order them and how to know about deletes. and the system takes care of the rest. So what you'll notice is writing robust logic for failures and out of order data can be very difficult. But declarative programming makes this easy by separating the how from the what should happen and lakeflow declarative pipelines has this and many other common ETL patterns and it makes them all declarative so you can focus on getting value out of the data for your business.

5:15 Conclusion and recap

So to recap, LakeFlow declarative pipelines help simplify ETL by making it declarative. LakeFlow provides simplified authoring, intelligent operations at scale, and unified streaming in batch so you can meet all of your cost and latency goals. The key features here help you solve your toughest data challenges. If you'd like to learn more, I encourage you to check out our website. Thanks for tuning in.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.