Skip to main content

Building reliable ETL pipelines with built-in observability - Data Engineering with Databricks

Summary

  • Lakeflow observability: End-to-end monitoring for ETL pipelines built into Lakeflow Jobs enables data engineers to diagnose bottlenecks, optimize performance, and manage resource usage in one place.
  • Monitoring layers: This video covers challenges including complex pipeline management, error detection with root cause analysis, and optimization at scale without risk to data quality.
  • Integrated solution: Native observability features across data ingestion, transformation, and orchestration let teams stay in control of their data and pipelines with native monitoring built into the data engineering platform.

Building reliable ETL pipelines with built-in observability - Data Engineering with Databricks

Watch: Building reliable ETL pipelines with built-in observability - Data Engineering with Databricks
As a data engineer, you bear the heavy responsibility of ensuring that the data pipelines you are building are healthy, reliable and efficient, enabling trustworthy downstream analytics and ML.
With Databricks Lakeflow, data engineers can enjoy a rich set of integrated observability features across data ingestion, transformation and orchestration so they can diagnose bottlenecks, optimize performance, and better manage resource usage and costs, in one single place. When you have access to end-to-end monitoring, you stay in control of your data and your pipelines.
Watch this video to learn about observability in Lakeflow Jobs and the latest tools we offer for pipeline monitoring, debugging and optimization.
Additional resources:

Chapters

FAQs

What observability challenges does Lakeflow address?

Lakeflow addresses three challenges: managing complex pipelines with scattered logs, diagnosing errors quickly to assess severity, and ensuring pipelines remain efficient at scale. Native observability features solve these by consolidating logs, lineage, and metrics in one place.

How does native observability in Lakeflow improve monitoring?

Native observability features integrate across data ingestion, transformation, and orchestration to provide end-to-end monitoring. This eliminates switching between systems and gives engineers direct access to logs, lineage, and performance metrics for faster troubleshooting.

What are the key use cases for Lakeflow observability?

Key use cases include discoverability (organizing and accessing jobs), error detection with root cause analysis, and optimization at scale. These capabilities help teams diagnose bottlenecks, optimize performance, and manage resource usage and costs effectively.

What makes Lakeflow different from external tools?

Unlike external orchestration systems that sit outside the lakehouse, Lakeflow is built natively into the Databricks Data and AI platform. This eliminates friction and integration complexity while providing observability designed specifically for lakehouse workflows.

Full transcript

0:00 Introduction

Hi, I'm on the product team here at Data Bricks and welcome to another installment of data engineering with data bricks. I'm going to discuss and demo how to build efficient and reliable data pipelines with native observability in LakeFlow. As a data engineer, you better be a responsibility of ensuring that your data pipelines are healthy, reliable, and efficient so that you can enable trustworthy downstream analytics and machine learning.

0:23 What is observability

This means that you're also accountable for monitoring their performance and resolving any issues should they arise. However, monitoring and keeping everything afloat isn't easy. In fact, most data engineers have the following challenges. Firstly, complex data pipelines, which makes it incredibly difficult to effectively manage and quickly access related jobs and pipelines. As a result, data engineers are not able to properly view the workloads and keep an eye on their performance and making sure everything is running as it should.
Secondly, we have error detection and root cause analysis. When issues arise, it can be difficult to identify that there is a problem, pinpoint the source and also assess its level of severity. This can lead to ineffective monitoring as well as time consuming root cause analysis and troubleshooting. Finally, we have optimization at monitoring at scale. With more data comes more complexity. Ensuring that your data pipelines are consistently efficient and running well becomes more challenging at scale.
Without the right observability capabilities, you risk impacting the quality of your data and your costs. These challenges can be resolved with observability natively integrated in your data engineering tool. something that data bricks can solve today with leg flow. Data bricks lakeflow is a unified intelligence solution for data engineering with streamlined ETL development and operations built on the data intelligence platform.
With LegFlow, data engineers can enjoy a rich set of integrated observability features across data ingestion, transformation, and orchestration. This allows them to diagnose bottlenecks, optimize performance, and better manage resource usage and costs in one single place. When you have access to endtoend monitoring, you stay in control of your data and your pipelines. Observability in native lakeflow consists of different use cases.

2:17 Use cases

To start off, we have discoverability. When we talk about discoverability, we mean making it really easy to organize and access your jobs and pipelines. This is important for proactively monitoring your workloads, managing them, and checking on their status. This can be especially key when you're developing new pipelines or are productionizing them. Then we have monitoring and alerts. Once your workloads are up and running, it is common to rely on a more reactive approach of monitoring.
Regardless of your entry point, this step will help you inform the decision if you need to drill deeper. If you've identified a problem and really need to understand what's happening, we have root cause analysis. This is really about understanding why something went wrong. Whether this is about debugging an error or resolving performance bottlenecks. This will equip you with the right insights to move on to debugging and optimization.
Let's take a look at what this looks like in practice. Let's assume I'm a data engineer and I'm working on data pipelines to ingest and transform sales data for my business. To achieve this, I use LakeFlow Connect to ingest my data from Salesforce and other sources. Lakefully declarative pipelines to aggregate the data and perform data quality checks and lakeflow jobs to orchestrate to make sure the data is always fresh and reliable and business user can rely on the dashboards and reports we produce.

3:15 Demo

Now I want to proactively monitor my project because we've made some recent changes. In order to do this jobs and pipelines is my central entry point from here I can filter for my tag. My project is called SE sales data and this will allow me to see all of the different related jobs and pipelines that are part of my project at one glance as well as key information about them and the recent runs. Here I can immediately see that my sales data processing job had an issue.
So I'm going to jump into this and see what went wrong. In order to do this I'm going to jump into the latest failed job and go to the timeline view. I really like to use this view because it adds a dimension of duration to my regular DAG view. This helps me to both immediately spot when something went wrong as well as identifying performance bottlenecks. In my case, I can see my aggregate sales data pipeline went wrong.
So, I'm going to click into this to understand what happened here. Looking into my pipeline that has failed, I can immediately see that there was an issue with a data quality check failing. If I want to understand this in more detail, I can look at the DAG view to understand which table was impacted. And if I wanted to directly jump into the code and fix the problem, let's assume I've now fixed a problem and I want to make sure I can automate this going forward so I don't have to proactively keep checking on my jobs and pipelines.
I can do this by setting up a notification. To do that, I'm going to jump back into my job and I'm going to find the notification section. Once I have it, I'm going to click add notification. And in this case, failure is already selected. That's exactly what I want to be notified about. Now, the only thing I need to do to complete it is to select a destination here. I can select from an email address or a system destination.
I'm going to select my email address here. And once I'm happy, I'm going to save this. Perfect. What this means is that now I don't have to proactively check on this job, but I will be automatically notified if there's a problem. So this was just one example of showcasing an end toend flow made possible with LakeFlow observability. But there's even more. So we've just seen an example of setting up an alert for a failure, but I could also set up more sophisticated alerts for duration thresholds or streaming backlog.

5:48 Outro

Other than that, I can also build dashboards to visualize observability at scale across my data bricks jobs and pipelines and across workspaces. Finally, LakeFlow allows me to track data lineage whether upstream or downstream with Unity catalog, data bricks unified governance solution. And that's about it. I hope this video helped you learn about observability in LakeFlow. Hear about the latest tools we offer for pipeline monitoring, debugging, and optimization. and see how easy it is to use Lake to help you solve your toughest data engineering challenges.
For more information, check out our website or dive into the technical documentation which includes demos and other resources. In the meantime, thank you for watching data engineering with data bricks.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.