Skip to main content

Authoring Data Pipelines With the New Lakeflow Declarative Pipelines Editor

Summary

  • Declarative editor: Databricks Lakeflow introduces a new multi-file editor with IDE-like features for building and maintaining declarative data pipelines.
  • Developer experience: Contextual data previews, DAG visualization, selective updates, and step-by-step debugging enable efficient, code-first development.
  • Productivity features: Built for experienced data engineers while providing a bridge for teams transitioning from imperative to declarative pipeline development.

Authoring Data Pipelines With the New Lakeflow Declarative Pipelines Editor

Watch: Authoring Data Pipelines With the New Lakeflow Declarative Pipelines Editor
Talk by Adriana Ispas, Sr. Staff Product Manager, Databricks ; Camiel Steenstra, Staff Software Engineer, Databricks
We’re introducing a new developer experience for Lakeflow Declarative Pipelines designed for data practitioners who prefer a code-first approach and expect robust developer tooling. The new multi-file editor brings an IDE-like environment to declarative pipeline development, making it easy to structure transformation logic, configure pipelines throughout the development lifecycle and iterate efficiently. Features like contextual data previews and selective table updates enable step-by-step development. UI-driven tools, such as DAG previews and DAG-based actions, enhance productivity for experienced users and provide a bridge for those transitioning to declarative workflows. In this video, we’ll showcase the new editor in action, highlighting how these enhancements simplify declarative coding and improve development for production-ready data pipelines. Whether you’re an experienced developer or new to declarative data engineering, join us to see how Lakeflow Declarative Pipelines can enhance your data practice.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What's new in the Lakeflow Declarative Pipelines editor?

The new editor provides a multi-file IDE environment with features like contextual data previews, DAG visualization, selective table updates, and step-by-step development capabilities designed for code-first data engineers.

How does declarative development differ from traditional SQL scripting?

Declarative pipelines let you define desired data transformation outcomes; the system handles scheduling, error recovery, and optimization. This reduces boilerplate code and makes pipelines easier to understand and maintain.

building declarative pipelines

Building declarative pipelines on the Databricks Data and AI platform simplifies development by letting engineers define desired outcomes rather than managing execution details. This video showcases the new multi-file IDE editor that provides code organization, contextual data previews, and DAG visualization for rapid iteration. The declarative approach reduces complexity while the new editor adds IDE-like features including selective table updates for step-by-step pipeline development.

how to build a data pipeline

Building a data pipeline begins with understanding your data flow requirements and transformation logic, which the new Lakeflow editor simplifies through an intuitive declarative interface. This video demonstrates how to structure transformation logic across multiple files, execute pipelines efficiently, and debug at each step using contextual data previews. The editor supports both SQL and Python, making pipeline development accessible to diverse engineering teams.

best platforms for building data pipelines

The Databricks Data and AI platform provides the best foundation for building data pipelines through its integrated Lakeflow family covering data ingestion, transformation, and orchestration. This video presents the new declarative pipelines editor that enables code-first developers to build production-ready pipelines with built-in optimization and error handling. Features like DAG-based debugging and modular code structure make it ideal for teams ranging from beginners to experienced data engineers.

Full transcript

0:00 Introduction to lakeflow

Hi everybody. Thanks a lot for joining us for this session where we will tell you all about a new editor which we have introduced to help you build data pipelines. I'm one of the product managers on the data bricks flow team and with me is Yeah, I'm a software engineer in the lake for pipeline team. Perfect. Right. So one could say that we live in times of data abundance and data literacy. On one hand side, companies have lots of data and their employees have come to expect access to this data so that they can use it for their day-to-day tasks.
Though it's not enough to just have data. This data often comes in a raw format and you need to first glance it, organize it, transform it before a data consumer such as the person here can actually use it for analytics. And that's where data pipelines come in. Data pipelines are like roads for data. They take data from its source and bring it to our data consumers. But like building a road, building a data pipeline can be difficult.
You require knowledge about ETL topics, software engineering patterns, distributed systems, and much more. To help re remove some of that complexity, we from datab bricks have introduced datab bricks lakeflow declarative pipelines. This is what you maybe know from before as DT. They are part of the lakeflow family together with connect and with jobs where connect is our solution to help you bring data in using point-and-click connectors and jobs is our solution to help you automate the pipeline as soon as you're ready as soon as you're happy with this logic.
So what is specific to late flow declarative pipelines is the declarative approach to data engineering like the name says we have introduced this some time back actually with a goal to reduce development complexity. So when you create a declarative pipeline you tell the system what is the outcome you're interested in. So in this case we tell the system we want to have two tables in our pipeline. We tell the system how the tables should look like and how they are connected with each other and then the system figures out what is a dependency graph and then handles updates for data sets also handles the execution in such a way that overall one gets a pipeline that it's easier to reason about.

1:53 New pipeline editor overview

It's optimized for execution and it's easier to maintain. Now us from the Leflow team didn't want to stop at this. So over the past year, we spent a lot of time with experts in house and also talking to customers to understand how they work and we've introduced this new developer experience to match the declarative paradigm. It is actually an editor which we've purposeuilt to help you develop declarative pipelines. We have built this editor in with users that prefer coding in mind also who are looking to put in place software engineering best practices.
And we also build this editor with a premise that when you are developing a data pipeline, you need to do much more than simply write code in a notebook. So you'll see that this editor helps you organize your code across different files and then folders. It also comes with different features that help you iterate on the pipeline logic. And then when you're finished, the editor allows you to also leverage other capabilities of the databicks platform that come into play when working on a pipeline.
So that includes modularizing code, doing code reviews, doing deployments, and much more. Now, we built this editor on top of four principles. One thing that was very important to us was that it should be easy for anybody getting started with the editor and with the declarative approach to get started. Second, it's our goal to make it very efficient to develop declarative pipelines. So the experience that you've come to expect from the SQL editor and from notebooks, we want to provide it with this editor but of course adapt it to building a data pipeline.
Third, like I already mentioned, there is much more to take care of when building a data pipeline. You know, I mentioned modularizing codes, code reviews, and so on. Well, our goal is to provide all of those into this editor on the same surface so that your work is most efficient. And finally, we also know that there are users who've already become experts either with a declarative approach and data engineering as a whole. they want sometimes more advanced features.
Those are provided in the editor as well in a way so that they don't get in the way when you're just getting started or when you don't need those advanced features yet. Now in the remaining part of the presentation we will go through this one by one and the presentation will be actually demoheavy and for these demos we chose to use a data set that comes from a free distribution service which is focused on articles for research community communities.

5:18 Creating a new pipeline

So with that said, I'll put myself now in the shoes of a new user, somebody that's new to this editor and I'll show you how you can actually, you know, figure out what are the capabilities of the editor. So I switch here. I have data bricks the homepage and I want to create a new pipeline. It's very simple. I create new. I press new and I choose ETL pipeline. Now what we have here is a landing page and it seems like the first thing that I need to do is give a name to my pipeline.
All right. So let's call it demo dice one two three and then it seems like you know the editor is guiding me. It's telling me okay now you need to give me a catalog for your tables. I'll choose one that I uh have for my user and then a schema. And we choose a schema that is relevant for our use case. Now, it seems like I have a few options to proceed. Um, there are some advanced options. They might be relevant to me later on, but now I'm just getting started.
So, probably the options on the part above are what I should be considering. Seems like I have the options to start with some sample code and some default folder structure for my pipeline. So, probably that's the right thing for me. And um yeah, it seems like I I can choose between SQL and Python. I'll choose SQL for now, but I I also like Python, so I'll probably want to choose that uh for our next pipeline. All right, so we are here on the editor.
Um it seems like we have already like the hints uh mentioned before, some sample code. Um I could extend this collection of files. It seems like if I run the pipeline I will see some execution insights and also I'll get the graph and you know it seems like the first thing that I need to do is run my pipeline. So I'll do just that and in order to save some time I've actually done that right before. So I've created a similar pipeline um from sample code and it's here and you know since this has already run it seems like I have two tables in my pipeline coming from those files presumably and it seems like you know this table has two columns all right let's quickly see the other one in this case it's also the same two columns so probably in order to actually figure out what's happening here I should look at a code And in order to do that, I can use this quick handy actions.
So it will now take me to the code for this table. And it seems like we are creating here materialized view that simply reads from a sample table in Unity catalog. All right, that seems pretty straightforward. Let's quickly see what the second table is all about. And here it seems that we are reading from the previous table which is obvious in a way because that's also what the graph has indicated and also we are doing some aggregations.
All right. That's uh pretty helpful. The thing is this is not my data. Remember I'm working with uh articles. I'm working with um data uh authors um and I'm a researcher. So let's actually see what it means to create my own files for this pipeline. So I want to go to Unity catalog and see you know I have some data about authors. Let's quickly have a look uh in my catalog here I have a table hopefully of authors. Yeah, that's great.
So I want to work with that and in order to use that data I'll just create a file for my pipeline and I'll use SQL again and um I'll give my table name of authors clean because I know that the table that I have there just authors is not really great yet and I seem to have some you know hints here. So I'll create the materialized view. Let's see what happens. All right. So, what we see here is that the system created for me a table that has the same name as the file, which seems quite handy for the future, but we still read from the same table, the sample table.
So, we will want to read from my table, from the one on authors. All right. So, it seems like I need to open this again. Let's quickly do that. Then I just use this quickly quick um uh action so that I populate the name of the table. Now the system already tells me hey these are not the right columns which is uh great that's very helpful because indeed I have uh data on authors in my table. So let's quickly change that.
All right I'll also take the last one the last column often. Awesome. So let's see what happens if I just run this file. Hopefully everything works. Hopefully I didn't do any errors. Uh what I expect to happen is that the graph here is extended with another table, the one that I'm just creating here. And that's really happening right now. So let's quickly see what we have here are some execution insights. Um let's see about the data.
Okay, it seems like we have maybe some problems. This author here, this author ID is null, which is not great. And I've read online that in DT in declarative pipelines expectations is what I can use to check my data quality. So I'll ask the assistant to help me with that. So I just press now command I. So I'm going to give the assistant a prompt to help me do some data quality expectations here. Let's see. Hopefully it get it gets it right.
I really need some code here. I never really remember the syntax. This looks pretty decent. So, let's run this. And in the meanwhile, I also want to, you know, extend the pipeline with another table. Do some aggregations on this data. So, I'll do that right away. So, we save some time. So, I'll create another file again, a SQL file, and I'll call this author's app. I'm not going to use a a template this time. Um, what I want to do is actually do some aggregations.
For example, um, just have a table with all of the authors that have more than 10 papers. So, I'll just use this quick hand here again, the assistant, and I'm going to give it a prompt that says exactly that. you know, given the author's clean, which is the table here, please help me create the code for another table that reads from it. Uh, let's quickly see if that's correct. Sometimes the assistant doesn't get the get this right.
And yes, it kind of made the mistake here because our column is called num papers and somehow it thought that it's number of papers. So, let's quickly correct that. All right, that looks right. I'm going to accept and I'm going to run this. And while this is running, you see here it seems like we have a hint that when I do did those expectations, actually it worked. Some records were dropped. So probably the data for that table now doesn't have nulls anymore.
Perfect. It seems like I have some nulls here. So I should have done some more quality expectations also for the number of papers. But in the meanwhile, it seems like I also have my aggregation table. Um, so it it worked as well. Great. So probably now is the next step. I want to get rid of those sample tables and uh, you know, further enhance my pipeline. So what we've seen is that when I get started, I have this guided setup if I want.
And you've also seen that there are some other options to create with a sample pipeline, an empty pipeline or pipelines with uh codes that maybe I have somewhere in Git. So maybe next time I'll do that. You've also seen that there were some recommendations in terms of how to organize my code. There was a transformations folder uh an explorations folder. We haven't really explored that too much. Uh we will see it in just a bit.
But you've also seen that I didn't do any settings. I just pressed run and things work. That's because we work a lot with defaults but of course one can change them as well. And finally we are trying to provide all of this efficiency features so that one doesn't have to do a lot of typing of uh you know boilerplate code but rather have the system do that work for us. So now in order to see how a user who has already gotten a better sense of the editor works I'll just pass it.

14:23 Iterative pipeline development

Okay, so I will talk in a bit more detail about all the features that we added in this editor to make it more efficient to iterate on pipelines and I will mostly be doing a demo as well and I wanted to start my demo here my workspace and I wanted to explain a new concept that we added here of pipeline folders or pipeline root folder and with this we wanted to make it easier to organize all the files related to a single pipeline into a single folder.
So you can see in my workspace I created a few demo pipelines already for this talk and each of these folders is a pipeline folder. So these pip these folders are associated with the pipeline and I now have this button open in editor. So I can open the editor for that pipeline or if you know the monitoring page already it also has a new button here edit pipeline. So if you click any of these entry points, you land on the pipeline editor.
And now I have the pipeline editor open for my pipeline, my demo pipeline here. And you can see all the panels related to the pipeline, the browser, the graph, and the bottom panel. And before explaining some features that we have in this editor, I wanted to give some background about my current pipeline. So my pipeline right now has two streaming tables and I'm actually reading this data set that we mentioned before.
So this data set as JSON files and I uploaded these JSON files to my volume here as separate JSON files and besides that I also extracted this category metadata. So it has some descriptions for categories and titles and ids and I also uploaded this in my foyer. Okay. So then coming back to the pipeline editor actually here I have the definition of my first data set and I'm using Python and autoloader. This is autoloader to read the data from the volume and I put my volume pads in the pipeline settings and I will show this a bit later.
And first I wanted to show some things that we added in this pipeline graph to make it easier to iterate on your pipeline. And this pipeline graph it highlights which table is defined my currently open file. So I have this table defined here and it's highlighted in the graph. And like shown before, you can also use this graph to navigate to definitions of different tables. So if I want to see the definition for this table, I can navigate to it like that.
So then I open the definition for my second table. So here I'm also using autoloader and Python just to get the category data to my pipeline. And then the another feature that we added here is data previews. So with data previews you can quickly see what's available in your tables in your pipeline. You can see a sample of the data and you can use it while developing to check if everything makes sense or what data is available in any upstream tables.
So you can do some basic searching or filtering as well similar to what you could do in a notebook. Okay. So then I wanted to focus a bit on this browser on the left. So like I mentioned before we have this pipeline folder and here I can see everything in my pipeline folder and in this folder we have kind of a new concept of pipeline files and non-pipeline files. So you can actually see here these files are marked with a pipeline icon.
So it means it's part of my pipeline and that means it's included in my settings here. So right now I just included these two files that I showed before. And besides that we also have excluded excluded files. So these are marked with this icon here. So it means they are not part of my settings. And you can now manage which files are included through this file browser here. I can include additional files. So I created another folder here with my silver layer.
And I can include these files individually. And if I include it like that, it's reflected in my settings panel. And we also added a new feature in pipelines to be able to include folders completely. So I can include this whole folder here and it will also be reflected in my settings and including files just means if I run the pipeline it will execute all those files. So all the data sets defined in those files will be in my pipeline.
So I will just run my pipeline after including the silver layer as well and I will briefly explain what I'm doing in the silver layer. So I added two additional data sets in this layer and I'm just reading the bronze data that I showed before and I'm doing some renaming some parsing of the column data like this and I also added a few expectations. So I'm using drop expectation here or a burning expectation here and this for the category data I'm doing a similarity.

18:58 Debugging and advanced features

So reading the bronze data and doing some renaming and actually when I run the pipeline I had an error here. So actually I added this intentionally but I wanted to show some features that we added to make it easier to debug errors. So mainly we added this issue panel at the bottom. And here you can see an overview of all the errors that happened when you run the pipeline. And you can also see in this fire of which which files are causing errors.
I can also use this utility here jump to code to open the file that has the error and there I can see in context what's happening. So apparently I have some issue here with my typing and these errors are actually integrated with assistant as well. So I can ask assistant to fix it error. So assistant actually found the issue. So I was using some column that I defined before passing it. So I will just accept that. And while that's running, I wanted to explain some features that we added in this editor to make it possible to um run specific parts of the pipeline that you're working on.
So in this example, I just fixed an issue in this table. And we added this cutter action that you can use to specifically update that table. So you can use it to iterate on only one table at a time and then update only that specific table. And I wanted to explain a bit here that in my demo pipeline I organized it so that I define one data set per file. But this is actually not a restriction that you have to do. You can define more uh things as well.
And for that we added also a new feature to run everything in a single file like this. So then it will execute all the data set definitions in that file. And like before we also have the ability to validate or try run the whole pipeline from here. And this will just check that all the typing and all the data flows make sense together. And you can of course also run the whole pipeline. And all these features you can also trigger a full refresh from the editor.
So this means it will recmp compute all the data in your pipeline. And it can be useful if you're working on streaming tables for example. Okay. So now I fixed the issue in my table and I can see the output in this preview and here I wanted to explain some other feature that we added to make it easier to have utilities that you can use in the whole pipeline in all the data sets in the pipeline. And in my preview I can see here my column naming is pretty inconsistent.
I'm using dashes in some places and underscores in others. And I actually added a utility already. So here I have this folder utilities and I created a utility to normalize all the column names in my pipeline and I was actually already using this in the getter query table as well. You can see here I'm importing this and this actually works in combination with the pipeline folder concept. So the pipeline folder is automatically added to the Python CIS path.
So I can just import anything in my pipeline folder like this. and then I can apply it to this table as well. Okay, then I will just also include my code layer and run my pipeline and then I will briefly explain what I'm doing in this code layer. So I have another folder here, my gold folder. And here I'm just aggregating some data about the articles to count per category how many articles there are. And for authors, I'm doing a similar thing.
I'm aggregating all the papers per author and counting some things as well. And then finally, I'm using this AI query to generate a summary per author based on all the titles they wrote. So I will just go back to the slides and briefly recap what I talked about. So we added several ways that you can run specific parts of the pipeline while you're working on the pipeline. So you can update only a single table. You can run everything in a single file or you can just run the whole pipeline.
We added data previews in the editor so you can quickly see what data is available in your data sets without having to go to a separate SQL query or a separate notebook. And we added several ways that you can debug pipeline issues through this issue panel or in context errors. And it's also integrated with assistant. And finally, we added several features in the pipeline graph that you can use to navigate between your sources and you can see what's happening while you're working on your pipeline.

24:04 Organizing and monitoring

Okay. So then I will continue a bit about how we incorporated different things that you need while you're developing into a single editor service. And for that I will just also go back to the editor. And I mentioned it briefly before but we have this concept of pipeline files and non pipeline files. And I wanted to explain a bit more about the purpose of non pipeline files. And one purpose of it is explorations. So in explorations you can just have a regular notebook and I used it there to just kind of investigate what data is available and kind of tweak my logic before I actually include it in my pipeline.
And so that's one use case for non pipeline files. So they are kind of related to the pipeline but you don't really want to execute them when you run the pipeline. And the second use case for it is utilities. I also showed it before. So I extracted several utilities that I can use throughout my data sets and making this modular makes it easier to test specific parts or to reuse code between all your data sets. So I have some logic here for parsing specific data in my pipeline and specifically this naming convention utility.
I wanted to talk a bit more about that. So this kind of utility is often useful across different pipelines. But if I put it in here in the utilities folder that's kind of scoped to a single pipeline and to make it easier to have utilities between different pipelines, we also added a new feature in pipelines for environments. So we now have this environment section as well and you can use it to manage any external dependencies that you have.
So you can install anything that you normally install through pip install and this can be any third party library but you can actually also move things in a package in your workspace or in a build file. So as an example I move this naming conventions to a separate folder and I can just install this in any pipeline as well. So this can make it easier to reuse code between different pipelines. So the next thing I wanted to mention is I already showed it a bit before but we added a settings panel.
So you can modify any settings of pipeline within the context of the editor. You don't have to switch back and forth to another page. And it can be useful while you're developing to kind of tweak your settings. For example, I have a lot of configurations here that I showed before that I was using. Okay. So then I also wanted to mention that this editor is compatible with version control. So my pipeline folder is actually in g.
I can open g here. So you can actually version control or your pipeline sources as well. I'll just close this. And here I just wanted to mention briefly that if you just put the pipeline folder in get then only the pipeline source files are version controlled. But the settings themselves are actually not. And for that you can also use data bricks asset bundles. So that's also compatible with this editor. And there you would actually source control your pipeline settings as well.
Okay. Then finally I wanted to show a bit that we added several execution insights within the context of the editor. For example in this pipeline graph you can see stats about how many rows were processed and you can also see info about expectations. So you saw before I added some exploitations in my data sets and I can see here what is happening with all these expectations how many were failing and that kind of thing.
Then we also have this performance tab. So the performance tab shows all the query profiles in your pipeline. So you can inspect what's happening in each query or each data set and you can optimize it that way. Then finally, if you want, you can also access the event log. So this just shows every event that happens when you run an update. And if you want to really know the details, you can go here. And finally, you can still go to the monitoring page.
So in the monitoring page, you can inspect every historic run as well. And you can see what happened in previous runs if it's loading. So yeah, you can go here to see what happened in the past. Okay, then to kind of recap what I talked about here. So we try to make it easier to have code adjacent to the pipeline like explorations or Python utilities that are not actually included in your pipeline and they can be added to your pipeline folder.

28:46 Conclusion and next steps

We added the settings panel so that you can edit any settings in the pipeline while you're developing. So you don't have to leave the editor. The editor itself is compatible with version control as well. You can use kit or database asset bundles. And finally, we surface several execution insights within this editor like expectations or any metrics about how many rows also the query plans and you can also access the event log.
So with that I will hand it. So there would be much more to say um and so much more to say that in fact we've prepared another session later today about some additional topics related to developing pipelines. Of course we invite you to join it's called in production. Um those topics will go more into topics related to taking the pipeline that you've developed in the way that we just showed here uh to production. So um today or right here I will just briefly touch on these topics just to give you a sense of it. already showed that there is also much more around execution insights historical insights about the pipeline that you can access either in the user interface or through an intent log.
So we've also packaged that into the surface as well. Also this editor integrates seamlessly with datab bricks asset bundles which is the solution that datab bricks provides to help you with your deployments across environments. Actually datab bricks asset bundles come with a user interface that is provided inside the datab bricks workspace. So that's available and also seamlessly accessible from the editor. This tab here allows us to manage deployments and you see it's accessible from the editor as well.
And in addition to that, what's relevant when working on a pipeline is testing. Often times we change the logic and we want to make sure that we are not introducing regressions. So we've intro integrated first class support for testing as part of the DT or declarative pipelines APIs. And uh we will show you that in action later today as well. Here is just to give you a sense of it. You see the editor provides the ability to run tests as well.
So with that said, a bit of a recap of what you've seen today. You've seen that we've introduced a whole new developer experience to go hand inhand with declarative pipelines. You've seen this multifile editor with a wealth of features to help you iterate on their on your pipeline logic. You've also seen how we thought about the experience of different users depending on their journey and the stage of their journey on data bricks.
We thought about users who are just getting started, users who've uh maybe already gotten some experience and also gotten advanced with declarative pipelines. You've also seen features that we've added to make the inner dev loop efficient. Selective execution. You've seen run file, run table, run the whole pipeline, data previews, graphs to give you a sense of uh all the tables in the pipeline, whether they are connected or not, whether you're reading from the right table or not.
We've also hinted at uh more advanced features that you can um hear about later today. So dabs, testing, config driven pipelines. And finally, what we wanted to show is that all of this is actually available to you today. So this editor is available in Pugly beta. So you can open your laptops and try it right after the session or tomorrow. Uh we are working very actively towards making this public public preview. So, exit the beta which is going to happen in the coming weeks.
And with that, we hope that we sparked your interest and uh we'd love to get some questions if you have any. Right, it seems we need to stop. Thank you so much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.