Why You Should Move to Lakeflow Declarative Pipelines Serverless

Summary

Lakeflow Declarative Pipelines serverless: This video presents the benefits of Lakeflow Declarative Pipelines Serverless, offering automatic infrastructure management and unified batch and streaming processing.
Cost and performance optimization: The serverless approach eliminates infrastructure management overhead, automatically scales to demand and optimizes costs while maintaining observability and reliability.
Simplified data engineering: Engineers write business logic without managing cluster sizing, autoscaling or orchestration complexities, enabling faster development and more reliable data pipelines.

Why You Should Move to Lakeflow Declarative Pipelines Serverless

Talk by Nandini N, Sr Backline Technical Solutions Engineer, Databricks
Lakeflow Declarative Pipelines Serverless offers a range of benefits that make it an attractive option for organizations looking to optimize their ETL (Extract, Transform, Load) processes.
Key benefits of Lakeflow Declarative Pipelines Serverless:

Automatic infrastructure management
Unified batch and streaming
Cost and performance optimization
Simplified configuration
Granular observability

By moving to Lakeflow Declarative Pipelines Serverless, organizations can achieve faster, more reliable, and cost-effective data pipeline management, ultimately driving better business insights and outcomes.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are the main benefits of Lakeflow Declarative Pipelines Serverless?

Key benefits include automatic infrastructure management, unified batch and streaming capabilities, cost and performance optimization, simplified configuration and granular observability for data pipelines.

How does serverless approach simplify data engineering?

Engineers focus on business logic and data transformation without managing cluster sizing, autoscaling or orchestration, allowing them to write declarative pipelines that automatically scale and optimize costs.

What is the cost impact of moving to serverless pipelines?

Serverless pipelines optimize both compute costs and engineering effort by automatically scaling infrastructure to demand, eliminating idle resource costs and reducing manual infrastructure management overhead.

How does DLT serverless handle changing data requirements?

Declarative pipelines automatically adapt to schema changes and varying data volumes without manual intervention, using the Databricks platform to manage scaling and reliability behind the scenes.

Full transcript

`0:00` Introduction and challenges

Welcome to the data NI summit. Uh, hi everyone. I started as a backline engineer in data bricks and now I work in enablement. Before we begin, here are some quick disclaimers. Uh I know probably you didn't expect this uh on this particular uh presentation but do you feed to struggle uh do you struggle to feed your kid or maybe your dog? I have a dog who is a fussy eater and some days he wants kibble. Other days he w even prefers grass and there are days when he doesn't even take the favorite food.
It really does not has to be so difficult with unpredictable situations coming in. I spend a lot of time feeding the dog without having to get him to do um get without him or Okay, I'm so sorry. I'm just getting a little nervous. Is it okay if I restart? Okay, thank you. Thank you. Okay. Um, yeah. I know you probably didn't expect this, but feeding a dog could be tough. Do you struggle to feed your dog? Um, sometimes my dog prefers kibble.
Other days he prefers grass. And sometimes he just won't eat his favorites. It's not so great to have unpredictable situations and I spend time a lot of time trying to feed him than I should. Feeding data pipelines isn't so different. Data engineers often wrestle with unreliable sources. constantly changing formats, manual pipeline maintenance, ingestion becomes a chore. It's slow, fragile, and resourceheavy. That's where DT serverless comes in.

`1:37` The case for DLT serverless

It's just like switching to a consistent balanced dog food that works. DT serverless simplifies ingestion, handles scaling automatically, and lets you focus on insights and not the infrastructure. Let's get started with a simple idea. Engineers never have to think about cluster sizing, autoscaling, retries or orchestrational complexities. They just have to write the business code and the platform handles the rest. In the next 20 minutes, we'll go over declarative pipelines which makes engineers life easy, lets you build reliable data pipelines, go over some of the cool features that enhance your injection with zero tuning.

`2:26` Data engineering pain points

When you build your next recommendation system, prediction, predictive maintenance or fraud detection system or any of the solutions, the foundation of all of that is data. But getting data could be difficult. Every data project starts with ETL, extract, transform, and load. Let's see a data engineer's journey. The focus of the data engineer is this box. They want to transform and aggregate the data to get business insights.
But first they have to think about where the data lives. Is it in data lake or data warehouse? Then they have to make sure it is good enough for the BI reporting, data science, machine learning. And there are streaming use cases, generative AI and orchestration on all of that. Then there comes in version controlling, deployment infrastructure, CI/CD, data control, quality checks, governance, data discovery, etc. And then there are moving parts of operational uh dependency management, backfill, uh partitioning, checkpointing, retries, etc.
While the focus of the data engineer was really the transformation logic. We really don't have to have the engineer to go and tailor through entire infrastructural details while they should have been focusing on giving the reliable data. The time which could have been spent in innovation is spent on troubleshooting. This is really unrewarding for the data engineers and it slows down business. The current process is complex.
It's difficult to maintain and with rigid architecture it makes making any changes to the business really difficult. As your data evolves, your business need evolves. There will be more and more ops changes. Now remember the idea that we started with that you don't have to have any operational changes. We don't have to worry about the operations but you could just build a reliable data pipeline. It should not be so difficult.

`4:23` Declarative pipelines overview

Having to create a data pipeline can be simple, scalable and optimized for both cost and performance. The solution is declarative pipelines which is fully managed assisted by AI powered by serverless unified system led by data governance and out ofthe-box solution which is intuitive to build automated to operate and intelligent and flexible to meet business requirements. Reliable data pipelines are built on top of data intelligence platform.
Your data sits on data lake. With unified data governance and the knowledge of your specific domain, you can apply AI to unlock untapped potential. All of this could be done at one single place, the data intelligence platform. As the summit logo says, u the theme says the data intelligence is for all. Let me show you my encounter with data intelligence. Just a couple of days back. I was struggling with mobile connectivity because I traveled international.

`5:41` Data intelligence in action

I tried some SIM resets, some recharges, activation, the holy restarts of the machine. So, uh none of them really was working. I tried to reach out to the local providers but my handset was something which was not supported and hence I could not even take a local SIM. This was really really frustrating. The telco in this case was Vodafone and this they were able to perform some realtime analytics on the data that had that was available on me.
They saw that I was trying to make some recharges. I was trying to make attempts on reaching out to the help and they were able to even figure out the bottleneck to be the network. They reached out to me over WhatsApp rather than a text or a call because I didn't have network and I also didn't have email configured. They offered me a personalized support with intelligent engagement. They helped me reach to a solution although the issue was not completely resolved but I felt supported.
I felt that I was heard and I had some solution. So now Vodafone gets a customer who is not going to go away. That's the intelligent data that we want. That's the intelligence that we want in our systems. To ingest data into data intelligence platform, we use LakeFlow. You'll hear a lot about LakeFlow in in the summit. LakeFlow is the future of data engineering. It is industry's leading data processing engine. Using LakeFlow, you can bring various small and big data sources into your lakehouse with native ingestion connectors.

`7:16` LakeFlow and DLT architecture

Using declarative pipelines, you can clean and enrich your data. And the lakeflow built-in jobs offers you fantastic observability triggers and orchestration. So it ensures that you have a automation from ingestion to transforming and to downstream consumption just at one single place. Let's move on to DT. DT also known as declarative pipelines is not a standalone tool. It's heart of the solution that delivers you fast and reliable trusted data.
It's part of LakeFlow, a unified system of data engineering experience built on top of data intelligence platform. With LakeFlow, the data engineer no longer has to perform patchworks of disconnected tools. It's streamlined experience that delivers data simple in a simple, scalable and reliable fashion. LakeFlow is tightly coupled with serverless with zero knobs and quick startups and scaling computes. We now understand where declarative pipeline sits in the entire data bricks.
So let's just get started with creating one. Creating a pipeline is really easy. With declarative pipelines, you just need to say what should be done and not how it should be done. You can say create a streaming table or a materialized view from a specified source and that's it. You can write code in plain simple SQL or Python whichever you prefer. The DT runtime figures out the best way to create or update this table saving you hundreds of lines of code in production and it's less errorprone.

`8:52` Authoring pipelines

Talking about less errorprone, there are some awesome features that we have an ID like editor for pipeline authoring with fast validations, inline error check and quick results for users who prefer coding. With our new multifile editor, we are taking developer experience to another level. This looks great, right? To try out the multifile editor which is currently in preview, make sure you switch on the ETL pipeline editor option while creating the pipeline or later on from the settings.
Let me walk you through some of the scenarios in the pipeline authoring experience. Although this is not a live demo, you'll be able to get a complete idea of how everything is set up. On the left hand side, you have pipeline assets browser where you can create, delete, rename, organize pipeline assets under a default pipeline folder structure. On the top, you have a multifile code editor option where you can work with multiple files across different tabs in the pipeline.
There's a pipeline specific toolbar with pipeline specific actions like dry run scheduling etc. On the right hand side there's interactive DAG to get the overview of how the DAG how the flow of the tables is. The data preview at the bottom and other table related actions could be seen. You can inspect your data of the streaming table and the materialized view. Next, there's table level execution insights where you can review insights about all the tables in the pipeline or a single one.

`10:40` Debugging and observability

The insights will refer to the latest run of the pipeline. The error tray on the right side summarizes across all the files whatever error you get. And in order to see the error, you can go to a specific file by clicking onto the error. The code editor has features of step-by-step development. That's selective execution. If you make any typo or syntactical error, you can see them in line. It lists down the stack trace and the details right there.
From the error tray or the event pipeline, you can get to the further details and use databicks assistant to help you diagnose the error. The datab bricks assistant is a contextaware AI specific tool. It works by analyzing user input including code queries and leveraging platforms metadata documentation and relevant suggestions and solutions. You could fix the error in line as well allowing the databicks assistant to take over from here.

`11:52` Databricks assistant and AI

It generates the suggestions real time. You can review and either accept or reject the suggestion. It highlights the error uh and the differences with color coding. If you do not want to write the code at all, you could use datab bricks assistant and give it instructions in plain natural language. The database assistant could be seen by this icon, the star symbol icon. You just have to look around for that. Or you can go to a new cell and create click on generate give instructions and review the generated code in the query history.
You can get the past runs and click to get further details like UI, Spark UI logs, etc. It's just pretty much like the Spark execution in DBSQL. to start exploring the DT serverless and materialized view that you have just created just with a few clicks and instructions again in natural query processing you you can use genie and genie understands the context and gives you accurate answers you can build in report on top of that and you can start querying the table that's a lot on the new UI from ingestion to quick consumption another reason to move to declarative pipelines is the unified batch and streaming solution in one single pipeline.

`13:07` AI/BI Genie and unified modes

Batch processing is costefficient but can introduce delays. Streaming introduces latency but can be complex and expensive to operate at scale. And once a pipeline is built one way, switching to another one is a complete rework. DT removes these trade-offs by unifying batch and streaming into one single framework. That means your team can adopt to pipelines and business needs without having to being logged into one approach.
To switch you can just change the trigger mode from pipeline in the pipeline mode from triggered to continuous and move from batched to streaming. When using when you're using serverless there's low latency and low cost. You can look at the numbers. There's 5x reduction in latency and approximately 50x reduction in the cost when compared for materialized view refreshed done for serverless against classic cluster. Serverless is fully managed and versionless.

`14:22` Performance and versionless ops

Version is one of my personal favorites. Many a times you might have raised tickets saying there's nothing that has changed from your end but the pipeline is still failing. So versionless how it works is there will be upgrades via health mediated releases that is HMR. So if there is a upgrade and there's a failure that's detected on your pipeline there will be a quick underlying roll back and that is done without you having to reach out support.
Your pipelines will continue to run as it was. Maybe you will definitely see in the one of the previous runs to be failed but the future ones would have succeeded. So the upgrades do not impact your scheduled runs. With Delta integration you see great performance because you can use features like liquid clustering, deletion vector, predictive optimization etc. For serverless, you also get incremental materialized view with enzyme.

`15:30` Serverless optimization features

That is, you can refresh a materialized view without having to do the complete recomputation but just the new changes. With pipeline execution where you can run multiple micro batches concurrently rather than waiting for all the tasks to finish, it gives further optimizations. There's also dynamic tuning for batches based on the amount of compute available. Photon offers faster read and write performance with vectorized IO and gives you approximately 10x increase with delta metadata queries.

`16:24` Feature comparison

We've already gone through a lot of features, but there's a quick comparison where serverless can be used when you're having CDC data, sedd data, and it offers you data quality expectation, observability features. Photon optimizations are by default enabled in serverless and you don't have to specify instance types. There's automatic instance scaling that prevents the out ofmemory failures. You get enzyme with incremental refresh for materialized view, streaming refresh only on serverless.
There's instant scaling with fast startup and single inclusive bill. When creating a pipeline, you can select serverless just by one single click and it handles all the problems of underallocating at peak hours or initial load seamlessly and excessive resourcing when off peak hours avoiding you to pay for idle resources. It's either you letting data bricks handle the compute for you or you can do the tuning yourselves.

`17:04` Conclusion and resources

The only difference between serverless and non-serverless architecture is where the data plane sits. In non- serverless it sits in the customer's workspace and in serverless it is in the datab bricks workspace. Creating declarative pipelines is simple, performant and it has low cost with full UC compatibility. There's production grade pipelines. DT serverless is indeed a gamecher with zero infra setup, enhanced autoscaling, ultra low latency, and native integration to UC and DT.
With serverless, there's no ops and just simple reliable pipelines. There's a lot more features being announced in data and summit, so stay tuned for hearing about more of them in the keynotes. And before you go, here are quick recommendations for learning about declarative pipelines in the sessions in the in some of the other sessions. Thank you. Thank you all. Please do fill a survey over here and have a good day. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.