Skip to main content

Introducing Lakeflow: The Future of Data Engineering on Databricks

Summary

  • Unified Data Engineering: Lakeflow is Databricks' end-to-end solution for data ingestion, transformation and orchestration, reducing complexity across the entire pipeline.
  • Components Overview: Lakeflow Connect handles no-code ingestion, Lakeflow DLT enables declarative pipelines and Lakeflow Jobs orchestrates end-to-end workflows on the Databricks Data and AI platform.
  • Simplified Workflows: By unifying ingestion, transformation and orchestration, Lakeflow enables scalable, reliable and efficient data pipelines in a governed, high-performing environment.

Introducing Lakeflow: The Future of Data Engineering on Databricks

Watch: Introducing Lakeflow: The Future of Data Engineering on Databricks
Talk by Bilal Aslam, Sr. Director of Product Management, Databricks ; Michael Armbrust, Distinguished Engineer, Databricks
Join us to explore Lakeflow, Databricks' end-to-end solution for simplifying and unifying the most complex data engineering workflows. this video builds on keynote announcements, offering an accessible introduction for newcomers while emphasizing the transformative value Lakeflow delivers. We’ll cover: What is Lakeflow?: A cohesive overview of its components: Lakeflow Connect, Lakeflow DLT and Lakeflow Jobs Core capabilities in action: Live demos showcasing no-code data ingestion, code-optional declarative pipelines and unified, end-to-end orchestration Vision for the future: Unveil the roadmap, including details of two major keynote announcements Discover how Lakeflow equips data teams with a seamless experience for ingestion, transformation and orchestration, reducing complexity and driving productivity. By unifying these capabilities, Lakeflow lays the groundwork for scalable, reliable and efficient data pipelines in a governed and high-performing environment.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are the three main components of Lakeflow?

Lakeflow Connect handles data ingestion with no-code connectors, Lakeflow DLT provides declarative pipelines and Lakeflow Jobs manages end-to-end orchestration.

How does Lakeflow simplify data engineering?

Lakeflow reduces complexity by unifying ingestion, transformation and orchestration into a single cohesive platform on the Databricks Data and AI platform.

What is the declarative pipeline approach?

Declarative pipelines let you define what data you want rather than how to get it, reducing development effort and improving pipeline maintainability.

Who should use Lakeflow?

Data teams of any size, from newcomers to experienced engineers, can benefit from Lakeflow's simplified, unified approach to data engineering.

Full transcript

0:00 Introduction to LakeFlow

Hello Bricksters. Hello. Welcome back. Thank you for joining us. We're so glad you're here. If you're just joining us now, I hope you get a chance to go back and take a look at our session that we just hosted prior to this with and take a look for knowledge assistant which is now in beta and also that multi- aent supervisor coming to beta very soon. Let's take a minute and review what we've got coming up for later this afternoon.
So following this introduction, we're going to hear from Bal and they're going to talk to us about LakeFlow. And then shortly after that, we'll hear about building AI powered apps. We heard quite a bit about apps in our keynote this morning. And then Lexi is going to show us how to build a roadmap for impact on the business in just 30 minutes. and then we'll finish things up with elevating SQL productivity and take a look at some things within notebook.
So, I'm super excited again for all of you to be joining us. This next talk is probably one of my what I've been one of the talks I've been looking forward to most because LakeFlow Connect Lake Flow is is just it's it's making everything easier. Again, we talk about the importance of democratizing data and data intelligence for everybody. And one of the hardest things is just getting all of the data in one place. And so, and Bal are going to tell us how you can use now these low code, no code approaches to ingest all of your data.
They're also going to share the road map with us. Super excited to introduce and. So, is one of our senior directors of product management and is the original creator of Sparks. So excited to have him in the room with us. So with that, I'll turn it over to the team. Gentlemen, take it away. Awesome. All right, good afternoon. Everybody had lunch? Awesome. Thank you. Was it good? Okay. Well, thank you so much. Uh this is a post- lunchunch meeting, which means, you know, you're probably a little tired.

2:00 LakeFlow vision and goals

You want to take a nap. Uh we hope it's an exciting conversation. Um, so Awesome. And we're excited to talk to you about LakeFlow. Uh, there's going to be a lot of stuff tomorrow. Uh, I do want to apologize and be very honest. The abstract says we're going to do two big unveils. Actually, this session was supposed to be after tomorrow's keynote. You would have seen the unveils and we would have got into depth. But you can see one of the big announcements today at talk.
Yes. Which is a little bit later. Yeah. It's uh declarative pipelines and Apache Spark. if you want to see that today and then we'll be doing a deep dive on Lake Flow declared pipelines tomorrow. Yeah. Awesome. And then there's one more which is actually massive. You have to wait till tomorrow for that. So I don't want you to walk away disappointed saying hey this was you know false advertising or something. It's just we had a little bit of a crosswires over here.
Okay. Awesome. Uh so welcome everybody. We have for half an hour. So I'm going to do something a little non-traditional. Uh you're going to see a lot of ingest, transform, orchestrate tomorrow. Today we're just I'm just going to tell you a little bit about what is LakeFlow just in case you don't know and just to bring you up to speed and then going to talk about the transform piece first. So a little bit of uh creative sequencing over here.
Uh forward-looking statement. Uh you've probably seen this a million times. Uh this is anformational presentation. It's not a promise. Uh perfect. Please complete your surveys. We do listen to feedback. Uh excellent. So what what is this talk about? We are going to be talking to you about how we are unifying data engineering with LakeFlow. But it's actually not just you know why we're actually going to get into the road map and there's plenty of time for Q&A.
So please hold on to your questions. There'll be time raise your hands ask those questions. Uh so one thing I wanted to do was very quickly talk about you know where does LakeFlow fit into the data intelligence platform and this is where it fits in. You have mosaic AI for artificial intelligence. You have data SQL for data warehousing. You have AIBI for business intelligence and lakeflow covers data engineering. It is how you ingest, transform and orchestrate uh data uh into data bricks.
And of course this is all built on top of unity catalogs. You get a single layer of governance and everything is unified. Uh I love this quote from one of our customers uh JetBlue. I think it's worth reading it out loud uh because I've actually found this to be true in 100% of places. And by the way, this is the big why behind Leg Flow. Uh Carol Clement says you can have all the AI in the world, but if it's on a shaky data foundation, then it's going not going to bring any value.
Uh the one way I've heard a customer say this is, you know, everything starts with good data. And that's actually the why behind LakeFlow. So if you take one thing away from this, this whole presentation conversation is LakeFlow is how you get great data into your business. And if you don't do that, if you don't use leg flow, what I see often happening in customers and out there in the wild is well, you have to stitch together a lot of these tools.
Uh you get this really fragmented stack. Uh this leads to governance problems. You get complexity and inefficient data pipelines. So don't do that. Uh you are going to see a lot of announcements around LakeFlow tomorrow. I do believe it's the future of data engineering and you should use it. And the one of the big things about LakeFlow is it's now generally available and we're going to announce that tomorrow. Uh we're seeing a lot of success with LakeFlow.
Uh we have thousands of customers already using uh different parts of LakeFlow. So it's generally available being used actually heavily and lots of gains in productivity. So I while we have I want to make sure that we talk about one super important part of LakeFlow. So take it away please. Yeah. Yeah. Thank you. And so you know one one big part of LakeFlow is the ability to transform data. All data starts off messy and you need to get it ready.
You need to clean it, filter, augment, join so that it's ready for consumption by the business. Traditionally we did this with LakeFlow DT and you know tomorrow we're going to be announcing that we're renaming this lakeflow declarative pipelines. You get a sneak preview into this by coming to this session. But you know before we get into that I want to kind of go into a little bit of a history lesson. Often declarative programming seems so simple you don't realize the power of what's going on under the covers.

5:53 Declarative pipelines

So I want to do a case study here where we look at an incredibly common uh data transformation you need to do called change data capture and look at what actually makes it difficult and how LakeFlow can simplify that. So the problem is super simple. I have this change feed coming from some other system. It could be Postgress. It could be my SQL. It could be you know some some other operational system. And I I'm getting this list of changes and the time that they happened and the new data for that record. there's a primary key and what I want to do is take the latest record for every key and then store it into a table.
Sounds really simple, but uh do there's a lot of hidden complexities due to out of order data and other other race conditions that you can run into. So if you go on to Google and you type in you know change data capture a spark, you're going to get some source code like this. And so it's actually kind of already looking a little bit complicated. We're going to read the stream of changes and then we're going to have this function that takes each change and merges it into the delta table.
Looks pretty good, but actually there's a bunch of subtle problems with this implementation. The first problem is it's not handling the case where there are multiple updates to the same key within a single microbatch. Merge is just going to explode and say, I don't know which of these rows to pick with the current implementation. And so you need to fix that. We we need to tell merge which one is the most important by looking at that time stamp.
So, you know, it's pretty simple. We'll just add an aggregation to each microbatch. We'll group by the primary key. We'll take the row with the greatest time stamp for each key. We'll explode it back out so it looks the same. You can see what that transformation is doing here. And so we'll add that into our code. And now it's a little bit more complicated. We're up to 18 lines of code. Unfortunately, it's still not correct.
There's another problem. When out of order updates come across different micro batches, now we're not, you know, we're not going to see them in that single aggregation anymore. And so we're actually going to end up picking the wrong one. we're going to end up with stale data in our table and we'll never correct it. Uh and so in order to fix that, we have to do something a little bit more complicated. We actually have to every time we do a merge, we need to store the timestamp for that row in the table so that when we get a new record, we can check to see if that time stamp is greater and only update the row in the table if it is greater.
So we'll go out, we'll kind of expand the merge. We have to do this for every single column in the table. Ends up being a little bit more complicated. We're now at over 20 lines of code and unfortunately it's still not correct. There are still some subtle problems with it. We want to also handle deletes and so often source systems will give you this is deleted flag. So we'll have to go and change the code to do that.
Okay. Well, that seems pretty simple. But unfortunately we actually introduced another race condition when we did that. We now have so we're even at 23 lines of code. There's still a problem with it. The problem is what happens when an update arrives late after a delete. So, a record should be deleted, but now a earlier update comes in. That record is going to be reborn as a zombie. And so, now this row that should be dead is going to be permanently trapped in our table, confusing all of our downstream consumers.
And so, the solution to this is getting a little bit more complicated. Instead of deleting records, we're actually going to soft delete them by recording the timestamp when the record was deleted. We'll call these tombstones. We don't want our records to see these tombstones, so we'll create a view on top that filters them out. And then we'll redirect all of our consumers to look at that view. And then of course we don't want these tombstones to pile up and continue taking up storage forever.
And so we'll have to build a cleanup process that removes the tombstones over time. So looking at the implementation, uh we're going to have to add a few more lines of code. We're going to have to create this view and redirect all of our readers to it. We're going to have to create a separate job that now we have to monitor and make sure it's always working to do this cleanup process. We're at 24 lines of code, one view, one cleanup job.
And that's pretty complicated. You shouldn't have to become an expert in distributed systems and race conditions in order to do basic data engineering. And that's why we created autocd with five lines of code does everything that I just talked about. And the magic trick here is something that we call declarative programming. Instead of procedural or imperative programming where you give the system the exact steps of how things are going to happen, with declarative programming, you say what you want to have happen and the system automatically figures out how to make that true.
And this is the core idea underneath LakeFlow pipelines. So looking at this, it's pretty simple. We're writing into this table called uh you know CDC data from this stream called source data. We tell it that the primary key is ID. We tell it which column is our timestamp. We tell it when to do deletes and that takes care of everything automatically. And if there's some other problems that you might need to add on to this, like for example, it's very common for source systems to save space by only sending you the uh columns that change.
So in this case, you know, we were only getting the name and the first record, we're only getting the city and the second record. And what the system should do is it should combine them to create a complete record. Uh as you can see here and adding that, you know, if you were doing this procedurally is actually quite complicated. You would actually instead of keeping a single version, you would have to start storing a vector clock, one for each column inside of the table.
But with declarative programming, it's as simple as adding one more clause to AutoC. We just say ignore null updates. Still five lines of code. And there's some more features that I actually don't even have time to talk about, but for those of you who need to keep history, if you're in a regulated industry, and it's very important for you to be able to go back in time and see exactly what the data looked like at any period in time, that's often called uh Sedd2, which is where you keep this history.
I would love to explain how that works, but we don't have time in this talk. And really kind of the key idea here is AutoC is over 2,000 lines of code that you're kind of getting for free when you write these five lines. And I think even more important than the implementation is the 10,000 lines of tests that are checking to make sure that it's always doing the right thing no matter what order the data arrives. , can I ask you one question?
Yeah, please. How many handrolled implementations of automatic CD sed2 have you seen? I have seen Oh, hundreds easily. And how many correct ones have you seen? I think apply auto CDC is the only correct one I have ever seen. It's actually like one of my favorite things to go and kind of snipe them now when I see them and point out the race conditions. Awesome. Great. And uh is it fair to say that uh LakeFlow pipelines is overall a declarative framework?
Yes, exactly. So this is one example of declarative programming inside of LakeFlow pipelines. But really we make lots of things from ingestion to change data capture to doing transformations. All of those become declarative when you're programming in LakeFlow pipelines. Awesome. And with that I'm gonna hand it. Awesome. Thank you. Okay, great. All right, so uh we're gonna uh I'm going to go talk about So that was Layflow pipelines, declarative pipelines, and you're going to hear a lot more about it, and we're going to I'm going to take questions.

12:56 LakeFlow Connect for ingestion

Um but I went out of order uh a little bit. Let me talk a little bit about layflow connect. So who here is ingesting data from more than one source? Great. So like every hand went up and let me ask one I promise one more question. Who here is ingesting data from databases? Okay, awesome. Great. Who here likes ingesting data from databases? Like nobody, right? Uh so look the uh layflow connect is how you ingest all your data into one place and it's it's actually the first step of layflow.
That's why I was saying we went out of order because layflow pip declarative pipelines is how you transform it. But the very first step when I said everything starts with good data. Well, you actually have to get the data into data bricks first and that's actually the job of LakeFlow Connect. Now, uh we're going to talk about road map and kind of what's coming in a little bit, but I did want to share this one uh insight that came across from hundreds of customer interviews, which is that ingestion is absolutely undifferiated heavy lifting.
By which I mean nobody likes doing it. It's a necessity. You have to ingest data. You have to do it, you know, securely, safely, at the right cost. Um, but I love writing connectors to ingest data. Said nobody ever. It's like literally a thankless job. Uh, very important for those people who do it, you know. Thank you. But data engineers don't like doing this. Uh, that's because it's not simple. It's actually very complicated.
It's a very difficult job. You have to write the code, which can seem easy. You could, you know, vibe code a connector in 10 minutes, but then you have to deploy it, which means you have to run it. Uh, you have to keep up with API changes. And this is actually very, very difficult. I think Salesforce uh the sales cloud is on API version 40. So some poor person has sat through 40 API changes kept up with them and possibly many more minor minor version changes.
Uh you have to deal with operations. So you have to deal with things like backfills which means hey what if what if I'm missing a day's worth of data? What if I what if the API went down? What if I have to go refresh a a particular table but not all the other tables? That's your job. Now if you're writing the code yourself, you have to monitor this. So you have to monitor both your connector but you also have to monitor the API whether it's a database you have to heartbeat that you have to make sure that it's not going down.
Um and it's costly. All of this adds up to time and money. And uh the tyranny of this is that this is time and money that's being taken away from the actual work we have to do as data engineers which is to help deliver value to the business. Um so connect makes injection point andclick. And when I say point-and-click, I mean there's literally a super awesome UI. I'll show it to you in a minute minute. But of course, we're data engineers.
We like to automate things and it comes with a really powerful API. So you can actually set up your uh sources and destinations and you can script it. So you go from having to write code, there's no code to write. Uh you use an API call or the UI to set up a connector. You don't have to deploy anything. We run everything for you. Uh we run the serverless compute or the classic compute. We take care of uh injection for you.
You don't have to keep up with API changes. So you don't have to care. You don't have to worry about Salesforce, Netswuite, Workday, various flavors of SQL Server and Postgress. We take care of that for you. You don't deal with backfills. And the the cool thing here is that under the covers, LakeFlow Connect just creates DTS or uh now they're known as declarative pipelines. You get the full semantics. You can do full refreshes, you can do partial refreshes, you have retries.
These are triggered entities. So they're actually managed by jobs which is the orchestrator. Uh so you're not you're not reinventing a new thing. Monitoring is already included. Again this is because these are pipelines. They're just standard first class pipelines. Um so you don't have to build anything secondary. So you get cost observability. You get data movement observability. It's just included. And it's not costly for two reasons.
First you can spend your time and money elsewhere. You can go actually do data engineering. Secondly, it's built on declarative pipelines which means that it's incremental and efficient from the get-go. Uh this is important and I'll tell you about kind of two two cases. One is most SAS sources support some kind of incremental ingest. So we do our very best to do incremental ingestion and there are like lots of cases where we have to do tricks on your behalf to keep doing incremental ingestion to avoid refreshes. uh Salesforce formula fields for example and for databases we use the binary log of the database so if you have SQL server there are a couple of ways to do change data capture we do that for you so we're always reading the changes uh whenever possible so I want to spend a little bit of time on this uh this is the current set of connectors that are available uh we are expanding in this is the so the way to think about this is there are three types of sources we want to make available first applications.
So we have Salesforce and Workday and we have previews of Service Now, Google Analytics 4 and Netswuite is in private preview and Dynamics 365 is in development right now. Uh these are the applications for file sources. We have the popular cloud storage sources. Uh we also have Sharepoint which is pretty awesome. So if you have a rag or AI use case, it's very common that your business has a repository. It could be SharePoint, it could be Google Drive, but that's where you store all your unstructured data.
And what we let you do is we'll let you bring that into Unity catalog into a volume. And soon we'll be able to bring in the ales, the access list into Delta Tables as well. So you can basically mirror SharePoint into data bricks and so you get uh you get to get all the data in and then you can do use it for AI use cases. This is super powerful. Um and then you can use it with the AI capabilities. We also have a secure FTP connector.
Uh very very popular in private preview and that's great for many legacy systems. For databases, this list is actually slightly old. You'll see I'll give you a little bit of a sneak peek for tomorrow. Uh we have SQL server in public preview. Postgress is coming. So we're actually announcing the uh private preview of Postgress tomorrow. Uh my SQL is under active development. Oracle is later this year. And you tomorrow you're going to see a lot more logos for databases on the slides.
And what we're doing, the cool thing we're doing here is if you have a database connected into Unity Catalog via Lakehouse Federation, you'll be able to use that from within LakeFlow Connect to do incremental ETL. So the way that'll work is it's going to be cursor based in uh ingest. So you you have to tell us hey here's the updated ad column for example or use this as the um yeah you use this to capture the changes for smaller databases under you know in the in the range of several hundred million uh you'll be able to load data from Snowflake terod data uh Oracle uh it's not binary log but actually unlocks a whole set of possibilities and if you already have it mounted in unity catalog it just works which is pretty awesome so that's going to be announced tomorrow Um, and lots more coming.
Cool. Uh, let's talk real quick about LakeFlow jobs. So, who here is using LakeFlow jobs today? You may know as workflows for example. Okay, awesome. Great. Uh, so uh we're settling on the name jobs for this capability because today we use workflows to create jobs. It's a little confusing. And jobs is an orchestrator. This is actually how you run workloads in production. And I want to talk a little bit about the evolution and to give you a sense of where we're going with it since this is a roadmap conversation.

20:15 Orchestrating with LakeFlow Jobs

So circa 2016 or so jobs was pretty awesome because prior to this, how would you run spark in production? Let's say you have a production spark workload. Well, you have to have some cron script sitting on a virtual machine and you know you you make an API call to a Livy API or some EMR API or data bricks API that spins up a cluster then you have to wait till the cluster is up and running then you submit the jobs to it and all that right how do you monitor this how do you deploy this how do you do logging so that was the first purpose of jobs in data bricks was to be a spe a scheduler for spark clusters for data bricks runtime clusters and that actually made life a lot easier And it took us a little bit of time to come up with multitask jobs.
And these are the workflows where you can have more than one task in it. So now you can start chaining these together. You can create you know uh DAGs of work to be done. Pretty awesome. And that unlocked let's say that was the first time we unlocked uh complex workflows on data bricks because previously you had to use airflow or azure data factory or some external orchestrator because you know most interesting workflows are multistep. uh in 2022 we just spend a lot of time scaling the service.
So no uh no surprise one of the most important guarantees an orchestrator can make is that it itself is up and running. Uh and not just up and running but it's actually able to handle the load of millions of concurrent workflows running on behalf of our customers. Right? And the guarantee that we made was that you as a customer will not notice that you can launch hundreds even thousands of concurrent runs and we'll take care of it for you.
And the thing we had to do here was really make sure that we make you we protect you from the problems of the cloud that if there's an outage in a region, we can go to another region. We can deal with cloud APIs returning errors and so forth. And then you know what have we been up to since then? There's actually a whole lot and I want to show you what jobs has become and where it's going. So today I think of jobs as a complete orchestrator for the data intelligence platform.
And let me let me share some things that we we we've done lately that are super powerful and you may not actually know about. So first is a set category of capabilities what I call advanced orch advanced control flow. For example, this means you can now set parameters at the job level and task level and these parameters are unified. So one thing you may not know this is jobs has this thing called task types. A task is a thing you can run.
It could be a Python script, it could be a DBT project, could be a notebook. So now you can set in a uniform way you can set and cascade parameters across job and it task types and into the code itself. You can do branching in jobs which is super powerful. So the reason we built this is because often in a job you want to take you want to stop and you want to review your execution. So you you may have a declarative pipeline that runs and you want to say hey I want to go and look at the number of rows that were written.
You may have a notebook that runs and you want to you want to inspect the output of that notebook. So this sort of branching and decision-m is extremely powerful and that's available in jobs as well. And if you're not using it, it's very simple. And of course you can you can pipe in task values the output of different uh parts of the of the of the uh workflow uh to make decisions. Then you have looping. Again, super super powerful.
Looping is exactly what it sounds like. If you want to do ex the same thing do lots of notebooks you want to do and you want to run it again and again that's exactly what looping is for and with looping and branching and all these capabilities you get retry semantics which is super powerful what mean what I mean by that is imagine if you ran a workflow and everything worked except for the last thing you really don't want to run all the others uh so with retry semantics you can choose what to run again uh you have task values which is how you pass data from task task to task.
So for example, you can pass along a few kilobytes of state uh along uh the workflow. And finally, this is super cool. We just announced this recently. It's called SQL references, which means that when you run a SQL script, you can actually inspect the result of that SQL script, get the result, and pipe it to the next task. So if you're building SQLcentric workflows, super powerful. So when I look at jobs, you know, this is a far cry and greatly expanded capabilities from 2016 when this this was a you know, its primary job was to bring up a Spark cluster and submit work to it.
Uh it it lets you model most I would say pretty much all potential workflows that you see in the real world. The one thing we don't have today is a sort of a human sort of pause the workflow and contact a human kind of operator or a task. Um after I'm done with the talk, we'll have some time. I'd love to hear from what's missing here. I I think there are a couple of things that are missing. Uh the other thing that we spent a lot of time doing was what we call orchestrate everything.
Uh and by the way, this is a little peak into the new UI uh for jobs. Uh and what's cool about this is that these are the task types that are now available. There are notebooks, Python scripts, DBT projects. Tomorrow we're announcing that we can orchestrate PowerBI. Um, and we're orchestrating DBT cloud as well. And in fact, the reason why I like this is there's so many we're running out of space. Um, so that's pretty awesome.
Uh, one of the very common things that I hear from customers is this is fine, but I want to orchestrate a REST API. Can you do that? And actually, we can. And if you're using serverless compute, a serverless notebook is a great way to do that. So typically uh the code to reach out to a REST API, get the data back, parse it and do error handling is pretty small. You could use the assistant to generate it. The bigger problem has been I don't want to spin up a large Spark cluster with a driver and wait for it to come up if I'm making a quick API call.
And that's exactly why you should use serverless compute. So you can use you can use a notebook with serverless compute. Spin it up. It spins up within seconds. Do your API call, shut it down. So with that, you know, pretty much everything is in within the realm of possible when it comes to orchestration. And we're we're going to keep adding uh one actually, I don't know if you noticed this, but at the very top, you also see these icons for LakeFlow Connect.
LakeFlow Connect is also now integrated into job. So from within your job, you can add data and orchestrate it. Uh so pretty cool. I want to talk a little bit about I just called it scale scale scale. Uh first of all, serverless compute is really big. So we spent a lot of time about a year and a half to build the capability to integrate serverless compute into data bricks jobs uh into lakeflow jobs. Of course classic compute is fully supported.
You can keep using it but now you have serverless compute. There are two modes of serverless compute. One mode is performance optimized which means it comes up really quickly comes up in order of seconds and it scales up really quickly as well. And this is suitable for you know certain workloads more let's say more interactive like workloads. we have a strict SL SLO you want to meet. But there's another mode of serverless which is called standard mode.
You basically turn off the per performance optimized mode. And then the spin up time for standard mode serverless compute is about 6 minutes. It's anywhere between four and six minutes which is sort of like a classic compute cluster but it's much cheaper. And so this way you can choose you can choose between freshness and latency. Uh and the great thing about serverless compute in jobs is that it's a way to kind of offload infrastructure.
So if you find yourself today thinking about clusters, cluster policies, what's the right size, is it at the right DBR version, serverless compute and jobs automatically upgrades. So we take care of the versions for you. We take care of the infrastructure. We take care of the scaling. So if you haven't tried it, it's pretty awesome. And this is actually a way to scale your organization. That's why I included in the scale scale scale bucket where you you know you may not have enough people enough experts who are you know great at managing clusters again to get you back to uh doing data engineering.
Um and there are there are several uh sessions on serverless compute uh so I won't dwell on it too much. The second thing which I'm super excited about is task queuing. So what would happen in the past with jobs is that uh there was a limit. It's a service. Every service in the world has a limit and if you try to run too many tasks it would say I'm sorry 429 back off and it wouldn't run your job which is very disappointing.
Uh so we spent a lot of time building task queuing and you can enable task queuing. I believe it's turned on by default now uh but you may want to uh take a look. Uh it's in the right sidebar and with task queuing what happens is that we hold on to the task for you. So if if if you've reached that limit and the limit is actually quite high. I think it's more than a thousand concurrent runs. We can just hold on to that and we'll run it when the queue empties up.
We've spent a lot of time just raising the limits on the number of tasks in a job. For example, I think you have more than a thousand tasks in a job now. It used to be 100. So you can truly build large workflows and we'll render them for you so you can actually manage them and then we can run them for you. DevOps is big and this is one where a lot of people don't know how much we've done uh to make layflow jobs friendly for people who are coming from the world of airflow, Daxter, Prefett really people who want to write their jobs as code and then manage the life cycle through CI/CD.
This was actually really hard several years ago. Uh so let me talk about two things. So first of all now you can write your jobs using Python code. Pretty awesome. You can express that this is a in Python code and uh if you're coming from the world of airflow it's quite similar super powerful then who here is using database asset bundles okay a lot of hands went up so if you don't know what database asset bundles is a way to express the objects that are deployed jobs pipelines notebooks that are deployed into the data bricks workspace so it's it's config driven so you can now integrate your job as code with data bricks asset bundles and have testing and CI/CD all along the way.
This is a great way to as you start maturing, as you start building larger and larger jobs, it's a great way to make sure they go through stage gates and have quality. So, the full DevOps life cycle is completely possible with jobs now. So, one thing I wanted to do, and we'll do a super quick thing here, is I I I spent a lot of time on jobs, spent a lot of time on connect, and gave you a little preview of how we're renaming um DT into declarative pipelines.

31:00 Demo and Q&A

Actually, I promised a demo in the session. I'm going to give you a demo. And what I'm going to show you is uh this is our dog food environment. So, you know, fingers crossed. I'm going to show you how easy it is to create a declarative pipeline. So, who here has created a DLT before? Okay. Uh, so if you created a Thank you. And if you created a DT, you're probably familiar with the old notebook based experience. And you know, notebooks are good.
They're good for exploring data. But for a declarative pipeline where actually the whole thing gets analyzed and run as a whole, the imperative model of notebook didn't work really well. So, I'm super excited. You're going to see this tomorrow. There's going to be a big demo for this, but you guys are all here, so I'm going to show it to you first. is uh we have a brand new IDE for data engineering. It's purpose-built just for building declarative pipelines.
So, I'll just do a live demo and hope it all goes well. So, you go click new and you click uh create declarative pipeline and I'm going to call this taxi analytics. Um so, the whole thing behind this editor, I'm going to hide this real quick. The whole thing behind this uh IDE is that we wanted to give you a collaborative UI first way to get started to make it super easy to get started with data engineering. Uh make it not daunting but also to give you all the data engineering is sort of unlike other disciplines.
Uh you care about things like tests, you care about things like data sampling, you care about things like quality. You wanted to make sure that this is a purpose-built editor uh just for data engineers. Uh so let me show you real quick. I'm going to call this taxi analytics because I'm going to be working off of a sample taxi data set and I'm going to choose uh just this one and uh I'm going to just say hey just give me some sample code in SQL and of course this is declarative pipeline so you can mix SQL and Python.
All right so over here you have your file tree in the IDE. Uh you can organize this is convention driven but you can organize your code the way you like. Uh this is not going to be in tomorrow's uh keynote, but we are adding testing. So very soon you'll see a native testing framework for declarative pipelines and a test runner and testing is going to be fully integrated into this IDE. And what I have here is also a pipeline graph that's going to show up here.
I'm going to run the pipeline real quick, but let me show you the code. So what I'm doing here is uh there is a data set uh in our test environment which is for uh taxi pickups and drop offs in New York City. Okay. And what I'm doing here is I'm creating a pipeline which computes um zip codes and the fair amount and then also computes the uh total fair. Okay. So what I want to show you here is that this is this is declarative programming, right?
So, whoops, sorry about that. Uh, it's basically if you're coming from the world of dbt, it's queries. You tell the system, here's a transformation, and we take care of it for you. Um, and this is our dog food environment. So, oh, there it is. So, the serverless compute is coming up. This is available with classic compute as well, but I'm using serverless compute. So, I'll show you the graph as soon as it has rendered to show you uh the new editor.
Give it a sec. There we are. Okay, perfect. So, what I'm doing here is I am I'm going to use the graph to navigate. So, okay, I am reading I'm just doing a select from samples. nyc taxaxi. trips. And the graph here showed you that it read that it read 22k records. Right? Then I have a follow-up transformation which is this one which is then reading from the previous table and it's just doing a group by unfair amount.
This is obviously a toy example. Uh, but notice that as I click on this, I see data samples at the bottom. And this is one of the big pain points with de with de developing DTS, which was, you know, you had to you wrote your code and you had to kind of walk away until sample data appeared. This is super snappy and serverless. Um, you get table metrics and performance built in. So, you get query profiles integration. So if anything is ever slow, you want to see what's going on is fully integrated.
And of course, I don't have this integrated with git, but git is integrated. Database asset bundles are integrated as well. And scheduling is integrated here too. And of course, you can manage all your settings from here. Um, so what and one thing that is kind of cool is I want to show you this and this is live demo. I'm going to show you the assistant integration. I'll just say add data quality checks. Perfect. And I'm almost done with the demo because I want to leave five minutes for questions.
Oh, that did not go well. Sorry. This is why it's a live demo. Maybe maybe I do something like here. Add data quality. There it is. Okay, perfect. So, that worked. Uh, so we're still working on the assistant integration. Uh, but generally the assistant is much better at understanding this code and generating suggestions for you. Um, okay, awesome. So, this is the new IDE for data engineering. Uh, this is how you build LakeFlow declarative pipelines.
Let me stop here. We have a little bit under five minutes. I want to make sure we have time for questions. Uh, happy to take any and all questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.