The Lakeflow Effect

Summary

• Lakeflow unification: Lakeflow brings simplicity and unification to Databricks' data engineering experience, consolidating pipelines, jobs, and data connectivity.
• Lakeflow components: The platform offers Lakeflow Connect for data integration, Lakeflow Pipelines for transformations, and Lakeflow Jobs for orchestration.
• Serverless compute: Lakeflow leverages serverless compute capabilities, simplifying infrastructure management compared to traditional Spark performance tuning.

The Lakeflow Effect

Talk by Bilal Aslam, Sr. Director of Product Management, Databricks ; Josue Bogran, VP of Data + AI Architecture, JosueBogran.com & zeb.co
Lakeflow brings much excitement, simplicity and unification to Databricks’ engineering experience. Databricks’ Bilal Aslam (Sr. Director of Product Management) and Josue A. Bogran (Databricks MVP & content creator) provide an overview of the history of Lakeflow, current value to your organization and the direction its capabilities are going toward. The session covers: What is Lakeflow? Differences and similarities between DLTs Overview of current Lakeflow Connect, Pipelines and Jobs capabilities How to get started What's Next? The session will also provide you with an opportunity to ask questions to the team behind Lakeflow.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Lakeflow?

Lakeflow is Databricks' unified data engineering solution that brings simplicity and unification to engineering workflows through integrated pipelines, jobs, and data connectivity capabilities.

What are the main Lakeflow components?

Lakeflow includes Lakeflow Connect for data integration, Lakeflow Pipelines for data transformations, and Lakeflow Jobs for workflow orchestration and scheduling.

How does Lakeflow relate to Delta Live Tables?

While Lakeflow and Delta Live Tables share capabilities, Lakeflow provides a broader unified engineering platform with Connect, Pipelines, and Jobs working together seamlessly.

What's the advantage of Lakeflow's serverless compute?

Serverless compute eliminates the need to manually tune Spark performance, provision clusters, or manage infrastructure, enabling teams to focus on data workflows instead.

Full transcript

All right. You want to kick it up? Yeah, let's do it, man. All right. So, thank you so much everyone for being here this morning. Uh, to Below's point a little bit earlier, it's 8 a. m. So, I think about it that there's three different reasons why you might be here this morning. One of them is you're new to data bricks and you want to understand lakeflow a little bit better before you know he covers a lot of things during the keynotes.
Second reason you might be here today is well you want to know what the heck is happening to delta live tables and how does it all play with lightflow. Third reason is you just needed an 8 am session so that you can expand some coffee at work and either one I hope you found you find this a little bit helpful. So I'm the VP of data and AI architecture at SEP which is a consulting company. I'm part of the data bricks MVP program which is like a ambassador program thing.
I'm part of the product advisory board for data bricks and I'm an adviser to a few other companies like sigma with me here. I have the one and only. Yeah, the one and only. Well, actually there's like five bal at data bricks now and I have a document which explains the differences between the bal but please continue. Is it in Excel though? Uh no it's a it should be. Okay. Okay. It's not uh so but I have the one and only Bal Aslam here who is the senior director of product.
Bal can you can I give a little intro and overview of what you do? Absolutely. So uh thank you. I've been at data bricks for about six years. Uh I was a customer of data bricks twice very early days you know the bad old days of hive meta store raw parquet files and uh tried to do some pretty crazy things with this platform very early uh I look after what is known as lakeflow our unified data engineering solution. I also look after our compute platform so serverless compute.
So if you want to throw some tomatoes at me or if you have questions about serverless very happy to take that. Uh my claim to fame is that uh I was one of the I was a customer of data bricks when this c company was 50 people. So I was once on a call with who's our CEO and I was complaining deeply about you know how hard it was to understand performance in spark and you was listening and nodding along and all that. Then when I joined data bricks years later he sent me the call notes and he said now go fix it.
I was like gosh darn it I have to go do this now. Um but yeah I'm excited to be here. No. Awesome. And well, we're going to start off with like I said, uh there Oh, this warning here. Don't chat GPT is down, but please forward-looking statement. We're not making promises about features and road maps and such. So, let's start it off with one of the questions that at the beginning some of you were wondering. It's really what is Lake Flow?
So, Blah, take it away. All right. So I'll I'll answer the question directly. So LFlow is unified data engineering from data bricks and uh I'll give a little bit more color to the why. Um so layflow is actually three things under this unified product umbrella. It's layflow connect which helps you get structured and unstructured data into data bricks. Uh and the point of layflow connect is it's pointandclick obviously has a great API so you can start bringing in data from everywhere.
And the thing to think about late for connect is nobody actually wants to spend time ingesting data. That's like pure commodity undifferiated heavy lifting. You need the data. You want it to be at the right price point. You need it to be efficient. Like no data engineer wakes up in the morning says, "Ah gosh, I'm so excited that I need to I get to write this connector for this random API and have to evolve as the API evolves."
Right? Nobody wants to do it. Uh and if you do want to do it, like tell me because that's crazy. Uh so that's late for connect. Um then once you have your data in data bricks so you have bringing on all this data. Sorry B you want should we how about here I'll I'll give you the No that's okay. Oh there's slides. Oh yeah. Okay. Uh so uh so connect has this notion of connectors. No surprise. It's pretty obvious. And a connector can connect to a source.
And the key thing is it can incrementally load data. So it actually knows how to load just the changes whenever possible. That's not always possible but it tries to do its best. So as your connector uh your pipeline runs, it's overall, you know, cheaper than you just doing it yourself or you doing a full load every time. Uh there's a little bit of thing here I want to talk about which is first of all there's like many different types of sources.
There are API sources but they're also databases and databases have been around forever. They're going to outlast us. Uh they're the most critical piece of infrastructure in the world. And as a result, the way I I used to work on operating systems a long time ago, databases are actually the order of complexity of oper operating systems. They're super old. They have lots of versions. There's this thing called the binary log.
And if you can read the binary log of the database, you can pick up just the changes. So the real magic here is not just the SAS connectors. It's the ability to read from SQL server and like the eight different versions of it out in the wild right now to read those changes. Same with Postgress, right? So that's connect. Can I help you get all the data? Uh DT is about efficiently transforming, cleaning, aggregating, aggregating and reshaping that data.
We'll get into more of that later. And then jobs is about orchestrating it. It's an orchestrator. So if you use Airflow, you've used Daxter, Prefed, Azure Data Factory, it job is to run workflows, keep them running, retries, alerting, things like that. So yeah, and I was going to say if you notice, if you go Thank you very much. Yeah. If you go back to this, you'll oh sorry I keep talking. You'll you'll see that I put in there a star in there and that the reason is LakeFlow is going to be great for a lot of use cases, not every use case and data bricks is aware of that and we'll get a little bit more into that uh a little bit later on in the presentation.
So Bal, I think you covered a lot more than we practiced there as part of the details. So I'm just going to get more details out of you on this. So absolutely let's go through each of those right now. So yeah talk a little bit more about connect. Okay great. So uh so the genesis for connect was so who here uses autoloader? Okay like quite a few hands. And if you don't know autoloader is a data bricks capability where you just point it at a source of files in say cloud storage like S3 or GCS or whatever.
It just slurps them up into delta tables and now you can use them for the rest of your business. Right? So we obviously have a lot of customers who use autoloader because again who wants to manage that ingestion themselves. How do you incrementally load it? How do you evolve schemas and all that? So connect is think of it as the evolution of data in ingestion at data bricks. I won't belver the point. We designed it from upfront from the get-go to be pointand-click right uh we talked to several hundred customers and just uniformly every customer said give me an API but also give me a UI.
And the key insight there is that there actually more people than just the data team that need to ingest data. So the cool thing behind connect is today if you go into new add data you'll see a couple of tiles. Uh and you're going to see more soon. Uh you're going to see the ability to slurp up data into data bricks in many parts like in the future in Genie in AIBI dashboards like anywhere where I'm working with data pretty much you'll see connect kind of as a add data button.
So like in your businesses uh in your companies you'll have hopefully fewer people coming to you and say hey how do I load data from here and there because they'll just have that capability. Um yeah I think that's covers connect. I think so. Yeah I'm sure you'll cover it more much more in detail and the teams are going to cover much more in detail throughout the session. Uh yeah so DT I think there are a lot of people who are familiar with it but I think uh so DT is a declarative ETL framework and by declarative and declarative systems are really powerful because unlike procedural or imperative systems instead of telling the system exactly the steps to execute think of SQL right when you write a SQL query you don't tell data bricks or any warehouse like you know here are the exact steps you have to perform to satisfy my query you ask it what you want and it figures out like it there's a query optimizer, there's a scheduler, there all these different pieces the system takes care of it for you.
So that's a declarative system and the key thing here is that DT is declarative. So you tell it what to do the transformation to make that transformation is a SQL query or a Python dataf frame operation. Uh and the key thing here is that if you do that and if everything works well then you just focus on that part of your workload. you just focus on reshaping the data instead of figuring out how do I materialize it? How do I scale the infrastructure?
How do I autoscale this and that? There's a lot of goodness that comes from it. There's a lot of like tricky things that come from it too. Well, I was going to say and like the way that I think about it is making it making pure data engineering feel a little bit more like the with the rules of software engineering. That would that be a first statement? Yeah, I think so. um the so yeah so I guess this is I'm very happy to talk about this because it's like one of those really powerful paradigms but also it's like to point it doesn't do everything so there are things like if you want to call an API that is not a declarative operation if you want to run a for loop and you know write stuff out to disk that's not a declarative operation so kind of my my thinking has evolved here which is that spark is super powerful it's like the Swiss army knife of the world it's the world's most popular data framework And Spark is great for everything.
You can do literally anything with Spark, right? Like you can do I don't know genetic algorithms if you want. Uh and then jobs. I think everybody like does there anybody here who doesn't know jobs? Pretty much everybody should know jobs. It's an orchest Oh, that's a hard question. Let let me let me rephrase this. Yeah. Who here has not used jobs before inside of data bricks? Okay. Thank you. Yeah. So jobs is an orchestrator.
So if you're familiar with Airflow, think of it as a much simpler point andclick but also scriptable and automatable orchestrator. Um it automates everything in data bricks. It scripts uh it orchestrates SQL, Python, notebooks. Um and this is sort of the Oh, this is the wrong screenshot, but that's fine. But basically, you build a DAG, a directed asylic graph of tasks, work to be done, and it runs them reliably. Yeah.
Yeah. Sounds good. And yeah, and in all fairness, this is so interconnected with everything else, it's even hard for me as we were putting this together. I was like, well, where do you really draw the line with it? Because it's so integrated to everything else. So, with that said, uh I think we had a little point there, end to end observability, and I call out, if anyone knows me, I like to joke about Excel. I call in there that it doesn't track when something gets exported to Excel, at least not yet.
Um, so that's a very high level overview of LakeFlow. Now, that's great if you're new to the ecosystem or first time you hear about LakeFlow, but for those of you that I'm going to take this poll again. How many of you have worked with Delta Life tables before? Well, can I ask a bold question too? Please, by all means. How many of you have not enjoyed working with Delta Life Tables? Please be be honest. We're like friends here.
Yeah, I'm one of them, guys. So, just just so you know. Okay. Yeah. Um, so so with that said, how does LakeFlow fit in with Delta Life Tables and vice versa? Yeah. Uh, so yeah, so Delta Life tables is the or DT is the declarative ETL portion. Uh, and by the way, like I'll take some wild guesses on the parts that this gentleman, you know, and by the way, I am a heavy user of DT and like I I started using it more and more a year ago and I was just like, "Okay, we got to fix this.
We got to fix this." So, the things I'll just tell you about what we fixed, okay, because there were some parts that were like, "Oh, come on, man. It's got to be easier." Uh, so I'll start with uh the thing that come up all the time. The biggest thing was like, hey, like incomplete compatibility with Unity Catalog. So, specifically, a lot of customers are like, "Hey, I have a pipeline. that's awesome. Can I write to different schemas in UC?
We were like, no. So, we fixed that. So, that's like generally available now. It's single. And by the way, if you think about it, it's like a really obvious things, but we fixed it. Um, the second category of issues that were were kind of these delta table format incompatibilities. So, things like, oh, can I do liquid partitioning on this table? Like, no, you can't. Or, you know, can I do alter columns? Like, nope, can't.
There's like a whole class of these. We fixed all of them. So, they're gone. So if you think about it from a data intelligence platform, it's like now you should have been there from day one where it's it's fixed now. So it's fully compatible with Unity catalog has almost every single delta feature in the world enabled on uh the data sets it creates. Then the third thing that came up a lot was hey the development process is like slow and buggy, right?
I actually have a slide for this. Let me There you go. There. Oh yeah. So there we go. Oh, you want to talk through this? No, no, no. You're you're doing a great job. Go ahead. Okay. Yeah. So this this was like this t this tough error handling. So you were building it. You would previously build DTS in a notebook. That sounds great, but actually there's a little subtle trick there which was that these because it's a declarative system.
You would write a command, you would run it, it would say okay, you actually have to run the pipeline separately. The pipeline actually compiles and runs as a unit. It was super confusing. So we fixed that and you're going to see a super awesome. We're calling it the IDE for data engineering. It's a brand new purpose-built IDE in the workspace with files and file structures and tests and everything and it's just designed for building pipelines, right?
Again, it should have been there from day one, but it's pretty awesome. And the other thing was like the execution. Oh, go ahead. Actually, it's available already. It's already in beta. So, you have to ask your account team uh to enable that beta for you. So, when when I say it's beta, it's not in every cloud right now. So, we're rolling it out. So, it's super cool and you'll never go back. So, we're seeing like 96% retention on this new IDE.
Like, you use it once, you'll never go back. And what's cool about this is that the we also spend a lot of time making the inner loop faster, the development loop. So, the time from when you type something and you run it, it could be minutes sometimes. It was like go get a coffee. Now, it's like order of seconds and it's getting faster and faster. We're rolling out some major changes. The errors are much better. the errors would be obscure because this things DT sits on top of Spark.
Sometimes you would see Spark stack traces which is nonsense. So we fixed almost every one of them. If you find one, send it to me. I'll fix it. Uh we rewrote all the documentation for DT like from scratch. We wrote like 50 plus articles. We like reorganized everything. Oh, the other thing is we integrated it completely with the assistant. So now the assistant actually understands some of the DT nuances and how to generate code for it.
Yeah. So, yeah, to give you a quick quick more backdrop again, like I already shared a little bit, I've been an anti-D person for the last three years, basically since it got released. So, uh so I don't take it lightly when I tell you that for the first time in three years, this is uh something that I actually would use myself. So, I don't take that lightly saying that. So, it's got a lot better. Uh I got to share a lot of feedback with and a lot of the a lot of the team behind the scenes and they were literally telling me telling me tell us everything that you think is wrong like we we we we want to know it because then they can prioritize uh the most impactful things and I would say 85% of the things that I called out have have been addressed and so it's very very promising where it's at today.
There there's one other thing that we did change which will not be in the keynote or anything like that but it's going to be communication to our customers. We're actually dropping the prices on it a good amount. So specifically the serverless part of D. If you run DT pipelines on serverless compute which means it spins up really fast. It autoscales really quickly. You get the enzyme incremental ETLs. It'll automatically figure out changes in data and only use those.
We drop prices on DT which are rolling out today throughout the end of the week through the end of the week where the serverless is now competitive with classic on spot. So like for in ingestion transformation change data cap it's like super competitive just actually and like it's also super competitive with jobs you can like use the declarative system you don't have to think about the cost anymore as much so that's the other thing we haven't really talked about now I'm going to take a quick segway so right now we just talked a lot about the the things that were troublesome with DTS that the team has addressed uh to the credit to it's not it's not all bad things and where things were at so I actually want to give all the opportunity to share some of the things that really the reason is why didn't and I'm going to go back to the slide there why not start from scratch if DT had this different issues why not start from scratch and Bal if you want to go ahead and share a little bit of here's some of the good things yeah um so a few things right so the big the big win with DT is just time savings so the thing that I I talked to like hundreds of customers who use DT and there's like one theme that came up again and again it's like man if you fix these bugs and these limitations.
It just like it's like adding a couple of data engineers to the team. It's like it's that powerful because I don't have to worry about infrastructure. I don't have to worry about orchestration. The there's three things that came up really well. One was you no longer have to think in terms of batch or streaming. It has both. So for example, uh if you're doing batch comput, you can actually mix and match batch computations and streaming computations.
And the semantics are like it's like literally you do a read stream versus a read. That's it. Uh you can tune latency versus cost. So you can have a streaming pipeline and run it in triggered mode so you can control the cost, right? You can run it every hour, every 20 minutes or whatever. So it's pretty awesome. So like that unlocks a set of capabilities for your business because you typically have like a small number of low latency use cases and a lot of these batch use high latency use cases.
You can mix them you can mix and match them on the same platform. And when you look at something like DBT, you can't do that, right? So that's just like a platform limitation. It was never built for that. So that's super powerful. Less code. There's like no extra code for it. Data quality is an area where something like half of our DT customers use data quality. I think we need to do a lot more here. I'm excited because right now you can express your data quality intent.
These are basically role level checks. I want to go make this aggregate checks. I want to give you like out of the box dashboards. I want to give you like you know ability to take action. the like the holy grail for me which I would love to do is to actually have a quarantine and reprocess flow. So when you find bad data, put it away then give you a way to reprocess it. I would love to do it. We have all the building blocks.
I literally had someone ask me yesterday like hey this is literally that scenario. Absolutely would love to do it. That's like on my radar. Then finally change data capture is big right? So this is uh slowly changing dimensions type one and type two. It actually is magical. It's like instead of thousands of lines of uh brittle scripts, it's like one line of SQL or Python. On this one, we greatly improve price performance.
So at scale with our largest customers, they were saying, "Hey man, this is getting really expensive." We found a bunch of performance problems. So it's actually much more performant now and you don't have to kind of manage it yourself. So this is a very common use case. Most companies have something where they need to maintain type one or type two dimensions. It just automates that one line of code. Uh so these are the good parts.
There's others. Yeah. Yeah. And I mean for me this is the recurring theme like I said for me personally I did not enjoy DTS for the last three years. That has changed. But I kept wondering like where where's the value? Every time I would ask folks these are the things that they would call out. This is what DTS does very well and the reason why so many still shows it and I think part of the reason why data bricks kept it around.
So now, like I said, I want to make sure we have time for Q&A. So how many of you use third party tools inside of data bricks um for orchestration, ingestion or transformation so like your five etc. Okay. So with that is I guess Bilal well actually no I'll speak to this part and that is I asked this question to Bal more than a few times. Uh I got the opportunity to work behind the scenes with helping lake flow but on top of that part of what I do really as a business sometimes is uh advise uh ISV companies that are partnered with data bricks and sometimes that can create a little bit of a conflict right because I want I want both to win.
So one of the questions that I asked Bal and I asked from that works with ISV with ISV teams too is how do you deal with a with a competitive nature of it? And below the way that he put it in an interview that I did with him is we want data bricks to have as much value out of the box as possible which is part of the value propositions for data bricks that I talk a lot that I talk about a lot. Data bricks offers a lot out of the box.
It's not a marketing pitch. This is really my my genuine conviction as to the value of data bricks versus other platforms but bilal I guess can you put it a little bit more in your words uh how do you deal with the overlaps absolutely yeah so uh you know we can take very concrete examples right so let's say data ingestion there is a rich set of partners out there there's five trend there's matilian there's yeah rivery there's many many more right uh if you just think about it from a company standpoint right like data bricks is not going to be in the business of building hundreds and thousands of connectors.
That's a company that builds connectors, right? Um so it's exactly what said which is there are certain abilities that the market has come you have come to expect out of the box in data bricks. Uh hundreds of customers told me hey I Salesforce is such a common data source almost like cloud files like I don't want to go use something else. I don't want to buy another contract for this. uh but you know but then take a consider like some regional CRM right like will we build that very unlikely so the way I think about this is exactly right which is hey for the very common sources for ingestion we'll have the capabilities in the inbox and we'll make them hyperefficient at scale so we'll make sure that those ones are absolutely topnotch um and for and by the way you should also feel free to go to other vendors right so five has great connectors they're battle tested so you're like I'm on five that's awesome and The rest of my job is to make sure they work really well.
Yeah, has a great relationship with prophecy. Yep. You have that choice. We could look at DBT, right? So, if you're using DBT, please like keep using it on data bricks. Like, we'd love for you to be using DBS SQL. We don't want you to have to do two migrations from DBT and uh over to, you know, DT or whatever. Um, in fact, with DBT, we're kind of going the other way. You can use materialized views and streaming tables inside your DBT project.
So you can some get some of the benefits of incremental ingestion and materialized views which are sort of views that are maintained for by the system right so we're like kind of going the other way and saying hey yeah use this partner tool please like we'll actually make that tool better uh and you know for jobs right jobs is awesome but if you're using airflow keep using it like there's an airflow operator set of operators that we maintain um and sometimes regard realistically too sometimes you have other platforms besides data bricks and sometimes there is value in being able to orchestrate things from a centralized location outside of data bricks and that's a valid use case.
There's trade-offs just like with anything. So yeah, so we're definitely not interested in sort of restricting your choice. You have that choice. The out of the box has to be great though. You can just get started, get work done. Yeah. Um so like I said, want to make sure we have time for questions. So allow 30 seconds. How to get started? Yeah. So LakeFlow is GA. So you can start using LakeFlow connect for data ingestion.
You can start using DT for uh you know transforming and reshaping your data. Jobs is available so please start using it today. There are certain features that are in beta or preview. So ask your account teams please ask them work with them and they'll guide you and turn the right things on for you. But uh yeah just get started with it. All right I think we are just about out of time here. Um I would encourage you think first off thank you so very much everyone for being here.
Thank you. Uh I don't take it for granted that you took the time this morning to be here with us. Uh if you want to connect with, come to him. If you want to connect with me on LinkedIn or anything, also happy to do so. Very. Thank you so very much for your time. Thanks everyone. Thank you.

Learn more about the Databricks Data and AI platform.