LakeFlow Designer: Production-Quality ETL with No Code Required
Summary
- No-Code ETL: LakeFlow Designer enables data engineers and business analysts to collaborate on production ETL pipelines using drag-and-drop visual tools, no SQL required.
- AI-Assisted Transformations: Databricks Assistant helps classify, translate, and transform data using AI, making it easy to prepare datasets for analysis.
- Built-in Path to Production: Visual pipelines run as standard LakeFlow Spark Declarative Pipelines with unified governance, eliminating handoffs between teams.
LakeFlow Designer: Production-Quality ETL with No Code Required

This video introduces LakeFlow Designer, a new tool that aims to bridge the gap between data engineers and business analysts. It demonstrates how LakeFlow Designer simplifies collaboration, provides a built-in path to production, and uses AI to boost productivity. The demo includes importing spreadsheets, transforming data with drag-and-drop, translating and classifying data using AI with Databricks Assistant, specifying pipeline transformations with examples, and deploying a visual pipeline to production.
Note: Databricks Lakeflow unifies Data Engineering with Lakeflow Connect, Lakeflow Spark Declarative Pipelines (previously known as DLT), and Lakeflow Jobs (previously known as Workflows).
Chapters
0:00The data collaboration gap1:41Introducing LakeFlow Designer2:20LakeFlow Designer demonstration7:43Deployment and monitoring8:23Key benefits and conclusion
FAQs
What gap does LakeFlow Designer fill between data engineers and business analysts?
Traditionally, business analysts build pipelines in spreadsheets and hand them off to data engineers, who rebuild them using code. LakeFlow Designer eliminates this handoff by providing a single unified tool that both roles can use directly.
How does AI assistance work in LakeFlow Designer?
Databricks Assistant uses AI to help classify and translate data, making it easy to transform and prepare datasets. You can specify pipeline transformations by providing examples, and the assistant learns from your patterns.
Can LakeFlow Designer deploy pipelines to production?
Yes. Visual pipelines created in LakeFlow Designer run as ordinary LakeFlow Spark Declarative Pipelines. They have a built-in path to production and are fully integrated with all Databricks governance features.
Does LakeFlow Designer support importing data from spreadsheets?
Yes. This video demonstrates importing spreadsheet data directly into LakeFlow Designer, which you can then transform visually and deploy as production pipelines.
Full transcript
0:00 The data collaboration gap
We've all seen just how powerful LakeFlow is. It's ready for your most demanding production data workloads. Now, when you hear production data workloads, you probably think of data engineers. Data engineers are experts at building scalable, reliable production pipelines. But the truth is data engineers don't work in isolation. Nearly every analytics project is a collaboration between data engineers and business analysts who have a deep understanding of business needs.
These two groups need to work together. But they often live in completely separate worlds when it comes to the tools they use for their work. Data engineers work in the data intelligence platform using products like LakeFlow to build robust production pipelines. Business analysts typically work in spreadsheets and similar analytics software. Every organization needs to connect these two worlds. Now, we're not the first company to notice this need.
There's a vibrant ecosystem of products in this space that aim to connect the business analytics worlds with data engineering. These products deliver tremendous value and many work really well with data bricks which is awesome. But even with all these fantastic products in the ecosystem, we still see gaps to be closed. The first challenge is that workflows are siloed. Analysts will build something and then hand it off to a data engineering team that often needs to rebuild it a second time with a completely different set of tools.
Or some teams may maintain many different systems in production, but that erodess the benefits of unified governance, increasing complexity. And if you're working outside the data intelligence platform, AI productivity is limited by having only a partial understanding of your full data. Now, what if these weren't separate worlds? What if data intelligence could bring them together? Now, we think that it can. That's why today we're announcing Leglow Designer, production quality ETL.
1:41 Introducing LakeFlow Designer
No code required. I'll show you LakeFlow Designer in just a minute, but first, let's talk about how it closes the gaps that I mentioned earlier. Sharing and collaboration is drastically simplified. There are no handoffs and no rewrites. Everyone can use LakeFlow and there's a built-in path to production. All visual pipelines run as ordinary LakeFlow SQL designer is completely integrated with all the features that Bal just showed you.
Because Lakeflow Designer is built on the data intelligence platform and understand your business, AI productivity features are also even more effective. Okay, I've talked about why we're building LakeFlow Designer. Next, I'm going to show it to you. We're going to go through a simple example of using designer to build a pipeline where I'm going to show you a few things. The first is importing spreadsheets and transforming data using simple drag and drop.
2:20 LakeFlow Designer demonstration
Next, we'll translate and classify data using AI with the data bricks assistant. We'll specify pipeline transformations completely using examples and then we'll deploy a visual pipeline to production. All right, let's get started. Okay, here we are in the LakeFlow designer canvas. We're going to build a pipeline that measures customer support efficiency. What we want to know is if support representatives with more experience have a higher rate of resolving customer calls.
So that's the question. Does more experience mean more resolved calls. Now it turns out the employee data that I need is outside of data bricks in a spreadsheet. And bringing a spreadsheet into designer is super simple. Drag and drop. Immediately I see the data which is imported as a regular Unity catalog table. Now here I've got the employee information for each customer support representative 0 to 3 months 3 to 6 months or more than 6 months on the job.
All right. So we brought in our spreadsheet data but we need to combine it with the call transcripts already in UC. Now add a data source node to the canvas here. Select the transcripts table. Let me give this a name. Transcripts. Okay. So that's our transcripts data. Now we can see here that each row has a transcript, an employee ID for the support agent, and a duration of the call. Now, this data isn't exactly what I need.
First, I need to know whether or not the support rep actually resolved the customer issue. And second, I see there's several different languages here. So I'm going to ask the assistant, translate transcripts label is resolved or not resolved, and compute percentage by employee. Okay. goes off and thinks for a few seconds and then we see two new operators here on the canvas. A transformation and this percent result aggregation here.
Let's see what the assistant did. The first step is translation and classification. I see the translated results over here. They look reasonable. We've got the translated transcript and a classification of whether or not the support call was resolved. Now, what's happening here under the covers? Well, the assistant uses our new built-in AI SQL functions to translate and classify these transcripts. But I didn't need to know any of the details about how to use these functions.
Now, let's look at the second operator counting the percent resolve. You know, I've got the resolve cases here, but what do I see here? I've got uh values like 1% 3% 1%. These support agents, they all look terrible. I know it's not that bad. Uh, okay. Now, what exactly happened here? Now, as I'm sure everyone in the audience knows after using Genai tools for a few years, they aren't perfect. Mistakes happen. And so we're building Leglow Designer to make it easier to recognize and correct those mistakes when they happen.
One example is what we see here showing each step of a pipeline with its input and its output so that you can spot check suggestions. So another example is operator summaries. If I hover over this suggestion, I can see a generated description of the operator. The summary here is percentage of all transcripts resolved by that employee. Okay, we can see what's wrong. This operator is calculating the percentage of all calls that a particular employee resolved.
And we can also see that shown in the SQL on the right. Now, language models are very smart, but it's also true that English is very ambiguous. So, we need to adjust this prompt a little bit. Let's add over their own calls. All right, the assistant goes off and thinks and now we can see the percent of resolved cases here are a lot more reasonable, as is the generated SQL. Uh, okay, that's great. So I'm I'm going to accept this.
Now we've got our present resolved cases per employee and we're ready to join with the tenure data that we brought in earlier from our spreadsheet. Okay. So we're going to combine these two. All right. Let me type in here. I want to join support rep data with percentage resolved. Okay, great. Now we've got a join node that combines these two. We see here the tenure information alongside the employee ID and resolved cases.
Just what we wanted. All right. So, we've got our employees bucketed by time and ro, but we need to tidy this up and look at the aggregated data. I've got a lot of extra columns here. I need to aggregate employees by tenure. I know I'm going to need to present this data. Basically, I need to fill out a table in a slide that looks pretty much like this. This is a situation where I've got an example of what I want the data to look like.
In these cases, it can be easier to show the assistant what we want to do rather than to describe it. So, let's see how that works here. I'll start with a a simple screenshot of my slide. Okay. Now, I'm going to switch back to designer and I'm going to add a new operator transform by example. Again, this is an operator that lets us show the assistant what we want with data or here even a screenshot. Let's attach our screenshot.
Okay. And here we go. The assistant goes off. thinks for a few seconds and fills in the data table below. And indeed, we do see here that logger tenured employees have a higher support case resolution rate. After 6 months, it's 80%. Whereas for new employees, it's closer to 20%. We figured all this out with no code whatsoever. Pretty awesome. Now, what about deploying this to production? Well, it turns out I don't need to show you all that much about this because Bal already did.
7:43 Deployment and monitoring
Under the covers, LakeFlow pipelines are completely standard LakeFlow SQL pipelines. If you need to collaborate with your data engineering team that prefers working with SQL, no problem. Edits you make to SQL automatically apply to the canvas and vice versa. Let's look at a few operators here to see what this looks like. I'm selecting operators. The corresponding SQL highlights. It's all just SQL. And similarly, you know, these are just ordinary LakeFlow pipelines under the covers.
There's no separate tool. It's all LakeFlow. So, here I'm showing you a completely ordinary uh LakeFlow pipeline monitoring page just like what you saw before. Okay, that is LakeFlow Designer. Pretty awesome. Let's go back to the slides and recap what we've seen. Okay, so first of all, sharing and collaboration between analysts and engineers is super simple. You can edit pipelines visually or as code, whichever you prefer.
8:23 Key benefits and conclusion
Second, there's a built-in path to production. Designer is fully integrated with LakeFlow. There's no separate tool. And finally, we saw how Designer uses AI throughout the product to boost productivity with features like transform by example. That's LakeFlow Designer, a visual editor that delivers production quality ETL with no code required. Designer will be available later this summer, and we're excited for you to try it out.
Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.