Skip to main content

Databricks LakeFlow Designer: Visual Pipelines, Real Code

Summary

  • Visual Data Transformation: LakeFlow Designer provides a no-code canvas for citizen data users and technical teams to build production-ready data pipelines visually.
  • Governed Data Access: Unity Catalog ensures users only work with authorized tables with full lineage maintained, enabling secure self-service data transformation.
  • AI-Powered Features: Designer includes AI operator search, in-canvas visualizations, forecasting, and just-in-time custom operators for accelerated pipeline development.

Databricks LakeFlow Designer: Visual Pipelines, Real Code

Watch: Databricks LakeFlow Designer: Visual Pipelines, Real Code
LakeFlow Designer is the visual, no-code data transformation experience in Databricks LakeFlow. Youssef Mrini (Developer Advocate) and Jason Messer (Sr. Product Manager) walk through the latest updates and build pipelines live on the canvas.
Designer is built for citizen data users who have outgrown Excel, and for technical users who want a faster visual path to production-ready pipelines.
In this video:
• Prepare and Unique operators, plus spreadsheet-style result table edits that generate operators on the canvas
• Enter Data, Metric Views as a source, and outputs to tables, materialized views, or CSV, Excel, and JSON in a volume
• Email a CSV of results for recurring reporting
• AI operator search, just-in-time custom operators, in-canvas visualizations, and AI functions including forecasting
• YAML, Python, and SQL under every operator, plus notebook export
• Lineage highlights, exact row counts, Run All, and visual revision history
Everything is governed by Unity Catalog, so users only work with the tables they are authorized to access, with full lineage maintained throughout.
Try LakeFlow Designer for free, no credit card required, with Databricks Free Edition: https://www.databricks.com/learn/free-edition
Learn more about Databricks LakeFlow: https://www.databricks.com/product/data-engineering
Chapters

Chapters

FAQs

What is Databricks LakeFlow Designer?

LakeFlow Designer is the visual, no-code data transformation experience within Databricks LakeFlow that enables both citizen data users and technical teams to build production-ready data pipelines on a canvas-based interface.

Who should use LakeFlow Designer?

LakeFlow Designer targets two main personas: citizen data users who have outgrown Excel and want to transform data without SQL or Python, and technical users who prefer a faster visual path to production pipelines.

What governance features does LakeFlow Designer provide?

Every operator in Designer is governed by Unity Catalog, ensuring users can only access tables they're authorized for. Full lineage is maintained throughout all transformations for complete traceability.

What AI capabilities are built into LakeFlow Designer?

The platform includes AI operator search, just-in-time custom operators, in-canvas visualizations, and AI functions like forecasting. All operators maintain code visibility with YAML, Python, and SQL available underneath.

Full transcript

0:00 Introduction to Databricks LakeFlow

Hello. Today, we're going to talk about Lake Flow Designer. I'm pretty sure you've heard about it. It was released 1 year ago at the Data and AI Summit. It's part of the Lake Flow offering, which is the data engineering offering from Databricks. We have actually four components: Lake Flow Connect. So, those are like we have more than 100 connectors to ingest data from databases or from SAS. Then, we have Lake Flow Declarative Pipeline.
This is the transformation part, where we can use Python or SQL. We have also Lake Flow Designer, that will introduce in a few minutes. And we have also Lake Flow Jobs, which is our orchestration layer. And we have. He's the product manager behind Lake Flow Designer. He's the best person we can host to talk about all the latest release, because I've read release notes, and it was way too long for me. So, Hi. >> Hey, Yousef.
Yeah, great to be back on the show. And yeah, hey everyone. product manager at Databricks. Working on Lake Flow Designer. As you kind of mentioned, Lake Flow Designer is our no-code offering to do data transformation. So, we've released a lot, a lot of cool stuff. And I think one of the things that I people complain about sometimes is like Databricks ships so fast. Like you release so many things, I can't stay up to date.
So, hopefully I can talk to you all about some of the new features we released, and then help you understand what they do. >> That's correct. We're going so fast. I was reading the release notes, and I started like deep diving on those enhancements, and it was too long for me. But, can you tell us, let's say in few words, who can use Lake Flow Designer? >> I think the short answer is everybody could, but I think there's maybe two personas that we're trying to go after.

1:31 Target Audience & Personas

One is kind of your citizen data user. So, these are people who maybe aren't as familiar with SQL or Python, but still kind of are familiar with the data and want to transform it. People that have maybe outgrown Excel. You know, you've passed the 1 million row limit for Excel, but you still want to transform your data. The other is more of your technical audience, so people who are trying to build out really production ready pipelines.
Lake Formation provides you a really easy visual way to do that. So, if you want to do that, you're also a yeah, it's also a great place to do that. >> And we have introduced Genie code a few months ago. So, when should I use Genie code to build my pipeline or my transformation compared to Lake Formation designer? >> Yeah, that's a great question. Yeah, people often ask us like, "Hey, with Agent TK I and Genie code, like what's the need for like a local portal?

2:07 Genie Code vs. LakeFlow Designer

Why can't I just ask AI to build it out?" And the answer is you definitely could, but like I think the hard part is once you've generated a bunch of code with Genie code, how do you understand and make sense of what it's doing? If you're not really super familiar with Python and it gives you 1,000 lines, like how do you actually understand what's going on? How can you validate that it's correct? With Lake Formation it's powered by Genie code, powered by the exact same thing, but it produces things in a way that creates a visual canvas that you can really easily see how the data flows.
So, you can understand what's going on. You can check at each step if the data looks like you'd want it to expect. And so, you can really easily validate and build out your workflow um using this visual canvas without having to understand code. >> With Lake Formation designer, I can I can write code, but you have a visual course interface where you can drag and drop the features to transform your data. And it does produce either SQL or Python.
Is it that easy to view the code behind the scenes? >> Yeah, yeah, now it is actually. One of the features I'll talk about that we released is we used to kind of tuck it away a bit, but now it's like if you just go to the table of contents, which I can show, you can just view for each operator you can see the code that's produced under the hood. And you can also of course, same as with everything that's that's code based, you can put it in Git, you can use a dab with it.
You know, there's a lot of stuff you can do with it because you have code under the hood. >> I think it's the best way to introduce what has been released since maybe the data and AI summit? >> Yeah, let me uh share my screen here and we can kind of go through everything. All right, so here we are on the canvas and my favorite thing now with Lake Designers to present using the canvas because we have these nice operators that let you kind of build out and describe things.
So, this is the note operator that we're looking at. This is one of actually the things that we improved here is we now have kind of inline editing, and this is a really really awesome and easy way to basically write and document things that happen on the canvas. So, today I want to talk about some of the new core transformations that we built out, some new operators, some improved output and input options, some new AI features, and then there's a lot of kind of nice quality of life improvements.
This is only like a various This is a subset of all of the stuff that we've done. So, we do have the full release notes if you want to really dive in, but this is These are some of the highlights for me. So, the first thing that maybe I'll call out is that we have a few new operators, and those are namely the prepare and the unique operator. So, the prepare operator is awesome because it provides you a way to um point and click to transform your data and prepare it for later down in the workflow.

4:25 New Core Transformations (Prepare & Unique Operators)

So, if I just hit the plus button here, you can see one of the options I have now is prepare. And then here we can add actions. And so, it says, "Hey, I want to add an action." And so, maybe for example, I want to do a regex replace. I was just looking over at this input data here, and we can see that on the um employees column, it's a string right now, and it has commas in there. So, I think it'd be more useful to have this as a an integer.
And so, for example, I can remove this comma by just going pattern comma here, and then say I want to replace with empty string and hit apply. Oh, sorry. I didn't specify column. I got too excited before. And so, we need to go to um employees. But this also shows a nice feature is we have AI error summaries, and so we can see really quickly like, "Hey, this just needs a a column name. Please provide a column name." So, then we can do that and apply here, and then in a matter of seconds we should see the transformation.
I mentioned this before last time, but with Lake Designers, we show your input on the left and your output on the right, so you can really easily compare side by side to see how to make sure that everything is is happening correctly. This is the first run of the day, so our our serverless compute is spinning up, so it might take a second longer. And maybe while that's going, I could also do another transformation where I could then say, "Oh, hey, I want to add another action.
Let's change the type here, and we can change employees again, and we can make this now an integer, and hit apply again." And then now we should see, if I scroll over to um the employees column, we have it as an integer, and we've removed the commas. So, as you can see, it's a really, really easy way to kind of um point to click and transform your data. Again, Genie code is really great at pairing with this, so you can use Genie code, and it can do all of these things for you, but I like to kind of show off some of the other features manually. >> So, I think it's we can describe it as rather than like writing down like one a one prompt, here you can split the real transformation in so or in the let's say small tasks, like start by filling null values, transforming the columns, then maybe adding new operators to compute new KPI.
So, it's like splitting the problem in small task, and then going building this pipelines step by step. >> I totally agree. So, cool, that's the prepare then the unique operator is another one that was pretty commonly requested. It's basically deduplication. Um so, this maybe is not the best table for this, but if I go over here and say, um you know, I want to go to add the unique operator again, then I can choose where I want to select to be unique by all columns, so basically just look at the entire column, look for any duplicates, or do I want to specify a selected column?
So, for example, um you know, I could say, "I want to um look at the code here, which we have some of the same values here." And I can just hit apply here, and then this will keep um basically one value for each of the code. So, it will look for unique values for the code column and do that. And then one other way that I'm actually really excited about is if I actually just go to just um output only here, is we now have the ability to transform your data directly in the result table.
So, this is something that I personally I'm really excited about because I wanted to do this for a long time. So, just for example, if I want to change this to employees here to make it num_employees, I can to the edits directly here in the canvas. If I want to filter maybe to the industry of home and living, for example, I can filter by this. I can make the edit directly here in the canvas. If I wanted to remove a column, for example, so say I want to like I don't want to remove this sales column, I can just say remove column here.

7:23 Core Workspace Editing & Data Manipulation

And so you can see as we do this, we have these pending changes that show up here. And so you can see, "Hey, it wants to remove this and rename this and then filter to this." And then if we just hit apply to canvas, it will then create the operators for me to perform that transformation. And then in just a second we should we can see that this will have actually run. And now we have the transform data that looked like the data in the table that I was manipulating.
So I think that's a really, really cool one. >> And we're bringing the Excel experience. >> Exactly. >> To uh the notebooks. >> That's a that's a great way to put it. So you can just work directly with your data and then I mean you can make the operators so that you can make it repeatable and you can schedule and whatnot. A few other features um now if we move over to improved um input and output options is um we now have the ability to have an enter data operator.

8:31 Input, Output, & Notification Options

So if I go over here, I can see I have this enter data. And essentially what this lets me do is it just lets me put in a spreadsheet of data. So if I want to manually fill in some values, I can do that. One of the things I love to do is combine this with Genie code and to do things like this like uh give me some sample data that I can use to test the unique operator. And uh this will go ahead and just um basically fill in a bunch of values for me.
So if you ever want to do any testing, I find enter data can be really, really helpful for that. And as you can just kind of, yeah, create data sets on the fly. The next thing is we do support metric views as a source now so you can drag in a metric view from your catalog explorer and basically say which measures and what you want to use. So that's that's a really nice thing. We also have a bunch of improved output options.
So if I was to go over here, you can just see here like, you know, just created this data set for me that I can kind of test with which is awesome. Other thing is if I now want to output, I have a bunch of different options here so I can actually output to a materialized view. So if you ever want to just store a kind of a a fast precomputed result, you can do that. This is the kind of the default we recommend. You can also output to a standard table.
If you do this, you have the ability to merge with an existing table, append rows, create or replace. And then we also have the ability to output to a file. So, if you want to just write to a CSV file, Excel file, or JSON file inside of a volume in Databricks, you can do that as well. I do also have to call out that you can always at every step download the result here. So, you don't need to output to a file, but if you want to output this a file to work with in a different workflow automatically, you can do that by just outputting here.
The other thing that I next thing I'm really excited to talk about is we do have now the notification operator. So, this is really cool. If I just go over here, I can say let's send the values here. I can basically say, "Hey, like who do you want to send this to?" And so, you know, I could say like I could type in myself here. And I can say what's the subject, email from Lake Flow Designer with your results. A JSON, here are some of the results.
And then you can just say like attach data. And then hit apply here, and then run. And then this will basically just email me the CSV from the results we have here. So, if you want to do any reporting, maybe I like build out something and then I have the results and I want to report, send the email. I want this to run every single week, for example. You can really easily like, you know, send you an email report from the stuff that you're doing within Lake Flow Designer.
And so, this is run. I'm correctly and if I was to click over to my email, we could see that um that would show up. So, that's a really really nice one for kind of doing more of your business automation, which we've heard a lot of. >> Yeah, I'm I think something important to remind people, everything that you see here is governed with Unity Catalog, which means you can only access the tables that you are allowed to. And of course, you can only create the tables if you have the um the permission.
So, everything is of course is governed. And you get to have all the perks of having the lineage, which means from where the data has been derived, which notebook has been used to create tables, everything you need to know to make sure that what you did is correct. >> That is very true. Yeah, thanks for calling that out. So, So everything here is a is fully governed under Unity Catalog, which is a very different from some of the other low code tools that uh people commonly work with.
Okay, the next thing I want to show off is uh we now have the ability to do visualizations directly in the canvas. So actually, let me go drag this one over here. If I wanted to just like do a simple visualization to maybe copy paste to a worksheet, or if I wanted to basically do a data quality check or check something out here, I could just go over here and I could say visualization, and I have the ability to build a visualization directly in the canvas.

11:35 In-Canvas Visualizations

One of the cool things I like to do is we can also combine this with our Polyglot AI Editor, which for every single operator we have a description. And this description, if you edit it, it will change the configuration. And then if I edit the configuration, it will change the description. So everything stays in sync. So you can work however you want to. So I could just say average life expectancy based on region, for example.
And I can just type that out. I can leave the leave the typos in there, for example. And this will just go ahead and build out a visualization for me that should show the average life expectancy based on different regions. So we can see here, um if I make this a bit bigger, we can then see the average life expectancy for across different regions. So now this is really easy. You can do this directly in the canvas if you want to. >> And I guess maybe in short term we'll have maybe possibility to copy paste the visualization you're building here in an AI BA dashboard. >> Yes, that is something that we do want to have on the road map.
There's a bit of a challenge because it we generate Python code under the hood, which is not totally compatible. But that is um that's something that we do have plans to do long term. It's enable you to just add directly to an AI BA dashboard from this. >> And I'm pretty sure we have AI somewhere. >> Yes, yeah, exactly. So yeah, moving over I want to show off a few of our new AI features. So one of the things that we've released recently is AI operator search.

12:57 AI Features & Just-In-Time Custom Operators

Um as our operator list builds out, and as you add kind of your own custom operators, it becomes a bit harder to find exactly what you're looking for. But with the AI operator search, it's really easy to find stuff. So you can just type like summarize rows, for example, and it will just do an AI search, and it will basically return, "Hey, if you want to do the aggregate operator, that will help you to summarize different things."
If I wanted to transpose, for example, um I could search this and it will say, "Hey, like the pivot where is how you transpose things." And so, you can just search in natural language what you want to do, and then it will recommend the right operator to use. So, I know this is a super helpful one. The other thing I mentioned is just the polyglot editor, which is on every operator. You can just go over here, and you can basically describe what you want to do.
So, for example, um if I was to go over again here and do kind of like an aggregation, or even let's just do a filter operator, for example. You know, I could just say, "Show me rows in Asia with a population of more than 10 million." For example, and then this will create the filter conditions necessary to do that. And so, for every single operator, it will do that. And then the cool thing is that it's again bidirectional, so you always have this description in sync.
So, if I was to go over it manually change this to say, "Oh, I don't want I don't want Asia. I want to look up in Europe now." and hit apply here, it will then, see, keep the rows in Europe um where the population is over 10 million, right? And so, these stay in sync, which is really, really helpful. The another feature that I'm really, really excited about is just-in-time operators. So, one of the things that we struggle with is we want to keep the product simple with a limited amount of operators, but we also want to give you the ability to do everything you want to do.
And we find there's a long tail of operators that you might need, and you could just write like a Python or SQL operator for that. That shows you a lot of Python code, is not super easy to use, and then if you want to reuse that across different workflows, you kind of have to port that over. And so, we created this idea to create an operator just-in-time. And so, what you do here is you just scroll down, and you have this ability to create a user-defined operator, and then I can say a name for it.
So, for example, for this one, I want to take my country ISO codes, so like NLD, Belgium, for example, all of this stuff here. And I want to add a latitude and longitude coordinates so that I can then put this on a map and visualize this data in a certain way. In order to do that, I can just like go and say, "Hey, create user-defined operator here." and I can say, "Country ISO code to latitude and longitude." And I can just say, "Give me a operator which takes in a three-letter ISO code and a pens a latitude and longitude column with the respective values.
I can say generate preview here. And so, what this is going to do is it's going to create basically one of these little blocks that we've been using, one of these operators for me that will then do this. And then I can then reuse that across every single operator. I think this is a a cool thing. It's something I think that we're going to see more as AI gets better and better is letting people you keep the UI simple, but then add certain custom elements for you when you need it.
I don't know if you have any thoughts on that while this is a while this is generating you so. >> I think it's pretty interesting like giving flexibility and keeping everything simple. I think this is really really hard to achieve, but I can see how like can you like data citizen can go and use it very easily. >> Exactly. So, it looks like it it's created. If I want to approve it, I can go and look and see that the config and the code that it's it's running.
And then I have a nice description of what it does and I can say approve and create here. And then if I look down here, I now have a new operator which takes this. And so, then I can just go over here. I can add this in. I can select the column that I want to have. I want to have name here. I can hit apply. And then voila, it looks like if I look over here, it looks like we have a latitude longitude, but it looks like it didn't totally check out this time.
Let's see maybe if we can debug uh >> If this is something that company like specific companies using a lot, they can configure this just-in-time connector by making sure that to standardize this access let's say box to every user using Lake Flow designers to make sure that each time it will produce the same results without like keeping things easy and customizing the UI or the interface for the end users. >> Exactly.
Yeah, you're 100% right. So, you can create a repeatable process that you can then share across everybody and you get the same result every time. Looks like I it was just a user error here. I put down the name instead of the code which we wanted the ISO code here. So, now if I look over now I have latitude longitude for everything which is great. And a cool way is I can then validate this with another user-defined operator I have, which is I can go over here and I can say globe visualization, and this basically takes the latitude column, takes the longitude column, for example, it takes the volume column, so maybe I want to look at the population, and then I want to label it by name here, for example.
And then if I hit apply here, um we should see an awesome globe visualization pop up here. So I have my data here. So I can zoom in here, I could stop it. Let's change it to be maybe crystals, for example, you know, and then we can see and you can see that yeah, looks like it's pointing to Canada here. Looks like you know, this would be the United States. And if we go over here, right, we can see the different um countries in Europe and their populations um based on this, right?
So you can see how easy it was to kind of build this out by just using the these user-defined operators. >> And also how easy it was to debug the issues. Like you were coding and you did two mistakes and were able to find them quickly thanks to the let's say the summary, the AI summary made that you were able to view in the UI. And this is really a real-life coding experience that you can also face. >> Exactly, yeah.
No, that's a great point. And I didn't even show, if I would have hit the fix me button from Genie code, it probably would have been able to fix those on their own as well. So yeah, so we the AI debugging is is really really helpful. >> Now that you created this pipeline, of course, you still need to show us how we can use the AI functions quickly. How can I view the code that has been built for me behind the scenes? >> Yeah, you're you're kind of jumping ahead.

18:24 Code Visibility & Transparency (YAML, Python, SQL)

So the next thing that I wanted to talk about is in the nice quality of life improvements, we now have the table of contents over here. So if I go over to the menu here, I can look at the table of contents. And as I click on each operator, it will then highlight it for me. So I can say, "Oh, I want to look at this." Or we can look at this. All right, in order to see the code that was that was generated here, I just have to open this and I can see the config, which is essentially like a little YAML representation, which helps us kind of build out the visual for it.
And then I can see the generated code as well for this. So you can see you know, the different things of different functions that we would take when when we execute this operator. If you want to see it all as a whole, you can say you can go file, you You export this. It will export as a notebook and you can see every single cell I'm in all of the code. Or again, you can view like per operator, um you can just click on it here and you can just expand the generated code to see that. >> Okay, we can maybe go back to the AI things cuz I know I was I was so excited to just to show people that it's really easy to view everything that has been developed.
It's not a black box. You can view the entire code and have a look at it, maybe share it with developers so they can also have a look at it. >> Yep, exactly. And we do have some mechanisms in place where if if they want to make changes on their own end, we do our best job to kind of be able to bring those back into the visual as well if you want to kind of work together across this with your data engineers and people who are working in the UI.
Yeah, the last thing I was just going to call on I don't we don't necessarily have to show this off, but we do have some new AI functions. So we have like AI forecast and a few new AI functions. So I just a reminder in Lake Designer, we have basically if I go over here, you can hit a plus here and I can say AI function and we have a bunch of options. So you can do sentiment analysis, classification, extraction. AI forecast as I mentioned, you can parse document, AI query, AI translate.
So there's a lot of different options here which are built out of the box in Data Bricks. It's just point and click. Don't have to set up any end points. You just basically click and use it. So we added some new ones. So hopefully those will be nice quality of life improvements for you all as well. Okay, the last thing I just want to mention real quick cuz we are running out of time here is we have a few new quality of life improvements that I'll just call out.

20:21 Platform Quality of Life & Version History Improvements

I don't necessarily need to show off all these, but we do have lineage um highlights now. So for example, if I go click on an operator, we can see that it makes the lineage from the source to sink a bit more prevalent here so you can really easily tell. It's not really a super important for this type of small workflow, but if you have a huge workflow, it can be really helpful to see like, "Oh, where is my data flowing across through all my operators?"
You we have the ability to give you the exact row counts now, which is great. So for example, I think this was all small data, but if you did run into a limitation where you say, "Oh, like there's more than 10,000 rows here, what's the exact row count?" We have the ability for you to calculate that now. We have count distinct in the aggregate operator. I know that was a big one cuz people want to see how many distinct values there are of this.
We have the ability to load all of your options in a filter operator. So, before with the filter operator, if the value you're looking to filter wasn't in the sampled options, so just let me help you understand this. It's like maybe one of the options you have here is not in the sample. We now have this ability to load more options, which will then give you every single option for this column that you can select from.
So, that's a nice quality of life improvement. We also have a run all button now. So, if you want to just quickly just make sure everything works correctly, you can hit run all to be able to do that. We have a visual revision history now. So, if I go over to my version history here, if I click to different little subsets, you can see like here like here I added the filter operator, and then you can see it's highlighted in green.
And so, you can really easily tell what changed across different versions. And if you want to go back, you can do your time travel to restore a past version, which is really helpful. We also have a visual data prep listing page. So, if I go actually over to the menu here, we now have this visual data prep page. So, I can I basically see all of the files that I've been working with or my coworkers have been working with in one simple place.
And then the last thing to mention is just we have so many more changes that we've made that will just be like more small small quality of life improvements. >> Exactly. And actually, we're we're fast than me. I was about to ask you how can I use Lakefront Designer, but you have showed it in the menu. It's the visual editor. And something very important to keep in mind, you can go and try Lakefront Designer for free.
You can create an account easily in Databricks free edition. No credit card is required, and you have access to the Lakefront Designer. Give it a try, test it. If you have any feedback, make sure to share us with. Thank you, and see you soon for in few months to show us what are the latest features you would have added by then. >> Yeah, we're adding so much. You might be in a few weeks. You might need to have me back again to talk about some of the newest features.
But yeah, just plus one. Hope you all get to try it, and definitely let us know if you have any feedback. >> Thank you. >> Thanks, everyone. Bye.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.