Lakeflow Designer: Removing the Technical Barrier to Self-Service Data Prep

Summary

Lakeflow Designer is a no-code, AI-powered visual canvas on the Databricks Data and AI platform that lets business analysts transform data and build production-ready workflows through natural language prompts and drag-and-drop operations, without needing to write code.
The tool addresses the gap between business analysts who have the deepest domain understanding and data engineers required to build production pipelines, replacing tools like Alteryx and Excel with a governed, scalable alternative integrated directly with Unity Catalog and Databricks Jobs.
Advanced capabilities include custom Python operators that extend the canvas, the ability to create new operators on-the-fly from a natural language description, and a production path that generates version-controlled, enterprise-grade code from visual workflows.

Lakeflow Designer: Removing the Technical Barrier to Self-Service Data Prep

For most teams, the biggest bottleneck isn't the generation of insights it is the work thats required to get the data wrangled the right way. For too long, the ability to transform data and automate workflows has been locked behind complex code, leaving business analysts dependent on overstretched data engineering teams.
Lakeflow Designer changes the equation with a no-code, visual canvas powered by Genie that lets anyone go from a natural language prompt to a production-ready workflow in minutes.
This breakout will cover:

How Lakeflow Designer fits into the overall Databricks ecosystem and what differentiates it from other low-code tools
How to get started and leverage natural language through Genie to build a workflow with zero previous experience
How Designer’s visual workflows translate into production-grade code that is governed, scalable, and ready for enterprise use

Talk By: Emanuel Zgraggen, Staff Software Engineer, Databricks ; Jason Messer, Product Manager, Databricks ;

Chapters

FAQs

What is Lakeflow Designer on Databricks?

Lakeflow Designer is a no-code, visual data preparation tool built into the Databricks Data and AI platform that uses an AI-powered canvas to let anyone build data transformation workflows through natural language prompts. It is designed for business analysts who understand the data domain but do not have the coding skills to build production pipelines independently.

How does Lakeflow Designer compare to Alteryx?

This video positions Lakeflow Designer as a replacement for tools like Alteryx and Excel-based data prep by offering an AI-powered canvas that generates production-grade code governed by Unity Catalog. Unlike those tools, Lakeflow Designer workflows schedule directly in Databricks Jobs and write outputs to Unity Catalog, keeping results within the same governed data platform rather than creating isolated local data silos.

Can I use Python with Lakeflow Designer?

Yes, Lakeflow Designer supports custom Python operators that extend the visual canvas with custom logic, allowing technical users to contribute reusable building blocks that business analysts can then use without writing code. New operators can also be created on-the-fly by describing their intended behavior in natural language.

How does Lakeflow Designer get data into production?

Visual workflows built in Lakeflow Designer translate into production-grade code that is governed, scalable, and ready for enterprise use. Teams can schedule the resulting pipeline through Databricks Jobs and write outputs directly to Unity Catalog, providing the same reliability and governance guarantees as hand-coded pipelines.

Full transcript

[00:07] Thank you so much for everyone for coming. It's the the last session of the conference, so we really appreciate you uh you taking the time to to be here. My name is Jason. I'm a product manager here at Data Bricks working on a Lakeflow Designer. And uh I've been saying it and it's true, but I've worked
[00:19] at Data Bricks for just over three years on a lot of cool products, but Lakeflow Designer is my favorite product that uh I've got to work on. I think there's a really cool intersection of AI and UI. So, I'm excited to show you all more about it uh today. >> And my name is Emanuel. I'm the tech
[00:32] lead on Lake Flow Designer. And uh yeah, also super excited to show you what we've been building over the the past year. >> Awesome. Okay, this is just like housekeeping. We have to put this forward-looking statement. Basically says if I say
[00:45] something that doesn't come true, you can't hold me accountable. Hopefully, I don't say something that doesn't uh doesn't come true. But we have the slide. Um and then the other thing is please do complete the survey afterwards. You'll get a notification in the app. Um it's really helpful for us to know what uh what you done better and
[00:58] whatnot. So, so please take the time to fill that out. All right. Today we want to do three things. I want to talk first about why we built Lakeful Designer. We'll do just some slides for a bit. Then we're going to do a live demo. So, for those of you
[01:11] who've been to any talks I've done before, I like to spend the majority of the time doing a live demo. So, we'll do that. Hopefully, the Wi-Fi and the demo gods are are in our favor. Um, and then after that, hopefully we'll have some time for Q&A. And then of course if you have other questions we can't get to I'm
[01:24] happy to chat afterwards outside as well. So with that let's jump into why we built LakeFlow Designer. So it kind of starts with the idea that everything you do on data bricks
[01:36] requires good data. Whether you're building AI models, whether you're doing dashboards, whether you're doing just basic analytics, you need a good foundation of data to to to build on.
[01:48] But the challenge is that the people that have the deepest understanding of the business are often not the ones who can build production data pipelines. So in other words, the people who need to make decisions with the data are not the ones that can kind of self-s serve and
[02:01] produce that data themselves. So that's where there's kind of this need for no code self-service data prep and analytics tools. And I'd be lying if I said that we're the first people to to ever think of this. There's a lot of other really great tools in this space
[02:14] like Alterix, Data IQ, Power Query, um, and even Excel or Google Sheets, um, that allow you to basically do this no code data preparation. But when we've talked to customers, we've seen that there's some challenges with this. Um, the the biggest thing is
[02:28] that the data work happens in two separate worlds with these existing low code tools. So on one end, you have the central data platform. You have data bricks, everything's governed. This is where your production stuff runs. On the other end, you have these low code tools. They often run on your local
[02:40] machine. uh you have to download data out of the central environment, put it on your local machine to do stuff and this just creates friction. When uh when talking to customers, we've kind of summarized uh maybe three core problems
[02:52] and then maybe one underlying thing uh that's challenged with this. The first as I kind of mentioned is you have siloed workflows. So your data work spans different environments. It's a nightmare for your IT team to manage the different environments. People are doing the same thing multiple times um on
[03:05] their local machines. People are not reusing work. Um and then there's also big governance problems with this where you have to take data out of a govern environment and put it into another. It also creates headaches when you want to bring these things to production. So you
[03:17] know you might have built like a 200 operator node workflow in Altrix and you want to bring that tip to to production and data bricks and it's like well how do I do this? Your data engineering team's like man like I do not know how I
[03:29] translate this into code. So that's really hard to to productionize. And the other thing is as these tools sit outside of the central data platform, they have um limited AI productivity uh by nature because they don't have all of the context and metadata that you have
[03:41] if you're sitting in the central data platform. And then kind of undercutting everything is there's expensive praise or license costing costs often for all of these tools. So you have to decide beforehand who is worth it to give access to these tools um before you can
[03:54] you can give access to them. And so it really limits the number of people that can use these tools. So that's where we've come up with uh with Lakeville Designer, which is our no code data prep solution that's native to to data bricks.
[04:06] I'll go over this quickly because I think we'll talk about a lot of this in the demo, but we've built it uh to be a drag and drop experience that's really really simple. Hopefully uh when you all use it, it's really really easy and intuitive. You just drag out and build your flow. We've built it from the
[04:19] ground up to leverage AI. So it's been actually really fun as we've built Lakeful Designer starting last year after kind of the Gen AI craze is is already uh well in in stream. And so we've been able to really think from first principles into how we can build
[04:32] an experience that leverages AI to be uh very intuitive and also make you very productive. And so we've worked really hard to do that. And then everything that you do in Lakeville Designer in the visual interface is backed by production ready code. So generates pi spark code
[04:45] so that if you want to take things to production, it's really really easy to do that. And ultimately if we look at those same three problems um we have an answer for them. So everything is unified and fully governed in one environment in data bricks. You get
[04:58] built-in governance and lineage with unique catalog by default. You have a simple path to production as it's just code behind the scenes. So if you want to schedule it, if you want to put it in a data bricks asset bundle, if you want to um push it to git, you can do that
[05:10] really easily because it's just code. Um and then as I mentioned before, we worked really hard to make it uh leverage AI and we get to take advantage of all of the metadata that we have being in data bricks in with Unity catalog to help provide really accurate responses and help you be very
[05:23] productive. And then finally undercutting everything is there are no per seat fees. It's all consumption based pricing and so you just pay for the value that you unlock out of it. So you could literally add everybody to it and you only get charged when your users start to get value uh from it.
[05:37] All right, that's uh that's enough slides. I feel like that's already more slides than I like to do. So let's jump into a live demo to kind of set it up. Um we have three personas today that we're going to kind of role play. So, we have Emma who's going to be my manager
[05:49] and she's going to ask me Jason who I'm going to represent the business analyst um to do something. And then Emanuel is going to represent our engineer um in this role play. So, Emma says, "Hey, Jason, I need you to help me put
[06:01] together some World Cup data from 2012 to now. To do this, I think you'll need to compile combine the all matches data in data bricks with the this tournaments Excel file I just put together. Can you fill out this table and visualize it on a map?" And so the table that then she's
[06:15] given us is we want to have the country, the average goals per game, the total matches won, a power rating which is calculated with this formula down here, and then the number of World Cups won. So that's what we need to do, and I'm
[06:27] going to show you how we can do it without writing a uh a single line of code with Lake Designer. So if we swap over here to the demo tab, um this is the the canvas of Lake Designer, the
[06:39] getting started screen. One thing I'll show you before we jump in here is how to create a file in Lake Designer. Um, if you go to the new menu, you can get hit visual data prep. Um, and that's kind of the the name of the the file name that you you have here. And then you'll be put into this entry point. I'm
[06:52] going to go back to to this one as I already have it in a git folder um for for later on in this demo. I'll go ahead and close out some things here. I'm actually going to Yeah, perfect. Okay. So, when you're in Lake Designer, the first thing that you you probably need
[07:04] to do is add some data. You need some data that you're going to transform. And we have a bunch of different ways to do that. Um, the first way that you can do that is just obviously by pulling in data that's already in uh, data bricks or using a source operator. So I can hit select a source here which gives me a
[07:16] nice little operator here. And then I have a few different options here. So I can browse an existing file like to find stuff or browse an existing table that's in unique catalog. I can create a table from a file. I can just drag and drop a local file and put it in a volume. I can
[07:29] also connect to other external sources. If I want to pull in from Google Drive or SharePoint, for example, you know, I can just uh put my connection and then put the fire file URL to to pull those things in. Um, for now, since my manager told me I need to pull in the all matches data, I'll just go to browse all
[07:42] m or all all tables and hit uh all matches here and I can just select this. Um, and then while my compute spinning up, I'll orient you towards the the rest of the UI. Oh, it looks like it was it was very fast, which is great. Uh, but
[07:55] on the lefth hand side here, you have your operators. So you have things like your AI functions if you want to aggregate, if you want to combine, if you want to filter, join, uh prepare, do pivots, uh select, you want to do Python operator, you have all these building
[08:08] blocks here on the left side. On the middle here is the canvas. This is where you drag things in to to use these building blocks to then build out your your workflow. So for example, you know, I could pull out this aggregate operator here, right? And I could chain it together um and do that.
[08:21] On the right hand side here, we have the operator configuration panel. So this is where you'll configure like if I added the aggregator, if I add a join operator, configure the keys um and whatnot on this right side. And then on the bottom here you have the output
[08:33] table which is where you can see a preview of your data. By default we work with a thousand row sample so that things are very fast um and also efficient. But you could change this to be whatever you want. I can make it a million row sample or if I wanted to I
[08:45] could do the work with the full data set uh by clicking that. All right. The other way you can bring in data is from your local machine. And we've made it really really easy to bring in files from your your local machine into Lake Designer. It's just
[08:58] literally drag and drop. So I have this Excel file here that my manager gave me. And I can just drag this onto the canvas. I basically say where you want to put it. So I can put it in this World Cup data. Um I can override existing file there and I can hit upload
[09:11] and this will just bring your data in. And as you can see we already have um a preview of the the data from my local machine. So we know in practice that uh it's often very common that you'll have not all of your data in data bricks like
[09:23] maybe in re the dream is to have that but in reality that's not always going to be the case and so we made it really easy to pull in local files. All right the next thing I'll show before we can start kind of jumping in and transforming our data is we have data profiling in Lake Designer and so
[09:37] this is great if you want to basically get a high level overview of your data before you jump in. So if I just select a column or a partial selection, I can hit this selection details um panel over here on the left or on the right, sorry.
[09:49] Um and then that will show me basically a profile with some stats uh about this. So this is uh information on people who have won the World Cup. And so we can see here there's eight distinct values, which it's kind of crazy that only eight
[10:01] countries have ever won the World Cup. Um, and then if we look, it looks like, you know, Brazil has won 22.7% of World Cups, which is kind of a wild stat uh to to me to think about how dominant they they've been there.
[10:13] Again, this works with like partial selection. Um, if you want to just like, you know, select a few numbers, you can get that. If you want to, you know, select the whole column, you can do that uh as well. And then you get different information, right? If you um look at integers, you know, you get some more statistics on things like the minimum,
[10:26] the median, uh, other values and whatnot. So this is super helpful for kind of getting an overview of your data. All right, now we can jump in and start uh start transforming our data. So let's
[10:38] briefly just look at um what our data is and then we can reference back to what we want to do. So looks like this table has different World Cup tournament data with who won. So we'll need this to find the total number of World Cups a country has won. And then if we go over here, we
[10:51] can see that uh that this is basically information on matches that were played. So you see like you know this is France, they played against Mexico, they were the home team, how many goals they scored and whatnot. So this will be
[11:03] helpful for all of our other um information here. And one thing we can notice here is that it looks like the tournament ID is an integer here whereas the uh the tournament uh ID over on this is a string and it has the prefix WC
[11:16] dash. And so with Lakeful Designer, we've made it really easy to transform um your data using natural language. And so I'm just going to show uh just that. So I can literally just click on the genie code input box here and I can say make oh sorry if I can type make
[11:30] tournament id and uh a string and prefix with wc dash. This is going to go off to to gen code.
[11:42] And within lakeflow designer genic code is is fully agentic. So everything that we do uh can take multi-step reasoning. It can basically look at it can try things. It can look at the output. it can debug itself if it runs into issues. And so it's a really really powerful way
[11:54] to to transform your data. Throughout the presentation, like if I'm being honest, the way that I primarily work with Lake Foresigner is just by typing and and asking what I want it to do. Um, but I'll kind of switch between both the point-and-click way and using AI to show
[12:07] you just just uh so we have some different things here. So it looks like it was able to um successfully do this. And so we see it added a prepare operator here where I have a summary of the changes which says hey this modifies the data by applying a formula and
[12:20] updating it. Um and then if we can see we have another AI description here that summarizes that we add a WC prefix um to it. And then uh we can see over here on my data that match or that uh
[12:32] tournament ID is now a string with WC dash. And one of the cool things in link designer is we can do the input and output side by side so that you can you can really easily um validate your changes. And so if we go over here, we can see tournament ID on the input was
[12:45] an integer and then tournament ID here is now a string with uh the right information. So this looks great. So I'm going to go ahead and hit uh accept. Now what if you want to use uh rather use the points and click way to do things?
[12:57] You can also do that super easily. So if I go over here to the input data, uh we can also see that match date here is a string. And if I want a filter to be after 2012, this isn't super helpful. So I'm going to go ahead and hit over here
[13:10] to add an action. And then I'll say parse date here. And then I can just select the the column that I want to to do here. So I'll go to match date. And then we can say, yeah, this format's okay. And we can set to null on parse error and hit um apply.
[13:24] So then again, it's really easy to to kind of use the point and click to to do that as well. Um the next thing that I want to do here is I've noticed that if I go and search this if I search east um we have East Germany and then we also
[13:37] have West Germany here. Um which you know I want to combine those all to just be Germany so we can rep represent those at the same country. And so uh so I want to yeah I want to basically replace the values for West and East Germany just with Germany. And another way that you can do this is with this um AI
[13:50] description here. This is one of the ways that we're trying to kind of bring AI and infuse it throughout the experience. But everything that you do via the point-and-click way um will be shown in the description here just naturally. And then also I can edit this
[14:02] description gen directly and it will also then show or change the operator configuration. So I can go here I can say um replace east and west Germany
[14:15] with Germany for example and then we can send that out. And so the way I like to use this is I'll tend to do kind of like big multi-step transformations with the the main input box here. And then if I want to make small tweaks, it's really easy
[14:27] to do that uh just by editing this uh this input box here and then having it regenerate the UI just for that specific operator. Um while that's um continuing to generate here, I can go ahead and talk
[14:39] about a few other parts of the UI that are nice. So one is we have this um fit view button. So if you ever get like, you know, lost or somewhere, you can hit out fit view and it will kind of put everything back into the center of your view. Obviously, you can zoom in and
[14:51] out. You can also have this auto format. So if I like, you know, move this over here to be weird and I hit this auto layout button, um, it will put it back to to get into a nice state. So if you ever like accidentally mess things up, you can get back really easily there. All right, it looks like it was able to
[15:04] do this. And um, here's just like a really cool thing, right? So I asked it to do this. I didn't tell it what columns to work on. Um, and it actually ended up saying, hey, look, I think both in the opponent and um the country
[15:16] column, we need to replace this. And so it was smart enough to just infer to replace the value in both of those columns. Um, so that looks great. The next thing uh that we need to do is we need to filter to after 2012. So again,
[15:28] this is really easy. You can just create a filter operator here. I'll hit the plus button here and I'll go ahead and hit filter. And then from here, we can select our column. We can say I want to filter on the match date which we converted to uh be a date and I can say
[15:42] we want to have after and then the date. We can just go ahead and select 2012 here and then we can go ahead and say January and we can select January 1st of 2020 for example and then hit apply and
[15:56] then now we should see um only data after 2012. Filter is also cool because you can you can kind of branch with this. So for example, I can say I can do things with the rows that were included from this filter and I can also branch
[16:08] from things that were excluded. It's also really helpful if you want to like do some data quality checking or something. You could check and say, "Hey, remove nles." And if you see any nles, you could basically filter those to a different uh branch and then do things with that uh if you need to
[16:21] afterwards. All right, so as you can see, it's really easy to use Lake Designer to to to build out and transform your data. I could go ahead and do this for the rest of the the presentation to build out um the aggregations and join the tables
[16:34] together, but I'm going to show you a pretty powerful feature, which is the ability to transform by an example now. Um and we'll we'll see how how we can give it basically a pretty open-ended prompt and have some pretty complex um responses from Geniode. So, I'll
[16:46] actually go back over to um this file here. I will do take a screenshot here of this table. We'll switch back over here. I will
[16:58] paste this in. You can see we have the table here. And I'm just going to say transform my data step by step to fit the format of the table in the image.
[17:12] And we're going to give that prompt to Genie code and we're going to see uh what it can do here. And while it's working just to kind of talk about the complexity of this. So I need to do some aggregations for this. I need to calculate the the power rating which is
[17:24] based on this formula here. So that's a separate step. Um I also need to do some aggregations on the World Cups table and I need to join those together to see uh to get the total information here. And then there's also some rounding here. So
[17:37] you know I've also specified like hey I want two decimal points for average goals per game. I don't want any decimal points for total matches one or for number of world cups um one. And for for power rating we want also some some rounding. Um and so I didn't specify
[17:49] this. I didn't say like hey when you do this like you know make sure to aggregate these and join these tables together. Like I just just gave it a prompt. I said, "Hey, this is what I want to get my input to look like. Can you help me with that?" And I don't know about you all, but I often know what kind of I want to do. Um, and then work
[18:02] backwards from there. And so, I find this really powerful to just basically start with what you want and give that to Genie Code and then see uh what we can do from there. So, let's go back over to uh the tab here. And it looks like it has um completed everything. And
[18:16] it's just maybe checking everything to see what's happening. Um, okay. So, a couple cool things that's just done automatically here is it also went ahead and over to this tournaments um file and it removed or replaced West Germany and East Germany. I did not say to do that,
[18:29] right? But it just noticed I did that as a as a step above and so it's like, hey, you probably want to do that for down here. It did the aggregations here. Um it calculated the power rating over here. Then it did an aggregate here and then it joined the two together and then
[18:41] it did rounding for us. And we can see now with our final output uh we have the country of Brazil. We have average goals per game with two decimal points. We have total matches one. We have the power rating. And then we have the number of World Cups um one which is uh
[18:54] which is which is pretty awesome. I don't know. I don't know. I think that's pretty cool. What do you think? >> Yeah. Okay. Awesome. And I think one of the really cool things about this, right, is that, you know, with cloud code, you maybe
[19:06] could have gotten a similar output, but then you'd have like a thousand lines of code and it wouldn't really be easy to understand what happened here. But with Lake Designer, we can see kind of step by step. And if we want to make tweaks to each specific area, right, you can really easily go in and you can do that.
[19:20] Awesome. I will go ahead and accept that. And now the next part of the the demo was my manager asked me to graph this on a map. Um, which you know, we don't have any latitude or longitude
[19:32] data. We don't like so that this is kind of a a bit of a challenging thing with Lake Designer. We can do visualizations do basic visualizations. So if I click you know add a visualization here uh you know then I can just describe like a
[19:47] power rating by country for example and hit enter here we can get a basic visualization and graph here but it's not going to be on a map like my manager asked. Um, so we can see I'll actually
[19:59] switch over to have just shown the output only here. So we can see this, right? So you can see we we have this, but there's no uh there's no map graphing here. And so um, one of the things maybe I need to do is I need to go get the latitude and longitude for
[20:11] each of these countries. And I could probably ask Genie Code to to just go ahead and do that for me. But maybe that's actually a reusable thing that I want to do more often. And so I'm actually going to pass over to Emanuel to show us how easy it is to extend the
[20:23] built-in capabilities of Lake Vo Designer.
[20:36] >> All right, thanks Jason. Um, yeah, so you know uh Lakefold Designer ships with a lot of built-in functionality and Jason has done a great job showing some of these, you know, very uh standard things like filtering, joining, combining data, preparing data, all sorts of stuff. Uh but there's always
[20:49] cases uh of uh functions or things you might want to run here in LakeFlow Designer that do not come with the built-in set of uh operators that we have. And of course, as Jason mentioned, you know, we could, for example, use a
[21:02] little SQL or Python escape hatch operator to kind of plug in some custom code to do stuff. Um but oftentimes, you know, the target audience uh for layflow designer are people who feel more comfortable in a no code, low code
[21:15] environment. And so for them ideally the experience should be something like they come in here and say I want to do latitude longitude um and they should be able to see a result here that they can then go and drag out uh and then connect
[21:29] like everything else uh to upstream data here you know I have some world cup data similar to to the one Jason has and then I could say I want to select a country column here uh and then this particular operator operator here behind the scenes
[21:42] will augment the in uh the input data to include a couple of new columns like um latitude, longitude, continent, and and ISO. And so, you know, this is not built-in functionality that chips with
[21:55] designer, but it feels and behaves like built-in functionality, meaning you get a nice no code way to interact with it. You know, very simple form here you can fill out. It's the same kind of UI that you can drag and drop onto the canvas uh
[22:08] and so forth. And then similarly we want uh Jason to also be able to do something like that for kind of some advanced visualization use case here in the example. So he should be able to say something like globe uh and then you
[22:20] know get a result there. Drag that out again. Here the UI is a little bit more complicated. There's a few more fields to fill fill out. Uh but kind of same idea. You know Jason could just go and plug this into any existing uh lakeflow
[22:32] designer workflow that he already has. So again, you know, this is not built-in functionality, but it behind the scenes, we're using um this uh concept that we call userdefined operators, which is
[22:44] essentially the best way to think about it is that it's the sort of extension mechanism for LakeFlow Designer. And so you can think of this as like being able to package organizational knowledge or complexity into reusable building blocks
[22:57] that you can then easily share with your entire organization. Um and so you know these uh are being used by customers already today and some internal use cases for various things. Just to name here a few things uh for
[23:09] example you might have a custom formula or KPI that you want all your business users to always compute the same way. Us is the perfect mechanism to sort of encapsulate that business logic once and then share it with everybody in the in
[23:22] the team. Maybe there's some advanced data science packages or statistical analyses or whatever that you want your business users to have access to. Um, you could package that that up into these UDOS's and then share share with
[23:35] the team. Maybe there's APIs you need to pull from uh or push data to external systems. So, type of integrations that you want to standardize. Again, great way to do that here with UDOS's. Uh and maybe lastly, maybe you're in like an
[23:48] industry that has specific needs, specific analytical needs, um that are not very common u not built in into LakeFlow designer. So this mechanism would allow you to kind of build up your own toolbox that you can then share with
[24:00] your team. Um not going to go too much into the details here, but the way you you would actually build those is relatively simple. There's these kind of YAML schemas you can fill out to give some context around like what is this thing
[24:12] doing? What are the inputs and outputs? what kind of UI elements do you want to show when the user is using this? Uh, and then I think the most important takeaway is that under the hood it's
[24:24] just plain vanilla Python code. Um, so everybody that feels comfortable uh writing Python code or you know everybody who's comfortable using Genie code uh can write uh some of these and then share with others on the team.
[24:36] So cool. Um you know uh we now have two new operators uh one to go from country to latitude longitude and one uh that does a nice map visualization. So, let's see if Jason can use those.
[24:54] >> Awesome. Thank you, Emanuel. That's uh yeah, that's just what I needed. So, now if we go back over to uh to Lake Flow Designer, I will delete this uh visualization. We do not need that anymore. And I will do exactly what Emanuel said. So, I'll go ahead and hit
[25:06] the plus button. I'll search latitude longitude here. I'll go and hit this country name to latitude longitude. I will select country here. and then hit apply.
[25:20] And then we should be able to see now I have appended on to my data. I have latitude, longitude, the continent, and the ISO code, which is which is perfect. Um, and then the next thing I need to do here is visualize this on a globe. And so thankfully, Emanuel created that for me as well. So I can go ahead and hit
[25:32] the plus button here and type globe. And I can hit the globe visualization. I can fill out latitude for this. I can go longitude for longitude column I can
[25:45] select let's we can do the power rating for the value column and then for the label column we'll do the country and go ahead and hit apply here and then I'll switch this again to be um output only so we can see everything here and as you
[25:57] can see now we have this awesome uh globe visualization here that basically plots different countries uh with their world cup power rankings for the last uh
[26:09] two world cups. I can change this and go ahead and make this crystals if I or orbs if I wanted to or crystals, right? And so you can see I have this really really rich visualization directly in Lake Designer. So there we go. We have
[26:22] now done everything my manager asked me to do uh without writing a uh a single line of of code. There are uh are a few more things that I want to want to show you as um this is not not all you can do with Lake
[26:35] Designer. So the next thing that uh that I want to show you is so it's great that Emanuel could create um his his own an operator for me but what if I wanted to create one of these reusable building blocks by myself and
[26:47] you know I'm I don't know how to write code. How am I going to going to do that? Um and that is where the ability to create your own uh custom operator um via Genie Code comes uh in line. So I could go back over here. Maybe we can uh
[27:01] unconnect these right here. And maybe what I want to do here is instead of having uh the country name, what if I want to have a flag um basically emoji represent uh the the country name um on
[27:14] my graph. Um so maybe I want to create an operator which then goes from an ISO code here to an emoji flag which then we can then use. I can go over here and I could look and I could say oh maybe
[27:26] there's like a ISO um to flag you know operator here. nothing uh nothing really comes up here and so I can hit this create custom operator button or I can just say ISO
[27:38] and we'll say ISO to flag emoji and I can just describe in natural language I can say um take an input column which has ISO country codes and append another
[27:52] column which contains a flag emoji for that ISO code I can go ahead and hit generate operator And uh while while this is working out,
[28:04] we think this is a a very cool uh technology and idea that helps keep the UI simple for you so that you only have what you need and then when you need to expand it, you can basically generate new UI on the fly um as you're working.
[28:17] So we'll uh we'll let Genie Code go ahead and uh and add this for us. U maybe while it's creating this, I can go ahead and talk a bit about some of the other organization um capabilities that we have um in Lake Designer. I could add
[28:29] um a note if I wanted to. So I could say, oh, let me add this in here and say this is where I convert from ISO value to a flag emoji. So if you wanted to
[28:41] deliver kind of like flow designer file as a final report, you could you could document um all you wanted here. Basically, this is just markdown. Uh so you could write whatever you want uh within that. Um and then it says, okay, it says looks like I created this. So
[28:54] now we can go ahead and go back here and we can hit the plus button here. Sorry, I keep clicking off of it. I can search ISO here. And we have ISO to flag emoji is one of the uh the options here. So I
[29:07] can go and pull this in. I can connect these. Now it asks me to select the column. So I can say I want to select the uh ISO column here. And I can hit uh apply.
[29:21] And then in uh in a matter of seconds, we should see if we were able to to generate uh this this on the fly and um if it worked. So we'll give it a give it a second more to to run this.
[29:35] All right, moment of truth. Let's see if the demo uh if the demo worked out. Okay, awesome. We have the flag emoji over here generated. So you could see how easy it was to to basically just uh expand. And now I have
[29:47] this reusable building block. So every time I want to basically go from ISO to flag, I can I can use that. So then we could then go over here and we could connect these together here. And I could change this to instead of doing the country, I could say let's do the flag
[30:00] emoji here. Hit apply. And now we should see if I go back over and I scroll down or scroll in, we can see we have the flags. We we clicked on
[30:12] the Switzerland uh flag it looks like there. Um so awesome. So now that you can see how you can build those custom operators for yourself even without writing any code. All right, a few things to to wrap up with before we open
[30:25] it up for questions. So we mentioned that everything that you do behind the scenes creates production ready code. Um and so that makes it really easy to productionize uh stuff. But what does what does that look like? Um well it looks like the fact that it's just a a
[30:37] file in the workspace. And so you can see I have this this file here. Um, and so if I want to put this into git, I can just, you know, click here to, uh, to open the git modal. And, uh, then we should see, uh, basically we have a source file that
[30:51] represents this. It's actually a notebook under under the scenes under the hood. So we can see we have like things for, uh, for the notebook, boilerplate code. Um, and then we'd have our actual code down there uh, as well. It also means that we can schedule
[31:03] things really easily. So if I want to to go over here and just hit the the schedule button, I can schedule this to run every day. I can schedule this as part of a job if I want to chain multiple different flows together. Um I can put this into an existing pipeline
[31:15] if I wanted to because it all is all just this code behind the scenes. The other thing that we can do here is in this example I was just kind of delivering uh uh an output visualization. But if I wanted to basically materialize this information.
[31:28] And so say hey I wanted to go over here and I wanted to create an output um to keep this you know you could have the option to write to a materialized view in uni catalog or I could write to a table and if I click table you know I
[31:40] could create or replace I could merge I could append rows you basically just select where in uni catalog you want to write that and then you're able um to do that we also uh could mention with userdefined operators there's a lot of
[31:53] stuff that you could do so for example if I wanted to send an email uh with an output if I wanted to write directly to share point uh you know I could do that if I want to send a Slack message right you could do that and so it's really easy to extend the built-in capabilities um of Lake Designer with these user
[32:06] defined operators

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.