Lakeflow Designer: No-code ETL— Authored With AI, Built for Production

Summary

Lakeflow Designer gives analysts a no-code, AI-first way to build production-grade data transformations on the same Lakeflow execution model used by data engineering teams, without moving data outside Databricks.
AI-powered operators and Genie Code enable analysts to ingest, clean, enrich, and normalize data through drag-and-drop interactions, including API enrichment and AI-generated screenshot parsing for unstructured inputs.
Designer-authored logic is production-ready and integrates directly with Unity Catalog for lineage and governance, allowing data engineers to operationalize analyst pipelines without rebuilding or refactoring.

Lakeflow Designer: No-code ETL— Authored With AI, Built for Production

As demand for data grows, central data teams can’t build every transformation themselves. But traditional “self-service” ETL tools often introduce new problems: logic leaves governed platforms, pipelines live in parallel runtimes, and productionization requires costly rewrites by data engineering teams. Lakeflow Designer addresses this gap by giving analysts a no-code, AI-first way to do data prep on the same Lakeflow execution model used by data engineering teams. Instead of introducing a parallel toolchain, Designer produces governed, production-grade transformations that fit directly into existing data engineering workflows. In this video, you’ll see how Lakeflow Designer: Enables analysts to do data prep without moving data outside Databricks Preserves centralized governance, quality, and lineage through Unity Catalog Lets data engineers scale pipeline development by directly operationalizing designer-authored logic— without rebuilding, translating, or refactoring
Talk By: Emanuel Zgraggen, Staff Software Engineer, Databricks ; Jason Messer, Product Manager, Databricks ;

Chapters

FAQs

What is Lakeflow Designer?

Lakeflow Designer is a no-code, AI-first data preparation tool on the Databricks Data and AI platform that enables analysts to create production-grade transformations using drag-and-drop operators. It runs on the same Lakeflow execution model as data engineering pipelines, ensuring governance and lineage are preserved throughout.

How does Lakeflow Designer preserve data governance and lineage?

Designer pipelines run within Databricks and produce governed transformations tracked in Unity Catalog, maintaining full data lineage from source to output. This prevents the governance gaps that arise when analysts use separate self-service ETL tools that operate outside the governed platform.

Can a data engineer directly use pipelines built in Lakeflow Designer?

Yes — Designer-authored logic is production-ready and can be directly operationalized by data engineers without rebuilding, translating, or refactoring. This allows central data teams to scale pipeline development by delegating initial transformation authoring to the analysts who understand the business logic best.

What role does AI play in Lakeflow Designer?

Genie Code in Designer helps analysts generate and modify transformation operators from natural language descriptions, including handling edge cases, normalizing data, and generating API enrichment steps. AI-augmented screenshot parsing can also extract structured data from images like tournament brackets or PDF reports, enabling unstructured inputs to be incorporated into governed pipelines.

Full transcript

[00:08] Hey everyone. Uh, my name is Jason. I'm a product manager here at Data Bricks that works on LakeFlow Designer. I've been at Data Bricks for just over three years. Worked on a lot of fun products, but I do have to say Lake Designer is I think my favorite product I've ever worked on. So, excited to talk to you all more about it today. And then this
[00:20] is Emanuel. >> Nice to meet you all. Yeah, my name is Emanuel. I'm the tech lead on Lake Flow Designer and uh, very excited to show you all what we've been building for the past year. Awesome. Yeah. So, we're going to do um Oh, I guess yeah, we'll do some slides
[00:33] and then we'll jump into a demo. We'll get a manual back, but for now, it'll be just for me. So, um just to start off, we have to put this forward-looking statement. This basically says um if I say anything that doesn't come true, you can't hold me accountable. Hopefully, that won't won't happen. Uh but we just
[00:45] got to put this up there. Um and then also, please do afterwards, you'll get a survey. Please do fill that out. It's really helpful for us to know what you liked, what we could have done better, what you'd like to see more of next time. So, please do uh take the time to to fill that out for us. All
[00:58] right, today um it's uh it's 5:20. Everyone's probably exhausted already after a long day. So, we'll hopefully make this uh this fun, but we have three things that we want to do today. One is I want to talk briefly about why we built Lake Designer. We'll do just some
[01:11] brief slides on that. Then, we're going to do a live demo for the majority of the time, and then we'll try and open it up for for Q&A afterwards. And then also, if we have if we have more questions, we can take those outside and uh because we'd love to talk to you and hear any of your questions. So, with
[01:24] that, let's jump into why we built LakeFlow Designer. So kind of the idea here is that everything that you do on data bricks starts with good data. Whether you're building AI models, whether you're building apps, whether you're building dashboards, everything
[01:38] needs a foundation of good data. But the problem that we've seen is that the people that have the deepest understanding of the business are often not the ones who can build production data pipelines. And so put in other words, the people who need to actually
[01:50] use an action on the data are not the people who know how to kind of self-s serve and procure that themselves. So, this kind of creates the need for a no code self-service data prep and analytics tool. And at data, I I would
[02:03] be lying if we said we're the first to come up with this idea. There are tons of other really great products that do
[02:15] transform their their data uh without writing code. But the challenge that we've seen when talking to to users with um these separate tools is that you kind of have a silo and that your work happens in two separate worlds. On one end, you have the central data platform. This is where
[02:28] everything is governed. This is where you have all your production stuff working. And then on the other end, you have these low code tools. A lot of them run on your local machines. And so you might download a file off the the central data platform, put it onto the low code tool, and then do it. Um and
[02:42] then if if you want to bring that to production, it's it's kind of hard. And so when talking to customers, we've seen there's kind of three core problems with this and then uh one maybe underlying problem with it as well. The first is that with these existing low code tools,
[02:54] you have siloed workflows. So you have data in your central data platform and then in this other tool, it's a headache for it to manage it. They're like, man, I've got a bunch of different tools to manage. You often have people do the same exact analysis like multiple times
[03:06] on these local tools because it's not put in a central shared place. And there's also governance gaps. All right, so you took your data, you downloaded it out of a govern environment onto your local machine, uh, which can be tough. The second is that it's really hard to
[03:18] bring these workflows to production. So, you know, you might build out uh with all a workflow that has, you know, say 200 different uh boxes on the screen and then it's like time to productionize it. And so, uh, when the time comes to hand
[03:31] it to your engineering team, they're like, man, what do I do with this? Like, this is like a bunch of boxes on a screen. Like, how in the world do I do I bring this to production? The third thing that we've seen is that sitting outside of the central data platform, there's really limited AI productivity.
[03:44] So, a lot of these tools have tried to to add on and bring in AI. Uh, but you can't reason on all of the data uh like you do if you're sitting within the central data platform. And so, we've seen that as a problem. And then kind of undercutting all of it is that these tools often have expensive per user
[03:57] licensing costs. So, you know, you have to decide beforehand who is worth it to give access to this tool because you're going to pay, you know, $10,000 to give access to this tool. And so that really limits who you can give access uh in the first place.
[04:09] So that's why we came up with with Lake Designer which is our Noode data prep solution that is native to to data bricks. I'm going to touch about this briefly because we'll we'll talk most a lot about this during the demo. But Lakeful
[04:21] Designer with everything it's drag and drop. We've worked really hard to create a really really simple UI that's um drag and drop for your business users and less technical people so that they can basically transform their data. It's also um built from the ground up to leverage AI. It's been a fun problem as
[04:34] we've developed it in the era of large language models. And so we've really been able to think from first principles like you know what is it like to build a product with that is centered on AI to help you be more productive. So I'll be able to show that in the demo and
[04:46] hopefully you all notice that as we as you use the product. And then third is everything in Lake Designer um creates production ready code behind the scenes. So you do everything in drag and drop but behind it is all production ready pispark code. It's just it's just plain
[04:58] code. Um and so if you want to productionize it you can do that. So if we look at those same three pain points, uh we kind of have a contrary slide here where we solve all those things. So the the first thing is that everything happens in one environment. It all
[05:10] happens in data bricks in your central governed environment. You have a simple path to production as everything's just code. So if you want to schedule it, if you want to make it part of a larger pipeline, if you want to um put it in a data bricks asset bundle or automation bundle, um you can do that. And then uh
[05:24] since we built it from the ground up to leverage AI and we have all of the rich context from being on the central data platform uh we are able to do a lot to make you very productive with AI. And one of my also favorite parts is that it makes AI very reviewable. You think with
[05:38] like cloud code you can generate thousands a thousand lines of code but it's like what does this do and withful designer we see uh that everything gets built out in a visual way so you can really see easily what's happening. And then kind of undercutting that is you have no perceived fees. So it's standard
[05:51] data bricks pricing which is you just pay for the compute that you use. All right is a bit of why we built Lakeful Designer. That was more slides than I even like to do. Um so now let's let's jump into a a live demo. And before we do that I do want to kind of
[06:04] set um set the scene. So kind of the the case that we're going to be interacting here is like so I have my boss Emma. I'm Jason going to represent the the business analyst persona and then Emanuel is going to represent the
[06:16] engineer persona and we're gonna all work together to kind of solve and build something out. So my boss says, "Hey Jason, I need you to help me put together some World Cup data from 2012 to now. To do this, I think you'll need to combine the all matches data in data
[06:28] bricks with this tournaments Excel file I just put together. Can you fill out this table and visualize it on a map?" And then we've been given a uh a table. So, she wants to have the the country average goals per game, the total
[06:40] matches won, the power rating, which is calculated with this formula down here, and then the number of World Cups um won. So, with that, let's go jump into a demo.
[06:56] All right. Hopefully the the Wi-Fi works out. It's it's been a little bit spotty today. Um so, awesome. So, you can see here we are on in Lake Designer. I'll just show briefly how to get into it, um just in case people are wondering. So you can go to the new menu here and say visual data prep and then this will
[07:08] create a file and put you into uh the interface here. I'll go ahead and close this. I have this file that's already in a git folder so we can we can work with that later. Really the first thing you need to do when you're in Lakeful Designer is you need to add a a source file. And so I'm
[07:20] going to go ahead and click select a source here. And then I have a bunch of different options for how I can bring data in. One is I can browse existing data. I can create a table from a file. I could drag and drop a local file. I could also connect to Google Drive uh or
[07:32] or SharePoint. For the purpose of this demo, I'm just going to browse the existing data that I have. So, I have to go in here, type all matches, and we'll hit that. And then while my compute spins up to show us uh the
[07:45] result, I can orient you towards the the UI here. So, within Lake Designer, I'll actually minimize this so we can have more space here. On the left, we have our operators. These are things that um you would use to build out your transformation. So things like your
[07:57] aggregate operator, your combine, filter, join, limits, prepare, pivot. You know, we have a bunch of different options here for how you can transform your data. In the middle, you have the canvas. This is where you'll kind of drag things out. So, you know, I could drag out onto the canvas. One of these
[08:10] operators here on the right hand side here, you have the configuration panel. So, this is where you would specify, you know, if you have the join, like which two tables I want to join on, what are the join keys, etc. And then, uh, on the bottom here, you have the output table.
[08:22] This is where you see um, your data. By default, we work with a a sample of your data so that it's very fast and efficient. But if I if you wanted to um to change this to work with the full data set, you could do that. Or you could also change the sample to be, you know, a million rows or however many
[08:34] rows you wanted to process with with each operator. All right. So, um we've just brought in this data, but now I need to bring in this Excel file that my boss gave me. And so, uh with Lake Designer, we've made that really easy. Uh we realize
[08:46] that uh in practice, you know, you might often have data that's not in data bricks and need to bring that in. Uh, and so if I just hit uh tab here, I have this Excel file here, and I can just drag this onto the canvas. I basically
[08:58] say where I want to put this, and I've already I've done this once before. Um, so I can hit uh upload here. And then you can see in just a matter of seconds, we should see um the uh the sample from
[09:10] the Excel file that we um uploaded. Um while this is um this is loading, um I can go ahead here and talk about this this data a bit. So basically this data
[09:22] is um different match data from the world cup. So we got the world cup now. So we want to maybe give some uh educational stuff about historical world cup matches. Um so we have like for example this is the country was France. They played Mexico. You had how many
[09:34] goals they scored, how many goals the other team scored and if they won um etc. And then in this one we have um just basically information on different World Cups. So you have like the the year of the World Cup was, who the host country was, and then who won. Um so
[09:47] basically to get the data that my boss asked for we'll have to um do some preparation. We'll have to join these two tables together um and then do some aggregations and uh and get the output here. Um so that is uh is all great. So before
[10:03] we start preparing our data I'll show one other thing in Lake Designer here we have is called data profiling and this is a really easy way to basically get a quick overview of your data. And this is not just available in Lake Designer but also in the SQL editor and notebooks uh as well. So, if I click on a column here
[10:16] in the result table, I'm then able to click on this selection details and I can see I'm just basically a quick overview of the data. So, like I can see here that Argentina has won 13.6% of the time. Brazil's won 22.7% of World Cups,
[10:28] which is kind of a wild stat to me. And then there's only eight countries in total who have won the World Cup. Uh, which is like I don't know, for some reason that's that feels really low to me. So, maybe we'll see a new country win uh this time around.
[10:40] Awesome. Okay, now it's time to uh to prepare our data. So the first thing that uh that we can notice here is that if I look at this tournament ID here, it's um it looks like it just has the year 1930. Whereas if I look at the
[10:52] tournament ID here, it has WC and 1930 and this is a string. Um and so maybe one of the things I need to do is um transform this uh this data. So with Lakeful Designer, we've made it super easy to to do this. Um I can just go
[11:04] over here and I can say um transform the tournament ID column into a string and and and prefix with WC dash for example.
[11:20] Lakeful designer leverages genie code and so everything that we do is basically in a gentic loop here which is uh which is awesome. So it's going to be able to um look at the the data. It's going to basically make the proposed transformation. It's going to then look
[11:33] at the output to see if it's correct um and then give me it can iterate based on that if it needs to and then give me a final output. And so we can see over here it created a prepare operator and in this we had a formula that looks like
[11:45] it transformed uh um it concated WC and then made it uh into a a string here. And my favorite way to validate that things are happening correctly in liquid designer is to look at the um output where we have the input and output side
[11:57] by side. And I can see tournament ID here 1930 and then I can see tournament ID here um is a string and then we have the WC prefix to that. Um throughout this presentation, you know, I could do everything pretty much with Genie code and just asking it to to build stuff
[12:09] out. I'm going to use the both the drag and drop way as well as uni code to kind of help you um see some of the different capabilities. So I'll go ahead and accept this one. But for this next one, I'll just use um the standard kind of points and click way. And so the next
[12:21] thing I want to do here is I want to do match date. Uh it's a string. I want to transform this to a date as we want to filter to after 2012. So I can go ahead and go over here. So if I scroll and hit add action here,
[12:33] I can say parse date for example. Uh, and then I can pick the the column here which is match date. I can say format. I can keep it the same. And then on parse error set to null. That looks good to me. And then I can hit apply here. Um,
[12:45] and then we should see uh this update here. And so now if we go over to the match date column, we can see that it is a date. So that's great. The other thing that uh that I realized is um when
[12:58] looking through this is that we have the uh country is East Germany um as well as West Germany, which we just want to make that all just Germany. Um and so there's a there's an easy way to to do this. And
[13:10] um we could use the the main input box, but I'm actually going to show another way that we've kind of embedded AI into Lake Designer, and that is by using the um this AI description here. So this is a really cool field because it stays um
[13:23] up to date and is birectional with the output configuration. So we can see here that you know I add things here and then this updates and vice versa if I want to update this it will update the configuration. So I can say um replace
[13:38] east and west Germany with Germany for example and then this will then go ahead and update the configuration. So, the way I tend to use this is I'll use the uh the main Genie code input box for bigger
[13:51] changes that are across multiple operators, but then when I want to tweak one thing specifically, I'll often go into this um little sub AI description and then tweak there. Um because this is really great for this. Like if you're filtering on a certain value, it will
[14:03] describe that um and you can change that really easily. Um while it's loading this, I can uh orient you to a bit of the the rest of the canvas here. Um so up on the the top left, we also have the ability to like fit view. So, for
[14:15] example, if I like was to to zoom in, for example, a lot, we could hit fit view and that would show everything here. Um, you have the ability to obviously zoom in and out. You can auto layout, which is really nice. If uh if you're manually putting things together and it's getting messy. Um, and you can
[14:27] also change from uh basically drag uh to selection mode. All right. It looks like Genieode was able to to do this for us. And uh I I mean, okay, one thing I think is really cool is it changed this in both the opponent and country. So it was
[14:39] smart enough to look and say hey like there's probably going to be multiple columns where this is going to be a problem in so let's change it for both and so if we then search here we can search east we have Dutch East Indies but we don't have East Germany and same
[14:52] with West for example we don't have West Germany um so I was able to do that next thing we can do is my manager asks for after 2012 so I'm going to go ahead and hit the the plus here and I will say um sorry not the prepare plus here and I'll
[15:05] hit the filter operator and again super simple ble to do this just say select a column so I want to go to the match date we have a bunch of different um filter conditions that are different depending on the data type so since it's a date we
[15:17] can say after for example and then I can just specify we want to go to we go to January and we can go to 2012 here and hit January 1st and we can hit apply here and that should uh filter for us
[15:31] really easily. All right, there we go. So you can see how easy it is to both use natural language to kind of transform your data um as well as to do point-and-click um ways to to transform your data. I could do this for the rest of the the presentation and kind of
[15:44] build out this um this uh table that my boss wanted me to build out, but I'm actually going to show a really cool way to and a powerful way to use Lake Designer to to save time and that is the ability to transform by example. So I'm going to hit control 4 here. I'm going
[15:57] to take a screenshot of this. We can go back here. I'm going to paste this in. You can see the the image here. And I'm just going to say transform my data step by step to fit the format
[16:12] of the table in this image. I'm going to hit enter. And while it's thinking about that, let's let's go back and look at how complicated this actually is. So the first thing is we need to do a bunch of aggregations um
[16:25] for this. So we need average goals per game, total matches won. We need to calculate this power rating which um has this formula down here to do that. Then we need to also do rounding. So you can see I have uh two decimal points for here, no decimal points for total
[16:38] matches one, three decimal points for power rating. Um and then three or no decimals for number of world cups won. And I didn't specify any of this, right? Like I could have been much more prescriptive and said like hey like you know have the two decimals, make sure to use aggregations, make sure to join with
[16:50] this. Uh but genie code I think we're going to find is is detail oriented oriented enough to know based on the screenshot alone. So, let's go back here and see how it's going. And it looks like it has already done it pretty much.
[17:03] And wow. So, it's even it's even cool, right? Like here's another thing like I didn't even realize um it went ahead and and replaced East and West Germany in this other tournaments um thing, right? Because it saw that we did that above and says, "Hey, you might need to do that as well." It then uh it calculated
[17:17] the power rating here. It aggregated by countries. It joined them together to get the World Cups. Um and then it rounded everything. And we can see our final output here is we have uh a country we have average goals per game matches one two decimals there power
[17:30] rating um looks looks good and then we also have the total number of world cups um one so it looks like uh we did a great job here and I think the cool
[17:42] thing just to to realize is that like you know maybe cloud could have just done this for me but what I I wouldn't have been able to understand right like what happened and so this is awesome because I can see step by step how my data changed and I can check every step
[17:54] of the way, right? So, I can look and I can see the power rating here and I can see if this is calculated correctly. Um, and I could tweak that if I needed to, right? And so, we're able to to really easily kind of build out this flow to transform our our data. So, this looks
[18:07] great to me. So, I'm going to go ahead and hit um accept here. Now, I think did I actually say this or did I maybe pull this out? I might have pulled this. Oh, I forgot to read this last sentence, but my boss says, "Hey, can you visualize
[18:20] this on a map?" Um, so that's also kind of a challenge. How would I visualize this now um on on a map? Uh with Lake Designer, we have the ability to do kind of basic visualization. So, for example, I could go here and hit the plus button and say visualization.
[18:34] And then, you know, for example, I could go ahead and I could use AI here instead of configuring it manually. And so, show like a power rating by country, for example. And that should show me
[18:47] just like a basic bar chart um of the the power rating by by country. But we're not visualizing this um on on a map. Oh, sorry. I have to connect the data. Looks like I hadn't connected it.
[18:59] Okay, perfect. And then this should Yeah, there they go. This should load. I'm actually going to switch to be the output only here. Um so we can see this better, right? But you can see now we have the power rating per country. Um but this is not on a map. And so in order to do that, I need some latitude and longitude data. And I probably could
[19:12] ask Giode to uh just basically go find an API and pull latitude longitude data. But I actually think this is something that might be useful for uh to have a more reusable component. And so I'm actually going to hand over to Emanuel
[19:24] to show us how easy it is to extend the built-in functionality of Lake Designer of what we're calling userdefined operators. Grab my water
[19:39] tab here. Um and so yeah, this is just a different designer file here, but um one of the things we want to do here is we want to make sure that JSON has a really no code experience. uh for doing this latitude longitude conversion. So ideally what he could do is he could
[19:52] come in here and say lat long and then he could see that there is a uh search result match here. He could then go and drag this out uh and then you know gets a nice UI like this where he can say um
[20:05] the column I need to uh select here that I want to transform is my country column. Hit apply. Uh and then uh hopefully in a second here we should see that the input data that Jason had was
[20:17] just country and total goals. But now this operator here um augments the augments the data set to give a few additional columns latitude, longitude, continent and ISO. Uh and then similarly
[20:29] we want Jason to be able to go in here and say okay I want to now do like a visualization on a on a globe or on a map. He should be able to search in here again, get a nice search result back here, be able to drag this out and again
[20:42] have a completely no code experience uh to do these two things uh that he wants to do. Um none of this actually ships with LakeFlow Designer and that's what we're going to talk here a little bit about which is this new mechanism that we have which is called userdefined
[20:55] operators which essentially the the way to think about it is it's the extension mechanism for um LakeFlow Designer. And so you can think about maybe there's like organizational knowledge or
[21:07] complexity uh that you want to package up into reusable building blocks so that business users like JSON can just use uh that through drag and drop without the need to to write code. And so obviously Genie code is really good at actually
[21:19] generating custom code. But these reusable operators here often provide a faster and more reliable way to share proven functionality across teams. You can make sure these things are tested.
[21:31] uh you can make sure um that users who use genie codes always get the same results. They're not there's no LLM non-determinism involved uh that kind of thing. And so internally and externally uh people have started using this
[21:44] extension mechanism to do things like encapsulate business logic. Let's say there's a particular KPI or formula that you want your business users to always compute the same way. UDOS's are the perfect way to sort of share that across
[21:57] your team. Maybe you want to uh give uh advanced statistics or data science tools into the hands of your business analyst. Uh again, UDIO is the perfect way to expose that. Maybe there's uh
[22:10] standard integrations like pulling data from an API or pushing data to an external system that you always want your business analyst to do the same way. Again, perfect example for when to use UDOS's. Uh and you know perhaps uh
[22:22] lastly here if you're in a a specific domain that requires particular analytical functions or whatever you can use this mechanism to kind of really build out a custom toolbox for LakeFlow Designer that your team can then use. Um
[22:36] this is all available today. I have a link to the documentation here. Uh you can go and play with this see if you can build your own operators here and uh you know we're hoping to get feedback and kind of see what cool things uh people build with this mechanism. So this was a
[22:49] little bit about the why we want to do this or use this userdefined operator mechanism. Uh but I also want to briefly show you a little bit how this actually kind of works under the hood. Um and so here uh I previously created one of
[23:01] these boxes here. You will actually see that there's a link that says view source file. If you click that it will bring you into a YAML file that looks a little bit like this. And so these YAML files are essentially kind of the
[23:14] blueprint of this userdefined operator that that I created. Um and to walk you through here a little bit uh in the top there's a little bit of boilerplate stuff like you can give a version. It supports versioning. Uh you could give
[23:26] it a name, you could give it a description that either the user or uh Genie code can use to kind of figure out when and how to use it. Um then there's a couple of sections that talk about how the inputs uh and outputs of this uh
[23:40] operator are configured here. Again really simple we have one config which is a country column. Everything in config is what ends up in the UI on the right hand side. Um so the user will see a country column that they need to
[23:52] select. We can specify what is the UI widget we want we want this to have. Um and so in this case we're using select which you know kind of like a dropdown uh where the options of that dropdown are populated from the input columns of
[24:06] the data that flows into that operator. We can define the input and output ports. These are the little handles that you see in lakeflow designers where you can connect edges to or from. Um and
[24:18] again really simple in this example here. You know we have one input data frame and then we do some massaging of that data and then we flow uh one data frame back out. Uh and then there's a way to actually configure the code that
[24:30] runs behind this uh userdefined operator uh in Python. To show you a little bit more about that, let me switch into the notebook I used to actually create this particular userdefined operator. And one
[24:42] important takeaway here is this is plain vanilla Python. There's nothing secret about anything I'm doing here. So whatever you can do in Python, you can do in one of these userdefined operators. So here I went ahead and I
[24:54] created a function um where first I pull out these input parameters that we discussed before. I get the input data frame. Uh I get the country column that the user specified. Um and then I'm
[25:07] doing my actual uh task here which is I'll I'll download some CSV files, publicly available CSV files to do the mapping from country to latitude and longitude. I do some caching here just to be smart. um and then you know a
[25:20] whole bunch of wrangling of that data and but ultimately what I'll end up uh doing is something like this down here where I take the original data that flows into this new operator and append all these columns the latitude longitude
[25:32] continent and ISO and then spit it back out. Um and so this is a a quick little example of a a custom userdefined operator. Uh I have another one here that we briefly looked at this um globe
[25:45] visualization one. You know kind of same idea here. Rather than doing some wrangling in pi spark uh what I do is I load some HTML file and then use display HTML to actually render it. Um so again
[25:58] you know kind of same idea. Um and then in these notebooks I uh also um can do a little bit of testing. Again it's plain Python codes. You know you can use your favorite unit testing environment whatever C you can put this
[26:10] in CI/CD. Uh here I just do a very simple thing where in the notebook I give it some test data run this through kind of manually check that this makes sense and so this looks all good. I also
[26:22] programmatically actually build up this YAML file um just to keep it a little bit easier for myself to manage. Um and we also do have support to for you to validate these YAML files. There's a
[26:34] schema so you can make sure the YAML that you're writing actually uh makes sense. And so yeah, that's sort of the the the way you would write one of these uos. Looks a little bit scary. There's a lot of boiler plate here, but um Genie Code is amazing at writing these. Um so
[26:47] a lot of this is actually just boiler plate. So we have a skill that you can use to do this. Um and so yeah, you can just point it at an existing example and it will be able to figure all the details out here. Uh and you know, you can really focus on actually writing
[26:59] sort of the the key logic in in Python here that you want the uh userdefined operator to have. Um one other um quick aspect I wanted to highlight is so these are how we define
[27:11] these operators. We also need a mechanism for lakeflow designer to find all these operators. Uh and here we kind of do a similar thing uh as we do with skills in Genieode for example where we have these do userdefined operator YAML
[27:24] files that can either be in the root of the workspace or in the user home directory. U and these are just simple lists essentially where you list all these YAML files that you want designer to know about. Uh and then ultimately
[27:38] what happens is if I go in LakeFlow designer I scroll down here a little bit I see that there's a section user defined operators and you know all these ones show up here country name to latitude longitude globe visualization these are the the ones we just wrote uh
[27:51] but then you can see our team has you know been testing this and you know somebody wrote a SMTP email sending operator somebody did some slack notification thing here um so a whole bunch of stuff um yeah if we go down the
[28:04] list whatever yeah rest API source so You know, again, really the sort of sky's is the limit because under the hood, it is just plain old uh vanilla Python. Cool. Um, so maybe Jason uh let's see if you can use some of these things I built
[28:17] for you. >> Let's do it. >> Awesome. Cool. Thank you, Emanuel. Um, let's go back to another we're working.
[28:29] Um, we can go and delete this visualization here. And yeah, so it's super easy as a business user to then just use these functions that the manual created. I just hit the plus button here. I can just search for uh latitude
[28:41] longitude, country name to latitude, longitude. I can select country here and I can hit apply. And then we should see um it will go out and fetch uh that. And so we see boom, we have uh our power rating, everything
[28:54] we had here, and then we have latitude and longitude as well as the constant and ISO. And then now my boss wanted me to visualize this. And so thankfully Emanuel helped me out with that as well. So I can go over here in here and hit the plus and I can say globe here and I
[29:07] can hit the plus button here and I can select latitude, longitude, I can select we can let's do the power rating. That'll be a fun one to visualize. And then the country as the label column here and hit apply. And
[29:21] we can make this bigger here so we can see it. And then now I have this uh awesome globe visualization uh right here in Lakeville Designer. Um I can hit space deposit if I want to change to maybe
[29:33] instead of having beams look at crystals. Um and then you know when I hover I can see the different power rating for um different countries. Now I can really easily visualize that. Um
[29:46] don't know maybe that that looks like deserves an applause like there you go.
[29:59] this right like I schedule this to run every day. If I wanted to store this in git I can you know have this in a git folder. Um and so I can just you know put this in git. I can do CI/CD with it. I can parameterize this. I can put it in
[30:11] a in a dab as I mentioned earlier right like so the sky's is the limit for what I can I can do with this file which is in production or to bring to production. There's um there's one more thing that I wanted to to show off um one more
[30:23] feature and that is what if I as a business user want to create my own user-defined operator um again without writing code and this is something that I think um I think we're hoping is going to be a bit of the future of UI and it's
[30:35] kind of the idea that you can keep UIs very simple and targeted to what you want to do and then when you want to expand them you can build your own UI uh essentially right so maybe something I could want to do here is I have this ISO
[30:48] Maybe I want to get an emoji that represents each of the countries here. Um, and then I can I can use that in my globe visualization. So, something I could do here, let me just uh delete this connection here. I could hit the plus button. Um, for example, I can
[31:01] search like ISO because maybe I'm looking for that. Maybe my team has it. Um, looks like I've done this before. Uh, so it looks like there might be one already. Uh, but I could go create a a custom operator here and hit create. And I can say ISO to flag. And I can say um
[31:16] take an input um column that has country ISO values and append a new column that has an emoji of uh with a flag for that
[31:30] ISO value for example and uh Lake Designer is going to uh go ahead and generate this uh this user defined operator on the fly.
[31:43] Um, while it's doing that, um, I'll talk briefly just about a few of the other, uh, operators and and things that you can do to organize your canvas. So, I could go add a note if I wanted to document like what's going on at each step. So, for example, I can say, oh,
[31:55] like in this step I do, you know, XYZ or whatever I want to put here. Um, I can also group things together. So, if I want to group things and minimize things, I can do that. Um, so you can have your have tools to basically
[32:07] organize your work here. Um, so you can kind of present this as a final report that makes sense to to anybody. Um, it looks like it has said that it's uh it created it. So, let's go ahead and um go back over here. We'll search for ISO.
[32:19] And then we had the arrow to flag here. I can drag this out here. And let's connect these. I have the ISO column. So, let's search for ISO for example. Um, and then hit apply here. Looks like
[32:31] I tried to run it before I had configured it. Um, and then let us see if my userdefined operator it worked.
[32:48] Okay. And it looks like we have a flag emoji now uh for uh for everything.
[33:02] that. And then when we get to our our visualization over here, we should now see the the flag emoji instead of the the country text. So if we go back over here, zoom in a little bit, hit space, we can see uh we have the flags. So uh
[33:14] so there you go. So hopefully with that you've seen uh how powerful it is to to use liquid design to transform your data without writing code and also how is how easy it is to extend it um to build things for your business users. Um and so uh yeah, that's it for uh us. We'd
[33:27] love to take any questions that the people have.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.