Skip to main content

Lakeflow In Action (Gourmet Pipeline Demo)

Summary

  • Unified platform: Lakeflow simplifies ETL and streaming through Spark Declarative Pipelines, while Lakeflow Jobs automates orchestration and Lakeflow Connect links enterprise systems.
  • Real-time transformation: The demo shows how a food company transforms messy, real-time sales data into targeted AI marketing campaigns with automated quality checks.
  • Business outcomes: Lakeflow enables dashboards and Genie spaces that deliver business solutions without exposing engineering complexity to business stakeholders.

Lakeflow In Action (Gourmet Pipeline Demo)

Watch: Lakeflow In Action (Gourmet Pipeline Demo)
Frank Munz, Principal TM Engineer at Databricks, showcases how Lakeflow—with Lakeflow Connect, Lakeflow Jobs, and Spark Declarative Pipelines—transforms data engineering.
See how a fictional food company converts messy, real‑time sales data into targeted AI marketing campaigns with automated quality checks.
Lakeflow simplifies ETL and streaming with Spark Declarative Pipelines (SDP), automates orchestration through Lakeflow Jobs, and connects enterprise systems directly to the Lakehouse using Lakeflow Connect—all within one unified platform for data and AI.

Chapters

FAQs

What are the three main components of Lakeflow?

Lakeflow has three key components: Lakeflow Connect (connects enterprise systems to the Lakehouse), Lakeflow Jobs (automates orchestration), and Spark Declarative Pipelines (simplifies ETL and streaming).

How does Spark Declarative Pipelines simplify data engineering?

Spark Declarative Pipelines provide a simplified approach to ETL and streaming, making it easier for data engineers to build and maintain pipelines without managing low-level complexity.

In the food company demo, what business outcome was achieved?

The demo showed how real-time sales data is transformed into targeted AI marketing campaigns with geolocation-based personalization and AI-generated nutrition advice with hallucination checks.

How does Lakeflow hide technical complexity from business users?

Business stakeholders see results through AI/BI dashboards and Genie spaces—visualizations like donut charts, maps, and sales ranks—without needing to understand the underlying data engineering.

Full transcript

0:00 Business Challenge

Hi everyone, I work as a TM engineer here at data bricks. And for this demo, I want to tell you about a food company that was drowning in its own success. The food was excellent, but they suffered because of their IT. They had data that lived in silos and pipelines that broke at every sprint and nobody trusted a single source of truth. In addition to that, a new chief marketing officer walked in and told the IT team, "Look, I think you're kind of missing the AI boat.
I want you to generate marketing campaigns for me based on fresh sales data and it should be targeted." So, I think it it should be different, you know, for San Francisco and for Sa Paulo and for Sydney. And since we're selling health food, I want you to also use AI to generate nutrition advice in those campaigns. And by the way, we have a lot of Spanish-speaking customers. So I expect automatic translations to Spanish.
And then I'm kind of worried that such an LLM might generate might hallucinate. So maybe we can have some basic check at least that there's nothing negative um going to the press. So now let me show you the solution that we built for that uh CMO. And if you think about this for a second, there is obviously a lot of data engineering required. But does that CMO want to see like privilege settings? Does he want to see triggers and streaming data pipelines?

1:16 AI Dashboard Solution

The answer is a clear no. He's expecting a business outcome, dashboards, genie spaces, apps. And this is what we use datab bricks one for. And you're now seeing datab bricks one. Um, which is like the portal we use to launch the business solution. And that business solution in our case is such an AI BI dashboard. And it's showing up here. Uh, it's launching now. And you see it comes with a nice graphics. And then on the top left you see this donut chart which is already taking three different uh data streams and displaying the outcome.
And the way I generated this was by just you know typing give me a donor chart for that uh solution for that data set. And that's the outcome. And another visualization is here the map. So we can use geolocations here. I I put this into t-shirt sizes from L to XXL. and you see where sales is happening, where we have franchises, etc. But the maybe the core outcome is this sorted list. Again, it's based on uh fresh data.
It's showing you the sales ranks. Uh the number one is here, San Francisco, but we also need we also know about the district, which is like district. And because it's combining different data sources, we also know this is using almonds as a local ingredient. Now remember the core ask was about these marketing campaigns and this is what I'm showing you here. So that's the solution for Bondai Beach which is actually taking the again the real time data like Bondai Beach and the amount of sales and the local ingredient generating this campaign and then it talks about like how it should look like and the snack is like a medium calorie snack that supports the surfers in the cold waters of Bondai Beach.
And the next column here is the Spanish translation. Um, depending on your Spanish, you can double check this. And then the right hand column is the sentiment analysis to double check that we're not generating anything negative. That's just like a very basic check um that we applied here. Now, how did we build that? And to show you this, I go back from datab bricks one to my lakehouse. Now you see the workspace of my lakehouse, which is the classic datab bricks workspace.

3:56 Data Workflow Overview

This is where we can do the implementation. This is where you see all the technical details. And this is what I want to use to dive deeper into lakeflow. Now clicking here on jobs and pipelines because if you want to understand about a bigger project, I always recommend to start with a workflow with a job. You see it here. It was running already like a couple of times. It's all green. So my workflow is doing well. I run it again just to show you that this is uh happening live.
And uh you can run this actually yourself. We share the we share the demo. Uh you see there's like three tasks spinning up at the same time. they start to blink which is a good sign. If I go into this workflow, you see them executing in parallel. So this is the scant view that helps you to find the critical path that shows you like which task is taking how long and where does it make sense to optimize. They're all running now at the same time.
And the way I define them is that they need to finish first before the next one like my pipeline that does the core processing um kicks in. That should take a little bit more. Let's see. Yeah, still executing. Oh, now you see the pipeline is kicking in and starting and um is starting to execute and the other three tasks are finished. Now I go back to my workflow. I want to look at the tasks. This is the task view, the graphical representation.
Let me zoom in a bit and move this. Okay. left hand side you see ingestion lakeflow connect three different uh manage connectors accessing sales data salesforce data XML pipelines it's actually deployed automatically this is why I want to disconnect from the source tell you about this more later I'm disconnected now so I have more freedom to add the node and this is where I want to show you how easy it is to in add another iness pipeline I use SQL server here just you know to show you how this would work.

5:54 Ingesting New Data

And basically, it's asking me for the classic database credentials that you all know, like host, username, password, and port. I would type them in, click create connections, and that's all I need to create a new inest pipeline uh in my workflow. Now, I don't want to do this because I already have three and they're working perfectly. I don't need the fourth one. That was just to show you. So left column inest pipelines from connect and then the second column the second uh task here is my processing pipeline.
Um and to get there I already found it and all I need to go is uh click on this uh little launch uh thing here. And now I'm in my new pipeline editor. So there's a built-in pipeline editor. I'm running the pipeline again again to show you that this is all happening live. And then let's start here on the left hand side. I have this little file browser that gives me all my transformations, all the transformations. The pipeline consists of and you see I modeled this as br silver gold.

6:51 Data Transformation Pipeline

So it's the classic medallion architecture. You're totally flexible. Uh you can use SQL, you can use Python. Uh I was asked if you could have like another layer like platinum and you could just add another folder. So this is all like uh best practices but you have all the flexibility and uh you can move files around uh you can add new folders, you can exclude folders from the execution. Uh I think this new pipeline editor is really excellent.
I use it all the time. And uh here is like let me just generate you a new folder. Let's just say, well, maybe we want to, you know, have special uh tables that we use as platinum tables. It's as easy as that to just create a new folder. Now, I'm not using it here. That was just to show you um the flexibility. And I could also take this file and just drag it into another folder. And um again, I'm not doing this here. Now, here on the top, you see the multi-tap view.
So it's uh also super easy, you know, to copy stuff over from one data set to another data set. And here you see we're using a materialized view which is a aggregation that is processed incrementally. And that's the full view of the pipeline that was running. And the cool thing is if I click on the nodes of this uh graph view, I immediately get the data that is in these nodes. So this is like a streaming table that we used for append only ingestion.
And um the cool thing is if you want to find out what's going on in your pipeline, it's like uh super convenient now to just click through these nodes and then um see the fresh data. This is another node. It's a materialized view. Again, if I click on this, this has a data quality constraint attached. It's an expectation that I model in the code. And here you see the outcome like I expecting the city to be uh in a correct setting and that actually works in like 2.
2% of that fails because I well I have data which is not like uh perfectly um perfectly well and two 2. 2% of my data sets fail. There is another expectation here that never fails. And um so you see this outcome is visible in the graph. But I can also go here to columns. I see um the schema which is super important if we detect the schema from you know a semistructured file or unstructured uh file. And then we can do schema enforcement and schema uh migration.
You see the performance of um such a statement. Again, that's the materialized view. Okay, so let me go back to the workflow. These are the inest pipelines. This is the pipeline with a transformation. This is a simple if then node based on a task variable. And this is where the magic happens. The AI magic happens. So it's a SQL task that looks like this. I define a function. The function is having having input parameters and those input parameters again they come from the gold table of my pipeline and then it's calling an AI query uh function that we host we provide that you can specify any LLM any frontier model that is available in datab bricks and then the cool thing is here we have these parameters and this is where we kind link the AI to your data so it's literally applying AI on top of your data in one unified platform that makes it such a big big difference and unlike uh Jet GPT where you just copy uh snippets of some text and this is the part where I define the calorie restriction.

10:00 AI Magic and Automation

So let me go back to the other node and this looks so different. This is a AI translate function. Very simple. You don't specify a prompt. You don't specify the LLM. And then we also do the uh sentiment analysis. We host these functions. We optimize them for you. Super simple to use. You have the choice between, you know, going for AI query or just using that. And that's the last task that is interesting. Updating the dashboard.
I have two more tasks, but they're just uh triggering notebooks. Um don't forget that we can always, you know, put a task uh in a notebook and then just run the notebook. I haven't showed you the triggers. This is how I could trigger my workflow uh based on a table update, based on a schedule, based on file arrival, have it running continuously. Here in this demo, I was doing this manually, but that's what you typically do.
Set triggers for a production workload.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.