Skip to main content

Lakeflow with Databricks One, AI Functions and Agents

Summary

  • Gourmet pipeline demo: Combines Lakeflow Connect for ingestion, Lakeflow Spark Declarative Pipelines for transformation, and Lakeflow Jobs orchestration to create dynamic marketing campaigns from live enterprise data.
  • AI at scale: Demonstrates operationalizing LLMs and AI agents with Databricks One dashboard view, AI/BI for visual analytics, AI functions for localized recommendations, and sentiment analysis for campaign safety.
  • Real-time personalization: Generated location-specific marketing campaigns (e.g., medium-calorie snacks for surfers in Bondai Beach), automatic translation to Spanish for Spanish-speaking countries, and sentiment analysis to prevent negative campaign generation.

Lakeflow with Databricks One, AI Functions and Agents

Watch: Lakeflow with Databricks One, AI Functions and Agents
In this demo, Frank presents the DAIWT Lakeflow demo Gourmet Pipeline from an AI perspective, showcasing how to create dynamic marketing campaigns that leverage your enterprise data. See how data ingested via Lakeflow Connect is transformed with Lakeflow Spark Declarative Pipelines, with the entire process orchestrated through Lakeflow Jobs.
Learn to operationalize your LLMs and AI agents with live enterprise data, demonstrating how Lakeflow integrates seamlessly with Databricks One and AI Functions to deliver real-time, data-driven marketing insights and campaigns.

Chapters

FAQs

How does Lakeflow enable AI-driven campaigns?

Lakeflow Connect ingests data, Spark Declarative Pipelines transform it, Lakeflow Jobs orchestrates workflows, and AI functions generate personalized recommendations. Databricks One provides the business view.

What personalization does it achieve?

The pipeline generates location-specific campaigns analyzing local sales and customer profiles. It recommends medium-calorie snacks for surfers in Bondai Beach and auto-translates for Spanish regions.

How does sentiment analysis protect quality?

Before generating campaigns, sentiment analysis prevents negative content creation. This ensures generated campaigns are safe before deployment.

What does the technical view show?

The technical view displays Lakeflow Jobs orchestration with graph showing concurrent pipelines, multi-tab SQL/Python files, real-time sample data for debugging, and if/then nodes controlling AI operations.

Full transcript

0:00 Introduction to the demo

Hey everyone, speaking from the tech marketing team at datab bricks. I want to talk to you about this gourmet pipeline demo. How they solved the data problem with the data intelligence platform and how this fictitious company unlocked AI at the core of their business. Talking about AI and talking about business, the core business view is data bricks one. This is where you see dashboards, genie spaces, and apps, but none of the lower level um techy things that we have in the lakehouse.
So, basically, I want to get to my uh dashboard. So, I type gourmet, and it's the first search result. And that's the AI BI dashboard. And um this donut chart is already showing me the live sales data that is updated like every second. We see the geographic location um of our stores. So that's a small one in Austin and that table that's interesting one again realtime sales data city local ingredient and the district but now we come to this marketing campaign for new recipes.

0:32 AI/BI dashboard overview

Uh so that's the one for Bondai Beach which is specific for the location. It's specific for the sales. It's also specific for the folks you know in this location. So for example, we recommend a medium calorie snack for the surfers in Bondai Beach cuz they spend all the time in cold water. And then our Simo said, "Look, you have a lot of customers in Spanish-sp speakaking countries make this translation automatically to Spanish.
That's what we did here." And then he's kind of scared like we would generate something negative. That's also why we do this sentiment analysis. So I switch back to the lakehouse view which is the more technical view and you know all these architectures that typically start with lakeflow jobs that's the orchestration system that orchestrates everything. So I'm just running that job again and this is the run. If I click on that run you can see the graph and these are the three uh lakeflow connect uh ingest pipelines.

1:43 Technical lakehouse view

If I click on that pipeline, you see the lakeflow connect uh pipelines. They're running at the same time. And now also this other pipeline is spinning up. This is where I do the ETL processing. You see the multi-tab views. I have SQL files. I could have Python files and at the same time the graph is updated. If I click on those nodes from the graph, you see the real time data at the bottom. So you see this sample data super helpful for debugging.
Back to the job. This is an if then node and we have two nodes, that one and that one for AI. So that's batch inference. We create a function which is this gen marketing campaign function. It's nicely documented. You see the parameters like district, city, country, ingredient. And then again it's nicely commented like what it does and you will understand why this is important. The core thing what this function does it's calling AI query.

2:44 AI function implementation

AI query needs the name of the LLM you want to talk to and then we have this lengthy string this lengthy prompt but that prompt is uh you know using the parameters that you get from your lakeflow pipeline like city country gradient district and this is where you link your data to the AI so it's not just chat GPT for the model you can select any of these hosted models and you just pick one you grab the name like set 4 and then once I copied the name I can go back and I could overwrite my claw 3.
7 with the set 4 model then at the end you need to use the function you uh use the function in this create table statement and this is how you add the functionality going to this one and this is like so different it's using just AI translate and that's basically the whole code it's a single line translate the description to Spanish and store it under this new column name or do the sentiment analysis for the transcription.
Much easier. You don't have to specify the LLM. You don't have to create the prompt. We do all this for you. We optimize it for you. Something else, if you go to the playground, of course, you could say, why do you need all this? And you just like again, we have all these models and uh let me pick one. Yeah, let's take this four. If I just take the model and don't you know don't have any data with that model and I type in a similar prompt like give me the matcha drinks for gou pipeline first of all it doesn't know about gourmet pipeline you know it's just creating something like LLMs are fantastic for always creating something it's giving us a marketing campaign it's talking about digital marketing about partnerships about launch events and success matrix like everything you would expect from such a campaign But it's not related to our data.

4:19 Experimenting in playground

Now talking about agents, an agent is giving a tool to such an LLM. The tool is an such a function like a Unity catalog function, the one we just talked about. Now it's assigned. Now what I want to do is I want to take the same prompt and I copy the prompt. I run it again and you will see what happens. It's thinking. It kind of matches the prompt calls the new function and it even fills the parameter for the function like city, ingredient, country, district.

5:06 Building and testing agents

And then look at the output. This is now coming from our agent. And the agent again is, you know, talking about the matcha cookies, San Francisco. And it says medium calorie. And let's read more. You know, that should be good enough for the neighborhood with the famous hills. I don't think I I don't think I need to say more about this. You all know about San Francisco. That's it. This is how you can create an agent. Now, you probably think, well, that's just the playground.
If I could code that, I would be in a different salary uh class. Well, you don't have to code it. We give you that notebook. You just say, "Show me the notebook." And then you have a a template notebook that basically does the same thing including everything including evaluation of the agent with the agent evaluation framework. One more thing the function should do that function that's the view in unity catalog. You see unity catalog gets the comments.

6:26 Summary and conclusion

It's using those comments to map the function that you want to call. Uh and it understands you know if you if you submit the right prompt it will call this function. That's why it is so important to have these comments. So to summarize, I kicked it off with data bricks one which is launching the AIBI dashboard. The AIBI dashboard is showing you the marketing campaign which is based on a SQL statement that is using AI query functions and those AI query functions they are based on lakeflow data on real time Lakeflow data with all the data quality and orchestration behind it.
And then at the end you can even use the same UC function generate an agent and then call this agent uh externally if you wanted. I want to say thank you for your attention and hope to see you soon.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.