Databricks on Databricks: Marketing with CustomerLake, the new Agentic CDP

Summary

Databricks' own marketing team built and launched CustomerLake, a fully agentic, embedded CDP on the lakehouse, and used it to automate real marketing campaigns already in-market.
The system covers onboarding, segmentation, activation, execution, and performance insights through agents that interact with governed data on the Databricks Data and AI platform.
Transitioning from channel-specific data silos to a unified marketing lakehouse enabled a single source of truth for customer data, analytics, and campaign activation.

Databricks on Databricks: Marketing with CustomerLake, the new Agentic CDP

In this marketing practitioner deep dive, the Databricks Marketing Technology and Engineering teams walk through how we built and launched a fully agentic, embedded CDP on the Lakehouse — and used it to automate real marketing campaigns that are already in-market. We’ll reverse-engineer the live demo you saw during the keynote, and show how onboarding, segmentation, activation, execution, and performance insights are carried out by the Databricks Marketing Team. Attendees will see how agents interact with governed data, how automation layers plug into the Lakehouse, and how marketers can deploy campaigns without fragile point-solution tooling.
Talk By: Melanie Chow, Senior Software Engineer, Databricks ; Paul Volk, Marketing Technology Solution Architect, Databricks ;

Chapters

FAQs

What is CustomerLake?

CustomerLake is an agentic, embedded customer data platform (CDP) built natively on the Databricks Data and AI platform. It allows marketing teams to run audience segmentation, campaign planning, activation, and performance insights using AI agents that operate directly on governed lakehouse data.

How did Databricks' marketing team use CustomerLake?

The Databricks marketing team used CustomerLake to automate real campaigns already in-market, with agents handling audience blending, personalized messaging plans, suppression logic, and content approval. This replaced fragmented point-solution tooling with a unified agentic workflow.

What problem did CustomerLake solve for the Databricks marketing team?

The team operated with channel-specific audience silos, meaning every campaign required manually building audiences, designing journeys, and writing messages across disconnected systems. CustomerLake replaced this with a unified lakehouse and specialized agents that handle campaign execution end-to-end.

What are the four pillars of CustomerLake?

The four pillars are audience (building and managing segments), content (personalizing messages), scheduling (determining when to send), and activation (executing campaigns across channels). These pillars are coordinated by AI agents working on governed data in the lakehouse.

Full transcript

[00:08] So, welcome to Databricks on Databricks marketing, where we are going to give you a sneak peek behind the scenes of how we run a gen-tech marketing campaigns on Databricks. My name is Paul, and I'm on the Databricks marketing team as the marketing technology solutions
[00:21] architect. Which basically means I look after our martech stack. I'll kick things off with a look back at our very own Databricks marketing team journey. From building our data foundation to discovering the need for a gen-tech
[00:34] execution. And I'm here with Melanie. >> Hi everyone, I'm Melanie. I'm an engineer on the new customer lake product we just announced. And I'm excited to talk today about how we're bringing our marketing and data teams together.
[00:49] >> But before we get into what we built, it's worth understanding how we got here. And honestly, we started like most marketing teams, executing directly in channels. An email here, paid media ad there, no
[01:03] shared view of the customer. All of our audience segments were built and activated in their own channel-specific silos.
[01:15] But we made a big bet. We wanted to centralize everything on Databricks. One single source of truth for all segmentation data and user behavioral actions, which means data analytics and marketing activation,
[01:27] they can happen in the same place. But we quickly realized that having all the data in one place was awesome for our data teams. But it didn't make campaigns any easier to run for marketers. Someone still had to build every
[01:40] audience, design every journey, and write every message. And this was our new challenge. And this is the realization that led us here today. What if instead of doing campaign
[01:52] execution manually, we got help from a specialized team of agents? And that's the story we're going to walk you through today. But first, let's start with the data foundation because we all know that a
[02:04] solid foundation is what makes all of this possible. So, you heard Ali in the keynote stress the importance of building a solid data foundation. This was a big deal for us. Going from a fragmented set of
[02:18] channel-specific systems to our unified marketing lakehouse where all of our customer data, our account data, and our user engagements, and ML scoring were all in one place for the first time.
[02:31] For our data team, this was a huge win. One source of truth for data analytics and campaign activation. For all For all the data edge folks in the room,
[02:43] you know where you're at in this journey, and it is a long one. But, I'm hopeful that today we can convince you that it's worth the effort. But just to set expectations, great data does not automatically mean great
[02:56] campaigns. We had everything the data team could want, but marketers and marketing ops still had to build everything manually to deliver a marketing campaign. So, it turns out the hardest part was actually
[03:08] in the execution. So, about a month ago, we were sitting here together thinking not here, but you know, on Zoom um thinking about uh what we would demo for you guys today. And we conveniently
[03:21] got this message from our VP of marketing ops. So, every marketer and marketing ops or data engineer in the room has received a version of this message. Short, urgent, notice the 5:00 p.m. timestamp,
[03:35] and carrying enormous hidden complexity. The real ask here is that we wanted an email campaign with personalized data and AI summit session recommendations based on each person's individual
[03:48] engagement history. So, how do we actually execute on this? Well, we did what any reasonable team would do and we looked at our existing campaign execution process.
[04:00] And it is rough. But don't worry, I'm not going to drag you through this whole thing. But I did have Claude read through our actual process docs. I see people taking pictures, it's not that great.
[04:12] And exaggerate it for comedic effect, so you'll get that. Um so this is either going to feel very familiar or very triggering. But let's just start with the color coding. Yellow is the marketer, blue is
[04:25] the data knobs team, and gray is all teams suffering together. So just to highlight a few areas, we start with our strategic planning. We have our classic slack intake from
[04:37] leadership and don't worry, there's a formal doc, a formal strategy doc, and a 47 question campaign intake form coming up next to dial in that strategic vision.
[04:49] That's three separate systems and at least two weeks planning. Next up, audience building. We probably need to build some new segmentation attributes, size the audience, then have
[05:02] a marketer review it only to find out they want a larger audience. Then QA it only to find out there's a 90% overlap with another audience that's getting bombarded by emails this same week.
[05:14] So let's give it a few more weeks to get that right. Next up, content development. Create the content brief, build the email, change the content, rebuild the email, repeat.
[05:26] Three weeks minimum. And finally, activation. We need to maintain ETL pipelines to push the user lists and the custom content to every channel.
[05:38] Every channel has its own integration, as I'm sure everyone in this room well knows. Your CDP is going to help here, but it doesn't come for free. So, the verdict? It'll take us about 8 weeks, and there's
[05:50] no way this happens before Data + AI Summit. But, if I had to sum it up, this process comes down to just four things. First, our strategy.
[06:02] Who do we actually target for our message to get the results that we want? Next up, building the audience. So, how do we build our audience without bringing a data engineer into the loop?
[06:16] Then, our content. What do we say to each audience? Not some generic template, but content that resonates and lands with each individual person. And then finally, our activation.
[06:29] When and where do we send our content without stepping on every other campaign going out at the same time? But, before we get to the solution, let me just show you how hard these execution steps actually are.
[06:43] So, here's what actually happens when you build an audience. And an audience is just a collection of rules that defines who we're sending a campaign to. Think of it as a giant where statement
[06:55] in your SQL query wrapped in a prettier UI. Here, you'll see an example audience. It includes three nested condition groups and or logic stacked inside more and or
[07:07] logic targeting high-value prospects in five key countries. And our resulting audience is about 1,400 or 4,100 users. Is that right?
[07:19] Is that the audience the marketer expected to reach? Did I mention marketers always want larger audiences? But wait, we forgot to add a filter for Data + AI Summit. So, I searched D A I S
[07:32] for Data + AI Summit to find any existing filter attributes, and I found about 20 options. So, is it past DICE attendee? Is it D A I W T engagement last 10 months?
[07:45] Or is it Dice years? These are all built by different people at different times. And even when I pick one, how do I know the attributes maintained? Do I just need to build a new one? Well, I guess I'm just going to go with
[07:57] is not past Dice attendee. And our target audience is gone. So, good luck explaining that to the marketer. So, all this back and forth is total process gridlock.
[08:10] But we haven't even gotten to the content. So, this is our campaign brief. And every campaign we run requires writing all of this content. So, just for the email,
[08:22] we have a subject line under 40 characters and body copy under 135 words. Very specific and very time-consuming. But here's the thing. Email's just one
[08:35] tab in this doc. There's also a landing page, there's creative promos, there's paid media ads. Each one filled out separately for every campaign. Now, multiply this entire document by
[08:47] the number of audiences you're targeting. Because you can't talk to a data engineer the same way you talk to a CEO. So, that's three audiences, three more content briefs. And true one-to-one personalization
[09:00] means one unique version per person. And humans can't maintain this level of scale. It completely falls apart at execution. Okay, so let's say you built your audience and you have your content.
[09:13] But here's the thing that you probably haven't thought of yet. Your audience doesn't exist in isolation when it comes to actually executing. This person right here, in the middle of these four different campaigns, is about
[09:25] to get four emails in a single week. One from our Data and AI Summit session recommendation campaign, but also one from a high-value prospect nurture, and two other campaigns competing for their attention.
[09:38] Not because anyone decided that was a good idea, just because the audience was built in its own silo, and no one checked for overlap, and now this person receiving all of these emails has no idea why Databricks won't leave them alone.
[09:50] And so they unsubscribe, and now they're out of every list forever. And that's the cost of building audiences in a silo without a system that knows the full picture.
[10:02] And it gets worse when you look at journey orchestration, which is basically just a ton of microsegments that fall follow this if this then that pachinko machine-style logic. And these journeys look totally
[10:14] reasonable in isolation, but as you can see when they run at the same time, they immediately start colliding. Someone in journey A is also getting picked up by journey B, and you thought you'd suppress them, but they just entered the wrong flow.
[10:27] And no one on the team has a complete picture of what campaign is going out to who and when. So journey orchestration is really designed for one campaign at a time. And the moment you're running more than
[10:39] one campaign or multiple simultaneously, you are responsible for your audience either getting hammered or getting nothing. And so, Melanie, we've established that manual audience building is hard. And
[10:52] one-to-one personalization of content impossible. And multi-channel orchestration, that's a mess. So we've got 1 month to launch this personalized session session recommendation campaign. What are we
[11:04] going to do? >> All right. Thanks, Paul. All right. So a lot of problems, huh? How are you actually going to solve this? And how are we taking that to transform how we do marketing at Databricks?
[11:16] So first, let's go back to Paul's process. When we thought about why this process was so long, we realized it's because today's marketing tech stack focuses on a bunch of point solutions.
[11:29] That means that every handoff between each process becomes this game of telephone across all these different teams. And so we realized about how to solve this, we realized the answer lies in the glue.
[11:43] You need a process where all these different teams can come together. And that's why we're excited to announce the launch of Databricks Customer Lake. You heard Ali talk about it in the keynote. Let's actually take a closer
[11:55] look. So what is Databricks Customer Lake? Customer Lake is an agentic CDP directly embedded in Databricks that equips your marketing teams and data teams with the workload of agents to deliver the
[12:08] perfect customer experience. Customer Lake is the only CDP directly embedded in your lakehouse. So it's the first for So for the first time, you no longer need to compromise
[12:20] on data governance for your most sensitive data. Security and trust is built in from day one. It's also platform built directly for marketers in mind. With this, marketers can now get a
[12:33] self-service access to your enterprise data in a way that actually speaks their language. Last but not least, Customer Lake is built for marketers and agents to work together.
[12:45] This is where we can utilize agents to come in, understand all your customer data, and personalize interactions at a never-before-seen scale and intelligence. And so with that, let me give you a
[12:58] sneak peek of the Customer Lake campaigns demo. Some of you were already in our session earlier today, so you may already seen this. We're going to talk more about it in the keynote tomorrow, so please tune in for that. For the sake of this demo, we're going
[13:10] to We're going to pretend we're an airline company, but don't worry, we're going to come back and talk exactly about Paul's problems. So with that, let me switch over to the demo screen.
[13:22] Cool. All right. So for this demo, I'm in our campaign page. And I'm going to think, "Okay, what do I actually want to do?" I think for this, actually my executives told me that we're actually trying to promote our new route from San Francisco
[13:35] to Lisbon. So, I'm going to tell exactly that. Uh so, I'm going to say, "Help me increase bookings
[13:49] from San Francisco to Lisbon this summer." All right. So, before I click start planning, I also want to call out that if you already have your plan, you can directly input it here. But, let's see what actually happens when I kind of give it something that's not too fleshed
[14:02] out. So, let's start planning. What you see here is that the Genie's actually going through all your customer data, figuring out your different destination platforms, and it's now going to ask me different questions to help me plan my
[14:14] campaign. So, in this case, you can see I have different options here. I think for this, I'm going to choose Braze, choose meta ads just so I have one email and one digital marketing.
[14:26] How many messages should the travelers receive? I don't actually know. I think I'm just going to let it decide for me. You know, I'm not actually sure what content actually exists. And when should this campaign end? I
[14:38] think because I'm going to target this summer, let's just choose 90 days because by then summer will be over. All right. So, let me draft the brief and let's see what happens. So, it's working. Awesome. So, you see here that Genie was
[14:51] actually able to come up with a whole campaign brief for me. You can see that I can edit the context, but it's actually able to come up with the gaps. It's able to come up with the primary goal, so it translated that to
[15:03] mean increase bookings on the new SFO to Lisbon route, as well as my conversion target, which are people who've actually completed this route. Lastly, I also have my channels listed here, my end condition, and my success metrics.
[15:16] Something super cool is talking about how we're actually blending planning with audience building. You can see here I only actually mentioned San Francisco, but it's actually taking that to translate it into our actual attributes. In this
[15:28] case, what I'm seeing is travelers with SFO as their home airport or people whose primary origin is the Bay Area. You can also see here that's actually translating things to things like suppressing marketing opt-outs and just
[15:41] giving me more details about my audience. Lastly, I have my goals. Came up with some messaging and what you see here is that it actually is connected to our destinations. So, it's only showing me messaging within Braze
[15:53] and Meta Ads. Lastly, we have this messaging plan that I let it type in for me. And what I want to call out is this section around guardrails and assumptions. You can see here I can still make sure
[16:05] I'm holding out 10% of eligible travelers. I'm going to make sure I cap Braze to one launch email per traveler. And I'm going to make sure I suppress customers immediately after they booked that SFO to Lisbon route just because
[16:18] once they book it, there's no need to target them again. I also want to make sure I'm actually not targeting customers with an active disruption, refund, or cancellation rate just because our company has this policy where we don't really want to give them another email just because, you know,
[16:31] usually leads to them not being too happy with us. All right. Something I want to call out is also assumptions. So, you can actually see what assumptions the agent is making. I can come in and tweak it in case these
[16:43] assumptions don't actually seem right. So, in this case you see that actually one assumption is that a traveler converts when they complete a booking regardless of whether or not they click the email or ad. Let's say this is something that my company doesn't actually want me to
[16:55] track. I can always come in and edit it and it can actually rebase itself. Lastly, I also want to call out the sources used. So, you can actually see why it's actually coming up with different details. This allows me to check my brief and see
[17:08] what I actually need to double-check on. So, you can see here it actually showed that I'm I'm the one that selected Braze and Meta Ads. Cool. So, I think this looks good. So, let me just build a campaign and see what
[17:20] happens. What I want to call out too is that this essentially is what that 47-page questionnaire that Paul was talking about and that really long Google Doc all in one.
[17:32] You can see here it's actually coming up with my audience directly for me. So, now I have my SFO Lisbon prospects and actually preview what that audience actually means. I can see that actually this audience means people whose home airport is SFO,
[17:46] um San Jose and Oakland, which is pretty cool, right? Because I only mentioned SFO, but it's or San Francisco, but actually it's meaning that to be in the whole Bay Area. Then I can see it's also looking at
[17:58] people whose website's destinations has been Lisbon or what else? Porto in the past 7 days. So, honestly that sounds pretty good to me. I mean, I didn't know that existed. All right, and people who've also searched for a direct flight in the last
[18:10] 7 days. And something I also want to really call out is this route affinity um attribute. This route affinity is actually something that our market data teams have been working under the hood to come up with what actually what routes are
[18:23] you usually able to book. And in this case, this is the AI predicted route where the customer is likely to fly next. And I actually want to make sure my route affinity is Europe just to make sure my campaign does well.
[18:35] And lastly, something really cool is this exclusion group. So, you can see here it you came up with people who've booked a destination to either Lisbon or different surrounding areas in the past 90 days.
[18:48] All right. So, I think that looks pretty good. Something else I want to call out is the section of the goals, which is our conversion metric. So, as as I mentioned, I actually want to increase my routes to Lisbon. As you can see it actually created a way to track that for me. In this case, my
[19:01] actual conversion metric will be people who are in these who have booked a route to Lisbon. So, you see here these are people who have booked a route. Essentially, my goal of my campaign is that this audience increases.
[19:14] And so it's all already tracked for me. You don't need to go back to your destination platform and figure out the numbers afterwards. That's how we're kind of blending in audience building and planning. But I also want to lastly touch upon the next
[19:26] phase of Paul's process, which is content. So you can see here, because I already know the destination platforms, it's actually already able to bring in the different content that it makes that it thinks it makes sense for. So SF photo Lisbon route launch, SF
[19:39] photo Lisbon route targeting, SF photo Lisbon route meta ads audience, I mean that sounds pretty good. And lastly for content, after that, we want to make sure we actually have activation.
[19:51] And we actually have this built in, too. We have your guardrails, but we also have frequency limits, so you can make sure you're not targeting the same people more than three times. So that four that person who's receiving four emails is now already handled for you.
[20:03] Databricks does it all, because we track everything. You can also make sure you have quiet hours, so you're not messaging your audience when they're sleeping. Cool. I also want to call out the section over here, in which you can
[20:16] always bring in more context, so that the genie can keep learning. So let me save draft. And what I have really here is also really cool. This is now a preview of my campaign, so I can actually send this
[20:28] out to my different team members, someone like Paul, and he can just review what makes sense. He can review if like this is the right way to track my campaign performance, and we can all come here to actually track what it actually look like on a
[20:40] message level. So I think this looks pretty good. So once I launch it, you will then be live. Cool. So that's it for the demo.
[21:00] All right. So those were all of those were all of Paul's steps. But what's actually happening under the hood, right? Like how is it actually doing all of that? And so, let's actually sit back and peel back the layers a little bit with our
[21:12] real-life example. So, we're going back to Data AI Summit. And so, in the demo, you saw that the agent was automatically able to take a plan and convert that to a built audience and more. But how does it actually know how to do
[21:24] that? And there's no secret, right? The answer all just comes back down to your data, which holds your semantic context. Because Customer Lake is an embedded CDP, it's actually comprised of the exact
[21:36] same tables already used by your data teams. And the best part is that it does it with no additional copy. So, your data gets to stay exactly where you want it.
[21:50] And because it's aware about how these actual tables join together, it's also able to turn your intent and turn that into attributes. It can come up with something as simple as age, or it could actually build a more complex attribute, like giving us all
[22:02] the years a person attended Data AI Summit. And how it do this is it would go through our events table, actually figure out what code means Data AI Summit, go through the timestamps, and put that all in one neat list.
[22:15] And it's not actual real magic, right? It's actually just SQL statements under the hood. But it's now SQL statements that you no longer have to manage yourself, and you no longer have to ask your data teams to build for you.
[22:32] And going back to Paul's issue about not knowing which what are the right attributes to use, because these are first-class citizens in our app, the agent is actually able to track all the metadata of how frequently these attributes are used, which one actually links to a live table, how recently that data you're
[22:45] looking at was, so that you can build audiences you never even knew you could make. And all those questions that you used to go back and forth on, like wanting to know the overlap, or wanting to find different ways to increase your audience
[22:58] size, or breaking it down by three three attributes, it's actually all already there for you to do. You can just ask Genie, and he'll surface exactly things like audience overlap and attribute breakdown.
[23:10] And lastly, over here, I really want to highlight the insight section. Regarding insights, we can actually do one better because we know all your contacts. Genie has information about your data, your brief, how are you how you even use
[23:23] Databricks, such that we can actually auto-generate insights that we think you find valuable. In this example, actually at Databricks, an important attribute is what life cycle a marketer is in, and you can see it's actually one of the
[23:35] first attributes that show up. All right. And for my data people in the room, guess what? It's actually already in your lakehouse.
[23:48] It's already in there without you needing to do any custom plumbing or any custom hooks. You no longer need to know if the data's already synced because audience segments are automatically entered into your catalog.
[24:01] This way, for my data analysts and scientists out there, if you ever needed to do more complex analysis, you can actually do it directly in the area you already love to work in. And the best part is that because we're
[24:13] embedded, you actually automatically get lineage tracking on these views. So, you can see what how the data's being used in audiences downstream. And I didn't forget about my data admins out there.
[24:26] All these query logs can actually be traced back down to customer lake. So, not only can you see who is accessing what, you get all the rich benefits we've already honed in around query insights.
[24:40] Lastly, because customer lake is channel agnostic, you can actually send it anywhere. However, customer lake is also customer aware or channel aware. For my marketing ops people in the room, there's no longer no need to figure out
[24:52] what columns exist in what destination channels, and what columns are required, or which one is snake case or which one is camel case? Because our agents are actually able to inspect the columns in your destination,
[25:04] track the different types, and ensure the proper mapping is applied. Back by Lake Flow, we're also building out direct integrations to many of your favorite ad channels.
[25:19] And for my data admins out there, don't worry. Each sync going out is automatically logged to the lake house so that you can audit exactly what's happening. With the power of agents, we can also help auto diagnose issues if there ends up being any sync failures.
[25:38] Last but not least, with bi-directional syncing, it's also never been easier to measure your data in one place. It's actually the same data bricks you already know and love for ingest and dashboards, but it's now actually baked in for your marketers out of the box.
[25:52] Super cool. All right. >> So, are we just done here? It looks like Customer Lake is going to help our team create their campaigns really easily with the strategy creation to build audience segments and a personalized
[26:05] content assisted by your team of campaign agents. So, like with all this talk about AI agents, how is Customer Lake preparing us for the future of marketing and taking us beyond
[26:18] that traditional campaign? >> So, great question, Paul. And you're right. I actually only really highlighted how customer how Customer Lake is going to solve your process. But with the way AI is shaping customer
[26:30] experience, we're thinking beyond just how to slap AI into every step. Quick poll from the room. How many of you actually use AI when deciding to buy something?
[26:46] See? AI is dramatically changing how people interact and buy with the world. And with the rapid pace of it all, sticking with old batch campaigns and journeys just won't last. And so, what we actually want to do is take your old waterfall process
[27:00] and throw everything you know about it away. Because we realized the question in that box is actually not the ones that matter. Right? Strategy is not about who do we target? It's actually what is our goal?
[27:13] Audience building is not about how do we target them? It's actually out of the people we can target, what exactly should be the message they're receiving right now? And that answer is fluid, right?
[27:25] Based on the user, it can shift by day, by time, as well as all the other campaigns your other teams is running. And lastly, content is not as simple as saying what do we say?
[27:38] It's actually what is the right messaging for this person? In fact, we actually think cohort level copy is almost anti-personalization. It's a cop out, right? Because beneath every segment are thousands more that
[27:50] actually overlap. And to the last part of your thing, activation. To achieve true personalization, it's clear that your content actually needs to be activation aware. Right? Like what
[28:02] channel are we targeting? When are they receiving it? What time of day are we sending it? And for true personalization, there's actually way too many things to take account for. There's your goal. There's the people you're talking to.
[28:15] There's the right channel. There's how they're viewing it. And there's all your different product signals that you already have. And it's clear that if you multiply it all, it's way more than a marketing team can handle.
[28:31] And so, what we really need is a system that replaces these rigid campaign waterfalls with a continuous agentic loop. And I'm happy to announce that this is what we're calling Infinity Campaigns, which is what Ali mentioned.
[28:43] Because what we really need is a system that already knows your business context and already has all your customer data. And we want to use that to decide, okay, what is the right action to take based
[28:55] on everything we know? Based on our goal priorities, how has someone like Paul behaved or engaged so far? And how should we take action? And then, once we actually decide to activate Paul, should we send him an
[29:08] email? Which one? Should this email be to take precedence over the other ones? And how do we best display this information And because we get actually all this data logged, when the customer receives
[29:21] that message, we can now assess how that action actually did and then utilize that in order to improve this loop. So, how does someone like Paul change over the course of this campaign?
[29:33] Did his engagement at least go up? Did he decide to come to Data AI Summit? I guess he's here. And if not, what can we learn from that? In what ways do other people who did convert look like Paul if he didn't?
[29:49] And lastly, to finally close off that loop by logging all the various sinks, audiences, and performance metrics, we can now know how to update Paul for the next phase of this loop.
[30:01] All right. So, you tell me, oh, the slides are off, but you may be telling me, why now and why only with Customer Lake? Because marketing has been trying to achieve this one-to-one personalization and next best action since forever.
[30:13] But traditionally, this has really been done via machine learning with models like reinforcement learning, multi-armed bandit, and gradient boost. But I'm going to tell you there's a reason there's historically low adoption, as many people in the team
[30:26] here know, because one, it's hard to build and maintain. You'll need a whole data science team for this, and it's hard to justify the cost and effort if it doesn't work out. Two, there's a cold start problem.
[30:38] What if I don't have enough data to start converging? Three, it takes time to tune. What if you pick the wrong attributes? What if the user you actually have doesn't have any values defined for the attribute you chose?
[30:51] And lastly, it's a black box. How is it making decisions? I don't know. What is it doing right now? I don't know. And so it's clear that traditional ML is not enough. But we think that with LLMs, this type
[31:04] of personalization is actually finally accessible with Genie. No training data? No problem, right? Because what we know is that LLMs are great at making up reasonable assumptions.
[31:17] You can trace through how it's reasoning. It can take your free-form data that may have a long tail of responses, like job roles, and actually go through what makes sense. Genie's also already great at data engineering and machine learning,
[31:31] so that you can imagine it can help train models for you on the side or orchestrate when each model will be best. And guess what? Her accent is not talking in theory. We actually tested the underlying model
[31:43] with the campaign we just talked about. And so what we did was we took all the requirements given to us by our VP. And what we do is actually make these as guardrails in our campaign. So things like our audience was going to be people
[31:56] who abandoned cart, our content was going to be session content, but everything else, like exact send time, the exact sessions to recommend, and who who got it and when, was actually all up to the agent to decide.
[32:11] We essentially wanted a big easy button that said, "Decide for me." Take all my models, all my goals, all my contacts, all my best practices, my templates, and tell me the optimal strategy based on the data for that one person.
[32:23] >> So, you're telling me that you're going to let AI make all the campaign decisions that a whole marketing team would normally make with no guardrails. No marketing team is going to let that happen. Do you remember the 20-page
[32:36] campaign brief? >> Don't worry, Paul. I got you. In this new world, we're actually taking all the issues with traditional methods and making things like auditing and traceability a first-class citizen. As we mentioned, LLMs are great at
[32:49] coming up with reasonable assumptions. And the beauty of the embedded CDP is that by being so close to the data and the sequel, so we can actually show you what data actually led to this decision, so that
[33:01] you can feel confident about how it actually got there. And all the issues we saw fixing with Paul's flow, we're not getting rid of that, right? There's still going to be different teams in the process. And what we did for our campaign was
[33:15] that we actually engaged our content team to tell us how to fine-tune our actual recommendations. You can see here we gave them a chance to give us good notes or actually flag which ones are not great.
[33:27] So, we can retrain and tweak it. You can start to imagine that in this world, with continuous learning and memory, not only do we have a system where humans can talk to agents, but agents can also talk to other agents
[33:39] across these teams. So, that the content team doesn't want any Oxford commas, we'll learn. >> So, back to me, back to my campaign. How do we actually do? Give me the results.
[33:52] >> Yes, we'll get to that. So, how did we do? Drum roll, please. It worked. And the numbers held up. And because we're rigorous scientists here at Databricks, we obviously did an
[34:05] AB test. And what we found was that our campaign actually beat out all other Data + AI Summit campaigns in engagement, having a 60% open rate and a 9% click rate.
[34:17] But as I mentioned, it's not just about click rates, right? Cuz what we truly cared about was that conversion metric. And what we saw was that our campaign had an 8% lift of people that converted.
[34:31] Cool. So, let's do a recap. With our approach, you no longer need to manually tune any models. It's transparent and explainable, so your business users can actually understand what's going on. And there's still guardrails in place and humans in the loop to make sure that
[34:44] it's actually making good decisions. But most importantly, it's easy. Honestly, this campaign wouldn't have happened if we didn't use Customer Lake. And so, it enabled us to actually have this lift.
[34:57] And so, what happens here is that Customer Lake is actually built in mind for that 95% of use cases where honestly custom ML is cost prohibitive and time prohibitive.
[35:11] >> So, this is awesome, Melanie. I want to get the my hands on this for every campaign that we're running at Databricks. But I'll be honest, I don't have half a million dollars in my marketing budget for yet another software license. >> Another great question. Well, I'm happy to announce that Customer Lake is actually in private preview, and we're
[35:24] actively working with design partners to help make this the best product. And regarding the pricing, don't worry. We won't have a traditional software licensing fee. It'll actually just be consumption-based.
[35:39] Cool. So, before we wrap this session up, I also want to call out that Customer Lake is actually more than just this. This session, we actually only focus on campaign agents, but we have a whole other side of the product called profile
[35:51] agents. And that section actually goes deep into that first aha moment that Paul was talking about. Taking your raw data, performing identity resolution to create a great data foundation.
[36:03] So, don't miss out on our next session, building an AI-ready Customer 360 on Databricks at 3:00, and visit our other talks on the CDP track.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.