Introducing: CustomerLake - Profile Agents

Summary

This video introduces CustomerLake Profile Agents, a new capability on the Databricks Data and AI platform that uses AI agents to convert raw source data into clean, unified Customer 360 profiles ready for marketing activation and analytics.
Profile agents handle identity resolution using ML-based scoring with configurable confidence thresholds, enabling organizations to generate unified customer IDs with optional third-party data enrichment from a marketplace.
The video walks through the full pipeline — from customer domain semantics and data normalization through identity resolution and a data steward review workflow — with built-in quality metrics and observability throughout.

Introducing: CustomerLake - Profile Agents

Databricks CustomerLake is how data and business teams can use agents to build a clean Customer360 on Databricks. this video will deep dive on new capabilities that help teams convert raw data into golden c360 profiles that are ready for business use, resolve customer identities, and continously quality-check data for accuracy – all while maintaining the security and governence of Unity Catalog your enterprise depends on. In this video, you’ll learn: - How to use agents convert raw data into golden C360 records, ready for business use - How to use agents to resolve customer identities and generate unified IDs using 3rd party data enrichment - How to train agents to run regular quality checks and improvements on data - How to use a marketplace of 3rd party data for enrichment
Talk By: Patrick Yang, Software Engineer, Databricks ; Phil Catterall, Sr Product Manager, Databricks ;

Chapters

FAQs

What is CustomerLake on Databricks?

CustomerLake is a capability on the Databricks Data and AI platform designed to help data and business teams build clean Customer 360 profiles using AI agents. This video focuses on Profile Agents, which automate the process of converting raw source data into unified, golden customer records ready for marketing activation and analytics.

How does identity resolution work in CustomerLake?

CustomerLake uses ML-based identity scoring with configurable confidence thresholds to match and merge customer records across disparate source systems. This video explains how the system generates unified customer IDs and how data stewards can review and approve low-confidence matches before they are committed to the golden profile.

What problem does CustomerLake solve for enterprises?

Building a golden customer record from scratch can take months or years, and many organizations make multiple failed attempts before succeeding. This video explains how CustomerLake Profile Agents reduce this friction by automating data normalization, identity resolution, and quality checks within the Databricks Data and AI platform.

What third-party data enrichment options does CustomerLake support?

CustomerLake supports enrichment with third-party data through a marketplace, allowing organizations to augment their Customer 360 profiles with external signals. This video describes how third-party data improves identity resolution confidence and fills gaps in first-party customer data, particularly for downstream marketing use cases.

Full transcript

[00:07] Hey everyone. Um Yeah, so uh just quick intros, Phil Caterall, I'm product manager here at Databricks um on the customer lake team. >> Yeah, I'm Patrick. I'm one of the engineers on the customer lake team.
[00:19] >> And uh today we're super excited to give you a sneak peek of customer lake profile agents uh tooling in Databricks to help you build customer 360, apply identity resolution, and get your data uh in a
[00:32] ready state for marketing, analytics, whatever else you might want to use it for. Um before we begin, quick show of hands, who knows what a CDP is?
[00:44] Who's built a CDP or a customer 360 on Databricks? Awesome. Still a lot of people. Um so uh yeah, so um
[00:56] you know, I want to start with some of the rationale for, you know, why Databricks is building kind of up the stack in the customer data space in particular. Um Ali gave a bit of an overview in the keynote uh this morning, and we have a deep dive on it in
[01:08] tomorrow's keynote, so be sure to uh stick around for that. Um but it all starts with kind of this basic idea, pretty uh non-controversial, the best companies differentiate on customer experience. In order to build a good
[01:21] customer experience, you need to have context about that customer, right? Who are they? What are they trying to achieve uh at this time, like as they're interacting with your brand? Um in other
[01:33] words, what data points do you know about these customers? Um and so when we talk to customers uh of Databricks, we hear this is still way harder than it should be, right? Going from a bunch of data sitting in source
[01:46] systems all the way through to some kind of marketing act activation or customer experience, uh there's too much friction in that process. And really we hear two main problems uh that come up. First
[02:00] is more on kind of the data engineering side, that process of building that golden customer record can take months if not years. And many organizations go through multiple attempts to build this. This is kind of a
[02:12] ever-ongoing project with a lot of our customers. And then two, even if you have that data kind of unified in a kind of golden customer view, taking action on that data is still harder than it should be.
[02:24] You have a lot of you know, channel-specific marketing applications in your stack. Um and none of them have kind of that full view of the customer and that full business context of your business to know
[02:37] how to message them in the right way. And this becomes even more pronounced in 2026 as these experiences are becoming more and more powered by agents. They need to have that full context in order to make the right decisions on
[02:50] your company's behalf. So, you know, when we were you know, designing customer lake um we thought you know, there are three main things we want to address as part of this. Three principles that we think modern
[03:04] CDP needs to have. One, it needs to be embedded within the lakehouse. Your data doesn't need to move to an external system, a subset of it. You don't need to pre-aggregate it. And it comes with all the security and governance
[03:17] guarantees that you have in Unity Catalog today. Second is that these tools need to be designed for the agentic era. Where agents are first-class citizens
[03:29] working alongside your users, your marketing teams. Where the humans provide the strategy, but agents to do most of the rest of that work. And then finally, you know, the holy grail of martech for the last 20 years
[03:42] is how do you actually get to truly one-to-one personalized experiences? Away from, you know, static campaign lists Uh, and humans in the loop having to iterate on that, experiment, you
[03:54] know, with months-long cycles uh, to get new campaigns out the door. Uh, we think, you know, as Databricks, being closest to your data and with all of our AI capabilities as a platform, we
[04:06] can get a lot closer to that uh, than previous attempts. And so, that's why we introduced Customer Lake uh, earlier this morning. So, Customer Lake uh, we're framing it as an agentic CDP that's really embedded
[04:18] directly within your Databricks warehouse. Uh, addressing all of those challenges that I outlined in the last slide, right? It's embedded. You're using agents to democratize uh, data to your less technical teams, your
[04:30] marketing teams. And it's autonomous. It can personalize those experiences on a one-on-one basis using all of that context that you have about your customers.
[04:43] So earlier, I outlined two main challenges. So, Customer Lake is a suite of products, two products, to address each of those challenges. Profile Agents is what helps automate that customer 360 work, a lot of that toil that goes into unifying that data.
[04:56] And then Campaign Agents offers the business-first interface for marketing teams and marketing ops users to get direct governed access to that data and take action on it.
[05:09] And so today in in this session, we're going to be focusing on Profile Agents um, you know, the those capabilities to build that customer 360. And you know, we think Databricks is already the best platform to build these
[05:21] pipelines, right? Uh, we have capabilities across the board with Lake Flow, Lake House, now Lake Base to be able to build these pipelines. And you know, generally our customers agree. Customer 360 is actually the number one most common
[05:34] workload across all all Databricks customers. But the problem today is the batteries aren't included. You have to stitch all of this together yourself using a bunch of capabilities that we have um,
[05:47] which ends up, you know, making it harder than it should be to build and maintain these things. Uh, right? It's expensive and tedious to build. Uh, especially if you're doing more advanced techniques around identity resolution, your teams might not have
[05:59] the expertise to build these models and train them. And it's never done. Your requirements are always changing, your data sources are always changing, and you need a system that can adapt with you,
[06:11] uh, to adjust to that over time. And so, really, that's what we're looking to address with profile agents. Um, again, a lot of the same principles to customer lake more broadly apply to
[06:23] customer, uh, profile agents. Uh, first and foremost, it's lakehouse native. There's no need to move your data outside the platform to apply things like, uh, identity resolution.
[06:35] You know, if all of your other ETL is happening on Databricks, why should customer data, uh, customer data mastering be any different? Uh, as part of that, you know, everything is a governed asset in Unity Catalog. Both
[06:48] your customer data assets and the outputs of, uh, profile agents, but also all the metadata, right? Why did these customer profiles get matched together in my identity resolution? Um, all of that is fully available as system
[07:01] tables, so you have full audibility of what's happening. It's really a, uh, we're focused on making a glass box solution. Um, second, it's powered by Genie. Um, so, our customers have told us they love
[07:13] Genie code. The majority of data pipelines in Databricks today are authored using Genie code, uh, and customer data should be no different. Um, so, Genie's really going to be a first-class citizen with special context around customer data to help you
[07:26] build these pipelines. And then finally, it's composable. Um, like I said, a lot of customers are already running these workloads on Databricks, and we don't want customer lake to be all or nothing. There should be no migration from what you're doing
[07:39] today to use these features. Uh, so, we're building in a way where uh if there are features that Profile Agents offers that complement what you do today, it'll be super easy to plug that into your existing pipelines.
[07:55] So what does this actually look like? What you know, that's a lot of kind of the theory and the strategy. What are the features that Profile Agents offers? Um so if you follow it kind of through the pipeline, you know, traditional ETL path, it starts with uh
[08:07] ingesting the data, right? So uh we have existing managed connectors through our Lake Flow Connect platform. Uh those will be available as kind of first-class citizens within uh the Profile Agents interface
[08:19] uh to load data into the platform. And again, it's composable. If your data's already in Databricks, that's great. You can just point it directly at any table that already exists in Unity Catalog. Uh second,
[08:31] you need to apply some kind of uh customer domain semantics to that data. So if I think if you think about identity resolution, what are the different attributes that I might want to match across? We call that semantic tagging.
[08:43] And with Genie, we can help apply that automatically for you, do quality checks on that data to find things like bad values, uh you know, everyone has had to filter out test@test.com email addresses, uh
[08:57] things like that. And apply normalization to uh improve that match rate. Uh and then finally, um we are working with a number of third-party identity providers uh to be
[09:11] able to integrate third-party data super seamlessly into your customer 360 pipeline uh for things like data hygiene, right? So if you want to use an external source to validate your postal address data, email deliverability,
[09:24] uh you'll be able to integrate that directly uh through Profile Agents. So next, the real core of the customer 360 pipeline is identity resolution.
[09:38] Um so, this is what we're going to go into a lot more detail uh a little bit later in the presentation, so I'll kind of talk quickly through this. Um but in general, you want to make sure that you have a deduplicated record of customers as unified across all these
[09:52] data sources. So, uh uh Profile Agents provides an out-of-the-box uh first-party what we're calling agentic identity resolution engine. So, you can do that identity resolution directly in Databricks.
[10:05] Um that's powered by an agentic learning loop to not only help you bootstrap what are those rules that you need to apply to do the identity resolution, uh but to help improve those match rates over time as your data changes and as
[10:17] your sources change. And finally, again, um the majority of a lot of our customers uh use third-party data to kind of enrich and supplement their first-party data. So, we'll have native clean room
[10:30] integrations with a number of third-party data providers out of the box and make it super easy to uh supplement your first-party data when necessary.
[10:43] All right. So, now I have uh you know, clean, standardized, resolved customer profiles. Um how what do you actually do with that? The output of uh Profile Agents is what we're calling customer 360 data products.
[10:55] So, this is tooling that allows you to map uh the schemas across all of those sources into that golden record, apply survivorship and aggregation rules. So, if I have uh four different addresses on a customer across sources,
[11:08] which is the quote best address to use for them? You can do that out of the box. Um and then finally, uh you know, even if you have a uh canonical customer 360, you often need to apply more transformations to
[11:21] that data for specific use cases. So, you might have a different view of that customer data for analytics compared to what you use for uh marketing activation. Uh Profile Agents makes it super easy to
[11:33] apply those transformations and publish them through a number of different consumption modes, uh, like Delta sharing, reverse ETL, um, metric views, all those capabilities
[11:45] that Databricks, uh, provides out of the box. So, this is kind of the end-to-end view of capabilities that I just walked through. Again, really looking to support the whole end-to-end pipeline to help you go from raw data to business
[11:58] data ready data products faster more easily than you can today. Really the way to think about it is, uh, it's a more ergonomic abstraction on top of a lot of what you can do in Databricks today without having to write
[12:11] all the SQL pipelines or notebooks yourself. And so, to illustrate this, let's walk through a quick example. Um, so again, walking through that process, first step is to ingest data.
[12:25] Uh, so you can see here I have a customer, his name is John, uh, and I have his record across a number of sources. And as a human, I can spot, "Hey, this is all probably the same person." Uh, I have a John Smith, a Jonathan Smith, a J. Smith,
[12:39] uh, but they're all coming from different systems and they're the key point is there's no common key to join these customers together. Uh, you can see slight variations in email addresses, business email versus personal, uh, addresses, they might have moved
[12:51] over time, etc. So, the first step that profile agents helps with is normalizing that data. So, you can see here, uh, if you have a single, you know, name string field, we actually break that apart using
[13:03] probabilistic algorithms into first, middle, surname, uh, basically things you want to do to help improve how you might match this record to others. Same thing with email standardization,
[13:15] phone number, address, uh, all out of the box in, uh, within profile agents so that you don't have to write, you know, the SQL logic or the UDFs to do this yourself.
[13:29] Next, uh, you need to resolve identities. So, you need to tie these records together and assign them a single ID. Uh you can see here, you know, we now have a resolved ID that applies across all these sources.
[13:42] And then finally, you need to create that golden profile. So, what is the best uh attribute to use across all these sources uh for each of those attributes. So, uh you might have different rules you want to apply for that. You might want, you know, the most recently updated address or uh the most
[13:56] frequently contacted phone number. Uh profile agents lets you apply those rules without having, again, to resort uh to writing manual SQL to do it. Now, you might have noticed I was a bit
[14:08] hand-wavy with step three here. How do you actually produce that resolved ID? Uh next we'll dive into a little bit more detail uh on identity resolution itself.
[14:24] So, just to level set to start, you know, identity resolution means a lot of different things different people. Uh there's many flavors of identity resolution. Uh you might be deduplicating a single messy set of data. Uh you might be looking at digital device identifiers
[14:37] from your clickstream data. Uh you might be working with third-party data providers. Um but for uh the rest of this session, we'll define identity resolution as when you want to link uh unique customer records across your
[14:50] own first-party data in cases where they don't share the same identifier. Uh so, you have different data sets, no common ID across them. We're putting a lot of effort into this aspect in particular because we think
[15:02] this is the hardest part to do on Databricks in kind of a do-it-yourself mode today. Why is that? First and foremost, like I said, no common keys across sources. You can't
[15:14] just do a left join across them because you have to look across multiple attributes and make a decision around whether this is the same uh person or organization. And that's one thing to call out here.
[15:27] You know, I've been talking a lot about customers in the context of individuals, but depending on your on your business, a customer is different things. It's organizations, it's entities, it's individuals. Profile agents will support all of that.
[15:41] Second, real world data is messy, right? People have nicknames. You know, I'm Phil, but I'm also Philip. My first name is actually Edward. That shows up as three different people in a lot of brands that I work with.
[15:54] Um And you know, if if you design kind of a rules engine for clean data, they're going to fail once you actually get running it on messy data.
[16:08] And third, in a lot of cases it's pretty straightforward. You can tell, you know, on exact matches or somewhat fuzzy matches if this is the same person or not. Um but it's the hard cases where you want to add more judgment. And today it
[16:20] requires a lot of human judgment. So, if you're using, you know, more traditional MDM solutions, you might have a data steward team who's reviewing individual pairs and assessing, you know, based on looking at it, is this the same person or not?
[16:33] Um depending on your business, you know, you might have different risk tolerances in terms of what you want to call a match versus a non-match. >>
[16:45] approach this problem? Uh the established way to do it is called entity resolution. Uh just a quick overview on what entity resolution is. Um it usually has three main steps in the process. So, you start with grouping together or
[16:58] blocking records to say, "What are potential matches?" Because if you compare every single customer to every other customer, that's an n-squared problem. Uh you know, the the volume of data you need to process explodes to unreasonable
[17:11] amounts. So, you want to start with, you know, what's a likely pair to begin with. And
[17:23] records together and assess those. Second, you need to actually compare the various attributes on those individual records to say make a classification, are these the same person or are they not?
[17:36] And then finally, you need to cluster together those records, merge them together. Basically, you have your records as the nodes and these matches are non-matches as the edges, you need to create that identity graph based off of that.
[17:49] So again, this is a common approach, um you know, this is not even specific to customer data, you know, entity resolution spans uh far beyond that.
[18:01] So, if you double click on that center box, uh how do you actually do the matching there? There's a number of different approaches, uh you know, you can use SQL, you can do kind of joins or substring joins, build that logic in SQL.
[18:13] Um a lot of existing identity resolution systems or MDM platforms use rule lists, so basically if-then logic. If it matches on email address and there's at least a fuzzy match on name, then let's
[18:27] call this a match. And there's also more classical ML approaches, both supervised where you need to provide labeled data up front or unsupervised um to basically output a confidence score
[18:40] that says how likely these people are to be the same or not. Um and there's even, you know, more new approaches today with things like vector search or using LLMs to try to do this.
[18:53] But, you know, we think all of these approaches today have some kind of trade-off. Uh typically across one of these dimensions. Uh so first, ease of use. Um how easy is it to actually get
[19:05] started with this, uh to go from zero to one to having a resolved profile? Interpretability. Can you go to your business teams and say, "Hey, why these records matched or did they not match?" Again, with some of
[19:18] the more uh machine learning-based approaches, all you get is a score. And, you know, it says, "Yeah, I'm 82% confident." But, you can't really explain why. Uh performance. Again, with a lot of these new approaches, things like uh
[19:30] vector search and LLMs, uh at least in 2026, inference isn't cheap and fast enough uh to do that across all of your data. And then finally, quality, right? Uh you can have it be performant if you
[19:43] sacrifice quality. Uh but, ideally, you want the best of both worlds. So, as part of Profile Agents, uh we're introducing uh a new framework for identity resolution that we're calling
[19:55] agentic identity resolution. Uh and really, the goal here is to optimize across all of those aspects that I just outlined. So, how does agentic identity resolution work?
[20:07] Uh so, it starts with a rules-based engine. Rules are explainable. They're easy to reason about. And, importantly, they're also easy for agents to reason about. So, we start with a rules-based approach that takes a mix of deterministic
[20:19] matching as well as probabilistic or fuzzy matching. Uh and we think that covers you know, in our testing, that covers the vast majority of uh candidates you need to compare.
[20:31] But, it doesn't cover everything. So, in this case, uh agentic identity resolution has a fallback to an LLM to actually uh assess an individual pair and make a determination, is this the same person
[20:44] or not? Again, super helpful in those edge cases where there's some nuance required, some understanding of your business context, and what a customer means to you. Uh where we think, you know, an LLM really
[20:57] helps in that aspect. And then finally, uh human review. So, for a lot of our customers, especially, you know, if you think in regulated industries, there's still times where you want humans to review these things
[21:09] because uh you have kind of a very high threshold uh or risk tolerance for what's a matter you know, matching different people together essentially. Um so, we include a workflow out of the
[21:22] box uh for that kind of data steward review process as well. And so, uh that's that's kind of the what we call like the inference path or the run time
[21:35] process that identity resolution as new records are coming in on an incremental basis. And what the key to this process really is is that green box uh on the right, uh the agentic learning loop.
[21:47] So, one of the big problems with rules-based engines today for entity resolution, how do you come up with those rules in the first place? That's what the agentic learning loop is meant uh to address. You can think of it similar to the training process for a
[22:00] classical ML model where uh but instead of having to label the data yourself, we actually use an LLM to label a subset of your data and then use Genie to actually fit a
[22:12] rule set to that data to give you a starting point. So, you're not starting from a blank page, you're starting with a set of rules that fits to your data based on context that you provide to Genie about what a match means for your business.
[22:29] Oop. Wrong button. And that's not just a static process. You don't just do that once when you're setting this up for the first time. Um agentic learning helps your customer 360
[22:42] evolve over time as well. Um so, you know, the old way uh you know, when I was a practitioner and when I was helping customers implement identity resolution, uh it usually looked like this, right?
[22:54] You profile and cleanse the data, that's a manual process that your engineers are doing. Uh you might label a a uh of the data and train and configure a model to match over it. And then you put it into production and
[23:06] it works. But you never look at it again afterwards, right? There's a high cost to iterating on this over time. And the cost is at the threshold where for a lot of times it's just not worth it to to go back and do that.
[23:18] So, with the Agentic learning loop, that's what we're really looking to address. You constantly have new data and new sources being added to this customer 360. And again, we always start with that rules list, which is super scalable and can
[23:31] resolve profiles uh at scale quickly. But and then we fall back to again to that LLM and or human review for the edge cases. But then Genie can take that and feed
[23:43] that back into the loop to suggest you new rules, new quality filters, even look at individual profiles where think if you had, you know, uh an army of 100 analysts to go profile individual records, what you could do there? That's
[23:56] what Genie's going to be doing under the hood for you as this pipeline is running in production. And again, it's always human in the loop. You make the final determination if you want to manually break up a cluster that was over merged or if you
[24:09] want to introduce a new rule. Um ultimately, you make that decision yourself. And so, you know, we really think Agentic identity resolution fills a major gap versus traditional approaches
[24:22] today um across those aspects of performance, interpretability, match quality, and ease of use. Uh and now to make this a little bit more real, I'll turn it over to Patrick to give a quick demo
[24:35] uh of the capabilities that we're working with customers to build right now. >> Awesome. Thanks, Phil. All right, let me switch to this screen. All right, cool. So, I'm I'm going to give you a quick demo, a quick preview, if you will, of what we've been working
[24:47] on so far. So, as a reminder, this is all kind of like um you know, preview state. This is just kind of giving a vision of what we've gotten so far uh building out profile agents. Okay, so the motivating example here that I'm going to um to use is
[24:59] essentially imagine you are building out a customer 360 for a certain global, you know, a sporting event and you're trying to integrate a new source into your customer 360 around ticketing sales,
[25:11] right? You work with a vendor, they're getting you they're giving you some data around customers, ticketing data. You want to integrate it back into your customer 360. So we're going to go through a quick slice of what that process might look like on profile
[25:23] agents. All right, so I'm going to go over to this fan 360 project over here. And just as just as a reminder, you know, like as Phil was mentioning the IDR the sorry the uh
[25:36] >>
[25:48] preparing it for the the matching process that we're going to do later, right? Second, we're going to go through identity resolution, which is essentially again doing that process of blocking and then matching and then clustering to essentially build
[26:00] out, you know, this deduplication as you can imagine for these sources and build out like uh essentially individual entities from your data. And then lastly, you know, we're going to take those clusters and produce
[26:13] entities out of them using survivorship and I'll walk through each of those different steps today. So on data prep, so as you can imagine we have three data sources already here. So as a quick reminder, I'm going to
[26:25] kind of plug this throughout the demo. Everything that we're building here today is built on top of the existing Databricks platform, right? So all of these sources that you see today are represented as tables within Unity Catalog. That's one of the amazing
[26:37] things about our design philosophy on profile agents is that we do want it to be built on top of the foundation of the Databricks Lakehouse as is so that you have the capabilities and you have
[26:49] the trust in the capabilities uh, we're building out. So. All right, So, let's go ahead and add new source here. Um, so again, we're pulling from tables that we're already seeing from Unity Catalog. Today, I know I want to work with this ticketing resale customers,
[27:02] uh, data set. Let me go add that. Okay. So, once we add it, um, you'll see that Genie's already, uh, hard at uh, hard at work away. You know, it's analyzing our data set. Remember, this is, um, you know, data that you see in a table in Unity Catalog. So, all of the
[27:16] the amazing capabilities of Unity Catalog are already applied there. So, you know, if you have comments on your columns, Genie's going to use that for semantic understanding, uh, of what your table contains, right? So, as you can see, um, Genie's analyzed our table, um,
[27:28] using all of that, um, you know, underlying UC knowledge, um, to say, "Okay, I've identified a couple of different semantic types, um, that I'm going to identify." And it's already uh, tagged them specifically, right? Um, so, as a quick reminder, these semantic
[27:40] types are basically our way of saying, you know, this column represents an email, right? So, if I'm going to do matching between this source and another source, I know, "Okay, this email might want to be a match with other emails." And that's how that's the source of
[27:52] doing this sort of fuzzy join, as we describe it, right? Um, awesome. So, uh, Genie's done a great job. It's identified a couple of columns for me. Um, a couple cool things that you can see here. Um, when we select a column here, we can get a quick look of
[28:05] a data profile for one of these columns, right? Um, so, that's a really critical step in getting the semantic type, as well as, you know, determining what we uh, we call the quality filters, right? So, you can see from this full name, um, you know, you can give you some
[28:17] statistics around like what's what's contained in the uh, in the column, and also a quick preview of some of the names, uh, that you can might see. So, for example, as as Phil mentioned, you know, um, during the data prep step, you want to get rid of all of these like
[28:29] kind of funny or wonky like, you know, fake data or essential essential pieces of data. Like, for example, you know, guest guest. We don't really want that in the process, cuz it's going to produce bad matching results, right? Um, so, that's why Genie has applied actually a quality filter for us to get
[28:42] rid of data like this, um, from from the data set. So, it's essentially resolved it to null. Um, otherwise, you can also see that uh, you know uh, the the data prep step is performing this like uh
[28:54] breakdown of the name, right? So, we can see this name getting broken down into the given name, you know, surname, suffix. This is just our way of ensuring that like the data is in the right format for a matching process going to occur.
[29:06] Right. So, you know, even though Genie's tags look great, maybe I want to add my own tags. So, I'm going to add here hey, this like marketplace customer ID. It looks like a unique ID, so I'm going to add this ID
[29:18] here. Yep. Also, so before I get off this page, I also like to highlight that we give you a quick preview of the data set before it's you know, from from raw as well as after normalization, right? So,
[29:30] you can see kind of this process at play where we're breaking down some of these types. For example, billing address we're breaking down into all these different subcomponents, which help us ensure that the the matching is performed accurately. All right. So, this all looks great. I'm
[29:43] going to mark this uh source as ready to be used in IDR. And then I'm going to switch over to the matching rules. So, as a reminder, you know, after we perform data prep, you know, we're going to basically
[29:56] get these sources ready for matching, right? So, remember we're doing matching across semantic types from two different sources. You know, we're taking all those types that we tagged in the previous step and using this as possible fields for to to build our rules on top of, right?
[30:09] So, just a quick preview of kind of like the the uh identity resolution resolution rule set. We define this as a rule list. It acts in practice very much so like a waterfall essentially. So, you define these rules in order and then that we we
[30:23] kind of go through them and try to evaluate each of them, right? So, as you can see here, I just want to quickly highlight that Genie has actually added a new rule for us based on the new source we added, right? So, Genie has analyzed essentially, okay, once we've added this
[30:36] source this new source, we might want to modify the rules a bit to affect um the matching process based on the new data that we just received. So, let me just accept that for now. Uh and then otherwise, you know, I'll give you a quick preview of like kind of
[30:49] what the rules look like, right? Um so, obviously, you have some fields that uh you know, are very clear as like a deterministic match. If the IDs match, uh if the unique IDs match, then let's resolve this as a match. That one makes a lot of sense. Um but then you have some interesting stuff here around a
[31:02] fuzzy matching, right? So, fuzzy matching is essentially where we'll take the data uh as Phil described, you know, even after it's cleaned, the the the data might not match up between different sources. Like a customer might use different emails or different phone numbers or they they might represent their name differently, you know, but
[31:14] they might use a nickname in a certain data source and a real uh full name, you know, uh in another source, right? Um so, a lot of the the work that we're doing right now is to essentially like build out um feature engineering here,
[31:26] if you will, to essentially uh enable you to do these fuzzy matching. Uh and we're we're managing this library on our own, and you know, that's where we're doing a lot of the R&D on um to make this process really robust and and functional. So.
[31:38] All right. So, and and other things you call out here, you know, at the end of the the the rule waterfall, you know, will allow you to essentially uh let Genie take a look. So, if um all the rules fall through and we haven't made uh you know, a a match from any of the
[31:51] rules, um basically, we'll let Genie take a look at it, take a look at the data sets, um and use its intelligence to determine whether or not it's a match or or or a non-match. Um and then as you can imagine, you know, this data and these evaluation results are used in a
[32:03] loop as as Phil mentioned, um to further inform uh rules as you go on, right? You know, we know that the nature of data is always a dynamic, right? Your the the the nature of the data that's coming in might change over time. Um you know, so,
[32:16] what you define here today might not apply later down the line if your data uh profiling uh does not match anymore, right? Um so, that's really where we think a lot of the uh capabilities uh are going to be proven here around where
[32:28] we have Genie um taking a look continuously as you do run over run of ID resolution, um what it should evolve the the quality rule uh sorry, what does she evolve the the matching rules for um over time essentially?
[32:42] >> One key thing to call out here, too, is again, we're trying to build this as lakehouse native as possible. So, every time this runs and every time this evaluates, all this gets logged as a system table that you have full access to to understand why did why did it make the
[32:55] determination either through the rules or for the genie review kind of a natural language description of why it either matched or uh not matched that pair. So, again, trying to make it as
[33:07] transparent and glass box as possible. >> So, if I switch over to the insights tab, you get a quick preview of some some of the uh you know, the data that we're logging essentially, right? So, on the previous run, um you know, we generated 4.2 million unified profiles,
[33:20] maybe 14 of them needed review, uh and then we actually you know, we have the actual have the capability of like conflict rules as well, right? So, um maybe we we saw some conflicts in those um and uh you know, that that will affect how the clustering performs
[33:32] essentially, right? Uh we'll also give you insights on um you know, essentially how your rules are being applied, you know, what percentage of uh profile pairs are being uh matched at a certain every certain rule layer. Um so, these
[33:45] are these are all helpful statistics for you to kind of audit how your um your essentially your rule model is performing, but also, you know, they're they're also great feedback for genie itself, right? Um so, it's a very central to our um you know, design
[33:57] philosophy around metric is to make sure that you know, we provide you observability, um but we also provide the agent observability um for when it needs to do uh perform that feedback loop effectively. Okay, for now, I'm going to maybe do a quick dry run with that my changed
[34:10] rules. Um uh you know, dry run we essentially say is like a kind of like a smaller run all of the uh the matching process. Um okay, awesome. We saw that like maybe a couple of the the um the profiles um got uh
[34:23] reduced in uh the review queue. Um we got some more profiles, which makes sense given that we have a new data set. You know, not every um piece of our uh row of data from um each data set is going to be matched to an existing cluster. Maybe some of them generate new
[34:35] clusters, right? All right. So, um this looks great to me. I'm going to launch a run. Um before I show off kind of the the run observability, um I I want to complete that process uh for you about
[34:48] um you know, the uh customer 360 pipeline essentially, right? So, we've done the the blocking, the matching, the clustering. We have all of our records kind of clustered together from different sources, right? Uh uh into uh uh Now, we basically need to
[35:01] evaluate what are the golden records that are going to come out of these these clusters, right? So, the last step is going to be evaluating into that that golden uh profile uh record uh essentially. So, um
[35:13] this is a process we generally call survivorship. Um if you uh if you can think you have like a duplicate data from multiple um different sources, um maybe have like multiple emails from two different sources, and they don't necessarily match up. How do I pick
[35:25] which one essentially wins, right? Which one is evaluated as the the canonical email in my uh golden table output? Um So, uh we we basically provide you with the ability to define these survivorship rules, right? So, if I give you a quick
[35:38] preview, you can take a look, you know, we have some very rudimentary rules that basically allow you to define do I want to take the most recent one? Do I want to take the most frequent uh like frequently observed record? Um do I want to take the oldest one? Do I want to just have all of them as an array uh uh
[35:52] of the the values essentially? Um and we allow you to define this for for all the different uh columns that you're going to see um in your data set. So, you can produce that one singular golden uh profile record at the end, right?
[36:04] And then we also give you a quick preview of this so that you can kind of take a look at what your um end like golden profile table is going to look like essentially. And and as Phil mentioned, remember this is all getting written out to Unity
[36:16] Catalog as well. So, it's going to be just a normal otherwise normal table. You can use that um for whatever you need to uh use downstream. Um whether that's, you know, campaign agents that you might have seen in another session uh or this you know for your reporting
[36:28] in your AI BI dash dashboards or if you want to output this to export this to another tool you know fully free to do so as well. Last thing I'll go over really quickly is essentially runs. So you know energy
[36:41] resolution as as Phil mentioned is a continuous process right it's something that you're going to be running over and over again as you get new data as you get new sources essentially. So we want to provide you with the observability out of the box to basically take a look at you know how how is my you know
[36:54] energy resolution process performing over time did something happen to the nature of the data that affected it you know do I need to make changes based on this so as Phil mentioned this is all going to be output as system table
[37:06] observability data. You know on the lake house so that you can analyze it and take actions as you see fit. Right? That's all I had today. Switch back to the slides.
[37:21] Awesome. >> Yeah so that's a quick overview of profile agents. So we're working with a handful of design partners today to build this out really stress test especially that ID resolution process make sure it works at scale make sure it works on
[37:33] messy data and it will be available in private preview in Q3 or I mean for September October time frame. So yeah Patrick and I are going to be available after the session either up here on stage or if we
[37:46] get kicked out in the hallway so feel free to come by ask questions and uh Thanks for listening.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.