How to Build Trusted CRM Enrichment with Delta Sharing and AI Agents

Summary

Databricks' own GTM team built a production-grade CRM data validation system on the Databricks Data and AI platform using Delta Sharing feeds from multiple enrichment vendors to create a triangulation framework that identifies the most trusted account data.
An AI governance layer reasons across competing enrichment signals to propose account re-parenting decisions, then a secondary validation agent checks those proposals against confidence thresholds to eliminate hallucinations before any changes are applied.
The system automates approvals for high-confidence changes and routes edge cases for human review, with agent reasoning traces explaining why each change was proposed and an account deduplication workflow backed by an opt-out approval process.

How to Build Trusted CRM Enrichment with Delta Sharing and AI Agents

Enterprise execution relies on trusted data, yet enrichment vendors often conflict and hierarchies drift. Manual validation is slow, and blind automation poses risks. this video details a production-grade validation system built on Databricks to transform CRM chaos into governed data layer. Using Delta Sharing feeds from various providers, we built a triangulation framework to reconcile signals across parent hierarchies, Salesforce structures, and revenue estimates. Rather than accepting data blindly, we implemented an AI governance layer that: 1. Reasons across enrichment signals to propose re-parenting. 2. Validates outputs via secondary agents to eliminate hallucinations. 3. Automates approvals based on confidence thresholds or escalates for human review. Attendees will learn to design multi-agent architectures and AI guardrails that preserve trust while accelerating planning cycles—enabling autonomous, governed workflows that improve revenue visibility at scale.
Talk By: Petar Zoric, GTM Senior Analytics Engineer , Databricks ; Yu Chen, Manager, GTM App Engineering, Databricks ;

Chapters

FAQs

What CRM data challenges motivated building this system on Databricks?

The GTM team had no single source of truth because multiple third-party data vendors provided conflicting account hierarchies, revenue estimates, and Salesforce structures. Manual validation could not scale across thousands of records from different providers, and inaccurate territory assignments directly impacted sales planning and revenue forecasting.

How does the triangulation framework choose the right enrichment data?

The framework compares enrichment signals from multiple Delta Sharing providers using confidence scoring and semantic similarity to determine which source is most trustworthy for each account attribute. Rather than accepting any single vendor's data, it reconciles parent hierarchy signals and revenue estimates across providers before proposing a canonical value.

Why is a secondary validation agent needed in this multi-agent architecture?

The primary agent proposes re-parenting changes based on enrichment signal reasoning, but a secondary validation agent independently checks those proposals to eliminate hallucinations and ensure outputs meet quality thresholds before being written to production. This two-agent validation pattern is presented as essential for governed, autonomous workflows.

How are agent decisions made transparent and auditable in this system?

Every proposed change is accompanied by an agent reasoning trace that explains why the re-parenting or deduplication action was suggested, referencing the specific enrichment signals that drove the decision. This trace is surfaced in the Databricks Apps-based orchestration UI so that human reviewers can understand and override any automated decision.

Full transcript

[00:09] Hello everyone, welcome to our session. I know it's the last day of the summit, so super appreciate your time. So before we start it, uh let's do a quick show of hands. Have you ever um need to make a decision or do some analysis based on
[00:22] your serum data? Please raise up your hand. Cool. most uh awesome and please keep your hands up if you find like later on the data was
[00:35] wrong. Nice, nice, nice. Uh I think uh welcome to the right session. So here today we're going to solve this problem. Uh I'm together with Peter. We are part
[00:50] of the go to marketing uh data governance and solution team here at data bricks and we're super excited to share our team story. We're going to walk you through how we turned the messy
[01:04] CRM data into a governant self-cleing systems to bring the trust data to our end user. And today we will cover the challenges, the problems uh and also the
[01:17] platform we've we've we've been designed it and also all the technical detailed. Okay, moving forward for the challenges it should be sounds super familiar for for the ones you just raised the hand
[01:29] but before we go even deeper uh I want to provide some context. So for someone who may not familiar with uh sales planning. So what does sales planning do in data bricks? At beginning of every
[01:43] physical year uh we are using sales planning to decide what to sell and how to sell. It is very important. Uh we need to decide how we need to divide it up divide it up our territories set up
[01:56] set up our goals and also assign the accounts to our reps. Um this this setup is super crucial for the entire company because it is basically our uh sales
[02:08] road map for the for the physical year and it connects our sales reps daily activities to the overall uh company's goals. Uh if we get it right we will max
[02:22] we will maximize our company's revenue but if not we cannot trust the data. on the other hand everything the the our road map will break up and uh we we might ended up losing it. So what are
[02:36] the challenges we've been facing? First uh no single source of truth um because of the quality of our internal serum data. We do buy in a lot of like
[02:49] uh third-party data vendors but uh it's just super hard for us to tell which uh data vendor is the is the right source of truth and also another challenge is
[03:02] uh enrichment doesn't scale because with thousand of records across different data providers um it's hard for us to manually check validate and even enrich our internal data set. just too slow and
[03:15] cannot scale. And also another challenges we've been seeing is like and governance changes because if we just blindly dump all the the vendor data into our CRM data, the the bad data may
[03:28] overwrite the right data and no one knows why. And lastly uh on the user side who rely on the systems uh the challenges is the worst part because uh the bad data may leads to the bad
[03:41] planning and we may ended up like assigning wrong account to the wrong reps. So all the uh head count and capacities are messed up.
[03:53] And continuing on the user side, uh we do ask our sales and planning team to describe their biggest pain point. What exactly went wrong and they summed up
[04:06] really well. So the problem is that the data scattered everywhere they cannot find it, they cannot trust it and uh they cannot fix it. So that's a problem.
[04:18] But here we have the solutions. Um this is where exactly the data bricks uh came in and today all our sales and planning teams are running their daily jobs on
[04:30] one single platform u where they we give their trusty data for them so they can finally use the data and plan with confident. Um so let's take a look at
[04:42] the high level arch uh the infrastructure of our system. uh we can look from right from left to right. First about the ingestion uh instead of
[04:54] like jumping to dozens of different system different cloud we're leveraging the delta share uh to give us the data immediately uh give us like a simple unified
[05:07] ingestion. uh we've been collaborating with different data vendors for example uh zoom info crunch base sumble and uh and by leveraging the data share I think
[05:20] this is one of the biggest uh secrets to make the whole system fast and scalable from beginning because we can just read the data data immediately from their live data it's it's always up to date
[05:33] it's always fresh and and also like we we've We can also ingest our Salesforce data to data bricks. And moving to the middle part, once all the data landed in data bricks,
[05:46] uh we start to clean it up and turn it into production ready a business ready data set. And on top of that, we're adding AI layer and also vector search um to automatically generate insights
[06:00] and uh help us to make decisions. And behind the scenes, everything is governant on data bricks. It runs on the serverless uh compute and governed in Unity catalog. Uh with all of those
[06:13] setups uh we've leveraging the setups to build our own data quality orchestration agent and also we feed all the insights we made um inside of data bricks to fit
[06:25] into into our agent. the agent will run daily um um make recommendations and take actions and it will automatically push claim data back to Salesforce.
[06:38] So that's the high level architecture about our system. So what are the advantage of running this solutions on data bricks? Uh we see like first of all
[06:50] it's a single platform. People don't need to jump into different system to find the data to clean their data. It's all in one place. Second is high trust because inside of our system, we're
[07:02] leveraging the multi uh AI uh uh agent and also human in the loop to validate data. So we know the data is right before we use it. And thirdly is scalability. Right now by leveraging the
[07:15] data bricks we can just process millions and all the better pies data um almost at the same time. and and lastly it's it's secure because all the data is
[07:28] saved uh safely and locked down in data brick. So that's the high level idea about how we did on the systems. Now I think it's time to show you the technical details about how we actually
[07:42] clean up the CRM data CRM data and turn it into trusty data to our user. So hand over to Peter.
[07:57] Thanks you. So I'll be walking you guys throughout the whole infrastructure, the architecture and basically the technicalities how we build the whole agent. So where should we start? Basically intelligent data quality for everyone starts with a good delta
[08:11] sharing and therefore a aentic enrichment. So at first what is delta sharing? Let me keep it simple and tell you a little story. So what happens basically every night? Every night our
[08:23] data providers surface their tables and we can read from those tables. They make it accessible via delta sharing service and we read throughout those tables, get them into the Unity catalog and
[08:35] basically we're able to query them in real time. That is our foundation layer here. This architecture enables us to get more and more data every day and to keep it fresh and to keep it in
[08:48] continuous loop which would be our key number one pillar for building this whole system. But then we are getting five different data providers in there is a lot of data and what is the biggest
[09:01] advantage of it living in the unity catalog is because everything is there everything is stored everything is easily accessible and it sits right there next to your Salesforce data. So basically we have our own account ids
[09:15] and all of those sorts of technographic signals across all of different enrichment providers. But now that we have one big table of enrichment, what do we do next? So this is where we
[09:28] created a bit of a problem with a lot of the data. So what are we going to do now? How do we know what is our single source of truth? How we can choose our data or who do we trust? Because if we're going to do waterfall enrichment,
[09:41] then we are just saying one provider is better than another one. So there is no logic behind that. Why did we buy this many providers? And on the other hand, how are we going to know what data point
[09:54] is the best and which provider is best for the revenue, for the parenting hierarchy, for intent signals? And that's we that's when we sat down, we were thinking about it and basically
[10:06] that's when we came up with data quality orchestration agent. So what is data quality orchestration agent? It actually lives on its four key pillars that keep it in continuous loop. That would be
[10:20] detect, reason, act, and govern. So let's start with the detect piece. We have developed families of agents that are living within our data brick system.
[10:33] They're running throughout the night. They're scanning all of the Salesforce fields that we uh that we defined. So basically, some of the agents are scanning the URLs, checking if the URLs are correct. Are they being redirected?
[10:46] We're checking if our hierarchy is correct, if our revenue rollup is correct, if the billing country is correct. We're do doing thousands of checkings and then we're going to the reason part. We defined our agents that
[10:58] way that we provided a context and reasoning. So when agents start working they know exactly the context behind it so that they can select uh select the skill they need to use identify the
[11:10] issue and then later on apply confidence threshold for that issue and that enables us to go to the act to the third pillar. So basically there is what actually everything happens what it
[11:23] means now when agent selected its uh its skill and it know its issue and it has a confidence it's going to loop into our triangulation framework within our data providers. So basically now we are
[11:36] choosing the best data point out of our providers based on the model's input. After that, we're going just to the step four, which is govern, which allows us to write that data automatically back
[11:48] into Salesforce and then consider Salesforce our single source of truth. And that is how we keep our continuous loop going. Once something is updated, it's being sent back to the system and when agent reads it, it's not going to
[12:00] do anything about it because now data is updated and best-in-class data points are provided here. But once something changes, if it goes stale, if it's invalid, if it's missing for some reason, agent is going to be triggered
[12:12] again for that account and the whole process will just be in the loop. So moving forward, how we started with everything. So basically it would be the AI powered matching and confidence
[12:25] scoring. So as we said, we're working with the five data providers which are Duns and Brad Street, AG Insights, Zoom Info, Sumble and Crunchb. So we're sending our list of accounts to them.
[12:37] They are doing the matchings on their end. They are returning the data with eographics, firmographics and everything, but at the end of the day, we want to make sure they did a correct match. So we're not going to blindly
[12:50] trust them in this one, but we want to assess if we're talking about the same account about same entity that is located within the same country because that can really impact our parenting hierarchies as well as revenue rollups
[13:04] which then scales to the planning and segmentation. We can get a lot of things wrong here. So when we started doing this before a couple of years so what we introduced is like character level
[13:16] comparison which would be 11stein distance and jakard similarity which would be pretty let's say at this point we have to agree pretty old pretty simple it's really efficient but it doesn't understand the semantics and the
[13:29] differences between the words and if they're actually matching correctly then we moved on into the vector search basically we are indexing all of our sfdc base directly in data bricks so
[13:41] that we can calculate coine similarity between every uh between our account names in Salesforce and all of the other enrichment providers and assign similarity score similarity would be a
[13:53] level up because it understands a context it's it has been trained upon a lot of things understands abbreviations so basically uh it understands when to remove incorporation
[14:05] LTD limited without you having to manually flag those things and at last we included the AI agent as a validation to mimic human reasoning. So basically we have these things and then we are
[14:18] evaluating each of our enrichment providers. So then if they pass this threshold they're going to go into our triangulation framework for selection of the best-in-class data points.
[14:33] So the question was when we were talking about data quality orchestration agent what is actually that where is everything going to live how are we going to navigate through it and we came to a conclusion that it should be an app
[14:48] production ready app and data bricks with built-in governance because we are building everything data bricks on data bricks and what are the advantages of our app so basically if we're deploying
[15:00] something In datab bricks, it's really easy to manage and to scale. So we can work with open-source stack like fast API and react that we use for this application as well as to easily manage
[15:13] and set up delta sharing in minutes and to keep data flowing in every day which is foundational. And as well once we have our backend and front end ready we just click deploy and the app is hosted
[15:25] and deployed within data bricks in a matter of minutes. So what is al also actually important is that here we have a really seamless integration because our data never leaves data bricks. Everything lives within the unity
[15:37] catalog. Delta sharing allows us to capture that data afterwards. What we are doing there is that we can also write data from data bricks to salesforce and everything integrates
[15:50] perfectly here and of course everything is governed and secure. How did we envision data quality orchestration agent? So when we were
[16:02] thinking about this, we figured out that we needed to have two key personas. One key persona that is going to be surfaced our app and the UI in order to action upon data enrichment points are account executives and on the other hand we have
[16:16] strategic leads. So when it comes to their roles, they're a bit different. Account executives care only about their books of businesses. is they're making decisions based on their targets and on the strategic leads as it says the word
[16:30] they're more of a strategic they looking at the whole business unit and they need to create some other uh to make some other type of decisions. So first for account executives we are holding them
[16:44] accountable for the data quality and data hygiene part which in this terms would mean for their book of business they're going to be notified if some of the URLs are being redirected if URLs
[16:56] are failing so that they can so they can archive those accounts and that we can keep our system clean. On the other hand they are going to be accountable for dduplication. We have a lot of duplicated accounts in
[17:09] Salesforce. So the agents are flagging everything. The only thing they need to do is to click merge or reparent under the same hierarchy or just dismiss this if they do not like what is actually surfaced there. But I can assure you
[17:22] what we are surfacing in the app is only high confidence because we are really on to building trust with us stakeholders here. And lastly, they're going to be accountable for formographics as for example billing country validation
[17:35] because sometimes that data cannot it's not correct and it impacts our segmentation. And why do we care about our data quality and data hygiene in Salesforce is because when we want to
[17:47] match our data to our external providers, these are the points that really matter. The URLs to be up to date, the billing countries and everything so that we can have a really good data flowing in. On the other hand,
[18:01] if we're talking about strategic leads, they're going to take also care about the data quality when it comes to unassigned accounts in their business units. But the more more important part here is they're going to take place in
[18:14] validating parenting hierarchies and then overall revenue rollups which directly impacts planning. While we were doing these things, we came to the conclusion that okay, we want to keep
[18:27] everything within Salesforce governed with the best-in-class data and we're protecting it that way. But on the other hand, what about the data that comes in?
[18:40] What are we doing with that? Are we allowing bad data to come in? And basically that was the part where we wanted to expand our agent, our application, and to include more things. So basically requests. One of the
[18:53] requests would be new accounts intake process because we do not want any more account executives or strategic leads go to Salesforce and open uh open accounts by themselves. We're going to do it in
[19:06] bulk for them via this application. What do they need to provide to us is six fields that are mandatory for us in order to get good match matches by our enrichment providers and they will not be able to submit data with uh to submit
[19:20] accounts without those six fields. After that, we are running everything throughout the dduplication process by the URL matchings and as well as vector search similarity by name within the
[19:32] same billing country. After that is being done, account executives and strategic leads are notified how many of their accounts passed actually the check and what is rejected because it's duplicated. And from that point, I think
[19:45] we won't have to deal with duplicates that much. So that was our idea. So we crafted the app for these two key personas and made everything visible and really accessible to them in this point.
[20:03] So this would be the UI of our data quality orchestration agent. Basically it's an app that is really intuitive. It is really user friendly and we wanted to gather it uh gather it in compliance with the needs of our internal customers
[20:17] in this terms account executive and strategic leads. So what you can see on the screen is when they log in their email is going to be automatically uh automatically recognized. So everything in the app is going to be filtered to
[20:29] them and that is also a great advantage of data bricks because we have rowle security. So you can easily control that what can somebody see and also what they can approve in this application because it is automatically going to trigger
[20:42] egresses to Salesforce. So we don't want some some other strategic leads that are not accountable for that business unit going around and approving something else. So UI is pretty simple. You are
[20:55] notified how many accounts are there that went throughout the dduplication URLs and that you need to validate them. But what we were thinking about is is at first we wanted this to be an opt-in
[21:07] process. So basically we're going to send Slack alerts to account executives and strategic uh leads. This is how many accounts you have to review. Please come into our app and finish this in two
[21:19] weeks and we are going to be done with this exercise. But we wanted to stay away from the opt-in process because we want to identify the whole enrichment and go towards full autonomous scale of
[21:34] always on enrichment and on that path we really need to build a high trust with our stakeholders make them trust the data so that we can in the next months or years I hope it's going to be months
[21:47] but move towards fully automated enrichment system with minimal human uh human in loop for some really complex decisions. So what are we doing here is
[21:59] that we introduce the opt out process. So basically we're surfacing all of these data points to you. We're saying you have two weeks to validate. If you accept that's cool you don't have to do that. But on the other hand if you
[22:12] reject that's the only thing we're not going to do any actions of. And then after that when we do a cut everything will be live in Salesforce. So that is our idea here how we want to make it uh
[22:26] how we want to make it identified as possible and we have already started with this process. So now I'm going to talk a bit uh about a couple of sections in our app. I'll
[22:38] leave some time also for the Q&A if you're interested in some particular session and how we uh section or how we did something. But here is what we did for the continuous account dduplication.
[22:50] So basically this is how the UI looks. Uh it's really intuitive and you can see a lot of reds and greens and notes and what is happening there. So basically we
[23:03] want to make sure that everybody knows what they're doing. We want to make sure that they know what is the prioritization for a target account over the duplicated account that should be deleted, merged or reparented. We're
[23:15] surfacing all of the data to them so they can see what is happening there. So all of the logic is transparent. So how did we actually build this? We started off with some simple let's say simple
[23:28] SQL queries that uh that are uh that are there and then we have our dimensions from our big enrichment tables. So basically once Dun and Brad Street does its own matching they're providing us
[23:40] with Dun's numbers. Dun's numbers are on the entity level which are a really good signal if some record is duplicated. So we're taking Dun's numbers into account. We are taking the URLs into account and
[23:52] we are also running our own vector search name similarity in order to identify the duplicates if maybe they do not have some URL address or they didn't go throughout the enrichment process. So that is the part that was done and then
[24:05] we're surfacing loads of duplicates throughout our system and then that's where our agent for dduplication comes in where we use llm as judge. So basically it is mimicking human behavior. We are passing all of the data
[24:18] points to him. We're saying these are the account names. These are the URLs we identified. They are duplicated by Dun's numbers. This is this is the data from DNB. This is what is happening in the
[24:30] back end. Can you please make a decision if this is actually a duplicated? Should we surface this one to our account executives and strategy leads? And if the confidence is high, we're going to
[24:42] surface it to them for now. Medium and low, we don't want to play with those things because we really need to build trust with our stakeholders and we want to keep them in the loop. Continuously put feedback, keep improving our models.
[24:57] On the other hand, we also have what we did is structured things for planning. You can see here that we wanted to make everything available in one place. So this would be a field. This would be a
[25:10] page for parenting hierarchy. So basically once you log in you're going to be able to see loads of accounts. So what are we doing here and how did we actually build this whole system?
[25:22] Usually for enrichment pro uh from enrichment providers when it comes to parenting hierarchy we're mainly using AG insights and DNB. So we're taking their insights and basically throughout
[25:34] our SQL codes we are doing the parentings and the new accounts that should be new ultimate parents. So we want to track if some account got acquired got merged should live in other hierarchy in Salesforce and we really
[25:47] need to know that. But how can we trust those signals? We are taking any signal into account and we're logging that into our tables, but we're not surfacing, we're not reparenting before we do a
[26:00] multi- aent validation on top. So what we did here is that we collect all of our data uh data providers data. We we connect it to new ultimate parents if
[26:12] there is any changes proposed in the system and then we are passing everything through really really huge contextual prompt with reasoning that we worked really hard with planning with
[26:25] planning team on. So basically what did we do there? We explained how we want our corporate hierarchy to look like what data bricks actually cares for. We do not want to see ultimate parents
[26:37] surfaced as holding companies. We do not want everything to roll under government accounts which can be uh which can be really often seen with enrichment providers. We don't want that. We don't
[26:49] want private equities. We don't want venture capitals there. If it's a joint venture, we don't care if it's 51% or more. We only want to parent it if it's 100% owned business because that is how
[27:02] our business operates and we tailored everything just to be in compliance with our planning standards and to make this process as seamless as possible. After that is being done. We are getting the data back from our AI agent with the
[27:16] whole reasoning. They're not just proposing the change. They're telling us why is this change correct or not correct or even neither of these two is correct. not even the parent that you have in Salesforce, not the parent that
[27:29] your enrichment providers suggested, but some third. So, we're just like, wow, this is cool. This is new. We're going to use this as well. But in this moment, when we run through that, we're also
[27:42] running it through a second agent because we want to we want to eliminate hallucinations as much as possible. So for the first scan, we're using the Opus models because they're they really really have a good and deep reasoning
[27:56] and afterwards we're validating everything with Sonat because when everything is already in place, it can be easily validated by Sonat and access the web on that on that side. So once
[28:09] again about the UI here what we are surfacing is basically the account their current ultimate parents and then recommended ultimate parents. In this terms, you would be able to see in this app all of the corporate hierarchy. If
[28:23] you're moving one account and that account is moving to another ultimate parent, all of the accounts underneath are going to move with it. And our account executives and strategic leads really need to know the whole hierarchy when they're changing those things. So,
[28:36] we introduced that also
[28:49] are just making the final call and final decision here. So basically what I would like to say now is I think that we're actually ready for the new era of
[29:02] agentic planning powered by delta sharing built completely data bricks on data bricks and um h can you tell us about the impact of this agent you
[29:15] >> sure of course actually we just launched our data quality orchestration agent last week and we already see some impacts. So, uh there are three key things. Uh first,
[29:29] more scale. Second, lower cost and more trust. What about the more scale? Today in the back end, actually our agent is scanning over 30,000 records across
[29:43] different data systems every night to do like validations, make recommendations, and automatically make decisions. and lower cost. Comparing to previous
[29:56] we're doing is like we're using human to do the data operation. It is just super expensive because with more data as on we need more human and more hours. It's just very expensive but our agent is not
[30:09] because our agent is scaled by the compute not by the high count. Um so and also we're also using AI wisely. We're only using AI when needed. We're not
[30:21] burning the expensive token for every single task. And thirdly, more trust because instead of human make decisions on what are the manual changes are uh without any logging or tracking right
[30:35] now for all the single update that the agents made we are we're doing fully audit and made all the changes by the high confidence score. So that's the impact we've made since
[30:49] last week. Before you guys go to next session, uh here are some key takeaways you can learn from this session. So first uh do not move the data sharing it. Please leveraging the data share at
[31:01] data bricks because you can read the data directly from where it lives. And second do not trust a single data source and do not trust a single agent. Please always triangulate your data source and
[31:13] use another agent on top of another agent to avoid the hallucination. And the third is like let the confidence throughout the work uh the easy scenes can down by the agent but always please
[31:25] include human in the loop for the complex decision. So thanks for thanks guys. I think that's it. >> Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.