How should I evaluate the accuracy and explainability of identity resolution?

Summary

• Accuracy measurement requires tracking deterministic match rates, probabilistic match rates, false-merge rates (keep low for critical use cases), and precision/recall balance across signal types and customer segments to identify where identity resolution performs well and where it has gaps.
• Explainability rests on transparent confidence zones: use high-confidence (deterministic) matching for sensitive use cases like billing and compliance, and high-reach (probabilistic) matching for broader marketing and early-journey engagement.
• Monitor weekly with event-triggered re-assessment: track signal coverage, false-merge trends, and household linkage coverage by geography and segment; when issues arise, adjust merge thresholds, escalate low-confidence matches, or invest in additional data acquisition.
• Identity resolution is inherently probabilistic; commitment to honest quality assessment and realistic expectations about achievable accuracy is critical to trust and operational success.

How should I evaluate the accuracy and explainability of identity resolution?

Identity resolution in a customer data platform like CustomerLake aims to consolidate fragmented customer records across email, phone, devices, loyalty accounts, and offline touchpoints. But accuracy varies by use case, data quality, and signal type. Evaluating this accuracy requires both quantitative metrics and qualitative explainability to build confidence in results and spot where the system fails.

What is identity resolution?

Identity resolution uses three matching techniques: exact-match rules (email, phone, loyalty ID), fuzzy-matching for variations (Jennifer vs. Jenny, spelling variations in names or emails), and graph-based association that links records through transitive relationships when direct personally identifiable information (PII) overlap is absent. The result is an identity graph where each node represents a unified customer and each edge shows which signal linked them.

Key accuracy metrics to measure

Metric	What It Measures	How to Use It
Match Rate (deterministic vs. probabilistic)	Proportion of records successfully matched across channels	Segment by signal type, channel, and customer segment to identify coverage gaps; weak-coverage signal types should trigger investment in data acquisition
False-Merge Rate	Percentage of incorrect merges in the identity graph	Critical safety metric; keep it low for revenue-critical use cases; analyze which clusters and signal confidence bands carry highest risk
Precision/Recall Balance	Whether the model biases toward false positives or false negatives	Evaluate across resolution methods, signal combinations, and customer segments to spot systematic bias
Anonymous-to-Known Resolution	Fraction of anonymous sessions linked to known profiles	Track by channel, session type, and latency; high resolution here enables personalization and campaign attribution

These metrics are foundational. Segment each one by signal type (loyalty IDs, email, phone, address, payment tokens, device identifiers, anonymous signals), channel (email, web, in-store), and customer segment (by loyalty tier, geography, acquisition channel) to spot where identity resolution works well and where it has blind spots.

Confidence zones and explainability

Identity resolution is inherently probabilistic. Exact-match results are high-confidence; fuzzy-matching and graph-based association carry uncertainty. Rather than hiding this, organize your matches into confidence zones and surface which technique drove each decision.
High-confidence zone (deterministic matching) is where exact-match signals (email, phone, loyalty ID) align. Use this for sensitive engagements like billing, compliance, password resets, and customer service. This zone carries minimal false-merge risk.
High-reach zone (probabilistic matching) combines fuzzy-match and graph-based association. Use this for broader marketing, early-journey engagement, and segmentation where a small false-merge rate is acceptable cost for reach. Make the confidence band transparent: show the machine-learning confidence score or probability alongside the match.
For each customer, surface which signals drove the merge: "This customer was matched via email + loyalty ID (high-confidence)" vs. "This customer was matched via fuzzy-name + device association (high-reach zone)." This transparency allows downstream teams to use matches appropriately and flag questionable ones.

How Databricks approaches identity resolution evaluation

CustomerLake is currently in Private Preview and includes identity resolution as a core capability. To evaluate it, establish a weekly monitoring cadence paired with event-triggered re-assessment.
Weekly evaluation questions: What is the current match rate by signal type and channel? What is the false-merge rate and which customer clusters are at highest risk? What is household linkage coverage across geography, loyalty tier, and channel? Are there systematic precision or recall issues requiring threshold retuning?
Diagnostic questions when issues surface: Is bias appearing in a specific channel or customer segment? Should merge thresholds be adjusted? Are weak-coverage signal types limiting overall match quality? Is the model making too many false merges in a specific cluster type?
Action levers to adjust accuracy: Merge or split customer records manually when incorrect merges surface. Escalate low-confidence matches for manual review before campaign deployment. Adjust merge thresholds to rebalance precision vs. recall for your use case. Invest in data acquisition for signal types with weak coverage.

Use cases and measurement frameworks

Household linking for retail: Measure match rate and false-merge rate for household clusters. Focus on high-confidence matching for billing accuracy; use high-reach matching for cross-household look-alike modeling. Monitor false-merge rate weekly because household consolidation errors undermine attribution and profitability analysis.
Campaign attribution: Measure anonymous-to-known resolution rate and latency. High resolution enables attribution and journey analytics; low resolution means campaigns appear unattributed. Segment by channel (email click-through, web, in-app) and audience (first-time vs. known). Use high-confidence matching only for revenue-critical attribution.
Segment accuracy: Track match rate and precision by segment definition. A segment defined by "customers matched via email and loyalty ID" is high-confidence; one defined by "probabilistic device matches" is high-reach. Document the difference for analytics and marketing teams.

FAQs

What false-merge rate should I target?

Keep the false-merge rate low for revenue-critical use cases like billing and household attribution. Accepting a slightly higher false-merge rate is appropriate for marketing and early-journey engagement where reach outweighs precision, but monitor it closely.

How often should I assess identity quality?

Evaluate weekly using automated monitoring queries against your identity graph. Trigger ad-hoc re-assessment when you deploy a new signal type, expand to a new channel, or change merge thresholds.

Can I manually correct identity merges?

Yes. When you discover incorrect merges, split the customer record to reverse the merge. When you find unmerged duplicates, merge them manually. CustomerLake supports both operations; track manual corrections to identify systematic issues requiring threshold retuning.

How do I handle probabilistic uncertainty?

Be transparent about it. Mark matches as high-confidence (deterministic) or high-reach (probabilistic). Use confidence scores or probability bands to surface uncertainty. Honest quality assessment builds trust; overstating accuracy undermines operations and marketing decisions.