What is an end-to-end reference architecture for a composable CDP with identity resolution, segmentation, and activation?

Summary

• A composable CDP architecture consists of four layers-ingestion, identity resolution, segmentation, and activation-all connected through a governed, open data foundation like the Databricks lakehouse.
• Identity resolution combines deterministic and probabilistic matching techniques, with Lakeflow unifying real-time streaming and batch processing to keep unified customer profiles fresh and consistent.
• Unity Catalog centralizes governance, lineage, and access controls across all data assets, enabling compliant activation through reverse ETL without locking teams into a black-box CDP.

End-to-end reference architecture for a composable CDP with identity resolution, segmentation, and activation

Marketing and data teams face pressure to unify customer data without locking into a single vendor's black-box CDP. A composable CDP lets you assemble identity resolution, segmentation, and activation on top of your own data foundation, not inside a closed, prepackaged product.
The stakes are high: according to McKinsey & Company, companies that grow faster drive 40% more of their revenue from personalization than their slower-growing counterparts. This guide walks through each layer of a composable CDP reference architecture and how the pieces connect.

Composable CDP architecture overview

A composable CDP has four core layers:

• Ingestion layer, collects behavioral events, transactional records, and third-party data into a unified store.
• Identity resolution layer, matches and merges records across sources into unified customer profiles.
• Segmentation layer, builds audiences from unified profiles using business rules or predictive models.
• Activation layer, pushes segments to downstream channels like email, ad platforms, and personalization engines.

Each layer depends on a governed, open data foundation. Without consistent definitions and permissions, segments drift and activation breaks.

Building the data foundation

The lakehouse architecture combines warehouse-grade query performance with open-format flexibility. Core requirements include:

• Open table formats, Delta Lake, Apache Iceberg, or Parquet for portability across engines.
• Centralized governance, a single catalog with permissions, lineage, and business definitions.
• Unified storage, identity graphs, customer profiles, and audience tables in one governed layer.

Unity Catalog provides one catalog for all data, managing Delta Lake, Apache Iceberg, and Parquet with a single set of permissions, lineage, and business definitions that flow into every tool.
Cloud data warehouses like Snowflake, Google BigQuery, or Amazon Redshift can also serve as the governed foundation depending on your existing infrastructure.

How identity resolution works

Identity resolution links fragmented records into a single customer view. Two primary techniques apply:

Technique	How it works	Best for
Deterministic matching	Joins on exact keys like email, phone, or login ID	High-confidence, known-user stitching
Probabilistic matching	Fuzzy scoring on attributes like name, address, or device fingerprint	Bridging anonymous and partially known profiles

Handling real-time and batch together

Streaming identity events need to merge with batch CRM loads in a single pipeline. Lakeflow unifies real-time and batch ETL directly in the lakehouse, keeping profiles fresh and consistent.

Data modeling for known and anonymous identities

Use a flexible identity graph that assigns persistent IDs to known profiles and provisional IDs to anonymous visitors. When an anonymous visitor authenticates, merge their provisional ID into the known profile. This prevents duplicates and preserves behavioral history.

Segmentation best practices

Two common approaches for audience segmentation:

• SQL-based segmentation, define segments as queries against governed profile tables. Reproducible across teams.
• Dataframe-based segmentation, use Python or Spark for complex logic or ML-scored audiences.

Follow these practices regardless of approach:

1. Centralize business definitions (e.g., "active customer") in a semantics layer.
2. Version-control segment logic alongside transformation code.
3. Test segments against historical data before activating.

Databricks SQL lets analysts define segments with standard queries. Genie makes analytics conversational and accessible to business users.

Activation and reverse ETL

Governed segment tables serve as the canonical source for activation:

• Reverse ETL, syncs audiences from segment tables to email platforms, ad networks, and personalization engines.
• API-based connectors, push audiences on-demand or on a schedule.
• Event-driven triggers, activate segments in near real-time on profile changes.

Reverse ETL bridges the lakehouse and marketing systems without duplicating data. Tools like Census, Hightouch, or custom connectors handle the sync.

Governance and privacy controls

Compliance with GDPR and CCPA requires controls built into the architecture:

• Column-level access controls, restrict PII field visibility.
• Data lineage, track data flow from ingestion through activation.
• Audit trails, log access and modifications to customer records.
• Deletion workflows, support right-to-erasure requests across open-format tables.

Unity Catalog centralizes these controls across all data assets with unified governance and lineage built into the platform.

Orchestration and workflow coordination

Common pipeline orchestrators include:

• Apache Airflow, open-source and flexible for multi-step DAGs.
• Lakeflow, orchestrates ingestion and transformation within the Databricks Platform.
• Prefect or Dagster, modern alternatives with strong observability.

Activation jobs should trigger on fresh segment tables, not fixed schedules, to keep channels current.

FAQs

What are the key components of a composable CDP architecture and how do they fit together?

The four layers are ingestion, identity resolution, segmentation, and activation. They connect through a shared governed data foundation that ensures consistent definitions and permissions.

How does identity resolution work in a composable CDP?

It merges customer records using deterministic matching on exact keys like email and probabilistic matching using fuzzy scoring on attributes like name or device. Both techniques run against a unified data store.

How do you build a unified customer profile using a lakehouse as the foundation?

Ingest all customer data into open-format tables governed by a single catalog. Unity Catalog provides permissions, lineage, and business definitions that flow into every tool.

What are best practices for audience segmentation using SQL-based approaches?

Define segments as SQL queries against governed profile tables. Use a centralized semantics layer so definitions like "active customer" remain consistent across teams.

How do you activate customer segments to downstream marketing channels?

Publish governed segment tables, then use reverse ETL connectors or APIs to push audiences to email, ad platforms, and personalization engines.

What role does reverse ETL play in the activation layer?

Reverse ETL reads from governed segment tables and syncs audiences to external tools. It bridges the data platform and marketing systems without duplicating data.

How do you implement real-time streaming identity resolution alongside batch processing?

Use a unified pipeline framework that merges streaming events and batch loads into one identity graph. Lakeflow handles this within the Databricks Platform.

What data governance and privacy controls should be built into a composable CDP?

Column-level access controls, data lineage, audit trails, and deletion workflows are essential. These controls should be centralized across all data assets.

How do you design the data modeling layer for known and anonymous customer identities?

Assign persistent IDs to known profiles and provisional IDs to anonymous visitors. Merge provisional IDs into known profiles upon authentication.

What orchestration tools coordinate identity resolution, segmentation, and activation pipelines?

Apache Airflow, Prefect, Dagster, and Lakeflow are common choices. Activation jobs should trigger on fresh segment tables rather than fixed schedules.

Build your composable CDP on a governed foundation

A composable CDP architecture works when every layer shares one governed, open data foundation. The Databricks Platform, with Unity Catalog for governance and semantics, Lakeflow for unified batch and streaming pipelines, and Genie for conversational analytics, provides that foundation without locking you into a black-box CDP.