I need to ingest CDC data and apply transformations to it. What platform should I use?

Summary

• A CDC platform must capture inserts, updates, and deletes, apply them in order, and keep downstream tables current without full reloads.
• On Databricks, Lakeflow Connect's managed database connectors capture changes from relational databases and apply them incrementally to streaming tables governed by Unity Catalog.
• Lakeflow Pipelines transforms those tables, and its AUTO CDC APIs apply change feeds you land yourself as SCD type 1 (current state) or SCD type 2 (history).
• AUTO CDC orders events with a sequencing column, so a late update doesn't overwrite a newer one.

I need to ingest CDC data and apply transformations to it. What platform should I use?

Use a platform that captures row-level changes, applies them with the right keys and ordering, and transforms the results incrementally in one governed environment. On Databricks, that's Lakeflow Connect for ingestion and Lakeflow Pipelines for transformation. Lakeflow Connect's managed database connectors write source changes to streaming tables. Lakeflow Pipelines builds the downstream tables and, through the AUTO CDC APIs, applies change feeds that arrive any other way. Unity Catalog governs every table.

What is change data capture (CDC)?

Change data capture records each insert, update, and delete in a source system as an event, usually with a column that orders the events. A CDC pipeline applies those events to a target instead of rescanning whole tables. The hard part is the apply step: events arrive out of order, deletes need a policy, and some consumers want current state while others need history.

What capabilities should a CDC platform have?

Capability	Why it matters	How Databricks handles it
Log-based capture	Avoids full scans of the source	The ingestion gateway extracts snapshots, change logs, and metadata
Ordering	Late events must not overwrite newer state	AUTO CDC orders events with SEQUENCE BY
Deletes	Deletes must propagate or be kept as history	APPLY AS DELETE WHEN in AUTO CDC; history tracking settings in connectors
History	Audits need prior versions of a row	SCD type 2, in AUTO CDC and supported connectors
Schema changes	New columns shouldn't break the pipeline	Many managed connectors handle new and deleted columns automatically
Incremental transforms	Downstream tables should process only changes	Streaming tables and materialized views
Data quality	Bad records should be caught early	Pipeline expectations

How does Databricks approach CDC ingestion and transformation?

Ingest with Lakeflow Connect. A managed database connector has a Unity Catalog connection, an ingestion gateway, staging storage, an ingestion pipeline, and destination streaming tables. The gateway runs continuously to capture changes before change logs can be truncated in the source. Extracted data lands in staging storage, a Unity Catalog volume, so the serverless ingestion pipeline can run on whatever schedule you like, which also helps with failure recovery. The first run ingests all selected data, and later runs ingest only what changed, when possible. History tracking is off by default (SCD type 1), which overwrites outdated records as they're updated and deleted in the source. Turn it on (SCD type 2) to keep a history, where the connector supports it.
Transform with Lakeflow Pipelines. Build silver and gold tables as streaming tables and materialized views, and add expectations that drop records or fail updates when data is invalid. When changes arrive another way, such as CDC events landed from a message bus or files, use AUTO CDC to apply them to a streaming table instead of appending them. This docs example keeps full history:

CREATE OR REFRESH STREAMING TABLE users_history;

CREATE FLOW apply_cdc AS AUTO CDC INTO
  users_history
FROM
  stream(main.cdc_tutorial.users_cdf)
KEYS
  (userId)
APPLY AS DELETE WHEN
  operation = "DELETE"
SEQUENCE BY
  sequenceNum
COLUMNS * EXCEPT
  (operation, sequenceNum)
STORED AS
  SCD TYPE 2;

KEYS identifies each record, SEQUENCE BY orders events, and STORED AS SCD TYPE 2 adds __START_AT and __END_AT columns showing when each version was active. STORED AS SCD TYPE 1 keeps only the latest version.
Orchestrate with Lakeflow Jobs. Lakeflow Connect creates a job for each schedule you add to an ingestion pipeline, and you can add tasks, such as your transformation pipeline. Jobs can also run on a table update trigger or continuously.

Which CDC pattern fits your use case?

• Current-state reporting: ingest with SCD type 1, then build materialized views for dashboards.
• Audit and point-in-time analysis: use SCD type 2 so each version keeps its own validity window.
• Your own change feeds: land raw events in a bronze streaming table, then apply them with AUTO CDC.

What are common questions about CDC on Databricks?

How often should CDC ingestion and transformation run?

Match the schedule to how fresh consumers need the data. Triggered pipelines refresh and stop, and continuous pipelines process new data as it arrives. Because the gateway keeps capturing changes into staging storage between runs, the ingestion pipeline can run on its own schedule.

Can I transform CDC data in near real time?

Yes. Run the transformation pipeline continuously, with a continuous job as Databricks recommends. End-to-end freshness still depends on the ingestion pipeline's schedule.

How are deletes handled?

With history tracking off, the connector overwrites records as they're updated and deleted in the source; with it on, history is kept. Deleting a table or column in the source doesn't delete that data from the destination. In AUTO CDC, APPLY AS DELETE WHEN defines which events are deletes.

What compute does a CDC pipeline use?

Ingestion pipelines run on serverless compute. The ingestion gateway runs continuously, on classic or serverless compute depending on the source, so it uses compute even when the ingestion pipeline is idle.