What is a good replacement for a brittle CDC pipeline into a warehouse?

Summary

• The right replacement depends on where the source of truth lives: Lakebase, an existing database, or a change feed you already produce.
• If your application data lives in Lakebase, Databricks' Lake Transactional/Analytical Processing (LTAP) architecture removes the external CDC copy, because analytical engines read the same underlying data.
• For an existing database such as MySQL, PostgreSQL, SQL Server, or Oracle, Lakeflow Connect managed database connectors run the change capture for you, and their release stages vary by connector.
• If you already produce a change feed, the AUTO CDC API in Lakeflow Pipelines applies it, including out-of-order events and slowly changing dimension (SCD) Type 1 and Type 2 history.

What is a good replacement for a brittle CDC pipeline into a warehouse?

Choose the replacement by where the source of truth lives. When the application database is Lakebase, Databricks' LTAP architecture removes the external CDC copy, because analytical engines read the same underlying data. When the source is an existing database, Lakeflow Connect managed database connectors run the capture for you. When you already have a change feed, the AUTO CDC API in Lakeflow Pipelines applies it to a Delta table. The release stages differ by option, so the table below labels them.

Why are CDC pipelines brittle?

Databricks' LTAP documentation describes the usual design: an external process polls the primary database, and a pipeline transforms row changes into columnar data. According to the docs, that pipeline "consumes resources on your primary transactional database, makes you handle schema changes and edge cases yourself, and trades data freshness against pipeline cost, all while adding points of failure." The docs also note that copying data between systems creates governance risk, because lineage can break as data moves.

How do the replacement options compare?

Option	Fits when	Where the data lands	Notes
Register Lakebase in Unity Catalog	Your application writes to Lakebase and the lakehouse needs governed read access	A read-only Unity Catalog catalog over the Postgres database	GA
Lakehouse//RT querying Lakebase	You need transactionally consistent analytics on live Lakebase data	Reads a columnar copy in object storage and merges recent changes	Beta
Lakebase Change Data Feed	Pipelines or audit need row-level change history	Unity Catalog managed Delta tables named lb_<table_name>_history	Public Preview; batches flush about every 15 seconds
Lakeflow Connect database connectors	The source is an existing database such as MySQL, PostgreSQL, SQL Server, or Oracle	Streaming tables, which are Delta tables	SQL Server is GA; PostgreSQL and MySQL are Public Preview; Oracle is Beta
AUTO CDC in Lakeflow Pipelines	You already produce a CDC feed or database snapshots	A streaming table with SCD Type 1 or Type 2 history	Bitemporal tracking is Beta

How do the Databricks options work?

Managed database connectors. An ingestion gateway extracts snapshots, change logs, and metadata from the source. It runs continuously on classic compute and stages data in a Unity Catalog volume, which is purged after 30 days. An ingestion pipeline on serverless compute then moves the data into streaming tables. Fully-managed connectors build on Lakeflow pipelines and add source-specific authentication, CDC, edge case handling, automated retries, and automated schema evolution. When a connector fails, it retries with exponential backoff.
AUTO CDC. AUTO CDC processes a change data feed, and AUTO CDC FROM SNAPSHOT compares snapshots when the source has no feed. Both support SCD Type 1 and Type 2. The docs say AUTO CDC handles out-of-order events without manual watermark code, and it requires serverless pipelines or the Pro or Advanced editions.
LTAP on Lakebase. Storage transcodes Postgres data into a columnar layout as it lands in the lake. The docs say "There is no pipeline for you to build, and no external process polling your database." Lakehouse//RT and Lakebase Change Data Feed both operate on that single copy, so the docs distinguish them from the external CDC that LTAP removes.

When should I keep a capture pipeline?

LTAP covers data that lives in Lakebase. If another database is the system of record, the docs' ingestion guidance applies: start with the most managed layer, which for supported databases is a Lakeflow Connect managed connector. If you only need ad hoc access, Lakehouse Federation queries sources read-only without moving data. Databricks still recommends ingestion for most use cases because it scales to high data volumes, low-latency querying, and third-party API limits, and it notes that a copy can go stale.

FAQs

Does Lakebase replace my warehouse?

No. Databricks describes purpose-built engines: Postgres serves transactions and the lakehouse serves analytics. LTAP removes the copy pipeline between them. It does not remove the analytical engine.

Which Lakeflow Connect database connectors are generally available?

The SQL Server connector became generally available on August 26, 2025. The PostgreSQL and MySQL connectors are in Public Preview and the Oracle connector is in Beta. Stages change, so check each connector's page.

How fresh is Lakebase change data?

Lakebase Change Data Feed (Public Preview) captures every insert, update, and delete from the write-ahead log, batched and flushed about every 15 seconds. Lakehouse//RT (Beta) reads the columnar copy and merges very recent changes from the pageserver.

Can I apply changes from a source that has no change feed?

Yes. AUTO CDC FROM SNAPSHOT determines changes by comparing in-order snapshots, which it can read from a Delta table, cloud storage files, or JDBC.