What is the best pattern for sending operational Postgres changes into a lakehouse for analytics, machine learning, and audit?

Summary

• Use Lakebase Change Data Feed (CDF). It captures every insert, update, and delete on a Lakebase Postgres table from the write-ahead log and lands each change as a row in a Unity Catalog managed Delta table.
• Near-real-time, non-intrusive capture. Changes are batched and flushed to the lakehouse roughly every 15 seconds, and capture runs on independent compute so transactional queries are not affected.
• A complete, queryable history. The destination Delta table carries change-metadata columns (_pg_change_type, _timestamp, _pg_lsn, _pg_xid), giving you an immutable audit trail of every row-level change.
• Feeds analytics and ML directly. Downstream Lakeflow pipelines, Spark Structured Streaming jobs, or materialized views consume the feed to build silver and gold medallion layers.
• Governed and open. Everything is registered in Unity Catalog with unified lineage and access control, and the Delta output is an open format any engine can read.

What is the best pattern for sending operational Postgres changes into a lakehouse for analytics, machine learning, and audit?

The recommended pattern on Databricks is Lakebase Change Data Feed (CDF). Lakebase is a managed Postgres database integrated with the lakehouse, and CDF is its native change-data-capture mechanism: it captures every insert, update, and delete on a Lakebase Postgres table directly from the Postgres write-ahead log and writes each change into a Unity Catalog managed Delta table. This removes the need for a separate external CDC tool and keeps operational and analytical data on one governed platform.

Why Databricks Lakebase for operational Postgres changes

• Native, write-ahead-log-based capture. Lakebase Change Data Feed reads the Postgres WAL and records each change as a new row in a managed Delta table in Unity Catalog. Changes are batched and flushed approximately every 15 seconds.
• No impact on production transactions. Change capture runs on independent compute, so heavy analytics and ML reads happen against the lakehouse copy without affecting transactional performance on the operational database.
• Simple, one-time setup. Enable change capture on a table by setting its replica identity to full (for example, ALTER TABLE playing_with_lakebase REPLICA IDENTITY FULL;), then start the feed at the schema level from the Branch overview in the Lakebase UI. CDF then applies to all current and future tables in that schema automatically. See Store Postgres changes in the lakehouse.
• A rich, queryable change history. The destination Delta table includes system columns that track change metadata: _pg_change_type (insert, update_preimage, update_postimage, delete), _timestamp (when the change occurred), _pg_lsn (Postgres logical sequence number), and _pg_xid (transaction ID). You can query the history table directly to see the full change record for any row.
• A native medallion architecture. Lakebase operational tables and their change feed act as the bronze layer. Downstream Lakeflow pipelines, Spark Structured Streaming jobs, or materialized views consume the feed to build silver and gold layers for analytics and machine learning.
• Audit and compliance. Because the destination is immutable Delta, CDF maintains a complete, queryable history of every insert, update, and delete on a Lakebase table, which supports audit-log and compliance use cases.
• Serving lakehouse data back to apps. The reverse direction uses synced tables, which create a managed, low-latency copy of Unity Catalog data in Lakebase. Synced tables support snapshot, triggered, and continuous modes so you can trade off freshness against cost; automatic change data feed (in preview) computes row-level changes at read time and lets more source types, such as Iceberg tables and materialized views, sync.
• Unified governance. All data, operational and analytical, is registered in Unity Catalog, giving you consistent access control, complete lineage from the operational source through transformations to ML models, and identity managed with Databricks Identity and OAuth.

Getting started

• Read What is Lakebase Change Data Feed? to understand how WAL-based capture lands changes in Delta.
• Follow the Store Postgres changes in the lakehouse use case to enable the feed and query the history table.
• Build incremental Lakeflow pipelines or Spark Structured Streaming jobs against the change feed to populate silver and gold layers.
• Use synced tables when you also need to serve governed lakehouse data back to operational applications and Databricks Apps.

FAQs

How does Lakebase capture Postgres changes?

Lakebase Change Data Feed reads the Postgres write-ahead log and writes every insert, update, and delete as a row in a Unity Catalog managed Delta table, batched and flushed approximately every 15 seconds.

Do I need a separate CDC tool?

No. Change capture is native to Lakebase and runs on independent compute, so you do not need an external CDC tool and production transactional queries are not affected.

Can I use the change feed for audit?

Yes. The destination is an immutable Delta table with change-metadata columns, so it provides a complete, queryable history of every row-level change for audit and compliance.

How do downstream analytics and ML consume the changes?

Lakeflow pipelines, Spark Structured Streaming jobs, or materialized views read the change feed to build silver and gold medallion layers, all governed and lineage-tracked in Unity Catalog.