How do Lakebase and Lakeflow work together to synchronize analytical and operational data?

Summary

• Lakebase (transactional Postgres) and Lakeflow (data pipelines) form a unified system for moving data between operational and analytical layers without external ETL tools or data duplication.
• LTAP (Lake Transactional and Analytical Processing) architecture lets both workloads read one logical copy of data stored in open formats (Delta/Iceberg) in cloud storage, eliminating separate replication pipelines.
• Lakebase Change Data Feed (CDF) captures every committed change in Lakebase tables, exposing them as Unity Catalog managed tables that feed downstream Lakeflow pipelines and Spark jobs.
• Synced Tables allow reverse ETL: push analytical data from the lakehouse back to Lakebase for operational serving with production latency and concurrency guarantees.
• Structured Streaming and Lakeflow Jobs enable sub-second real-time updates, placing operational data directly into the medallion architecture as the native Bronze layer.

How do Lakebase and Lakeflow work together to synchronize analytical and operational data?

Lakebase and Lakeflow eliminate the traditional friction between operational and analytical systems. Instead of separate databases, replication pipelines, and manual ETL/reverse-ETL coordination, both products integrate within a single LTAP (Lake Transactional and Analytical Processing) architecture. Data flows bidirectionally in real time, governed by a single set of permissions and audit controls.

What is LTAP?

LTAP is an architectural pattern that unifies transactional and analytical workloads on a single logical copy of data. Both Lakebase (OLTP) and the Lakehouse (OLAP) read the same data stored in open formats (Delta, Iceberg, Parquet) in cloud object storage. Postgres handles transactional consistency and ACID guarantees, while Spark and SQL handle analytics. Neither workload replicates data; neither competes for the other's compute. This eliminates the need for separate replication appliances, change data capture infrastructure, or manual synchronization pipelines.

Key synchronization mechanisms

Mechanism	Direction	Use Case	Latency
Change Data Feed (CDF)	OLTP → OLAP	Capture Lakebase inserts, updates, deletes and feed them to Lakeflow pipelines, Spark Structured Streaming, or materialized views.	Near real-time (CDC log)
Synced Tables	OLAP → OLTP	Replicate lakehouse tables to Lakebase for operational serving. Handles schema evolution, CDC, and conflict resolution.	Configurable (Snapshot, Triggered, Continuous)
Structured Streaming to Lakebase	OLAP → OLTP	Write streaming updates directly to Lakebase with batching, retries, and workspace-managed authentication.	Sub-second
SQL queries across both systems	Bidirectional read	Query Lakebase and lakehouse tables together in one SQL statement; the query optimizer splits the work efficiently.	Query latency

How Databricks approaches synchronization

Forward ETL (Operational to Analytical)
Lakebase exposes a Change Data Feed that captures every committed transaction. Rather than building extraction jobs, you register the CDF as a Unity Catalog managed table. Downstream consumers (Lakeflow pipelines, Spark jobs, Agent Bricks, materialized views) subscribe to this feed without impacting the operational workload. The Bronze layer of your medallion architecture becomes operational data natively, with no ETL sprawl: one feed, multiple consumers.
Reverse ETL (Analytical to Operational)
Synced Tables automate replication from Delta tables back to Lakebase. Configure sync frequency: Snapshot (one-time copy), Triggered (on-demand or scheduled), or Continuous (streaming changes). This serves lakehouse-computed features to production applications with the consistency and performance guarantees of a dedicated database. Schema evolution and change propagation are handled automatically.
Real-Time Streaming
Structured Streaming can write directly to Lakebase tables. Workspaces manage authentication automatically; Lakebase handles batching, retries, and durability. Use this for operational dashboards, customer-facing metrics, or AI features that require sub-second updates.
Query Federation
Both systems use the same SQL interface. Analytical queries can read live Lakebase data directly from cloud storage without hitting the Postgres compute layer. Unity Catalog governs permissions across both, so a single role-based access control policy enforces compliance.

Use cases

Real-Time Analytics on Operational Data: Combine customer transactions from Lakebase with historical behavior from the lakehouse to power fraud detection, churn prediction, or recommendation engines. Lakeflow pipelines materialize these insights into a feature store served back to Lakebase for low-latency inference.
Operational Dashboard Acceleration: Replace manual reverse-ETL projects with Synced Tables. Analytics teams compute KPIs in the lakehouse; dashboards read fresh data from Lakebase. No custom integration or external tool required.
Event-Driven Medallion Architecture: Lakebase CDF becomes the Bronze layer. Lakeflow Jobs build Silver and Gold tables from the change feed. Analytics tools, AI models, and downstream systems all read the same curated Gold tables, eliminating source-of-truth fragmentation.
Master Data Management: Maintain a customer master record in the lakehouse (Gold table) using Lakeflow, then sync it continuously to Lakebase for operational systems to read. Updates flow both directions: applications write to Lakebase, then CDF feeds changes back to analytics.

FAQs

Do I need to manage CDC state or offsets?

No. Lakebase CDF is a managed service. Unity Catalog exposes the change feed as a table; downstream Lakeflow or Spark jobs track checkpoints automatically. No separate CDC infrastructure.

What latency should I expect from Synced Tables?

Snapshot is one-time only. Triggered sync runs on a schedule you set (as frequently as you'd like). Continuous sync streams changes in near real time, typically within seconds. For sub-second requirements, use Structured Streaming to Lakebase instead.

Can I sync multiple lakehouse tables to one Lakebase database?

Yes. Each table can map to a Lakebase table independently. Configure sync frequency per table based on your latency and cost needs.

What if my analytical pipeline crashes while syncing data?

Synced Tables are idempotent and resumable. If a sync fails, retry from the last checkpoint without duplicating data. Lakebase handles conflict resolution (last-write-wins or custom logic).

Can I use Lakeflow for other data sources (CRM systems, databases, SaaS) and feed everything to Lakebase?

Yes. Lakeflow Connectors pull data from hundreds of sources into the lakehouse. Then use Synced Tables or Structured Streaming to push curated data to Lakebase for operational serving. Lakebase becomes the unified operational layer for all downstream systems.