Skip to main content

When should I use a Lakebase synced table instead of querying a Unity Catalog table directly?

Summary

  • A Lakebase synced table copies a Unity Catalog table into Lakebase Postgres, a process commonly known as reverse ETL, so applications can read lakehouse data with standard Postgres drivers.
  • Use a synced table when an application needs fast lookup-style queries and transactional consistency over data the lakehouse produces.
  • Query the Unity Catalog table directly for analytics, because Databricks describes the lakehouse as optimized for analytics and enrichment.
  • Synced tables flow one way, from Unity Catalog into Lakebase. If your application writes the data and analytics needs it, use the other capabilities of Lake Transactional/Analytical Processing (LTAP) on Lakebase.

When should I use a Lakebase synced table instead of querying a Unity Catalog table directly?

Use a synced table when an application needs low-latency reads of lakehouse data through Postgres. Query the Unity Catalog table directly when the consumer is analytical. Databricks draws the line this way: "The lakehouse is optimized for analytics and enrichment, while Lakebase is designed for operational workloads that require fast lookup-style queries and transactional consistency." The decision comes down to who reads the data and which system owns the writes.

What is a Lakebase synced table?

A synced table is a managed copy of Unity Catalog data in Lakebase. Creating one gives you a synced table in Unity Catalog that references the sync pipeline, plus a read-only Postgres table that applications query. The Unity Catalog schema name becomes the Postgres schema name. Supported sources are managed and external Delta tables, managed and external Iceberg tables, views, and materialized views. Sync pipelines are managed Lakeflow pipelines that keep both the Unity Catalog synced table and the Postgres table up to date, as described in the synced tables documentation.

How does a synced table compare with a direct Unity Catalog query?

Question Lakebase synced table Direct Unity Catalog query
Who reads it Applications using standard Postgres drivers Analysts and dashboards on a SQL warehouse
Access pattern Fast lookup-style queries with transactional consistency Scans, joins, and aggregations
Copy of the data Managed copy in Lakebase that counts toward the branch storage quota No second copy to sync or store
Freshness Set by the sync mode: Snapshot, Triggered, or Continuous The source table's own freshness
Writes Read-only in practice; direct writes are overwritten on the next refresh Not applicable, because there is no synced copy
Access control Postgres privileges for application connections; Unity Catalog manages the synced table Unity Catalog permissions

Which sync mode should I use?

  • Snapshot copies all data on each sync, so the source only needs to support SELECT *. Databricks lists it for sources where more than a tenth of rows change per cycle.
  • Triggered runs on demand or on a schedule and propagates inserts, updates, and deletes. The source must provide a change data feed.
  • Continuous streams changes with seconds of latency. The docs rate it lowest lag and highest cost, with a minimum 15-second interval.

After the first sync, Snapshot and Triggered syncs must be triggered explicitly, for example with a Database Table Sync pipeline task in Lakeflow Jobs. Continuous mode manages itself. Triggered and Continuous modes support only additive schema changes.

Which workloads fit a synced table?

The docs list personalization engines that serve fresh user profiles to Databricks Apps, applications that serve model predictions or feature values computed in the lakehouse, customer-facing dashboards that serve KPIs in real time, fraud detection services that serve risk scores, and support tools that serve enriched customer records.

When is a direct query the better choice?

Choose a direct query when the consumer is an analyst, a dashboard, or a batch job. Size and type limits matter too: Databricks recommends not exceeding 1 TB for tables that need refreshes, a source table can have up to 20 synced tables, and GEOGRAPHY, GEOMETRY, VARIANT, and OBJECT columns are not supported.
If an application only needs read queries with sub-second responses on Unity Catalog tables, Lakehouse//RT is a serverless SQL warehouse type built for that. It is in Beta, runs ANSI read (SELECT) queries only, and the docs tell you to contact your Databricks account team to enable it.

What if my application writes the data?

Then the direction reverses. Databricks' LTAP guidance is that each table has a single writer, either Lakebase or the lakehouse. When Lakebase owns the writes, register the database in Unity Catalog (generally available) to query it read-only from the lakehouse, or use Lakehouse//RT (Beta) for live analytical queries.

FAQs

Can a synced table copy data from an external database such as MySQL or PostgreSQL?

No. Synced table sources are Unity Catalog tables, views, and materialized views. To bring external database data in, Lakeflow Connect database connectors ingest from relational databases such as MySQL and PostgreSQL into Unity Catalog (release stages vary by connector), and Lakehouse Federation queries those sources read-only.

Can my application write to a synced table?

Databricks strictly recommends running only read queries. The identity that creates a synced table is not granted INSERT or UPDATE, and the pipeline owns the data, so direct writes are overwritten on the next refresh.

Does a synced table use extra Lakebase resources?

Yes. Each synced table can use up to 16 connections, which count toward the project's connection limit, and the copy counts toward the branch storage quota. During a full refresh, the old and new versions both count until the sync completes.

How do I change a synced table's primary key?

Updating a synced table's definition in place is not supported. To change the primary key or timeseries key, or to make a non-additive schema change, delete the synced table and create a new one.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.