managed Iceberg tables in Unity Catalog vs external Iceberg tables

Summary

• Managed Iceberg tables in Unity Catalog give Databricks full control over metadata, storage lifecycle, and optimization; Unity Catalog automatically handles snapshot expiration, file compaction, and liquid clustering.
• External (foreign) Iceberg tables registered in Unity Catalog via Lakehouse Federation are read-only in Databricks and carry limited platform support; time travel works only for previously-accessed snapshots and partition evolution is not supported.
• Managed tables support full read-write access from both Databricks compute and external Iceberg-compatible engines through the Unity Catalog Iceberg REST catalog endpoint, with credential vending.
• Foreign Iceberg tables require a manual REFRESH FOREIGN TABLE command to surface new snapshots and do not support credential vending through the REST API.
• For most Databricks-first workflows, managed Iceberg tables are the recommended default; external registration makes sense when another system must own the data location or write directly to storage.

managed Iceberg tables in Unity Catalog vs external Iceberg tables

Managed Iceberg tables are the recommended default for Databricks-native workflows: Unity Catalog owns the storage lifecycle, runs automatic optimizations, and provides full read-write access to both Databricks compute and external Iceberg clients. External (foreign) Iceberg tables, registered via Lakehouse Federation, let you query data governed by another catalog system without moving it, but Databricks treats them as read-only and provides limited platform integration.

What are managed Iceberg tables in Unity Catalog?

A managed Iceberg table in Unity Catalog is a native table type where Databricks controls the storage location, metadata, and lifecycle. When you drop a managed table, Unity Catalog deletes the underlying data files after a configurable recovery window (default seven days). Unity Catalog also runs serverless compute to maintain Iceberg metadata and applies automated optimizations including liquid clustering, predictive optimization, and incremental refresh for materialized views. Managed tables require Databricks Runtime 16.4 LTS or later and serverless compute enabled on the workspace.

What are external (foreign) Iceberg tables?

Foreign Iceberg tables are registered in Unity Catalog through Lakehouse Federation. They represent Iceberg tables whose metadata and files live in an external catalog such as AWS Glue or another Iceberg-compatible system. Databricks reads the metadata from the external catalog on demand but does not manage the storage lifecycle. Foreign Iceberg tables are read-only in Databricks; write operations are not supported. Time travel is limited to snapshots that have already been accessed through Unity Catalog, and partition evolution is not supported. To surface new snapshots, you must run REFRESH FOREIGN TABLE manually.

Managed vs external Iceberg tables in Unity Catalog: how they compare

Capability	Managed Iceberg	Foreign/External Iceberg
Write support (Databricks compute)	Full read-write	Read-only
Storage lifecycle management	Unity Catalog manages files, purges on DROP after recovery window	Owner's external catalog controls files; DROP removes catalog entry only
Snapshot handling	Automatic expiration and compaction	Requires manual REFRESH FOREIGN TABLE for new snapshots
Liquid clustering	Supported	Not supported
Predictive optimization	Supported	Not supported
Materialized views (incremental refresh)	Supported	Not supported
Partition evolution	Supported	Not supported
External engine access via Iceberg REST catalog	Full read-write with credential vending	Read-only; credential vending not supported
Time travel	Full	Limited to previously-accessed snapshots
Databricks Runtime requirement	16.4 LTS or later	Standard UC-compatible runtime

How Databricks approaches Iceberg governance in Unity Catalog

Unity Catalog exposes an Iceberg REST catalog API endpoint (/api/2.1/unity-catalog/iceberg-rest) that lets external systems such as Apache Spark, Apache Flink, and other Iceberg-compatible query engines interact with registered tables. For managed Iceberg tables, the REST API supports both reads and writes and automatically vends temporary cloud-storage credentials so external engines can commit changes without requiring direct IAM configuration.
For external engines writing to managed Iceberg tables, Unity Catalog enforces the Iceberg specification: it blocks commits that would introduce duplicate data files in a single snapshot, protecting table correctness when multiple compute engines access the same table concurrently.
Foreign Iceberg tables accessed through this endpoint are read-only. They do not participate in credential vending, meaning clients must supply their own credentials to reach the underlying storage.
Managed Iceberg tables also integrate with the full Unity Catalog governance stack: column-level lineage tracked automatically across workspaces, fine-grained privileges and attribute-based access control, audit logs captured in system tables, and data quality monitoring through built-in profiling.

When to choose each type

Choose managed Iceberg tables when:

• Your primary compute is Databricks and you want automated storage management, optimization, and the full governance feature set.
• You need external Iceberg clients to write to the table alongside Databricks (Unity Catalog coordinates concurrent commits).
• You want lineage, predictive optimization, liquid clustering, and materialized view support.

Choose foreign/external Iceberg tables when:

• Data is already managed by another catalog system (for example, AWS Glue or a third-party Iceberg catalog) and you need to query it in place without moving files.
• Another system must own the data location and write directly to storage.
• You accept the limitations: read-only in Databricks, manual refresh for new snapshots, no platform optimizations.

FAQs

Can external Iceberg engines write to managed Iceberg tables in Unity Catalog?

Yes. Managed Iceberg tables support full read-write access from external Iceberg-compatible engines through the Unity Catalog Iceberg REST catalog API. The REST endpoint vends temporary cloud-storage credentials, and Unity Catalog enforces the Iceberg specification to prevent duplicate-file commits from concurrent writers.

What happens to data files when I drop a managed Iceberg table?

Unity Catalog schedules the underlying data files for deletion after the recovery window expires (the default is seven days). During that window you can restore the table. For foreign/external Iceberg tables, dropping the table in Unity Catalog removes the catalog registration but does not touch the files in your cloud storage.

Are foreign Iceberg tables integrated with Unity Catalog lineage and governance?

Foreign Iceberg tables are registered in Unity Catalog and subject to UC privileges, so access control applies. However, automatic lineage capture and platform optimizations (liquid clustering, predictive optimization) are not available for foreign tables.

Do managed Iceberg tables require any special compute configuration?

Managed Iceberg tables require Databricks Runtime 16.4 LTS or later and serverless compute enabled in the workspace. Foreign Iceberg tables accessed via Lakehouse Federation work with standard Unity Catalog-compatible runtimes.

Can I convert a foreign Iceberg table to a managed Iceberg table?

Yes. You can create a managed Iceberg table and use CREATE TABLE AS SELECT or DEEP CLONE patterns to copy the data. The foreign table's history is not preserved during this process, and you'll need to configure storage credentials and external locations as part of the migration setup.