Managed Iceberg versus self-managed Iceberg?

Summary

• Managed and self-managed Iceberg tables use the same open table format. The difference is operational: who runs the catalog, who schedules maintenance such as snapshot expiration and compaction, and who controls storage paths and credentials.
• In a self-managed setup you deploy a catalog (for example a Hive Metastore, a JDBC database or a REST catalog service) and run Iceberg's maintenance procedures yourself, because snapshots and old metadata files accumulate until something removes them.
• In open-source Spark, DROP TABLE on an Iceberg table removes only the catalog entry. Deleting the data and metadata files takes DROP TABLE ... PURGE.
• Unity Catalog managed Iceberg tables hand that lifecycle to the catalog: it manages the storage location, runs snapshot expiration and file compaction, vends temporary credentials to external engines and purges dropped tables after a recovery period.
• Self-managed fits teams that need direct control over catalog choice, table properties and maintenance timing. Managed fits teams that share tables across engines and don't want to run the maintenance loop.

Managed Iceberg versus self-managed Iceberg?

Choose managed Iceberg unless you have a specific reason to run the plumbing yourself. The table format is identical either way, down to the metadata tree and data files. What changes is the operating model. Self-managed means you run the catalog service, schedule every maintenance procedure and wire storage credentials into each engine. Managed means a catalog service owns those jobs, and you give up some low-level control in exchange.

What self-managed Iceberg involves

Iceberg is a table format and a library, not a running service. Every table needs a catalog that tracks its current metadata file, and each commit replaces that pointer with an atomic swap. That swap is the basis for Iceberg's serializable isolation and its optimistic concurrency between writers, so the catalog has to stay available and correct. In Spark you register one with org.apache.iceberg.spark.SparkCatalog and a type such as hive, hadoop, rest or jdbc. Whatever sits behind it (a Hive Metastore, a database reached over JDBC, a REST catalog service or a file system path) is yours to deploy, secure and upgrade.

from pyspark.sql import SparkSession

spark = (SparkSession.builder
    .config("spark.sql.catalog.lake", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.lake.type", "rest")
    .config("spark.sql.catalog.lake.uri", "http://localhost:8080")
    .getOrCreate())

The larger commitment is maintenance. Snapshots accumulate until they're expired, and old metadata files are kept by default. The Iceberg docs list three recommended jobs (expire snapshots, remove old metadata files, delete orphan files) plus optional ones such as compacting data files and rewriting manifests. In Spark they run as stored procedures in the catalog's system namespace:

CALL lake.system.expire_snapshots(table => 'db.events', older_than => TIMESTAMP '2026-10-01 00:00:00', retain_last => 10);
CALL lake.system.remove_orphan_files(table => 'db.events', dry_run => true);
CALL lake.system.rewrite_data_files(table => 'db.events');
CALL lake.system.rewrite_manifests('db.events');

Defaults shape the risk. expire_snapshots uses an older_than of five days ago and keeps one ancestor snapshot unless you set retain_last. Orphan-file removal defaults to a three-day interval, and the docs warn that a shorter interval than your longest write can corrupt the table. On Spark 3.x the procedures need the Iceberg SQL extensions; Spark 4.0 supports them natively. Drops need care too: since Iceberg 0.14, Spark's DROP TABLE removes only the catalog entry, and DROP TABLE ... PURGE deletes the contents.

Who owns each job

Responsibility	Self-managed Iceberg	Unity Catalog managed Iceberg
Catalog service	You deploy and operate it	Unity Catalog, which implements the Iceberg REST catalog API
Storage layout	You choose paths and write.* properties	Unity Catalog manages them; properties such as write.data.path can't be set
Snapshot expiration and compaction	You schedule the Spark procedures	Unity Catalog runs them; external clients can't run maintenance operations
Engine credentials	You configure storage access for each engine	Credential vending issues temporary, scoped credentials
DROP TABLE	Removes the catalog entry only, unless you add PURGE	Recoverable with UNDROP TABLE for 7 days by default, then purged
Partitioning	Any Iceberg transform, such as bucket(16, id) or days(ts)	Expression transforms such as bucket() and years() aren't supported; partition evolution works only through external engines on the REST catalog

How Databricks approaches managed Iceberg

In Databricks, Iceberg tables you create in Unity Catalog (with USING iceberg) are managed Iceberg tables. Per the docs, Unity Catalog manages lifecycle tasks like snapshot expiration and file compaction, using serverless compute to maintain table metadata. Predictive optimization, which runs OPTIMIZE, VACUUM and ANALYZE on managed Delta Lake and Iceberg tables, must be enabled before you can create one. The requirements are a Unity Catalog workspace with serverless compute and Databricks Runtime 16.4 LTS or above, and data files must be Parquet.
Other engines use the same tables through the Iceberg REST catalog endpoint, /api/2.1/unity-catalog/iceberg-rest, which gives read and write access to managed Iceberg tables once you enable external data access on the metastore and grant EXTERNAL USE SCHEMA. Unity Catalog blocks external commits that would add duplicate data files to a snapshot. A dropped managed table stays recoverable with UNDROP TABLE for 7 days by default before an asynchronous purge deletes its files; configurable recovery periods are in Public Preview. Tables that stay in another catalog can be read as foreign Iceberg tables, which are read-only in Databricks.

FAQs

Does managed Iceberg change the file format?

No. Iceberg tables in Databricks follow the open Iceberg table format specification, with Parquet data files and spec versions 1, 2 and 3. External clients read and write managed tables through the standard REST catalog API.

Can I run expire_snapshots against a Unity Catalog managed table?

No. External Iceberg clients can't run table maintenance operations, such as expiring snapshots and removing orphan files, on managed tables. Unity Catalog runs that maintenance itself.

How do I run maintenance safely on a self-managed table?

Expire snapshots on a schedule, keep the orphan-file interval longer than your slowest write, and run remove_orphan_files with dry_run => true before deleting anything. If frequent commits pile up metadata files, set write.metadata.delete-after-commit.enabled=true.