Delta Lake vs Apache Iceberg

Summary

• Delta Lake and Apache Iceberg are open table formats that add a metadata layer to data files in object storage to provide ACID transactions, time travel and schema changes.
• Delta Lake tracks table state in a transaction log of JSON files and checkpoints. Iceberg tracks it in metadata files, manifest lists and Avro manifests, and each change swaps in a new metadata file atomically.
• Iceberg documents hidden partitioning, partition evolution, and branching and tagging. Delta Lake documents liquid clustering, RESTORE and shallow clones.
• Databricks supports both formats: Delta Lake is the default table format, Unity Catalog manages Iceberg tables, Iceberg reads (UniForm) let Iceberg clients read Delta Lake tables, and CONVERT TO DELTA converts Iceberg tables.
• Both projects list Apache Spark, Flink, Trino, Presto and Hive among the engines that work with their tables.

How do Delta Lake and Apache Iceberg compare?

Both formats add a metadata layer to data files so that engines can treat files in object storage as tables with transactions and snapshots. They differ in how that metadata is laid out, in a few documented features, and in how each project describes its engine support. Delta Lake keeps a transaction log of JSON files plus checkpoints. Iceberg keeps metadata files that are swapped atomically on every change, with manifest lists and manifests underneath. Databricks supports both formats, so choosing one does not force a separate platform.

What is an open table format?

An open table format defines how a set of data files forms a table. Delta Lake "extends Parquet data files with a file-based transaction log for ACID transactions and scalable metadata handling", according to Databricks. Iceberg "brings the reliability and simplicity of SQL tables to big data", according to the Iceberg project. The Iceberg specification covers Parquet, Avro and ORC data files, and Databricks supports Iceberg tables that use Parquet.

How do the two formats compare feature by feature?

Area	Delta Lake	Apache Iceberg
Project	Open source; the Delta Lake Project is a sub-project of the Linux Foundation Projects (joined in 2019)	An Apache Software Foundation project under the Apache License, Version 2.0
Table state	JSON files in the _delta_log directory, plus checkpoints	Each change creates a new metadata file through an atomic swap, with manifest lists and Avro manifests
Transactions	ACID transactions, with optimistic concurrency control between writes	Serializable isolation based on the atomic swap of table metadata files
Time travel	Time travel to an older snapshot, and RESTORE to an earlier state	Time travel and rollback to a table snapshot
Schema changes	Add and reorder columns; column mapping enables metadata-only rename and drop without rewriting data files	Add, drop, rename, update and reorder columns without rewriting files
Data layout	Hive-style partitioning, or liquid clustering, which lets you redefine clustering columns without rewriting existing data	Hidden partitioning and partition evolution on existing tables
Named versions	Shallow clones for testing a workflow on a production table	Branches and tags, named references to snapshots with their own lifecycles
Engines	Spark, PrestoDB, Flink, Trino and Hive, among others	Spark, Trino, Flink, Presto, Hive and Impala

How does Databricks support Delta Lake and Apache Iceberg?

Delta Lake is the default format for all operations on Databricks, and Databricks originally developed the Delta Lake protocol and continues to contribute to the open source project.
For Iceberg, Databricks supports tables that use Parquet and versions 1, 2 and 3 of the Iceberg specification. Iceberg tables you create in Unity Catalog are managed Iceberg tables. Iceberg tables need a Unity Catalog workspace and Databricks Runtime 16.4 LTS or above, and managed Iceberg tables also need serverless compute. Iceberg tables managed by a catalog outside Unity Catalog are foreign tables, which are read-only in Databricks. Through the Iceberg REST catalog, clients such as Apache Spark, Apache Flink and Trino can read from and write to Unity Catalog-registered Iceberg tables.
Iceberg reads (UniForm) configure a Delta Lake table to generate Iceberg metadata asynchronously, so Iceberg clients can read it without rewriting the Parquet data files. Iceberg client access is read-only. The table must be registered in Unity Catalog, have column mapping enabled, and be written with Databricks Runtime 14.3 LTS or above.

FAQs

Which table format should I choose?

Databricks documents both formats, so choose by the engines, catalogs and features you need. Delta Lake is the Databricks default. Iceberg's documentation covers branching, tagging and partition evolution. With Iceberg reads (UniForm), one copy of a Delta Lake table's data files serves both Delta and Iceberg clients, so reads do not force a single choice.

Can Iceberg clients read Delta Lake tables?

Yes. Delta Universal Format (UniForm) lets Iceberg clients read Delta tables. UniForm generates Iceberg metadata asynchronously, and a single copy of the data files provides access to clients of all formats. On Databricks, Iceberg client access to these tables is read-only.

Can I convert a table from one format to the other?

Yes, in both directions. Delta Lake can convert an Iceberg table in place when its files are Parquet, with no data copy or rewrite, and the Iceberg and Delta tables keep separate histories. Iceberg's snapshot action creates a new Iceberg table from a Delta Lake table and leaves the original Delta table unchanged.

Can Unity Catalog hold both Delta and Iceberg tables?

Yes. Delta Lake is the default table format on Databricks, and Iceberg tables you create in Unity Catalog are managed Iceberg tables. External Iceberg engines can read and write those Iceberg tables through the Iceberg REST catalog, but they cannot read views defined in Unity Catalog.