Are open table formats worth the complexity?

Summary

• Open table formats such as Delta Lake and Apache Iceberg are worth their extra moving parts when several writers, engines or workloads share data, or when you need updates, deletes, schema changes or time travel on files in object storage.
• Both formats add a metadata layer over the data files that commits each change atomically, so readers see a complete table version and never a partial write.
• The cost is operational: a transaction log or catalog tracks table state, and tables need routine maintenance such as compacting small files and removing unreferenced ones.
• A single job appending immutable Parquet files that one engine reads may not need a table format yet.
• On Databricks, Unity Catalog managed tables support both Delta Lake and Apache Iceberg, and predictive optimization runs OPTIMIZE, VACUUM and ANALYZE on them automatically.

Are open table formats worth the complexity?

Open table formats are worth the added complexity when several writers, engines or workloads share the same data, or when you need updates, deletes, schema changes and history on files in object storage. Apache Parquet is a column-oriented data file format: it defines how data is laid out inside a file, not which files make up a table at a given moment. Delta Lake and Apache Iceberg add that table layer and commit every change atomically, so readers never see a half-finished write. If one job appends immutable files that one engine reads, plain Parquet can be enough, and you skip the catalog and maintenance a table format brings.

What does a table format add on top of Parquet?

A table format keeps metadata alongside the data files that records which files belong to the current version, the schema, and the history of changes. The Iceberg spec puts it plainly: table state is maintained in metadata files, and every change creates a new metadata file that replaces the old one with an atomic swap. Delta Lake versions every write, and DESCRIBE HISTORY returns the operation, user and timestamp for each one.

Delta Lake vs Apache Iceberg: what you get and what you maintain

Capability	Delta Lake	Apache Iceberg
Transactions	ACID transactions with serializable isolation; readers never see inconsistent data	Serializable isolation; table changes are atomic and readers never see partial changes
Concurrent writers	Optimistic concurrency control; a conflicting write fails with a concurrent modification exception instead of corrupting the table	Multiple concurrent writers use optimistic concurrency and retry so compatible updates succeed
Row-level changes	MERGE, UPDATE and DELETE	MERGE INTO, UPDATE and DELETE FROM in Spark
Schema changes	Validates writes against the table schema; mergeSchema updates the schema as part of a write	Schema evolution that won't inadvertently un-delete data
History	Data versioning for rollbacks, audit trails and reproducible experiments	Time travel to an exact snapshot, plus version rollback
Streaming	A table is a batch table as well as a streaming source and sink	Spark Structured Streaming reads and writes
Partitioning	Partitioned tables plus Z-ordering to colocate related data	Hidden partitioning and partition layout evolution
Routine maintenance	OPTIMIZE coalesces small files; VACUUM removes unreferenced files older than the retention threshold (7 days by default)	Expire snapshots, remove old metadata files, delete orphan files; compact with rewrite_data_files

When the complexity pays off

• Several jobs or engines share a table. Atomic commits keep each reader on a consistent version while other jobs write.
• Rows change after they land. Change data capture feeds, corrections and deletion requests become MERGE, UPDATE and DELETE statements instead of hand-managed file rewrites.
• Schemas drift. Delta Lake can update a schema as part of a write with mergeSchema, and Iceberg supports schema evolution.
• You need history. Time travel gives reproducible reads and a rollback path after a bad write.

It may not pay off yet for a single writer producing append-only files that one engine reads, with no row-level changes and no audit requirement. There, a table format mostly adds a catalog and maintenance jobs to operate.

The complexity you take on

• Something has to own the current version. Delta Lake records each transaction in the table's _delta_log folder. Iceberg clients connect to a catalog (REST, Hive and JDBC are among the options), whose most important job is tracking each table's current metadata.
• Maintenance jobs. Every write adds files and a new table version, so compaction and cleanup (OPTIMIZE and VACUUM, or snapshot expiry and orphan-file deletion) need a schedule.

How Databricks approaches table formats

Databricks supports both formats in one catalog. Unity Catalog managed tables are the default and recommended table type in Databricks for Delta Lake and Apache Iceberg, and Unity Catalog manages their read, write, storage and optimization responsibilities. Predictive optimization runs OPTIMIZE, VACUUM and ANALYZE automatically on Unity Catalog managed tables in both formats, which removes the manual maintenance described above.
For interoperability, Iceberg reads (UniForm) generates Iceberg metadata for Delta tables without rewriting the Parquet data files, so a single copy of the data serves both Delta and Iceberg clients. External engines reach Iceberg tables in Unity Catalog through the Iceberg REST Catalog API.

FAQs

Is plain Parquet ever enough?

Yes, for a single writer appending immutable files that one engine reads, with no updates, deletes or audit needs. Add a table format when concurrent writes, row-level changes, schema evolution or history enter the picture.

What maintenance does a table format need?

Compaction of small files and cleanup of files that old versions no longer reference: OPTIMIZE and VACUUM in Delta Lake, or snapshot expiry, orphan-file deletion and rewrite_data_files in Iceberg.

Do I have to choose between Delta Lake and Apache Iceberg?

Not on Databricks. Unity Catalog managed tables support both formats, and Iceberg clients can read Delta tables through Iceberg reads (UniForm).