what is a lakehouse table format

Summary

• A lakehouse table format is an open specification for storing a table on object storage as data files plus a metadata layer, so engines can read and write it as one transactional table. Delta Lake, Apache Iceberg and Apache Hudi are open-source examples.
• The data stays in open file formats such as Parquet. The format adds the metadata, such as Delta's transaction log or Iceberg's metadata files and manifests.
• Commits are atomic, so readers see a consistent snapshot while writes happen, and conflicting writers fail or retry instead of corrupting the table.
• That metadata enables ACID transactions, schema enforcement, time travel and row-level updates on a data lake. Existing Parquet data converts in place with CONVERT TO DELTA or Iceberg's migrate procedure.

what is a lakehouse table format

A lakehouse table format is an open specification for storing a table on object storage as data files plus a metadata layer, so that engines such as Apache Spark, Apache Flink and Apache Hive can treat a set of files as one transactional table. The metadata records which files make up each version of the table, along with its schema and partitioning. That record is what makes ACID transactions, schema enforcement, time travel and row-level updates possible on a data lake. Delta Lake, Apache Iceberg and Apache Hudi are open-source formats in this category.

How does a table format work?

Each format keeps data in open files and tracks table state in metadata:

• Delta Lake stores data as versioned Parquet files plus a transaction log in the table's _delta_log directory. Writers optimistically write new data files, then commit by adding a log entry that records which files were logically added and removed. A table has a single serial history of atomic versions.
• Apache Iceberg tracks individual data files rather than directories. A table metadata file holds the schema, partition config and snapshots. Each snapshot's manifests, listed in a manifest list, record every data file with its partition data and metrics. Every change writes a new metadata file and replaces the old one with an atomic swap.
• Apache Hudi describes itself as an open data lakehouse platform built on an open table format, with tables, transactions, efficient upserts and deletes, and indexes.

Readers pick files from this metadata, not from a directory listing, so a reader keeps a consistent snapshot even while another job commits.

What does a table format add to plain Parquet files?

Capability	What it is	Why it matters
ACID transactions	Delta's serializable isolation means readers never see inconsistent data; Iceberg's changes are atomic, so readers never see partial or uncommitted changes	A crashed or concurrent job can't leave a half-written table
Optimistic concurrency	Writers check for conflicts at commit. A conflicting Delta write fails with an exception; Iceberg writers retry so compatible updates succeed	Delta's docs note that the same conflict would corrupt a plain Parquet table
Schema enforcement and evolution	Delta raises an exception for writes with unknown columns or mismatched types, and adds new columns when schema evolution is enabled (for example the mergeSchema option); Iceberg schema evolution supports add, drop, update and rename	Bad records stay out, and schemas can change without rewriting data
Time travel and rollback	Query an older version with VERSION AS OF or TIMESTAMP AS OF, or restore one	Reproducible queries, audits and recovery from bad writes
File pruning	Delta keeps per-file statistics in the log; Iceberg prunes data files with partition and column-level stats and supports hidden partitioning	Engines skip files using metadata instead of scanning everything
Batch and streaming on one table	A Delta table is a batch table as well as a streaming source and sink	One copy of the data serves both kinds of pipeline

How do you convert existing Parquet data?

Both formats convert in place. Delta Lake's CONVERT TO DELTA lists the files in a Parquet directory and creates a transaction log that tracks them. Iceberg's Spark procedures include snapshot, which creates a lightweight test copy without changing the source table, and migrate, which replaces an existing table with an Iceberg table:

-- Delta Lake: convert, then read an older version
CONVERT TO DELTA parquet.`<path-to-table>`;
SELECT * FROM table_name VERSION AS OF version;

-- Apache Iceberg (Spark procedures)
CALL catalog_name.system.snapshot('db.sample', 'db.snap');
CALL catalog_name.system.migrate('db.sample');
SELECT * FROM prod.db.table TIMESTAMP AS OF '1986-10-26 01:21:00';

How Databricks approaches lakehouse table formats

Databricks originally developed the Delta Lake protocol and still contributes to the project, which is independent and part of the Linux Foundation. On Databricks, Delta Lake is the default format: unless you specify otherwise, every table is a Delta Lake table. Unity Catalog also manages Apache Iceberg tables that use Parquet and versions 1, 2 and 3 of the Iceberg spec, and every Iceberg table in Unity Catalog is reachable through the Iceberg REST Catalog API for reads and writes from external Iceberg engines. For Delta tables that Iceberg clients need to read, External Iceberg Reads generates Iceberg metadata alongside the Delta metadata, so a single copy of the Parquet files serves both.

FAQs

Is a table format the same as a file format?

No. Parquet, ORC and Avro are file formats that store the rows and columns. A table format sits above them and records which files make up each version of the table, plus its schema, partitioning and history.

Does a table format tie my data to one engine?

The formats are open specifications. Iceberg is designed as an open community standard with a specification for compatibility across languages and implementations, and Delta Lake has connectors for engines such as Apache Spark, Apache Flink and Apache Hive. The files stay in your own object storage.

How do I undo a bad write?

In Delta Lake, run RESTORE TABLE db.target_table TO VERSION AS OF <version>. In Iceberg, the rollback_to_snapshot Spark procedure resets the table to an earlier snapshot.