What is Apache Iceberg?

Summary

• Apache Iceberg is, in the project's words, "the open table format for analytic datasets": a high-performance format for huge analytic tables that lets engines work safely on the same tables at the same time.
• A table is a tree of metadata (table metadata file, snapshots, manifest lists, manifests) above data files in Parquet, Avro or ORC. Each commit swaps the metadata file atomically.
• Schema and partition evolution are metadata changes that rewrite no data files, hidden partitioning lets queries filter on source columns, and snapshots enable time travel.
• Spark and Flink connectors live in the Iceberg repository, other engines integrate as third parties, and a catalog (Hive, REST, JDBC, Nessie and others) tracks the tables.
• On Databricks, Unity Catalog supports managed Iceberg tables, foreign Iceberg tables and an Iceberg REST catalog API.

What is Apache Iceberg?
Apache Iceberg is an open table format for analytic datasets. The project site describes it as a high-performance format for huge analytic tables that brings the reliability and simplicity of SQL tables to big data, so multiple engines can safely work with the same tables at the same time.

How is an Iceberg table organized?

The Iceberg table spec describes a tree of metadata above the data files.

Layer	What it holds
Table metadata file	Schema, partitioning config, custom properties and snapshots. Every change creates a new metadata file and replaces the old one with an atomic swap
Snapshot	The state of the table at some time, used to access the complete set of data files
Manifest list	The manifests that make up a snapshot, with metadata such as partition stats and data file counts
Manifest	An immutable Avro file that lists data files or delete files, with each file's partition data tuple, metrics and tracking information
Data files	The table rows, in Parquet, Avro or ORC

What problems does Iceberg solve?

The Iceberg docs point to the Hive table layout. Hive keeps partitions in a central metastore and files in a file system, which makes atomic changes to a table's contents impossible, and job planning needs many slow listing calls. Iceberg avoids file listing and rename operations, so its tables are compatible with any object store. A commit replaces the current metadata file path with an atomic operation, concurrent writers use optimistic concurrency and retry on conflict, and all table changes form a linear history of atomic updates (serializable isolation).

Which capabilities does Iceberg provide?

Capability	What the Iceberg docs say
Schema evolution	Add, drop, rename, update (widen a type) and reorder are metadata changes, so no data files are rewritten
Partition evolution	Data written under an earlier partition spec stays unchanged; new data uses the new spec
Hidden partitioning	Iceberg produces partition values and skips unneeded partitions, so consumers do not add extra filters
Time travel	Query an earlier snapshot, branch or tag

How do I create and query an Iceberg table in Spark?

Spark needs the Iceberg runtime JAR and a catalog. The SQL extensions add features such as stored procedures. This follows the Spark getting-started guide, with a local Hadoop catalog (limits below).

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.local.type=hadoop \
  --conf spark.sql.catalog.local.warehouse=$PWD/warehouse

CREATE TABLE local.db.events (id bigint, data string, ts timestamp)
USING iceberg
PARTITIONED BY (day(ts));

INSERT INTO local.db.events VALUES (1, 'a', TIMESTAMP '2026-01-15 10:30:00');

-- filter on the source column; no partition column is needed
SELECT * FROM local.db.events WHERE ts >= TIMESTAMP '2026-01-15 00:00:00';

ALTER TABLE local.db.events ADD COLUMNS (status string comment 'event status');

-- list snapshots, then read one by ID
SELECT * FROM local.db.events.snapshots;
SELECT * FROM local.db.events VERSION AS OF 10963874102873;

How do engines and catalogs fit together?

Connectors for Spark and Flink are maintained in the main Iceberg repository. The Iceberg site lists other engines, such as DuckDB, as third-party integrations. A catalog tracks the tables. In Spark, the catalog type property accepts values including hive, hadoop, rest, jdbc and nessie.

• REST: an OpenAPI-defined catalog API for managing table metadata and performing catalog operations. It exists so that new languages and engines can support any catalog with one client implementation.
• Hadoop: needs no Hive metastore, but can only be used with HDFS or similar file systems that support atomic rename. Concurrent writes with a Hadoop catalog are not safe with a local file system or S3.
• Nessie: adds multi-table transactions and git-like operations (branches, tags, commits) on top of Iceberg.

How does Databricks support Iceberg?

The Databricks docs list three ways: managed Iceberg tables, foreign Iceberg tables (read-only access to tables in external catalogs through Lakehouse Federation), and the Iceberg REST catalog API. Through the REST endpoint, /api/2.1/unity-catalog/iceberg-rest, external clients such as Apache Spark and Apache Flink can read and write managed Iceberg tables in Unity Catalog. Foreign Iceberg tables and Delta tables with Iceberg reads enabled are read-only there.
Iceberg reads (UniForm) configure Delta Lake tables to automatically generate Iceberg metadata, so Iceberg clients can read Delta Lake data without rewriting files. The table must be registered in Unity Catalog with column mapping enabled, and writes must use Databricks Runtime 14.3 LTS or above.

FAQs

Is Iceberg a file format?

No. Data files are Parquet, Avro or ORC. Iceberg is the table format that tracks them.

How do I bring an existing Hive or Parquet table into Iceberg?

The Spark procedures (they need the SQL extensions and run with CALL) include snapshot, which creates a temporary Iceberg table that shares the source's data files; migrate, which replaces a table with an Iceberg table loaded with the source's data files and keeps the original as table_BACKUP_ by default; and add_files, which adds files from Hive or file-based tables to an existing Iceberg table without moving them.

How do I inspect a table's history?

Query metadata tables with dot notation, such as local.db.events.snapshots or local.db.events.partitions.