What is the Iceberg REST Catalog (IRC)?

Summary

• An Iceberg REST catalog is any catalog service that implements the Iceberg REST Catalog API specification, so one client implementation works with any compliant server.
• Commits are change-based. A client sends requirements and updates. The server validates the requirements, applies the updates and writes the new metadata.
• Servers can delegate storage access with short-lived, table-scoped credentials (credential vending) or remote signing.
• Unity Catalog on Databricks serves the API at /api/2.1/unity-catalog/iceberg-rest.

What is the Iceberg REST Catalog (IRC)?

An Iceberg REST catalog is any catalog service that implements the Iceberg REST Catalog API specification. Instead of requiring a catalog-specific client in every engine and language, the catalog logic lives behind an HTTP API, and a single client implementation works with any compliant server. Apache Iceberg publishes the API as an OpenAPI document and describes the protocol as analogous to Hive's thrift protocol for the Hive Metastore (HMS).

What problem does the REST catalog protocol solve?

As the Iceberg project grew to support more languages and engines, pluggable catalogs caused practical problems. Catalogs had to be implemented in multiple languages, and it was difficult for commercial offerings to support many different catalogs and clients. The Iceberg docs give five reasons the REST protocol matters:

Concept	What it is	Why it matters
Language and engine compatibility	One client implementation per language or engine	Works with any compliant catalog
Change-based commits	Requirements plus updates	Server-side conflict detection and retries
Catalog-written metadata	The catalog service writes the root metadata	Easier metadata version upgrades
Advanced features	Lazy snapshot loading, multi-table commits, caching	Lighter table loads, optional atomic multi-table commits
Security	Credential vending or remote signing	Secure table sharing

How does a client talk to a REST catalog?

1. Discovery. The client calls GET /v1/config. The response can carry defaults (suggested properties), overrides (required properties) and endpoints (what the server supports). A common override is prefix, inserted into later paths (/v1/{prefix}/namespaces/...) so one server can host multiple catalogs.
2. Load. A loaded table returns its metadata JSON in metadata and its file location in metadata-location, plus optional config and storage-credentials.
3. Commit. A REST client never writes a metadata file and swaps a pointer itself. It sends POST /v1/{prefix}/namespaces/{namespace}/tables/{table} with requirements (what must still be true of the table's current metadata) and updates (the changes). If every requirement holds, the server applies the updates and writes the new metadata. If one fails, the commit is rejected (HTTP 409, CommitFailedException), and the client refreshes the table, reapplies its changes and retries.
4. Storage access. The optional X-Iceberg-Access-Delegation header lists the mechanisms the client supports, vended-credentials and remote-signing. The server may supply either, both or none. Credential vending returns short-lived, table-scoped credentials. With remote signing, the client sends each storage request to the catalog's signing endpoint and the server returns the signed request.

What does a create-table request look like?

The create route is POST /v1/{prefix}/namespaces/{namespace}/tables, and prefix is optional. The body requires name and schema, and field names use hyphens, such as partition-spec and source-id:

{
  "name": "events",
  "schema": {
    "type": "struct",
    "fields": [
      {"id": 1, "name": "event_id", "required": true, "type": "long"},
      {"id": 2, "name": "event_ts", "required": true, "type": "timestamp"}
    ]
  },
  "partition-spec": {
    "fields": [
      {"source-id": 2, "name": "event_ts_month", "transform": "month"}
    ]
  }
}

How does a REST catalog differ from a metastore-backed catalog?

In the Iceberg table spec, a metastore-backed catalog keeps a pointer to the current metadata file. The writer creates a new metadata file, then asks the metastore to swap the pointer with a check-and-put operation. With the REST protocol, the server writes the new metadata after validating the client's requirements. The two can coexist: Apache Hive documents that the Hive Metastore offers Iceberg REST API endpoints alongside its Thrift API (HiveCatalog).

How does Databricks approach the Iceberg REST catalog?

Unity Catalog implements the Iceberg REST catalog API specification at /api/2.1/unity-catalog/iceberg-rest, and Databricks documents clients such as Apache Spark and Apache Flink. Enable external data access on the metastore, grant the principal EXTERNAL USE SCHEMA on the schema, and authenticate with a personal access token or OAuth. Databricks recommends credential vending to control cloud storage access. The older read-only endpoint, /api/2.1/unity-catalog/iceberg, is documented as legacy. Unity Catalog OSS also serves an Iceberg REST catalog, documented for Delta tables with UniForm enabled.

Table type	Read	Write
Managed Iceberg	Yes	Yes
Foreign Iceberg	Yes	No
Managed Delta (Iceberg reads enabled)	Yes	No
External Delta (Iceberg reads enabled)	Yes	No

The Databricks docs show this Spark setup for personal access token authentication:

pyspark \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:<iceberg-version>,org.apache.iceberg:iceberg-aws-bundle:<iceberg-version> \
  --conf "spark.sql.catalog.<spark-catalog-name>=org.apache.iceberg.spark.SparkCatalog" \
  --conf "spark.sql.catalog.<spark-catalog-name>.type=rest" \
  --conf "spark.sql.catalog.<spark-catalog-name>.uri=https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest" \
  --conf "spark.sql.catalog.<spark-catalog-name>.token=<token>" \
  --conf "spark.sql.catalog.<spark-catalog-name>.warehouse=<uc-catalog-name>"

External Iceberg clients can't access views or run table maintenance such as expiring snapshots and removing orphan files.

FAQs

Is the Iceberg REST catalog the same as the REST catalog protocol?

The protocol is the specification, an OpenAPI document. An Iceberg REST catalog is any catalog service that implements it.

Can a REST catalog serve Delta Lake tables?

The load-table response carries Iceberg table metadata, so a Delta table must expose Iceberg metadata. On Databricks, managed and external Delta tables with Iceberg reads (UniForm) enabled are readable through the endpoint, read-only.

How do vended credentials stay fresh?

They expire. The spec defines a credentials endpoint, GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/credentials, for fetching fresh ones. In Unity Catalog the default expiration time is one hour, and expires-at-ms shows when they expire.