How do I use the Iceberg REST Catalog with Spark?

Summary

• An Iceberg REST catalog is a service that implements the Iceberg REST Catalog API specification, so catalog logic lives behind an HTTP API instead of in a catalog-specific client for every engine.
• In Spark, register it as a named catalog: set spark.sql.catalog.<name> to org.apache.iceberg.spark.SparkCatalog, set type to rest, and set uri to the endpoint. Add the iceberg-spark-runtime package that matches your Spark and Scala versions.
• Authentication is also set per catalog, with rest.auth.type and properties such as credential, token, oauth2-server-uri and scope.
• On Databricks, Unity Catalog serves an Iceberg REST endpoint at /api/2.1/unity-catalog/iceberg-rest, and warehouse is set to the Unity Catalog catalog name.

How do I use the Iceberg REST Catalog with Spark?

An Iceberg REST catalog is any catalog service that implements the Iceberg REST Catalog API specification. The catalog logic lives behind an HTTP API, so engines do not need a catalog-specific client each. Spark reaches it through Iceberg's SparkCatalog class with type set to rest.

What do I need before I start?

• The Iceberg Spark runtime. Pick the iceberg-spark-runtime artifact that matches your Spark and Scala versions. The Iceberg docs show org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0, added with --packages.
• A server that implements the spec. The spec page mentions the community-maintained iceberg-rest-fixture Docker image. Hive Metastore offers Iceberg REST API endpoints, Unity Catalog OSS implements the Iceberg REST Catalog APIs, and Unity Catalog on Databricks serves one too.
• Credentials, if the server requires authentication.

How do I configure the catalog?

In the Iceberg docs, a catalog is created and named by adding a property spark.sql.catalog.(catalog-name) with an implementation class for its value. The type property picks the underlying Iceberg catalog implementation, and rest selects RESTCatalog. The docs' REST example names the catalog rest_prod:

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.12.0 \
    --conf spark.sql.catalog.rest_prod=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.rest_prod.type=rest \
    --conf spark.sql.catalog.rest_prod.uri=http://localhost:8080

Set these properties after the spark.sql.catalog.<name>. prefix:

Property	Meaning
type	rest selects RESTCatalog; other values are hive, hadoop, glue, jdbc and nessie
uri	The REST URL for a REST typed catalog
warehouse	Base path for the warehouse directory; the REST example passes an identifier such as my_warehouse
default-namespace	The default current namespace for the catalog
cache-enabled	Whether to enable the catalog cache; the default is true
cache.expiration-interval-ms	When cached entries expire; the default is 30000 (30 seconds), -1 disables expiration and 0 disables caching
table-default.<key>, table-override.<key>	Default or enforced table property values

How do I authenticate?

The Iceberg REST docs define rest.auth.type (default none; supported values are none, basic, oauth2, sigv4 and google) and these OAuth2 properties:

• credential: client_id:client_secret, exchanged for a token in the OAuth2 client credentials flow.
• token: a Bearer token.
• oauth2-server-uri: the token endpoint, default v1/oauth/tokens, required if the REST catalog is not the OAuth2 authentication server.
• scope: additional scope, default catalog.

The docs warn that credential and token are secrets, and that Spark lists catalog configuration in the Environment tab and writes it to event logs.

How do I create and query tables?

Use the catalog name as the first part of the table name:

CREATE TABLE rest_prod.db.sample (id bigint, data string) USING iceberg;
INSERT INTO rest_prod.db.sample VALUES (1, 'a'), (2, 'b'), (3, 'c');
SELECT count(1) AS count, data FROM rest_prod.db.sample GROUP BY data;
ALTER TABLE rest_prod.db.sample ADD COLUMNS (new_column string);
SELECT * FROM rest_prod.db.sample VERSION AS OF 10963874102873;
SELECT * FROM rest_prod.db.sample TIMESTAMP AS OF '1986-10-26 01:21:00';

VERSION AS OF accepts a long snapshot ID, a branch name or a tag name.

How does it work with Unity Catalog on Databricks?

Unity Catalog exposes an Iceberg REST endpoint at https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest, and Databricks lists Apache Spark and Apache Flink among the supported clients. Managed Iceberg tables can be read and written. Foreign Iceberg tables, and Delta tables with Iceberg reads (UniForm) enabled, can be read only.
The requirements are Unity Catalog, external data access enabled on the metastore, the EXTERNAL USE SCHEMA privilege on the schemas, and a personal access token or OAuth. The Spark settings differ from the open-source example in a few places:

• uri is the endpoint above, and warehouse is the Unity Catalog catalog name.
• With a personal access token, set token.
• With OAuth, set rest.auth.type to oauth2, oauth2-server-uri to https://<workspace-url>/oidc/v1/token, credential to <oauth_client_id>:<oauth_client_secret> and scope to all-apis.
• The Databricks example adds org.apache.iceberg:iceberg-aws-bundle:<iceberg-version> next to the runtime package.

FAQs

Can I use a REST catalog and a Hive catalog in one session?

Yes. Every catalog gets its own name. The Iceberg getting-started command registers spark_catalog as a SparkSessionCatalog of type hive next to a local SparkCatalog of type hadoop, and tables are addressed as catalog.database.table.

How long does Spark cache catalog entries?

The catalog cache is on by default, and entries expire after 30000 milliseconds unless you change cache.expiration-interval-ms.

Do I need the Iceberg SQL extensions?

The Iceberg docs set spark.sql.extensions to org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions in their getting-started command. The extension module adds SQL commands such as CALL for stored procedures.