Best catalog for a multi-engine lakehouse

Summary

• No single catalog is best for every lakehouse. For several engines sharing Iceberg tables, pick a catalog that implements the Apache Iceberg REST catalog protocol, a common OpenAPI-based API that lets a new engine support any compatible catalog with one client implementation.
• Apache Polaris and Unity Catalog OSS are open-source catalogs that implement the protocol. Polaris targets Iceberg tables. Unity Catalog OSS lists Delta Lake, Iceberg via UniForm, Parquet, JSON and CSV, and serves UniForm Delta tables to Iceberg clients through its Iceberg REST endpoint.
• Compare catalogs on table formats, engine clients, read versus write access, credential vending and access control.
• Delta Lake and Iceberg each consist of Parquet data files and a metadata layer. They are not catalogs, and catalog support for each varies.
• On Databricks, Unity Catalog implements the Iceberg REST catalog API, with read and write access for managed Iceberg tables and read-only access for Delta tables with Iceberg reads enabled.

Best catalog for a multi-engine lakehouse

What is the best catalog for a multi-engine lakehouse?

There is no single best catalog, but for several engines the Iceberg docs point to the REST catalog protocol. The community created it as "a common API (using the OpenAPI spec) for interacting with any Iceberg catalog." In the spec's words, "New languages and engines can support any catalog with just one client implementation." An Iceberg catalog stores each table's current metadata pointer and performs atomic operations to update it.

Why does the REST protocol suit multiple engines?

The Iceberg REST catalog spec lists these benefits:

• Language and engine compatibility. One client implementation is enough for a new engine to support any catalog.
• Improved reliability. Change-based commits enable server-side deconfliction and retries.
• Advanced features. Lazy snapshot loading, multi-table commits and caching.
• Security. Secure table sharing using credential vending or remote signing.

How do I point Spark, Flink and DuckDB at one catalog?

Each engine needs a catalog entry that targets the REST endpoint. These examples come from the Iceberg and DuckDB docs.

# Spark
spark.sql.catalog.rest_prod = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.rest_prod.type = rest
spark.sql.catalog.rest_prod.uri = http://localhost:8080

-- Flink
CREATE CATALOG rest_catalog WITH (
  'type'='iceberg',
  'catalog-type'='rest',
  'uri'='https://localhost/'
);

-- DuckDB
CREATE SECRET iceberg_secret (TYPE ICEBERG, TOKEN 'bearer_token');
ATTACH 'warehouse' AS iceberg_catalog (
  TYPE ICEBERG,
  SECRET iceberg_secret,
  ENDPOINT 'https://rest_endpoint.com'
);

In DuckDB, attaching an Iceberg REST catalog gives access to the full feature set, including writing.

Which open-source catalogs can I use?

Catalog	What its docs say	Table formats
Apache Polaris	Open-source catalog that implements Iceberg's REST API; role-based access control for catalogs, namespaces and tables; vends temporary storage credentials	Iceberg; other formats such as Delta through a Generic Table API that the docs mark beta
Unity Catalog OSS	Apache 2.0; compatible with the Hive metastore API and the Iceberg REST catalog API; asset-level access control and credential vending; a sandbox-stage LF AI & Data project	Delta Lake, Iceberg and Hudi via UniForm, Parquet, JSON, CSV
Iceberg's own catalogs	The Spark type property accepts values including hive, hadoop, rest, jdbc and nessie; CatalogHandlers or the community iceberg-rest-fixture image expose them over REST	Iceberg

What should I evaluate?

Question	What the docs say
Which table formats must it serve?	Polaris targets Iceberg tables, and its beta Generic Table API does not support credential vending. Iceberg's Hive-based catalog only loads Iceberg tables.
Which engines write, and which only read?	The delta.io integrations page describes the Flink connector as writing to Delta Lake, the Hive connector as reading from it, and the Spark connector as reading and writing.
How are storage credentials handled?	Polaris vends temporary storage credentials, and the REST spec lists credential vending or remote signing.
Which engine versions do you run?	Iceberg provides a runtime connector jar for each supported Spark, Flink and Hive version. Parity across maintained versions is expected for most features, not for those that need newer engine functionality.

How does Databricks approach multi-engine catalogs?

Unity Catalog on Databricks provides an implementation of the Iceberg REST catalog API at /api/2.1/unity-catalog/iceberg-rest. Clients such as Apache Spark and Apache Flink can read from and write to Unity Catalog-registered Iceberg tables. Managed Iceberg tables are readable and writable. Foreign Iceberg tables, and managed or external Delta tables with Iceberg reads enabled, are read-only.
To connect, enable external data access on the metastore, grant the principal the EXTERNAL USE SCHEMA privilege on the schema, and authenticate with a personal access token or OAuth. Add the Iceberg Spark runtime JAR and the cloud-specific bundle JAR to your Spark packages, then configure the catalog:

spark.sql.catalog.<spark-catalog-name>=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.<spark-catalog-name>.type=rest
spark.sql.catalog.<spark-catalog-name>.uri=https://<workspace-url>/api/2.1/unity-catalog/iceberg-rest
spark.sql.catalog.<spark-catalog-name>.token=<token>
spark.sql.catalog.<spark-catalog-name>.warehouse=<uc-catalog-name>

Iceberg reads (UniForm) configure Delta Lake tables to generate Iceberg metadata automatically, so Iceberg clients read them without rewriting files. The table must be registered in Unity Catalog and have column mapping enabled. Two limits to plan for: branching and tagging are not supported on Iceberg tables in Databricks, and external Iceberg engines cannot read views.

FAQs

Can Delta Lake and Iceberg tables share one catalog?

It depends on the catalog. Unity Catalog OSS lists Delta Lake and Iceberg via UniForm among its formats, and UniForm tables are Delta tables with both Delta Lake and Iceberg metadata. Polaris registers non-Iceberg tables such as Delta through a Generic Table API that its docs mark beta.

Do I need a managed service to run an Iceberg REST catalog?

No. Apache Polaris and Unity Catalog OSS are open source, and the Iceberg docs also point to a community-maintained iceberg-rest-fixture docker image.

How does Iceberg handle concurrent writers from different engines?

Iceberg uses optimistic concurrency. A writer commits by atomically swapping the table metadata file. If another writer committed first, the failed writer retries on the new table state, and the commit fails if its assumptions no longer hold.