What's the best catalog for multi-engine access (Spark, Trino, DuckDB)?

Summary

• For Spark, Trino and DuckDB to share tables, choose a catalog all three can reach through one open protocol. The Apache Iceberg REST catalog API is the interface all three engines document clients for.
• Spark connects through Iceberg's SparkCatalog with type set to rest, Trino through its Iceberg connector with iceberg.catalog.type=rest, and DuckDB by attaching a catalog with TYPE iceberg.
• If you also keep Delta Lake tables, check Delta access per engine. Unity Catalog OSS serves Delta tables to Spark through its connector and to DuckDB through the unity_catalog extension, and implements the Iceberg REST catalog API for engines such as Trino.
• Evaluate a catalog on protocol coverage, table formats, write support in each engine, access control with credential vending, and where the catalog service runs.
• On Databricks, Unity Catalog exposes an Iceberg REST catalog endpoint and the Unity REST API, and vends short-lived credentials that carry the configuring principal's Unity Catalog privileges.

What's the best catalog for multi-engine access (Spark, Trino, DuckDB)?

The catalog that serves Spark, Trino and DuckDB well is one they can all reach through the same open protocol, and the protocol each engine documents a client for is the Apache Iceberg REST catalog API. Spark uses it through the Iceberg Spark runtime, Trino through its Iceberg connector, and DuckDB through its iceberg extension, so any catalog that implements the spec can give the three engines one shared namespace. Unity Catalog, an open-source project hosted by the LF AI & Data Foundation, serves Delta Lake tables directly and implements the Iceberg REST catalog API, through which Iceberg clients read Delta tables that have UniForm enabled. A shared protocol doesn't mean identical features, so verify writes and access control engine by engine.

What does a catalog do for multiple engines?

A catalog maps table names to table metadata. In Iceberg's own words, the most important responsibility of a catalog is tracking a table's current metadata, which the catalog provides when an engine loads a table. When Spark, Trino and DuckDB share one catalog, each resolves the same table name to the same current version, and if the catalog enforces access control, you manage permissions in one place instead of three.

Spark vs Trino vs DuckDB: how each connects to a catalog

Capability	Apache Spark	Trino	DuckDB
Iceberg REST catalog client	Iceberg SparkCatalog with type = rest and a uri	Iceberg connector with iceberg.catalog.type=rest and iceberg.rest-catalog.uri	iceberg extension: ATTACH with TYPE iceberg and an ENDPOINT
Other Iceberg catalog options	Types include hive, hadoop, jdbc and nessie	Types include hive_metastore, jdbc and nessie	iceberg_scan reads a table from its metadata path with no catalog (read-only)
Iceberg writes	INSERT INTO, MERGE INTO, UPDATE and DELETE FROM	INSERT, UPDATE, DELETE, TRUNCATE and MERGE	Read and write access through an attached REST catalog
Delta Lake tables	Delta Lake's Spark integration (DeltaSparkSessionExtension)	Delta Lake connector with read and write access, using a Hive metastore	delta extension reads Delta tables and supports blind appends (INSERT INTO)
Unity Catalog OSS	io.unitycatalog.spark.UCSingleCatalog for creating, reading and writing Delta tables	Iceberg connector pointed at Unity Catalog's Iceberg REST endpoint	unity_catalog extension reads Delta tables and supports INSERT

How to evaluate a catalog for Spark, Trino and DuckDB

• Protocol coverage. Prefer a catalog that implements the Iceberg REST catalog API, since all three engines speak it. Unity Catalog OSS also documents compatibility with Apache Hive's metastore API for Hive-compatible tools.
• Table formats. If your tables are all Iceberg, any catalog that implements the REST spec is a candidate. If you also run Delta Lake, check that the catalog serves Delta tables and that each engine has a Delta path, as the table above shows.
• Write paths. Reads over REST are the common denominator. Writes vary by engine and by table type, so test the exact MERGE, UPDATE or INSERT your pipelines run.
• Access control and credentials. Look for asset-level permissions plus credential vending, which hands engines short-lived storage credentials instead of long-lived keys. Unity Catalog OSS lists authentication, credential vending and asset-level access control.
• Where it runs. An open-source catalog server is yours to deploy and operate. A managed catalog shifts that work to a provider.

How Databricks approaches multi-engine access

On Databricks, Unity Catalog governs Delta Lake and Apache Iceberg tables and opens them to external engines once a metastore admin enables external data access and the connecting principal is granted EXTERNAL USE SCHEMA. Iceberg clients connect to the Iceberg REST catalog at /api/2.1/unity-catalog/iceberg-rest, and Databricks lists Apache Spark, Apache Flink and Trino among its supported clients. Through that endpoint, managed Iceberg tables are readable and writable, and Delta tables with Iceberg reads (UniForm) enabled are readable.
Delta clients use the Unity REST API, which supports direct reads of Delta tables. With credential vending, Unity Catalog grants short-lived credentials that inherit the privileges of the Databricks principal used to configure the integration.

FAQs

Can Spark, Trino and DuckDB share one Iceberg catalog?

Yes. Each documents an Iceberg REST catalog client: Spark's SparkCatalog with type = rest, Trino's iceberg.catalog.type=rest, and DuckDB's ATTACH with TYPE iceberg. Point all three at the same endpoint and they share one set of tables.

Does DuckDB need a catalog to read Iceberg tables?

No. The iceberg extension can read a table directly from its metadata in storage, read-only. Attaching an Iceberg REST catalog adds write support.

Can Trino query Unity Catalog tables?

Yes. Unity Catalog OSS documents pointing Trino's Iceberg connector at its Iceberg REST endpoint to read Delta tables with UniForm enabled, and Databricks lists Trino among the supported clients of its Iceberg REST catalog.

Is Unity Catalog open source?

Yes. Unity Catalog OSS is Apache 2.0 licensed and hosted by the LF AI & Data Foundation, with an OpenAPI specification and support for Apache Iceberg's REST catalog API.