Best open-source data catalog 2026?

Summary

• No single open-source project is the best data catalog, because "catalog" covers different jobs. The Hive metastore, Apache Polaris and Unity Catalog OSS record which tables exist and serve their metadata to engines, and Apache Atlas provides governance and lineage services for Hadoop.
• Apache Iceberg and Apache Hudi are table formats, not catalogs. Iceberg tables are loaded through a catalog, and Iceberg's Spark integration supports catalog types including Hive, Hadoop, REST, JDBC and Nessie.
• Unity Catalog OSS is Apache 2.0 licensed, hosted by the LF AI & Data Foundation, and compatible with the Hive metastore API and the Iceberg REST catalog API.
• On Databricks, Unity Catalog is built in as the governance layer for data and AI, and it implements the Iceberg REST catalog API.

Best open-source data catalog 2026?

No single open-source project is the best data catalog, because "catalog" covers several jobs. The Hive metastore, Apache Polaris and Unity Catalog OSS record which tables exist and serve their metadata to engines. Apache Atlas covers governance and lineage in Hadoop environments. Apache Iceberg and Apache Hudi are table formats, one layer below a catalog. This page covers projects used with Spark and Iceberg stacks, not every open-source metadata tool. Decide which job you need done first, then check that your engines support your pick.

What is the difference between a table format and a data catalog?

A table format defines how a table's data files and metadata are organized so engines can read and write the table safely.
A catalog sits above the format. In Iceberg's architecture it is the top-level layer that returns the current metadata when a table is loaded and handles creating, dropping and renaming tables. Iceberg's Spark integration supports catalog types including Hive, Hadoop, REST, JDBC and Nessie, so the format and the catalog are separate choices.

Which open-source projects are catalogs, and which are table formats?

Project	Role	What the project documents
Apache Iceberg	Table format	Schema evolution, hidden partitioning, time travel and version rollback. Tables are loaded through a catalog.
Apache Hudi	Table format	An open table format built for high-performance writes on incremental pipelines, with upserts and deletes.
Hive metastore	Catalog	A central repository of metadata that engines use to parse, authorize and execute queries. Since Hive 3.0 it can run without the rest of Hive installed.
Apache Polaris	Catalog	An open-source catalog for Apache Iceberg that implements Iceberg's REST API.
Unity Catalog OSS	Catalog	Covers tables, files, functions and AI models. Apache 2.0 licensed, and compatible with the Hive metastore API and the Iceberg REST catalog API.
Apache Atlas	Governance and lineage	Governance services for Hadoop, with lineage enriched by business taxonomical metadata. Uses Apache Ranger to block unauthorized access paths to data at run time.

When should I use Iceberg and when Hudi?

Both are open table formats that support updates and deletes, so the choice follows each project's emphasis. Iceberg's documentation centers on engine-neutral tables: an open specification for compatibility across languages and implementations, with Spark, Flink and Hive connectors maintained in the main repository. Hudi's documentation centers on incremental pipelines: Copy On Write tables for read-heavy workloads, and Merge On Read tables that balance write and read performance by combining log files with base files. Both support TIMESTAMP AS OF time travel in Spark SQL.
Configure a REST catalog in Spark, then create and query an Iceberg table:

spark.sql.catalog.rest_prod = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.rest_prod.type = rest
spark.sql.catalog.rest_prod.uri = http://localhost:8080

CREATE TABLE rest_prod.db.sample (
  id bigint NOT NULL COMMENT 'unique id',
  data string)
USING iceberg;

SELECT * FROM rest_prod.db.sample TIMESTAMP AS OF '2026-01-15 00:00:00';

What does Unity Catalog OSS add?

Unity Catalog OSS manages tables, files, functions and AI models. Its server, OpenAPI specification and clients are Apache 2.0 licensed, and the project describes authentication, credential vending and asset-level access control as built in. Per its README, it is a sandbox project hosted by the LF AI & Data Foundation (part of the Linux Foundation, not the Apache Software Foundation), and its APIs "are currently evolving and should not be assumed to be stable."

How does Databricks approach open-source catalogs?

Unity Catalog is built into Databricks as the unified governance layer for data and AI. It enforces access control, tracks lineage, logs activity for auditing and supports discovery through Catalog Explorer. Unity Catalog is also available as the open-source implementation described above, and the Databricks docs call the older Hive metastore "legacy."
Databricks supports Delta Lake, an open-source project that no single company controls, and Apache Iceberg tables in Unity Catalog. Unity Catalog implements the Iceberg REST catalog API, so clients such as Apache Spark and Apache Flink can read and write Iceberg tables registered in it. Iceberg reads (UniForm) generate Iceberg metadata for Delta Lake tables, so Iceberg clients can read them without rewriting files.

FAQs

Can I move an Iceberg table to a different catalog?

Yes. Iceberg's register_table procedure creates a catalog entry for an existing metadata.json file. The docs warn that registering the same file in more than one catalog can lead to missing updates, loss of data and table corruption, so use it only when the table is no longer registered elsewhere or you are moving it between catalogs.

Do I need a separate catalog if I only use Spark SQL?

Spark can create a local one. With Hive support enabled and no hive-site.xml, it creates metastore_db in the current directory and a warehouse directory set by spark.sql.warehouse.dir. Applications that must share table definitions need a shared metastore, configured through hive-site.xml.

Is Iceberg a replacement for a data catalog?

No. Iceberg defines the table format, and its tables are loaded through a catalog. Access control and lineage come from the catalog or from a governance project such as Unity Catalog OSS or Apache Atlas.