Skip to main content

Unity Catalogs vs Open Source Unity Catalog

Summary

  • Unity Catalog on Databricks is the managed governance layer built into the Databricks platform. Unity Catalog OSS is the Apache 2.0 open-source catalog server that Databricks open sourced in 2024, hosted by the LF AI & Data Foundation.
  • The two share an API heritage, and Databricks says its Unity Catalog deployments implement the same open APIs. The OSS project's own docs say the Databricks service has a larger feature set.
  • Unity Catalog OSS is a server you run yourself, with one metastore per server and users as the only principals. Unity Catalog on Databricks adds account-level groups and service principals, regional metastores shared across workspaces, and attribute-based policies.
  • Pick Unity Catalog OSS to self-host an open catalog for Delta Lake, Iceberg and AI assets. Pick Unity Catalog on Databricks when you need governance, lineage, auditing and AI governance across a production platform.

Unity Catalogs vs Open Source Unity Catalog

Unity Catalog on Databricks and Unity Catalog OSS are two implementations of one catalog design. Unity Catalog on Databricks is a managed service that runs beneath every data and AI interaction in a Databricks workspace. Unity Catalog OSS is an open-source server that you deploy and operate, governing tables, volumes, functions and models for whatever engines connect to it. The OSS docs put it plainly: the Databricks service has "the same API heritage and a larger feature set", so a Databricks guide isn't automatically a guide for the open source server.

What is Unity Catalog OSS?

Unity Catalog OSS keeps track of which tables, volumes, functions and models exist, where their data lives, and who may use them. Engines such as Apache Spark, DuckDB and Trino ask it those questions over open REST APIs. It's Apache 2.0 licensed, ships an OpenAPI specification, and the project README describes it as a sandbox project of the LF AI & Data Foundation. Databricks created Unity Catalog and open sourced it in June 2024.

Unity Catalog on Databricks vs Unity Catalog OSS: how they compare

Capability Unity Catalog on Databricks Unity Catalog OSS
Delivery Built into Databricks; enabled automatically for workspaces created after November 8, 2023 Server you deploy, for example with Docker Compose, with PostgreSQL as the metadata database
Metastores One per region, linked to any number of workspaces in that region Exactly one per server
Principals Users, groups and service principals, defined at the account level Users identified by email; no groups or service principals
Privilege model Hierarchical grants including BROWSE, MANAGE and ALL PRIVILEGES, plus attribute-based policies with row filters and column masks Ownership plus privileges such as USE CATALOG, SELECT and MODIFY; no ALL PRIVILEGES, MANAGE or BROWSE
Governed assets Tables, views, volumes, functions, models, and model and MCP services Tables, views and metric views, volumes, functions, registered models
Table formats Managed Delta Lake and Apache Iceberg tables, plus external tables Catalog-managed Delta tables, managed Iceberg tables, plus external tables in other formats
Engine APIs Unity REST API, Iceberg REST catalog, OpenSharing Unity Catalog REST API, Delta API, Iceberg REST API
Storage access Storage credentials and external locations, with credential vending Storage credentials (AWS IAM roles) and external locations, with credential vending
Authorization switch Enforced whenever Unity Catalog is enabled for a workspace Checked only when server.authorization=enable is set

The principal row matters most in practice. In the OSS server, every grant names an individual user. On Databricks, best practice is to grant access to account-level groups synced from your identity provider and to run production jobs as service principals.

When to choose each

Choose Unity Catalog OSS when you want an open catalog you can run anywhere: on a laptop for development, inside your own infrastructure, or as a shared catalog for engines like Spark and DuckDB. You own the deployment, upgrades and security hardening. The OSS docs warn that anyone holding direct storage credentials can bypass the catalog's checks, so locking down storage is part of a secure setup.
Choose Unity Catalog on Databricks when governance has to hold up in production across teams, workspaces and AI workloads. We believe it's the stronger choice for enterprises because the governance features come built in and are operated for you.

How Databricks approaches Unity Catalog

On Databricks, Unity Catalog enforces access control when you query a table or call a model, tracks lineage as assets are used, and logs activity for auditing. Beyond grants, it adds attribute-based access control with row filter and column mask policies, automated data classification, data quality monitoring, column-level lineage across tables, notebooks, jobs and pipelines, and AI governance through Unity Gateway for LLM endpoints, agents and MCP servers.
Openness carries over. When Databricks open sourced Unity Catalog, it noted that existing Databricks deployments implement the same open APIs. Today external engines reach Unity Catalog on Databricks through the Unity REST API for Delta clients and the Iceberg REST catalog for Iceberg clients, with credential vending handing out short-lived credentials scoped to the configuring principal's privileges.

FAQs

Is Unity Catalog OSS the same code as Unity Catalog on Databricks?

They share an API heritage, not a feature set. The OSS docs describe the Databricks service as having a larger feature set and say they cover only the open source server.

Can I move from Unity Catalog OSS to Databricks later?

Databricks says its Unity Catalog deployments implement the same open APIs, so clients written against them are a good starting point. The OSS docs call the Databricks service a larger feature set, so check each feature you rely on, and plan how metadata and grants will carry over.

Does Unity Catalog OSS support Apache Iceberg?

Yes. It implements the Iceberg REST catalog API, and its current docs show Iceberg clients creating, writing, reading, renaming and dropping managed Iceberg tables through it.

Who governs the Unity Catalog open source project?

It's hosted by the LF AI & Data Foundation, part of the Linux Foundation, and released under the Apache 2.0 license.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.