Common Unity Catalog migration mistakes to avoid

Summary

• Most Unity Catalog migrations stumble on process, not technology: skipping assessment, copying legacy permissions verbatim, and cutting over everything at once are the mistakes that cause outages and ungoverned data.
• Assess before you move. UCX, the open-source Unity Catalog migration toolkit, inventories your workspace and maps every table, external location, and job so nothing is a surprise mid-cutover.
• Fix identity first: provision account-level groups with SCIM and SSO, grant to groups instead of individual users, and don't run workspace-level and account-level SCIM in parallel, which creates duplicate groups and inconsistent policy.
• Prepare storage up front: create storage credentials and external locations before migrating external tables, default to managed tables, and remember that dropping an external table never deletes the underlying data.
• Migrate in waves: update workloads to the three-level namespace (catalog.schema.table), test downstream dependencies, and only then decommission the legacy Hive metastore.

Common Unity Catalog migration mistakes to avoid

Most Unity Catalog migrations run into trouble for process reasons, not technical ones. The expensive mistakes are predictable: moving tables without a full inventory, recreating legacy Hive metastore permissions instead of redesigning them for groups, and flipping every workload at once. Treat the move as a governed re-platforming. Assess with tooling, federate identity, migrate in waves, and validate downstream dependencies before you retire anything.

What migrating to Unity Catalog involves

Unity Catalog is Databricks' unified governance layer for data and AI. A migration typically upgrades tables, views, and workloads off the legacy Hive metastore into Unity Catalog's three-level namespace (catalog.schema.table), registers cloud storage as storage credentials and external locations, and moves identity and grants to the account level. Compute also has to be Unity Catalog compliant: clusters on Databricks Runtime 11.3 LTS or above using standard or dedicated access mode, while SQL warehouses support Unity Catalog by default.

The mistakes that cause the most pain

Mistake	Why it hurts	Do this instead
Skipping assessment	Unsupported patterns surface mid-cutover, after you've committed	Inventory the workspace with UCX before moving anything
Granting to individual users	Permissions drift and become impossible to audit as the estate grows	Grant to account-level groups; assign object ownership to groups
Workspace and account SCIM running in parallel	Duplicate groups and inconsistent row-filter evaluation	Provision identity at the account level and retire workspace-level SCIM
Migrating external tables before storage is set up	External-table upgrades block without credentials in place	Create storage credentials and external locations first
Defaulting to external tables	You give up full lifecycle governance for no reason	Default to managed tables; use external only to share storage with outside systems
Assuming a dropped external table is gone	Orphaned data becomes an ungoverned copy	Clean up the underlying storage explicitly after dropping
A single "big bang" cutover	One untested dependency takes down many jobs	Migrate incrementally and validate before decommissioning
Leaving code on the old namespace	Jobs keep reading the Hive metastore	Update queries and jobs to catalog.schema.table

How to migrate to Unity Catalog the right way

• Assess with UCX. Run the open-source UCX toolkit to profile the workspace, generate a table-mapping file, create the migration principal, and identify the storage credentials and external locations you'll need. Review the mapping before you migrate.
• Federate identity before data. Provision users, groups, and service principals at the account level with SCIM and SSO, and grant permissions to groups using a clear taxonomy (for example readers, writers, pii_readers, and data stewards). Assign ownership of production catalogs and schemas to groups, not people.
• Prepare storage and pick the right table type. Define storage credentials and external locations up front. Default to managed tables, which Unity Catalog governs end to end; reserve external tables for storage that must be shared with non-Databricks systems. Use deep CLONE to copy managed-table data, and SYNC or the upgrade wizard (both metadata-only) for external tables.
• Migrate in waves. For a gradual path, Hive metastore federation exposes the legacy metastore as a foreign catalog you can govern in Unity Catalog while you move. Annotate deprecated Hive tables with a pointer to their Unity Catalog replacement, and use Databricks' built-in assistant to find and update the queries and jobs that still reference them.
• Validate, then decommission. Test every downstream dependency against the Unity Catalog objects, update workloads to the three-level namespace, and only then retire the legacy tables and jobs.

Start with the Hive-to-Unity-Catalog migration docs, the Unity Catalog best practices, and the enterprise-scale migration guide.

Behavioral changes to plan for

A few differences trip up teams who expect Hive semantics. Unity Catalog manages partitions for you, so direct partition-manipulation commands aren't supported. CREATE TABLE ... CLONE treats the result as a new table, so pre-migration Delta Lake time-travel history doesn't carry over. And grants require account-level groups, so groups defined only at the workspace level can't be used in Unity Catalog GRANT statements.

FAQs

What's the fastest way to assess a Unity Catalog migration?

Run UCX, the open-source migration toolkit. It assesses the workspace, maps tables to target catalogs and schemas, and flags patterns that need attention before you move.

Should I use managed or external tables after migrating?

Default to managed tables, which Unity Catalog governs for lifecycle, location, and access. Use external tables only when the underlying storage must be shared with systems outside Databricks.

Do I have to migrate everything at once?

No, and you shouldn't. Migrate incrementally, and use Hive metastore federation to govern legacy data through a foreign catalog during the transition so workloads keep running.

Does dropping an external table delete its data?

No. Dropping an external table removes it from the catalog but leaves the underlying files in cloud storage. Clean up that storage explicitly to avoid orphaned, ungoverned copies.

What compute do I need for Unity Catalog?

Clusters on Databricks Runtime 11.3 LTS or above using standard or dedicated access mode. SQL warehouses support Unity Catalog by default.