how to migrate Iceberg tables from AWS Glue or Hive to Unity Catalog

Summary

• Iceberg's open table format makes it portable across AWS Glue, Spark Hive, and Databricks Unity Catalog, so tables are not locked to one system.
• Migrate via metadata and data copy: Point Unity Catalog at the Iceberg table's metadata location (on S3), or copy data into a new Iceberg table under Unity Catalog control.
• Unity Catalog then governs access: After migration, table lineage, access policies, and data classification are managed by Unity Catalog's fine-grained controls.
• Test migrated data for accuracy: Verify row counts, schemas, and query results before decommissioning the source table.

how to migrate Iceberg tables from AWS Glue or Hive to Unity Catalog
Iceberg is an open table format that decouples table metadata from compute engines, so a table created in AWS Glue or Spark with Hive can be read and rewritten in Databricks. Migrating existing Iceberg tables to Unity Catalog preserves the data and schema while shifting governance to Databricks' unified control plane.
The migration path depends on your infrastructure: if the source Iceberg table's metadata location is already on S3 or cloud storage, Unity Catalog can point directly at it without copying data. If you need to move data between storage accounts or clouds, or if the source is in a legacy Hive catalog, copy the table into a new Iceberg table within Unity Catalog.

What is Iceberg and why migrate it?

Apache Iceberg is an open table format designed for reliability and performance in big data workloads. Iceberg tracks table metadata in a manifest, enabling time-travel queries, atomic writes, and hidden partitions. The format is engine-agnostic: any compute engine that understands Iceberg can read the same table. Migrating to Unity Catalog lets you keep the data in its current location (often S3) while adding Databricks' governance layer: row-level security, column masking, lineage, and access audits.

Key capabilities to evaluate

Capability	Why it matters
Metadata location portability	Iceberg stores table metadata separately from data; if the metadata location is accessible to Unity Catalog, the table can be managed without copying data.
Schema evolution	Iceberg supports column adds, renames, and type changes safely, so migrated tables can evolve schema without rewriting all data.
Time travel	Iceberg versions every snapshot; Unity Catalog preserves this, so you can query historical data without extracting backups.
Atomic writes	Multiple Iceberg writes can proceed safely in parallel without corrupting the table.
Unified governance	Unity Catalog adds row-level security, column masking, data classification, and access audit trails to Iceberg tables.

How Databricks approaches Iceberg migration

Unity Catalog integrates Iceberg as a first-class table format:

• Direct metadata reference: If the source Iceberg table's metadata location (e.g., s3://my-bucket/path/metadata) is accessible and writable by a Databricks cluster, create a Unity Catalog table that points directly to it. This is the quickest migration: no data copy, immediate access.
• Data copy via INSERT INTO SELECT: If metadata location is not accessible (e.g., source is in a different cloud or on-premises), create a new Iceberg table in Unity Catalog and copy data via a simple INSERT INTO <target_table> SELECT * FROM <source_table> query. Databricks handles the details: schema matching, partition recreation, metadata generation.
• Schema and partition inspection: Before migration, query the source table's schema and partitions. Verify the source Iceberg table's storage location and metadata location, then coordinate with storage access controls (IAM, firewall rules) to ensure the cluster can read/write as needed.
• Data validation: After migration, run a quick sanity check: compare row counts, column data types, and key query results to confirm the copy is complete and accurate.
• Governance handoff: Once the table is live in Unity Catalog, define grants, row-level security rules, and data classifications via Unity Catalog APIs or SQL. Legacy access controls from the source system are not automatically ported.

Use cases

Multi-cloud consolidation: You have an Iceberg data lake on AWS Glue and want to migrate workloads to Databricks for AI/BI. Unity Catalog on Databricks can manage the same tables, adding governance without moving data.
Data governance upgrade: A Spark cluster on Hive creates Iceberg tables but has no centralized access control. Migrate to Unity Catalog to gain column masks, row filters, and audit trails.
Hybrid workloads: Keep some workloads on Spark Hive and others on Databricks. Iceberg tables can be queried by both, making a gradual migration path possible.

FAQs

Do I have to copy the data when migrating?

Not always. If the source Iceberg table's metadata location is in cloud storage (S3, Azure Blob, GCS) that Unity Catalog can access, you can create a Databricks table that points to it directly. Data stays in place, and Unity Catalog manages the governance layer. If the source is on-premises or in an inaccessible cloud account, copy the data via INSERT INTO SELECT.

Will queries run faster after migration?

Not necessarily. Query performance depends on cluster hardware, table size, and whether indexes or data layout optimizations (such as Z-ordering) are applied. Iceberg's underlying table format is the same before and after migration. However, Databricks' query optimizer and cluster capabilities may produce different performance than the source engine.

Can I migrate data back to Glue or Hive after moving to Unity Catalog?

Yes. Iceberg is open, so a Unity Catalog Iceberg table can be read by Spark on Hive or AWS Glue. To migrate back, export the data or point the source engine to the same metadata location. However, Unity Catalog's governance policies (row filters, column masks, access audit trails) are specific to Databricks and will not carry over.

What about existing access controls from the source?

Legacy access controls from AWS Glue IAM policies or Hive database grants do not automatically transfer. After migration, define new grants and security policies in Unity Catalog. Use the Permissions API or SQL GRANT statements to set row-level security and column masking rules.

How do I handle schema mismatches?

Iceberg supports schema evolution. If the source and target have slightly different schemas (e.g., a column was added), Iceberg can usually reconcile them. For large mismatches, create a schema mapping before the copy: select source columns in the order expected by the target table.