How do I govern Delta Lake and Iceberg together?

Summary

• Delta Lake and Iceberg are complementary table formats in the Lakehouse: Delta is optimized for transactional workloads and AI, while Iceberg excels at cross-platform analytics and schema evolution.
• Unity Catalog governs both formats uniformly: the same access controls, tag policies, and governance rules apply whether data is stored as Delta or Iceberg.
• You can migrate data between Delta and Iceberg without losing governance, lineage, or access controls, making format choice an implementation detail rather than a governance boundary.
• Multi-format governance allows teams to pick the right table format for each use case while maintaining a single, consistent access control layer.

How do I govern Delta Lake and Iceberg together?

Many organizations use both Delta Lake and Iceberg in the same Lakehouse. Delta powers transactional operations and AI pipelines; Iceberg handles cross-platform analytics and schema evolution. Governing both formats consistently is essential: different formats shouldn't create governance seams or duplicate policies. Unity Catalog solves this by treating both as first-class table formats, applying the same governance rules uniformly regardless of which format you choose.

What are Delta Lake and Iceberg?

Delta Lake is Databricks' transactional table format, optimized for streaming, ACID updates, and AI workloads. It stores data as Parquet files and uses a transaction log for consistency and time-travel.
Apache Iceberg is an open-table format designed for analytics at scale, with strong support for schema evolution, hidden partitioning, and cross-platform query engines. It uses a metadata layer to track snapshots instead of a write-ahead log.
Both are table formats stored in cloud object storage (S3, Azure Blob, GCS); both can store petabyte-scale data. The choice between them typically depends on workload: transactional AI pipelines favor Delta, while multi-engine analytics favor Iceberg.

Key capabilities to evaluate

Capability	Why it matters
Unified access control	Row filters, column masks, and privilege denials apply to both Delta and Iceberg tables, enforced at the SQL engine level.
Consistent tag-based policies	Govern by tags: a policy on the data_classification: sensitive tag applies to any table (Delta or Iceberg) that carries that tag.
Lineage tracking across formats	See which jobs or AI models read and write Delta tables, Iceberg tables, or both, in a single lineage graph.
Governed format conversion	Migrate data from Delta to Iceberg (or vice versa) while preserving access policies and metadata.
Audit trails per format	Track read and write events separately by table format to troubleshoot issues or validate data movement.
Cross-format analytics	Query both formats in the same SQL statement; governance applies uniformly.

How Databricks approaches multi-format governance

Unity Catalog treats Delta and Iceberg as two formats within a single governance framework. Here's how it works:

1. Register both formats in a single catalog. You define a catalog in Unity Catalog and register both Delta and Iceberg tables within the same schema. There's no separate governance boundary between them.
2. Apply tag-based policies universally. When you tag a table environment: production, the policy ("only data engineers can write") applies whether the underlying table is Delta or Iceberg. The enforcement happens at the SQL layer, not the storage layer.
3. Migrate between formats safely. The Databricks CONVERT TO DELTA and CONVERT TO ICEBERG commands preserve metadata and governance tags. After migration, the same access controls apply.
4. Track lineage across both formats. When a Lakeflow Pipelines job reads an Iceberg table, transforms it, and writes a Delta table, Unity Catalog records the entire lineage. Lineage reports show both tables linked in the same dependency graph.
5. Audit queries on both formats. Query audit logs capture who accessed Delta tables, who accessed Iceberg tables, and which columns were masked or filtered, all in one queryable table.

For more details, see Unity Catalog governance and the Delta to Iceberg conversion guide.

Use cases

Phased migration to Iceberg: A large analytics platform runs on Delta. To support cross-platform query engines and open-source query tools, you plan to migrate to Iceberg. Unity Catalog ensures that as you convert tables one by one, existing access policies (row filters for customer isolation, column masks for PII) remain in place. Teams don't see governance gaps during the transition.
Workload-optimized format selection: A financial services firm stores transactional data as Delta (for fast updates and ACID semantics) and historical snapshots as Iceberg (for efficient analytics queries). Both are tagged classification: trading_data and governed by the same policies. Traders access both via SQL; the format choice is transparent to the security model.
Multi-team collaboration: Data engineers build pipelines that write Delta tables; analytics teams query Iceberg tables converted from Delta nightly. Unity Catalog's lineage links both, and any change to access policies applies to the entire pipeline automatically.
Cross-platform Iceberg queries: An organization stores Iceberg tables in cloud object storage and queries them from multiple platforms: Databricks for AI workloads, and open-source or third-party engines for analytics. Unity Catalog governance applies across all query sources: the same column masks and row filters enforce regardless of which engine executes the query.

FAQs

Can I run SQL queries that join Delta and Iceberg tables?

Yes. A single SQL query can join a Delta table and an Iceberg table. Governance rules apply to each table separately: if a Delta table has a row filter, it applies; if an Iceberg table has a column mask, it applies. The join proceeds on the filtered and masked results.

Does converting a table from Delta to Iceberg change its governance?

No. The CONVERT TO ICEBERG command preserves tags, ownership, and policies. After conversion, the same rules apply. The only thing that changes is the underlying table format.

What if I have a policy that should apply only to Delta tables, not Iceberg?

Create a tag specifically for that use case (e.g., format_target: delta_only) and apply the policy to that tag. This is rarely necessary; most policies are workload-driven, not format-driven.

Can I enforce different column masks for the same column in Delta versus Iceberg?

No. If a column contains PII, it should be masked the same way regardless of format. Masking rules shouldn't differ between formats.