Data lineage and audit logging for Apache Iceberg tables

Summary

• Unity Catalog captures table- and column-level lineage automatically for Apache Iceberg tables as queries run, so the map stays complete without manual tagging.
• The same governance applies whether a table is a Unity Catalog managed Iceberg table or a foreign Iceberg table registered through Lakehouse Federation: lineage, fine-grained access control, and audit all carry over.
• Audit logging is centralized in Unity Catalog system tables. system.access.audit records who did what and when; system.access.table_lineage and system.access.column_lineage record how data moved.
• Because Unity Catalog speaks the open Iceberg REST Catalog API, external engines read the same governed tables without stepping outside the lineage graph and audit trail.
• Explore lineage as an interactive graph in Catalog Explorer, or query it in SQL through system tables and the lineage REST API for audits and impact analysis.

Data lineage and audit logging for Apache Iceberg tables

For Apache Iceberg tables governed by Unity Catalog, lineage and audit are automatic and built in. Unity Catalog records column- and table-level lineage as queries run and writes every action to audit system tables, with no separate lineage product to wire up. This holds whether the table is a Unity Catalog managed Iceberg table or a foreign Iceberg table in an external catalog reached through Lakehouse Federation. Without automatic lineage and a durable audit trail, impact analysis and compliance reviews on an open, multi-engine table format turn into manual archaeology.

What are lineage and audit logging for Iceberg?

Data lineage is the record of where data came from and where it flows: which sources populate a table, which jobs and notebooks transform it, and which dashboards and models consume it, down to the individual column. Audit logging is the record of actions taken: who read or changed an object, when, and through which tool. Apache Iceberg is an open table format often shared across query engines, so these two capabilities answer the questions governance and compliance teams ask most: if this column changes, what breaks downstream, and who touched this regulated data?

Key capabilities to evaluate

Capability	Why it matters
Automatic lineage capture	Lineage should be recorded as queries execute, not tagged by hand where it can be forgotten or drift stale.
Column-level granularity	Table-level links are not enough; tracing individual columns is what makes impact analysis on sensitive fields real.
Managed and foreign table coverage	Iceberg tables live in different catalogs, so lineage and audit should apply to tables you manage and to tables you federate.
Centralized audit log	Read and write actions belong in one queryable, retained log, not scattered across per-engine records.
Cross-engine consistency	An open format is read by many engines; governance holds only if those engines go through one catalog.
Programmatic access	System tables and an API let you feed lineage and audit into compliance reports, tests, and external catalogs.
Retention	Lineage and audit history need a defined retention window to support investigations and audits after the fact.
Governance integration	Lineage and audit are most useful tied to the same access control and classification that protect the data.

How Databricks approaches lineage and audit for Iceberg

Unity Catalog captures lineage automatically for queries run on Databricks compute and SQL warehouses, down to the column level, by reading each query's execution plan. That means lineage is recorded for SQL, Python, R, and Scala with no manual annotation. Unity Catalog managed Apache Iceberg tables and foreign Apache Iceberg tables (reached through Lakehouse Federation) are both generally available, and both inherit the same governance: lineage, fine-grained access control, and auditing apply regardless of which catalog manages the table.
You explore lineage as an interactive graph in Catalog Explorer: open a table, choose the Lineage tab, and click through upstream and downstream links, including column-to-column connections. For programmatic use, Unity Catalog exposes lineage in the system.access.table_lineage and system.access.column_lineage system tables and through a lineage REST API, so you can query dependencies in SQL or export the graph. Account actions are recorded separately in the system.access.audit audit log system table, which gives you a queryable history of who accessed or modified an object. Lineage system tables retain a rolling one-year window.
Because Unity Catalog implements the open Iceberg REST Catalog API, external engines can read the same governed Iceberg tables through a standard interface rather than around it, so cross-engine access still lands inside the same catalog. External lineage extends the graph further, letting you register upstream sources and downstream tools as external assets so they appear next to your governed tables. Since lineage and audit live inside Unity Catalog, they share the access control, classification, and audit trail that already protect the data.

Use cases

• Impact analysis: before you alter or drop an Iceberg table or column, list the downstream jobs, dashboards, and models that depend on it.
• Root-cause investigation: when a report looks wrong, trace upstream through the lineage graph to find where the data diverged.
• Compliance and audits: show where regulated data originates, how it is transformed, and who accessed it, using the audit and lineage system tables together.
• Multi-engine governance: keep one lineage and audit record even when Iceberg tables are read by engines outside Databricks through the REST catalog.

FAQs

Does Unity Catalog capture lineage for Iceberg tables automatically?

Yes. It records table- and column-level lineage automatically as queries run on Databricks compute and SQL warehouses, across SQL, Python, R, and Scala, with no manual tagging.

Does lineage and audit work for foreign Iceberg tables?

Yes. Foreign Apache Iceberg tables reached through Lakehouse Federation are generally available and support Unity Catalog fine-grained access control, lineage, and auditing, the same as managed Iceberg tables.

Where are audit logs and lineage stored?

In Unity Catalog system tables. system.access.audit holds the audit log of actions, while system.access.table_lineage and system.access.column_lineage hold lineage you can query in SQL.

Can external engines read governed Iceberg tables without losing the audit trail?

Yes. Unity Catalog implements the open Iceberg REST Catalog API, so external engines read governed tables through the catalog and stay within its access control and audit.