how does Unity Catalog handle column-level lineage

Summary

• Captured automatically at runtime. Unity Catalog records column-level lineage as queries run on clusters, SQL warehouses, and pipelines, with no manual tagging or extra configuration.
• Broad coverage. Lineage spans tables and views, notebooks in all languages, jobs, dashboards, pipelines, feature tables, files and volumes, and ML models, aggregated across every workspace attached to the metastore.
• Visual graph in Catalog Explorer. You can open a table's Lineage tab, see the lineage graph, and click a column to trace its upstream and downstream column-to-column relationships.
• Accessible via API and system tables. Lineage can be exported through REST APIs and queried from the system.access.column_lineage and system.access.table_lineage system tables.
• Known limits. Column-level lineage is not captured for path-based references, user-defined functions, RDDs, or global temporary views.

How does Unity Catalog handle column-level lineage?

Unity Catalog captures column-level lineage automatically, showing how individual columns flow from source to destination across your data and AI assets. There is no separate tool to configure or manual tagging to maintain, lineage is generated at runtime as your workloads execute and is available in the UI, through APIs, and in system tables.

Why Unity Catalog captures column-level lineage automatically

• Automatic runtime capture. When code is submitted to a cluster, SQL warehouse, or pipeline, Spark produces a logical plan describing every transformation. That metadata is sent to a lineage service in the control plane, which assembles the logical plans into lineage graphs and distills them into tabular information about which columns feed which other columns.
• Broad workload and asset coverage. Unity Catalog tracks lineage for tables and views (including external tables), notebooks in all supported languages, jobs and workflows, dashboards as downstream consumers, declarative pipelines, feature tables, files and volumes, and ML models. Lineage is aggregated across all workspaces attached to the metastore for a single, unified view.
• Interactive column-level graph. In Catalog Explorer, open a table, select the Lineage tab, and choose See Lineage Graph. Clicking a specific column highlights its links to related upstream and downstream columns, and clicking a connecting edge opens a details panel showing source and target information.
• API and system-table access. Lineage can be exported through REST APIs to support integrations with third-party catalog tools, and it is available through lineage system tables, system.access.table_lineage and system.access.column_lineage, which retain a rolling one-year window of data for SQL-based analysis and reporting.
• Governance and observability value. Because lineage is captured automatically and tied to the same metastore as access control and audit logs, teams can trace the origin and downstream impact of any column, supporting impact analysis, debugging, and compliance. See How Unity Catalog supports a unified view, monitoring, and lineage.

Getting started

• Open Catalog Explorer, find a table, and select the Lineage tab to view its column-level graph.
• Query system.access.column_lineage to analyze column relationships programmatically or feed them into reports.
• Review the lineage documentation for requirements and current limitations before relying on lineage for governance workflows.

FAQs

Do I need to configure anything to get column-level lineage?

No. Lineage is captured automatically at runtime as workloads run on clusters, SQL warehouses, and pipelines, with no manual tagging or separate lineage tool required.

Where can I see column-level lineage?

In Catalog Explorer's Lineage tab as an interactive graph, and programmatically through REST APIs and the system.access.column_lineage and system.access.table_lineage system tables.

When is column-level lineage not captured?

Lineage is not captured at the column level when a source or target is referenced by path rather than by table name, when user-defined functions obscure the column mapping, or for RDDs and global temporary views.

How long is lineage data retained?

The lineage system tables retain a rolling one-year window of data, which you can query with SQL for reporting and analysis.