How do I view lineage for an SDP pipeline?

Summary

• In open-source Spark Declarative Pipelines (SDP), lineage is the pipeline's dataflow graph: each flow reads from sources and writes to a target dataset, and SDP analyzes those dependencies to order and parallelize execution.
• The upstream SDP guide documents no lineage viewer or lineage API. You read the graph from your definitions, where every spark.table(...), spark.readStream.table(...), FROM or FROM STREAM reference to a pipeline dataset is an edge.
• spark-pipelines dry-run checks the pipeline without reading or writing data and reports syntax errors, analysis errors such as a missing table or column, and graph validation errors such as cyclic dependencies.
• During spark-pipelines run, the Spark UI's SQL tab lists the queries the pipeline executes, and each query's plan shows which tables or files it read.
• On Databricks, Lakeflow Pipelines draws the pipeline graph, records each flow's inputs and output as flow_definition events in the event log, and Unity Catalog captures table and column lineage.

How do I view lineage for an SDP pipeline?

In open-source Spark Declarative Pipelines (SDP), you view lineage by inspecting the pipeline's dataflow graph, because that graph is the lineage. Every flow reads from one or more sources and writes to a target dataset, and SDP works out the dependencies between them to order execution. The upstream SDP programming guide, for the framework that arrived with Apache Spark 4.1.0, doesn't document a lineage viewer or a lineage API. So the workflow is to read the edges from your definitions, validate the graph with spark-pipelines dry-run, and check each flow's query plan in the Spark UI. A rendered graph, plus table and column lineage across pipelines, is what a managed platform adds.

How SDP builds the graph

Three concepts from the SDP guide define the graph:

• A flow reads data from a source, applies your processing logic and writes the result into a target dataset.
• A dataset is a streaming table, a materialized view or a temporary view. Flows can read datasets that other flows in the same pipeline produce.
• A pipeline contains the flows and datasets. While it runs, SDP analyzes the dependencies of your defined objects and orchestrates their order of execution and parallelization automatically.

You reference a dataset defined in the pipeline the same way you'd reference a table defined outside it. Each such reference is an edge in the graph:

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, SparkSession

spark = SparkSession.active()  # required in Spark 4.1; optional from 4.2

@dp.table
def orders() -> DataFrame:
    return spark.readStream.table("raw.orders")

@dp.materialized_view
def customers() -> DataFrame:
    return spark.read.format("csv").option("header", True).load("/data/customers")

@dp.materialized_view
def customer_orders() -> DataFrame:
    return spark.table("orders").join(spark.table("customers"), "customer_id")

That project produces this graph:

Dataset	Type	Reads from
orders	Streaming table	raw.orders, a table outside the pipeline
customers	Materialized view	CSV files under /data/customers
customer_orders	Materialized view	orders and customers

SQL definitions work the same way: each table named in a FROM or FROM STREAM clause of a CREATE MATERIALIZED VIEW or CREATE STREAMING TABLE statement becomes an edge.

Three ways to see lineage in open-source SDP

Method	What it shows	Limits
Read the definitions	Every dataset and the datasets each flow reads	Manual work; names generated in a for loop need extra care
spark-pipelines dry-run	Whether the graph resolves, with syntax, analysis and graph validation errors	Validation output only; the guide documents no diagram
Spark UI SQL tab during spark-pipelines run	Each executed query's plan, including the tables or files it scanned	One query at a time, with no pipeline-level view

Run the CLI from the project directory; when you don't pass a spec path, it looks for spark-pipeline.yml or spark-pipeline.yaml in the current directory and its parents.

cd my_pipeline
spark-pipelines dry-run
spark-pipelines run

The dry run reads and writes no data, but it catches the errors that break lineage: a flow selecting from a missing table or column, or a cycle between datasets. Once run starts, open the Spark UI of the application running the pipeline. The SQL tab lists the SQL and DataFrame queries it executes, each with a graph view of its operators and a details link to the logical and physical plans. The scan operators show what each flow read.

How Databricks approaches pipeline lineage

On Databricks, Lakeflow Pipelines runs declarative pipelines like these and captures lineage for you at three levels:

• Pipeline graph. The monitoring page shows the pipeline graph, or directed acyclic graph (DAG), as soon as an update has successfully started. Arrows represent dependencies between datasets, and "Navigate to code" jumps from a table to its definition. The Lakeflow Pipelines Editor shows the graph after you run or validate the source files, and Dry run validates the pipeline without updating any data.
• Event log. Events of type flow_definition carry output_dataset and input_datasets, which define each relationship in the graph. By default, only the pipeline's run-as user can query the event log.
• Unity Catalog. Unity Catalog records lineage on its own for queries that run on Databricks, including column-level links. Open a table in Catalog Explorer, select the Lineage tab, then See Lineage Graph, or query system.access.table_lineage and system.access.column_lineage.

SELECT details:flow_definition.output_dataset AS output_dataset,
       details:flow_definition.input_datasets AS input_datasets
FROM event_log('<pipeline-id>')
WHERE details:flow_definition IS NOT NULL
ORDER BY timestamp;

FAQs

Does open-source SDP have a lineage UI?

The Spark SDP programming guide doesn't document one. It documents the spark-pipelines CLI (init, run and dry-run), so use the dry run, your definitions and the Spark UI.

Can several flows feed one dataset?

Yes. dp.create_streaming_table plus @dp.append_flow(target=...) lets multiple flows append to the same streaming table, so that dataset has several incoming edges.

How do I get column-level lineage?

Upstream SDP doesn't document it. On Databricks, Unity Catalog tracks column lineage for Lakeflow pipeline workloads on Databricks Runtime 13.3 LTS and above; click a column in the lineage graph to see its upstream columns.