How do I fix a dependency cycle between datasets?

Summary

• A dependency cycle means datasets read each other in a loop, so no run order exists. Spark Declarative Pipelines (SDP) analyzes the dependencies between the objects you define and orders execution automatically, and circular dependencies are not supported in a pipeline.
• SDP reports a cycle as a graph validation error that names the objects involved. Two errors you may see are CircularDependencyException and PIPELINE_GRAPH_NOT_TOPOLOGICALLY_SORTED, and spark-pipelines dry-run catches cycles without reading or writing data.
• Fix it by cutting one edge of the loop. Point the downstream reader at the original source, or move the shared logic into a dataset that both sides read.
• Several flows appending to one streaming table are not a cycle. A loop appears when an upstream dataset reads one that sits downstream of it.

How do I fix a dependency cycle between datasets?

A dependency cycle is a loop in the pipeline's dependency graph: dataset A reads B and B reads A, or a longer chain such as A reads B, B reads C and C reads A. SDP, which shipped in Apache Spark 4.1.0, analyzes the dependencies of the objects you define and orchestrates their order of execution, so a loop leaves no valid order and the pipeline fails validation. To fix it, find the two datasets named in the error and remove one of the reads between them.

What does a cycle look like?

In SDP, streaming tables, materialized views and temporary views are datasets, and a flow reads datasets and writes its target. A cycle appears when an upstream dataset reads one that sits downstream of it. In this SQL, each materialized view reads the other:

CREATE MATERIALIZED VIEW order_totals AS
SELECT customer_id, sum(amount) AS total
FROM flagged_orders GROUP BY customer_id;

CREATE MATERIALIZED VIEW flagged_orders AS
SELECT o.* FROM orders o
JOIN order_totals t ON o.customer_id = t.customer_id
WHERE t.total > 0;

In Python, the same loop appears when a function decorated with @dp.materialized_view reads a pipeline table through spark.table("...") and that table's definition reads it back.

How do I find the cycle?

Run spark-pipelines dry-run. The Spark documentation says it launches an execution that does not write or read any data and catches graph validation errors such as cyclic dependencies. Then match the error to its cause:

Symptom	Likely cause	Fix
CircularDependencyException: the downstream table is referenced when creating the upstream table or view	An upstream dataset reads a dataset that sits downstream of it	Remove the dependency between the two datasets named in the message
PIPELINE_GRAPH_NOT_TOPOLOGICALLY_SORTED: there is a cycle between a flow and an input	A flow's input cannot be placed ahead of it in dependency order	Trace the named flow and input, then cut one edge
Two datasets feed each other	Each one reads a result that the other computes	Compute the shared result once in a dataset upstream of both

How do I break the cycle?

Compute the shared piece from the original source instead of from the dataset that depends on it. Here order_totals reads orders, so the data flows one way:

CREATE MATERIALIZED VIEW order_totals AS
SELECT customer_id, sum(amount) AS total
FROM orders GROUP BY customer_id;

CREATE MATERIALIZED VIEW flagged_orders AS
SELECT o.* FROM orders o
JOIN order_totals t ON o.customer_id = t.customer_id
WHERE t.total > 0;

If several datasets need the same intermediate logic, put it in a temporary view. The SDP guide describes temporary views as scoped to an execution of the pipeline and useful for intermediate entities that multiple other elements depend on. Make sure the shared dataset reads only upstream sources.

How does Databricks approach this?

Lakeflow Pipelines is the Databricks product that extends SDP. The documentation says pipelines infer dependencies between datasets and arrange them in a directed acyclic graph (DAG), with upstream datasets computed before downstream ones. Because the pipeline analyzes dependencies across all source files, you can organize code across files in any order, so reordering files does not fix a cycle. Other points from the documentation:

• You can view the pipeline graph in the Lakeflow Pipelines Editor, and Dry run validates the pipeline without updating any data.
• A self-reference (or circular reference) is not allowed in the default publishing mode and generates an error.
• The error class PIPELINE_GRAPH_NOT_TOPOLOGICALLY_SORTED reads There is a cycle between <flowName> and <inputName>. in the error-class reference.

FAQs

Does writing to one table from several flows create a cycle?

No. SDP documents multiple flows that append to the same target, using CREATE FLOW in SQL and @dp.append_flow(target=...) in Python. Those flows form a cycle only if one of them reads the target or a dataset downstream of it.

Do temporary views avoid cycles?

No. SDP lists temporary views among its dataset types and analyzes the dependencies of everything you define, so a loop that runs through a temporary view is still a loop.

Can a dataset read its own output?

Not in Lakeflow Pipelines with the default publishing mode, where a self-reference generates an error. Read the upstream source the table is built from instead.