What's the difference between declarative and imperative pipelines?

Summary

• An imperative pipeline is code that spells out each step and its order: read, transform, write. A declarative pipeline defines the datasets and the queries that produce them, and the framework works out the execution.
• Spark Declarative Pipelines (SDP) is the open-source declarative framework in Apache Spark, added in Spark 4.1.0. You "define what tables should exist and what their contents should be, and SDP handles the orchestration, compute management, and error handling automatically".
• In Python, @dp.materialized_view defines a batch dataset and @dp.table defines a streaming table. In SQL, CREATE MATERIALIZED VIEW and CREATE STREAMING TABLE do the same.
• On Databricks, Lakeflow Pipelines extends SDP and runs on Databricks Runtime. Code written with the open-source pipelines module runs without modification there.

What's the difference between declarative and imperative pipelines?

In an imperative pipeline you write the steps: read this table, transform it, write the result, then run the next step. You sequence the work, and for streaming queries you set the checkpoint location yourself. In a declarative pipeline you describe the end state, meaning which datasets exist and which query defines each one, and the framework works out the rest. The Spark 4.1.0 release notes describe SDP as a framework where "you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints, and retries."

How does an imperative Spark pipeline look?

orders = spark.read.table("raw_orders")
customers = spark.read.table("raw_customers")

(orders.filter("status != 'cancelled'")
    .join(customers, "customer_id")
    .write.mode("overwrite").saveAsTable("customer_orders"))

The write at the end is an explicit action in your code. Anything that must run before or after this script is ordered outside it, for example by a scheduler.

How does a Spark Declarative Pipeline look?

from pyspark import pipelines as dp
from pyspark.sql import DataFrame, SparkSession

spark = SparkSession.active()  # Spark 4.1 files declare this; Spark 4.2 injects spark

@dp.table
def orders() -> DataFrame:
    return spark.readStream.table("raw_orders")

@dp.materialized_view
def customers() -> DataFrame:
    return spark.read.table("raw_customers")

@dp.materialized_view
def customer_orders() -> DataFrame:
    return (spark.table("orders").filter("status != 'cancelled'")
        .join(spark.table("customers"), "customer_id"))

Each function returns a DataFrame and none of them writes anything. Because customer_orders reads orders and customers, SDP "analyzes the dependencies of your defined objects and orchestrates their order of execution and parallelization automatically". To try it, run pip install pyspark[pipelines], create a project with spark-pipelines init --name my_pipeline, check it with spark-pipelines dry-run, and run it with spark-pipelines run.

What changes between the two styles?

Aspect	Imperative Spark code	Spark Declarative Pipelines
You write	Reads, transformations and writes, in execution order	Dataset definitions: Python functions that return a DataFrame, or SQL statements
Execution order	Set by your script or an external orchestrator	Derived from the dependencies between datasets
Writes	Explicit calls such as saveAsTable()	Never in dataset code; SDP writes each dataset it defines
Batch and streaming	Separate entry points, spark.read and spark.readStream	One framework: @dp.materialized_view for a batch read, @dp.table for a streaming read
Pre-run checks	Run the job or its tests	spark-pipelines dry-run catches many errors, such as syntax, analysis and graph validation errors, without reading or writing data
Re-runs	Whatever your code does	In the Spark 4.2 guide, spark-pipelines run performs "a default incremental update" and --full-refresh recomputes chosen datasets from scratch

What rules apply inside SDP dataset functions?

SDP evaluates the code that defines a pipeline multiple times during planning and pipeline runs. A dataset function should contain only the code needed to define the table or view, and it must return a DataFrame. The Spark 4.2 guide lists operations that should never appear in SDP code: collect(), count(), pivot(), toPandas(), save(), saveAsTable(), start() and toTable(). The PIVOT clause is not supported in SDP SQL.

When should you pick each style?

• Use SDP when the work is a set of tables derived from other tables. The guide describes SDP as "designed for both batch and streaming data processing", including ingestion from cloud storage and message buses and incremental transformations.
• Use plain Spark code when a step cannot be a dataset definition, for example one that must call collect() or toPandas() or write to a destination SDP does not manage. The Spark 4.2 guide's sinks are streaming-only, Python-only and append-only.

How does Databricks approach declarative pipelines?

Lakeflow Pipelines extends SDP and is interoperable with it, while running on Databricks Runtime. The docs state that "Code written with the open-source pipelines module runs without modification on Databricks". The Lakeflow Jobs docs draw the same line as this page: "Lakeflow Jobs provides a procedural approach to defining relationships between tasks. Lakeflow pipelines provide a declarative approach to defining relationships between datasets and transformations." A job can run a pipeline as a task. The docs list three benefits of pipelines over manual orchestration with Spark and Structured Streaming APIs: automatic orchestration that retries transient failures progressively, from the Spark task to the flow to the entire pipeline; declarative processing; and incremental processing that keeps materialized views current. Pipelines run in triggered or continuous mode, the tables they create are Delta tables, and expectations (@dp.expect(...)) are a Databricks feature that is not part of Apache Spark.

FAQs

Do I need Databricks to use SDP?

No. SDP is part of Apache Spark beginning in Spark 4.1. The spark-pipelines command is built on spark-submit, so it supports all cluster managers that spark-submit supports.

Can I write a pipeline in SQL?

Yes. Pipeline source files can be .py or .sql. For example, CREATE STREAMING TABLE basic_st AS SELECT * FROM STREAM samples.nyctaxi.trips; defines a streaming table.

Can I mix imperative and declarative code?

At the workflow level, yes. On Databricks a job can run a pipeline as a task. In Python source files you can also use ordinary constructs, such as a for loop that defines several datasets, as long as the dataset functions follow the rules above.