SDP versus Airflow for orchestration?

Summary

• Spark Declarative Pipelines (SDP) and Apache Airflow work at different levels. SDP defines datasets and flows and orders the work inside one pipeline. Airflow schedules and monitors Dags of tasks that can run almost any code.
• They combine. The Apache Spark provider for Airflow includes a SparkPipelinesOperator that runs a pipeline with the spark-pipelines CLI.
• On Databricks, Lakeflow Pipelines are the declarative layer and Lakeflow Jobs is the orchestrator. Airflow can also trigger a Databricks job with DatabricksRunNowOperator.

SDP versus Airflow for orchestration?

SDP and Airflow solve different problems, and they work together. The Apache Spark programming guide calls SDP "a declarative framework for building reliable, maintainable, and testable data pipelines on Apache Spark": you define what tables should exist and what their contents should be, and SDP handles ordering and execution inside the pipeline. The Airflow documentation calls Airflow "an open-source platform for developing, scheduling, and monitoring workflows", and Airflow workflows are defined entirely in Python.

What does each one control?

SDP analyzes the dependencies between the objects you define and orchestrates their order of execution and parallelization automatically. You write datasets, and the references between them set the order:

CREATE STREAMING TABLE orders
AS SELECT * FROM STREAM orders_source;

CREATE MATERIALIZED VIEW orders_by_state
AS SELECT state, count(*) order_count
FROM orders
GROUP BY state;

An Airflow Dag, in the project's words, "doesn't care about what is happening inside the tasks; it is merely concerned with how to execute them - the order to run them in, how many times to retry them, if they have timeouts, and so on". Airflow tasks can run Spark jobs, move files or send email.

Question	Spark Declarative Pipelines	Apache Airflow
What you define	Datasets (streaming tables, materialized views, temporary views) and flows	A Dag of tasks that run operators
Authoring	Python with pyspark.pipelines, or SQL	Python
Ordering	Inferred from dependencies between datasets	Declared between tasks, for example hello >> world
Scheduling	The guide documents the spark-pipelines CLI and no scheduler	A schedule argument on the Dag, or manual and API triggers
Backfill	--refresh, --full-refresh, --full-refresh-all	airflow backfill create over past dates of a Dag
Failure handling	The Spark 4.1.0 release notes list checkpoints and retries	Reruns only failed tasks, and the Dag controls how many times tasks retry

When should I use each?

Use SDP when the work is data transformation on Spark that you can express as datasets and flows, in batch or streaming. Use Airflow when, in the Airflow documentation's words, your workflows "have a clear start and end and run on a schedule", or when one workflow spans several systems. The same documentation notes that defining workflows as code is central to how Airflow works, so if you prefer clicking over coding, Airflow might not be the best fit. To use both, run the pipeline as a task. The provider needs a configured Spark connection and the spark-pipelines CLI to be available.

from datetime import datetime

from airflow.sdk import DAG
from airflow.providers.apache.spark.operators.spark_pipelines import SparkPipelinesOperator
from airflow.providers.standard.operators.bash import BashOperator

with DAG(dag_id="daily_sdp", start_date=datetime(2026, 1, 1), schedule="0 0 * * *"):
    refresh = SparkPipelinesOperator(
        task_id="refresh_pipeline",
        pipeline_spec="/path/to/spark-pipeline.yml",
        pipeline_command="run",
        conn_id="spark_default",
    )
    notify = BashOperator(task_id="notify", bash_command="echo pipeline finished")

    refresh >> notify

How does this look on Databricks?

The Databricks documentation frames the split directly: Lakeflow Jobs provides a procedural approach to defining relationships between tasks, and Lakeflow Pipelines provide a declarative approach to defining relationships between datasets and transformations. Lakeflow Pipelines extend SDP.

• A pipeline task in a Lakeflow Jobs job runs a pipeline. Adding a schedule to a pipeline creates a job with a single pipeline task.
• Jobs can start on a schedule, a table update, file arrival or continuously, and they support branching with if / else and looping with for each.
• Pipelines retry transient failures progressively, from the Spark task to the flow to the entire pipeline, and support expectations that validate data as it flows through the pipeline. The Apache Spark programming guide does not cover expectations.
• Airflow can trigger an existing Databricks job with DatabricksRunNowOperator from the open source Databricks provider. Databricks recommends that operator, and notes the provider packages are not directly supported by Databricks.

FAQs

Can Airflow run an SDP pipeline?

Yes. The SparkPipelinesOperator wraps the spark-pipelines binary and takes pipeline_command values of run and dry-run. For --refresh or --full-refresh, call the CLI from a BashOperator.

Do I need Airflow if I run on Databricks?

Not for scheduling a pipeline. Lakeflow Jobs provides schedules and triggers for pipeline tasks. Teams that already run Airflow can keep it as the outer orchestrator and have it start Databricks jobs.

Does SDP replace a scheduler?

No. The Apache Spark guide documents running a pipeline with spark-pipelines run, and a scheduler decides when that command runs.