How do I schedule and backfill an SDP pipeline?

Summary

• Open-source Spark Declarative Pipelines (SDP) runs through the spark-pipelines run command. The programming guide documents no scheduler, so recurring runs come from a scheduler that calls the command, such as the SparkPipelinesOperator in Apache Airflow's Spark provider.
• Backfills use the run options: a default incremental update, --refresh for chosen datasets, --full-refresh to reset and recompute chosen datasets, and --full-refresh-all.
• On Databricks, a pipeline runs as a pipeline task in a Lakeflow Jobs job with a schedule or trigger. Historical data is backfilled with one-time append flows, a full refresh, or the job Run backfill feature.

How do I schedule and backfill an SDP pipeline?

spark-pipelines run launches an execution of a pipeline and monitors its progress until it completes. The Apache Spark programming guide documents that command and no scheduler, so a recurring run comes from a scheduler you already operate, and a backfill comes from the run options.

How do I run an SDP pipeline on a schedule?

Any scheduler that can run a shell command can call spark-pipelines run. The Apache Spark provider for Apache Airflow also has a SparkPipelinesOperator that wraps the spark-pipelines binary. It needs a configured Spark connection and the spark-pipelines CLI to be available. This example uses the Airflow 3 import from airflow.sdk import DAG.

from datetime import datetime

from airflow.sdk import DAG
from airflow.providers.apache.spark.operators.spark_pipelines import SparkPipelinesOperator

with DAG(dag_id="sdp_nightly", start_date=datetime(2026, 1, 1), schedule="0 0 * * *"):
    SparkPipelinesOperator(
        task_id="run_pipeline",
        pipeline_spec="/path/to/spark-pipeline.yml",
        pipeline_command="run",
        conn_id="spark_default",
    )

pipeline_command accepts run or dry-run. The operator's documented parameters cover the spec path, the command and Spark settings, but not refresh selection. For --refresh and --full-refresh runs, call the CLI from a BashOperator.

What do the run options do for a backfill?

Goal	Option	Behavior in the guide
Normal run	none	A default incremental update
Update chosen datasets	--refresh orders,customers	Triggers an update for those datasets without clearing existing data
Reset and recompute chosen datasets	--full-refresh orders,customers	Clears all existing data and checkpoints for those datasets and recomputes them from scratch
Reset and recompute everything	--full-refresh-all	A full graph reset and recompute, equivalent to --full-refresh for all datasets

--full-refresh-all cannot be combined with --full-refresh or --refresh. The other two can be used together. spark-pipelines dry-run checks for syntax, analysis and graph errors without writing or reading data, which is a safe step before a large reprocess.
A full refresh recomputes from scratch, so the source must still hold the data you need. Databricks warns that when a source no longer holds the data, such as Kafka with a short retention threshold, a full refresh drops those records from the target table. For sinks, the Spark guide notes that full refresh updates reset checkpoints but do not clean previously computed results.
The guide has no backfill-specific command. To reprocess only a window, have your dataset code read the window from a configuration value, the way the Databricks backfill guide reads begin_year and backfill_years with spark.conf.get. Airflow can also create runs for past dates of a Dag with airflow backfill create --dag-id sdp_nightly --from-date 2026-01-01 --to-date 2026-01-07, but with the Dag above each run calls the same command, so the pipeline itself must decide what to reprocess.

How do I schedule and backfill on Databricks?

Lakeflow Pipelines extend SDP, and Lakeflow Jobs schedules them. Adding a schedule to a pipeline creates a job with a single pipeline task, and in a triggered or scheduled job that task starts one update and stops when it completes. Jobs also support table update, file arrival and continuous triggers.

resources:
  jobs:
    sdp_nightly:
      name: sdp_nightly
      schedule:
        quartz_cron_expression: "0 0 9 * * ?"
        timezone_id: UTC
        pause_status: UNPAUSED
      tasks:
        - task_key: refresh_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.example_pipeline.id}

Backfill approach	Use it when	Notes from the docs
One-time append flow (once=True in Python, INSERT INTO ONCE in SQL)	You are backfilling an append-only streaming table	Databricks recommends it for pipelines. The flow stays in the graph and re-runs on a full refresh
Full refresh (full_refresh on the pipeline task, full_refresh_selection in the API)	The source still holds all the data you need	Set the table property pipelines.reset.allowed to false to block full refreshes
Run backfill on the job	The job takes a date or time parameter	A backfill that would create more than 100 runs cannot start, and jobs with pipeline tasks run sequentially

A one-time flow looks like this in SQL:

CREATE FLOW registration_events_raw_backfill_2024
AS INSERT INTO ONCE
  registration_events_raw BY NAME
SELECT * FROM read_files(
  "/Volumes/gc/demo/apps_raw/event_registration/year=2024/*/*",
  format => "json",
  inferColumnTypes => true
);

FAQs

Does open-source SDP retry a failed run?

The Spark 4.1.0 release notes say Spark handles the execution graph, dependency ordering, parallelism, checkpoints and retries for SDP. Whether the command itself is rerun is up to the scheduler. Airflow can rerun only failed tasks, and a Lakeflow Jobs task takes a retry policy that, for most configurations, defaults to no retries.

Does the append_flow decorator in Apache Spark support once?

The PySpark reference lists target, name and spark_conf for append_flow. The once option appears in the Databricks documentation for Lakeflow Pipelines.