When should I use SDP instead of hand-written jobs?

Summary

• Use Spark Declarative Pipelines (SDP) when the work is a graph of datasets, each defined by a query, and you want Spark to handle "the execution graph, dependency ordering, parallelism, checkpoints, and retries".
• SDP covers batch and streaming. Streaming tables process new data incrementally, materialized views are precomputed into tables, and Spark 4.2.0 adds Auto CDC for declarative SCD Type 1 upserts.
• Keep a hand-written job when dataset code needs actions or writes such as collect(), toPandas() or saveAsTable(), uses SQL PIVOT, is written in Scala or Java, or needs a sink that isn't append-only.
• SDP doesn't schedule itself: spark-pipelines run launches one execution. On Databricks, a Lakeflow Jobs pipeline task can start it.

When should I use SDP instead of hand-written jobs?

Use SDP when your pipeline is a set of datasets, each defined by a query, with dependencies between them. Use a hand-written job when the work is mostly actions and side effects that don't fit datasets defined by queries. The two can coexist, because a dataset is, in the guide's words, "a queryable object", so a hand-written job can read what a pipeline produces. Here, a hand-written job means a Spark application you write and submit yourself with DataFrame, SQL or Structured Streaming code.

What does SDP take off your hands?

The Spark 4.1.0 release notes describe SDP as a framework "where you define datasets and queries, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints, and retries". The programming guide says SDP handles "the orchestration, compute management, and error handling automatically" and is designed for both batch and streaming: ingestion from cloud storage and message buses, plus incremental batch and streaming transformations.
You write only definitions. In this chain, nothing says which statement runs first:

CREATE STREAMING TABLE orders
AS SELECT * FROM STREAM orders_source;

CREATE MATERIALIZED VIEW customers
AS SELECT * FROM customers_source;

CREATE MATERIALIZED VIEW customer_orders
AS SELECT c.customer_id, o.order_number, c.state
FROM orders o
INNER JOIN customers c
ON o.customer_id = c.customer_id;

While the pipeline runs, SDP "analyzes the dependencies of your defined objects and orchestrates their order of execution and parallelization automatically".

How do the two approaches compare?

Concern	Hand-written Spark job	SDP
Order of steps	You sequence reads, transforms and writes in code	Inferred from the dependencies between datasets
Streaming checkpoints	You configure a checkpointLocation for each streaming query	One storage directory in the pipeline spec holds checkpoints for the streaming tables
Reruns	Reset and rerun logic is yours to write	Default incremental update, --refresh for chosen datasets, --full-refresh or --full-refresh-all to recompute
Checking before a run	Run the job	spark-pipelines dry-run reads and writes no data
Side effects	Any action or write is allowed	Dataset code must avoid collect(), count(), pivot(), toPandas(), save(), saveAsTable(), start() and toTable()
Source languages	Java, Scala, Python and R, plus SQL	Pipeline source files are .py or .sql

When is a hand-written job the better fit?

Stay with a hand-written job when any of these apply:

• Dataset code needs operations the guide says "should never be used in SDP code", such as collect(), count() or toPandas(), or writes with save() and saveAsTable().
• The logic uses SQL PIVOT, which "is not supported in SDP SQL".
• The output has to go to a sink that isn't append-only. SDP sinks are Python only, streaming only and append only.
• The code is Scala or Java, because pipeline source files are .py or .sql.

Moving to SDP is simplest when each step already reads its inputs and returns one DataFrame, because every Python dataset function in SDP "must return a pyspark.sql.DataFrame".

How Databricks approaches this

On Databricks, Lakeflow Pipelines "extend and are interoperable with Spark Declarative Pipelines, while running on the performance-optimized Databricks Runtime". The Lakeflow Jobs docs describe the difference: Lakeflow Jobs "provides a procedural approach to defining relationships between tasks", while Lakeflow pipelines "provide a declarative approach to defining relationships between datasets and transformations".
You can run a pipeline as a task in a job. In a triggered or scheduled job, the task "starts a single update and stops when the update completes". A pipeline's own mode is triggered (it stops after refreshing all tables) or continuous (it processes new data as it arrives), and Databricks recommends running continuous pipelines with a continuous job. Pipelines can also carry expectations, which apply data quality checks to each record, and the editor shows the table dependency graph. Databricks recommends serverless for new pipelines, and serverless pipelines always use Unity Catalog.

FAQs

Does SDP support streaming?

Yes. A streaming table is defined with a streaming read, spark.readStream in Python or the STREAM keyword in SQL. The guide says SDP loads data from "all the formats supported by Spark Structured Streaming", and its example reads from a Kafka topic.

Does SDP replace my scheduler?

No. The guide describes spark-pipelines run as launching "an execution of a pipeline" and monitoring its progress until it completes, and it doesn't describe a scheduler. Your existing scheduler starts each run. On Databricks, that can be a Lakeflow Jobs pipeline task.

Can I mix SDP and hand-written jobs?

Yes. Pipeline datasets are queryable, and pipeline code can read tables defined outside the pipeline in the same way as tables it defines itself. On Databricks, one job can combine a pipeline task with other tasks.