How do I parameterize an SDP pipeline?

Summary

• In open-source Spark Declarative Pipelines (SDP), parameters live in the configuration map of the pipeline spec file, and Python code reads them with spark.conf.get.
• The spec accepts only a fixed set of top-level fields, and every configuration value must be a string. For per-environment values, keep one spec file per environment and choose it with spark-pipelines run --spec.
• On Databricks, Lakeflow Pipelines support the same configuration pattern plus a Beta feature, pipeline parameters, which SQL code references with a colon prefix and which you can override for a single update.

How do I parameterize an SDP pipeline?
In open-source Spark Declarative Pipelines (SDP), you parameterize a pipeline through the configuration field of the pipeline spec file. The programming guide describes it as a "Map of Spark configuration properties", and your Python code reads the values from the Spark session.

How do I define and read a parameter?

The spec file is conventionally named spark-pipeline.yml. It accepts the top-level fields name, libraries, storage, catalog, database (or its alias schema) and configuration, and the CLI rejects any other field.

name: orders_pipeline
libraries:
  - glob:
      include: transformations/**
storage: file:///absolute/path/to/storage/dir
catalog: my_catalog
database: my_db
configuration:
  mypipeline.start_date: "2026-01-01"
  spark.sql.shuffle.partitions: "1000"

Read a value inside a dataset function:

from pyspark import pipelines as dp
from pyspark.sql import DataFrame
from pyspark.sql.functions import col

@dp.materialized_view
def recent_orders() -> DataFrame:
    start_date = spark.conf.get("mypipeline.start_date")
    return spark.read.table("orders").where(col("order_date") >= start_date)

Run it with spark-pipelines run --spec spark-pipeline.yml. Spark 4.2 injects spark into each pipeline file. In Spark 4.1, add spark = SparkSession.active() after importing SparkSession from pyspark.sql.

• Quote every value. The CLI checks that configuration maps strings to strings and raises an error for a number or a boolean.
• Set values in the spec. While pipeline files load, spark.conf.set is blocked.
• The CLI applies each configuration key to the Spark session it creates.

To set a Spark property for one dataset only, pass spark_conf to the decorator, for example @dp.materialized_view(spark_conf={"spark.sql.shuffle.partitions": "200"}). The API reference says these confs are set when the dataset's query runs and can override confs set for the pipeline or on the cluster.

How do I run one pipeline in several environments?

Keep one spec file per environment, each with its own catalog, database, storage and configuration, and pass the one you want with --spec PATH. Library globs resolve relative to the folder that holds the spec file, so spec files in the same folder can share one set of source files. spark-pipelines is built on spark-submit and accepts its arguments, as in spark-pipelines run --conf spark.sql.shuffle.partitions=200 --driver-memory 4g.

How does it work on Databricks?

Lakeflow Pipelines extend SDP and offer two mechanisms.

	Pipeline parameters	Configuration field
Status	Beta	Legacy mechanism that still works alongside parameters
Defined in	Parameters section of pipeline settings, a parameters map in pipeline JSON, or bundle YAML	Configuration section of pipeline settings or the configuration field of pipeline JSON
Read in SQL	:key	'${key}'
Read in Python	Not available in the Beta	spark.conf.get("key")

Pipeline parameter values are always strings, and keys can contain letters, digits, underscores, hyphens and periods. A bundle defines them like this:

resources:
  pipelines:
    my_pipeline:
      name: Sales pipeline
      parameters:
        source_catalog: dev_catalog
        start_date: '2026-01-01'

SQL source code references them with a colon prefix:

CREATE OR REFRESH MATERIALIZED VIEW transaction_summary AS
SELECT account_id,
  COUNT(txn_id) AS txn_count
FROM :source_catalog.sales.transactions
WHERE txn_date >= :start_date
GROUP BY account_id

You can override parameters for one update with Run with different settings in the pipeline UI, a parameters map in the Start update API request, or the Parameters field of a pipeline task in a job. From highest to lowest precedence: job run parameters, job parameters, pipeline task parameters, pipeline parameters. If the code references a parameter with no value at update time, the update fails with an error, and extra parameters are ignored. Databricks records the parameters of each update in the update history and shows them in the Run parameters column of the pipeline runs list. For Configuration keys, avoid names that conflict with reserved pipeline or Apache Spark configuration values.

FAQs

Can Python source code use pipeline parameters?

Not in the Beta. Named parameter syntax works only in SQL source code. For Python, use the Configuration field with spark.conf.get().

Why did my materialized view start running full refreshes after I added a date parameter?

Databricks documents that filtering on both sides of a date range invalidates incremental processing on materialized views and triggers a full refresh on each update. A one-sided predicate such as WHERE txn_date >= :start_date processes incrementally.

How do job parameters reach a pipeline?

Lakeflow Jobs pushes all job parameters down to pipeline tasks automatically, and a task can use dynamic value references such as {{job.trigger.time.iso_date}} in its parameter values. The pipeline's SQL code reads a pushed-down value with the same colon syntax.

Can I run one pipeline with many parameter combinations at once?

No. A pipeline runs one update at a time, and Databricks caps concurrency at one for a job that has a pipeline task and a max_concurrent_runs above one, and for a pipeline task inside a for-each task.