What are Spark Declarative Pipelines, and what problem do they solve?

Summary

• Spark Declarative Pipelines (SDP) are a declarative framework for building and running batch and streaming data pipelines in SQL and Python. On Databricks they are delivered as Lakeflow Spark Declarative Pipelines.
• You declare the datasets you want — streaming tables, materialized views, and views — and the engine figures out how to build them, inferring dependencies, ordering execution, and processing data incrementally.
• The problem they solve: they remove the operational burden of stitching together tools for orchestration, incremental processing, data quality, backfills, and recovery, so engineers write business logic instead of glue code.
• Built-in capabilities include automatic orchestration and dependency resolution, inline data quality expectations, an incremental processing engine, AUTO CDC (SCD Type 1 and 2), and a queryable event log.
• They extend Apache Spark Declarative Pipelines, so your transformation code stays portable across SDP runtimes while running on the performance-optimized Databricks Runtime.

What are Spark Declarative Pipelines, and what problem do they solve?

Spark Declarative Pipelines (SDP) are a declarative framework for developing and running batch and streaming data pipelines in SQL and Python. On Databricks, they are delivered as Lakeflow Spark Declarative Pipelines, which extend the open Apache Spark Declarative Pipelines project with production-grade features. A pipeline is a collection of source code and configuration that declares datasets — streaming tables, materialized views, and views — along with the queries that produce them. Rather than writing step-by-step code that says how to build each table, you declare what datasets should exist and how they are derived, and the engine determines the execution plan.

What problem do they solve

Data engineering teams spend much of their time on the operational burden of stitching tools together rather than writing business logic. With traditional hand-coded pipelines, engineers manually encode execution order, incremental processing, and data-quality logic, and rely on separate orchestrators for retries, alerting, and monitoring. As data volumes and complexity grow, this burden compounds and turns data engineering into a bottleneck. Spark Declarative Pipelines apply the same idea that made SQL powerful — you specify the result you want and the system works out how to produce it — to entire pipelines. See why data engineering needs to become end-to-end declarative and the procedural and declarative approaches explained in the docs.

Why Databricks Lakeflow Spark Declarative Pipelines

• Automatic orchestration and dependency management. The framework infers dependencies between datasets, builds a single execution plan, and runs the steps (called flows) in the correct order with maximum parallelism — no external orchestrator to wire up. See the benefits of pipelines.
• Incremental processing built in. You write transformation logic with batch semantics, and the incremental processing engine keeps materialized views current by reprocessing only new or changed source data, handling backfills and late-arriving data without manual intervention.
• Inline data quality with expectations. Data-quality rules are expressed inline as expectations that can flag, drop, or block bad rows, and the framework tracks them automatically.
• Streaming tables and materialized views. Streaming tables handle incremental, append-only processing (with Auto Loader, Kafka, and change feeds supported declaratively, each row processed exactly once), while materialized views are derived tables that refresh automatically when their dependencies change.
• AUTO CDC. The AUTO CDC API handles change data capture, including SCD Type 1 and Type 2, without hand-written code for out-of-order events or streaming concepts like watermarks.
• Batch and streaming together. SDP natively supports both batch and streaming flows, covering batch ingestion from cloud storage and incremental ingestion from message buses such as Kafka, Kinesis, and Pub/Sub.
• Automatic recovery and observability. The framework provides automatic checkpointing, progressive retries (from the Spark task to the flow to the whole pipeline), and a queryable event log for monitoring.
• Open and portable. Because Lakeflow pipelines build on Apache Spark Declarative Pipelines rather than proprietary APIs, your transformation code stays portable across SDP runtimes while running on the performance-optimized Databricks Runtime. See bringing declarative pipelines to Apache Spark.

Getting started

1. Read the Spark Declarative Pipelines concepts to understand pipelines, flows, and datasets.
2. Create a pipeline and declare your datasets in SQL or Python — start with a streaming table for ingestion and materialized views for transformations.
3. Add data-quality expectations inline, and use AUTO CDC where you need change data capture.
4. Run the pipeline; the engine resolves dependencies, orchestrates execution, and processes data incrementally.
5. Explore recurring patterns in modernizing SQL ETL with declarative patterns.

FAQs

What is the difference between declarative and hand-coded pipelines?

With a declarative pipeline you specify what datasets should exist and how they are derived, and the engine handles execution order, incremental processing, and optimization. Hand-coded pipelines require you to encode all of that manually and to manage a separate orchestrator.

Do Spark Declarative Pipelines support both batch and streaming?

Yes. SDP natively supports both batch and streaming flows in the same framework, including batch ingestion from cloud storage and incremental ingestion from message buses like Kafka, Kinesis, and Pub/Sub.

How do Spark Declarative Pipelines relate to Apache Spark?

Lakeflow Spark Declarative Pipelines are built on the open Apache Spark Declarative Pipelines project and extend it with production features such as AUTO CDC, data-quality expectations, and a queryable event log, while keeping transformation code portable across SDP runtimes.

What do they handle automatically?

Dependency resolution and orchestration, incremental processing, backfills and late-arriving data, checkpointing and retries, data-quality tracking, and event logging for monitoring.