How do you build declarative data pipelines?

Summary

• A declarative pipeline inverts the work: you declare the tables and views you want and how they derive from sources, and the framework resolves dependencies, orders execution, and processes data incrementally. You write business logic, not orchestration glue.
• You build one in SQL or Python by declaring three dataset types: streaming tables for incremental, append-only ingestion; materialized views for transformations that refresh when inputs change; and views for intermediate logic.
• Data quality goes inline. Expectations validate records as the pipeline runs and can track, flag, drop, or block bad rows, and the framework records the outcomes for you.
• On Databricks, declarative pipelines are Lakeflow Pipelines, built on the open Apache Spark Declarative Pipelines project, with AUTO CDC for change data capture, automatic orchestration, and a queryable event log.
• You develop in the Lakeflow Pipelines Editor with a multi-file SQL and Python workspace, an interactive dependency graph, data previews, and selective execution, then deploy to production as code with Declarative Automation Bundles and Git.

How do you build declarative data pipelines?

You build a declarative pipeline by declaring the datasets you want to exist, streaming tables, materialized views, and views, together with the queries that produce them, then letting the framework resolve dependencies, order execution, and process data incrementally. This is the reverse of hand-coding each step. Rather than writing procedural code that spells out how to load and refresh every table, you specify the end state and the engine builds an execution plan to reach it. The result is less orchestration glue, fewer custom retry and checkpoint scripts, and pipelines that backfill and recover on their own. On Databricks, declarative pipelines are delivered as Lakeflow Pipelines, built on the open Apache Spark Declarative Pipelines project.

What is a declarative pipeline?

A declarative pipeline is a collection of source code and configuration that declares datasets and the flows that populate them, not a script that runs steps in a fixed order. You define what each dataset is and how it derives from its inputs. The framework analyzes dependencies between datasets, determines execution order and parallelism, and reprocesses only new or changed data. It applies the idea that made SQL productive, state the result you want and let the system plan the work, to an entire pipeline. See procedural versus declarative approaches.

What to get right when building declarative pipelines

Capability	Why it matters
Dataset declarations	Streaming tables, materialized views, and views cover ingestion, transformation, and intermediate logic without hand-written state management.
Dependency resolution and orchestration	The framework should infer the dependency graph and run steps in order, so you don't wire up a separate orchestrator.
Incremental processing	Reprocessing only new or changed data keeps refreshes efficient and handles late-arriving records and backfills.
Inline data quality	Rules that validate records during the run, and act on failures, stop bad data before it reaches downstream tables.
Change data capture	Built-in CDC, including SCD Type 1 and Type 2, removes fragile hand-written merge logic.
Recovery and observability	Automatic checkpointing, retries, and an event log make runs resilient and auditable.

How Databricks builds declarative pipelines with Lakeflow Pipelines

• Declare datasets in SQL or Python. A pipeline is a set of .sql and .py files that declare streaming tables, materialized views, and views. Streaming tables handle incremental, append-only ingestion (with Auto Loader, Kafka, and change feeds), processing each row exactly once; materialized views refresh automatically when their inputs change; views hold intermediate logic.
• Add data quality with expectations. You declare expectations inline to track, flag, drop, or block records that fail your rules, and the framework tracks the outcomes for monitoring.
• Handle change data capture with AUTO CDC. The AUTO CDC API applies inserts, updates, and deletes and maintains SCD Type 1 and Type 2 history in a few lines, without hand-coding out-of-order events or watermarks.
• Let the engine orchestrate. Lakeflow Pipelines infers dependencies, builds one execution plan, and runs flows in the right order with parallelism, managing checkpoints, exactly-once processing, retries, and schema changes for you.
• Build in the Lakeflow Pipelines Editor. The editor gives you a multi-file SQL and Python workspace, an interactive graph of dataset dependencies and state, data previews, selective execution to refresh a single table, and inline debugging, with Git built in.
• Deploy as code. Package pipelines with Declarative Automation Bundles and Git for repeatable promotion from development to production. Pipelines run on managed compute, stay governed by Unity Catalog, and expose a queryable event log for monitoring. See the Lakeflow general availability announcement.

FAQs

Do I build declarative pipelines in SQL or Python?

Either, and you can mix them in one pipeline. You declare streaming tables, materialized views, and views in .sql or .py files, and the framework builds them.

What dataset types can I declare?

Three: streaming tables for incremental, append-only ingestion; materialized views for transformations that refresh when their inputs change; and views for intermediate logic reused elsewhere in the pipeline.

How is data quality enforced?

Through expectations, rules declared inline that validate each record as the pipeline runs. Failing records can be tracked, dropped, or blocked, and the results are recorded so you can monitor quality over time.

How do declarative pipelines relate to Apache Spark?

Lakeflow Pipelines are built on the open Apache Spark Declarative Pipelines project, so transformation code stays portable across runtimes while running on the performance-optimized Databricks Runtime.