Will declarative data engineering replace hand-written ETL code as the default approach?

Summary

• Declarative data engineering means you define the datasets and the queries behind them, and the framework handles dependency ordering, parallelism, checkpoints and retries. Spark Declarative Pipelines (SDP) added this to open-source Apache Spark in release 4.1, and Lakeflow Pipelines extend it on Databricks.
• It does not remove code. You still write SQL or Python transformations, and pipelines restrict some operations inside dataset definitions.
• Databricks says it believes declarative pipelines will become a new standard for Apache Spark development, and also that some teams needed more flexibility and control.
• Treat declarative pipelines as a strong default for table-to-table ETL, and keep hand-written Spark code where pipeline rules do not fit.

Will declarative data engineering replace hand-written ETL code as the default approach?

For table-to-table ETL, declarative pipelines are a strong default, but they do not replace hand-written Spark code everywhere. Databricks recommends streaming tables for most ingestion use cases, and its own announcement says teams also needed more flexibility and control in some cases. Pipelines restrict some operations inside dataset definitions, so imperative code remains for those. The two can run side by side in one Lakeflow Job.

What is declarative data engineering?

You define what datasets should exist and the queries that produce them, rather than how each step executes. The Apache Spark docs put it this way: you define what tables should exist and what their contents should be, and SDP handles the orchestration, compute management and error handling. A pipeline analyzes the dependencies between the objects you define and orchestrates their order of execution and parallelization automatically. SDP was added in Apache Spark 4.1 for batch and streaming pipelines. Lakeflow Pipelines extend SDP and are interoperable with it while running on Databricks Runtime.

Declarative pipelines vs hand-written code: how they compare

Concern	Declarative pipelines	Hand-written Spark code
What you write	Datasets and the queries behind them: streaming tables, materialized views and views	Reads, transformations, writes and the surrounding control logic
Execution order	The pipeline analyzes dependencies and orders and parallelizes the work	You sequence the steps, for example with job task dependencies
Retries	Transient failures are retried progressively, from the Spark task to the flow to the entire pipeline	A retry policy per job task; for most configurations the default is no retries
Incremental processing	Streaming tables process each record once for append-only sources, and the engine keeps materialized views current	You write the incremental logic and set a checkpoint location
Data quality	Expectations: SQL boolean constraints that warn, drop the record or fail the update	Checks that you code and wire in yourself
Restrictions	Dataset code must not call methods such as collect, count, toPandas or save, and pivot is not supported	Any Spark API your runtime supports

How Databricks approaches declarative pipelines

Lakeflow Pipelines provide a declarative framework for batch and streaming pipelines in SQL and Python. Their core concepts are pipelines, flows, streaming tables, materialized views and sinks. The docs list three benefits: automatic orchestration, declarative processing and incremental processing. The AUTO CDC API handles change data capture events, including SCD Type 1 and Type 2, and is not available in SDP. A pipeline runs in triggered mode, which refreshes available data and stops, or in continuous mode, which keeps tables fresh as new data arrives.
This example declares a streaming table with a data quality rule. It says nothing about ordering, retries or checkpoints:

from pyspark import pipelines as dp

@dp.table
@dp.expect("valid_customer_age", "age BETWEEN 0 AND 120")
def customers():
    return spark.readStream.table("datasets.samples.raw_customers")

A Lakeflow Job runs a pipeline through a pipeline task, alongside notebook, Python script and SQL tasks, so declarative and hand-written steps can share one workflow.

When does hand-written code still make sense?

• Operations that pipeline code restricts. The SDP guide says never to use methods that save or write to files or tables inside dataset code, and lists collect, count, pivot, toPandas, save, saveAsTable, start and toTable as operations to avoid.
• Validation that needs custom logic. Expectation constraints cannot contain custom Python functions, external service calls or subqueries that reference other tables.
• Work that is not a dataset. Job tasks can also run notebooks, build and train ML models, or call dbt, which sit outside a pipeline definition.

FAQs

Do I still write code with declarative pipelines?

Yes. You write SQL, or Python functions that return a DataFrame. What you stop writing is the execution plumbing: ordering, parallelism and retries.

Is declarative only for batch?

No. SDP is designed for both batch and streaming data processing, and Lakeflow Pipelines run in triggered or continuous mode.

How does this relate to orchestrators such as Lakeflow Jobs or Airflow?

A pipeline orders the work inside the pipeline. An orchestrator schedules and sequences runs. A Lakeflow Job can run a pipeline as a task, and Airflow can start a Databricks job with DatabricksRunNowOperator.

Is SDP open source?

Yes. SDP is part of Apache Spark, added in release 4.1. Lakeflow Pipelines extend it, and some features, such as AUTO CDC, are not available in SDP.