What does declarative data engineering mean, and how is it different from procedural ETL?

Summary

• Declarative data engineering means specifying what data transformations you want rather than writing detailed instructions on how to build them step-by-step.
• Procedural ETL requires you to orchestrate jobs, manage compute, handle retries, and monitor pipelines manually through code.
• Declarative platforms like Lakeflow Pipelines automate orchestration, compute allocation, error handling, and monitoring so you focus on business logic.
• The declarative approach is simpler to write, easier to maintain, and automatically optimized for performance, making it the modern standard for data engineering.

What does declarative data engineering mean, and how is it different from procedural ETL?
Declara data engineering is fundamentally about shifting the burden of orchestration and operations from the data engineer to the platform. Instead of writing imperative code that says "do this, then do that", you declare the desired outcome and let the system figure out the most efficient way to achieve it.
The difference is practical and profound. Procedural ETL puts engineers in control of execution but demands they manage every operational detail. Declarative engineering trades some control for automation, letting engineers focus on business logic instead of infrastructure.

Comparing the approaches

Aspect	Procedural ETL	Declarative (Lakeflow Pipelines)
What you write	Step-by-step orchestration code	Dataset definitions and transformations
Scheduling	You write the job scheduler calls	System handles scheduling automatically
Compute	You allocate and manage cluster resources	System optimizes allocation on your behalf
Failure handling	You code retry logic and error checks	System retries transient failures automatically
Optimization	You tune query plans and parallelism manually	System analyzes and optimizes the execution plan
Monitoring	You instrument logging and alerts	System provides built-in lineage and observability
Data quality	You write validation code	You declare expectations; system enforces them

Declarative data engineering in practice

In a procedural approach, a data engineer writes PySpark code that orchestrates jobs:

# Procedural: You manage orchestration
raw_df = spark.read.parquet("s3://data/raw/events")
raw_df.write.mode("overwrite").parquet("s3://data/bronze/events")

clean_df = spark.read.parquet("s3://data/bronze/events")
# Manual validation
assert clean_df.count() > 0, "No data in bronze layer"
clean_df.write.mode("overwrite").parquet("s3://data/silver/events")

The engineer must also manage job scheduling, retry policies, and error alerts outside the code, often through a separate orchestration tool.
With declarative pipelines, you describe datasets and dependencies:

# Declarative: The system handles orchestration
@table
def raw_events():
  return spark.read.parquet("s3://data/raw/events")

@table
def clean_events():
  return spark.read.parquet("s3://data/bronze/events") \
    .filter(col("timestamp").isNotNull())
  
clean_events.expect("count > 0")
clean_events.expect_or_drop("timestamp > '2020-01-01'")

The platform automatically handles scheduling, parallelism, retries, and data-quality enforcement. The system also generates a dependency graph showing how datasets connect, making lineage transparent and debuggable.

Why Databricks calls it Lakeflow Pipelines

Databricks' declarative platform is called Lakeflow Pipelines, built on Apache Spark Declarative Pipelines (SDP). The "Lakeflow" name emphasizes data flowing through the lakehouse as transformations. You define transformations declaratively; the system orchestrates the flow.
Lakeflow includes a dedicated development environment with an IDE-like editor. You write and organize transformations in files, and the editor shows the dependency graph in real time. You can run single files or entire pipelines, iterate quickly, and see lineage visually.

Operational benefits

Automatic optimization: The system analyzes your declared transformations and chooses the most efficient execution plan. Procedural code often requires manual query tuning; declarative systems optimize by default.
Resilience: Transient failures are retried automatically with progressive backoff. In procedural pipelines, you code this yourself or rely on external orchestrators.
Observability: Every transformation is tracked in a lineage graph. You can see which dataset depends on which, trace data quality issues to their source, and audit who changed what and when. Procedural pipelines require you to instrument logging.
Reduced complexity: You write less code. Business logic stands out; operational boilerplate is gone. This makes code reviews faster and makes onboarding new engineers simpler.
Compliance and auditability: Declarative systems can enforce policies uniformly. For instance, you declare a table should be GDPR-compliant, and the platform applies those rules across all dependent transformations. Procedural code often has inconsistent data-quality checks across different jobs.

When to use each approach

Use declarative (Lakeflow Pipelines) for:

• Regular ETL jobs: ingesting, transforming, and loading data
• Data warehousing: SQL transformations with dependencies
• Data quality monitoring: enforcing expectations and catching issues early
• Standard patterns: incremental loads, slowly changing dimensions, fact tables

Procedural ETL remains valuable for:

• Custom algorithms: machine learning feature engineering, anomaly detection
• Complex orchestration: conditional branching, dynamic parallelism, custom error handling
• One-off analysis: interactive exploration where you need precise control
• Legacy systems: migrating from existing procedural pipelines gradually

The shift in the industry

Declarat data engineering has become the modern standard because it aligns with how data teams actually work. Engineers want to focus on business logic and data quality, not infrastructure management. Declarative platforms deliver this by automating the operational burden.
Databricks' approach with Lakeflow Pipelines is to make declarative the default for routine ETL while preserving procedural capabilities for cases that require them. This hybrid model gives teams flexibility without forcing them to choose between simplicity and control.