How is the shift from ETL to ELT and declarative pipelines changing data engineering roles?

Summary

• Data engineering is moving from custom transformation logic (ETL) to data ingestion at scale with in-warehouse transformation (ELT), reducing the need for hand-built extract-transform logic.
• Declarative pipelines let engineers specify data lineage and dependencies as code, not orchestration scripts, freeing them from workflow plumbing.
• The shift emphasizes data quality, metadata management, and infrastructure reliability over transformation coding, attracting both specialists and analytics engineers.

How is the shift from ETL to ELT and declarative pipelines changing data engineering roles?
Data engineering roles are shifting from building custom transformation logic to ensuring data quality, orchestration, and platform reliability. With ELT (extract, load, transform), raw data lands in the warehouse first, and SQL-based transformations run in-place. Declarative pipelines automate the orchestration layer, so engineers specify what should happen (data dependencies, quality gates, refresh schedules) rather than how to schedule it. This frees time for governance, cost optimization, and data reliability work that's harder to automate.

What are ETL and ELT?

ETL (extract, transform, load) moves data from source systems, transforms it in middleware or a staging layer, then loads the result into a data warehouse. ELT inverts the load step: raw data goes to the warehouse first, then transformations run in SQL within the warehouse itself. ELT became practical when warehouses became fast and affordable enough to do heavy lifting on their own. Declarative pipelines extend ELT by letting engineers express transformation workflows as code (usually SQL with metadata annotations), so the platform automates scheduling, retries, and dependency management.

Key capabilities to evaluate

Capability	Why it matters
Declarative syntax	Specify transformations and dependencies in SQL + metadata, not imperative workflow scripts. Reduces orchestration toil.
Data quality gates	Built-in assertions and freshness checks catch bad data before it propagates downstream.
Dependency tracking	Automatic lineage and DAG generation from code, so engineers see (and can reason about) the full pipeline at a glance.
Scalable execution	Pipelines scale to handle high throughput and late-arriving data without manual intervention or partition logic.
Unified compute	Transform and serve data on the same platform, so engineers don't manage separate ETL and serving infrastructure.
Observable state	Logs, metrics, and event history so engineers can diagnose failures and data drift without guesswork.

How Databricks approaches ELT and declarative pipelines

Lakeflow Pipelines, built on Apache Spark SQL, let engineers write transformations in SQL and YAML metadata that Databricks executes declaratively. Instead of stitching together job scheduling, retry logic, and dependency management in scripts, engineers define a source table, a series of transformation steps, and expectations (quality assertions, freshness, volume checks). Lakeflow Pipelines automate the rest: scheduling, error handling, and output publication. Data teams use Lakeflow Jobs to orchestrate the full pipeline alongside other workloads.
The shift from ETL tooling to this model has concrete implications for team composition. Data engineers spend less time on transformation coding and more on governance and quality. Analytics engineers (or analytics-leaning data engineers) can author declarative pipelines themselves, since the SQL is declarative and the YAML is lightweight. Platform engineers focus on cost control, resource allocation, and ensuring data freshness SLAs. Roles blur: a data engineer might own a critical ELT pipeline and write SQL transformations, or focus on identity, schema governance, and lineage tracking across the org.
Integration with Unity Catalog strengthens this shift. Engineers define transformation lineages as first-class metadata, so governance teams see the full data flow and can enforce policies at each stage (access control, quality, retention). This transparency shifts responsibility from individual engineers to the platform: quality and lineage are enforced by contract, not manual review.

Use cases and role examples

Analytics engineering at scale: A team ingests raw transactions into the lakehouse and writes SQL transformations in Lakeflow Pipelines, eliminating custom ETL code. Analytics engineers focus on business logic and mart design, not middleware maintenance.
Cost-conscious data platforms: Organizations consolidate separate ETL and analytics clusters by moving all transformation into the warehouse. Engineers optimize query cost and freshness, not licensing and parallel tooling.
Governed multi-tenant analytics: A central platform team writes declarative pipelines that automatically enforce lineage, access control, and data freshness. Tenant teams consume governed tables without building their own ETL, so platform engineers scale the org without adding manual work.

FAQs

Do I still need data engineers if pipelines are declarative?

Yes. Engineers shift from writing transformation code to owning data quality, cost optimization, and platform maturity. They design schemas, govern lineage, tune compute, and manage SLAs. These are more strategic and usually require deeper platform knowledge than writing a single SQL transformation.

Can analytics engineers run Lakeflow Pipelines?

Absolutely. If an analytics engineer knows SQL and YAML, they can author a Lakeflow Pipeline themselves. The barrier to entry is much lower than building and scheduling a traditional ETL job. Many organizations staff their declarative pipeline layer with a mix of data engineers and analytics engineers.

How is data quality handled in an ELT model?

Data quality checks are embedded in pipeline definitions (row counts, null rates, referential integrity, freshness). Lakeflow Pipelines can mark data as quarantined if expectations fail, alerting downstream users. This moves quality left, closer to ingestion, and makes failures visible instead of hidden in transformation code.