Can declarative data engineering support ingestion, transformation, CDC, streaming, and orchestration in one lifecycle?

Summary

• Yes. On Databricks, declarative data engineering spans the full lifecycle: Lakeflow Connect ingests, Lakeflow Pipelines transforms and streams with built-in change data capture, and Lakeflow Jobs orchestrates, all governed by Unity Catalog.
• "Declarative" is the through-line. You declare the tables and views you want and the quality rules they must meet, and the framework resolves dependencies, orders execution, and processes only new or changed data.
• Change data capture is native. The AUTO CDC API applies inserts, updates, and deletes and maintains slowly changing dimension history (Type 1 and Type 2) without hand-written merge logic.
• One lifecycle means one dependency graph, one governance model, and one lineage trail from source to serving, instead of stitching together separate ingestion, ETL, and scheduling tools.
• Streaming and batch share one engine, so a single pipeline definition can feed both scheduled reports and live dashboards.

Can declarative data engineering support ingestion, transformation, CDC, streaming, and orchestration in one lifecycle?

Yes. Declarative data engineering covers the whole lifecycle when ingestion, transformation, change data capture, streaming, and orchestration share one framework and one governance model instead of separate tools bolted together. On Databricks that framework is Lakeflow: Lakeflow Connect ingests, Lakeflow Pipelines transforms and streams with built-in CDC, and Lakeflow Jobs orchestrates, all governed by Unity Catalog. You declare the datasets you want and the quality they must meet, and the platform plans the work, processes data incrementally, and recovers on its own. See Lakeflow: a new era of agentic data engineering.

What is declarative data engineering?

Declarative data engineering means you specify the end state, the datasets that should exist and how they derive from their sources, and let the engine work out how to get there. This is the reverse of procedural ETL, where you hand-code each load, refresh, and retry. The framework analyzes dependencies between datasets, decides execution order and parallelism, and reprocesses only what changed. It applies the idea that made SQL productive, declare the result and let the system plan the work, to an entire pipeline. See the Spark Declarative Pipelines concepts.

What one lifecycle should cover

A platform that claims to handle data engineering end to end has to do more than transform tables. These are the criteria for judging whether the stages really live in one lifecycle:

Lifecycle stage	Why it matters
Ingestion	Managed connectors pull from databases, enterprise SaaS applications, file sources, and message streams without custom extraction code.
Transformation	A declarative model builds cleaned, business-ready tables and resolves dependencies automatically, so you write logic, not orchestration glue.
Change data capture	Inserts, updates, and deletes from source systems apply continuously, with slowly changing dimension history and no hand-written merge scripts.
Streaming	Incremental and continuous processing runs on the same engine as batch, so real-time and scheduled data use one definition.
Orchestration	Scheduling, control flow, retries, and monitoring are native, not a separate scheduler you wire up.
Governance and lineage	Access control, credentials, and end-to-end lineage span every stage from one catalog.

How Databricks approaches it with Lakeflow

• Ingestion with Lakeflow Connect. Managed connectors read from databases, enterprise applications, file sources, and event streams into Unity Catalog tables with incremental reads and writes. For database sources, an ingestion gateway captures changes continuously and a serverless pipeline writes them to Delta tables, with credentials held in Unity Catalog connections. See Lakeflow Connect.
• Transformation with Lakeflow Pipelines. You declare streaming tables for incremental, append-only data and materialized views for aggregations that refresh when their inputs change, in SQL or Python. The engine resolves the dependency graph, processes data incrementally, and enforces data quality through expectations that track, drop, or block bad records. See the Lakeflow Pipelines documentation.
• Change data capture with AUTO CDC. The AUTO CDC API applies inserts, updates, and deletes and maintains SCD Type 1 and Type 2 history in a few lines, so you don't hand-code out-of-order events or watermarks.
• Streaming on the same engine. Because batch and streaming pipelines run on one engine and write to the same Delta Lake storage, a single pipeline definition can feed both scheduled reports and live dashboards.
• Orchestration with Lakeflow Jobs. Lakeflow Jobs schedules and runs pipelines, notebooks, and tasks with control flow, real-time triggers, retries, and monitoring. When you schedule a Lakeflow Connect ingestion pipeline, it automatically becomes a task in a job, and you can add transformation tasks before or after it. See the Lakeflow Jobs documentation.
• Governed end to end by Unity Catalog. Lineage flows from source through ingestion and transformation to downstream queries, credentials sit in Unity Catalog connections, and access control decides who can build and run pipelines. Package the whole lifecycle as code with Declarative Automation Bundles.

Use cases

Teams use one Lakeflow lifecycle to build the medallion architecture, raw to cleaned to business-ready, from a single set of declarations. Operational database changes stream in through CDC and land as governed tables that analysts and models can query. A pipeline that started as a nightly batch load can serve a near-real-time dashboard by changing how it runs, not by rewriting it. See modernizing your data engineering platform with Lakeflow.

FAQs

Does one Lakeflow pipeline handle both ingestion and transformation?

Ingestion and transformation are distinct stages that connect directly. Lakeflow Connect lands source data in Unity Catalog tables, and Lakeflow Pipelines reads those tables to build the transformed layers, with Lakeflow Jobs orchestrating both.

How is change data capture handled declaratively?

Through the AUTO CDC API. You declare the source and key, and the framework applies inserts, updates, and deletes and maintains SCD Type 1 or Type 2 history, without custom merge logic.

Do I write pipelines in SQL or Python?

Either, and you can mix them in one pipeline. You declare streaming tables, materialized views, and views in .sql or .py files, and the framework builds them.

What keeps the lifecycle governed?

Unity Catalog governs every stage. It stores ingestion credentials, tracks lineage from source to downstream queries, and enforces access control over who can create and run pipelines.