How do declarative pipelines handle schema evolution, data quality, backfills, and failed runs?

Summary

• Declarative pipelines make resilience the default: you declare the tables, quality rules, and sources, and the framework handles schema changes, quality enforcement, incremental refresh, and retries without hand-written orchestration or merge code.
• Schema evolution is automatic for compatible changes like adding a column or widening a type from INT to BIGINT; narrowing, incompatible type changes, renames, and hard deletes require a full refresh, which Lakeflow Connect can trigger automatically.
• Data quality runs on expectations: per-row rules that warn and keep a row, drop it, or fail the update, with pass and fail counts written to the event log every run.
• A backfill is a full refresh that reprocesses data from the source; normal runs process only new or changed data, so you full-refresh only when logic, schema, or source state changes.
• Failed runs retry automatically from the Spark task to the flow to the whole pipeline, and automatic checkpoints with atomic Delta commits give exactly-once processing so a restart doesn't duplicate or lose data.

How do declarative pipelines handle schema evolution, data quality, backfills, and failed runs?

Declarative pipelines treat these operational concerns as built-in behavior, not code you write and maintain. You declare the tables you want, the quality rules they must meet, and the sources they read; the framework works out execution order, processes only what changed, enforces your rules on every row, evolves the schema where it safely can, and retries failures on its own. On Databricks that framework is Lakeflow Pipelines, built on the open Apache Spark Declarative Pipelines standard. Schema drift, bad records, backfills, and transient errors become engine behavior instead of bespoke orchestration and merge logic.

Key capabilities to evaluate

Capability	Why it matters
Automatic schema evolution	Additive source changes shouldn't break the pipeline or force a manual migration for every new column.
Declarative data-quality rules	Constraints enforced on every row of every run keep bad data from silently reaching downstream tables.
Incremental processing	Reprocessing only new or changed data controls cost and runtime as volumes grow.
Explicit, safe backfills	A full reprocess from source should be a deliberate, repeatable action, not an ad hoc script.
Retries, recovery, and audit	Transient failures should retry on their own, a failed run should resume without duplicating or losing data, and an event log should record what happened.

How Databricks approaches this with Lakeflow Pipelines

Schema evolution. Lakeflow Pipelines evolve the schema automatically for compatible changes: new columns are added as they appear, and types can widen, for example from INT to BIGINT. Changes that can't be applied safely, such as narrowing a type, incompatible type changes, renames or hard deletes, and changes to deduplication or aggregation keys, require a full refresh so the table rebuilds correctly. Databricks recommends adding a new column with the new type or name and exposing a view that unions old and new values. Ingestion built with Lakeflow Connect can enable auto full refresh to recover from unsupported schema changes automatically. See when to run a full refresh.
Data quality with expectations. You attach expectations to a dataset as declarative constraints, and the pipeline evaluates them on every row of every run. Each expectation sets an action for a violating row: warn and keep it while recording the violation, drop it from the target, or fail the update so bad data never lands. A SQL expectation reads like CONSTRAINT valid_email EXPECT (email LIKE '%@%') ON VIOLATION DROP ROW. Pass and fail counts go to the pipeline event log, so data quality is a continuous, measured property rather than a one-time check. Teams usually put the strictest gates at the bronze-to-silver boundary. See pipeline expectations.
Backfills and refresh. Normally, streaming tables and materialized views process only new or changed data, which keeps runs fast and inexpensive. A backfill is a full refresh: the pipeline reprocesses data from the source and rebuilds the table. Full refresh is explicit, from the pipeline UI or API or auto full refresh in Lakeflow Connect, and it's the right move when a change would make previously processed data inconsistent, such as new business logic, a schema or partitioning change, an upstream source change, or a lost checkpoint. Materialized views refresh incrementally when the query and sources allow and recompute in full otherwise, returning the same result either way. See the pipeline lifecycle.
Failed runs, retries, and recovery. Lakeflow runs each step in dependency order and retries transient failures progressively, from the Spark task to the flow to the whole pipeline, so a brief outage doesn't require a manual restart. Development mode fails fast and keeps compute running for debugging; production mode applies more retries and releases compute when the run finishes. For managed Delta-to-Delta tables, each micro-batch commits its inputs and outputs together with automatic checkpoints, giving exactly-once processing so a restart doesn't double-count or drop rows. At custom sinks or non-Delta targets, writes are at-least-once, so make them idempotent by upserting on a key. Every run streams progress, quality results, and errors to an event log you can query. See the benefits of pipelines.

FAQs

Do declarative pipelines evolve the schema automatically?

Yes, for compatible changes. New columns are added automatically and types can widen, such as INT to BIGINT. Narrowing, incompatible type changes, renames, hard deletes, and changes to deduplication or aggregation keys require a full refresh, which Lakeflow Connect can trigger automatically.

What happens to rows that fail a data-quality expectation?

It depends on the action you set: warn and keep the row while recording the failure, drop it from the target table, or fail the update so no bad data is written. Pass and fail counts are recorded in the event log for every run.

How do I run a backfill, and how does a run recover from failure?

Run a full refresh from the pipeline UI or API to reprocess from source and rebuild the target; normal runs process only new or changed data, so reserve full refresh for changed logic, schema, or source state, or a lost checkpoint. When a run fails, the framework retries transient errors on its own, and automatic checkpoints with atomic Delta commits let a restart resume without duplicating or losing data.