What's the best way to manage data pipeline errors and fix them at scale?

Summary

• Sort errors into classes and automate the response for each: retry what is transient, enforce rules on records, notify an owner for the rest, and re-run only what failed.
• Lakeflow Pipelines retries recoverable failures automatically for updates started from a job or the Pipelines API, and for all continuous updates.
• Expectations decide what happens to invalid records: keep and record them, drop them, or fail the update.
• Schema drift is expected. With schema inference, Auto Loader's default mode adds new columns and stops the stream with an UnknownFieldException, and the stream resumes after a restart. Data that does not match the schema can be rescued instead of dropped.
• At scale, query events instead of reading logs: the system.lakeflow tables for jobs and pipeline updates, and the pipeline_events system table (Beta) for pipeline events.

What's the best way to manage data pipeline errors and fix them at scale?
Failures are routine, so the goal is to classify them, automate the response to each class, and save human attention for the class that needs judgment. Databricks gives you a response for each class, from automatic retries to targeted re-runs.

What kinds of pipeline errors should I plan for?

Error class	Example	Automated response
Recoverable infrastructure failure	A failure to start a cluster, a memory leak, stale credentials	For updates started from a job or the Pipelines API, and all continuous updates, Lakeflow Pipelines retries specific errors and restarts the cluster for specific recoverable errors. Lakeflow Jobs tasks retry according to the retry policy you set.
Invalid records	A value that breaks a business rule	Expectations: warn (keep and record), drop, or fail the update.
Schema change	A new column appears in incoming files	Auto Loader adds the column to the schema, then stops the stream with an UnknownFieldException. Restarting resumes with the new schema.
Code or configuration error	A wrong table or column name	Run Validate to check the graph and source code without materializing data.
Upstream not ready	Source tables or files arrive late	Table update and file arrival triggers start a job when the data arrives.
Invalid streaming checkpoint	A changed stateful operation or source	A retry alone will not fix it. Run a full refresh or a selective checkpoint reset.

How do I get the right person notified?

Lakeflow Jobs can notify on failure by email, Slack, PagerDuty, or any webhook-based service. Job-level notifications are not sent when failed tasks are retried, so use task notifications when you need every failure. Pipelines can send email when an update fails with a retryable or non-retryable error, only when it fails with a non-retryable (fatal) error, or when a single flow fails.

How do I find and fix a failed update quickly?

1. Start at the Issues pane in the pipeline editor or the bottom pane of the monitoring page. Failed tables and flows are highlighted on the graph.
2. Open the event log with View logs. It carries the full error, and the UI can truncate long messages.
3. Fix the code or data, then use Refresh failed tables to update only the failed tables and their downstream dependencies.
4. For a code or configuration problem, run Validate before a full update.

How do I manage errors across many pipelines?

Query events instead of reading logs one pipeline at a time:

• system.lakeflow.pipeline_update_timeline records each pipeline update with its result_state (COMPLETED, FAILED, or CANCELED), and system.lakeflow.job_run_timeline does the same for jobs.
• The pipeline_events system table (Beta) holds events for all pipelines across the workspaces in a region and is the recommended way to query events. During the Beta, its path is system.lakeflow_pipeline_events_preview.pipeline_events, and at general availability it moves to system.lakeflow.pipeline_events.
• The audit log system table, system.access.audit (Public Preview), records who created, edited, started, or stopped a pipeline under the deltaPipelines service name, which helps post-mortems.

SELECT workspace_id, pipeline_id, COUNT(DISTINCT update_id) AS failed_updates
FROM system.lakeflow.pipeline_update_timeline
WHERE period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
  AND result_state = 'FAILED'
GROUP BY ALL
ORDER BY failed_updates DESC

How do I recover without reprocessing everything?

Use Refresh failed tables for pipelines and Repair run for multi-task jobs. Both re-run only what failed and what depends on it. For jobs, make writes idempotent first, because a repair restarts each unsuccessful task from the beginning and Lakeflow Jobs does not make tasks idempotent. Run full refreshes only when necessary: a full refresh of a streaming table can drop records whose source data no longer exists, for example after a short Kafka retention period or a storage lifecycle policy. To block full refreshes on a table, set pipelines.reset.allowed to false.

FAQs

Should I retry automatically or require approval?

Retry automatically for recoverable failures, which Lakeflow Pipelines does for updates started from a job or the Pipelines API. Do not retry data quality failures. An expectation that fails the update requires manual intervention before reprocessing.

How do I handle schema changes without breaking pipelines?

When you do not provide a schema, Auto Loader's default addNewColumns mode adds new columns and then fails the stream with an UnknownFieldException. Databricks recommends running Auto Loader streams in Lakeflow Jobs so they restart automatically. Data that does not match the schema, such as a type mismatch, goes to the _rescued_data column instead of being dropped. Set cloudFiles.schemaEvolutionMode to rescue, failOnNewColumns, or none for other behavior.

Can pipelines share a workflow with other tools?

A job can combine pipeline tasks with dbt, notebook, and other task types, and Apache Airflow can trigger Databricks jobs.