How do I reduce the overnight on-call burden when my data pipelines break?

Summary

• Page people only for failures that need a human. Put failure notifications on tasks, mute them until the last retry, and use duration warnings for runs that hang instead of failing.
• Let retries absorb transient failures: task retry policies in Lakeflow Jobs, serverless auto-optimization, and automatic flow and update retries in Lakeflow Pipelines.
• Contain failures. With the default task dependency, downstream tasks do not run after an upstream failure, and pipeline expectations can drop or fail bad records.
• Cut recovery time with Repair run and Refresh failed tables, which re-run only what failed and what depends on it.
• Speed up diagnosis with the run page, the pipeline event log, system tables, and Diagnose Error.

How do I reduce the overnight on-call burden when my data pipelines break?
On-call load comes from alerts that did not need a human, failures that spread to downstream tables, and slow diagnosis once someone is paged. Lakeflow Jobs and Lakeflow Pipelines have documented settings for each. Work through them in the order of an incident: alert, retry, contain, diagnose, recover.

Which failures should wake someone up?

Jobs can send notifications when a run or task starts, succeeds, fails, or exceeds a duration threshold. Destinations are email addresses or third-party systems such as Slack, PagerDuty, or any webhook-based service. An administrator configures system destinations, and each job or task allows up to three per event type. These settings cut noise:

• Retries: job-level notifications are not sent when failed tasks are retried. Use task notifications to hear about every failed task, or select Mute notifications until the last retry to hear only about the final failure.
• Skipped and canceled runs: select Mute notifications for skipped runs or Mute notifications for canceled runs.
• Slow runs: set an expected duration, then select Duration warning to be told when a run exceeds it. A Timeout stops a task that never finishes and sets its status to Timed Out.
• Pipelines: email notifications can fire when an update fails with a retryable or non-retryable error, only when it fails with a non-retryable (fatal) error, or when a single flow fails.

How do retries reduce alerts?

Mechanism	What it does
Task retry policy	Re-runs a failed task. For most configurations the default is no retries, so you add a policy.
Serverless auto-optimization	On serverless compute for workflows, retries failed tasks. It is on by default and can be turned off for tasks that must run at most once.
Continuous jobs	Retries with exponential backoff when consecutive failures exceed a threshold.
Lakeflow Pipelines	Retries transient failures progressively, from the Spark task to the flow to the entire pipeline. Updates started from a job or the Pipelines API, and all continuous updates, retry automatically.

For pipelines, pipelines.maxFlowRetryAttempts defaults to two retries per flow, and pipelines.numUpdateRetryAttempts defaults to five for triggered pipelines. Manual updates of non-continuous pipelines from the pipeline editor disable retries so you see errors immediately.

How do I stop one failure from spreading?

• Task dependencies: the default Run if condition is All succeeded, so a task whose upstream task failed does not run and is marked Upstream failed. Conditions such as All done or At least one failed let you run cleanup or alerting tasks after a failure.
• Data quality gates: pipeline expectations keep invalid records and record metrics (the default), drop them, or fail the update. In a triggered pipeline, a failed expectation fails and rolls back only that flow, and other flows keep updating. To stop downstream work, split the validation and the downstream tables into separate pipelines, and make the downstream pipeline task depend on the validation task in a job.
• Right trigger: a table update trigger starts a job when source tables are updated instead of at a fixed time.

How do I diagnose a failure quickly?

For a job, open the failed run, click the failed task, and read its output and error message. Select Diagnose Error on the failed job to have Genie Code help.
For a pipeline, failed tables and flows are highlighted on the pipeline graph, and View logs opens the event log. The event log holds the full error, data quality metrics, and lineage events, and the UI can truncate long messages.
For trends across everything, query system.lakeflow.job_run_timeline and system.lakeflow.pipeline_update_timeline. Job records are typically available within one hour.

How do I recover without redoing everything?

For a multi-task job, Repair run re-runs only the unsuccessful tasks and their dependents. For a pipeline, Refresh failed tables updates only the failed tables and anything downstream of them. For jobs, make writes idempotent first, because a repair restarts each unsuccessful task from the beginning and Lakeflow Jobs does not make tasks idempotent. For code or configuration problems in a pipeline, run Validate first to re-check the graph and source code without materializing data.

FAQs

Can pipelines recover from every failure automatically?

No. Retries cover specific recoverable errors, such as a failure to start a cluster. An expectation set to fail the update needs manual intervention before reprocessing, and an invalid streaming checkpoint needs a full refresh or a checkpoint reset because a retry alone will not fix it.

Does serverless compute remove cluster-related alerts?

Serverless compute for workflows lets you run a job without configuring and deploying infrastructure, and Databricks optimizes the compute for you. It does not remove failures in your code or data, and jobs that need large amounts of memory or include many tasks can see longer startup times.

How do I get alerted when a table goes stale but no job fails?

Anomaly detection in data quality monitoring (Public Preview) learns each table's commit history and row counts. It marks a table stale when a commit is unusually late, and incomplete when the last 24 hours of rows fall below the predicted range. Alert rules that notify users by email, Slack, or webhook destinations are in Beta.