How can I automatically detect and get alerted on data pipeline failures across my stack?

Summary

• Automatic detection needs three layers: monitoring that notices failed or slow runs, data checks that notice bad data in runs that succeed, and alert routing to the right people or tools.
• Lakeflow Jobs can notify on start, success, failure, a run that exceeds a duration threshold, and a streaming backlog, by email or to Slack, PagerDuty, or any webhook-based service.
• Lakeflow Pipelines expectations check each record and can warn, drop, or fail the update. Unity Catalog anomaly detection (Public Preview) flags tables that are stale or incomplete.
• For a view across your stack, query the system.lakeflow tables, which record job runs across the account, and use Databricks SQL alerts to notify on a condition you define.
• Genie ZeroOps is a background agent that investigates failures and proposes fixes. Its announcement describes a private preview, and nothing reaches production without approval.

How can I automatically detect and get alerted on data pipeline failures across my stack?

Combine three things: monitoring that notices a failed or slow run, data checks that notice bad data in a run that succeeded, and alert routing that sends each signal to the right people or tools. On Databricks, these map to Lakeflow Jobs notifications, Lakeflow Pipelines expectations with Unity Catalog data quality monitoring, and Databricks SQL alerts.

Which Databricks capability covers each kind of failure?

Failure to catch	Capability	What it does
Failed, slow, or backlogged job runs	Lakeflow Jobs notifications	Notify on start, success, failure, duration above a threshold, or streaming backlog
Failed pipeline updates	Pipeline email notifications	Email on a successful update, any failed update, a fatal failure, or a failed flow
Invalid records	Pipeline expectations	Apply a SQL constraint to each record, then warn, drop, or fail
Stale or incomplete tables	Anomaly detection (Public Preview)	Evaluates freshness and completeness from each table's history
Failures across jobs	system.lakeflow system tables	Record job runs and tasks across the account
Any custom condition	Databricks SQL alerts	Run a query on a schedule and notify when a condition is met

See Add notifications on a job, pipeline expectations, and Databricks SQL alerts for setup steps.

How do I set up failure alerts in Lakeflow Jobs?

Add notifications on a job or on individual tasks. Choose an email address or a system destination such as Slack, PagerDuty, or an HTTP webhook. An administrator must configure system destinations. A few documented behaviors shape what you receive:

• Job-level notifications are not sent when failed tasks are retried. Use task notifications to hear about every failed task, or select Mute notifications until the last retry to wait for the final attempt.
• Duration warnings need an expected completion time. A separate Timeout sets a maximum, and a job that exceeds it gets the status Timed Out.
• You can mute notifications for skipped or canceled runs.
• Retries are off by default for most configurations, so set a retry policy where retries make sense.
• Webhook payloads carry event types such as jobs.on_failure. Databricks recommends a user-defined webhook if you need a specific message schema.

How do I catch bad data in a run that succeeds?

A run can finish and still write invalid data, so add checks inside the pipeline. An expectation applies a SQL constraint to each record, and you choose the action: warn (the default) keeps invalid records, drop removes them, and fail stops the update. For example:

CONSTRAINT valid_count EXPECT (count > 0) ON VIOLATION FAIL UPDATE

Metrics for warn and drop appear in the pipeline UI and the event log, and a failed update can trigger the pipeline email notifications above. For custom alerting, Python event hooks (Public Preview) run on pipeline events, but Databricks does not guarantee that every queued event is processed before compute terminates. Anomaly detection (Public Preview) monitors the tables in a schema you enable it on. It marks a table stale when a commit is unusually late and incomplete when it receives fewer rows than its history predicts.

How do I see failures across my whole stack?

The system.lakeflow tables record job runs and tasks from all workspaces in your account that are deployed in the same cloud region. This query, adapted from a sample in the Databricks documentation, counts recent failed runs:

SELECT COUNT(DISTINCT run_id) AS failed_runs
FROM system.lakeflow.job_run_timeline
WHERE period_start_time > CURRENT_TIMESTAMP() - INTERVAL 1 DAY
  AND result_state IN ('FAILED', 'ERROR', 'TIMED_OUT')

Save it as a Databricks SQL alert with a condition on failed_runs, and it runs on a schedule and notifies you when the condition is met. External lineage extends the Unity Catalog lineage graph beyond Databricks.

Where does Genie ZeroOps fit?

Genie ZeroOps is an autonomous background agent that monitors jobs, pipelines, tables, and ML workloads. It investigates failures using metrics, events, logs, run history, and the Unity Catalog dependency graph, and it tests proposed fixes on a shallow clone in an isolated sandbox. Per the June 2026 announcement, nothing is applied to production without your approval, and early access is a private preview requested through your Databricks account team.

FAQs

Should I alert on every failure?

No. Use the documented filters: mute skipped or canceled runs, mute task notifications until the last retry, and set duration warnings only on jobs where a slow run matters.

Can alerts go to an on-call tool?

Yes. Job notifications can go to PagerDuty or any webhook-based service. An administrator configures these system destinations, with a maximum of three per event type for each job or task.

Does a failed expectation send an alert by itself?

Not on its own. The fail action stops the update, and you hear about it through the pipeline email notifications or job notifications that you configured.

How do I know what a failure affects?

Use Unity Catalog lineage. It captures lineage automatically for queries run on Databricks, down to the column level, so you can identify the downstream tables, jobs, and dashboards that depend on a source before you change or fix it.