Is there a way to auto-fix my pipelines?

Summary

• Yes, in layers. Lakeflow Pipelines retries and restarts automatically on retryable failures in job-run, API-started, and continuous updates, expectations keep invalid records out of your tables, and Genie Code can diagnose and fix pipeline errors with your approval.
• Retry limits are pipeline properties. A flow gets two retry attempts by default, and an update gets five retries on a triggered pipeline and unlimited retries on a continuous pipeline.
• Expectations let you warn on, drop, or fail on invalid records, so data problems do not flow downstream silently.
• Genie ZeroOps, announced on June 16, 2026, is a background agent that proposes fixes and validates them in isolation. It is in private preview, and nothing is applied to production without your approval.

Is there a way to auto-fix my pipelines?

Yes, for failures that do not need a code change, and with your approval for the ones that do. Lakeflow Pipelines retries and restarts on its own when a failure is retryable and the update came from Lakeflow Jobs, the Pipelines API, or a continuous pipeline. When the cause is a bug in the pipeline code, Databricks gives you tools that diagnose the problem and propose a fix, and a person approves the change.

What can be fixed automatically, and what cannot?

Auto-repair has four layers, from fully automatic to human-approved:

1. Retries and restarts handle recoverable infrastructure errors.
2. Expectations stop invalid records from spreading.
3. Partial re-runs let you repeat only the parts that failed.
4. Agent-assisted fixes diagnose code and logic errors and propose a change.

Some failures still need a person. Incorrect transformation logic needs a code change, and an expectation set to fail the update stops it when invalid records appear: the documentation says "Manual intervention is required."

Key capabilities to evaluate

Capability	Why it matters
Automatic retry and restart	Recoverable failures resolve without anyone being paged
Configurable retry limits	Teams can bound how long a pipeline retries
Data quality enforcement	Invalid records are flagged, dropped, or stopped before consumers see them
Partial re-run	A fix does not require recomputing work that already succeeded
Error diagnosis	Root cause is easy to find
Human approval for changes	Automated changes to production get reviewed
Alerting	Failures that need a person reach one

How does Databricks approach auto-repair?

Automatic retry and restart. Updates started from Lakeflow Jobs or the Pipelines API, and all updates for continuous pipelines, use automatic retry and restart behavior. The cluster restarts for specific recoverable errors, including memory leaks and stale credentials, and execution is retried for specific errors, such as a failure to start a cluster. Two pipeline properties set the limits: pipelines.maxFlowRetryAttempts (default: two retry attempts, so a flow runs up to three times including the original) and pipelines.numUpdateRetryAttempts (default: five for triggered pipelines, unlimited for continuous pipelines). Non-continuous updates started from the editor, and Validate updates, are not retried, so you see errors immediately.
Expectations. Expectations apply data quality checks to each record passing through a materialized view, streaming table, or view. The default action is warn, which writes invalid records to the target. You can instead drop invalid records or fail the update. In Python, use the @dp.expect_or_drop decorator. In SQL, the documentation shows:

CONSTRAINT valid_current_page EXPECT (current_page_id IS NOT NULL and current_page_title IS NOT NULL) ON VIOLATION DROP ROW

Partial re-runs. If an update fails because of errors in one or more tables, click Refresh failed tables on the pipeline monitoring page to update only the failed tables and their downstream dependencies. In Lakeflow Jobs, Repair run reruns only the unsuccessful tasks of a multi-task job and their dependent tasks, each from the beginning, so make tasks safe to rerun.
Genie Code in the Lakeflow Pipelines Editor. The Issues panel lists errors, warnings, and insights across the pipeline's files. Click Diagnose error to troubleshoot with Genie Code, or use Quick fix on a code error. In Agent mode you can ask it to "Fix the failure in this pipeline." Genie Code asks for approval before it runs code or updates pipelines, and a dry run validates the pipeline without updating data.
Genie ZeroOps. Databricks announced Genie ZeroOps on June 16, 2026 as a background agent that monitors data and AI assets, starting with jobs, pipelines, tables, and ML workloads. It uses Unity Catalog lineage to trace a failure to its root cause, generates a fix, and tests it against shallow clones of production data in isolation. The announcement states: "Nothing is applied to production without your approval." Genie ZeroOps is in private preview, with early access requested through your Databricks account team.

FAQs

How does automatic error recovery work?

For updates that use automatic retry and restart, the pipeline restarts the cluster for specific recoverable errors and retries specific failures within the retry limits. If a failure persists, the update fails and the event log records the outcome.

What types of failures can auto-repair handle?

Retry and restart behavior covers recoverable errors such as memory leaks, stale credentials, or a cluster that fails to start. Expectations handle invalid records. A failure caused by incorrect pipeline logic needs a code change, which Genie Code can propose and a person approves.

Do I need to configure anything?

Retry behavior has defaults that you can change with the pipeline properties above. Expectations are optional and added per dataset.

How do I monitor pipeline health?

The pipeline monitoring UI shows the pipeline graph color-coded by status, per-table row counts and data quality metrics, update history, and streaming backlog metrics. The event log is a structured Delta table you can query for update outcomes, data quality trends, and resource usage. To alert on pipeline events, configure notifications on the pipeline, or as job notifications when it runs inside a Lakeflow Job.