My dbt models keep breaking in production. Is there a way to auto-diagnose or self-heal them?

Summary

• You can automate detection and diagnosis. dbt data tests, freshness checks, and model contracts catch common causes of breakage, and Lakeflow Jobs archives dbt logs and artifacts from job runs.
• Re-running is the self-healing that is safe to automate. Lakeflow Jobs can retry a failed dbt task, repair only the failed and dependent tasks, and notify an owner. A failing test usually needs a data or code fix.
• Run dbt Core as a dbt task in Lakeflow Jobs to get scheduling, retries, notifications, and run history around the project.
• Splitting a project into several dbt tasks with --select can let a repair run re-run only the part that failed.

My dbt models keep breaking in production. Is there a way to auto-diagnose or self-heal them?

Partly. Detection, diagnosis, and retrying transient failures can be automated. Fixing the cause of a failing test or a logic bug cannot, and an automatic fix there can hide the real problem. For schema changes, dbt lets you choose per incremental model whether to fail the build or adapt. The practical pattern is to fail fast, tell the owner exactly what failed, and re-run only what needs it.

Why do dbt models break in production, and what catches each cause?

Cause	What catches or prevents it
Source schema changed	Model contracts (enforced: true under contract) make the build fail when the output does not match the declared columns and data types. For incremental models, on_schema_change accepts ignore (the default), fail, append_new_columns, and sync_all_columns.
Source data is late	dbt source freshness (dbt freshness in dbt v2) compares sources with the warn_after and error_after thresholds you configure and reports a warning or error. A table update trigger starts a Lakeflow Jobs run when source tables are updated.
Bad values	The four built-in data tests: unique, not_null, accepted_values, and relationships. Use severity, warn_if, and error_if to decide which failures block the build.
Failing SQL	The SQL warehouse query history shows the SQL that dbt generated.
Missing permissions	The Run as principal needs CAN USE on the SQL warehouse and the Unity Catalog privileges the models require.

With dbt build, models, tests, snapshots, and seeds run in dependency order, and a failing test makes its downstream resources skip instead of building on bad data. Set a test's severity to warn if it should not cause skipping.

How do I run dbt in Lakeflow Jobs and diagnose a failure?

Add a dbt task to a Lakeflow Job to run a dbt Core project. The task runs the dbt commands you list, in order, and every command must start with dbt. The defaults are dbt deps, dbt seed, and dbt run, and commands such as build accept --select and --exclude. You select a serverless or pro SQL warehouse to run the SQL that dbt generates. Databricks recommends the dbt-databricks package, pinned to a specific version so development and production match.
When a run fails:

• Open the failed task's run page for its output and error message.
• Fetch logs and artifacts with databricks jobs get-run-output <task_run_id> --output JSON. Use the dbt task's run ID, not the parent run ID. The response includes logs and dbt_output.artifacts_link.
• Check the archived artifacts, which include logs, results, manifests, and configuration. dbt's run_results.json records timing and status for every node that executed, so it shows which model or test failed.
• Open the failed job in the Jobs UI and select Diagnose Error to have Genie Code help diagnose it.

What should retry automatically, and what should alert instead?

Situation	Automate	Why
Failure that clears on its own	Task retries. On serverless compute, auto-optimization retries failed tasks by default.	Retries suit failures that resolve without a code or data change.
Failing test or contract	Fail the run and notify the owner.	Re-running the same SQL on the same data usually repeats the failure.
One part of a multi-task job failed	Repair run.	It re-runs the unsuccessful tasks and their dependents only.
Source not ready	A table update trigger or a freshness check.	It avoids building on stale data.

These Jobs API task settings retry a dbt task up to twice and email an address when it fails:

{
  "task_key": "build_models",
  "dbt_task": {
    "commands": ["dbt deps", "dbt build"],
    "warehouse_id": "<warehouse-id>"
  },
  "max_retries": 2,
  "min_retry_interval_millis": 60000,
  "email_notifications": { "on_failure": ["oncall@example.com"] }
}

A repair restarts each unsuccessful task from the beginning, and Lakeflow Jobs does not make tasks idempotent. For incremental models, set unique_key so a re-run updates existing rows instead of appending duplicates.

FAQs

Can dbt retry a failed run by itself?

Yes. dbt retry re-executes the last invocation from the point of failure, using run_results.json to decide where to start. If nothing ran before the failure, for example because of a warehouse connection or permission error, there is nothing to resume from.

How do I keep one failing model from rebuilding the whole project?

Split the project into several dbt tasks, each with its own --select argument and dependencies. A repair run then re-runs the unsuccessful tasks and their dependents instead of the whole project.

Does Lakeflow Pipelines run dbt models?

dbt projects run as dbt tasks in Lakeflow Jobs. Lakeflow Pipelines is a separate declarative framework with its own expectations and automatic retries for transient failures.