My Airflow DAG keeps failing intermittently and I can't find the root cause. What tools can help me investigate faster?

Summary

• Intermittent Airflow failures usually fall into a few classes: lost workers, timeouts, flaky external services, capacity limits, and inputs that differ between runs. Each class leaves different evidence.
• Start with the Grid view, one log per task attempt, and callbacks. Compare durations across runs, and check for task instance heartbeat timeouts, which the Airflow docs tie to workers that ran out of memory or were moved.
• If the failing task triggers Databricks, follow the run URL the Databricks operators write to the Airflow log, then use the Jobs UI, error codes, and system tables to find patterns.
• Lakeflow Jobs can orchestrate Databricks workloads with matrix run history, repair runs, and duration warnings, and Airflow can keep triggering those jobs.

My Airflow DAG keeps failing intermittently and I can't find the root cause. What tools can help me investigate faster?

Start with the evidence Airflow already keeps: Grid view history, one log per task attempt, and task durations across runs. Then sort the failures into classes, because each class points to a different tool. If the failing task triggers Databricks work, that run has its own history, error codes, and system table records.

What causes intermittent Airflow DAG failures?

Failure class	What the Airflow docs say	Where to look
Worker lost mid-task	A task instance heartbeat timeout marks the task failed or retries it. Documented causes include a worker that ran out of memory (OOMKilled), failed its liveness probe, or was moved by the system	Worker or pod events and memory metrics, not only the task log
Timeouts	execution_timeout raises AirflowTaskTimeout. A sensor timeout (reschedule mode) raises AirflowSensorTimeout, which fails without retrying	Task durations across recent runs
Flaky external service	Retries use a fixed count and delay whatever the error type, unless a retry policy says otherwise	The log of each attempt; retry policies that act per exception type
Capacity limits	When pool slots are full, runnable tasks queue	The task instance Details tab, which shows pool and slot usage
Inputs that differ between runs	Each task instance keeps rendered templates and XCom values	Compare a failing run's values with a good run's

Which Airflow features speed up the investigation?

Grid view. It marks failed and retried tasks by color, opens logs from a task cell, and lets you pick a run range such as the last 25 runs. Its sidebar shows task durations and states across recent runs, which helps spot recurring issues.
Per-attempt logs. Airflow writes a separate log for each task and attempt, named with attempt={try_number}. Compare the first failed attempt with later ones.
Callbacks. on_failure_callback and on_retry_callback can capture context when a task fails or is retried. They run only when a worker executes the state change, and errors inside a callback appear in the dag processor logs, not the task logs.
Retry controls. retries, retry_delay, retry_exponential_backoff, and max_retry_delay tune retries. AirflowFailException stops retrying when a retry cannot help.
Alerts on slow runs. Deadline Alerts, new in Airflow 3.1 and experimental, set time thresholds for DAG runs and run a callback when one is missed. They replace the SLA feature that Airflow 3.0 removed.

What can Databricks add when the DAG runs Databricks work?

The Databricks Airflow operators write the job run page URL to the Airflow log every polling_period_seconds (default 30 seconds), so follow it from the failed task. DatabricksRunNowOperator triggers an existing job, and its runs appear in the Jobs UI. DatabricksSubmitRunOperator creates one-time runs that are not backed by a saved job, so you cannot find them by job name. Filter by run ID, Run as, or start time instead.
In the Jobs UI, the matrix view shows run history for each task, the Runs tab lists runs started by external tools such as Apache Airflow, and the Top 5 error types table shows the most frequent errors in the selected range. Open a failed job and select Diagnose Error to have Genie Code help diagnose it. The UI keeps run history for up to 60 days, while the system.lakeflow tables keep 365 days, so query them for patterns:

SELECT to_date(period_start_time) AS run_date, termination_code,
       COUNT(DISTINCT run_id) AS failed_runs
FROM system.lakeflow.job_run_timeline
WHERE result_state IN ('FAILED', 'ERROR', 'TIMED_OUT')
  AND period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
GROUP BY ALL
ORDER BY run_date

Codes such as CLOUD_FAILURE, CLUSTER_ERROR, and WORKSPACE_RUN_LIMIT_EXCEEDED point to platform or capacity causes. RUN_EXECUTION_ERROR means the run completed with task failures.

FAQs

Should I move this DAG to Lakeflow Jobs?

Consider it when most tasks run on Databricks. Lakeflow Jobs gives you a task graph with dependencies, matrix run history, a timeline view for finding slow tasks, duration warnings, and notifications. You can also keep Airflow and trigger those jobs from it. The Databricks provider for Airflow is open source and not directly supported by Databricks.

What does a repair run do?

For jobs with two or more tasks, Repair run re-runs the failed or canceled tasks and their dependents. Each task restarts from the beginning, and Lakeflow Jobs does not make tasks idempotent, so a task that wrote partial output can duplicate data on a repair.

Does Lakeflow Jobs support branching and loops?

Yes. If/else condition tasks add branching, and For each tasks loop over an input array that can be specified manually or generated dynamically.