My Databricks job has this error message, what does it mean and how do I fix it?

Summary

• Open the failed run under Jobs & Pipelines, click the failed task, and read its output and error message on the Task run details page. The runs list also shows the error code the run terminated with.
• The error code points to the layer that failed: task execution, the driver or cluster, libraries, permissions, configuration, or a concurrency limit.
• A message that starts with a bracketed name, such as [TABLE_OR_VIEW_NOT_FOUND], carries a Databricks error condition and a SQLSTATE. Look it up by that name, because message text can change.
• After you fix the cause, use Repair run to re-run only the unsuccessful tasks and their dependents.
• Add retries for transient errors and failure notifications for the rest.

My Databricks job has this error message, what does it mean and how do I fix it?

Work from the failure outward. Find which task failed, read its error message and the run's error code, then match the code to the layer that broke: your task's code or data, the compute that ran it, a permission problem, or a limit on concurrent runs. These steps follow Troubleshoot and repair job failures.

Where do I find the full error message?

1. Click Jobs & Pipelines in the sidebar, then click the job name. The Runs tab shows a matrix view of past runs, including each task's result.
2. Hover over a failed task to see its status, duration, cluster details, and sometimes an error message.
3. Click the failed task. The Task run details page shows its output, error message, and metadata.
4. In the graph view, click a task node to see the cluster that ran it and links to its query history and logs. In the Spark UI, failed stages show the failure reason.

With the default All succeeded condition, a task whose upstream dependency failed does not run and is marked Upstream failed, so look upstream for the real error.

What do the common error codes mean?

The runs list shows the error code a run terminated with. The termination_code column in system.lakeflow.job_run_timeline documents these values, among others:

Code	What the docs say it means	What to check
RUN_EXECUTION_ERROR	The run was completed with task failures.	Open the failed task and read its output and error message.
DRIVER_ERROR, CLUSTER_ERROR	An error communicating with the Spark driver, or a cluster error.	Check the task's logs, metrics, and compute settings.
LIBRARY_INSTALLATION_ERROR	The library install failed, for example because the library is invalid or permissions are missing.	Check the library name and install permissions.
UNAUTHORIZED_ERROR	A permission issue occurred while accessing a resource.	Check the privileges of the identity in the Run as field.
MAX_CONCURRENT_RUNS_EXCEEDED	The scheduled run exceeds the job's maximum concurrent runs.	Wait for other runs, or raise Maximum concurrent runs.
INVALID_RUN_CONFIGURATION	The run failed due to an invalid configuration.	Click Edit task, correct the settings, and save.

How do I fix the most common causes?

The docs name a data quality issue, a misconfiguration, and insufficient compute resources as common reasons a task fails.

• Out of memory: the Apache Spark tuning guide says an OutOfMemoryError can come from one task's working set being too large, and that the simplest fix is to raise the level of parallelism so each task's input is smaller. Databricks generally recommends against configuring most Spark properties, and serverless compute does not support most of them, but spark.sql.shuffle.partitions is one you can set.
• Insufficient cluster resources: change the number of workers, the instance types, or other cluster settings, or ask an administrator to raise resource quotas in the cloud account and region.
• External source unavailable: wait for the source to recover, then repair the run.

How do I re-run the job after fixing it?

For a multi-task job, open the failed run and click Repair run. Databricks re-runs the unsuccessful tasks and their dependents with the current settings and skips tasks that already succeeded. Repair needs a job with two or more tasks, so re-run a failed single-task job with Run now.
A repair restarts each unsuccessful task from the beginning, and Lakeflow Jobs does not make tasks idempotent. If a task wrote part of its output before failing, a repair can duplicate that data, so prefer overwrite or merge over append.

How do I stop the same error from paging someone again?

• Retries: click Add next to Retries on a task. For most configurations the default is no retries, and serverless jobs auto-optimize retries by default. The Jobs API fields are max_retries, min_retry_interval_millis, and retry_on_timeout.
• Notifications: notify on failure by email, Slack, PagerDuty, or any webhook-based service. Job-level notifications are not sent when failed tasks are retried. Use task notifications to hear about every failed task, or select Mute notifications until the last retry to hear only about the final failure.
• Diagnosis: open the failed job in the Jobs UI and select Diagnose Error to have Genie Code help.
• Trends: count recent failures by code across your jobs.

SELECT termination_code, COUNT(DISTINCT run_id) AS failed_runs
FROM system.lakeflow.job_run_timeline
WHERE period_start_time > CURRENT_TIMESTAMP() - INTERVAL 7 DAYS
  AND result_state IS NOT NULL
  AND result_state <> 'SUCCEEDED'
GROUP BY termination_code
ORDER BY failed_runs DESC

FAQs

What does a bracketed name like TABLE_OR_VIEW_NOT_FOUND mean?

Databricks errors carry an error condition (a descriptive string such as TABLE_OR_VIEW_NOT_FOUND), a five-character SQLSTATE, and a message with the parameters filled in. The message text can change between releases, so look up the condition name in the error conditions reference. For TABLE_OR_VIEW_NOT_FOUND it says to verify the spelling and correctness of the schema and catalog.

Why doesn't my job show Failed when a task failed?

Databricks decides the outcome from the leaf tasks, the tasks with no downstream dependencies. If a task failed but every leaf task succeeded, the run is Succeeded with failures, which counts as successful. Select Success when you configure notifications if you want to be told about that state.