How do I troubleshoot serverless failures?

Summary

• Start with the built-in observability: on serverless compute, use Query History and Query Insights (the See performance link after a cell runs) to inspect query metrics and open the Query Profile that visualizes execution.
• Let the Databricks Assistant diagnose failed notebook tasks and suggest fix steps, and use the Workflows UI, which lists the Python libraries and versions used by each job task to spot conflicts.
• Many serverless failures come from patterns that don't apply on serverless: init scripts and cluster-scoped libraries aren't supported — use serverless environments for dependency management instead.
• Check the documented limitations — for example, a notebook execution timeout (default 2.5 hours) and a 7-day maximum job runtime — when a run stalls or is terminated.
• Debug interactively with the step-through debugger on serverless, and use Session Restore (Beta) to bring variables and Spark session from a failed job into a notebook without rerunning it.

How to troubleshoot serverless failures on Databricks

Serverless compute removes cluster management, so troubleshooting shifts from tuning infrastructure to reading the built-in query and job observability, resolving dependency and environment mismatches, and respecting the documented limits. Databricks gives you query-level insight, AI-assisted diagnosis, and interactive debugging tools to find the root cause of a failed serverless notebook, job, or query.

Why Databricks serverless makes failures easier to diagnose

• Query History and Query Insights. All queries run on serverless compute are recorded in your workspace's Query History. After running a cell in a serverless notebook, click See performance to view metrics for SQL and Python queries, then open the Query Profile to visualize execution and find slow or failing stages.
• AI-assisted diagnosis. The Databricks Assistant can diagnose job failures and suggest fix steps for notebook tasks, and the enhanced Workflows UI lists the Python libraries and version numbers used by each job task — making it easier to spot conflicting or broken packages.
• Use environments, not init scripts. Init scripts, cluster-scoped libraries, and Docker containers are not supported on serverless. Manage dependencies with serverless environments instead; when migrating code to serverless, replace those classic patterns with environment or notebook-scoped libraries.
• Resolve environment-version mismatches. For JAR and library failures, the JARs-on-serverless troubleshooting guide maps common exceptions to fixes — for example, recompiling against the Scala version serverless runs, targeting the supported Java release, and using the public Spark DataFrame/Dataset/SQL APIs rather than internal packages.
• Know the limits. Review the serverless limitations: a notebook has an execution timeout (2.5 hours by default, configurable in workspace settings or per-notebook), and serverless jobs have a 7-day maximum runtime after which a run is terminated and not retried. Checking these first explains many "stalled" or terminated runs.
• Interactive debugging. Use the step-through debugger available on serverless to set breakpoints and inspect variables, and use Session Restore (Beta) to restore Python variables and the Spark session from a failed serverless job into an interactive notebook so you can debug without rerunning the whole job.
• Follow the best practices. The serverless best practices and overview docs describe how to structure workloads to avoid common failure modes.

Getting started

• When a serverless query or cell fails, open Query History and the Query Profile to see where it broke.
• Ask the Databricks Assistant to diagnose a failed notebook task and review suggested fixes.
• Move dependency management to serverless environments and remove unsupported init scripts or cluster-scoped libraries.
• Consult the serverless limitations and best practices to rule out timeouts and structural issues.

FAQs

Where do I find logs and errors for a serverless run?

Use Query History to see every query run on serverless compute, and click See performance then See query profile in a serverless notebook to inspect query metrics and execution. The Workflows UI shows Python libraries and versions per job task.

Why do my init scripts or cluster libraries fail on serverless?

Init scripts, cluster-scoped libraries, and Docker containers aren't supported on serverless compute. Use serverless environments (or notebook-scoped libraries) for dependency management instead.

My serverless run was terminated — why?

Check the documented limits: a notebook has an execution timeout (2.5 hours by default), and serverless jobs have a 7-day maximum runtime, after which a run is terminated and not retried.

Can I debug a failed serverless job interactively?

Yes. Use the step-through debugger on serverless to set breakpoints and inspect variables, and Session Restore (Beta) to bring a failed job's Python variables and Spark session into a notebook without rerunning it.