How do I update my Spark runtime?

Summary

• On Databricks, your Spark runtime is the Databricks Runtime version of your compute. To update classic compute, edit it, pick a newer version in the Databricks Runtime Version drop-down menu, and restart it if it is running.
• For a job task on a new classic cluster, change spark_version in its new_cluster settings. The Jobs API describes new_cluster as a cluster created for each run, so the next run uses the new version. Serverless compute is versionless, and Databricks upgrades it for you.
• Each Databricks Runtime moves from Beta to GA, then to long-term support (LTS) for three years. After support ends you can't select the version in the UI, and at end of life existing workloads on it fail.
• Before upgrading, read the target runtime's release notes and the Spark migration guide, and test on a non-production job first.

How do I update my Spark runtime?

On Databricks, the Spark runtime is the Databricks Runtime version your compute uses. For classic compute, edit the compute, choose a newer version in the Databricks Runtime Version drop-down menu, and restart it. For a job, change the spark_version of its cluster definition. Serverless compute is versionless, so there is nothing to update. A new runtime can carry a new Apache Spark version, so test your code before you move production workloads.

What is a Databricks Runtime version?

Databricks Runtime is the set of core components that run on your compute, and every version includes Apache Spark. The release notes for each version list its Spark version, system environment (operating system, plus Java, Scala, Python, R and Delta Lake versions) and installed libraries. Photon is not a separate runtime version. On classic compute it is a setting.

How do I update classic compute?

1. Open the compute details page and edit the configuration.
2. Choose the new version in the Databricks Runtime Version drop-down menu.
3. Restart the compute. Editing any attribute of a running compute except its size and permissions requires a restart, which can disrupt people using it.

Notebooks and jobs attached to the compute stay attached after you edit it, and installed libraries stay installed. Databricks recommends the most current version for all-purpose compute. For job compute running operational workloads, it recommends considering an LTS version so you can test thoroughly before upgrading.

How do I update the runtime for a job?

A job task that runs on a new classic cluster describes it in a new_cluster object, and the Jobs API says that cluster is created for each run. List the available versions, then set one:

databricks clusters spark-versions

new_cluster:
  spark_version: <a key from the list above>

To use Photon through the API, also set runtime_engine to PHOTON. A task that points at an existing cluster through existing_cluster_id runs on that cluster, so update that cluster as described above.
Lakeflow Pipelines use a channel setting instead. The current channel is the default and what Databricks recommends for production, while preview lets you test upcoming runtime changes.

Which version should I choose?

Version	Apache Spark	Release date	End of support
19	4.2.0	June 15, 2026	Determined at LTS transition
18 LTS	4.1.0	June 10, 2026	June 10, 2029
17.3 LTS	4.0.0	Oct 22, 2025	Oct 22, 2028
16.4 LTS	3.5.2	May 9, 2025	May 9, 2028
15.4 LTS	3.5.0	Aug 19, 2024	Aug 19, 2027

This table reflects the Databricks Runtime release notes page as checked on 2026-10-09.

What happens when a runtime is retired?

• Beta: early evaluation only, not recommended for production, with no support SLA.
• GA feature development: lasts about six months. New features and fixes ship under the same version number, and clusters receive them when they restart.
• LTS: major stability and security fixes are backported for three years.
• End of support: the version is no longer selectable when you create or update compute. Workloads might keep running, but without support or fixes.
• End of life: Databricks aims for six months after end of support, but can remove a version at any time after support ends. Existing workloads on it fail.

FAQs

Can I roll back to an older runtime?

You can edit the compute and select an older version from the drop-down menu, as long as it is still selectable. Tables written by a higher runtime might use Delta table features that lower runtimes don't support, so test reads and writes first.

Will updating break my code?

It can. Read the target version's release notes, including behavioral changes, and the Spark migration guide for the Spark version it includes. The guide URL follows the pattern https://spark.apache.org/docs/<version>/migration-guide.html.

How do I find compute that still runs old runtimes?

Query the dbr_version column of system.compute.clusters. The Databricks support lifecycles page has sample queries for clusters and jobs on legacy versions.

What about serverless compute?

Databricks upgrades serverless compute automatically, and workloads always run on the latest runtime version. For libraries, you pick a serverless base environment instead.