How do I migrate my Databricks jobs from classic clusters to serverless?

Summary

• Databricks' migration guide says most classic workloads can move to serverless compute with minimal or no code changes. Workloads that depend on R or Scala notebooks stay on classic compute, and patterns such as RDD code and df.cache() need rework.
• Serverless jobs use Spark Connect on standard access mode, so check Spark APIs, Spark configurations, init scripts, DBFS paths and streaming triggers before you switch.
• A quick compatibility test: if the job already runs on classic compute with standard access mode and Databricks Runtime 14.3 or above, it can migrate without code changes.
• Run the job on classic and serverless side by side, diff the output tables, and compare DBU usage in the billable usage system table before you cut over.
• Roll out in phases: new jobs first, then low-risk PySpark and SQL jobs, then jobs that need code changes.

How do I migrate my Databricks jobs from classic clusters to serverless?

Check that each task type and its code fit serverless compute, fix the incompatibilities, run the job on serverless next to the classic version, compare the results, and then switch the job's compute. Serverless compute for jobs is the default for every task that supports it, and Databricks recommends it for most new workloads.

What changes when a job moves to serverless compute?

Area	Classic jobs compute	Serverless compute for jobs
Compute setup	You choose runtime, node types and autoscaling bounds	Databricks manages the compute, and autoscaling and Photon are enabled automatically
Runtime	You pick a Databricks Runtime version	Databricks upgrades the runtime for you
Spark properties	Set on the compute or in the notebook	Only a short list is supported, and a run fails if it sets an unsupported one
Dependencies	Cluster libraries and init scripts	Serverless environments; init scripts are not supported
Data access	DBFS paths, mounts, instance profiles, Hive metastore tables	Unity Catalog volumes, external locations and tables
Spark APIs	RDD APIs (dedicated access mode), caching calls	Spark Connect APIs only; RDD APIs and df.cache() are not supported
Streaming triggers	Any trigger	Trigger.AvailableNow(); ProcessingTime and Continuous are not supported
Debugging	Spark UI and Spark logs	Query profile; the Spark UI and Spark logs are not available

Serverless jobs support notebook, Python script, dbt, Python wheel and JAR tasks. Spark Submit tasks run only on classic jobs compute, and that task type is deprecated.

How do I migrate a job step by step?

1. Check prerequisites. The workspace needs Unity Catalog. Replace VPC peering with NCCs, Private Link or firewall rules, and replace instance profiles and DBFS mounts with Unity Catalog external locations.
2. Update the code. Move dbfs:/ paths to Unity Catalog volumes, use three-level table names, rewrite RDD operations as DataFrame operations, remove df.cache() and df.persist(), and delete unsupported Spark configurations.
3. Expect ANSI mode. Serverless enables ANSI mode by default, so an invalid cast raises CAST_INVALID_INPUT and a division by zero raises DIVIDE_BY_ZERO. Use try_cast or try_divide where you want NULL.
4. Set a supported streaming trigger. Use .trigger(availableNow=True). The default trigger is ProcessingTime("0 seconds"), which serverless rejects with INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED.
5. Move dependencies. Replace init scripts and cluster libraries with a serverless environment, and pin package versions.
6. Test. Run the job on classic compute with standard access mode and Databricks Runtime 14.3 or above. If it succeeds, it can migrate without code changes. For production jobs, run the classic job as the control and the serverless job as the experiment, then diff the output tables.
7. Switch the compute. In the Job details side pane, click Swap under Compute, or pick Serverless in the Compute drop-down.

In a Declarative Automation Bundle, omit the cluster settings for a notebook task. Python script, Python wheel and dbt tasks need an environment_key. The performance_target setting defaults to PERFORMANCE_OPTIMIZED; STANDARD suits batch jobs that tolerate slower startup and consumes fewer DBUs.

resources:
  jobs:
    my_serverless_job:
      name: my_serverless_job
      performance_target: STANDARD
      tasks:
        - task_key: task
          spark_python_task:
            python_file: ../src/main.py
          environment_key: default
      environments:
        - environment_key: default
          spec:
            environment_version: '6'
            dependencies:
              - my-library==<version>

How do I compare cost and roll out safely?

Serverless billing is based on DBU consumption, not cluster uptime, so Databricks recommends benchmarking a representative workload and reading the billable usage system table. Usage can take up to 24 hours to appear.

SELECT usage_metadata.job_id,
       product_features.is_serverless AS is_serverless,
       SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE billing_origin_product = 'JOBS'
  AND usage_date >= '<start-date>'
GROUP BY ALL

Roll out in phases: start new jobs on serverless, then move PySpark and SQL jobs that already run on standard access mode, then jobs that need code changes, and review the rest as capabilities expand. A Beta migration agent can propose changes for one job at a time: a workspace admin opts into the Compute Agent preview, and you run /compute migrate to serverless in Genie Code.

FAQs

Can some tasks stay on classic compute?

Yes. You can set a different compute type per task, which helps when a task type is not supported on serverless.

What if the job fails on serverless but works on classic?

Check the usual causes: an unsupported Spark configuration, RDD or caching calls, init scripts, DBFS paths, ANSI-mode errors and the default streaming trigger. Use the query profile to investigate, because the Spark UI is not available.

Is there a runtime limit?

Serverless compute has a maximum runtime of 7 days. Runs that exceed it are terminated and not retried, so split longer workloads or use classic compute.

How do I back out?

Keep the classic job definition until the serverless run is verified. If you keep both jobs, pause the schedule on the one you are not running, or the same trigger fires both and can duplicate writes.