Classic clusters versus serverless: when to switch?

Summary

• Switch a workload to serverless when it uses DataFrame, SQL and supported Structured Streaming APIs. Serverless runs code through Spark Connect, the client-server architecture Apache Spark introduced in 3.4, which doesn't support RDDs or the SparkContext.
• Stay on classic compute for R, Scala notebooks, init scripts, custom containers, most custom Spark configurations, the Spark UI, or a single run longer than 7 days.
• Databricks' quick readiness test: run the workload on classic compute with standard access mode on Databricks Runtime 14.3 or above. If it succeeds, it can move to serverless without code changes.
• Serverless billing follows DBU consumption, not cluster uptime. Migrate in phases: new workloads first, then low-risk PySpark and SQL workloads, then code that needs changes.

Classic clusters versus serverless: when to switch?

Switch when a workload uses only the APIs serverless supports and you don't need cluster-level control. Serverless compute on Databricks runs your code through Spark Connect, so DataFrame and SQL code, and streams that set a supported trigger, move with minimal or no changes. Code that reaches into RDDs, the SparkContext or cluster-wide Spark settings needs a rewrite or stays on classic compute. Databricks' migration guide recommends starting all new notebooks and jobs on serverless and moving existing workloads in phases.

What actually changes when you move off classic compute?

Classic compute is provisioned compute that you create, configure and manage: you pick the Databricks Runtime version, instance types, autoscaling bounds and an auto-termination inactivity period. Serverless compute is a Databricks-managed service. You run workloads without provisioning compute in your cloud account, Databricks allocates and manages the resources, and the runtime is versionless, so Databricks upgrades it automatically.
The open-source piece underneath is Spark Connect. Apache Spark 3.4 introduced it as a decoupled client-server architecture: the client sends unresolved logical plans to the server over gRPC. By design the protocol doesn't support all of Spark's execution APIs, most importantly RDDs, and the client can't reach the static Spark configuration or the SparkContext. The PySpark API reference labels each API that supports Spark Connect, so you can audit code before you move it.
On a self-managed open-source cluster, elastic sizing comes from dynamic resource allocation (spark.dynamicAllocation.enabled), which is disabled by default. Don't turn it on for Databricks classic compute: Databricks autoscaling manages executors at the platform level, and running both can cause conflicting scaling decisions.

Which workloads should stay on classic compute?

Your workload needs	On serverless	What to do
RDD APIs, sc.parallelize, spark.sparkContext	Not supported	Rewrite to DataFrame APIs, or stay on classic
R, or Scala in notebooks	Not supported	Stay on classic; JAR tasks in jobs are supported
Init scripts, custom containers, instance pools, compute policies	Not supported	Use a serverless environment for libraries, or stay on classic
Custom Spark configurations	A short supported list only	Remove unsupported configs
df.cache(), df.persist(), CACHE TABLE	Raise an exception	Remove caching calls
The Spark UI and Spark logs	Not available	Use the query profile
A stream with the default or a processing-time trigger	Not supported	Use Trigger.AvailableNow()
A single run longer than 7 days	Terminated, not retried	Split it into smaller runs, or use classic
DBFS mounts, Hive metastore tables, instance profiles	Not supported	Move to Unity Catalog volumes, tables and external locations

How do you test whether a workload is ready?

Databricks documents a quick compatibility test: run the workload on classic compute with standard access mode and Databricks Runtime 14.3 or above. If that run succeeds, the workload can migrate to serverless without code changes. For production, the guide recommends an A/B run: the same workload on classic (control) and serverless (experiment), diffing the output tables until they match.
Most code changes are mechanical. Replace DBFS paths with Unity Catalog volumes and use three-level table names:

# Classic
df = spark.read.csv("dbfs:/mnt/datalake/data.csv", header=True)
df = spark.table("my_database.my_table")

# Serverless
df = spark.read.csv("/Volumes/main/sales/raw_data/data.csv", header=True)
df = spark.table("main.my_database.my_table")  # three-level namespace

Set an explicit trigger on every stream. Omitting .trigger() defaults to a processing-time trigger, which serverless rejects:

query = (df.writeStream.format("delta").outputMode("append")
    .trigger(availableNow=True)
    .option("checkpointLocation", checkpoint_path)
    .start(output_path))

How Databricks approaches the switch

The migration guide lays out four phases. Start all new notebooks and jobs on serverless. Next, migrate low-risk PySpark and SQL workloads that already run on standard access mode and Databricks Runtime 14.3 or above. Then migrate workloads that need code changes (RDD rewrites, DBFS updates, trigger fixes), and review the rest periodically as capabilities expand.
Pick a performance mode per workload. Performance-optimized, the default, suits interactive work that needs fast startup; Databricks keeps a pool of warm resources ready for it. Standard mode, available for Lakeflow Jobs and Lakeflow Pipelines but not notebooks, accepts longer startup and, per Databricks, typically offers the best value for scheduled jobs where startup latency isn't critical. Serverless billing follows DBU consumption rather than cluster uptime, so validate cost with representative workloads before migrating at scale.

FAQs

Can I keep calling df.cache() on serverless?

No. DataFrame and SQL cache APIs raise an exception on serverless. The migration guide says to remove caching calls, and it shows writing an expensive intermediate result to a Delta table instead.

Which Spark configurations can I still set?

Only a short list, including spark.sql.shuffle.partitions, spark.sql.session.timeZone, spark.sql.ansi.enabled and spark.sql.files.maxPartitionBytes. Databricks generally recommends against setting most Spark properties, and job runs on serverless fail if you set an unsupported one.

Does Structured Streaming work on serverless?

Yes, with Trigger.AvailableNow() (recommended) or the deprecated Trigger.Once(). Fixed-interval and continuous triggers aren't supported. For always-on streams, use Lakeflow Pipelines in continuous mode or a continuous job that runs with AvailableNow.

Do I need Unity Catalog to use serverless?

Yes. Workspaces with Unity Catalog enabled get serverless compute automatically; legacy workspaces without it don't. Serverless also requires Unity Catalog for connecting to external data sources.