What are the limitations of Databricks serverless compute?

Summary

• Serverless compute for notebooks and jobs supports only Spark Connect APIs. R and Spark RDD APIs are not supported, and notebooks don't support Scala or R.
• It removes classic compute controls: compute policies, compute-scoped init scripts and libraries, instance pools, compute event logs, Databricks Container Services and most Spark configurations. A job run fails if it sets an unsupported Spark property.
• Debugging and caching differ. The Spark UI and Spark logs are not available, so use the query profile, and DataFrame and SQL cache APIs raise exceptions.
• Streaming triggers are limited: Trigger.AvailableNow is recommended, Trigger.Once is deprecated, and time-based triggers fail. A serverless run also ends at a seven-day maximum. Always-on streams belong in Lakeflow Pipelines in continuous mode.

What are the limitations of Databricks serverless compute?

Serverless compute is a Databricks-managed, versionless service, and Databricks recommends it for most new workloads. The trade-off is fewer supported features: fewer languages and APIs, no compute-level controls, different debugging tools, and limits on streaming, caching and job runtime. The Databricks serverless limitations page holds the current list, and the table below summarizes the main ones.

Which limits matter most?

Area	Documented limitation	What to do
Languages and APIs	R is not supported. Only Spark Connect APIs work, so Spark RDD APIs do not. Notebooks don't support Scala, and JAR libraries aren't supported in notebooks (JAR tasks in jobs are).	Use DataFrame or SQL APIs, or run the workload on classic compute
Compute controls	No compute policies, compute-scoped init scripts or libraries, instance pools, compute event logs, or Databricks Container Services.	Use a base environment or notebook dependencies
Spark configuration	Most Spark properties are not supported. The documented settable list is short: spark.databricks.execution.timeout, spark.sql.legacy.timeParserPolicy, spark.sql.session.timeZone, spark.sql.shuffle.partitions, spark.sql.ansi.enabled and spark.sql.files.maxPartitionBytes. Setting any other property fails the job run.	Remove unsupported properties before migrating
Debugging	The Spark UI and Spark logs are unavailable. You get client-side application logs.	Use the query profile
Caching	df.cache(), df.persist(), df.unpersist(), df.checkpoint(), CACHE TABLE, UNCACHE TABLE, REFRESH TABLE and CLEAR CACHE raise exceptions.	Remove these calls from migrated code
Streaming	Trigger.AvailableNow() is supported and Trigger.Once() is deprecated. Trigger.ProcessingTime and Trigger.Continuous are not supported, and a query without a supported trigger fails with INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED.	Set Trigger.AvailableNow(), or use Lakeflow Pipelines in continuous mode
Job runtime	A run is capped at 7 days. Jobs have no query execution timeout by default, and task logs are not isolated per task run.	Split long runs, or set spark.databricks.execution.timeout
Data access	Unity Catalog is required for external data sources. DBFS access is limited, and Maven coordinates, global temp views and Hive SerDe tables are not supported.	Use external locations, Unity Catalog volumes or workspace files
Networking	VPC peering is not supported, resources reachable only over IPv6 can't be reached, and cross-workspace access works only within a region and only if the destination workspace has no IP ACL or front-end PrivateLink.	Create private endpoints from serverless to your VPC resources, or allow Databricks serverless traffic through a firewall

When should I use classic compute instead?

Choose classic compute when the workload depends on something in the table: R, Spark RDD APIs, Databricks Container Services, compute-scoped init scripts, instance pools, compute policies, or Spark properties outside the settable list. Databricks also lists classic jobs compute as the recommended option for JAR and Spark Submit tasks. On classic compute, standard access mode doesn't support R or RDD APIs either, so those workloads need dedicated access mode.

Which behaviors change on serverless?

Some code runs but behaves differently:

• ANSI mode is on by default. Invalid casts, arithmetic overflow and malformed input raise runtime errors instead of silently returning NULL.
• The session time zone defaults to Etc/UTC. Set spark.sql.session.timeZone to change it.
• Spark Connect defers analysis and name resolution to execution time. Use unique names for temporary views, especially inside loops.
• UDF code can't use the internet, and custom code such as UDFs, map and mapPartitions can't exceed 1 GB of memory.
• Working directories aren't guaranteed, so relative paths and relative imports can fail. Use absolute paths.
• Auto-optimization can retry failed tasks. For workloads that must run at most once, clear Enable serverless auto-optimization in the task's retry policy.

What about cost and startup time?

Pricing for Lakeflow Jobs is tied to the compute that runs the tasks, and serverless usage appears in system.billing.usage. Standard performance mode consumes fewer DBUs and tolerates a startup latency of four to six minutes. Performance optimized mode starts and runs faster. Jobs that need large amounts of memory or include many tasks might also start more slowly.

FAQs

Is there a cost cap for serverless jobs?

The docs describe alerts, not a hard cap. A budget emails the addresses you list when monthly spend crosses a threshold, with up to 24 hours of delay. A serverless job has no query timeout unless you set spark.databricks.execution.timeout, and every run ends at the seven-day maximum.

How do I install libraries without init scripts?

Use serverless environments, which cache installed packages and reduce startup latency on later runs. Pin package versions, because unpinned packages can resolve differently and add install time.

Can one job use serverless and classic compute?

Yes. You can specify different compute types for tasks in a job, which may be required when a task type isn't supported on serverless.