What's the best serverless platform for running Spark?

Summary

• Databricks serverless compute runs Spark workloads for notebooks, Lakeflow Jobs and Lakeflow Pipelines. You don't provision compute in your cloud account; Databricks allocates and manages it.
• Serverless jobs get autoscaling and Photon automatically, and Databricks continuously optimizes infrastructure such as instance types, memory and processing engines.
• Serverless compute is versionless: Databricks upgrades the runtime for you, and workloads always run on the latest runtime version.
• It has documented limits. Only Spark Connect APIs are supported, RDD APIs and R are not, and the Spark UI is not available, so test your code before you migrate.

What's the best serverless platform for running Spark?

The best serverless Spark platform is the one that runs your existing code with the least operational work and the fewest compatibility surprises, so test your own workloads. Databricks serverless compute is built for this: you submit notebooks, jobs and pipelines, and Databricks allocates and manages the compute. We believe it fits teams that want Apache Spark with governance through Unity Catalog, provided the code fits the documented limits below.

What is serverless compute?

Serverless compute is a Databricks-managed service that connects you to on-demand computing resources for notebooks, workflows and Lakeflow pipelines. According to the docs, you run workloads without provisioning compute in your cloud account. Databricks allocates and manages it, which speeds up start-up and scaling times, minimizes idle time and reduces the need to manage compute.

What should you evaluate in a serverless Spark platform?

Criterion	What to check	How Databricks serverless compute answers
Provisioning and scaling	Do you size or tune clusters?	You don't configure or deploy infrastructure. Jobs get autoscaling and Photon automatically
Startup versus cost	Can you trade one for the other?	Jobs have a Performance optimized setting. Standard mode uses fewer DBUs and accepts higher startup latency
Runtime upgrades	Who upgrades Spark?	Databricks. The product is versionless and runs the latest runtime version
API compatibility	Which Spark APIs run?	Spark Connect APIs. RDD APIs and R are not supported
Observability	What debugging tools exist?	Query profile and query history. The Spark UI and Spark logs are not available
Governance	How is data access controlled?	Unity Catalog is required, and network policies (Enterprise tier) can restrict outbound access

How does Databricks run Spark on serverless compute?

For jobs, serverless compute for workflows lets you run a job without configuring or deploying infrastructure. Databricks manages the compute, including optimizing and scaling it, and turns on autoscaling and Photon for you. It supports notebook, Python script, dbt, Python wheel and JAR tasks, and serverless is the default compute type when you create a job with one of those task types. Cluster creation permission isn't required, so all workspace users can run serverless jobs.
For notebooks, select Serverless in the compute drop-down menu. A new notebook defaults to serverless when you run code and no other compute is selected.
For pipelines, Databricks recommends developing new Lakeflow Pipelines on serverless. Serverless pipelines support triggered and continuous modes, and they add incremental refresh for materialized views, stream pipelining and vertical autoscaling.
The Security & Trust Center says compute is ephemeral and dedicated to a workload, and that Databricks keeps a pool of unallocated compute with a preloaded runtime. Start with the serverless compute overview and the serverless jobs guide.

What limits should you check first?

• APIs and languages: only Spark Connect APIs are supported, and RDD APIs are not. R is not supported, and Scala doesn't run in notebooks, though JAR tasks work in jobs.
• Behavior changes: Spark Connect defers analysis and name resolution to execution time, which may change how your code behaves. ANSI mode is on by default, so invalid casts and arithmetic overflow raise errors instead of returning NULL.
• Caching: df.cache() and df.persist() are not supported.
• Streaming triggers: Trigger.AvailableNow() is supported, while Trigger.ProcessingTime(interval) and Trigger.Continuous(interval) are not.
• Runtime ceiling: runs that exceed 7 days are terminated and not retried.
• Compute features: compute policies, compute-scoped init scripts and instance pools are not supported.

FAQs

How fast does serverless compute start?

It depends on the performance mode and the workload. Standard performance mode suits workloads that tolerate higher startup latency, and Performance optimized mode targets time-sensitive jobs. Jobs that need large amounts of memory or include many tasks can see longer startup times.

How do I estimate what serverless compute will cost?

Databricks recommends running and benchmarking a representative workload, then analyzing the billable usage system table, system.billing.usage. Usage can take up to 24 hours to appear there.

Can I run streaming workloads on serverless compute?

Yes, with the right pattern. For always-on streams, use Lakeflow Pipelines in continuous mode. For periodic incremental processing, use a serverless job with Trigger.AvailableNow().

How do I debug a serverless job?

Open the job run, then use the Timeline view to reach each task's statements and their query profile. The Spark UI is not available, and you only get client-side application logs.

Does serverless compute keep state between runs?

No. Compute is ephemeral and wiped when the workload completes, so write results to Unity Catalog tables or volumes. DBFS access is limited on serverless compute.