Skip to main content

Do I still need to manage infrastructure with serverless?

Summary

  • No. With Databricks serverless compute, Databricks manages the infrastructure so you focus on your workloads and data instead of provisioning and tuning clusters.
  • Databricks handles cluster configuration, capacity, autoscaling, patching, security updates, and runtime upgrades on your behalf.
  • You no longer pick instance types, set cluster scaling parameters, or configure Spark settings.
  • Security patching is automatic: serverless compute is limited to a maximum of 7 days of uptime before it is recycled in the background to apply the latest patches.
  • Runtime upgrades are automatic too: a versionless architecture removes manual Databricks Runtime upgrades, with 25 upgrades performed across more than 4.5 billion workloads at a 99.998% success rate.

Do I still need to manage infrastructure with serverless?

No. The whole point of Databricks serverless compute is that Databricks manages the infrastructure for you. You choose the workload; Databricks provisions, scales, patches, and upgrades the compute automatically. That frees your teams to focus on writing code and analyzing data rather than on capacity management, patching, upgrading, and performance tuning of clusters.

What Databricks manages for you

  • Cluster configuration. You no longer pick instance types, set cluster scaling parameters, or configure Spark settings. Databricks automatically allocates and manages the compute resources your workload needs. See What is serverless computing?.
  • Capacity and autoscaling. An intelligent autoscaler scales capacity both horizontally (more executors) and vertically (larger executor sizes) in response to workload demand, and winds resources down when they are no longer needed.
  • Patching and security updates. Databricks owns hardening and patching. Serverless compute is limited to a maximum of 7 days of total uptime before it is recycled seamlessly in the background so it always runs on the latest security patches.
  • Runtime upgrades. Serverless uses a versionless architecture that eliminates manual Databricks Runtime upgrades. Over the past year, Databricks automatically performed 25 Databricks Runtime upgrades across more than 4.5 billion workloads at a 99.998% success rate. See Announcing GA of serverless compute.
  • Infrastructure selection and resilience. The system continuously monitors workload patterns and resource utilization, automatically moving to larger instances when memory constraints are detected and failing over to compatible instance types during cloud provider outages.
  • Instant, permission-free access. Serverless compute starts in seconds using warm pools of pre-initialized instances, and no cluster-creation permissions are required, so all workspace users can run serverless workloads by default.

What you are responsible for

With serverless compute you are not responsible for infrastructure management tasks such as capacity management, patching, upgrading, or performance optimization of the cluster. Your focus stays on your code, queries, models, and data, along with governance and access policies for that data through Unity Catalog.

Getting started

FAQs

Do I still manage clusters with serverless?

No. Databricks manages cluster configuration, capacity, patching, upgrades, and performance optimization. You do not pick instance types or tune Spark settings.

How are security patches applied?

Automatically. Serverless compute is limited to a maximum of 7 days of uptime before it is recycled in the background so it runs on the latest security patches.

Do I have to upgrade the Databricks Runtime myself?

No. Serverless uses a versionless architecture that removes manual runtime upgrades; Databricks performs them automatically in the background.

What am I responsible for with serverless?

Your code, queries, models, and data, plus governance and access control through Unity Catalog. Databricks handles the underlying infrastructure.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.