Serverless GPU Capacity: How Databricks Cloudless Computing Avoids GPU Shortages and Outages

Summary

Cloudless computing: Routes serverless workloads across clouds to available GPU capacity.
Data residency: Workspace and data stay in home region while compute routes elsewhere.
Security preserved: Private backbone, per-workspace encryption and policies remain unchanged.

Serverless GPU Capacity: How Databricks Cloudless Computing Avoids GPU Shortages and Outages

Talk by Sheng Lu, Sr. Engineering Manager, Databricks
When a cloud region runs out of GPUs or suffers an outage, Databricks keeps serverless workloads running by routing them over a private Databricks network backbone to available compute in another cloud or region, while your workspace, data and security policies stay in your home region. This video demos fixing a no-GPUs-available error in a notebook by enabling cross-cloud GPU and compute, then covers the private connectivity, per-workspace encryption and existing network policies that stay enforced.
Sheng Lu demos the fix for a "no GPUs available" error in a notebook using the Databricks AI Runtime and serverless GPUs: an admin turns on the "allow cross-cloud GPU and compute" setting and the same notebook runs with MLflow tracking. He then covers GPU and CPU capacity problems, the backbone's private traffic, per-workspace authentication and encryption, unchanged IP access lists, Private Link and outbound controls, and data access that stays in the home region. Databricks plans to bring this to more serverless products, with a preview expected within the next few months.

Chapters

FAQs

What problem does serverless GPU cross-cloud capacity solve?

When a cloud region runs out of GPUs or suffers an outage, traditional setups fail. This video demos fixing a 'no GPUs available' error by enabling cross-cloud compute, allowing the same notebook to run in another cloud with GPUs.

How does Databricks route workloads while keeping data in place?

Your workspace, data and security policies stay in your home region, but serverless workloads route over a private Databricks network backbone to available compute in another cloud or region. This video explains that data access stays in the home region for compliance.

What security measures protect cross-cloud workloads?

This video covers private connectivity on the Databricks backbone, per-workspace encryption, unchanged IP access lists, Private Link controls, and outbound controls. Existing network policies stay enforced despite cross-cloud routing.

When will cloudless computing expand to more Databricks products?

This video notes that Databricks plans to bring cloudless computing to more serverless products, with a preview expected within the next few months. The feature is currently in public preview for serverless GPUs.

Full transcript

`00:00` Introduction

Greetings to all and good day. My name is Sean. I am a senior engine development manager at Databricks. I manage several teams working on network infrastructure. I'm excited to be here today to share with you why AI can't wait and how we're building cloudless computing that will help you overcome the GPU shortage and keep your mission- critical workloads running. Let's get started. We live in the age of AI. I want to jump right into the demo to show you exactly what we've created for you.
And then we'll discuss the slides and how it works "under the hood." This will be a very short one-minute demonstration. We'll start with a simple Databricks notebook to train the model. The contents of the notebook are not that important. Importantly, we will be using the Databricks AI Runtime and serverless GPUs, which we launched into public preview yesterday. So, let's start the demonstration . Here is a simple notebook for training the model.

`00:47` Demo: fix "no GPUs available" with cross-cloud compute

In the upper right corner you can see that a serverless GPU is being used. You click " run", and an error appears. There are no GPUs available in your cloud and your region. This is unfortunate. I want to train this model on my data, right? Luckily, I am also the administrator of the Databricks account. I remembered that there was a feature I heard about at the Data + AI Summit that could be enabled to solve my problem.
So I'm looking for the option " allow cross-cloud GPU and compute". This is exactly what I need to solve the problem. I'm turning this feature on. Done? Now I'm going back to my workspace. It's the same workspace in the same region, the same notebook. I click " run" and now there is no error. Execution has started, MLflow is tracking this experiment, and everything is running successfully. That's the whole demonstration.
It's pretty simple, right ? So what actually happened is: with a single click of a button, you enabled cloudless computing and bypassed the limitations on GPU availability in your cloud and region. And this happened because Databricks is now allowed to run this task in a completely different cloud where there are free GPUs. This is the seamless cloud computing experience we're talking about . Going back to the slides, let's take a step back and rethink the problem.

`02:59` GPU shortages and CPU outages: the problem

So, whether you're working on large-scale deep learning AI workloads or even large-scale CPU workloads, there are several challenges. The first one, on the left, is the GPU shortage. We've all felt it before. GPUs may not be available in the cloud and region you are working in. And most importantly, you don't want your infrastructure to depend on where the GPUs are located, right? You probably don't want to recreate your workspace, move data, or reconfigure everything just to access the GPU.
The second problem is that this can also happen with the CPU. Of course, the shortage is less felt with CPUs, but there are failures at cloud providers and limitations that lead to a temporary lack of computing power. These may only be a few hours of downtime, but the consequences can be very serious if you have critical workloads that need to be completed on time. Isn't that right? Let's move on to the third point. When addressing these two issues, we must put security and compliance first .
Regardless of what we do “ under the hood,” it is critical not to compromise your security, compliance, and data privacy policies. So how did we decide this? We are building a Databricks network backbone. This is a key component that will provide a cloud-agnostic computing experience. This is a layer of private network infrastructure connecting different clouds and regions, managed and supported by Databricks. This network backbone can be imagined as a system consisting of many pairs of regions.

`04:44` Databricks network backbone across clouds and regions

On the screen we show an approximation of one such pair. On the left is your home region. This is something you are already familiar with. Here is your workspace, data , settings, and connection options. Yes? And on the right is a remote region. This could be a completely different cloud or a different region within the same cloud. The important thing is that this is not where your workspace is. The Databricks backbone acts as a bridge connecting these two regions in a secure and private way.
This allows us to search and find the best and most affordable computing power across different clouds and regions. We search for them for you and run your workload there. And we use the backbone to connect back to the main home region. So the appeal of this solution is that it's completely managed for you, and you continue to use the same workspace. This does not require any changes or reconfiguration of your existing security policies.
So what does this give you? Let's go through the points below. This gives you GPU on demand, just like we showed in the demo. This ensures high CPU availability as we can now bypass cloud provider outages. And we continue to strictly adhere to your existing security policies, which we will discuss in detail on the next slide. And what's really exciting in the future is, as this feature evolves and becomes more intelligent, we'll be able to be very precise about where to run the computation .

`06:28` Benefits: GPU on demand and higher availability

We will be able to optimize costs , availability or performance by analyzing the characteristics of each workload and doing what is best for it. OK. One last thing: this might sound a bit intimidating since we're talking about crossing cloud and regional boundaries, right? Therefore, I want to emphasize that security is the top priority when developing this feature. And we can go into the specifics of how we deliver that.

`07:25` Security: private connectivity and per-workspace encryption

So, the first item on the left is about connectivity between clouds and regions. This is the network backbone we just talked about. The important thing is that it remains completely private. The traffic does not pass through the Internet in any way. Additionally, we are adding authentication and encryption for each workspace. This means that traffic for each workspace will have a unique identifier. This will allow us to isolate it from the traffic of all other workspaces.
Although they share the same network backbone between the two regions. So, let's move on to the second column. Again, policy implementation is very important. Existing inbound traffic controls continue to be consistently applied. This means that your IP access list, inbound private link, and inbound context control remain unchanged. The same applies to outgoing traffic. This means that your allowed domains and storage destinations in your outbound traffic settings also remain unchanged.

`08:27` Existing network policies stay enforced

All existing policies will be followed. Of course, the third point is connecting to your data. Most importantly, the outgoing traffic will remain in the same home region. The fact that we ran the workload in a remote cloud in a different region does not affect this aspect . So, accordingly, none of your network routes to your storage buckets and resources will change. You know, if you set up a private link there, you can still use it.

`09:05` Data access stays in the home region

Nothing changes. And it also means you don't need to reconfigure any policies for your storage bucket. So, that's all. This is all my material. We are very excited to work on this network backbone and, more broadly, on serverless computing for you. We believe this will be extremely valuable to our customers, which is why we are here talking to you about it to get your questions and feedback. A little note about terms.
So, a lot of teams at Databricks are working very hard on this. Our goal is to implement this capability in many serverless products. We start with AI runtime and serverless GPUs because the GPU shortage problem is more acute. And just like I showed in the demonstration, right? You can expect a preview version to launch soon, within the next few months.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.