Cost Management for Serverless Compute: Notebooks, Jobs, and Pipelines

Summary

This video addresses three categories of serverless cost concerns—runaway spend, uncontrolled access, and accountability—using a real example of a UK customer who accidentally spent £5,000 in one night from a forgotten notebook.
Piyush Singh walks through new Databricks primitives including usage policy tags, statement timeouts, per-user entitlements to grant or revoke serverless access, and t-shirt-size rate limits for individual jobs and pipelines.
The video closes with a concrete action plan—configure mandatory tags, audit current usage, and enroll in the rate-limiting preview—and maps out the migration path from classic cluster policies to the new serverless cost controls.

Cost Management for Serverless Compute: Notebooks, Jobs, and Pipelines

Serverless compute removes infrastructure overhead, but platform teams still need to govern who can use it and how much each workload can spend. this video covers the cost-control model now for serverless Notebooks, Jobs, and Lakeflow Pipelines, current features, recent developments, and the roadmap ahead.
We walk through the new primitives: entitlements that grant or revoke serverless access per user, group, or service principal, and per-workload rate limits that cap the spend rate of an individual Job or Pipeline.
Attendees leave with a concrete migration path from cluster policies to serverless cost controls.
Talk By: Piyush Singh, Staff Product Manager, Databricks;

Chapters

FAQs

How do you prevent runaway serverless compute costs in Databricks?

This video covers three mechanisms: statement timeouts that automatically stop runaway interactive notebook sessions, per-workload rate limits that cap spend for individual jobs and pipelines, and usage policy tags that enforce cost attribution on every serverless workload. Together these provide the guardrails that classic compute had, now available for serverless.

What are serverless entitlements in Databricks and how do they work?

Serverless entitlements are a new access control primitive that let platform administrators grant or revoke the ability to use serverless compute on a per-user, per-group, or per-service-principal basis. This video demonstrates how to configure them from the Serverless tab on the Compute page, making it straightforward to restrict access for new users, interns, or teams that should not have unrestricted serverless capacity.

What are rate limits for serverless jobs and pipelines in Databricks?

Rate limits let administrators assign a t-shirt-size spend cap—such as small, medium, or large—to individual serverless jobs or pipelines, preventing any single workload from consuming the full available capacity. This video explains the distinction between per-workload limits for jobs versus shared limits for notebooks.

How do you migrate from classic cluster policies to serverless cost controls in Databricks?

This video provides a concrete migration path: first set up mandatory usage policy tags for cost attribution, then audit current serverless spending to understand baselines, and finally enroll in the entitlements and rate-limiting preview features to replicate the guardrails that classic cluster policies provided. The goal is to reach the same level of governance on serverless that teams already have on classic compute.

Full transcript

[00:07] Good afternoon everyone. Thanks for joining us here today. My name is Piyush. I'm on the product team here at Databricks. I work on serverless. And I'm going to talk about cost. So we're going to get into all of the details around serverless cost
[00:20] management for notebooks, jobs, and pipelines. And if you're you know a little bit worried about serverless costs, you're in the right place. So let's get into it. So this is our standard disclaimer. You
[00:33] know, I'm going to talk about some things that are coming up in our road map. Of course, I can't guarantee everything's going to shake out exactly like I'm presenting on these slides, but I think it's going to be pretty darn close.
[00:46] Cool. So let's start with a story that I think is going to sound a bit familiar to everyone here. So this is a real you know, customer case study. I was speaking with one of our customers based out of the UK. They had a data scientist who left a notebook running overnight.
[00:59] And you know, it was just one day. They came back you know, the following day and they realized they had accidentally spent 5,000 pounds. So this wasn't you know, the user being malicious, right? It was just purely an
[01:12] accident. It wasn't a bug either, right? It was just an accident on behalf of the user. And because serverless didn't have the same guardrails you're familiar with in classic, you know, the compute scaled and they were surprised by the bill.
[01:25] And so even if you haven't heard of you know, this exact example internally, I'm sure this is something that you're worried about and that's why you're here. So what can we do about it? So when I talk to customers, I hear kind of a few categories of concerns, right?
[01:38] So you know, the main concern like I just highlighted is runaway spent. What happens when something gets stuck or something gets forgotten and it racks up a huge bill.
[01:50] Or alternatively, serverless is extremely powerful. I'm not sure if I want every user to have the full power available to them at all times. I have new college grads, I have interns. They shouldn't really be able to scale serverless compute to the moon and you
[02:03] know, throw thousands of dollars away by accident, right? And then lastly, um how do I make sure the right people are accountable? How do I know that uh a charge that I see in my billing records actually belongs to one team or
[02:15] another? How can I How can I make sure that the teams that are spending too much are held to account? So, what are we going to do about these three things? First, why doesn't this happen on
[02:27] classic compute? What tools do we give you that structurally prevent these things? And there's three things. So, the first one is tagging, right? And that's used mostly for chargeback cost attribution. So, if I have several
[02:40] clusters, yeah, knowing the name of the cluster or even the cluster ID when I see my billing records is great, uh but I might want to understand which groups of clusters are are kind of coming from one specific cost center, project, team.
[02:52] Tags help you aggregate your costs along the boundaries that you actually care about within your org. And so, you can do that today on classic. The second thing is entitlements. So, using classic cluster policies, you can
[03:05] restrict who is able to create clusters, right? So, who can create all-purpose compute, who is able to create job clusters, and then you can assign access to those. So, if somebody creates all-purpose compute, you can control who
[03:17] can use that cluster by assigning individual users or groups or service principles access. And the last thing is rate limits. So, in classic compute, that is very manually configured. You pick machine types, auto scaling limits,
[03:31] uh all of the different knobs and dials that manage your costs. But of course, many of these things are missing today on serverless. So, what can you do today and what's coming?
[03:43] The first thing I want to highlight is cost attribution is already possible today on serverless. There are some small gaps, but for most, I think this is a hopefully solved problem. The second thing is access controls.
[03:57] That is something we actually just launched in private preview this week. The last thing is limiting spend. So in classic, again, you can set kind of guardrails on how quickly people can spend.
[04:09] We will be rolling out similar controls in July, and I'm going to talk about how those will work in just a second. So starting with attribution first, because it's a mostly solved problem.
[04:23] We have a tool called usage policies. Usage policies let you enforce tags on serverless workloads. You can kind of think of it as the equivalent for mandatory tags and cluster policies on classic for serverless.
[04:36] Those tags flow through to your system.billing.usage records, and then you can aggregate costs on those tag boundaries. And of course, those tags will flow through to tools like the governance hub, cost dashboard, your cost reporting, and
[04:49] your account console, um all of those places. This is an actual query you can use to aggregate by any arbitrary tag you define. Uh so you can group your serverless cost by which team or which project or which cost center is
[05:01] responsible. Once you have those tags in uh you get really great observability, right? So account budgets can trigger email alerts when your spend crossed
[05:13] over a certain threshold. Um you have the new cost governance hub, like I just mentioned. Uh it gives you detailed breakdowns and drill downs into exactly where cost spikes are coming from. But of course, these are all reactive,
[05:25] right? An alert tells you uh after the fact that you've spent too much, not before. How can we prevent it? The first control I want to talk about is a relatively new one. So we actually
[05:38] just introduced a workspace setting, which will allow you to define a timeout for serverless interactive for every query that runs in your workspace. So, this is just like DB SQL, where you
[05:50] can set a statement timeout. Um by default in DB SQL, I think the timeout is 2 and 1/2 days. On serverless interactive, we've capped it at 2 and 1/2 hours, but of course this is configurable by you.
[06:02] The thing that I'll add here is DB SQL is actually in the process of making statement timeouts specific to warehouses. So, you will actually be able to set different timeout thresholds on a per warehouse basis. That's coming very
[06:15] soon. We're going to do exactly the same thing for serverless interactive. I'll talk about how in a second.
[06:27] Next, controlling access. So, the idea on classic, right, is on a per user, per group basis, you get to decide what people do. And this is what we're rolling out in private preview right now. You will soon be able to independently
[06:39] define which users and groups get access to serverless notebooks and separately serverless jobs and pipelines. Let's take a quick look at how that's going to work.
[07:06] So, I have a small >>
[07:19] they actually will see serverless entirely disabled for them. So, when they hover over the serverless option, you can tell it's disabled, and we actually indicate to them, "Hey, you have to go talk to your admin in order to get access."
[07:31] We're going to do exactly the same thing for jobs as well. Once you create a job, you'll be warned if you don't have access to serverless compute. You'll still be able to create a job because you might want to create it and then update the run as to a service
[07:43] principal that does have access. Uh but we will warn you that you don't have access. And in notebooks, you won't be able to connect at all. So, we'll define access independently, and where that will live
[07:55] is on the compute page. So, in the compute page, we're going to introduce a new serverless tab. And there, we will have independent compute for interactive, and uh which means notebooks and DB connects, and
[08:09] automated compute, that's jobs and pipelines. You can configure access independently for users, groups, and service principles. Very simple to do. All you have to do is uh open this little menu off to the
[08:22] right. Let's see if I can grab it really quick. And you can configure it directly here. Alternatively, you can also configure this via the rule set API, and I'll show you how in just a second.
[08:41] Great. So, that's access controls. One of the most common questions I get is, "Hey, I I don't define things in the UI. I have hundreds of workspaces. I have hundreds of thousands of jobs. I have, you know, thousands of users." All of this will be accessible via our
[08:53] APIs and via other surfaces like Terraform and uh you know, SDK, CLI, etc. Everything downstream from our APIs will support configuring access to serverless in this way.
[09:11] The next piece of control that we're going to introduce is rate limiting. So, you can now control who gets access to what. And in July, we will start introducing the capability to control how much
[09:24] serverless compute your users actually get access to. We're going to start with jobs and pipelines, and you will be able to define a t-shirt size-based limit to control how much those jobs can scale.
[09:38] You can really think of this just like the max auto scaling limit in your classic clusters.
[09:50] One thing to note there, once you hit the limit, your workload will get throttled, right? Oops. Um so, I mentioned jobs are going to have limits defined. They're going to be per workload.
[10:03] The reason for that is we don't want your jobs to contend for resources. So, when you pick a t-shirt size in your in your compute when you're defining it, you each job that's executing or each pipeline update that executes gets an
[10:16] independent copy of that limit. So, one production job will never get throttled because another one's running at the same time. We know that's not ideal for interactive usage, right? So, if users are doing exploratory work in notebooks, right,
[10:28] exploratory data science work, it's kind of hard to, you know, understand what costs you're going to be and you want to bound costs for a particular team overall. So, for interactive, we're going to introduce shared limits, just like classic all-purpose compute.
[10:41] You'll be able to define for a group of users the overall amount that they're allowed to consume as a rate limit, defined as a t-shirt size.
[10:54] What do these t-shirt sizes actually mean in practice? So, DB SQL also has t-shirt sizes, but the way they behave compared to notebooks and jobs will be different. In DB SQL, when you pick a t-shirt size, it's a fixed size.
[11:08] The scaling of warehouse endpoints is done by incrementing or decrementing the number of clusters behind that endpoint, but they're always a fixed size. So, if I create a warehouse with an XL t-shirt size,
[11:21] it's going to scale in increments of that XL t-shirt size. Alternatively, for jobs in notebooks, the t-shirt size that you select is a max cap.
[11:33] So, if you pick XL and you have a workload that only needs a small amount of compute, we'll only allocate a small amount of compute. You'll only get billed for a small amount of compute. So, it's still usage-based pricing.
[11:46] It's just you can cap how much people can use at any given point in time. And again, the way that works is roughly a max auto scaling cap. You can think of it as a limit on the max number of VMs a particular job or notebook execution
[11:58] gets. So, bringing it all together. Now, we have cost attribution through usage policies.
[12:11] You can access by limiting who gets access to serverless compute. And soon, you'll be be able to define your own custom compute and set t-shirt size-based rate limits inside of it to
[12:23] control how quickly users' jobs and notebooks can execute.
[12:35] some of you may be in a preview for usage policies, which similarly restricted access. We're going to make this more fine-grained. The feedback that we got from users over and over again is uh granting a policy that gives uh people access to everything in the
[12:47] platform is too broad. Uh and so, that's what we're starting to do. And you're you're seeing that here at Summit, right? So, now we have independent controls for serverless notebooks and jobs and pipelines. AI Gateway will independently and separately control AI
[13:00] spent. Um if you have questions about that, I can answer after the talk. Uh and then, we're going to introduce similar controls for apps and Lake Base in the future. So, across everything, we're going to start giving you more and more control
[13:13] uh at a finer grain, so you can actually limit what users are doing on different parts of the Databricks platform. So, what should you do after this talk if
[13:26] you haven't already. Well, the first thing is understand what usage you already have. So, start enforcing tags through usage policies. This will get you a crystal-clear picture on which org,
[13:40] which team, which project is driving your costs. And this kind of aggregation gets pulled into every cost surface that we have in Databricks. So, it's really important to set that up. And then of course, once you have those tags, you can start monitoring using our
[13:53] cost governance hub, which we'll show you shortly, and budget alerts. Next, if you want to restrict access to serverless, go and check who is consuming what.
[14:06] Before you remove a user's access or a group's access, go and see what they're spending. See where it's coming from and understand why they're doing it, so that things don't start suddenly breaking if you revoke access.
[14:21] Next, you can decide which users and groups will get access to interactive and automated computes. And if you want to start controlling access, you can enroll in the preview that will let you do this starting this week.
[14:36] So, to summarize, you will soon be able to control access to serverless compute, and next month, you will be able to limit how quickly your users can consume
[14:48] serverless compute for jobs and pipelines. Interactive will come a little bit later, probably more like Q3, but we're working on that as well. And hopefully, you can already attribute costs, and if you haven't set that up
[15:01] already, you'll go ahead and do that now. If you take a quick picture of this QR code, it actually opens up a little survey form. So, if you're interested in this preview, you can just give us your
[15:13] details that way, and we can go ahead and start enrolling you. Well, of course, if you have an account team, keep them in the loop as well. You can add the account account reps email in that form. Um just so we're all on the same page. But if you're interested in
[15:26] this preview, if you're interested in these controls, definitely check out this link, fill out the form, let us know, and we'll go ahead and take you in to that preview.
[15:39] And I will pause here for questions. One other thing while we're taking questions, uh there are several related talks about serverless. So um if you haven't checked out some of these and they're of interest, definitely go uh check out the sessions that are going to be later today and tomorrow.
[15:51] Uh and of course there's that QR code again if you're interested in the preview that I just mentioned. All right. Let's open it up for questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.