Serverless Compute for Notebooks, Jobs and DLT

Summary

• Infrastructure Simplification: Eliminate infrastructure management overhead by using serverless compute on the Databricks Data and AI platform, reducing time spent on networking, security, and cluster configuration.
• Dynamic Resource Allocation: Enable rapid scaling with dynamic resource allocation that automatically adjusts to workload demands, reducing idle costs and optimizing performance for notebooks, jobs, and DLT.
• Development Acceleration: Streamline development cycles with serverless architecture that removes the burden of managing clusters, upgrading DBRs, and monitoring resource utilization.

Serverless Compute for Notebooks, Jobs and DLT

Talk by Roland Fäustlin, Product Manager, Databricks
Discover how Databricks serverless compute revolutionizes data workflows by eliminating infrastructure management, enabling rapid scaling and optimizing costs for Notebooks, Jobs and DLT. this video will delve into the serverless architecture, highlighting its ability to dynamically allocate resources, reduce idle costs and simplify development cycles. Learn about recent advancements, including cost savings and practical strategies for migration and optimization. Tailored for Data Engineers and Architects, this talk will also explore use cases, features, limitations and future roadmap, empowering you to make informed infrastructure decisions while unlocking the full potential of Databricks’ serverless capabilities.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction to serverless
• 1:03 Why choose serverless
• 3:52 Simplifying compute
• 5:03 Cost management practices
• 6:32 Performance and KPI benefits
• 9:35 Versionless infrastructure
• 12:43 Conclusion on serverless
• 13:45 Deep dive into architecture
• 23:49 Roadmap and future features

FAQs

What are the main challenges serverless compute solves?

This video highlights that managing infrastructure is hard; central data engineering teams spend significant time on networking, IP ranges, subnets, security, compliance, efficiency tuning, and DBR upgrades—up to 20% of team time on upgrades alone.

How does serverless compute reduce costs?

Serverless architecture dynamically allocates resources based on workload demands, reducing idle costs by eliminating overconfigured clusters and enabling automatic scaling without constant manual tuning.

What workloads can use serverless compute?

According to this video, serverless compute on the Databricks Data and AI platform supports notebooks, jobs, and DLT (Delta Live Tables) pipelines, covering diverse data workflow requirements.

What are the latest serverless performance enhancements?

The video covers recent advancements in serverless DLT, including cost savings, migration strategies, and optimization best practices drawn from real-world customer implementations.

Full transcript

0:00 Introduction to serverless

All right, we're at time. This is about the future of compute, serverless compute for notebooks, jobs, and pipelines. I'm super excited to talk about this. I'm a product manager on serverless compute, and I'm here with, one of the masterminds behind of what's happening under the curtain of serverless compute. This is one of many talks of serverless compute. Um so you know feel free to check out that list. We'll also talk about that later.
Um and the things that we'll cover is why serverless compute, why would you choose that? What are the benefits of it? I'll throw a bunch of KPIs uh and things that we've achieved over the last couple of years actually in serverless compute. Then Aaron's going to give you a look under the hood of how this actually works and how the magic is happening. And then we'll talk about what further improvements we're doing over the coming year.
All right, so let's get started. Of course, all the things we talk about, it's forward-looking, so no guarantees. First of all, why serverless for notebooks, jobs, and pipelines specifically? Well, what's the problem we're trying to solve here in serverless? Managing infrastructure is really hard. So, when we talk to many of you, you know, you're like, you know, okay, I can use data bricks. There's a lot of things that make my life a lot easier, but managing clusters and setting up the things is still pretty hard.

1:03 Why choose serverless

And we find that a lot of the central data engineering teams spend a ton of time on setting up networking, IP ranges, subnets, etc. This is pretty hard. Consumes a lot of time uh getting new IPs and things like that. Security on compliance, making sure that there's no data excfiltration risk, that everything is secure, right? including like egress and all the things managing efficiency making sure that the clusters are sufficiently large that you don't run into any out of memory issues all the time even as data grows but then at the same time that they're not too large so that you're wasting the resources of the company being efficient about it finding the right trade-offs between speed and performance right that requires constant attention and tuning and then ultimately also upgrading DBRs we've spoken to many of do and we we've heard numbers like up to 20% of the time of a central data engineering team is spent to just upgrading DBRs and we launch two um long-term supported DBR versions per year.
So that's a lot of effort, right? And if you don't do that then you're missing out on the latest performance improvements, meaning also you're missing out on a lot of cost savings you could be having. All right, those are the problems we're tackling with serverless compute because serverless compute automatically optimizes for those challenges. Serverless comput is simple. Instead of selecting individual infrastructure knobs like you know cluster sizes, instance types, number of workers, you're selecting business goals.
Do you want this to run fast or simple? And we'll see that in a second uh with screenshots and an example. We have strong cost governance. There's actually a separate talk tomorrow about that where we'll deep dive into that and I'll just touch on that in a bit as well. Serverless is very performant. It allows you to do fast execution. We have 5 to 10 seconds startup that we can achieve. We cache the environment uh so you no longer need to install things and it's maintenance-free.
We automatically upgrade DBR versions for you. We have some great stats to show about that uh in a bit. We also automatically prevent any issues from running your workloads like you run into an out of memory issue. We automatically fix that and give you a larger machine. Right? So this is all built on this serverless compute platform that powers all of these products. All right. So let's go in a little bit more detail.
How do we make it simple? I talked about choosing the business goal. Don't expect you to be able to read this on the left side, but this is the cluster form. This is very complicated. There's, you know, thousands of combinations you could choose here. So, this is hard. Serless makes this very simple here. Serverless is the default. It's selected. The only thing that you need to choose is, do you want this fast or do you want this efficient?

3:52 Simplifying compute

These are the performance goals. And then we continuously optimize towards those performance goals for you. This is how it looks like in practice. On the right side, you see a screenshot of the right side panel of a job where you can select this performance optimized mode. And when you select that, we give you, you know, a warm pool of machines under the hood and we start it really quickly for you. And that means if you choose performance, we're typically twice as fast than a classic cluster.
But if you don't choose that, then we run standard mode. We know efficiency is more important for you than speed. And standard mode is up to 70% cheaper than performance optimized mode. And this has been in public preview for a bit. And I'm super happy to announce that actually this is GA now. So this is rolled out now. So it's pretty awesome, right? 70% savings right there. And then we talk about cost management. Cost management is also much simpler in serverless compute.

5:03 Cost management practices

There's four levels to this and we'll go through it one by one. The most foundational piece of fors is I want to know how much I'm actually spending. You know, if spends too much, I want to know and I want to get an alert. And that's possible. We have system tables. We have templatized dashboards that you can freeclick install into your workspace. then tells you like okay who are the most spenders what are the jobs that are most spending and you can build alerts on top of that you know should go overboard you can attribute the cost to certain teams you can enforce tags and then you do a group buy uh in the analysis and you'll actually see that automatically rolled up in those dashboards and then you can say how much did the finance team spend and you can do chargebacks based on that and then you might have people who are not writing the most sophisticated code who are not like super experienced and so you need to prevent accidents from happening that could cost you a lot of money.
We already have infrastructure limits in place that limit the maximum amount of spend that you could have on a given workload for a given user. We also have timeouts now in place and we will be starting to predict uh how much uh a query actually takes and so you can shut that down or provide certain warnings for that user. And then the fourth thing that is coming so that is work in progress is choosing how much spend can actually be done on a per user per hour basis right so that's coming and we'll talk a little bit more about in the future things all right throwing some comparisons and KPIs of what we've achieved here uh at you so fast startup and execution in classic this was four to six minutes to spin up the cluster you know until you get it from the cloud provider and it's all ready and you paid for that And after it spun up, you had to scale it potentially up, took again time, then you have to spin it down again.

6:32 Performance and KPI benefits

So there's always like a bit of a lag. In serverless, we bring this down to 5 to 10 seconds for interactive workloads. So it's really snappy. You don't need that allpurpose cluster around all the time anymore. And we typically achieve uh twice as fast execution environment caching. So in classic, you had all of these init scripts. You know, typically you would set up an init script on a workspace basis and every cluster starting up in that workspace would have to, you know, download and install all of the libraries.
So that took a significant fraction of the actual execution time and you had to do it every single time you would start up that jobs cluster. And then it's also really hard to manage like okay I've got an analyst team, they need this environment, there's another team that needs another environment. It's pretty hard to manage that on a per teamam basis. In serverless we've introduced this concept of an environment. Think about this as like a requirements txt.
We analyze that. We download all of the libraries that you need. We install them. So yeah, we still have to do that one time, but then we take a snapshot and cache that. So every subsequent run, every subsequent user that uses that environment can load that from cache. And that happens in seconds. No download, no installation required. So that's pretty awesome too. And that reduces particularly for smaller workloads significantly the run duration.
Automatic infrastructure selection in classic you had to use this form that I showed before. It was pretty complicated to see which cluster which instance type I should choose right and you had to manage that over time and then data volumes grew and still you would get paged in the middle of the night. You would get an out of memory issue. This workload's not running. You had to go in there. You had to fix that find out what's the next cluster that I should choose. what's the configuration to fix the issue and then rerun everything.
You have to manually choose trade-offs between cost and performance. In serverless, we do this all for you. There's a bunch of advantages here. We pick the infrastructure for you and when we see, hey, you're running out of memory, we automatically give you a larger instance. We can fail over when there's stock outages from the cloud provider to other instance types that are compatible with your workload because we have all of that history, right? how much memory did you need and all the things.
And so what we actually see is the success rate uh is much higher on serverless. We have 89% fewer outages on serverless compute than we observe on classic compute. And that's just purely for the fact that there's a lot of things we can automatically remedy and that we can shield you from certain cloud uh exposures. So serverless compute is actually our most stable compute product. All right. And then we talk about versionless.
This one's pretty awesome. So, as I mentioned before, we have these LTS versions that we roll out, the DBRs, uh, the runtimes. Uh, there's typically two per year, and you have to upgrade to those DBRs to take the benefit or the advantage of all of the new improvements that come with it that come performance, that come cost benefits with it. And that's a lot of effort. I've heard people say like up to 20%. Right? And if you don't do that, then you're missing out on the performance improvements.

9:35 Versionless infrastructure

So that also costs you something and that leads to a lot of customers being months behind. You know, I think on average even like a year behind the latest LTS, right? And so with serverless, we have a completely new architecture under the hood. going to talk about that in a little bit. And that allows us to swap out the DBR effectively under the hood without any breaking changes. And we've been doing that for quite a while.
And the stats are pretty stunning. So we've done that in fact like 18 times over the last year for more than 1. 3 billion workloads with a success rate of 99. 998%. And even the little remainder there that just means okay we pin you on a previous version and there might be some human interaction required but it still means yeah you can continue running right and sometimes we did get complaints right hey why is this suddenly faster right why is this suddenly cheaper am I missing something and like that brings me to that last KPI here right and I think even reold said this last year at data and summit on DBSQL we see on average 52% better per performance year-over-year when you look at the average query duration and we're on a really good track record to do this for all of the other serverless products as well.
So serverless is only getting better and you're automatically benefiting from those improvements from the new DBRs from all of the performance improvements that we roll out, right? All right. But now you might be like, "Okay, but you know, how is this going to how much is this going to cost me?" And we actually looked at that as well. And so we we did a benchmark and we looked at, you know, what is a typical ETL workload like one terabyte ingestion into bronze layer, some transformation, and then going into silver and gold, right? cleaning up data, dduping, and all these kinds of things.
We're doing pretty well on that. So, what you see here is the cost on the y-axis for this one terabyte. Um, you see the performance optimized mode and you see standard mode here. And there's like some other people. And so, we see that we're typically uh 3. 4x cheaper than the competition. And so, that's a pretty awesome number, right? But how fast is it actually? Am I losing speed or anything else on that? So let me show you that number as well.
So this is the performance and as you would expect standard mode is you know the efficient mode not as fast as performance optimized mode um but performance optimized is which you would want to choose if you want to be fast is you know 3. 8 times faster than the competition. All right so we're actually doing pretty well in terms of both performance and speed. So that's awesome. All right, so key reasons why you would choose serverless.
Hopefully I could convince you. It's simple. I showed you a bunch of examples. It's performant, right? We just saw benchmarks. And it's maintenancefree. So you get all of the advantages of it continuously improving, right? So Spark is amazing, right? We've all used it and you know want to continue using that, but managing it and managing the infrastructure, these DBR upgrades is hard. And we're, you know, in the age of Vibe coding, right?

12:43 Conclusion on serverless

Don't do this yourself. let AI do this for you. There's a caveat of course, right? And the caveat is, you know, classic is there to stay, of course, right? Um there's we know there's a lot of workloads out there uh that, you know, need to have root access that have like very specialized structures that are, you know, migrating from somewhere else. Classic is here to stay. It's our most configurable and our most compatible product, right?
So, no worries there, right? With that said, I'll hand it over to who's gonna show you what's under the hood. Oh, and there's like lots of customers that this too. Customers like it too. Yeah, customers like it too. It's like Yeah. most important thing. Yeah. Uh, cool. So, yes. I want to talk a little bit about like what makes this problem hard uh and how do we solve these problems? Like I I can I can do that. I can create some warm pools.

13:45 Deep dive into architecture

I can I can make things spin up and spin down quickly. Um, so obviously there's a lot that goes into building serverless. There's, you know, the worm pools. There's how we manage the number of clusters, the size of clusters, how do we implement versionless, one of the hardest problems. Uh for this talk, I'm going to focus specifically on these two parts of the problem which are very core to the engine. You know, we're at data summit.
We often talk about spark. These are very spark ccentric problems. Um so since the very first version of spark, customers have had to manage clusters. That's within data bricks. It's outside of data bricks. And this has many configurations when I'm managing clusters. What how what size to give my to my clusters, you know, how many VMs, what type of VMs, and that directly uh dictates the price and performance that I see.
So there's a trade-off there. And also the number of clusters, you know, does every job get its own cluster leading to some inefficiency? Do I binack interactive workloads? And does that start to cause uh reliability issues? Um so these are all things that people have had to forever configure. So in serverless we took as one of our primary goals not to require users to configure these infrastructural knobs that uh result in clusters.
And so how do we solve this? Well three parts and I'll just talk through each of them briefly. So the first is spark connect. The second is gateway managing the number of clusters and third is our autoscaler managing the size of those clusters. So probably people have heard of spark connect. Uh it's a it's a fundamental innovation within spark itself. It's probably the biggest architecture change we've ever made to open source spark.
Uh in the traditional Spark model, if you've run a lot of Spark programs, you're probably familiar with this. Uh there's this idea of a driver, which is the head node of a cluster and executives, which are the workers that do the distributed processing. But the one key thing you may not have intuited is that there's this customer application like your code, right? The Python code or Scola code that you're running. And in traditional Spark monolith that would actually run on the same VM, the same machine as the driver.
It' be colllocated with the driver. That was a core architectural assumption. And that kind of works. But you might imagine that well what happens if my app uh uses a lot of memory or use a lot of CPU or have multiple applications packed on the same cluster? This can cause reliability issues, things to crash, things to become slow and then the entire job fails. So this was a core limitation within Spark with this tight coupling between the application and the cluster.
So Spark connect which is available in open source. So it's just uh part of spark uh is is a change in how this architecture works where now the ripple the application is there on the lefth hand side and it communicates to the driver over you know gRPC uh and so this means that queries are now the unit of submission as opposed to the low-level like py forj rddd kind of language um and so this core innovation is kind of the baseline is the fundamental building block that we need to now start to decouple applications and compute and really build more reliability seamlessly. ly.
Um so given that we have that layer of of RPC that we can separate the applications uh the first thing we did is introduced a layer of indirection the first topic of any computer science uh lecture. Uh so we built a spark connect gateway and application the way it works is an application comes they say hey I want to use spark they talk to the gateway the gateway says aha give you a cluster there's no other cluster in this workspace and that creates like this kind of dotted line between this app and the cluster that's the spark connect uh connectivity so requests will be sent from the app to the cluster another application comes in the gateway says okay actually I have enough room still on this cluster uh so I'm going to go ahead and assign you to the same cluster and then finally maybe some large application comes in maybe we know from the history or by looking at the query as incoming and so we actually grant it its own cluster.
Um and so this like kind of smarts this intelligence of trying to figure out uh when to spin up clusters and when not may sound trivial but it's actually a core uh complexity and I want to just spend one minute on that point of like well why do we actually need more than one cluster? Why can't you just have a single cluster for everyone? So even ignoring potential reliability issues, I want to give an explanation that relies on how there's a fundamental efficiency versus predictability trade-off when you talk about multiclustering um and that this is everyone has to deal with this today implicitly or explicitly um and then we have to try to solve it in serverless.
So here I have on the screen five worker nodes each worker node can do one task at a time part of a single cluster and in comes a request that request sorry that request is a single query. It's a green request like a notebook or a job and that query needs three workers for some period of time. So it's paralyzable. It uses most of the cluster. While it's running, you know, there's enough space. So other queries come in and something else comes in and runs.
Uh and then maybe by minute five, everything is is quested and all the queries have completed. So in this case, the green query finished within four minutes. So what we could do is we could sort of isolate those uh slots those workers uh for the green session. So that if the green comes back, it gets them back and it can run really quickly. That would be creating a separate cluster essentially. But this is the shared cluster case.
So those slots are free for anyone to use. And indeed in the next minute comes the yellow session and actually is a very large query. And of course everything is happy except that green comes back and now observes that it can't use the whole cluster and so it starts to queue. Uh so we have to wait for yellow to finish over you know minute 10 and then these guys get scheduled. The first one finishes and we have to wait.
So at the end of the day, uh the green query took seven minutes instead of four. So if you think about that from a admin perspective, this was actually awesome. This query was highly utilized. Every one of those workers was almost always busy. But if you think about from a user perspective, what they saw was that well sometimes I run it, it takes four minutes, sometimes it takes seven minutes. It's like twice as slow.
And so this is a core trade-off of how do you maximize efficiency while also granting as much predictability as possible. And so that is the uh core responsibility of the gateway is to estimate the size of incoming queries, take a look at their history over time if jobs are scheduled, keep track of the underlying clusters and their load, manage the spinning up and down of those clusters, and even we have the capability to migrate sessions uh to different clusters if we made a mistake.
So green got assigned to a cluster that's under load, we can actually move it to a different cluster uh so that maybe the next query uh will will see a better performance. So I just wanted to kind of showcase kind of one of the core features we were building in Trivolus. You can see how there's a lot of intelligence that goes into this that we have to build up over time and improve across the entire fleet to uh to solve these fundamental trade-offs and not have to expose clusters to you.
And so the step three uh the last step to talk about is cluster sizing. Um so clusters are not fixed capacity. In the previous diagram I showed five workers. Well, that's not really true. We can we can increase and decrease that number. uh and in fact the cluster count is even more important than the number of clusters for talking about how expensive or how fast a query is. So it's actually the primary uh indicator. Um so for this I want to give one more example of why there's not a single solution to this problem.
Why this is a hard problem as well. So suppose a query comes in and it it has 10 tasks this time not three but 10. Uh and let's say each task takes five minutes to run and every we're going to have a no warm pool. So every VM takes two minutes to start up. So in this case, let's say you know you're given this problem. Okay, 10 10 uh units of work come to you each take five minutes. How many VMs, how many workers do you want for that query?
Uh and maybe we'll take look at two options. Option one, let's spin up five workers. Of course, if we do that, uh we'll have to go two waves. So five tasks run for five minutes complete and then another five tasks run for five minutes complete. The end result is that the query runtime is uh 12 minutes. So the 2VM startup time plus the 10 minutes for two waves. So the user sees 12 minute latency and the cost is just that runtimes the number of VMs 60 minutes in this case.
Another option might have been to have 10 VMs and in this case um we have the cost is actually wrong in this case but okay sorry uh in this case the uh query runtime is 2 minutes plus 5 minutes because we only have to run it through one wave. So that's seven minutes and the total cost is actually 70 uh not not 84 sorry for the math there error. Um but as you can see what we've done is we've traded off. So if a user, if this were interactive, a user ran this query in a notebook, let's say they ran it, they go get some coffee, come back in seven minutes, the query is done.
But in option one, they might have to wait another five minutes. So they're actually sitting there having to wait. But if this is a query that's running overnight and no one's watching it, maybe you don't care whether it ran for seven minutes or 12 minutes, and you would have actually preferred to save the VM minutes from that. So uh this is a core trade-off in cluster sizing. Uh there's actually no single solution. That trade-off I mentioned is is quite fundamental but I will mention there's actually two things going on.
So the first is uh this graph is a Pareto frontier meaning that like you know you can move you can improve your query runtime and that will improve uh the query cost and that's kind of like the fundamental trade-off based on this kind of nonlinear system that exists in reality. Uh most of us when we try to configure clusters manually are actually not on this frontier. It's actually quite hard to correctly size and have the number of clusters a number of VMs correct.
And so usually we're inefficient. we we get worse runtime and worse cost both. So the first problem and the core problem is let's actually make sure that when we run workloads they reach at least some optimal point along this frontier. So that's like a core optimization issue. But the second point is actually where do you want to be along here? How do you value uh runtime versus cost and as showed before this is where we rely on um both form factor like jobs versus notebooks as well as that knob of peroptimize versus standard.
So, as opposed to having to tell us these low-level semantics like VM type and count, you tell us your target, your goal, but we do need that information to know where on the frontier to sort to live. Um, and over time we learn from customer workloads individually in an aggregate to sort of hit these targets and get reach that frontier. So hopefully this kind of showcases uh some of the complexity we have to go through to build the serless architecture.
So, I just ch sorry I just chatted about uh the gateway and sizing, but you can see there's a whole bunch of features and core infrastructure that we're building to make this better uh every day. So, I will now turn it. Awesome. Thanks. Cool. So, let's talk a little bit about the future of this, right? There's already a lot of cool KPIs that you've seen. You've seen that there's a lot of cool tech under the hood. There's more coming.

23:49 Roadmap and future features

So, one of the first things is default base environments. So, we heard about these environments that once you set them up for your workloads, they're really easy uh to manage and they save you a lot of installation time. But as an admin, you might want to say all of my analysts get that one environment. All of this, you know, machine learning scientists get the other environment. And that's what defaultbased environments enables you to do.
And you can use that QR code to sign up for the preview that is happening right now. The other one is Scala jobs. I know a lot of you are still using Scola. That's great. We love Scala and we have a preview for that. So you can develop Scala in your local ID which is what you see happening here on the screen. Um you can deploy fat jars and it includes things like JDBC OBC that's going to be custom JDBC OBC that's going to be possible with this.
So here's another link to sign up for that preview. And then we have ML support. Of course, you know, we live in the age of ML and AI and there's a lot of things you can already do on that single ripple node. That's single node ML psychic learn for example. But there's two more things that are coming. One is that you can get GPUs. That's actually being announced this week um that you have a beta where you can turn on the availability of accelerated mode where you can pick GPUs.
Initially that's going to be A10s, but later it's going to be H100s as well. And in Q3, we're also getting SparkML. So distributed ML as you know it from classic Spark clusters is then also going to be possible on serverless compute. Cost management. I already mentioned that we've got the first three layers of cost management fully done. And this is the fourth layer of like being able to limit the amount of compute per hour that an individual user can use.
So that's the rate limits and entitlements. That's the first bullet here. We're also going to predict how long an individual query is going to take and we will show you hey you probably as an individual you're probably running over the timeout because we can do that once we have the query plan and so then you can decide oh this was probably a mistake maybe I shouldn't have done that so that's coming and we're also adding more details to system tables to allow you to say was this run performance optimized was this run standard and there's more details coming there's actually a separate talk tomorrow specifically about cost management by push So check that out.
All right, this is like an overview road map. There's a ton of new features coming. Um I think I mentioned most of them already. So a lot of the stuff on the left hand side of that graph. I don't even mention anymore. As you have learned also from, there's a lot we've built and there's even more to come. With that said, thank you very much. And I think we're ready to take some questions. Oh.

Learn more about the Databricks Data and AI platform.