Versionless and Environments for Notebooks, Jobs, Pipelines: Safe DBR Upgrades and Cached Packages

Summary

Engineering teams historically spend about 20% of their effort on DBR version upgrades that deliver no direct business value — Databricks versionless serverless runtime eliminates this entirely, with 6.7 billion auto-upgrades completed since general availability with 3.5x fewer regressions and a 24% performance gain.
Lakehouse Replay continuously shadow-tests a representative sample of real production workloads against incoming Databricks runtime changes before they ship, and a workload pinning service has kept 229 pinned workloads stable last quarter with zero customer-impacting incidents.
The Environments model delivers a unified, portable dependency strategy for notebooks, jobs, and pipelines — with a PyPI proxy, workspace-level default package sets, serialized notebook dependencies, and Maven support on the roadmap — extending eventually to classic compute as well.

Versionless and Environments for Notebooks, Jobs, Pipelines: Safe DBR Upgrades and Cached Packages

On classic compute, upgrading a DBR version and stitching together cluster libraries, init scripts and Spark configs consume a meaningful share of a platform team’s time, and every upgrade risks breaking production pipelines. Serverless eliminates both with versionless runtime and Databricks’ Environment model. this video covers the two pieces that make versionless trustworthy at enterprise scale. Lakehouse Replay continuously replays a representative sample of real production workloads to validate Databricks runtime changes before they ship, catching regressions with zero customer effort. Environments deliver a unified, portable dependency strategy with faster startup, tighter governance and clearer observability for notebooks, jobs, and pipelines. Attendees leave with a clear picture of how versionless and environments fit together and how to plan their move off manual DBR upgrades.
Talk By: Justin Breese, Staff Product Manager, Databricks ;

Chapters

FAQs

What is versionless runtime on Databricks and how does it work?

Versionless runtime on Databricks serverless means the platform automatically upgrades the Databricks Runtime on your behalf without any customer action, so teams no longer need to test, plan, and execute DBR version migrations. The Spark Connect client-server architecture separates the client environment you control from the server runtime Databricks manages, allowing the server side to upgrade transparently. This video explains how this model has processed 6.7 billion automatic upgrades since general availability while maintaining a regression rate 3.5x lower than classic upgrade cycles.

How does Lakehouse Replay protect production workloads from regressions?

Lakehouse Replay continuously shadow-executes a representative sample of real production query plans against new Databricks runtime versions before those versions are released, catching performance regressions or correctness issues with zero effort from customers. When a workload is flagged as sensitive — through a process described in this video as workload fingerprinting and pinning — it is held at its current runtime version until the regression is resolved, preventing any customer-visible impact. Last quarter, 229 workloads were pinned through this system with no customer pages.

What are Databricks Environments and how do they differ from classic cluster libraries?

Environments are a unified, portable dependency model for notebooks, jobs, and pipelines on Databricks serverless that replace the combination of cluster libraries, init scripts, and Spark configs required on classic compute. They support workspace-level default package sets for teams, serialized per-notebook dependencies that travel with the notebook across jobs, and a PyPI proxy for faster and governed package installation. This video explains how Environments reduce startup latency to 4.8 seconds at P50 and make dependency management observable and consistent across all compute types.

Why do DBR upgrades consume so much engineering time and how does serverless fix this?

According to this video, engineering teams at large Databricks customers spend roughly 20% of their total engineering effort on DBR version upgrades — testing compatibility, updating cluster configs, and fixing regressions — work that delivers no business value. Classic compute requires manually managing the full stack of runtime version, cluster libraries, init scripts, and Spark configuration, creating a fragile combination that breaks whenever any component changes. Serverless versionless runtime eliminates this entirely by delegating the upgrade responsibility to Databricks and providing safety guarantees through Lakehouse Replay and workload pinning.

Full transcript

[00:08] This is versionless runtime in environments. Oh my. I am Justin still. Um I'm a PM at data bicks and I focus a lot on runtime and things. So
[00:20] real quick, who am I? Again, I'm Justin for those listening. Um been at Data Bricks about seven years. I've worked on some greatest hits like Spark uh was the original PM for Photon. I did Delta way
[00:32] way back in the early days and I've been spending the past three years on making the runtime versionless and unifying the the DBR across all of data bricks. So a lot on the release infrastructure side
[00:45] and then I also uh lead our dependency management efforts. So dependency management things like cluster policies, containers, firstparty artifact registries, you
[00:57] know, fun things that we're working on like that. Um environments and serverless, all sorts of things. I live in Tanga, California. That is in LA region up in the mountains. Um don't
[01:09] worry, I'm not in LA, just above. It's fun. And I have a wife and then two little tyrants that are really cute and they tell me to do things and I listen. Uh, you can message me, send me email, jdatabbricks.com. You can troll me on
[01:22] Reddit, just join Justin. I I do reply there. Um, but be careful. If you troll me, I'll troll you back. I'm really good at it. Anyway, enough about me. 18 minutes and
[01:35] 37 seconds left. So, a few years ago, we set off on a bold new direction, something where no one has gone before. We wanted to create the best-in-class data engineering system from first principles.
[01:48] And so three of those things were stop managing infrastructure. Just stop it. Don't do it. Not cool. Friends, don't let friends do that. Stop worrying about upgrades because they suck. Um,
[02:01] traditionally when you talk to like engineering teams at customers, they spend about 20% of their effort upgrading. It means nothing to the business. It really doesn't. They hate it. I hate it too. and then make the entire UX crazy fast. Like so people
[02:16] would traditionally like start the cluster, go get coffee, come back, drink half a cup, the cluster's up, go fill up your coffee again while it's bootstrapping. Now sit down, it's 30 minutes past the hour, you're ready to
[02:28] work. Yeah. So stop that, too. Dev loop, slow dev loop is bad dev loop. So where are we now? This is like a little bit of a check-in. We've been generally available for 2 years and change like 2 years and
[02:43] 1 month. So I don't know 2 years and 12% of a year I don't know. Um so where are we? The start feels instant. So when you're in serverless notebooking job the first command latency P50 is 4.8
[02:56] seconds. Sorry if you're in a P90. Ah I you know I can't help you. We're working on knocking that down too. Uh but the P50 half of you people it's going to be 4.8 or better. Um, dependency
[03:08] installation time, something that's super near and dear to my heart. We're at about 6.9 seconds for P50. Now, let's compare these two classic. In classic, the average Python or PISpark job takes
[03:20] about 5 minutes of runtime. I'm not including the cluster spin-up time, but it takes about 5 minutes of runtime, of which two two minutes of that or 40% for you mathematicians is just dependency installation time. So, we go from
[03:35] 120 seconds to 6.9 seconds. That's big and it just happens out of the box. And we'll talk about all these things, how and why. And then on the upgrades part, cuz they suck. Um, so
[03:49] a fewish years ago when we started telling customers like we're going to do this model, we're going to auto upgrade you, they said, Justin, you're on crack. And I said, yes, I am, but I'm right and it's going to be fine. So, um, we've
[04:01] done scary upgrades from DBR14 all the way through, uh, so when I say 14, 140, 1, 2, 3, 15. I don't have to keep counting, but we're all all the way, we're up to the realms of 18.3
[04:15] in Spark Parlance. We've gone from Spark 3 to Spark 4. We've auto upgraded everybody 6.7 billion times and no one really knows about it, which
[04:27] kind of is weird. As a PM, I want everyone to know about my feature. But if people know about my feature, that means they're pissed off because I broke them. And so I don't get yelled at as much. So I guess that kind of means it's doing a good job. It's not perfect. And
[04:39] we'll talk about that. But really, when I look at it in aggregate, um, in terms of customers reporting regressions, it's about 3.5x. You know, when you normalize it, it's about 3.5x less fewer customer
[04:52] reported issues than classic compute. Does that mean we we don't have issues? No. We hide them away from you. And I'll talk about that in another slide. Um over that course in time, your workloads since we've auto upgraded, they gotten
[05:05] better, faster, stronger with zero effort. 24% improvement on performance. So all the way from 14 to 18, 6.7 billion times, less uh customer visible
[05:17] breakages, 24x speed up. Now the the ones the one thing I really love is there are in the past year we've had five months when zero customers reported
[05:30] any regressions and that's awesome. I can't say that on classic. I get yelled at a lot on classic but never in serverless. There's been five months where it's been super duper quiet and we've upgraded. So you know all these
[05:42] upgrades there's about been 150 upgrades in this time. So we've done a bunch of things. It's been scary to customers, but the reality is it just works. Now, what is the magic of this model? How did
[05:54] we do it? Great question. Glad you asked. Uh, we have a client server model. So, Spark Connect underneath the hood. What it does is um we we take all the dependencies out of the server, we put them into the client. The client's
[06:07] job is to create an unresolved logical plan, a data frame. Then via gRPC sends it over to the server, this big server that then does things. Uh the optimizer
[06:20] is there. It distributes off to executors and then it streams the result back and there you have the results. So super simple model. So it's the client server architecture. Um again based on
[06:33] Spark connect built into OSS Spark and language agnostic. There are um let's see I think five you can write clients in five different languages I think actually five that are out there and
[06:45] there's another five in the works uh greatest hits like go swift scola pispark and one more that I can't remember um for you cobalt fans no we're not working on one uh but we are working
[06:57] on one for C++ as well so fun stuff that's the magic now how does it work so by having this separation by having this client and server what we actually can do is um on the server side that means
[07:12] we can operate the entire fleet on a single image a single warm image that is what allows that very very fast startup and scale up. That's the magic behind the scenes because we're not letting people customize the images.
[07:25] So what we have underneath the hood is um you know you've got your app. So think about it in this this red box the that's a mixture of the serverless environment version Think of it as a a
[07:39] base image Docker underneath the hood mixed with your code and and packages. And so um these environment versions I think we've now shipped five of them. They're each one is supported for for
[07:51] three years. So think just like traditionally how we support DBR supported for three years. It comes with a fixed environment. Um sorry to use the word environment over and over but whatever it is what it is. Uh, so it comes with a fixed OS version, fixed
[08:04] Python version, fixed set of Python packages. You see where I'm going? It's a versioned client. Then when we release a new environment version, oftent times the OS version can change, the Python version can change.
[08:17] But again, this is all up to you as the as the user, as the developer to define, say, hey, I want to use version one, I want to use version two. We never autoupgrade those. Think of that as the contract that you're interacting with.
[08:29] You own that. You own your dependencies. We own the server side and make it magical. So when you all powers combined, you combine the environment version that you're using with your code. That is
[08:42] what creates the unresolved logical plan sends it over to the wire and in the event of a UDF for my astute colleagues here uh we actually replicate that entire environment from your app inside
[08:55] of the UDF in a sandbox. So you don't have to worry about like traditionally in classic spark you would say like oh make sure you add these packages to the driver oh and these ones go to the executor like you don't have to do that with this model we just do that
[09:08] automatically now auto upgrading is really easy right anyone can just go into a cluster manager and say yeah go from DVR 14 to 15 start what could go wrong everything
[09:21] so auto upgrading is easy auto upgrading and it actually works is hard and so to do to that is the release stability system. Um this this is the magic behind the scenes um that has enabled us to
[09:33] upgrade the 6.7 billion workloads. Uh it includes a lot of things and I'm not going to run over all of them here, but uh basically how it works at a high level is we have a workload fingerprint. So we can track this fingerprint and
[09:46] runs over time. We can we can get all sorts of metadata like how many bytes did it consume, how many bytes did it read, how many bytes did it write, how many executor hours did it consume the last time, what does the job look like and we have we can we package that into
[10:00] historical runs. Um there's ML models that we use for error classification, triaging, anomaly detection, reports and alerts. And then the the real star of the show is this pinning
[10:12] service. pinning service can when it when it starts to understand there could be a regression for an image, it will actually pin a fingerprint back to a previous version. Now, that might sound complicated, but here's a really simple
[10:26] way of how it works. And I have black arrows in between all these, which you can barely see, and I'm sorry for that. Missed that one. And anyway, how it works. Let's just say you have a given fingerprint or a given workload and it's
[10:39] been running fine for weeks on 18.1.1 underneath the hood. All of a sudden, we introduce 18.1.2 and it breaks. Oh no, terrible. Um, what we do is um since the
[10:51] workload has retries enabled, it will just retry on the last version that was good with 181.1. And if we see that now succeeds in the previous version, we say, "Aha, there
[11:04] could be a regression that kicks off an automatic triaging service." Um, so again, the user has no idea any of this is going on. We triage, we fix our bug and say 18.3 and as soon as 18.3 is out,
[11:17] we unpin that workload from 18.1 and we move on. So I did put a caveat in there. I said auto retries. Now, by default, all serverless workloads have auto
[11:29] retries enabled. And when you look at it in the fleet, it's about 95% of workloads have these retries enabled. So, what that means is for 95% of the workloads out there, if a user runs into a regression, don't worry, you got some
[11:41] protection here because we keep track of based on the release stability system. Again, we have historicals and we can remember, you know, what happened.
[12:06] single customer. All that you would see is like, hey, maybe the run took a little bit longer this time versus yesterday because maybe it failed on the it failed at first and then it retrieded onto the second version. So, all you see is looking at your runs, you're going to
[12:19] see a little blip in the graph in total runtime, but they're all going to be green. Same thing when we when we unpin it, it'll just be green. So people don't know. And what is you know what is the impact of that? That alleviated 42
[12:32] SEZero tickets on our end that customers would have had to have reported. So you guys would have gotten paged in the middle of the night. We would have gotten paged in the middle of night. This avoided that. But 15 regressions
[12:46] still were customer reported. These weren't breaks. These were mixture of correctness. um hangs, etc. So, what are we doing about these? Like, come on, Justin. Be perfect. And I'm working on
[12:58] it, and here's how we're doing it. Lakehouse Replay. Lakehouse Replay is in beta as of last week. This is a super super cool piece of technology. Think of it as an AB testing shadow or an AB
[13:10] shadow testing service. So, your production workloads happen. They're just running. Doesn't matter. They're in serverless. Um doesn't matter if it's DBSQL or serverless notebooks and jobs. And what we do is we take that
[13:23] production workload and then we take that spark plan and we run that on a B-cluster. That Bcluster could be Raiden like Ali and Reynold talked in the um the other day during the keynote. It could be an upcoming nightly build of of
[13:36] a you future release either way. Um and now we can operate this across the fleet. So we can basically look at all the distinct hashes out there and we can run them against upcoming releases and we can look for things like performance
[13:48] regressions, breaks, correctness, all sorts of fun things. Now the super duper cool thing, this costs zero dollars in compute cuz we're eating the compute. We're spinning up serverless clusters to
[14:00] do this. So unit testing is great. Unit testing isn't perfect, but to me, the holy grail has always been like, how do we run safely on production workloads with zero customer effort? And this is
[14:13] exactly what Lakehouse Replay does. We've been in private preview for about a quarter on 31 customers, beta as of last week. So now there's hundreds of customers using it. Um, right now it's only for reads, but don't worry, users
[14:25] don't have to do anything for this. It's just we we understand which query is viable for this service. It just runs, think of it as a background service that just runs behind the scene. So, um, this is coming out. It will be, um, you know,
[14:39] this is something good. Everyone wants this because if we can we can better understand workloads that users are running in production, we can better avoid regressions. All right, environments. Five and a half
[14:52] minutes left. We will get out on time. All right. So, how does dependency Oh, we have what? Wow. Hey, uh, sound guy. Can you come and help, please?
[15:05] Friend,
[15:19] >> Yeah, >> just fell off. >> Yeah, >> I didn't do it. I swear >> he did it.
[15:34] It just wants to be there. >> Yep. I got you. >> It was a slight disconnection on it is all. >> Oh, all right. >> Yay. We're back. Thank you. Good.
[15:47] >> Oh, it's all good, man. Thank you. >> All right. Slideshow. We're back. Okay. won't go through this in the interest of time. So, dependency management in
[16:00] serverless, it's different. Yes. Um, you can blame me for it. Don't worry, it's going to make sense as we get there. So, we spent a lot of time interviewing customers to try to understand like a
[16:13] better way of doing dependency management. Um, in server or in classic, it's really complicated. There's a lot of different layers. There's many ways of doing it. There's cluster scope libraries, there's init scripts, there's
[16:25] docker, there's a lot of different ways. And what we notice is a lot of time in init scripts, people are just doing setting pip.com things like they're just sending the default package repository. So we said, hey, how about this? Let's
[16:37] just make a super super simple feature that enables customers to just enables an admin to set it once at a workspace level and then it just works across classic, serverless, STP, etc. So, this
[16:51] has been out there for uh been GA for nearly a year. Admin settings, compute, and then you'll see like pack uh Python package repositories or default Python repositories. Either way, super duper
[17:04] simple. It just works. Um so, Python only today. However, we will be um I was actually reviewing the design doc yesterday for Maven. So, Maven friends, Scola friends, this is coming. Don't
[17:17] worry. So yeah, friends don't let friends set index URLs. The next thing the treat we gave to admins was um workspace based environments. This enables admins to define a default set
[17:29] of packages for users on their workspace. What it does is admins define it. We then create a virtual environment of it. We store it on object storage. Then when a user is using a new notebook, all they do is click on the
[17:42] environment panel dropdown. They click on the the um workspace base environment that they want. Then they click apply and they have the packages. You don't have to start up, wait, install, wait two minutes. I mean, this is where the
[17:55] sevenish seconds come from. So, super simple. It just works. And you can choose a default one. So, whenever anyone creates a a new notebook, it will by by default use the default.
[18:09] All right. And then you can also reference these in a job. Now, with um serverless, we took a different approach. So in the notebook itself, we actually already serialize the dependencies. So if you use the
[18:21] environment panel and you put the dependencies in there, they're serialized into the notebook. So when in terms of running a job, you just click schedule and it will have the dependencies. However, we do let users um override that notebook if they want
[18:34] if they want to use different set of dependencies. And um recently, I think in the past couple weeks, it's now in beta. you can leverage those workspace-based environments directly in a job as well. So, that is out there.
[18:48] And then also, oh, and I'll also little pro tip, little side fun fact cuz we're all friends now. Um, we actually just put a another base environment out there called ML. You can kind of see it in the
[19:01] drop down really quick when it shows up, but it's basically all the ML libraries you've known and love in classic are now in serverless in a base environment. All right. Then other things admins will like um and security infosc can't forget
[19:15] our friends there is we are uh putting uh coming very very soon like probably like a monthish out from putting this on a um we're going to have a proxy for pi. We've all been impacted by like the
[19:28] trivy exploits and all those fun things. What this is is every single data bricks workspace will um by default get a pi proxy. It's just a proxy that sits in front of default pi. And what it'll do
[19:42] is it'll have a 24-hour delay on net new packages. That gives the security community time. In average, we see they find vans in about 3 to four hours. So, this is something that's really, really good. Um, helps keep vulnerabilities
[19:56] away. What about the users? We'll move quick because I know we're pretty much out of time. I talked about how dependencies um live. They're already serialized in the notebook. And yeah, there's our friend ML. You can see it's
[20:08] brand new. Um, oh, environments. We're going to be bringing them to classic, too. That's upcoming. So, if you want in on a preview for that, contact info is at the
[20:20] end. Jay at databicks.com. So, the idea is like you'll be able to use environments on classic as well as serverless and and back and forth to your heart's content. Oo, environment variables. This one's coming out in July. We'll have a beta.
[20:33] So at a job level be able to define uh environment variables. So super easy. Um ooh and we'll have M files as well as key value pairs for you M lovers. The
[20:46] weight is no longer. All right. So this is where I'm like closing it out because we're out of time. So I talked about a lot of things today and so like if if you like nerd research I mean you know research papers
[20:59] not nerd papers. This is one that I wrote last year for Sigmod, which is about like the whole release stability system. It's about the whole pinning service and how that works. It's like a five-page read. It's not too bad, but
[21:11] this is like trying to get the world's more comfortable with automatically uh upgrading. This is the the magic behind the system. So, it's it's all patented and super duper cool. Um, then last,
[21:24] like Roland, who's another PM, asked me to put these up. I don't know the I think these are surveys, but I don't know. You can take surveys. You don't have to take surveys. I did my part, Roland, if you're listening to this recording. Cool. So, that's all I have.
[21:38] Um, questions, you can always, if you're on databicks Slack, you can slack me Jaydatabicks.com or Justin at Reddit or Justin Databicks on Reddit. Thank you. We did it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.