Migrating to Serverless: The Databricks Journey

Summary

Databricks' internal Data Platform team began migrating mission-critical ETL and BI workloads to serverless after repeated 2 a.m. quota failures and a pattern of whack-a-mole cluster sizing made manual capacity management unsustainable.
The migration used an automated pipeline to inventory eligible workloads, stop new classic job creation, and perform automated cutover with API calls, config snapshots, and owner notifications, beginning with SQL warehouses before moving to compute jobs.
Approximately 9% of migrated workloads experienced regressions — from blockers like memory limits, DBFS dependencies, and SQL edge cases — which were fed back to the Databricks engineering team as a real-world quality signal.

Migrating to Serverless: The Databricks Journey

The future of the data intelligence platform is serverless, and Databricks is leading by example. We aren’t just telling customers to migrate; the Data Platform team is actively moving our own mission-critical ETL and BI workloads to Databricks Serverless SQL and Compute first. this video chronicles our internal migration journey. We will share the raw “before and after” metrics on cost, latency and engineering overhead. We will be transparent about the challenges we faced as early adopters and the specific best practices we developed to run a massive-scale, serverless-first platform that powers the business.
Talk By: Alan Jackoway, Sr. Staff Software Engineer, Databricks ; Zack Loebel, Staff Software Engineer, Databricks ;

Chapters

FAQs

Why did Databricks decide to migrate to serverless compute?

Repeated 2 a.m. quota failures — where streaming jobs fell behind due to resource exhaustion and engineers had to manually resize instances — exposed how unsustainable manual capacity management had become. The team decided to invest in serverless rather than continuing the whack-a-mole approach to quota and cluster sizing.

What was the migration strategy for moving to Databricks Serverless?

The team built an automated pipeline that inventoried all workloads, assessed eligibility for serverless, stopped the creation of new classic jobs, and performed automated cutover with API calls, config snapshots, and owner notifications. SQL warehouses were migrated first, followed by compute jobs.

What blockers did Databricks encounter during the serverless migration?

Common blockers included workloads hitting serverless memory limits, dependencies on DBFS paths not supported in serverless, and certain SQL edge cases that behaved differently on serverless compute. These blockers, along with a 9% regression rate, were reported back to the engineering team as a quality signal for ongoing improvement.

What are the key benefits of migrating to Databricks Serverless?

Serverless compute automatically handles runtime and version upgrades, eliminating the need to manage quotas, instance types, and cluster configurations. The internal migration at Databricks showed average cost and runtime improvements, and agent-assisted migration tooling is now available to help customers replicate the same process.

Full transcript

[00:08] We are talking today about migrating to serverless and specifically data bicks's internal journey to migrate to serverless. We talk a little bit about gener generic migrations and things that you can do here. Uh but our focus today
[00:20] is our migration to serverless. But before we do that, let's talk about why, right? Why why migrate to serverless in the first place or embark on any migration? So this is what this this
[00:32] experience is what led us to really begin our journey in earnest. Hopefully you recognize yourself or maybe hopefully you don't recognize yourself in the slide yet. Um but maybe you do.
[00:44] You're woken up at 2 a.m. It says, "Oh man, one of my streaming jobs is behind. Let's wake up." You get out the laptop. It's some critical job. It looks like it's out of resources.
[00:56] There's been some spike in traffic or natural growth or whatever. Something has gone wrong. Your job's falling behind. You try to raise the size, but you're out of quota on this. Maybe I should use a new instance type that's more efficient. Do we have quota for
[01:09] that? Yeah. You go talk to someone. They say yes. You make the change. You wait for it to restart. You wait for it to start processing records and convince yourself that it's doing enough work and going to catch up so that you can go
[01:21] back to sleep. And then if you're unlucky or not paying enough attention because it's 2:45 in the morning, you forget to commit that to your Terraform or your other CI tool and a week later when you hit the same weekly spike, the
[01:33] next person gets woken up and does this exact same thing. Whoops. Right? This was happening to us a lot. And even if you're not hitting the specific quotota aspect of it, job sizing and the whack-a-ole that we were doing of making
[01:46] sure that jobs were growing at the right rate and using quota that was available to us had become a huge pain point for us. This is a real chart we were seeing of quota related job failures um coming
[01:58] into last summer and uh fall. We and we were dealing with it by that whack-a-ole like look at this job, it has failed. Oh, it's out of resources. it's falling behind. We need more resources. We have
[02:10] to ask the cloud provider, here's an urgent ticket. Please give us more. Right? This is a, you know, a scaling problem that we've run into a lot and that we see customers running into. And when you're running in classic data
[02:22] bricks, this is your problem, right? Like you are working with your cloud provider to make sure that they can allocate quota for you in the region of the type that you need. And we eventually decided like this is we are
[02:34] spending too much work on this. um we should just spend that work moving to serverless. And for a little while, you can kind of see as we get through after the like late October, early November
[02:46] spike, it starts to go down. We're feeling really good. We hit December and everyone's freaking out again. They say, "Are you sure this was a good idea? Why didn't you just ask for quota? Why are you doing this serverless thing?" But this is what happens afterwards, right?
[02:59] This is the right side. You can still see that December spike. Now we're down to almost zero issues with quota. And the reason for that is because we have offloaded something that used to be our team's problem to databicks. Now data
[03:12] bicks is negotiating with the cloud vendors, projecting capacity, making sure that there's enough available in the region of the right types. And as a special bonus, another thing that was very painful for us, which was adopting
[03:25] new, more efficient instance types. Every cloud provider releases these. Let's call it every 12 to 24 months. There's some new generation. it's 10% better, whatever. They'll make their claims, but to get there, you have to
[03:37] change all your jobs, right? You have to go through and make hundreds or thousands of job changes and make sure that they're all working. But data bicks is doing that in the background for you and letting you capture these values
[03:49] without doing the work. This was a case where we found that we could offload the problem to data bicks and solve it with serverless. Right? We have fixed a real problem that was plaguing our data teams by just moving that issue to the
[04:02] platform side. What else can we solve with serverless? DBR upgrades. Right? This is another thing that caused us a lot of pain. Your company probably has a policy that says you are not allowed to run out of
[04:15] support software. You have to run software that has support supported updates and security updates. Data bricks is going to keep releasing a new version every six months or so. historically is what we've done. Uh and
[04:28] that means that every six months or so, one of the old versions goes out of support, right? You have to upgrade those versions. That's your job on classic. But on serverless, it's not your job. It happens in the background.
[04:41] And that has allowed us on products like DBSQL. This is a slide from our blog last year was presented at data AI summit last year where DBSQL which never exposed this concept of DBR version to
[04:53] the customer was able to give 5x improvements with no work from the customer. Serverless jobs and interactive give you that same kind of benefit in the background on the other
[05:05] products. This is another case where you can solve it with serverless. No more updating hundreds of jobs to get to the newest versions. All right. Another one is the that was really hurting us is what I have termed here as the decision that everyone
[05:18] always makes wrong. This is again a real chart from our real production environment which was thousands of jobs running at uh on average less than 20%
[05:30] CPU utilization. Like this is what it looked like in our real data bricks deployment prior to the move to serverless. We are essentially overspending and sizing jobs for that 5
[05:42] to 10% case so that you don't get woken up at 2 in the morning like we showed in the cartoon at the beginning and then the rest of the time oops those machines are mostly idle. You're you're burning cash to avoid getting woken up.
[05:55] Serverless removes your ability to make that decision and in the process solves this. It runs at a much higher utilization than what we were seeing internally and what we're seeing in the customer metrics. All right, hopefully
[06:07] that is enough to convince you that it is worth at least trying to move to serverless. We are going to talk to you about how we have done it. But first, a quick little introduction about who we are and why you should listen to us,
[06:20] right? Uh our team um Zach and my team is a kind of unusual team within data bicks because we are running the data bicks platform for data bicks. We're kind of halfway between customer and
[06:33] vendor because we are operating the data bicks platform and running lots of jobs. These this adds up to one of the largest data bricks workspaces in the world and it's running missionritical business use
[06:47] cases for the whole company. You've got your product usage, your real time data, real time apps and lakebase and everything you could imagine. We are trying to build on this environment. So it's a real environment with real production uh expectations. It has high
[07:02] very tight SLAs's that have to be met even through a migration. It has streaming pipelines that have to stay up and running. And in simultaneously we're dealing we've been doing this for years. So we're dealing with years of
[07:15] accumulated debt. Like the thing about not running out of support DBR was not a joke. Like we were running really old out of support DBRs that even inside data bricks, right? So like we we know it is hard to keep up with this and we
[07:29] want to be able to offload it to serverless. Um so hopefully you believe that when we say this works, it really works. We we're migrating more than 10,000 jobs. We're trying to cut down on how many of them need manual review. We'll talk about that in a second
[07:42] because the more we can automate it, the faster we're going to be able to move. So that's who we are. That's why we're doing it. I'm going to let Zach talk to you about how we have done it. Thanks, Ellen. So let's talk a little bit about
[07:56] how this actual migration worked. So how did we think about starting? I'll I'll first talk we'll talk about jobs. That's really the elephant in the room. That will take the majority of the time. But start easy. There are things that you
[08:08] can do later today in this meeting. It won't take very long. Upgrade your SQL warehouse or change Spark declarative pipelines. Those are both opportunities where you just click a button and before
[08:20] you know it, you're running on serverless. You can do it literally during this meeting. Alan will show you how later. But something that takes a little bit longer, all-purpose clusters, right? These are where depending if you don't have the right policies in place,
[08:32] it could be a little bit more complex. You might need to work with some customers, maybe on the order of a week or two. And of course, the hard thing, the scheduled jobs. We'll talk about that quite a bit more. That will take a
[08:45] while for a number of reasons, but it's also where the most savings are. It's where all of the money probably is. So, let's walk through SQL warehouses. Here we have the super important BI
[08:57] dashboard. This might be the warehouse that Ali was using for his demo earlier on in the keynote, right? We want to make sure this thing is always running hot so there isn't a cold start problem. We have only turn off after 30 minutes
[09:10] of inactivity. We've got quite a few clusters. This is an expensive thing to run, right? You're paying for that always on cost because we want to make sure people have snappy experiences. But can we do better? Could we say solve it
[09:24] with serverless? Yeah, we can. All you do change that button pro down there to serverless. And now you actually have a lot of those benefits. You can see we've reduced the inactivity time because
[09:36] serverless starts up much faster. And so now you're not paying that always on idle cost. You also it the autoscaler works in such a way where those new clusters will as well come up faster. So
[09:48] you get the exact same experience but for less money. All you need to do change that button easy. Let's talk about allpurpose clusters. Get rid of them. This was a big pain for
[10:01] our team right similar to the warehouses. They take a really long time to start 5 to 10 minutes. You might have complicated libraries. Someone could be running like some notebook that's doing
[10:13] a huge query and all of a sudden the cluster can't be used anymore because this one notebook is using all of the resources as well. If you don't want to deal with that wait time then you have to pay that always on idle cost. Again
[10:27] you should just solve it with serverless. So we had cluster policies enabled so it was more straightforward to move to serverless for us. I'll talk about that in a moment. But with serverless, it
[10:40] starts in seconds. You automatically have isolated compute. So that runaway notebook is no longer your problem. It's the user of the notebook's issue and data bricks problem, which is the
[10:52] business we want to be in. Hopefully, you don't. And you only pay for active work. If the cluster is not on or in use, you're not paying for it. So, like I was saying, we had cluster policies
[11:04] set up. So, we could just change the policies such that people can't create those all-purpose clusters anymore. New clusters have to be on serverless. And we were able to make that change pretty quickly. It was the first time we did a migration. People actually said, "Wow,
[11:16] this is a better experience." They complimented us complimented us. That never happens in a migration. So, don't look back. Start using interactive uh clusters,
[11:28] serverless clusters, excuse me. Now, let's talk about jobs. Why are they hard? They're hard because a lot of times they're in Git. Not only that, you might have jobs in Git. You might have
[11:40] jobs just existing ad hoc. Um, and so that means they can be in either place, but it's also, like I said, where the money is. The m about half or more of our compute was running via jobs. And like Alan was saying, people size jobs
[11:53] for the incident that happened years ago. Here you can see there's a job that's running. It's a pretty big cluster, 150 nodes. It maybe peaks up to 50% utilization,
[12:05] right? All of that white space between the red and the black line, that's money you're leaving on the table, right? That's compute that you are paying for but not using. Again, this is something where you should just solve it with
[12:18] serverless. So, let's get into the weeds. And I want to talk a little bit first though about the philosophy. We are data bricks. So that means we wanted to show that we could get to 100%
[12:30] serverless. We wanted to run into problems for you, the customers do. That way we can feed it back to the product and get those issues fixed and resolved so that you don't have to. We wanted to
[12:42] show it's possible that all workloads can go to serverless. That may not be what your goals are, but it certainly is what ours are. So just remember that as we go through this. The other thing we
[12:54] didn't want to change code, we only wanted to change config. So that means we don't necessarily want to change how your job works. We just want to change the compute it runs on. We didn't want to make changes to your job, have it
[13:07] fail, and then suddenly become the owner because we touched it last. We only wanted to say your job was classic before, now it's on serverless. And finally, we wanted to make changes
[13:20] centrally such that we would reach out, we would make a change to the job on behalf of users, and you would have to opt out if you didn't want to participate. Of course, we'd come back for you and change it later. But we we were worried it would take way too long
[13:33] if we let people come to us and make the change at their own pace. So, we wanted to proactively go and change for everyone. So, now let's get into it. So first off, make it hard to use classic,
[13:46] right? We want to make it so any new job that's created is running on serverless by default. First thing you do, change the default. And at that point, start to make it so that there's limitations on who can actually create or run classic
[13:59] jobs. Once you do that, start to make them request an exception if they want to run on classic. Because if you never do those things, people will keep making classic jobs and you'll keep chasing them down. You have to stop that inflow
[14:13] before you start your migration otherwise you'll just keep going forever. So the core principle that we did was this sort of flywheel. So we have it's all built on system tables that you get
[14:26] for free building data and job metadata. So we start off we build an inventory right what are all of the jobs that are out there and that are potentially eligible. Once we have an idea about
[14:39] what's in the inventory, we make a selection. Which jobs are we going to upgrade that day? Okay, we've picked the jobs that day, the candidates. Let's notify the owners. Tell them, hey,
[14:51] we're going to upgrade your job. Once they've been notified, that PR has potentially been merged or we we cut the PR. And once that's done, we wait and see. We keep an eye on the job to see
[15:03] whether it succeeds or fails. After that, we say, "Hey, this migration for this job is complete, or we had to roll it back because there's a problem." And then the final step that makes this a flywheel, we reinccorporate those
[15:16] learnings. And that's what makes this go faster and faster and get better and better. As we run into new novel failures, we can use that to identify in those other candidates to say, "Hey, wait a second. We need to make a
[15:28] platform change or we need to involve the product team and we shouldn't migrate some of these jobs." It allows us to improve this process as it goes along so that it can get smarter and better. So let's go into each of these
[15:41] pieces in detail. So first off, we start with our inventory. Like I said, this is all built on data bricks primitives. We make our inventory snapshot. This is a daily data bricks job. It consumes system
[15:54] tables, stuff like flow jobs, lakeflow job run timeline, and building data to give us an idea of the opportunity cost. We also have a special blocker table. If you want access to it, well, your SA or
[16:06] your account team should be able to give you a list. That's the only specialized thing here that we were able to use. So, we join that together. We look to see and of course, like, have we tried to migrate that? And we create a snapshot
[16:18] each day that shows us, hey, what is our total addressable market that we can use? How many jobs can we migrate? So, now we have our snapshot. We filter that down to specifically what is
[16:30] eligible. Have we tried to migrate it before? Did this user opt out? Is it potentially a migration we're waiting to be completed? And of course, is it healthy? This is a really important point. You don't want to migrate a
[16:43] failing job. If the job's failing before, it'll probably fail on serverless. So, we make a point to only look for jobs that have succeeded the last n runs. I think we did something like five. And in that case, we say,
[16:57] "Okay, this job is eligible. Let's migrate it." From there, we write to like the spine of our framework, the action log. This is where we record all of the migrations we're planning and the results. But all of this is a delta
[17:10] table with views on top of it that say, what do we plan to do or what have we done? So, we have our cohort selected. We've written to the action log. Let's actually look into kicking off a
[17:22] migration. So, we consume a certain number from the action log. We have our cohort selected. Sorry. And either it's a repo manage job or it's something that just exists in the
[17:34] workspace. And so we split based on that. If you live in a repo, that's where we use an agent to go make that config change, cut a PR, write that back to say, "Hey, this upgrade has been initiated and wait and see what
[17:46] happens." If it lives in the workspace, that's much easier. It's just an API call. You make that API call. And of course, you want to make sure store the config snapshot. We want to know what it was running before because if there's a
[17:58] problem, it's not in git. We can't just roll back. So, we need that state from before. Again, the goal here, we don't want to change the code. We just want to change the configuration.
[18:10] So, great, upgrades been initiated. We write it back to the action log. At that point, we wait. We don't necessarily want to try to wait for the PR to be merged or anything. We just say, "Hey, this job has been initiated. Let's pay
[18:23] attention every 15 minutes. Let's do a scan of the jobs we're waiting and let's see what has happened. Right? So we can query what have we attempted to migrate and we can use again system tables to
[18:35] say was it a success? Was it a failure? And what type of compute did it run? Did it actually run on serverless? If it did, did it succeed? In which case, congratulations. That job's migrated. If
[18:48] it failed, that's where we need to write back to our action log and initiate a roll back. And that's okay. Roll back is expected, right? We've captured the compute before
[19:00] so that if it's a if it's in the workspace and not in git, we just write back to the API and we notify the owner to say, "Hey, we tried to migrate your job. It failed. If it lives in a PR or in a repo, we just cut a revert PR
[19:14] and notify the owner, let them merge it." But we also want to incorporate that information about why did it fail? Why didn't weren't we able to identify that? and we feed that into our blocker taxonomy so that we can like I said
[19:26] about the flywheel improve we know hey there's a problem in maybe this framework we should incorporate that and look for those jobs on subsequent runs so once you do that you need to identify
[19:40] who's your first batch you're going to start with us we have our internal metrics platform there's a talk about this tomorrow Alan will speak to it later um but it's called metric store
[19:52] but But it's a platform and that's what's important here. It's a platform that our team owns and manages and that means we understand it really well. We have really good testing infrastructure. It helps that we own it and it's
[20:04] controlled controlled centrally which means we can just go and make the change. It's very familiar and it's a way for us to sort of prototype this out. So find your metric platform. It also has well understood SLAs's. We know
[20:18] if a job's important or if it's just dev or test. And so we can sort of work our way through that. But it's something owned by us. And if it has all these things, it makes a really good candidate because it means you can sort of walk before you can run. And so it's those
[20:32] SLAs's that we use to really start with the basics. Let's just start with our dev or our testing jobs, the lowest SLA jobs, so that we can kind of test out this process and start that flywheel.
[20:44] Right? So we start with the low SLA jobs and we work our way up. We wait to see if it runs successfully and if it does, move your way up to higher tiers. Uh, and as you do that, rinse and repeat. Find the next cohort and work your way
[20:57] through that platform until you're ready for the next platform. And so, like I said, there's sort of this flywheel here. As we keep working, we are able to improve our inventory. We
[21:09] know we can better identify which projects work a certain way. We can feed that into our automation to improve prompts to say, "Hey, this project should get migrated this way or that way." And as we hit blockers, we either
[21:21] add them to rules to say, "Oh, make sure we roll back these jobs." Or we can also feed it to the product to say, "Hey, there's a problem here. This is something you should consider fixing, right? We can run into issues." But this
[21:33] kind of continues and feeds on itself. And so I want to talk a minute about some of those blockers that we saw and problems that we ran into. Again, your mileage may vary. We are all on Unity catalog, but one of the things that we
[21:47] ran into first was Anie SQL. We were doing some of those DBR upgrades that Alan was talking about earlier, and by implicitly by doing that, it turns out an becomes required. We were worried we're going to have to rewrite all of
[21:59] these SQL queries, but it turns out there's a setting. It wasn't a problem. You can actually turn off an SQL if you need to. It wasn't a real blocker for us. As well, UDF's map and map partitions.
[22:11] This is one where you can't use more than a gigabyte of memory in any of these sort of lambda type functions on serverless. This turned into a big issue for us. There are quite a few jobs that are blocked because of it. And we took
[22:24] it to the product and we'll talk a little bit more about that in the results as well. You can't use RDD APIs. We thought this was going to be an issue. Turns out they aren't really used anywhere. So that wasn't a problem whatsoever.
[22:36] Streaming limitations is another blocker for serverless. This is sort of a big one for us. We have a lot of streaming workloads and those teams are going to have to own those changes. They're going to have to go and rewrite some of their frameworks to do it, but they've
[22:49] committed to it because they also want to get away from that quota pain. So, this these were problems for us that we ran into. Like I said, we don't really use DBFS. We're all on Unity catalog. So, your mileage may vary and you may
[23:02] have other issues. Now, I'll hand it back to Alan for the results. Okay. Hopefully, you know, I don't know how many of you are are looking forward to this part where we tell you did this
[23:14] actually work, but let's talk about what worked and what didn't. One more quick little thing about the metric store metric platform that we were talking about before. This was the first cohort we targeted and it's used by every uh employee or it's available to every
[23:27] employee at data bicks to try to make it easier for every employee at data bicks. We had given this sense of um sizing. We didn't expose the full like cluster UI
[23:39] where there's a bunch of different things and min and max sizes and DBR version and instance type and all that. We just said small, medium, large, extra large. And essentially everyone in the company said, "Oh yeah, I work with lots of data. Extra large." They all chose
[23:52] extra large. And the result was that we had a lot of clusters that looked like this. This gigantic cluster running at it says 5 to 10% utilization. I think that's less than 5 to 10% utilization.
[24:04] So we had a sense that um because for one reason or another a lot of our users had oversized their jobs and had felt safe to do so or were you know not sufficiently incentivized not to do so.
[24:16] If you want to you know go talk see our cost governance talk recording or maybe they got another session um you can learn how to incentivize them better not to do this. Uh but for one reason or another we had a lot of these we had a good sense that at least this metric
[24:29] platform would have strong results. That said, the results themselves were impressive even to us. We saw on average an 80% reduction in DBU, which is to say an 80% reduction in cost, and paired it
[24:42] with a average runtime reduction. So, it's not just reducing 80% by cutting 80% off of that chart where you saw like 80% of the nodes weren't used. It's actually taking advantage of
[24:55] improvements that have been made in the serverless runtime, newer DBRs, newer instance types, all of this stuff to run faster to get the job done. Um, this was our this was for our largest single metric store migration batch. It was
[25:07] more more than a hundred jobs. And the downside that we ran into is that this is the batch where we really slammed into an SQL. This is the batch where I mean you can see nearly one in five jobs that we migrated here failed almost
[25:21] exclusively because of any SQL that wound up being very easy to fix once we learned that it was a blocker in retrospect. Might have been nice to learn that on a smaller batch of jobs.
[25:33] Um but it was still relatively easy to exercise the roll back then make the config change to say okay look these run in non-ansancy mode and then run them again through the next batch. So that's that's what we saw on this big batch.
[25:45] This is by far the single batch that looked best. Let's talk a little more generically about how we think about any migration. This is actually applicable beyond metric store or beyond serverless. This is applicable to
[25:57] anything. But this is my like abstract framework in you start kind of in the middle of the chart. And in the perfect world, my dream job that I never get to
[26:09] experience would run in 0 seconds and cost zero dollars and get the work done. Wouldn't that be nice? Right? You're all the way down into the left, all the way into the bottom left corner of quadrant A. It's just done for you. You're pretty
[26:23] much never going to hit that because your jobs are doing real work. That does take some amount of time and cost some amount of money. But in general, when you want to when you make a change, you want to see the job moving down and left on this chart. You don't want to see it
[26:35] moving into the red, right? Because that means that somehow you've made a change and your job got slower and cost you more. That's no good, right? No one is happy with that. So what does it look like on our actual results? We're now
[26:49] expanding beyond that first hundred or that large batch of 100 jobs. I think this is more than 10,000 unique job runs, over a thousand distinct jobs, and they're weighted by DBU, weighted by
[27:02] cost. Essentially, what we found is that nearly 3/4 of our jobs had moved down into the left some amount the way that we were hoping, right? So, we saw that a lot of our jobs were be able to simultaneously
[27:15] speed up and get cheaper by moving to serverless. And again, there there's a lot of reasons behind this. There is variability. you can see in the chart. Um, but it comes from the fact that the platform keeps getting better and a lot
[27:27] of our jobs weren't getting upgraded because people don't upgrade stuff, right? Whereas serverless is doing these upgrades for them in the background. Our second largest category was um this
[27:39] slower cheaper side. So that 20% there, the jobs got slower but they got cheaper. This happened to us on serverless because we primarily move jobs to standard mode. Serverless has a
[27:51] standard and a performance mode. Usually use performance for something more like interactive, something where there's a human waiting on it, maybe your most critical SLA dependent jobs. We said by default, we're going to start on
[28:03] standard, save a little money, and essentially considered that bucket to be also a success. These jobs didn't have such tight SLAs's that going a little slower was a problem for us. But it is something you have to at least watch out
[28:15] for, right? because you may make a different decision and instead you may change to that other serverless mode and try to take that 2% right. The reason we're so low on uh more expensive but
[28:27] faster is that we intentionally moved to the version of serverless that's trying to run a little bit cheaper and a little bit slower. Um you might make a different decision depending on exactly the characteristics of your jobs. Okay,
[28:39] but we do have the red zone. Almost 10% of our jobs or job runs in our real data set, production data set got worse on both metrics. We promised at the
[28:52] beginning or at least in the description that we would give you the real like real life as a customer experience here. 10% is not great. 9% is not great. You
[29:04] in the room, a normal customer of datab bricks, would probably just roll these back to classic. You'd probably look at it and say, "Serverless didn't work, right? And I'm going back to classic." I
[29:17] would encourage you to think of that as uh serless didn't work yet. It does keep getting better over time. Uh but it's very reasonable to see a result like that and move back on those workloads.
[29:29] However, that's not what we did. So what we did was we essentially put on our we're a data bicks customer hat, not a data bricks employee hat, and went to our account team, which is to say the serverless engineering team, and said,
[29:42] "Hey, 10% of our jobs got worse. You promised us this would be better, but one in 10 of the time it's getting worse. What's going on? This is an unacceptable level." and and we're
[29:55] working with them to do job reproductions and try to dig into what's going on with the idea that when you in the room do the same migration, instead of that number being 9%, we want it to be 0.9% or 0.09%. Right? You're you're
[30:10] realistically no migration's ever going to hit 100% better. Like it's just not how any migration works. But we are trying to drive that number in the right direction using our internal workloads
[30:22] to make it easier and more successful for people in the room. That's one of the ways that our migration internally at data bricks should be benefiting you. Right? We're make taking that 9% of jobs
[30:34] and using them as test cases to make sure that the platform keeps improving. Some of these blockers like the Python UDF blocker was something the product knew about, right? like it was it was a known blocker, but it was very hard for
[30:46] us to estimate. It was very hard for us to look at a job or look at the code and analyze the code and say, "Yeah, that UDF is going to use more than a gigabyte of RAM, but that one's not." It turned out to just be very hard to analyze and
[30:58] hit at a much higher rate than the product people had estimated when they put it on the blockers. So, we sort of raised it to them and said, "Look, this is really a problem and it's going to be a problem for customers because it's
[31:10] very hard to predict and somewhat hard to deal with." like we've been some of them you have to rewrite UDFs or make tweaks to various things. Um so the product is now like fully committed to fixing this problem and increasing that
[31:22] limit so that a higher percentage of jobs that do rely on UDFs which we know are very common in the real world um so that a higher percentage of those jobs will be successful and similarly we're putting pressure on them for all the
[31:35] other blockers that we're running into again with the goal of making it easier for you. So hopefully we've convinced you it's worth doing this. Hopefully we've convinced you that the results, at least on average, are going to benefit
[31:47] you and that it's possible to do it and that you can get it moving. How do you get it moving? Well, if you haven't started, you could, if the Wi-Fi is vaguely working, like you could right now go change SQL to serverless. It is
[32:01] literally a button click, maybe a few buttons if you want to mess with the auto stop and all the other things so you can capture a little more value from the change. But you can do it in one click in the UI. Same with uh Spark
[32:13] declarative pipelines. If you're a little further along, you've done the really easy stuff, really encourage you like moving people towards serverless interactive, getting rid of those standing team clusters you may have or
[32:25] maybe you have a shared cluster for your company depending on the size. We had a lot of trouble operating those as we scaled because there's just too much noisy neighbor and like contention going on. And as Zach was saying, really
[32:37] everyone has had a pretty good experience since we have pushed those use cases into serverless interactive. Uh the other thing you can do is you can ask Genieode to migrate something for you. Um this QR code that's up on the
[32:50] slide here is something that was recently released as a like lab's best effort. Please don't come complaining too much if this goes wrong as with everything with the agents. But it's a set of skills that was released by data
[33:02] bicks out on GitHub that you can use with uh Genieode or any other agent, but you can use it with Genieode and then it's smart enough to actually make job changes for you. It'll analyze the job and look for known blockers and and help
[33:15] you make sure that the thing's going to work. So that's up there. Um if you are further along, maybe you're mostly migrated. Firstly, thank you for coming to the talk. I don't know why you came to the talk about how to migrate if you're mostly migrated, but maybe you're
[33:28] scoping it out for your own talk next year. Uh, but what we want from you is to help make it easier from everyone in the room, right? Raise the problems that you did have to your account team, get it prioritized so that the blockers that
[33:41] hit you as you go through a migration to serverless or really anything else in the platform, breezing those to your account team and back to data bricks is what makes it better for everyone in the room. it. So, it allows us to make sure
[33:54] that the product keeps improving based on real use cases. All right. To do any of this, you are going to need a team. You've got Zach and me talking to you uh with the magic of automation and agents and all of this
[34:07] stuff. You can do these centralized migrations with relatively small teams. We moved thousands of jobs with basically four people and essentially only one of them. As with most software things, all the work's done by our
[34:20] junior engineer who's not on the stage talking to you, right? So Henry is the one who did, I would say, most of the work. And then you have, you know, you have a manager, Krishna, who's in the audience. You've got Zach and me helping guide the ship. But it's, you know, it
[34:33] took us four people to migrate thousands of jobs, many of them not owned by our team, right? we run as an internal platform and we just ran the communication and the flywheel that Zach was was talking to you about in a way
[34:46] that allowed us to make those changes, convince the users that they're safe, let them opt out if they're not or if they're not comfortable with what we're doing and make like pretty broadscale changes across our company without
[34:59] having to go through like months and years of chasing people around and saying, "Hey, can you please upgrade this thing? Hey, we're blocking you after this date." all the pain that you've maybe gone through on federated migrations. All right, that's that's the team.
[35:12] That's what we've done. If you want to hear more from our team, we have uh many talks. Some of them are already done. These are still out in the future. So, these are all from the same data
[35:24] platform organization. You can learn more about how data bricks is using datab bricks. They're specialized on other topics. I don't think any of these are migration specifically, but if the topics are interesting to you, we'd love to have you. Uh, thank you for coming
[35:38] and

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.