Serverless Compute at Petabyte Scale: Deloitte's Global Migration From Classic to 70% Cost Savings

Summary

Databricks Serverless Compute has matured to support petabyte-scale, mission-critical workloads, achieving 89% fewer outages, eliminating library download delays through environment caching, and completing 25 Databricks Runtime upgrades at 99.998% success rate through a versionless architecture.
Deloitte Global migrated its data platform serving 60,000-plus auditors with more than 20,000 daily jobs across Americas, EMEA, and APAC from classic compute to serverless, achieving 70% cost savings and 50% performance gains while eliminating idle billing and shared driver support issues.
A partnership with Databricks introduced workspace-level base environments that reduced a 150,000-notebook runtime upgrade challenge from 12 hours of manual work to three API calls, demonstrated as a concrete example of the enterprise serverless migration playbook.

Serverless Compute at Petabyte Scale: Deloitte's Global Migration From Classic to 70% Cost Savings

Serverless Compute has matured from convenience feature to production workhorse. this video covers recent launches across runtime, performance, and cost for Notebooks, Jobs, and Pipelines, plus upcoming innovations like cost-optimized execution with fallback. More importantly, Databricks' largest enterprise customers now run petabyte-scale, mission-critical workloads on serverless with dramatically improved economics and simplicity.
Deloitte Global migrated its data platform serving 60,000+ auditors with 20,000+ daily jobs across Americas, EMEA, and APAC from classic compute to serverless, achieving 70% cost savings on interactive and SQL workloads while eliminating idle billing, shared driver issues, and manual cluster tuning. Key migration steps included infrastructure setup for private endpoints, notebook transformations to handle V1 to V4 runtime changes, and endurance testing against peak production loads. The partnership introduced workspace-level base environments, turning Deloitte's 150,000-notebook upgrade challenge from 12 hours to 3 API calls.
🤝

Chapters

FAQs

What are the main advantages of Databricks Serverless Compute over classic cluster compute?

Databricks Serverless Compute eliminates four classic cluster challenges described in this video: manual networking and IP range setup, security configuration overhead, resource inefficiency from idle billing, and Databricks Runtime version management across teams. The serverless architecture auto-optimizes compute allocation, charges only for actual execution time, and uses a versionless approach that eliminates manual cluster upgrade cycles.

How did Deloitte migrate its workloads from classic Databricks compute to serverless?

Deloitte's migration proceeded in two phases: first, infrastructure setup including private endpoint configuration and environment variable migration, then notebook transformations to address API constraint changes between the classic V1 and serverless V4 runtimes. The team validated the migration by replaying peak production loads in endurance testing before moving more than 20,000 daily jobs to serverless.

What cost savings did Deloitte achieve by moving to Databricks Serverless?

Deloitte achieved 70% cost savings on interactive and SQL workloads after migrating to Databricks Serverless, primarily by eliminating idle billing on the 25-plus managed clusters that previously ran continuously across three global regions. Additional savings came from eliminating shared driver support costs and the operational overhead of manual cluster management.

What are workspace-level base environments in Databricks and how did they help Deloitte?

Workspace-level base environments allow a Databricks administrator to define default Python library environments for all notebooks in a workspace using API calls, without modifying each notebook individually. Deloitte needed this capability because upgrading 150,000 notebooks to new runtime versions would have taken approximately 12 hours manually, but workspace base environments reduced that work to three API calls.

Full transcript

[00:07] I'm Roland. I'm a product manager at Databricks and about 4 years ago, I wrote the specifications for serverless compute for jobs, notebooks, and pipelines. And so I'm super excited to talk about that today, of course. And um we'll talk a little bit about uh
[00:22] why serverless, how it works under the hood. I'll give you a little bit of a glimpse of that. And of course, we'll talk about the road map. And then I'm also super excited that I'm here with Santosh who has migrated a lot of workloads over to serverless compute and he's going to talk about how to do that. Uh his experience and um also the KPIs.
[00:40] What was the outcomes? So before we dive right into it, um who of you has used uh serverless SQL, serverless warehouses? Okay. That's like vast majority. How about serverless jobs? Who has used serverless jobs? Okay. Okay.
[00:56] So I don't need to talk too much about the how I suspect, right? Like plenty of you have been using it already. But let's talk a little bit about, you know, why serverless at all and what problems are we solving with it. All right. So first of all, you know, you typically have lots of different
[01:12] types of teams. Um data engineers, ML engineers, analysts. All of these users, they want to get all of the data that they need. And this is structured data, unstructured data. It could be in any form.
[01:28] And so you need to provide them that data in a reliable and performant way. And you need to govern that. Only the data that they should have access to, they they should actually access. And so you need Unity Catalog, you need security layer. And then in order to serve that diverse
[01:46] set of uh data and user groups, um you need to also um have a compute fabric that can be dynamically reacting to whatever the need of that specific user group is or whatever that specific data is.
[02:01] So, you can of course do that by focusing, okay, what do those ML engineers need or do those agents need now that we're in agentic world? What do they need and how can we cater best to that? But doing that at scale across all of those groups is really hard. So, how do you scale that?
[02:16] So, we find when central data engineering teams or also just generally like users try to solve this, they spend a lot of time specifically on four problems. So, the first problem is setting up networking. And this includes things like IP ranges.
[02:32] This includes often things like talking to other teams that are not even your team. It might be a security team that needs to open up a certain networking and and certain setups. And it's also security and compliance. You know, you need to make sure that there's no exfiltration risk. You need
[02:47] to make sure that you're on the latest security patches. And then you need to also manage efficiency. This includes not making the clusters too large in classic, so you're not wasting resources, but also not too small because otherwise, you know, you might get paged in the middle of the night because one
[03:03] of the one of the clusters ran into an OOM because your data volume increased. And it includes things like compaction and other optimizations that you have to manually do all the time. And it's also DBR upgrades. Um I've heard that it's up to 20% of the
[03:18] time that it takes to upgrade to the next LTS version uh for these central data engineering teams. So, it's like a lot of effort that goes into this. At the same time, if you don't do it, it means you're losing out on all of the performance benefits uh and efficiency increases that come with the latest DBRs.
[03:36] And so, these problems is what Serverless is meant to solve. Serverless is auto-optimizing. So, what does that mean? It's fully managed. Not only the infrastructure, so that means we pick the instance types, we pick the memory sizes for you to
[03:53] optimize them for your workloads. And um you instead of tweaking these infrastructure knobs, you tell us, "Do I want this fast or do I want this efficient, right?" You give us the business goals that you have for this specific workload, and then we tweak the infrastructure knobs
[04:09] automatically towards that business goal. And we do that not only once, we do that continuously for every run, we learn. It's also performant because we have a warm pool of machines, and we'll see that uh how that works in a bit more detail later. And so we can deploy VMs
[04:24] very quickly uh and also cost-efficiently. And uh we have environment caching, so that means you no longer have to download and install for every run the libraries that you need for that uh job to complete. It's also versionless. So it's not only fully managed when it comes to the infrastructure, but it's
[04:40] also fully managed when it comes to Spark and maintaining Spark on the latest DBR versions. And we'll talk a little bit about the architecture that makes that possible. All right. And all of this now powers um you know, the serverless compute uh layer powers SQL that you're all
[04:57] familiar with. Many of you are also using um serverless jobs and also notebooks and pipelines. All right. We talked a little bit about that trade-off between, you know, performance and efficiency. So how does that play out in the actual product?
[05:12] So what you see here is um at the bottom there's this performance optimized toggle that you can switch over. And that's on the right side panel of your Lake Flow job. When you switch that on, you're in performance optimized mode. And what we'll do automatically for you is we
[05:29] will tune the auto scaler that we've completely developed from scratch for serverless, we'll tune that to be more aggressive, so give you machines faster. We will get those machines from the warm pool that we have that is set up per region. And so your workload runs much faster and starts up much faster. So on average
[05:45] we see twice as fast execution uh that we typically see on classic. If you disable this toggle, then you're in standard performance mode. And so, that means we're more conservative. It takes 4 to 6 minutes to start up the machines, which is kind of equivalent to classic with the difference you're not
[06:02] actually paying for those 4 to 6 minutes. And it's up to 70% cheaper. The auto scaler is less aggressive and we don't need the warm pool for it. And so, this is what we mean with you picking the business goal, you know, fast or efficient, and then we automatically pick the infrastructure
[06:18] for you that and, you know, the the tuning that meets your goal. All right. But, let's do a few more comparisons between the classic world and the serverless world. In classic, you had to manually pick the infrastructure, of course. Gave you a lot of freedom,
[06:34] but it also meant you had to think about out of memory issues when your data volumes increase and then suddenly, you know, the cluster goes down because your driver ran out of memory. And you also had to tune the performance and efficiency trade-offs through many, many knobs. And that was, you know, often led to,
[06:50] you know, getting paged in the middle of the night, led to stability issues, led to, you know, SLAs being violated. On serverless compute, we see 89% fewer outages compared to classic. And that is because we shield um you in serverless through a variety of
[07:06] mechanisms. For example, we automatically fail over to other instance types. We have a warm pool of machines that's ready to go and so, that is a buffer even if when there's no more VMs from the cloud provider. And there's a variety of other mechanisms. So, we can now say that serverless compute is
[07:22] actually our most stable compute product. In classic, uh we already said like there's 4 to 6 minutes startup time and you actually pay for that. Um and when it's 4 to 6 minutes, it also means it takes a while when you have a scaling event later during the execution to get
[07:38] new VMs, get more VMs. And because it takes that while, you'd rather hold on to those VMs for a little bit longer because in case you might need them right away again. In serverless, that's not a problem because we can react much more elastically to the demand of your workload.
[07:54] And so, in serverless, we see 5 to 10 seconds startup time for performance optimized and interactive. And we see typically twice as fast execution. The other thing that contributes to performance here is that in classic, you
[08:10] have to download the libraries for every run of your job. And then you have to install them. This can be a significant fraction of your actual execution time, and you're paying for that. And this is specifically, you know, can be specifically large when it comes to ML workloads where these libraries are,
[08:26] you know, CUDA can be like really large. So, in serverless, you download them once, then we install them once, and then we take a snapshot before we execute your workload. And that snapshot is what gets loaded for subsequent runs. And that loads in seconds. We're basically caching that
[08:43] environment for the future runs. So, no more downloads, no more installation waiting for the subsequent runs. And then, of course, there's versionless. We talked about, you know, it's really important to keep up with the latest DBR releases
[09:00] to take advantage of all the performance and all of the efficiency gains that come with them. And but it's very hard to do this, right? And and and we know there's obviously like breaking changes between major versions and things like that. In serverless, that becomes much easier.
[09:17] And in fact, we've done 25 DBR upgrades over 6.7 billion workloads with a 99 point, I always need to look how many nines it is, 998 percent success rate. And so, that means even for the 0.002%,
[09:32] it's not like your workload's failing, it means that we actually detect something's wrong. We pin your workload on the previous version, and then automatically we send out tickets to the engineering teams to fix the thing before we upgrade you again. So, it's really successful. And we see
[09:48] around 50% performance improvement year-over-year here. All right. So, just to summarize a few key reasons why you should choose serverless, because it's fully managed, we manage the infrastructure, you choose the business goal and then we manage the
[10:03] infrastructure optimizing towards your goal. It's performant um because we've got environment caching, we've got warm pools of machines that are always ready to go for you. And it's versionless, so we also manage the Spark experience fully for you. Yeah. And you've all used Spark, right? Spark's amazing. Um but uh
[10:21] it's hard to manage Spark and the infrastructure. We live in the in the age of AI and agents, right? So, let's have AI do that for us. That's the whole idea with serverless compute. All right. But, let me give you a little bit of a glimpse of how this actually works under the hood. So, what you see here on the one side is
[10:38] you've got all these users, they are on different workspaces, and on the other side you have a pool of machines that are ready to go. We maintain that pool, and we do that on uh on the for the whole uh region. So, across all customers in that region.
[10:55] And then once you submit a workload, we allocate these VMs to form the cluster and execute your queries accessing the data. This sounds an awful lot like instance pools. And it kind of is, but we
[11:11] actually did a bunch of performance tuning to actually start this even faster up than instance pools. Now you're wondering like, "But how can this be efficient? Like instance pools are so expensive." And the way we do that is uh well, we first of all do this across the whole region, so across all customers. A lot
[11:27] of the individual fluctuations average out. And then once these machines, of course, they are assigned to you, then they're only assigned to you, so there's full isolation. But we have a few more tricks up our sleeve. Most of these workloads, we actually know what the last run
[11:44] looked li- looked like, and we're actually scheduling the next run. So, we not only know when they're supposed to run, but also what's the profile going to be. There's typically relatively small fluctuations workload per workload. And so, that means we can very efficiently learn, predict, and pre-provision
[11:59] that warm pool of machines. And so, that makes it very cost-efficient. Let's talk a little bit about how versionless work, because the architecture here is actually quite different from classic. In classic, you just had a cluster with
[12:15] a driver and executors. And there was a very tight coupling between what is on that driver, like the REPL, the the versions that you installed, and the code that was running, your your code that was running on there, and like, let's say, the Python version and things like that.
[12:30] And if you changed one thing, let's say, for example, the DBR, you had to adjust all of the other things. It was like really monolithic. Think like, you know, Windows 95 or something like that. And what we've done in serverless compute is we've actually taken the REPL out of the driver in its own VM here.
[12:46] And so, that's running all of your user-defined code. You have full control over that, which libraries get installed there, and you have full control, like, let's say, over the Python version, the library versions, and your code here. And then we have a standardized uh client-server architecture here through Spark Connect,
[13:03] which talks to this serverless Spark cluster. And because this is standardized, we can swap out effectively the DBR here without affecting any of your code. And this is how versionless is possible. We can upgrade to the next version here,
[13:19] giving you all of the performance, all of the efficiency benefits, without you actually having to change the code, because that interface is standardized. Right? And I hear of course, well, but there are still client versions. Yes, there's client versions. And there's three year support for the client versions. This includes things
[13:36] like at some point you might want to upgrade Python or like some of the libraries, right? And then you have full control over it when you want to do that. But even if you don't do anything over that three year period, you're still getting throughout those three years all of the performance benefits from the latest DBRs on Spark.
[13:53] And so that's the huge advantage here of of of version lesson how we do it. All right. And we've got plenty of customers like, you know, Airbus, but also Compass highlighting the benefits of simplicity, performance, and efficiency here.
[14:09] But, you know, uh let's talk a little bit about the road map, because I think you're all excited about what's to come next. There's plenty of items on here. I don't have time to go into all of them, but, you know, private network gateway is already in preview. It's coming.
[14:24] Uh I'll talk a little bit more about DF Cache. I think you're probably super excited about entitlements and rate limits means you can select who gets access to serverless and uh how much are they allowed to spend. We have uh migration agents. There's talks on on cost control and migrations
[14:40] tomorrow specifically. Uh and uh we have more observability with log delivery. There's environment variables, and there's more features coming for debugging as well. I'll dive into a few of these. Select few. One of them is cost optimized with fallback.
[14:56] And so, we talked about this mode where we optimize for performance. We talked about the standard mode. And then, what we gotten a lot of feedback on is, "Hey, there's this class of workload where standard mode is still too fast.
[15:11] We actually want this cheaper, and we have more time to run this. Like, this is the typical overnight large ETL workloads where they're like, 'Yeah, you know, if I spend a bit of time, you know, maybe running it on spot, maybe I can make this still a little bit cheaper, but I don't need it that fast. So, we said, "Hey, serverless is so big
[15:27] now. We've got plenty of capacity. If you give us more freedom in terms of time when to run this, or or we can stretch it out in time a little bit more, then we can actually use some spare capacity, because we know the demand curve really well, and we can predict this well." So, the idea here is you pick a goal of
[15:44] like, "Hey, this needs to run within like, say, 4, 8, 24 hours." We're still defining what those limits are, so happy to talk to you uh about that. Um and then if that workload comfortably, with, you know, enough buffer for us to optimize this, fits in there, we can provide this 20 to 30%
[16:01] cheaper for you. So, this is effectively a third execution mode, and the third performance target. We're aiming preview in Q3. And if that's something that's interest to you, here's a QR code that not only contains links to all of the other uh serverless relevant talks and a few publications,
[16:16] but you can also sign up for this preview and a bunch of other previews. All right. Uh migration tooling. There's another awesome talk about this. We have been working a lot uh on agents to help you move from classic to
[16:31] serverless, because, you know, there's a lot of people who are asking like, "How do I bring my workload over?" So, the first thing that we've done is we open sourced a migration skill. That's on the Databricks GitHub. It's also linked in the same QR code that I showed earlier. And you can try this out already. Uh I've received a lot of positive feedback so far, but I'm
[16:46] looking forward to like getting more challenges on how to improve that skill. But more importantly, we're integrating that fully into the Databricks UI, um which means you can go in there and say, "Hey, here's like 50 jobs. I want to migrate them all." Then Genie is going to go in there and
[17:04] um scan them all for compatibility, flag things to you that it can't migrate. And for those that it can migrate, it's going to adjust the configurations, as well as update the code for you. And then you can run it in parallel. See, hey, does it work? Does it not work? You can always roll back.
[17:20] So that's a feature that I'm super excited about. There's going to be an amazing demo uh by Prashant who's sitting back there. Um and so you should check that out as well. There's a link to that presentation as well in the QR code. And then of course we've been working a lot on closing compatibility gaps. Since
[17:35] the architecture that I've shown you is quite different from the classic architecture, that means some of the features did not work out of the box initially, right? And so we've been working really hard to support a lot more features in serverless compute, including things like DF cache that I'll talk about. Uh we will also have standard um
[17:53] standard performance mode for run submit. So if you're using Airflow or ADF, you can actually uh soon use the standard performance mode. Um then there's support for environment variables coming and we're also exporting uh the logs.
[18:09] Let's talk a little bit about DF cache. DF cache initially wasn't supported on serverless compute. Now we're in private preview and you can sign up to the preview again with this QR code. And I think the implementation that we've done for serverless compute is actually way better than the classic implementation
[18:25] because well first of all, you don't need to change anything. It's the same API. So no code changes required in order to use uh DF cache on serverless. And then it follows your data frame which is quite different from what it was in classic. In classic it was um it was query plan matching. So that
[18:41] meant if somebody else did something on that cluster that you were working on, it might mean that you get matched on that uh on that cache data and you might get stale data from that cache. And it was very unpredictable. Now it's actually predictable and it's fully isolated to you. So I'm I'm very happy about that implementation.
[18:58] All right. Um we are providing uh more performance insights. One of the other things that I've heard is like, "Hey, but you know, maybe some of the people in my team are not writing the code as efficiently as it should be and serverless is just compensating for that, right? Just by scaling up more. And so,
[19:14] we're providing a lot more insights already today through these Gantt charts and also through wall and task time in the insights panel in jobs, for example, but also in SDP. And that gets you a sense of the scaling. But even more important, you're getting these this little light bulb icon, which
[19:29] tells you, "Hey, something about your code is not super efficient." For example, you might have disk spills, right? And then there is the optimize button here on the right side, where Genie is going to help you optimize your code.
[19:45] And then of course, uh you know, um we have uh cost controls. And specifically, you'll be able to uh tell say and determine who in your team gets access to serverless and how much they're allowed to spend, which is the rate limits piece here. So, if you're interested in that, again, sign up here. Um there's a specific talk about cost
[20:02] controls also tomorrow, which is also linked from here. So, check that out. And with that said, I want to hand over to Santosh, who's going to talk us about the practical side of how to migrate over and um what they've achieved in their migration.
[20:18] Thank you, Roland. Hello, everyone. I'm Santosh Tandon. Can you hear me? Is it working? Let's see. Maybe it's
[20:44] Hello. Okay, just use mine. There you go. Minor technic- minor technical difficulties.
[21:14] Hello everyone. Good evening. I am Santosh Standy. I'm a senior manager at Deloitte Global. And I lead a large-scale data platform uh at Deloitte. Today I came here to uh share our story with uh Databricks
[21:30] and how we transitioned from classic computer to serverless. And our experience with it. So the use case. Deloitte performs financial statement audits for its clients and provides audit opinion.
[21:46] So we have over 60,000 auditors across the globe performing this work. As part of the audit uh our auditors are required to perform financial statement uh data-driven audit procedure.
[22:02] The which is audit sampling analytical procedures and general entry testing. Which requires our auditors to connect to the financial uh client's financial data and check for errors and fraudulent fraud patterns.
[22:18] That's the challenge. Not all the client the each client data is different. A client who is using an Oracle ERP system general ledger and chart of account is different from a client who using PeopleSoft.
[22:34] And on top of that, every client customizes their systems. So we built a platform that allows all these different client data to come into uh come into our platform, transform them into one uh
[22:49] one common data common data model and uh one common data model. So our practitioners ingest the data transform them into common data model and run data quality checks for accuracy and completeness and securely store the
[23:06] data for downstream consumption. And the result, every auditor can run the same procedure consistently, accurately for every client.
[23:24] Databricks is the core for our our platform. It covers every step what you see in uh every step you see in this picture. With the exception of data extraction, we use few different tools. After that, all the data processing happens in Databricks.
[23:41] So, for data extraction, we use combination of multiple tools, uh vendor vendor tools. Second is we we use the SFTP tools and custom scripts to bring in data into the platform. Uh and then we provide uh
[23:58] we use interactive clusters for uh for our data practitioners to help transform the client data into the common data model. And we built a custom data quality checks uh using PySpark that allows our our
[24:14] practitioners to define data quality rules uh for each data model, apply them into the incoming files, and store the uh data quality indicators within the common data model so that our auditors review those data quality
[24:31] reports, and uh and use them during their audit procedures. Data management is the core is the core for everything what we do. We use Unity Catalog to govern everything. And
[24:47] we we we strictly follow uh uh regulatory guidelines and keep the data isolated completely data isolated between the clients.
[25:02] And on the right, we see our data products, which are modern applications built using React application React with .NET. They all connect to uh Unity Catalog through Databricks SQL Warehouse.
[25:19] So, the result is Databricks is a backbone for our platform. It provides trusted, standardized auditory data uh for the Deloitte's data-driven audit procedures at scale.
[25:37] The platform is live and deployed across all geographies, which is Americas, EMEA, and APAC regions. It is a one-governed Databricks Lakehouse used across the whole Deloitte global network. About 60,000 auditors and thousands data
[25:53] practitioners use this platform every day. And our our teams ingest data, prep the data to common data model, and use them for audit processing 24/7. The platform process around 20,000 jobs
[26:11] daily across geographies uh with terabytes of data uh uh with terabytes of data daily. So, here is the architecture we built with a classic compute. At the top, you have the front-end apps we built using
[26:28] React JS uh supported by the .NET microservice. And they all integrate with the Databricks classic compute with API integration layer. We use Jobs API to submit jobs and statement APIs to run the queries.
[26:44] And we have a Power BI and Tableau they connect to the SQL Warehouse directly. And we have a uh for governance, we use external high metastore with SQL. And yeah, I want you to pay attention to the classic compute. This is the compute we
[27:00] provision, we size it, and we manage per region. So, for our use case, we provision multiple computes. The first compute is data prep notebooks. These are all interactive clusters we provision for our data data practitioners.
[27:16] We provision around 10 to 15 clusters depending upon the busy season. And we have the other clusters called jobs. So, we have a jobs that get triggered on demand based on the UI interactions. We have we need to support about 20,000
[27:32] jobs daily. And for that, Deloitte built a custom uh for that Deloitte built a to keep the cost low and provide better performance. Deloitte built his own orchestrator. During peak season, the system scales
[27:48] out that uh all our purpose compute from 1 to 20 to process multiple jobs parallelly and scale back to 1 during normal hours. And the and the third cluster type is SQL warehouse. We provision this for our
[28:04] front-end apps to interact with the data in the Unity Catalog and for our BA workloads. And what you see everything here is like Databricks control plane orchestrates the clusters, but the compute leaves within your environment.
[28:21] And you manage it. This setup uh this carried the global audit workload for years, but it comes with uh cost. There are two pain points major pain points for us. One is billing, and another is a driver behavior.
[28:38] You pay for idle cost. To provide better performance for our uh auditors, we keep the clusters running. So, whenever a job completes, your driver and executor nodes goes to the ideal pool. And it waits for the next job to come
[28:53] in. And while it is waiting, the computer is running in your resource group. You're paying for it. And imagine with the clusters the light manages, that becomes huge. We manage around 25 plus interactive clusters across the
[29:10] regions. And the next thing is you get two billings. For us, you have a DBU's Databricks charges for the for the use, and you have a second bill that comes for the VM. And that is coming from your cloud
[29:25] provider. For us, it's Azure. And you pay for all the idle instances here. The next one is driver behavior. So, we provision interactive clusters for data preparation, for data practitioner. Up to 20 users can connect
[29:43] to a single cluster and run their transformation notebooks. So, it means a day driver is shared between multiple users. And this causes frequent driver restarts. You would see driver not responding.
[29:58] And to find your to minimize these issues, we fine-tune the cluster with multiple Spark tuning parameters like aggressiveness and GC garbage GC collections, and also increase the memory of driver to limit the frequency
[30:15] of these driver restarts for shared clusters. This is the two problems exactly what serverless solves for us. It is paper query. You pay for only the compute what you use. There is no idle
[30:31] compute in serverless. The second is you have a single billing. Databricks charges you for the compute as well. There is no separate bill for the compute. Third, you have a one serverless option.
[30:46] You have nothing to size and manage. The next one is the driver behavior. So, there is a architectural change. What you call as a driver in the classic, now it is a rebel and they provision a rebel per user. It is around 16 GB or 32 GB options.
[31:03] And this helps us with limiting the driver issues what we have seen when you have a shared compute. So, so that to close it, serverless removes both the pain points. We pay only for what we use.
[31:20] And we get one simple bill, single bill, and we stop managing clusters and drivers. And here is the architecture after serverless. If you see it, all this front-end .NET microservices,
[31:36] your integration layers, everything remains constant. The only thing changes, two two things changes. One is your Unity Catalog, which is replaced by the external high metastore. And you have a There's a significant change we have to make. And for
[31:52] serverless compute, it is replaced with your classic compute. That's a simpler change. And all this compute is managed within control plane. Databricks control plane orchestrates it. And the compute lives within the control
[32:07] plane. So, there is no compute to manage. I will walk you through our next steps on how we are uh migrated from classic to serverless. So, here are the our migration journey. The first step is Unity Catalog.
[32:23] We have around 70,000 schemas that you had to migrate. We had to uh perform a lot of steps. Uh it took us around 6 months to stand up uh Unity Catalog. I For this uh for this
[32:38] talk, I'm skipping the details of Unity Catalog and I'll focus more on serverless. So, we divided them into two phases. The first one phase one is MVP phase. And in the MVP, we focused on uh interactive notebooks and Databricks SQL
[32:55] warehouses. Uh on purpose, we have chosen these two things because it's a swap in techno infrastructure and we want to bring in serverless to the hands of our practitioners as fast as we can. So, it took us around four steps to do
[33:11] that. I'll walk you through it. The first step is infrastructure setup. We need to enable serverless to connect to your components. So, for us, we have a lot of components our notebooks will connect to. One is storage accounts, service bus, event hub,
[33:26] Redis cache, and all of them with a combination of service endpoints and private endpoints. Uh Databricks engineer we we do not have private endpoints when we started it and we had to work with Databricks engineering team to enable private
[33:42] endpoints for all our components. So, once we have that private endpoints and service endpoints connection established, the next step is notebook transformations. Before I go to notebook transformation, I will tell you about the init scripts. So, in classic compute, we used to use
[33:58] environment variables in init scripts and there is no init scripts in serverless. So, we built a custom utility that reads the policies from cluster policy uh and your environment variables and sets it on every notebook.
[34:14] So, once that uh infrastructure setup and init script is com- completed, the second step is notebook transformations. There are two major uh changes for us in notebooks. One is uh ANSI is enabled by default for V1
[34:30] version V1. And already low-level APIs are not supported. What we did is we scanned through all of our data practitioner notebooks and uh and and added a setting in the notebook to
[34:45] uh to disable the ANSI for ANSI to keep the backward compatibility. And the next thing we have trained all of our data practitioners to move away from audit ease. Once it is done, the step step step is endurance testing.
[35:01] Our platform is always on 24/7 and thousands of data practitioners use this system every day. And so what we did is like we want to ensure that serverless is able to support our
[35:16] busiest workloads. So what we did is like we followed endurance testing, we ran we took the busiest day in the entire year and we replayed back that entire load in our lower environment with synthetic data 24/7 24 hours. One with classic and then
[35:35] with serverless and we compared the results. This gives us confidence that serverless can handle the busiest load in our production busiest production workloads. Once it is done, we went with with the confidence we went live with uh
[35:51] serverless for interactive clusters and SQL warehouses. The second phase is where we did the transformation notebooks. We have a a lot of jobs that gets triggered using the UI. Those jobs like we we took around uh we transitioned around currently 40%
[36:08] of them and it did that that involves migrating from serverless Scala to Python notebooks and V1 to V4 notebooks. And here are the results after with with our journey within phase one and phase two.
[36:28] We observed consistently across every month that serverless has provided us 70% cost savings across SQL warehouse and interactive notebooks. And we have observed 30% cost savings so far uh so far while we are running serverless and classic compute side by
[36:45] side. We expect the jobs uh compute uh savings would grow faster would grow significantly while we transition to serverless in coming months. The next observation is we have seen for the larger files, greater than 100 million records to 1 billion records,
[37:02] serverless performed very well. We have seen 50% performance improvements. For example, we have a complex transformation notebook that used to the for a billion records, it used to take 2 hours with our classic compute. It took 1 hour
[37:19] uh in uh for the same thing. And the next uh thing is operational side. We have no more driver support tickets from our practitioners. And our infra- infrastructure is simplified.
[37:35] So, across the board cost, performance, and operations, serverless has paid off for us. And I have a uh one of the during the transition of phase two, we want to upgrade from V1 to V4. V1's
[37:52] uh maintenance is ending next year. So, we are upgrading all our workloads from V1 to V4. During the transition we uh we ran ran into problem. For classic compute, whenever you want to run the if you want to do the runtime upgrades,
[38:07] you just send to it it is centrally managed with your policies. So, you uh you edit your edit your policy and manage your edit your policy, apply the new runtime, and all the notebooks would use that same policy.
[38:23] But it's different for serverless. In serverless, your base environments are pinned at each notebook {{}dot} ipynb files. So, for us to edit uh for us, what we need to do there is like we have to edit all of our notebooks.
[38:38] Uh ex- uh export, edit the notebook metadata, and change it to V1 to V4, and re-import it back. And we have around 150,000 notebooks that our data practitioners create across the around the globe.
[38:55] And this cost us around with the efficient scripts we built, it took us 12 plus hours to perform this activity. And we pay this penalty every release and every version upgrade.
[39:12] So, we connected with Databricks team through our RSAs, and they connected with the product teams uh with Jess uh Jessi and Roland, their team, and they partnered with us on the solution. A workspace default base environment. It is now generally available. The idea is
[39:29] simple. You set the runtime once at a workspace level, and your notebook stays untouched. So, for that, Databricks provides you three APIs. One API is for get API to get the workspace default version, patch API to apply the new version, and
[39:47] post API to refresh your cache. And what it what it used to take 12 hours for us, it is three API calls, and everything is automated. So, the next upgrades, it is a one pipeline step.
[40:05] So, our partnership with Databricks turned our scale our scale problem into a platform solution. And here are the lessons learned based on our experience with serverless. The first one is keep your architecture flexible. Not all regions have serverless.
[40:21] So for us, we deploy this in API and APA and China regions, serverless is not available. We have to use feature flags to use classic compute where serverless is not available. The second, use base environment. That will be a single source of truth. Avoid managing your
[40:38] versions even in the .ipynb files. The third one is invest in observability. Log delivery is not available. It's coming in the future. Uh what we did is we implemented our custom log 4J appenders to uh push the logs into telemetry.
[40:55] The fourth one, you have a limited Excel support with serverless. The REPLs uh when we have a large uh Excel files with a billion million records with multiple Excel multiple wider columns, it performance degrades. Our recommendation
[41:10] is to use the text and CSV files. The overall the takeaway for this talk is serverless transition from classic to serverless is straightforward. And the collaboration with Databricks powered our journey. The team helped us enable private endpoints and a workspace
[41:27] level base environments uh for Deloitte. And there we have seen around 70% cost savings and 50% performance improvements for the larger files. That's our journey. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.