Sponsored by: Zipher | Dynamic Compute Allocation: Runtime Placement to Reduce Costs by up to 60%

Summary

This video frames Databricks compute selection as a runtime placement problem: the same job can face 40% more data one run than the next, spot prices vary by time and availability zone, and static configurations cannot respond to this variability.
Dan Coster presents Zipher's ML-based dynamic allocation engine, which benchmarks serverless against classic compute using TPC-DS and selects the right compute type, configuration, and instance mix per run, achieving 35–60% cost reductions across enterprise customers.
Samuel Sterling from Hertz shares a case study showing a 43% overall Databricks cost reduction achieved through dynamic compute allocation, demonstrating how the approach scales to large enterprise data platform workloads.

Chapters

FAQs

Why does static compute configuration fail for Databricks workloads?

Production workloads are inherently variable: the same job can process significantly more data during peak business hours, spot instance availability changes by time of day and availability zone, and business cycles create weekly and monthly demand swings. This video shows that a configuration tuned for average load will either overspend during low periods or miss SLAs during peaks.

What is Zipher and how does its dynamic compute allocation work?

Zipher is an optimization layer for Databricks that uses ML-based models to select the right compute type, configuration, and instance mix for each job run at runtime rather than relying on a static cluster configuration. This video explains that the system considers serverless versus classic compute, driver and worker types, spot versus on-demand availability, and autoscaling parameters simultaneously.

What cost savings have customers achieved with dynamic compute allocation on Databricks?

This video reports that enterprise customers using Zipher's dynamic allocation approach have achieved 35–60% cost reductions on their Databricks spend. The Hertz case study presented by Samuel Sterling shows a 43% overall Databricks cost reduction achieved through this approach.

What is the difference between serverless and classic compute for Databricks jobs, and which is cheaper?

This video presents an empirical TPC-DS benchmark comparing serverless and classic compute, explaining that neither is universally cheaper because the answer depends on workload characteristics, data volume, and current spot market conditions. Zipher's approach is to run this comparison dynamically at runtime so each job uses the most economical option for that specific run.

Full transcript

[00:08] Hi, good morning everyone. I'm Dan Coster. Um I'm the CEO of Ziper. Um we developed an optimization layer for database workloads. Here with me Sam from Herz, Samuel Sterling. He's a data architect in Hertz. And it will be our
[00:21] pleasure uh to introduce you today uh our talk around dynamic computer location. Um uh so uh with your permission I'll be happy to start. Uh we'll have a a
[00:34] question sessions at the end of the talk. So if you'll have some questions we'll have a few minutes at the end.
[00:54] Great. Oh yeah. So, so this going this talk is going to answer the question that comes up again and again for most data engineers and data platforms engineers. How can we allocate the right compute to each workload per run and
[01:06] meet the SLA? The reason this question is so challenging is because in order to answer this it you would need to consider a multiple factors.
[01:18] So first for example whether to use serverless or use classic compute. If you choose to run on serverless you can select between two modes performance or optimized. And if you're using notebooks
[01:30] you can even select between low memory or high memory. If in case you're using classic compute you first should select on which SKU you would like to run your workload on job clusters on all purpose clusters or on
[01:43] instance pools. For each of them, you need to select the driver type and the worker type. You can tune multiple factors, multi multiple Spark configurations.
[01:55] You should select whether you want your workers and drivers to run on spot or on demand instances. Oh, I have a clicker so it would be easier. And you should select whether you want to use a fixed
[02:07] number of clust of workers, sorry, or whether to use autoscaling and define the band of the min and max workload. You can also use Photon in order to help you optimize the duration of your
[02:19] workloads. And if you're using AWS, you can even select the availability zone where your cluster will be deployed and also you can select whether or not to allocate external disk in Azure or
[02:31] EBS in AWS. So basically there is a lot of to configure and the decision space here is huge. There are multiple options across different parameters here. Um so usually
[02:46] what happens then you go do some trial and error cycles make sure your workloads meet the SLA and from that moment you push it to production um and and it's all very critical parameters uh
[02:59] to tune proper so you would be able to meet this SLA so this pro is now getting 10x bigger uh for two main reasons or for two for one
[03:11] main reasons there is an exponential growth grow in the number of data workloads and that is driven for from two uh core reasons behind this uh shift. The first one is the fact that cost and um and also uh the time
[03:26] required to generate another data workload is essentially reduced to zero. I guess multiple uh folks here and also in your companies use their favorite decoder. It could be claude, it could be
[03:38] um cursor or any other copilot. And the in essentially the time for workload generation is go uh to zero and also the cost of generating yet these workloads
[03:50] is also getting lower. So for the same amount of effort you can now generate much more data workloads. The second factor is the fact that agentic applications are by design data inensive workloads. So as agent takes
[04:04] over time more business function they replace human knowledge memory and manual tool usage with the queries context management rugs all of these are data intensive workloads uh that needed
[04:17] in order to provide uh youric apps with the right capacity.
[04:29] So we did some analysis um from our customer data. So it's all anonymous data. So here what you can see in blue and in green these are the this is the growth rate in the number of uh SQL
[04:41] queries and jobs within uh more than 15 accounts 50 accounts sorry eight of them are like big enterprises Forbes global 2000 companies and we can see here the growth rate in the number of workloads
[04:54] between 23 to 24 we can see it's around 10% of both jobs and and um and uh and SQL queries in 2025 the growth of a bit higher between so we just measure
[05:08] between my 23 to May sorry May 23 to May 24 uh what in compared to the year before what was the growth rate in the net number of of workloads you can see that in 24 uh it there was a
[05:22] higher h like a higher growth rate and the question is what would be the grow what is the growth when we compare 25 to 26 to the year before. So, um let's do
[05:34] it a bit more interactive. So, uh we'll do a quick survey. Um who thinks that the growth rate between uh 25 to 26 in compared to the previous year was higher than 20%. Between 20 and 30%. Raise your
[05:47] hand. Okay. Who thinks it's between 30 to 40%. Who thinks it's more than 40%.
[05:59] Okay. So um the popular answer is also the right one in this case. So between 25 to 26 there is a 40% higher number of new workloads or number of workloads in
[06:12] compared to the previous years and the reason is clear it's much more easier to generate new data workloads and it's we can also see it in our data.
[06:28] So the point that I would like to mention now is um to show you how cluster configurations that are allocated in static manner might not be the most optimal solution. So a developer choose the infrastructure
[06:40] config once that is had that that is has the sufficient uh resources to support the workloads and does and does not revisit it until something bad happens. It could be a failure, it could be SLA
[06:52] breach etc. As a result, every run has the exact same cluster configuration. You can see here like the um the dashed line and and this static config doesn't
[07:04] meet the inherent variability of the workload. So here we can see an hourly workload. H it process different amounts of loads across different times of day. So we can see that this workload is
[07:16] overprovisioned for most of the time of the day in order to meet the peak utilization that is usually around noon time sometime around 2 let's say and this leaves all other hours of day with
[07:29] unutilized compute resources that you actually pay for and this patterns is not only for hourly workload. So you can see here on left that you can have a day of week
[07:42] seasonality pattern for your workloads. You can have an incremental workloads. For example, if you calculate some monthly process with sliding windows that will be will get bigger day after day. You can have a a parameter based
[07:54] variability. Um so for example um if you um you have like a customer ID that uses a parameter for each of your workloads and each of customer ID h um in
[08:07] introduce a different load. So this all uh leaves you in order to meet the peak utilization with a lot of overprovisioned resources
[08:23] and low variability also occurs in continuous spark structure streaming jobs where it is often more nuanced and hidden in stream specific metrics like microbatch sizes and Q Q lags. As you can see here across time the different
[08:36] micro batch size. So now we have established that workloads are inherently variable. Let's look at the compute options teams typically typically have at high level
[08:51] you have a several SKUs and each of them has a different characteristic. First uh is the all-purpose compute. This is usually used for interactive work, notebooks, debugging, um,
[09:04] development and in many organizations some produ production jobs still end up running there as well. Oops, sorry. I think a pro
[09:33] the bottom of the remote. >> Sorry for that. I just pressed this one by by mistake. Okay, that was easy. Sorry for that. Um so
[09:46] then you have classic job compute. Here you define the cluster configs. uh you can see um well I don't press you can see on the right upper part and the cluster typically spun up uh for the job
[09:59] run and terminated afterwards it's an ephirmal cluster and uh you also have um instance pools um when you have some pre-wormed instances and you can allocate the workloads to these instance
[10:11] pools and utilize uh uh like uh utilize it it's usually more uh relevant for short period jobs and lastly it's serverless jobs compute when you don't need to do any cluster tuning.
[10:26] So there in many enterprises the decision is still made statically like this job runs on serverless that job runs on classic compute these workloads use a pool and those streaming jobs run on
[10:38] allpurpose streaming allpurpose clusters. The problem is that the workload itself is not static as we showed before. So infrastructure layer around it shouldn't be static as well.
[10:52] So I'll show now a few examples uh for the different aspect of utilizations of different uh um compute type. So here we see uh uh data from one of our customers
[11:05] an energy company who spends more than $2 million a year on data bricks. These are aggregations of 10 instance pools that they h used before we start working with them across different hours of day.
[11:17] So we can see here that in some part of the day or across all you can see here the memory utilization and the CPU utilization the average ones across different hours of day. So you can see here that within this instance pool
[11:30] there is a very low utilizations of CPU a bit higher utilization of memory across different times of day. So generally speaking, these instance pools most of the day are underutilized.
[11:42] And so this is exactly the kind of problem that cannot be solved with a one-time static configuration. The right answers shouldn't be like use pools or don't use pools. The right
[11:54] answers should be dynamic. When should they use pool? How large should they be? Which workload should they lender? And when should we shift workloads to a different compute option? That is exactly where runtime placements become
[12:07] critical. Here is another example of utilization of allpurpose cluster. This is for from food and beverage company uh spends more than $10 million annually on data
[12:20] bricks. They had 27 all-purpose clusters across different workspaces. So we can see here that there is a very low utilization of CPU across day. peak is around noon time uh with higher memory
[12:34] utilization. So allpurpose clusters are extremely useful for notebooks, exploration, debugging and development. But in many organization they also become a convenient place to
[12:46] running recurrent workloads, headoc pipelines and even streaming jobs. Um but we see that um they are still not utilized enough. So runtime runtime placement of workloads can still be
[12:58] relevant in this case. Now I would like to show you a different angle. So we closely monitor and collect data uh anonymously from many of our customers and we also use our own
[13:11] collectors to collect the spot termination probabilities of different instances. So in this case um I would like to illustrate the the spot reclaim probability of different instances. For example, we choose here the instance R
[13:23] six ID to Excel. It's a common instance in AWS and we compare here the metrics of the probability the spoke reclaim probability across different hours of
[13:35] the day across different availability zones. I think the interesting thing here is to see the difference of this probability between weekdays and weekdays both across different availability zones. It's all data from US East one from multiple customers. So
[13:49] what we can see here is that for the selection of the same instance type it could be different um different probability for spot termination event throughout day. Um so the key observation here is that weekends behave
[14:02] a lot different from weekends with overall better spot stats across across the board. But this is getting even more interesting when comparing different month time. We also see a shift in how
[14:16] spot market behaves. Here we compare April 26 to May 26 of the instance IE38 XL with a clear difference evolving over
[14:28] time which naturally requires adjustments to workloads and infra configuration. A key takeaway here is that the availability also stems from the your underlying cloud provider not just for the from the workload itself. A
[14:42] data team wanting to make a costefficient usage of spot instance at production would have to constantly adjust their compute configuration to accommodate this variation. So for example here we can see that during April um it was very high spot
[14:55] probability um on availability on availability zone two. So for those of you who are not working with AWS, so in AWS besides selection the region when you will place your workload, you can also select the availability zone and
[15:08] that provide you another layer of variability. Now to compare uh data brick serverless compute and classic compute objectively
[15:21] we run an empirical benchmark using TP 2pcds uh which is a a common benchmark. Uh we use the serverless default configuration which was um standard and um or performance and we use compare it
[15:35] to six instances um also on AWS memory optimized compute optimized um and R six ID just mentioned and we use uh instances of spot versus on de and on
[15:47] demand uh we used autoscaling band of four and 12 and run it with uh 14.3 database version and the driver was MC 6 ID except we run each workload three
[15:59] times per day at 6:00 a.m. 2 p.m. and 10 p.m. for 3 weeks and we run sim simultaneously sorry a serverless job um to reduce the timing bias because as I
[16:12] mentioned for spot instances there is a variance in the probability of these instances. So what we can see here that the cost of serverless compute was almost 3x higher even in compared to on demand um and and
[16:26] much higher if we compare it to to spot instances. Another um experiment that we have done is around photo. So we utilize the um so
[16:39] we create a um we want to evaluate the effectiveness effectiveness of photo. So this is something we tested pretty recently. We used two synthetic data set simulating real world data volumes that
[16:52] were stored into bronze layer. A small data set with 100,000 users and 1 million transactions and a large data set with 1 million users and 10 million trans transactions. Each data set um included a users table
[17:06] and a transaction table containing randomized usernames, countries, transaction amounts, and dates. To evaluate performance under different processing scenarios, we created three variations of the transformation logic
[17:19] for converting bronze data into silver silver layer and enrichment and filtering, categoriz categorization and windowing and rolling metrics and dense ranks. We run each scenario with and without
[17:32] photo. So you can see here that in on the large data set uh photon duration which you can see here on the like fill like the full color um was uh pretty
[17:45] short uh the the duration of the job was short when we enabled photo. However, the cost was um between two to six higher uh when we use photo
[17:58] also here uh but on the small data set the uh duration of the workloads was almost the same but the cost was significantly lower. So I think the point here is that photons should be
[18:10] super helpful when you use it for the right workloads you need to meet their SLA but if you have workloads with with more loose SLA then you need to consider it carefully when and if to use photo.
[18:28] Okay. So I demonstrate here a few examples of like how variable work how variable workloads are and how right allocation of compute type could impact um uh the performance and the
[18:40] cost of uh of the workload and with that permission with your uh so now I'd like to elaborate a little bit about what we're doing in Ziper. So Ziper is a optimization layer for
[18:52] data bricks workloads. We compute we develop sorry a full abstraction uh for compute allocation for database workloads. You can install Ziper within 15 minutes. It's a zero touch platform which means that we are not a
[19:05] recommendation system. Our manage our system autonomously allocate the right compute per each workload and it's based only on metadata which means that we don't have any access to your actual data PII or code. The way it works is
[19:20] that we install a very lint collector on the driver of each of your clusters and this collector collects only operational metadata. So I'm talking about two types of metrics. The first one are resource utilization metrics like CPU, memory,
[19:33] disk and network. And the second types is spark logs. Based on this metadata, we send it to Zeler core and we actually learn a profile per each of your workloads. Now based on this profile we can now apply a
[19:47] set of optimizations um that will help you to get the most out of your workloads with the lowest cost. The first one is the most and the most important one is what we call dynamic cluster configuration. So as I showed
[20:00] before workloads tends to have variability in their load. So we don't need to use the same static configuration for every job. I just exemplify for example in the early job. What we are doing here using our dynamic cluster configuration is that we to we
[20:13] take into account two main aspects. The first one is the seasonality pattern of the workload itself. We show the examples of daily jobs and hourly job. So instead of using the same static
[20:25] configuration for every run, we know what would be the predicted load in the next run. The second parameter here is the spot instance probability. So instead of using the same instance type for every run, we fuse these decisions
[20:38] and before every run we select the most optimal uh worker and driver types in terms of the predicted load but also the most available one in term in the spot market. We also take full ownership of
[20:50] the full cluster config. So our models select the most optimal spark configuration, whether or not it's beneficial to use photon, the driver and the worker type, the number of workers, whether or not the number of instances
[21:03] of spot versus on demand, and we actually take full ownership of all the cluster configuration. Second, we have two runtime optimizations.
[21:16] The first one is our proprietary sparkware autoscaling. So what we are doing here we developed a spark aware autoscaling that is based on your actual load. So because we learned already the profile of your workload using using our
[21:28] collector we can provide an efficient scaling pattern that will be ML based and based on the on the profile that we learned. So it we make sure that we provide the most efficient scaling
[21:40] pattern and both reduce the cost of the jobs and the duration of it. The second optimization is is around spot instances. So instead so we developed a runtime optimization that
[21:53] enables you to enjoy the p the good price of spot instances in production. What we are doing here is that we closely monitor the spot termination signals in the cloud provider and before
[22:05] the instance is being taken by the cloud provider we reshuffle the data from this instance across all the other workers on the cluster. We are doing it in runtime and prevent the long spark retry and in some cases even prevent fetch fail.
[22:19] Sorry for that. Lastly is our compute auto routing feature. So what we are doing here is that we closely monitor the utilization patterns across the instance pools and
[22:32] the all-purpose cluster and we can identify when there is an active cluster which is underutilized. In that case, we can route the workloads to this cluster dynamically and do it um before every run.
[22:49] In general, we help our customers to reduce between 35 to 60% of their cost. Um 35 is really our lower bound. On average, we save between 40 to 45. We reduce both the data bricks cost and the cloud provider cost. We also reduce
[23:02] pipeline failures mostly info related failures like out of memory out of disk space um qua exceptions fetch failed errors and um um AWS insufficient
[23:14] capacity events and some others but these are like the main ones and we help our customers to reduce the completion times of their jobs and with your permission now I'll uh
[23:26] give the um microphone to someone who will elaborate a bit about our work with Herz. >> Hi. Um,
[23:38] you all know Herz. We operate in 150 countries or more. Um, over half a million vehicles and over 25,000 employees. So, we're pretty
[23:50] goodized company. Um, and the Herz data platform. Now, a couple days ago, u some of you attended a session by a couple of my
[24:04] co-workers that I worked with on migrating from terod data and other platforms to uh data bricks and um
[24:16] so we're going to talk a little bit about that. We lifted and shifted and they did a great job. I've never seen a migration go that well. Um but it wasn't I wasn't in charge. I
[24:29] was just on the terod data side. So what we're doing is
[24:41] you know at be not unexpectedly cost went way up. So we brought in Zepher for a quick hit, reduce cost, you know, quick ROI and then leave the
[24:54] the remaining moles to go whack afterwards. And so we were able to bring down cost significantly. But the other aspect of that is
[25:08] um that people are using it much much more. They're doing things on data bricks they couldn't do on terodate or postgress or Oracle or whatever. So these are the the
[25:23] compute mix that's running now. Um we brought in um we brought over the the batch jobs Um, we have things running in
[25:38] all-purpose clusters, various instance pools, serverless, and a lot of that was driven by um, the II ML from Cipher. They moved
[25:51] things around very quickly, brought savings down a lot for many, many jobs. And then we went out and tackled the the still the still
[26:04] um large jobs, the expensive jobs. Um
[26:18] but the work is growing exponentially and if you watch the keynotes, the last half hour of both of those keynotes were all about AI and how we're going to be doing more and more and more. And we've got some really
[26:31] aggressive users who are using AI for all it's worth. And staying on top of that is almost impossible, which again is
[26:43] what Zepher brings for us. Um so load is constantly changing. Static
[26:56] well static just doesn't exist for us. Workloads change hour to hour because we're across 24 time zones and
[27:09] you know and and different time zones have different rental volumes and things like that. Moving cars around, all of that stuff is done on data bricks now.
[27:23] But it's different at 3:00 a.m. California time than it is at 2:00 p.m. California time.
[27:38] Um, the other problem as Dan mentioned about static is you have to provision for the largest um workload and that means you're overprovisioned for half the day.
[27:52] So that's what Zepher does is it knows what times are busy. It knows where it can get spot instances and
[28:04] that's you know that's how we've reduced cost significantly.
[28:20] And so overall um we've seen about a 43% reduction in cost overall. Um per workload it's it's higher. It's 46%.
[28:34] We're projecting over a million dollars. Um actually we we suspect and and Dan and his team and I have done some analysis on that. Actually, the reason those numbers are so low is because
[28:50] we tackled the production stuff first, which is fairly well tuned. The stuff that's not well tuned is all in data labs and ad hoc stuff. And those are going to be a huge savings.
[29:05] And you know, we're just not done. We're we're about six months into it. And my my mission the next six months is to get towards that.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.