Skip to main content

Saving Millions From Millions: Navigating Towards Cost-Efficiency in Pinterest's Spark Jobs

Summary

  • Cost efficiency at scale: Pinterest runs millions of Spark jobs monthly and has developed strategies saving tens of millions of dollars through performance optimization and cost management.
  • Analytical approach: This video uses insights from billions of collected metrics to identify performance bottlenecks and implement targeted technical solutions across job instances.
  • Infrastructure innovations: Remote shuffle services address shuffle slowness, improve memory utilization, and reduce costs while supporting hundreds of millions of pods efficiently.

Saving Millions From Millions: Navigating Towards Cost-Efficiency in Pinterest's Spark Jobs

Watch: Saving Millions From Millions: Navigating Towards Cost-Efficiency in Pinterest's Spark Jobs
Talk by Nan Zhu, Staff Software Engineer, Pinterest
While Spark offers powerful processing capabilities for massive data volumes, cost-efficiency challenges are always bothering users operating at large scales. At Pinterest, where we run millions of Spark jobs monthly, maintaining infra cost efficiency is crucial to support our rapid business growth. To tackle this challenge, we have developed several strategies that have saved us tens of millions of dollars across numerous job instances. We will share our analytical methodology for identifying performance bottlenecks, and the technical solutions to overcome various challenges. Our approach includes extracting insights from billions of collected metrics, leveraging remote shuffle services to address shuffle slowness and improve memory utilization and reduce costs while hosting hundreds of millions of pods. The presentation aims to trigger more discussions about cost efficiency topics of Apache Spark in the community and help the community to tackle the common challenge.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How many Spark jobs does Pinterest run?

Pinterest runs millions of Spark jobs monthly, making cost efficiency optimization critical to support rapid business growth.

What were the results of Pinterest's cost optimization?

Through various strategies, Pinterest has saved tens of millions of dollars across numerous job instances while hosting hundreds of millions of pods.

What role do remote shuffle services play?

Remote shuffle services address shuffle slowness, improve memory utilization and reduce costs—key techniques in Pinterest's overall cost efficiency strategy.

Who should watch this video?

This video aims to trigger discussions about cost efficiency topics of Apache Spark in the community and help others tackle similar challenges at scale.

Full transcript

Hello everyone. Uh welcome to my talk. Uh I'm N from Pinterest and uh today I will present uh seven millions from millions navigating towards cost efficiency for Pinterest spark jobs. Uh a little bit about myself. I'm currently a software engineer in Pinterest. Um I work in batch processing platform there and my uh I primarily focus on build a sparkle on Kubernetes ecosystem. Uh optimize our job cost and also build a data infrastructure for faster uh AI and machine learning iterations.
Uh I'm also involving in multiple open source projects. Uh I'm known as a coding cat in GitHub. You can see from my hoodie and uh I also I work previously I work on spark and later on iceberg and recently I focused on option kible and uh for today uh first I will briefly introduce the spark platform in Pinterest and then I will talk about what are the real challenges we we face with when we talk about manage the cost yeah in Pinterest for spark jobs and to address this Because we have multiple layers of solutions ranging from how we improve the debugability of cost issues.
How we build a platform innovations which uh which are the general solutions applying to all of our spark jobs and how we build a fine grand cost management strategy for those very very expensive or very very like a uh challenging projects and uh finally I will give a summary about this talk and happy to take any questions. First of all, Spark platform in Pinterest. Basically, our platform consists of three layers. Uh client platforms.
These are the channels for our users to submit a spark jobs. They can submit at a schedule pipeline. They can use interactive notebook. They also submit SQL queries uh based on Spark SQL to our servers uh job servers and bunch of our customer build their own platform on top of ours like our machine learning platform team build the back feature back fair platform uh on top of spark and the next layer is a spark submission servers.
We have two types of uh submissions uh servers uh submit our spark jobs to Hadoop cluster and kuberneti cluster respectively and we build currently we still have two types two types of cluster here hadoop and kuberneti. We build uh these clusters on top of Amazon EC2 and you can imagine we are actually in the middle of consolidating our Hadoop cluster into kuberneti. So when you open the Pinterest app um the experience you you have is basically backed by a lot of sophisticated machine learning models.
Uh why you see these pins in your uh first page? Why you see this ads instead of those? Uh so all of these are backed by those machine learning models. And to build this large scale machine learning models uh there you can you can imagine there will be tons of like data processing work and all of these use cases are built on top of AI spark and because of we have so many use cases we have one of the largest spark platforms in this world we have 60,000 workflows running our system and we have tens of thousands of computing nodes ranging uh in more than 30 computing clusters every Today we have hundreds of thousands of spark jobs and these spark jobs are also very big.
They process hundreds of parabas of shuffle and input every day. And most importantly our usage is actually growing in a databases and because of this fast growth we are facing significant challenges in manage our spark job cost. First of all as I said uh we have a very fast growth of our spark usage. uh for some months we can see up to three to four percent of usage comparing to the earlier ones and we also have so many use cases uh user can submit a pure spark SQL jobs which are mostly user resource with those built-in spark operators and some of complicated spark jobs consist of some memory heavy user defined data structures that requires a lot of onep memory for them and we also have complicated jobs like the mix scala and C++ code that have a very different resource usage pattern than others and also we have so many jobs every week we need to deal with probably not millions just tens of millions of jobs.
So our goal here is how to save xo millions of dollars from these millions of jobs. We build a holistic framework addressing all these cost efficient challenges. First we improve the cost debugability and second we build some general solutions in our platform level. And in this talk I will mostly talk I will uh mainly talk about how we optimize the shuffle performance with a remote shuffle service and how we use burst aware memory allocation algorithm to save our uh to uh allocate resources to more spark jobs with uh with less memory spa spaces.
And um I will also talk about how we uh apply the fine grand cost management strategy for those like a very expensive jobs and how we scale this with some work in progress project called manage the job set. Uh first how we improve the cost issue debug uh debugability. The goal for us to improve the cost debugability is basically to surface the root cause of cost issues. First for different scenarios whether uh too high cost is caused by uh over supplying the resources uh whether there are some logical implementation becomes a bottleneck or whether the efficiency is caused by some platform components.
We need a we need a tool to identify these issues and help our engineers to address them. And also we need a tool for different audience. Uh our audience can be platform engineers which develop general solutions to improve the cost efficiency of our spark jobs. And we also need to help our spark users who are primarily focused on individual job tuning. To achieve this uh we have uh the key is to improve the spark job observability.
We have the f two types of metrics extracting from all the spark jobs. The first type is called uh real-time matrix. Uh we can use the built-in matrix sauce in spark like a JVM source, executed source uh like uh periodical thread dump to help us to identify issues like over supply over supply resources, logical bottleneck and we use a real-time metrics mostly for individual job tuning. some examples of real-time metrics we collected like a memory utilization, GC states and um uh target and allocated the number of executors.
We also monitor monitor shuffle performance uh with the metrics like fresh failure to avoid the two minute shuffle retries. We also uh have some work in progress uh project like periodically uh generate the snapshot of a thread dump of those stock tasks so that we can uh we can find the bottleneck of those uh implementation. Here is the example uh about how we use real-time metrics to identify cost issues. Uh in this graph we can see the JVM RSM memory.
We see how much memory we allocated to on hip space and how much memory we really use in on hip space. You can see for majority of these jobs life cycle the on usage is pretty low. So that indicate us we can potentially split this job into two jobs and the first one only need a very small amount of memory and the second one probably you need to allocate more memory for that. But even for the second one in the P usage there is always a gap between the allocated onip space and the used onip space that means we are actually over supplying this memory uh me memory space.
So this graph can clearly show to our users that you need some tuning or refactoring of your job. The second type of metrics uh we collected is through event logs. Event logs are essentially the JSON files generated by Spark to if you use Spark you know there is the Spark history UI. Basically event logs are use are those JSON files. Uh we use building event log uh log sauce like uh to capture events like task end, stage end and or persist associated metrics.
We also develop a lot of customized event uh event log source like uh we we track the shuffleer data expiration rate aggregated GC data and for such for such like event logs generated metrics we use injection pipeline to proceed to parse this JSON files and uh save that as a hype table and we use that for platform wide analysis or and also extract the trend for example for a long uh period of time and for such metrics we mostly use them to develop like a general solutions for our spark jobs.
Here is the example about how we leverage this event lock generated metrics. Uh the background is that we have two types of CPUs in our cluster. Uh one is ARM, the other is Intel and ARM CPUs are cheaper. uh but uh for those jobs which requires very power powerful like a per core or core computation for example some jobs need a rejax matching uh with our observation is that the entire CPU usually uh does better job than ARM.
So the challenge here is we have millions of jobs every week. how we can effectively identify which jobs should be run in ARM CPU to save cost without uh sacrifice the performance. The solution here is to have the to uh capture the spark internal accumulator called executor CPU time is based on the JDK API string to track the own CPU time and this accumulator can be captured in the task and event. So here is our architecture.
First of all, we define a matrix called CPU intensity. Basically, that's the sum of all the own CPU time for all the tasks and divide the total task durations. And when spark job is running when task when task is finished the event log listener will capture that and save the JSON object containing that executor CPU time uh to the for example to S3 and then we persist that JSON file. Oh, sorry. We parse that JSON fires and proceed at the park load at load as a hive table and then eventually we can run this query to do to have analysis over millions of jobs and uh sort and know oh okay uh the top ones should be run in uh in because they have very intensive CPU computation and for the uh bottom ones they they don't use CPU intensively so they can easily be rotated to ARM cluster to save our cost based on the improved cost debugability um we can develop a lot of platform innovations to help to cut the cost for majority of our spark jobs.
So the first platform innovation I want to talk about is uh how we optimize the shuffle performance with the remote shopper service. Our start our story start with some customer reports. uh they say oh our spark job is just so slow and it runs for more than 24 hours it takes us a lot of money and when we look at them we find this spark jobs are slow because they spend more than 80% of time just reading shuffer data and we go deeper into that we find most of these spark jobs just run in yarn with the external shuffer service and the root cause of the solunice is that they have a large number of mapper tasks and large number of reducer tasks when the reducers need to read the uh shuffle shuffer data it actually create M m*s unform connection that really saturate our cluster network and to resolve this problem we introduce the remote shuffle service we want the idea of remote shuffer service is basically instead of saving the shuffer data together with your computing cluster it use a dedicated cluster to save that and for every partition of the shuffle it saves in different workers.
In this way when we have a reducers to read the shuffle data you only need to establish uh connections that will avoid that for mesh connection problem. So seems our shuffle read problem has been resolved. However, later we observed our shuffle right performance becomes very very bad. Uh up to some for some jobs with high number of partitions up to like more than 70% of task running time is spending in shuffle writing.
So we have a deeper analysis of why uh deeper analysis of the root cause. We find that um in the client implementation of our uh remote shuffer service we want which is based on one of the earliest open source solution for remote shuffle it just create the buffer time and time again. So we spend quite a amount of CPU time in create a CR serializer strings uh that makes everything so slow. So we decided to switch to a new version of remote shuffleer based on Apache Calibon.
Apache Calibone is the open source project uh started by Alibaba group. uh it integrated with multiple computing engines and it is used by many peer companies uh including linking stripe and it also had a very active open source community in Calibon uh we use sort based shuffle writer so all the partitions that share the same buffer when the buffer for it just send each partition to different workers and uh it doesn't need to create the buffer time and time again so it doesn't have the problem of remote shopper service one and this is very lightweighted to maintain this uh these buffers.
And when we compare the performance of Calibone and the yarn external shuffles and also our first version of remote shuffle, we find that the overhead brought by the remote cluster of Calibon is ignorable and comparing to our first version of remote shuffle, it just significantly improve our uh shuffle right performance. This is a example about one of our production jobs which get a significant benefit when we switch that to use calibbo.
You can see right after we switch to kalon the cost just uh reduced significantly. Uh so because we have dedicated hardware for remote shuffer service where that remote shuffer service itself become a cost center. So due to time limitation I will not go into too many details. The idea basic idea is to we can offset the cost with other innovations in our cluster like a burst aware memory allocation algorithm I will um talk about that immediat immediately after this part we can use the cheaper CPUs.
We also contribute a lot of features for Calibon to uh in Calibon to uh open source community to save the cost of running such a calibbon cluster and we have some upcoming engineering blog to talk about some new features we just developed. we can run four parab four pabytes of KS joining with only like 200 terabytes of disk footprint. That's a significant save saved our cost. So uh the next platform innovation uh I want to mention is the burst aware memory allocation algorithm.
Our customer complained to us saying that uh so why my spark jobs cannot get enough resources and when we look at the uh metrics of our cluster we find that our computing cluster just frequently get overloaded. Then in this graph the red curve is the allocated memory percentage and the green one is the used memory percentage. We find that for this uh for this cluster frequently all the memory are allocated to this spark jobs.
But if we look at the green green curve we find that there is a big gap between the allocated and the user cluster memory. That mean that means our cluster is underutilized. Even our customer is complaining we cannot get the enough resources to address this issue. Uh we implement a burst aware memory allocation algorithm in spark. This idea is inspired by a academia paper from bite dance uh in last year's vb conference.
So uh it's primary focused on optimized the allocation of memory overhead uh which is uh specified by spark. exeutor. mmemory executor memory overhead and the observation is that for use cases of memory overhead space like a direct memory serializ serialization derialization comparation all of these show very like a dynamic usage pattern you can you will not serialize dialized data in the whole running time of your uh spark spark job running time you you just do in some phases in so when you you are not serialized or realize that this memory overhead spent just the wasted.
So how to address such a burst memory demand in Kubernetes cluster? We can leverage the concept concept of request and limit. request just means how much resource how many resources you need to get your Kubernetes pod be assigned to a particular machine and the limit means after you your pod is assigned to that machine what's the max size of resources uh you can use for example if I have a part have a request of 28 28 megabytes and limit at 256 megabytes that means any node with more than or not less than 128 megabytes of free memory can be a candidate for hosting this pod.
But after this pod is assigned to a host, it can use as much as 256 megabytes before it's secured by killer. Applying this this concept to memory overhead space I just talk about the idea is that we can split the memory overhead space into two parts. fixed memory overhead and the burst memory overhead. And when we applying resources for our executor part from Kubernetes, we use on heap space which is a fixed part and plus the fixed memory overhead uh to get a res resource allocation and then the burst memory overhead will be shared among all the parts in the same host.
The assumption is that it's very rarely for order pots in the same host to ask for max size of memory overhead at the same time. So we can share some kind of time sharing some memory uh space in the host. Here is the example. Uh by default uh spark just sets the same value for request and the limit. And uh if we have 100 GB of memory space and we have a three execut uh allocate three parts. First one needs uh 50 GB, second 40 and the third one 10.
But if we can split this memory request this kind of resource request into request and a limit part for example the first one only need a 30 GB to be allocated. Second one need 24 and the third one need need a six and we can allow this part to use as max as the original size. For example, the first one uh you need a 30 gigabytes to be allocated but you can use uh as much as 50. Then we can use the same size of memory space to host the more ports.
For example, in the bottom of this example, after we allocate the original three parts, we can split another port into into this host. Uh for example, that's also six six gigabytes at request and 4 GB as uh limit. Oh, sorry, 10 gigabytes at limit. So that will save us a lot of memory memory memory space to uh allocate to more spark executor uh parts. After we applying this uh algorithm to our cluster uh we we thought it benefited a lot about our resource problem.
For example, here after we launched Canon uh for this uh canon in this cluster, there are less opportunities for our cluster to be overloaded. And even when the cluster is allocating 100% of memory space to all the Spark executors, the gap between the allocated and the utilized memory is way smaller than before. So this is the second uh platform innovation we have burst aware algorithm b aware memory allocation algorithm in spark and the next part is fine grand cost management you may ask uh you have made a lot of improvement in in your platform why you still need to have such like a fine grand cost management that's because some of our spark jobs are just so creative ical and so expensive.
Even we apply those canon the burst aware memory uh algorithm we optimize it sharper performance it's still like takes our millions of dollars to run that to run that spark job every year uh and for example three out of our tens of thousands of daily spark applications actually contribute 2% of our o overall data uh data computing budget and uh the other case is that our customer can be in a very like uh urgent project uh they need to run uh like a many experiments every day but they don't have enough bandwise to manage the cost at the same time that's they also need our help to have some fine tuning or fine grand management for their cost they just focus on business implementation so uh we have some like a powerful tools uh for fine-tuning the first one is look at the memory utilization uh this is part of the real-time metrics that we collected from different metric source. uh in this case we look at the memory utilization we find uh even the customers uh spark job executor had used 100% of allocated memory space but the GC is pretty lightweighted in there that means we can probably let GC JVM GC to do more job and squeeze uh the allocated on space more so that we can save the And uh the second one is uh uh use the spark UI thread dump.
Uh when we have a running spark application uh we can always use the thread dump in the spark UI and keep refreshing that and for the thread dump frequently appear in uh every time that must be the bottleneck implementation. So here is the example. Uh I mask the internal code but basically the idea is that we find that our some internal code in the most expensive job in our system repeatedly create the J JVM uh Java array that makes the job pretty slow.
So with the on space tuning based on memory utilization st and the bottleneck code optimization. Here is a example about how we optimize theize the job. We can see that we can reduce about half of the resource consumption for this job after we tune on keep space and after uh after we optimize the bottleneck implementation we can save more uh about the another 30 to 40% of cost saving there. So next problem is uh uh we cannot just assign engineers to tune every job uh every day for our customers.
How we can scale this practice? Uh the problem here is uh our users are not really interested in saving the cost. Uh they have so many urgent task to finish. They they need to run their business logical successfully and um uh there are also many like many complicated operations needed for like a capping the cost and sim uh at the same time meet the SRA of every job. So how we do this? We are basically developing something called uh management job set.
The customer just give the input to us as the total budget uh SR of each job and the set of jobs sharing the sharing a trunk of budget and we can provide them the initial optimal configuration for their job and then we uh can do some automated or semi-auto automated cost monitoring and tuning for their jobs. uh this is something we are we are currently working on with our customer. So summary of this talk uh uh we have our solution to address the cost efficiency challenges uh in our imprint is basically in three layers.
First we improve the cost and performance observability uh and we build a comprehensive observability platform for uh for with the real-time matrix and alo event logs to help us to address to surface different root cause of the uh of the cost issues and we build a general solutions in platform wide. Uh for example, we optimize our shopper performance with the remote shuffle service and we save our cost allocation uh with the burst aware memory allocation algorithm and for those very very expensive and critical jobs.
We apply the fine grand cost management strategy and to scale this practice we have man we are developing manage the job set to man to provide the manage the experience for cost budget for our customers. So yeah uh that's pretty much about my talk. Thank you very much. Happy to take questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.