High-Throughput ML: Mastering Efficient Model Serving at Enterprise Scale

Summary

• High-Throughput Inference Architecture: Build inference pipelines on the Databricks Data and AI platform that efficiently scale to handle thousands of ML predictions per second while maintaining low latency.
• Feature Store Optimization: Leverage Feature Store for consistent, low-latency feature lookups essential for real-time predictions in recommender systems and fraud detection models.
• Scaling and Monitoring: Implement auto-scaling strategies that optimize both performance and cost, with monitoring systems that provide visibility into inference performance and latency optimization.

High-Throughput ML: Mastering Efficient Model Serving at Enterprise Scale

Talk by Mingyang Ge, Software Engineer, Databricks ; Yucheng Qian, Staff Software Engineer, Databricks
Ever wondered how industry leaders handle thousands of ML predictions per second? this video reveals the architecture behind high-performance model serving systems on Databricks. We'll explore how to build inference pipelines that efficiently scale to handle massive request volumes while maintaining low latency. You'll learn how to leverage Feature Store for consistent, low-latency feature lookups and implement auto-scaling strategies that optimize both performance and cost. Key takeaways: Determining optimal compute capacity using the QPS × model execution time formula Configuring Feature Store for high-throughput, low-latency feature retrieval Managing cold starts and scaling strategies for latency-sensitive applications Implementing monitoring systems that provide visibility into inference performance Whether you're serving recommender systems or real-time fraud detection models, you'll gain practical strategies for building enterprise-grade ML serving systems.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and approach
• 3:22 Model training and setup
• 5:15 Demo: Training and deployment
• 9:14 Endpoint configuration
• 12:35 Load testing and performance
• 23:29 Recap and conclusion

FAQs

How can I achieve thousands of predictions per second?

This video reveals the architecture behind high-performance model serving systems, explaining how to build inference pipelines that efficiently scale to handle massive request volumes while maintaining low latency.

What is the QPS × model execution time formula?

According to this video, determining optimal compute capacity requires using the QPS (queries per second) × model execution time formula to calculate the resources needed for your serving infrastructure.

What database powers online feature serving for real-time fraud detection?

The video demonstrates that online feature stores must be co-located with model serving endpoints. Feature tables published to online stores enable sub-millisecond lookups during inference, critical for fraud detection and personalization systems processing thousands of requests per second.

Do you need a feature store for production machine learning?

This video shows that production ML systems require feature stores to separate the features you need at training time from those available at inference time. Without a feature store, you cannot reliably serve consistent features for high-throughput prediction workloads.

What makes a feature store enterprise-ready for ML serving?

The video describes how Databricks' feature store integrates model training and online serving through a unified API that binds features to models. This automation eliminates manual glue code and ensures models automatically know where to fetch online feature tables at serving time.

Full transcript

0:00 Introduction and approach

Good afternoon everyone. Uh welcome to this session. So this is a high throughput ML session and this is Mang. Today we're going to walk you through how we can do high throughput ML with data bricks and the subtitle of this talk is well mastering efficient model and feature surv. Yes. So we're going to talk about both model training and the feature serving at enterprise scale which means IQPS. Um so with ML there is a lot of uh lot of infrastructure involved and for this particular talk let's talk about one very let's oversimplify this a bit to see what are the problem we are trying to achieve today is saying we have a classification model are very difficult and uh how can I achieve a highqps that's the goal of this talk right you have your models you have your features and how can I serve that with very high traffic and okay so what kind of a model do we have today cool thanks um thanks that's a great question um how do we achieve 15k QPS um I will take an example to um demonstrate how do we achieve this goal um we have a a very typical but uh very like uh very typical but uh uh very like normal use case we have in production.
Um let's imagine I'm a fintech company and I'm a client of data bricks. Um am I I'm asked to solve this fraud detection question. So this is a very simplified overview of a typical setup we have. Um imagine this is a component in your whole like transaction processing system and we're we are asked to detect some of the transaction if they are fraud. Um imagine you have client sending us request um we have a machine learning model returning if the if the transaction is a fraud.
Um that's very simple but normally in production use cases we don't have all the information in the request. So we also need help from the online feature store to return all the feature values we have from the online database. So this is simple. Then how do we um achieve 15k QPS? That's the main question we have. Um okay so uh this is the approach we are going to take. There are couple of steps. First of course we need to deploy this model um with help of feature store.
So the model and the features should be connected and it should al also be hosted together. Um and then we will need to do a scale testing to see how do we scale this entire system um to serve the anticipated traffic we want. Through the low testing uh we will know which part should we scale and how much they should be scaled and then we are uh we can productionize our product with confidence. Okay, so first step, how do we uh train our model with features?

3:22 Model training and setup

Data bricks provide us a very simple API that can um help us connect the model with the feature. Um it's only needs three steps. First thing is we need to define a training set. So this is all done through the model training process. So you have a um a feature engineering client that's the API data bricks provide us and then um I can create a training set with some feature lookup. So the feature lookup is the reference to some feature tables. um the table is stored in unique catalog and I also need to specify some details like what's the primary key for lookup and uh in this training set what labels uh what columns are the labels and then second step is log this model it basically store this model in unit catalog and by logging this model we combine the model and the feature so the model knows where the feature is and it automatically knows like if the model is served online, where to look up this online feature.
Um, and then the third step is to publish the feature table to online feature store. Um, you may notice that this is the only three steps we have, but we don't need to tell the uh models like deploy the model where to look up from the online table. Um, it's all connected. That's a m like the magical part. It's all connected together. data bricks model serving will automatically know where the online table is because all the information is stored in this training set when you log the model.
Cool. So now we have a model and all the features supported. Let's jump into some code. Um cool. So in this example, I'll just show you the steps I took previously for training some models and then uh we will have you help like doing the scale testing to find out how do we scale the system and then we'll see some results. Cool. First step is training the model. Um first thing we should look at what what data we have.

5:15 Demo: Training and deployment

So in this example I have two feature tables. One is a user feature table um and another is merchant because we're processing some transactions and then we also have some label data. Um let's to let's take a look at the table. Um data is always the first. So user table have some user ID um some features there are aggregated features and time stamp time stamp can be used for point in time joining um and merchant table is also very similar.
It has features ID and time stamp. And then this is the label data. We have one of the column is uh like the label which contains a boolean value. Uh we also have some other features. These are not stored in feature store. They only exist in the real time traffic. Um this is very typical. You cannot have all your features pre-processed and stored in feature store. Um this is all taken care of. Cool. This is a step we have in the previous slides.
Basically, you create the feature lookup reference the feature table and then we have a training set. So there are a couple of steps to create to train the train the model um in like I'm taking a very simple XG boost classifier for this example and I train it with the help of the training set. Now we have the model prepared. we need to store it in unit catalog. Basically, I call this log model and pass the training set into it.
At this point, the information of all the features are stored together with the model. They're binded together. So, the model will work automatically both online and offline. They know uh where to find the feature tables. And then the last step is like to make the feature table online. um this the this is the last step to make the the model ready for serving. So how do we do that? It's also very simple. Um we have SDKs but we also have UIs which is what I want to show you.
So this is a feature table we have. You can see the schema matches what we have and then basically you can go to create and create online table. So this is a small like spoiler uh what we are going to launch soon. Um if you want to learn more you can come to our keynotes tomorrow. Um but basically it's a brand new online table we're going to launch. Um but for the example today I will just use the OG online table. For the creating a long table we'll just give it a name.
Uh for this one I will use user feature online. You can see the primary key and time stamp keys are prepopulated. Just click confirm and then oh it's already exist. So let me try another one. All right. Now it created online table. You can see we are provisioning some resources. These resources are pipelines that we use to copy the data from offline to online. Cool. Now this model is ready. We have all the model trained and the feature table published to online.
Next step is serve and scale testing. Thanks M. All right. So now we have the online feature store ready. Right. And we have also registered the model as well which is XG boost fraud detection classifier. Very simple. As you can see that from the UI is very easy to start serving the model. Okay. So we can click on this on the right side. uh serving the model and uh it will bring us to the creation form of ed serving endpoint.

9:14 Endpoint configuration

You might have seen this already uh but let's see uh what we would show. Uh you can give it a name and the model and the version has been pre-selected and this is going to be a CPU model right the next one or the important pieces is scale out. Right? So we can see that we can choose different concurrencies. You may be wondering like how do I know which one to select right so we say it's a small medium large what exactly should we select and how do we choose the right one to meet our goal of 15k QPS right that's the problem uh well we don't know that yet because as you can see the QPS highly depends on the model performance itself right if the model takes a long time to make a prediction then naturally it's not going to reach a high QPS and unless you give it a lot of currency So what we can do is the key thing of this is we need to perform load testing of the endpoint and understand what the model takes how long it actually takes to make prediction.
So we want to start with small right so this is the mo smallest scale we can choose. So the theory is that if we know how the endpoint will perform with a scaleless scale then if we want to increase that we just horizontally scale out right. So you see that if four concurrency is the smallest if we want to give four times of that then we would choose say a medium or even higher. So it depends on the multiplier based on the smallest scale how the model performs here.
So we're going to do the perform uh uh load testing here on this scale and you can see that scale to zero was uh enabled by default that is usually for experimental use cases and if you want to save cost and for low testing we don't want to that right we want to have the model working and we're going to stress test it. So we're going to uncheck the scale to zero option choose the smallest scale and the next important piece is enable route optimization.
So this is for model solving a particular mode that we will eliminate any unnecessary hops during the network. So this will be the ideal expected behavior when you have a very low latency and high through use case. So we want to enable the route optimization. That's pretty much it when you config the endpoint. Right? So again we choose the smallest scale. We disable scale to zero and enable route optimization. And then we're going to create the endpoint. that in the interest time I have previously created this endpoint already.
So assume this is the endpoint that's created. As you can see uh reemphasize is the smallest scale. It does not have scale to zero. And here it has a very unique URL for query. So this is a route optimized URL. Okay. So then next step we're going to load test this end point right that's a key right we have a we need to know how this model performs. Um so this is a bit of something uh we're going to roll out yet very soon.

12:35 Load testing and performance

But we see on the right top right corner we have a load test button right we're going to make this very easy accessible we have prepared a load testing notebook how to perform load tests on endpoints we click on that and yes it will generate a notebook with necessary information that to perform this there's a lot of instructions as you can uh imagine so I'm going to start choosing a cluster and actually I'm going to close that um we have uh we're going to have a very detailed documentation page as well just to walk you through to understand the different concepts but for this talk let's just uh quickly go over the things I would be important to note uh okay first I'm going to start installing some of the dependencies right but the key thing is the notebook will instantiate using the locust framework to instantiate many concurrent users to send a request to the endpoint we just created and it will measure the performance, the latency, the requests per uh RQPS so that we know how this model performs.
So I have instant uh installed the dependencies and I'm going to keep going importing some of the item uh libraries and here is a configuration about the endpoint itself but they are all prepopulated because we it knows which endpoint we are created this endpoint for so the uh one thing I want to notice that yes we need a a client to authenticate with the endpoint and we use h we can use a service principle here which is our recommend mented a more standard and a secure way to query the endpoint.
I have previously created a service principle with their secrets and clients. So I'm going to refer them here. So this I have created this beforehand. Okay. So this means I'm going to use this identity to query the endpoint and we're going to run that. Keep running this. So that's pretty much the only thing we want to specify and we have a lot of uh helper uh functions in included as well. So first thing is we're going to run very short test right.
So you're going to run it for 30 seconds to warm it up to see how this looks. So if you run that as you can see this is the locust framework and it will create a UI if you want to monitor the load testing in progress. So as you can see that it keeps increasing the requests and measuring the latency and which shows this in real time graph as well. You can see the cube RPS or requests per second keeps increasing as we uh slowly to ramping up the number of concurrent users that's sending the request.
So here we are simulating a real use case or traffic load to see how this model endpoint is responding. Right? So in the end we reached about like 30 or 29 concurrent users and uh we reached about 240 RPS. As you can see the higher the more users concurrent users sending request naturally the latency goes up right so there is a bit more queuing happening. Okay, so that would be the uh warming up of the endpoints and the next we will see that so one thing we want to know is for the smallest scale.
So imagine if it has a one replica, you can still send as many concurrent requests to it as you like is except that the more you send to it, the high uh the more requests will be queued up, right? So the latency will go up. But it also that means you can have a higher you can handle more higher concurrency if you can tolerate a bit higher of the latency. So here is a trade-off is if you have a target of the latency then maybe you can tolerate more concurrent users.
What we want to do test next of this part is say if I'm give it a range of different concurrent users sending requests to the endpoint what results do I get? So naturally would expect the more current users we send to it the higher the latency is but maybe at a certain point even if the latency increases it may be still within our target right that means could be acceptable. So this request will do that and then see how this endpoint performs and the different concurrent user settings.
Uh this is this test is going to take a while. So every number will take like a five minutes or so. It's going to take a total of 45 minutes. I'm not going to do that here. So I'm gonna I have a jump to jump ahead of time. Right. So this is the results I have previously run on a similar endpoint and see what we end up seeing is yes given the different user accounts on the small endpoint. We will see the different requests and the different average response time and their distribution as as expected.
So maybe a better presentation is this graph form right naturally you can see the more concurrently users the latency starts to go up here is where we can make the decision uh what is our target here if we look at this table form for say for example for P90 right so it go as low as 26 milliseconds or up to almost 50 milliseconds if we say 50cond is acceptable. That's our target. That means we could allow up to 10 users.
We could accept that, right? So what the next step of the notebook is say if this is our target and then so in this case I'm going to choose 10 that represents this row. This latency target is what we accept and this is our say expected request per second. How much concurrency do I need? So now you see this is the calculation. I already know the small scale how the endpoint performs and I have a target number. Now I calculate how much scale I need.
So the notebook will do the perform uh calculation for you. Uh it basically gives you a recommendation right. So basically based on the previous results and your target you will say this is the concurrency you probably want to look at. So certainly this is not absible number. It's a starting point as a guideline. As you can see this notebook you can change any part of it. If you want to change even uh different uh concurrencies or different user accounts feel free to do so.
And this is the point of the no testing. You want to try different parameters to see how this endpoint performs in but this one will give you a starting point to say if I want to use the concurrency for 324 right how do I do that now I would go back to the endpoint and then I edit it right so it's a small I to go edit it and now I know what scale I need to choose that's the purpose of the load testing And I in this case is none of them right it's not small medium or large right so this is the thing we're actually going to roll this out very quickly but you may have experience with this before how do I specify customer page or concurrency we're going to have a custom option there and let you type what you want that's it right so you can type 384 uh actually that's right so then we also choose this 34 right if I'm continuing doing the load testing.
I could enimulate the autoscaling by specifying the same minimum and max concurrency so that I know I'm always going to have this many parts running, right? So, they're going to do that. But in a more production scenario, you may change that depending on if your traffic pattern is day of the day of the time of the day pattern, then you can have a different minimum and a maximum. So, it will autoscales. So in this case if I'm continuing load testing I know the concurrency I'm want to test next I specify here update the endpoint uh I'm not going to do that but just say if you update it it's gonna yeah update the endpoint to that concurrency and then you would to go back to the notebook you say now I changed my endpoint I'm going to run the test again right that's basically this loop here I know this is I know what I'm testing the concurrency and I'm going to see how much traffic can get out of it and here yes so pretty much the same thing is going to run the locust again for a while and this particular results let's see what we get from uh last time I run it right have a low test report in the end it will give you how much request is send like a failure rate what most interesting thing here is the charts right so what do we end up achieving for uh the model is Okay, we actually did get to like say this is a 50k QPS 50,000 that's our goal and then that achieved about like 29 milliseconds on P50 for like still less than 50 milliseconds on P95 it's pretty good right but a more interesting thing is yes this is a starting point you can keep stress testing it right if you want to go higher I say so naturally the response time goes up by let's see in the end what do we end up doing is almost 50,000 users sending requests to the endpoint we got like a 20k QPS that's what we got as a high highest we got with the more certainly higher latency so here is again depending on your particular use case you want to decide what latency you want to tolerate or expect right because certainly the more concurrency the more cost it is right so there's a cost trade-off, the cost, the concurrency and the latency target.
So those are the ones to consider, but this notebook or the no testing helps you to decide what you why you're making these trade-offs and how. Okay, so that's the end of this particular demo on no testing, right? Let's go back to our uh presentation. Okay, cool. Thanks. So yeah, that's great. It's a lot of content, but we got a really good result. We um set up a goal for 15,000 QPS, but we end up reaching 20,000 QPS.

23:29 Recap and conclusion

Um let's do a little recap on how do we scale this system. So basically there are two components um in this serving stack. One is online feature store, another is model serving. So online feature store this is also a little like spoiler um we are providing some new options for you to tune the capacity of online feature store. So you can calculate um like it's it's both like horizontal scaling and vertical scaling. So based on the data size you have we have the in-memory cache that can improve a lot of performance.
Um and that you can get the capacity unit to specify and then uh for to serve the anticipated traffic we need we need to scale the online feature store and this comes in with the readable secondaries. So if you read from more secondaries uh each one have a like a fewer load. The second part is model serving. Um as we showed in the demo, you can enable router optimization. This give you the best network latency we can provide.
Um it eliminates a lot of the network hops within the system. And then selecting the proper concurrency is crucial. Um that gives you the proficient capacity for serving the anticipated traffic. Uh and the formula we use in the notebook is quite simple. is just QPS times the execution time aka the latency of the model. Um and uh yeah basically with this formula we can calculate how much concurrency we want. Um and then just to remember that in production uh the traffic will not stay flat in most of the case.
So if uh you are anticipating some like high traffic spikes you can tune that uh beforehand so we can capture all the all the requests that we want to serve. Um yeah so basically that's u that's what we can scale. Mhm. Yeah. And well, this is pretty much the almost the end of our presentation. Before we get to the point uh quite Q&A, right? Some key takeaways to summarize. It's very easy with data bras to deploy model and the feature endpoints, right?
Simple as three steps, right? Create that training data set, lock the model with the training data set, and then publish to the online feature store. So now the model and the feature connection is established. And now with the data bricks, you can also deploy with the confidence, right? So we're going to provide you the out of the box load testing environment to help you guide you through every step of the way to determine how you can achieve the target and we have a very scalable infrastructure underneath as well.
So uh we are constantly improving our infrastructure. So we showed a 20k example results but we actually can go way beyond that like as high as 250k QPS. that's not a problem and they can go even higher on concurrency for a thousand concurrency if that's what you need your enterprise desire needs we can provide that as well so again as the improvement is always underway and in particular want to just plug in for tomorrow come to tomorrow's keynote because we're gonna have some interesting announcements very exciting ones about the next steps for online feature store and particularly how online feature store can support this very high QBS uh workload as well.

Learn more about the Databricks Data and AI platform.