Scaling Custom LLMs with vLLM and Databricks Model Serving: Fast, Flexible, and Production-Ready

Summary

This video walks through deploying custom and open-source LLMs on Databricks Model Serving using vLLM as the inference runtime on serverless GPU endpoints, enabling high-throughput inference for models not covered by Databricks' built-in provisioned throughput options.
The deployment workflow involves downloading a model from Hugging Face, packaging it with MLflow, registering it in Unity Catalog, and configuring a custom GPU endpoint — demonstrated live with an A10 GPU running a Databricks-specific question-answering LLM.
Serverless GPU compute on the Databricks Data and AI platform simplifies the operational complexity of GPU management, batching, and caching that has historically made custom LLM hosting challenging, with vision model support also available on custom GPU endpoints.

Scaling Custom LLMs with vLLM and Databricks Model Serving: Fast, Flexible, and Production-Ready

Databricks Model Serving supports deployments ranging from classical ML models on custom-CPU workloads to foundation models with dedicated provisioned throughput endpoints. But what about the use-cases that need any of the 1000s of other open-source or fine-tuned LLMs? Deploying them efficiently can be challenging. This breakout session explores deploying LLMs on custom-GPU endpoints with vLLM. In it, we’ll examine: Serverless GPU Compute: How it simplifies the deployment process, saving hours on configuration work alone. vLLM + GPU workloads: How they work in tandem on the platform to deliver high-throughput inference with scalable infrastructure. Implementation: A step-by-step code walkthrough for packaging models, configuring vLLM runtime, and deployment! Serving LLMs on GPUs doesn’t have to be scary. Learn more about how Databricks enables you to deploy even your most demanding model serving workloads!
Talk By: Colton Peltier, Senior Staff AI FDE, Databricks ; Mohamad Aboufoul, Senior AI FDE, Databricks ;

Chapters

FAQs

What gap does custom GPU endpoint support fill in Databricks Model Serving?

Databricks Model Serving already supports provisioned throughput for popular model families and CPU endpoints for classical ML, but there was no straightforward path for deploying custom or open-source LLMs outside those supported families. Custom GPU endpoints with vLLM fill this gap for any of the thousands of available open-source or fine-tuned models that teams want to host themselves.

Why use vLLM as the inference runtime for custom LLM serving on Databricks?

vLLM is an open-source inference engine that handles the complex aspects of LLM hosting — including request batching, KV cache management, and GPU memory optimization — that make production-grade serving difficult to implement manually. It runs as the inference backend for Databricks custom GPU endpoints, providing high-throughput performance without requiring users to manage these internals themselves.

What is the step-by-step process for deploying a custom LLM on Databricks?

The workflow involves downloading the model from Hugging Face, running local validation with vLLM shell commands, packaging the model using MLflow's log-model API, registering it in Unity Catalog, and then creating a custom GPU endpoint through the Databricks Model Serving interface. The video demonstrates this end-to-end with a live inference test on a Databricks-specific question-answering model.

What are the current limitations of custom GPU endpoints on Databricks?

The video notes several known friction points including timeout configurations, correct file path handling for model artifacts, and a single-replica constraint that was in beta at the time of the demo. These are described as paper cuts that the Databricks model serving team is actively working to address in future releases.

Full transcript

[00:07] So, thank you for attending this talk. Like the title says, we're going to be talking about hosting models with vLLM on Databricks model serving. Show of hands, has anyone here used vLLM before? Got one person. Okay, awesome, too.
[00:21] Cool. Well, we'll go through this. We'll do a brief demo as well, and hopefully it's pretty informative. Cool. Hey everyone, my name is Colton Peltier. I'm a senior staff forward
[00:34] deployed engineer here at Databricks, but I also work with the model serving team like sort of like on the side. And they have some new functionality I wanted to talk about today, which is really exciting. So, I want to talk a little bit about why we're talking about this, right? So,
[00:47] LLMs can be hard to host. If you've ever tried to serve your own LLMs, spinning up one maybe is a little easy, but actually like getting it to host in a production grade way can be challenging. So, you run into things like caching and batching,
[01:02] GPU management, it all gets very complex very fast. There's some open source technology like vLLM, which really helps here. But again, you know, like that's not easy to serve on Databricks up until now. So,
[01:15] the second thing I'll talk about is there's a different playing field from 3 years ago, right? Like LLMs are launching constantly. There's certainly a big focus now on switching back into cost optimization. So, there's big needs that each team has
[01:29] around serving very specific LLMs for their use case. Not everybody wants to throw everything at Claude Opus, right? Like people might want to use a small fine-tuned model. Databricks today does support provision throughput for supported families of
[01:42] models, which is really helpful. It'll It'll serve a fine-tuned variant of that model for you. But if you have like a smaller open source model or one that you created yourself, there currently was no real good solution to putting those on
[01:54] Databricks. You go to the next slide. So, for those unfamiliar with Databricks AI model serving, I'll talk a little bit about what it is. Um right now it serves two different
[02:06] variants. You can choose when you're deploying your model if you want to support support CPU endpoint. So, think like small models, maybe SK learn, things like that, not LLMs, or GPU endpoints.
[02:18] Um GPUs are obviously useful for things like LLMs, but also for models like embedding models. Um we do support the foundation model API. So, this is sort of like a chat API that can talk to these models. We have
[02:31] paper token models for some of the most popular models out there, like Claude and like llama. Um and we support provision throughput for all those families of models if you fine-tune.
[02:43] You get management, governance, and scaling out of the box, um which is fantastic, but again, the the missed need here was if you have a model that's not supported by provision throughput, you didn't really have any way of
[02:55] getting that put onto Databricks model serving and getting all that management, governance, and scaling built in. >> All right. So, I'm sure everyone here knows that typical model serving is very difficult. There is a lot that goes into
[03:08] it and a lot that you have to maintain um and worry about. And LLM serving is significantly harder. Everything that you have to handle with traditional model serving does not go away, and then
[03:21] you also have to worry about GPU infrastructure, uh long times, configurations, um and making sure that the uh model that you're using is one available and two usable, be it an
[03:33] open-source model or a fine-tuned model that you fine-tuned or even a model that you trained from scratch. But, there are benefits to uh hosting your own uh LLM, uh especially if it's a small one for a
[03:45] specific use case. You have a plethora of options options, especially in this day and age. Uh Hugging Face alone has a lot of open source models available for various use cases. You have more control
[03:58] when you host your own LLM as opposed to just relying on the proprietary ones, which could change be it in model and quality of the model that comes out in
[04:12] the cost structure as well. And you also have the potential to save a lot of money when you choose to use a small model which has as we've seen in various use cases can often outperform a
[04:26] proprietary model in a specific use case. Again, we are going to go through a brief demo, but before we do so I wanted to cover the generic or high-level architecture.
[04:39] What we'll be doing in this example, we will be downloading a small model from Hugging Face. Once we get the model weights into Databricks, we'll be logging the model artifacts into MLflow
[04:51] along with the configuration for the vLLM server. Once it's in MLflow, we will register it to Unity Catalog. This will serve as our model registry. And once it's there, we will deploy to a serving endpoint
[05:06] through which we will have the custom GPU compute and this will be accessible to users through the APIs that Databricks offers.
[05:18] Why do we use vLLM? Because it's a pretty good inference runtime. It is pretty robust, pretty comprehensive. It's great for GPU backed high
[05:30] throughput serving. And you can again as Quentin mentioned serve models outside that standard catalog that Databricks offers for proprietary and the large open source ones that just take a lot of compute to
[05:43] serve. The vLLM uh essentially becomes the the engine inside this custom endpoint. So, let's get into the demo.
[05:59] Uh what we first want to do is make sure that we are on serverless uh GPU, and we want to select an A10 uh GPU uh to support this. You can also just go into configuration and select the accelerator
[06:12] that you want to use. This is taking a while to run, so uh I'll I'll let this run in the background, but walk through the rest of the notebook. Um but the outline here is essentially just showing what or replicating what we
[06:26] discussed in that architecture. Um so, first we need to set up the environment. We need to make sure that we have the right uh compute. We have to install a couple of libraries, and then we have to
[06:38] uh have a working directory for some local work that we're going to do before we deploy. Um I know this seems like a lot. This is in beta, so there is uh a couple of configuration steps that have to be handled, but don't be intimidated. It's
[06:51] uh pretty readable and uh straightforward. Um so, with regards to the configuration, we have a couple of items related to the model that we're going to select. Uh in this case, we're downloading from Hugging Face, so we
[07:03] just specify the location, and then we specify some like local path we want to save it to, uh as well as what the model endpoint name will be once we serve it. Um we're going to define a couple of
[07:17] uh elements that are relevant to the model. So, in this case, it's a floating point 16, has a max length of about 16K, and we're going to use a GPU utilization of about 85%. Um these are some local ports that are
[07:29] necessary for the VL OM. Um and then this is going to be where we'll store it or register it in Unity Catalog. And then these are related to the endpoint itself within uh Databricks Model
[07:41] Serving. Um I'm not going to run this right now since it seems like our uh workspace is being bombarded with uh a lot of different requests, but um essentially what we're going to do here or what we would do here is just make
[07:54] sure we've set up the configuration, uh download the model locally, which I thankfully did before this talk began. Um and then, once it's downloaded, we're going to test the uh functionality of
[08:06] this model within our notebook. It's really important to test it locally because you don't want to end up spending half an hour or more just waiting for this endpoint to build and
[08:18] then find out that there's something wrong with it or constantly like go back and forth trying to redeploy several times. Um this is going to save you a lot of the headache. Just make sure that this is working locally uh as expected.
[08:30] Here, these are just the commands that will be run, the shell commands essentially, to set up the VLOM uh server. Um we'll set it up. See that it uh was set up right here. We
[08:42] can do a test uh request. This is a simple hello. See that it responds. And then we can ask it something a bit more complex. We can say, you know, "Tell me a story uh that's about 300 words." Seems to do the job right here.
[08:55] And then we close the server uh once we're done just to, you know, not keep it running locally. Um now that we've confirmed that, we can go straight into uh the registration or the logging and
[09:08] registration process. It's pretty straightforward for MLflow. We just use a log model. We store the artifacts of the model or rather the path to the artifacts and uh then the entry point
[09:22] and some extra requirements as necessary. And then registration is just a one uh line command. We take the MRI, the piece from MLflow, register it. We can see it
[09:34] right here. We did this in advance. We have a couple of versions that we've run before. This is the latest one that we've registered into Unity catalog. And then we can also uh basic well, sorry, right here we're going to uh do
[09:48] the actual deployment piece. Um and we have to do this programmatically since it's in beta, but again, pretty straightforward. We just define the configuration with the Databricks SDK, um make the endpoint, and uh if it's not
[10:01] already made or if it is already made rather, we um update the existing one. So, uh that process is going to take 20 to 30 minutes. Um and once it's done, we can then hit the endpoint itself uh to confirm that
[10:16] it is working programmatically. Uh same as before, but we do have one that's warmed up already. So, why don't we try it out? Just a simple hi. You can see that yes, it is working. We can try something a
[10:29] bit more complex. Ask it what is Databricks Data and AI Summit.
[10:57] Major industry conference hosted by Databricks. Perfect. I'm going to assume that the rest of this is accurate, but this is a small model, so it could have gotten it wrong. Um but that that's essentially it. It's it's pretty straightforward, pretty simple to implement. And you are not
[11:11] limited to just text models. You are limited in this beta to chat completion models, so we can't use embedding models, but uh chat completion includes a plethora of models including uh vision
[11:23] ones as well. So, I don't have the time to run an example for this, and I don't have the endpoint set up, but I did try a coin model that was a vision language model. Apologies for the orientation of this picture.
[11:35] Um a year or two ago, I went to a couple years back actually, I went to a Real Madrid game in my hometown of Charlotte cuz I'm a big fan for about 20 years now, and I wanted to see if this Quen
[11:47] model could actually tell what was in it. Um so, the score is 0-0 between Chelsea and Real Madrid. Game just started at 4:15. And I have run this before.
[12:00] You can see right here that it does make an attempt of parsing this this this the contents of this image. It does correctly find that it is Real Madrid playing. It
[12:12] incorrectly thinks that Atlético Madrid is playing against it, and it also incorrectly thinks that 4 minutes and 15 seconds are remaining, but that's what started. But point is that you can actually get models like these up and running.
[12:24] There will be some slight differences in implementation, particularly in that you have to convert your language model into binary essentially into base 64 rather.
[12:37] So, you just have to programmatically include some kind of encoding helper functions, and once you do that, you take your image itself,
[12:50] convert it, and then you can add your prompt as you'd like. Prompt I'd asked was what's in that image, and you essentially see that the payload is a little bit different, but point still stands. You have a lot of options to
[13:03] work with here. All right. >> Awesome. So, let me talk through a little bit about like what are the paper cuts you might face during this beta. Um always check the paths that you're
[13:15] saving the file to. If you save it to a temporary file path, it may not be accessible um when you're downloading your your actual model from the internet. Um check permissions. If you download something and someone else tries to use it, there may be permissions issues. So, always
[13:27] double-check that. Um in our notebook examples, you might have noticed that we were using the the Databricks SDK for deploying the model and waiting for the endpoint to come up. There is some default timeouts in there.
[13:39] So, the default timeout is 20 minutes to wait for the endpoint to come up, but with these bigger GPU models, they might take longer. So, double-check that and when you look at the example notebooks that are provided in the beta, they will have this in there, but again, just uh
[13:51] make sure that you uh time your timeouts properly. Um again, and the other piece is it only supports one single GPU serving right now for one H100. Um so, we kind of give you a reference there. The biggest model you would probably use would be something like a 120 billion parameters.
[14:09] And so, just to kind of go back through one last recap, I think this is going to be most applicable for people who are trying to serve small models that are specific to their team's needs, that they want the um the good stuff that comes with Databricks model serving, right? They want the scalability, they
[14:21] want the auditability, they want the governance. Um but maybe that model's not supported in the Databricks provision throughput families. So, this will really help them out. Um the second piece is it's not a single quick setup
[14:33] yet. We're in beta right now for this and we'll show you a QR code in a moment of how to sign up for the beta, but uh it's, you know, you do have to run the notebook right now. There is some configuration needed. >> All right. >> And finally, yeah, the good news is that
[14:45] it is doable now on Databricks to serve your custom LLMs on GPUs. >> Right. Uh one clarification, this QR code is going to take you to the documentation and uh sample code. If you want to sign up for the beta, just uh
[14:57] ask your workspace admin to enable it. Um and sorry to put you on the spot, Cyrus, but I want to give a huge shout-out to the engineer who set this up for us, uh Cyrus right here. Um wouldn't have been able to do this without him, so uh big thanks.
[15:11] >> All right. Well, thank you so much for your time, everyone. If anyone wants to stop by afterwards and talk, just let us know. >> Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.