Mission-Critical Inference: Powering High-Scale AI in Production

Summary

• This video explains the inference scaling challenge on the Databricks Data and AI platform — how growing token demand requires efficient multi-tenant GPU serving with sophisticated routing, admission control, and autoscaling to manage costs.
• Databricks Model Serving uses token- and cache-aware routing, prefix caching with a multi-layer cache hierarchy, and a cache-aware load balancer called Dicer for session-sticky routing, particularly benefiting agentic workloads with shared prompt prefixes.
• High-throughput batch LLM inference is available through the ai_query function in Databricks SQL, bringing model capabilities natively into data pipelines, while custom model serving with VLLM supports any transformer-based architecture.

Mission-Critical Inference: Powering High-Scale AI in Production

The explosive growth in token consumption has made deploying LLMs at scale increasingly complex. Teams must balance latency, cost, and reliability. Databricks Model Serving is designed to absorb these operational challenges across the full model spectrum: proprietary APIs, open-source foundation models, fine-tuned variants, and fully custom architectures. In this talk, we'll go under the hood of how we built token- and cache-aware routing, intelligent scheduling, and autoscaling to deliver efficient, resilient inference at scale. We'll then show how we extended this infrastructure with high-throughput batch support through the ai_query function in Databricks SQL, bringing LLM capabilities natively into data pipelines. Finally, we'll explore how we enable customers to serve their own post-trained/finetuned models or custom transformer-based architectures. Whether you're optimizing costs, scaling to millions of requests, or bringing a custom architecture to production, this video offers a clear picture of what production-grade LLM serving looks like end to end.
Talk By: Brian Law, Sr. Specialist Solutions Architect, Databricks ; Mike Eastham, Senior Staff Software Engineer, Databricks ;

Chapters

• 00:00 Introduction: the inference scaling challenge and token cost growth
• 01:46 Why open-source models are essential for cost-efficient enterprise AI
• 03:14 LLM vs. classical ML serving: GPU scarcity, variable latency, and caching
• 04:43 Multi-tenant GPU serving: admission control, routing, and auto-scaling
• 08:40 Proactive GPU health monitoring and automatic node draining
• 15:44 Token cost estimation with model units and exponential moving averages
• 21:00 Prefix caching and multi-layer cache hierarchy for agentic workloads
• 25:38 Cache-aware routing with Dicer for session-sticky load balancing
• 28:49 High-throughput batch processing via ai_query in Databricks SQL
• 36:51 Custom model serving with VLLM for any transformer architecture

FAQs

How does Databricks Model Serving handle GPU scarcity for LLM workloads?

Databricks Model Serving uses multi-tenant GPU serving with admission control, intelligent request routing, and autoscaling to maximize GPU utilization while maintaining low latency. The platform also includes proactive GPU health monitoring with automatic node draining to prevent serving failures before they impact production applications.

What is prefix caching in Databricks Model Serving?

Prefix caching stores previously computed key-value representations of repeated prompt prefixes so the model does not recompute them on every request. This video explains how Databricks implements a multi-layer cache hierarchy combined with cache-aware routing (Dicer) for session-sticky load balancing — which is especially valuable for agentic workloads that share long system prompt prefixes.

How can you run batch LLM inference in Databricks SQL?

The ai_query function in Databricks SQL allows teams to run high-throughput batch LLM inference directly within SQL pipelines, without requiring a separate serving infrastructure. This video demonstrates how this brings model capabilities natively into data engineering and analytics workflows.

Why are open-source models becoming more important for enterprise AI inference?

This video notes a wide cost spread in the inference market and argues that open-source models — now trailing frontier quality by roughly 3.5 months — will become increasingly important as enterprises optimize costs. Databricks Model Serving provides an efficient stack for running open-source models as part of a cost-optimized, mixed-model AI strategy.

Full transcript

[00:07] Thank you everyone for making it here today for this late session. Today we're here to talk a bit about the inference stack on the Databricks platform. So I'm Brian Law, I'm one of the AI specialists from our Australia team, and we've got
[00:19] one of our engineers as well. >> Yeah, I'm Mike Easterham. I'm a tech lead on the Databricks foundation model serving team. >> So here today we'll peel back a little bit of what's under the hood so you can understand all the challenges of going
[00:31] through and doing large language model inference. >>
[00:44] well, but people are using more and more and more tokens. And that means we need to be more and more efficient. So as some of you might have already seen inside your organizations, as you
[00:56] go on, as you use live coding tools, as you go through and build more AI agents, then tokens is becoming more and more important. And then in order to be able
[01:08] to make it efficient, you need to start thinking about costs, you need to start thinking about the way that you host models and you serve models as well. So with with the token demand growing so
[01:20] much, we need to be able to be better at inference. We need to be able better at hosting models, running them cheaply, and giving you access to the tokens you need in order to build the applications
[01:32] that you're building. So the key is inference cost. And if you have a quick look kind of at the market, you see like there's this big broad spread in terms of how much it costs for inference, all the way from the
[01:46] dearly departed Fable model, all the way through to some of the cheaper open-source models. So for you to be able to be efficient within your organizations, you're probably going to need a mix of different models. So, it's
[01:59] not just about being able to be really efficient with say a Sonic class model or something, it's going to be able to It's going to be important to be able to have an efficient stack for going through and running open source models
[02:11] as well, and everything in between. You don't need Mythos, you don't need Fable for a cookie recipe. And from what we see, we feel that open source is
[02:23] going to become more and more important. And as that becomes more and more important, it means that you need an efficient platform in order to be able to host these models and run these models and integrate these through into
[02:36] your workflows, as well. >>
[02:50] very challenging problem. And I'll pass it over to Mike. >> Yeah. Uh so, at Databricks, we've built uh pretty sophisticated multi-tenant uh
[03:02] model serving system to be able to serve all the different open source models uh that you might want to use uh directly on our platform. Um and I'm going to walk through some of the key like design decisions we had to make make to do this.
[03:14] Um so, compared to classic ML serving, uh serving LLM models has a number of kind of unique challenges. Um I put three of them up on this slide here. Um so, in classic ML, we're
[03:27] typically serving these models on practically infinitely available, highly uh reliable CPU hardware. In contrast, uh when we're serving LLM models, we're using scarce,
[03:39] uh very expensive, and very faulty GPU hardware. On the classic ML side, uh most models you might encounter have low, predictable request latency.
[03:51] Um whereas in LLM serving, we experience highly late highly variable request latency. For some models, you might see request latencies as low as tens of
[04:03] milliseconds, and then for others, you might be going up into multiple minutes. On the classic ML side, your requests are typically fully independent of each other. You can kind of, you know, round robin route them to any given replica
[04:18] that's serving them. But when we're looking at LLM inference, particularly for agentic workloads that we're seeing where we're seeing a lot of growth, you need to share pieces of the work
[04:30] that you're doing for each request via cache. And this introduces a lot of complexity in the routing setup that you have to use, which I'll go into later in the talk. So first, we're going to look at the
[04:43] serving the GPU serving infrastructure that's kind of like the substrate for all of this model serving stuff. At a very high level, here's how a request flows through our system. So starting on the left, we've
[04:57] got Databricks users in their workspace. They're all sending us their LLM inference requests. Those are going being directed towards multi-tenant endpoints that we have per model.
[05:09] The first stop that they arrive at is on the control plane. We have admission control. And here's where we're enforcing lots of different policies that pertain to a given tenant or given request. The most
[05:23] important and kind of like technically challenging one to enforce is quotas. And so we have quotas for different types of products that we serve. It might be a kind of provisioned amount of throughput that a customer's bought from us. It
[05:35] might be for batch, which we'll talk about a little bit later, sort of dynamic amount of quota that's assigned based on what we have available. Once we pass admission control, we decided that we're going to serve a
[05:47] request. It's then passed along to the data plane. Uh and the first step there is the router/gateway. Um here's where we determine an appropriate GPU node uh to send a request to.
[06:00] Um that's a bit of a complex problem to figure out what the best place to send the request is. I'll be talking about that later. Um and then the final stop is that we arrive at what we call the LLM server. Um and this is the node that actually has the expensive GPUs that are doing
[06:13] the inference for us. So, it's kind of the journey of a request through the system. Um having this multi-tenant setup that I just showed you um has some advantages,
[06:25] but it also presents challenges. So, um like probably the biggest advantage this gives us is that by pooling demand across all the tenants, we're able to smooth out peaks and curves in each uh individual
[06:37] uh each individual tenant's request stream. Um and this by adding those all together and getting kind of like a smooth request shape, we're able to achieve higher uh utilization on our GPUs.
[06:49] Um the flip side of this is that when you're adding a bunch of demand together, you can sometimes get peaks that like exceed your instantaneous capacity to serve them. Uh and that's why we've designed a system of priorities for different uh types of traffic. And so, this allows us
[07:02] to have a policy that says, you know, provision throughput traffic that's very high priority. We don't want to drop that under any circumstances. Uh some lower priority types of traffic, say coming from batch, uh this is uh
[07:14] it's less critical those be served right away and we can drop them and retry them in a few seconds or minutes. Um and so, in this way, if we do have like peaks in the demand curve that kind of exceed the capacity we have, we can we can drop some requests uh gracefully
[07:27] without um violating any SLOs uh that we have. Um another aspect of this infrastructure is that um we're we have to take these nodes that we're
[07:39] buying from cloud service providers, which typically come in an arrangement where there's eight GPUs on one node, um and then we have this like very large catalog of open source models that we're trying to serve on them. Uh each of them have different requirements in terms of
[07:52] the amount of hardware they require. Um so, your largest GPUs might take up all of uh or sorry, your largest models might take up all of the GPUs on the system or potentially require even multiple systems in order to serve. Uh
[08:04] but a lot of smaller ones, um you know, with fewer parameters uh can actually uh get by just one GPU or maybe two or four. Um and so, to mix and match all these together, what we've done is we built a system where we can
[08:16] bin pack multiple models onto one node uh while still maintaining uh performance isolation between them. So, each model gets basically a slice of the system, it gets some number of GPUs,
[08:28] it gets some number of uh CPU resources associated with those and some amount of memory. And those are all there's strong isolation between them. Um we uh we can't, however,
[08:40] uh for every model in the system provision it at its peak demand. Um and so, we have to have a system of auto scaling where if one model is getting a lot of traffic at a given point in time, we'll add replicas uh to
[08:52] it. Um and if another one is not, then we'll scale it down. That allows us to, you know, with a limited pool of hardware that we have serve a larger variety of models by being um uh responsive to the demand we're seeing
[09:04] at any given point in time. Uh to make auto scaling work well, though, you do need fast request startup. Um I don't know how many people in here have tried doing like local LLM inference, but uh the the systems that
[09:17] serve these models can sometimes take tens of minutes to start up if you're not careful. Um and so, a big area of investment for us has been trying to drive down the startup time on our LLM servers. Um I've
[09:29] put on the slide a few examples of things we've done. Um one is kind of just like basic Docker image optimizations. Some of our infrastructure teams have helped us with this where we uh instead of downloading the whole Docker image all at once,
[09:41] we'll pull pieces of it on demand base depending on what's actually being read. Um similarly for model weights, uh some of the largest models we're serving have like hundreds of gigabytes of model weights that need to be downloaded and
[09:53] loaded into GPU memory um when the job starts up. So, uh we invested a lot in optimizing that download process from our internal uh model registry, making sure that we're fully maxing out the the network
[10:06] interfaces that we have uh available for making these downloads. Um and then finally, uh a typical inference engine uh actually does a lot of like compilation work when it starts up. And the reason for this is that uh
[10:19] some uh CUDA kernels that you use in these systems do like uh you want to auto tune them and that process actually depends on the GPU that you're serving on. Um so, the simplest approach that most people take is just do all that
[10:31] auto tuning right when the process starts up. However, that can add several minutes. Um and so, what we've done is we uh kind of boot the boot each uh server image up once on the appropriate
[10:43] hardware, do all that compilation and auto tuning, save it, and then reuse it uh later on when we're we're scaling up so that it doesn't happen within the critical path of auto scaling. Um so, combining all these together
[10:55] saves us like several minutes um on our model startup which allows us to to to run our system at a much higher efficiency than we'd be able to otherwise.
[11:10] Uh another issue we have to contend with in this infrastructure is that GPUs fail far more often than CPU-based hardware that people are used to working with. Um so, I pulled out a statistic. Uh this is from the Meta Llama 3 pre-pretraining technical
[11:24] report. Um it's based on a fairly small time period, but they found that the annualized failure rate of an H100 GPU in their system was 9%. Um so that means 9% of the GPUs in their
[11:36] cluster would fail every year um based based on the short uh sample they had. Um and keep in mind this is for one GPU, we're putting eight of these in a box and so like the per box failure rate is actually even higher
[11:49] than this. Um and to make matters even worse uh often times when uh GPUs fail, they don't do so in a clean way. So uh if you had, you know, something like a power supply failure, you'd probably
[12:01] just see your whole server drop off the network. Uh GPUs tend to go in the into a state where they just are like not working correctly and you'll find that uh your model serving system can start just serving 100% errors or hang or all
[12:14] sorts of kind of like nasty outcomes. Um and so you can't really just take the approach that you would take with most CPU-based systems where you say just let it crash if something goes wrong and then we'll restart it.
[12:26] Um so to address this, we built this like uh a proactive GPU health monitoring system. Um and so the way this works is we have a probing job that runs on every node um
[12:38] and it's got this kind of pluggable framework where we can uh plug in different health signals. Um I've put in the slides a couple of examples of of some things we actually have. So one is like XID error, DCGM uh
[12:50] diagnostic monitoring. These are sort of like Nvidia specific uh diagnostic systems that get exposed to you. So we look for errors in there. Um we keep an eye on the temperature of the GPUs. So a
[13:02] lot of failure modes are associated with like higher core temperatures on the chips themselves. Um and so that's like a good indicator that something's about to go wrong if the temperature gets too high. Uh we look at NVLink counters. We look at the RDMA bandwidth. We basically
[13:15] have this whole list of things that we keep an eye on. Um and then if any of those go outside of acceptable bounds uh for a period of time, we'll basically automatically drain the node. So we'll take all of the
[13:29] model replicas running on that node, schedule them somewhere else, uh take the node out of service, and then kind of hand it back to the cloud service provider for remediation at that point. Um so, with this system, while GPU
[13:42] failures still impact us, it takes some capacity offline and causes a few requests to fail before um our health monitoring kicks in, uh this really like minimizes the damage um and and allows
[13:54] us to to have a much more reliable system. Uh so, another problem that we have to deal with uh when engineering the system is like highly variable request costs.
[14:06] Um and you might be wondering like what I'm talking about with request costs or why they matter. Um and so, uh to motivate this a little bit, um I just wanted to point out there's sort of uh three different components of the system
[14:19] that depend on having some accurate view of how much a request is going to cost. Um the first is admission control. So, like I mentioned before, we enforce quotas uh on different users. Uh we need
[14:31] to make people make sure people stay within their quotas uh if we're going to have the capacity of the system divided up fairly. Um and so, we have those quotas defined in terms of the actual cost of the request um to make sure that
[14:43] we have uh resource fairness. Um the next place it matters is in the router. Um like I mentioned, the job of this thing is to uh send the request somewhere where there's capacity available to
[14:55] serve it. Um the way we do that is just having a view of the sum of all the requests that are running on all the replicas uh at any given point in time and then choosing the least loaded one
[15:07] uh to send a request to. Um and obviously, for this to work well, you have to have kind of an accurate estimate of like how much your request is going to impact the capacity of the server. Um and then similarly, uh there's a very uh
[15:19] you know, similar use case with auto scaling. Uh so, I mean, this is almost the same as load balancing, but basically the idea here is when the ongoing cost gets too high or too close to the total capacity, you want to start adding replicas so that you'll you'll
[15:31] have capacity available if more requests come in. Um and so to make more concrete what I actually mean by cost, um we have this concept that we've defined internally
[15:44] and it's also part of our pricing called model units. And a model unit is basically an abstract or normalized unit of throughput for for one of our servers.
[15:56] And so you can say something like this configuration where we've got four H100s and it's serving this particular query model can serve 20 model units of throughput. Um and then to connect from
[16:09] this throughput model unit concept to the cost of a request, what we do is for each request comes in, we we calculate a certain number of MU seconds that request requires to serve.
[16:22] Um and in the next few slides I'm going to talk about exactly how all this this math works. Um so the starting point is that well, let me say a key a key observation here is that like the
[16:34] cost of a LLM inference request depends very much on the shape. So by that I mean the number of input tokens and the number of output tokens. The kind of like fundamental math of
[16:46] transformers and LLM inference is that there's a quadratic relationship between the size of the tokens in the sequence and the cost to serve it. Um and so we can't just do something very simple
[16:58] like counting requests. We have to like map out this quadratic relationship. And so what we do is we take our server that you know, that has the configuration that we're going to be serving and we benchmark benchmark it at a bunch of
[17:11] different input output token shapes and we see what the maximum throughput we can get for that shape is just experimentally by sending a bunch of requests. Um once we have the maximum throughput
[17:23] for a certain request shape, you basically can just invert that. Uh so you take the the total model units of the model divided by the maximum throughput that you get for a certain input shape, and then we say that's the
[17:35] MU second cost of a certain request shape. Um so this gives us a way of like turning our benchmarks into some sort of cost for each request, which is grounded in reality.
[17:48] And then in the bottom right of this slide here, I have a couple of worked examples, so you can see for this fake model, a 2,000 input and 500 output request would cost 0.11 MU seconds,
[18:00] whereas a 16K output, or sorry, 16K input, 2K output would cost 0.71 MU seconds. Um so this is the same model, two requests that are both kind of like reasonably shaped, and they end up
[18:12] having a factor of like six or seven difference um in this cost model. Um a problem with this is that we need to know the number of output tokens to compute the cost of a request,
[18:26] but at the beginning of a request, we don't know how much output it's going to produce. And so we have to have a system for estimating this. Um one thing that doesn't work very well, and I can tell you because we've tried it, is just estimating say the
[18:39] output is going to be some multiple of the input. Um but I put like an example up here on the slide of why this doesn't work very well. You can have two requests where one is what is the capital of France, and another one is write a short story
[18:52] about Paris. They're both seven or eight input tokens, but you know, one can produce like 300 times the output of the other one. Um so we found that input and output size is weakly correlated in practice, and it doesn't work super well.
[19:06] Um so what we do instead is we track for each model, tenant, and session, we track an exponential moving average of the output lengths that we've
[19:19] observed for that particular sequence of requests. And we use this to to estimate what the output is. This gives us a little bit of a bootstrapping problem. So like if a session is just starting and we've never
[19:31] observed any requests before, we don't really know very much about what the outputs are going to be. And so to resolve that, we basically just like delete of the hierarchy. So if we don't know much about the session, we'll just look at
[19:43] what we know for the model and the tenant instead. Similarly, if we don't know much about the tenant, we'll just look at what we have for the model. So this gives us like a fairly reasonable way that's pretty cheap to predict output tokens.
[19:56] So end end this whole process looks like we get a request in, we consult the exponential moving average we have to estimate the output tokens. We have this MU curve which is based on
[20:09] the benchmarks I talked about earlier. We'll interpolate between measured points on that curve based on the the shape that we're expecting to see. And then we get a certain number of MU seconds that come out at the end of that. We report those to all the
[20:22] relevant systems. So we've got like the pod auto scaler, the admission controller, things like that. And then we let the request run. And finally, at the end when we know how many output tokens actually came out, we'll do a correction. So
[20:35] if we actually produced fewer tokens than we were expecting based on the exponential moving average, we'll actually report a negative amount of MUs to all the systems at that point. And so in that way, everything kind of converges to the
[20:48] correct result. So that was how we estimate requests weight. The next thing I want to talk a bit about is caching.
[21:00] So with these kind of like multi-turn agentic workloads that have become increasingly popular as I'm sure you all are aware. If you serve them naively,
[21:13] each turn in the request, you're basically redoing the work of all the previous turns. And so turn one, that would be your first prompt, maybe your cloud code session, you're like, "Hey, I want to build this thing." You get a bunch of output. And then in
[21:26] turn two, when you're going to say, "Okay, I actually want you to why did you do this? I want you to fix this thing." Naively you would be reprocessing all the all the output from turn or your original prompt and all the output from turn one. Obviously this is
[21:40] kind of a silly thing to do in practice. The architecture of LLMs is such that you can save the intermediate state that you have between these turns. And so there's this concept of prefix
[21:52] caching, which is basically you you cache the intermediate state and for requests that share common prefixes, you reuse it. And in practice when you implement this well for
[22:04] a generative coding workloads, you can avoid up to 90% of the cost of serving them. So it's really if you want to be competitive in serving these types of workloads, which we do, it's really imperative that you can have an
[22:16] effective caching system. The challenge with this is that like KB cache is actually very expensive. So it's we're storing essentially a bunch
[22:28] of intermediate values from this very large neural network. It it the size of the cache grows with the size of the model architecture. So if you want to do some really large model like Kimi 2.6 or 2.7, which we announced
[22:41] we're going to be serving today, you're you're dealing with very large KB cache values. So some examples here, if you have a single 4,000 token sequence for Kimi 2.6 with an FP8
[22:54] cache quantization, you end up needing 131 megabytes of storage for that part of the cache. And so if you do the math here, if you have like 1,000 concurrent sessions that uh uh of 4,000 um token sequences, that
[23:08] requires more memory than you would have in a single H100 GPU. Um, so just kind of doing the naive thing and trying to store all this in GPU memory, you don't get very far. You basically only get like a couple minutes of cash if you work that way.
[23:21] Um, so what we've done to engineer our way around that is we built this multi-layer cache hierarchy. Um, so at the top you have the L1 cache that's sitting in the really like high throughput expensive GPU memory.
[23:34] Um, the figure I put up here is what you would get for a system with eight uh B200 GPUs, um, which are sort of like near the the leading edge of what's available on the market. Um, you get about a a terabyte and a half of uh RAM,
[23:47] uh but only a few hundred megabytes or sorry, a few hundred gigabytes of it are actually usable for caching. There's a lot of other stuff has to go in there. You have like model weights, model activations, things like that. Um, so uh
[24:00] once uh cache items uh run out of capacity on L1, we evict them down to L2, which is your normal system RAM. Um, typical configuration for machines uh
[24:12] like this would have about 2 TB of regular system RAM. Um, and so we get two two terabytes of storage there. And then the final level is that we'll evict things out to a uh 30 TB of um attached
[24:24] solid state disks we have on these machines. Um, so as you go lower in this hierarchy, obviously accessing uh the cache entries gets a lot more it gets a lot slower.
[24:36] Um, but it's still much faster than doing all that work again on the very expensive GPU. Um, so uh using this hierarchy, we're able to, you know, increase our total cache capacity by a factor of about 100. Um,
[24:48] and we find uh that this gives us enough uh enough storage space to store about an hour or more of of cache sequences, which is about what you need to to get good cache hit rates for agentic coding
[25:00] requests. Um so, having the storage for all the cache is only one part of the battle. It's obviously pretty useless like in this scenario if server one is cached
[25:13] your previous turn, but then your load balancer sends turn two to server two, you're not going to be able to use the cache from server one. So, you basically run into this problem with any sort of classic load balancing
[25:26] algorithm whether it's round robin, least loaded, random, whatever it is. So, doing this you basically just wasted all the work that you did to get all this stuff in your cache. Um so, we basically need a smarter way
[25:38] of doing this like a router needs to be cache aware or have a smart brain as as this slide says. Um so, a a hypothetical system you could have that does this is that for incoming requests,
[25:52] you kind of have two cases. One, there's an explicit session key. So, you can imagine say like you're using Claude code and every time you start up Claude code it generates a random session ID and then sends that to the inference system. That way you know
[26:05] you know the deterministically you know what session a request is part of. That doesn't always work. Not all of our use cases have explicit session keys. So, we have a fallback system where we compute the key using a fixed
[26:19] length prefix of the request. And the idea with this is that you want to make the prefix large enough so that it's larger than the system prompt and other material that's that's shared between every request going into the system, but
[26:31] you want to make it small enough that it it covers like uh after the first one or two turns, you're outside of the the shared
[26:43] prefix of the prompt. Um once you have this key, you would have some sort of shard map where you map a key value to a specific server. And then since these keys are meant to be the same for every request in a
[26:56] session, this kind of solves your problem with cache routing. Um so obviously the hard problem with this is like how do you keep this map updated? And if one of the servers ends up with shards that are too you know hot shards that are getting a lot of
[27:08] requests, how are you going to avoid overloading the servers? Uh so thankfully Databricks has this really nice piece of infrastructure called Dicer, which is built basically to solve this exact problem. And we
[27:21] actually have a pretty cool blog post that we just published when we open sourced this a couple months ago. So if you're interested in it, you can go to the blog to find out more. Um but the way Dicer works for inference is that
[27:33] um so kind of starting in the bottom left of this diagram, requests come into the load balancer, and we look them up in that map. So Dicer maintains that magic map that we were talking about. Uh load balancer
[27:45] forwards the requests onto the server that's specified to the map. And then the key piece is that the LLM server is reporting the key and the load for each request to Dicer. And Dicer is asynchronously keeping a
[27:58] view of all of the load on all the replicas, and it's figuring out, "Okay, this shard covering these keys is going to it's getting too much traffic, and it's going to overload this server." And so it'll
[28:11] shuffle shards or split them or merge them depending on what's needed to rebalance things with the minimal number of reassignments because each reassigning is going to end up being cache hits down the road.
[28:23] So this system kind of takes care of like optimizing this the situation for us where we want to keep traffic as sticky as possible, but essentially no stickier.
[28:36] All right, I'm going to hand it back now. Yep. >> So we've just gone through a lot of the kind of the internals of going through and hosting and scaling with large language models. And we went through a sort of the lifetime of a request as it goes through the system, but for most of
[28:49] the organizations we work with, it's not about real-time chatbots. It's about going through and doing batch processing, going through and looking at documents, doing tagging, going through and doing OCR and things like that as
[29:01] well. And batch is a very different workload. So with batch, we've got high amount of compute that will need in order to go and run through that, but at the same time, latency is not as important. We can go wait 10 15 minutes,
[29:14] maybe even more for that to go through and process. So other platforms don't necessarily do batch very inefficiently. If you see this flow we've got here, some of the key things that make it complex is that
[29:27] you need to export your data out into some sort of object storage format or into like a JSON L or something. Then you have a very long SLA of even up to 24 hours in order for that to finish.
[29:39] And once that data is processed, you'll need to go through and restructure that and then load that back into your data platform, into the information ecosystem that you need the results to be stored within. So that means that you're
[29:51] writing dedicated code in order to go through and run this batch process. So batch on Databricks though is seamless. So we use all of the stuff that we talked about before with the
[30:03] model units to look at load sizes, with the caching abilities that we had there, and with the auto scaling architecture that we went through, in order to make it easier for customers to go through and run batch.
[30:16] So how do we go from the raw system that we talked about to sort of a batch aware design? So what we have is one more component that's the batch gateway that sits on top of the architecture. So
[30:28] that's that little component that sits over there. And then that one has a smart routing feature in order to go through and make batch run more efficiently. So some of the things it does is that it aligns with the Spark
[30:41] executors that our customers use when they're going through and processing data on Databricks. It looks at the scaling of the load and the number of tokens required in order to go through and process that, and then it aligns that with the GPU capacity that we have
[30:55] in order to make sure that we can push that load through, but at the same time not crash other real-time loads that might be going through the system as well. So, the key objective of our batch system is firstly, we still want to be
[31:08] timely. We don't want you have to wait hours in order things to get processed. We want to be able to process the job and start on it as soon as it comes in. We want it to be predictable in terms of how long it'll take to run each time.
[31:22] So, once you've start to run these loads, you'll be able to get a really good idea of how long this will take and how long your pipeline will run. And in order to make it efficient and cheap, we want to be able to utilize all the
[31:34] resources that we've got on the cluster all together. So, then we can save on the amount of GPU time you need to run that. So, the way that we've set it up within Databricks is it's all wrapped up
[31:47] essentially within this AI query command. So, this will go through and if you've got like a small request, then it can go through and just send it straight through. If you've got a larger request, then we'll go through, trigger the batch
[31:59] scheduler, and then work out how we can go through and scale and run that load. So, with the way that batch runs on top of Databricks, then firstly, there's no
[32:11] separate setup for you to go through and set up the model. From our benchmarking, you get up to 10 times faster execution than if you were to just to write loops and go through and send those requests through.
[32:23] We have the ability for you to go through and fine-tune and deploy other models on top of the platform and use them together with the batch execution as well. And we have the ability for you to do things like structured output out
[32:35] of that as well. So, then you can go through and feed the outputs of your batch processing into other systems, maybe as JSON requests or something as an API, or for you to maybe spit out auxiliary tables as a part of that. And
[32:49] because it all runs on top of our Spark infrastructure as well, on top of all the work that we've put in for you to track the status of your cluster and make that run well, you're able to go through and track the query performance,
[33:01] look at what's going on, and understand your batch workflow as well. So, I'll go through and show you a little bit about how it works. So, let me find my mouse first.
[33:21] So,
[33:33] customer system over here with ratings and with different lengths of characters. And then, we've got these raw text chunks that we want to go through and process. So, the way we can do that is through
[33:45] the AI query function here. So, inside the AI query function, then you just go through and specify the model. So, I can go through and select, say, the Quinn model here.
[33:57] >>
[34:10] just got to start my cluster first. Because that's all running on Databricks serverless, it doesn't take too much time for that cluster to kick in. You can see here from the progress bars,
[34:23] we're able to go through and track that as it runs. And because we've got a chunk of information that we're pushing through with the larger batch, then that will trigger the batch scheduler in the back there.
[34:35] So, that will take I think last time I tested it, maybe 15-20 seconds in order to go through and process with all of those rows.
[34:51] We are using the 122 billion coins, so that's going to take a little bit more time to go through and execute.
[35:15] And live demos are always a bit dangerous, but
[35:33] Oh, there we go. So, now we've got all the reviews that have come through over there. Because our platform is able to work with a lot of different models, we can go through and we don't have to worry about anything like spinning up different models and all the configurations for that as well. I can
[35:47] go through and just drop in a Sonnet model instead and run that through. Uh maybe I'll drop that down to 10 just so that executes faster.
[36:02] So,
[36:14] worry about tuning anything in terms of the cluster sizing. So, that's kind of like uh batch processing. These are kind of the models that we've got on Databricks, so we've got third-party models from Claude,
[36:26] we've got the OpenAI models with the GPT models. We've got access to Gemini models, and we announced today that we're going to add Grok in there as well. And we've also give you access to
[36:38] a wide variety of open-source models, including Kimi, which we announced. Now, one question that comes up then is what if you want other models? So, not everyone just wants from that fixed
[36:51] list. There might be some sort of esoteric model that you want to go through and deploy as well. So, one thing we've added into the system is custom model serving on Databricks as well.
[37:08] So, let me just go through and show you a little bit of that. Find my mouse again. If I go back into my Databricks environment there,
[37:20] one thing that you can do is you can go through and register and set up models inside of our MLflow system. And then from there, you can go through and host that on top of the serving infrastructure that we explained just
[37:33] before as well. I'm not going to go through all the details of this notebook, but one thing that we've added that we didn't have before is full support for VLLM as a language model server on top of the Databricks
[37:45] platform. The way that works is that we've got this entry point command here where you can go through and define your VLLM configuration as you do with the open-source. And then from there, when we go through
[37:57] and set that up inside of MLflow, there's a new ability to go through and set a custom endpoint that will trigger that function that will load that onto the VLLM infrastructure and make sure
[38:09] that that model is served efficiently. So, then that opens you up to be able to go through and host all sorts of different models on top of the Databricks platform, regardless of whether we have a specifically tuned configuration, like say with Kimi, like
[38:23] with some of the Llama models, and GPT-OSS, and others. So, once that's all been set up, then all I need to do is to take that model that we've deployed,
[38:38] and then I can use that through our AI query command again. Again, it'll be sort of batch aware, because it's only a 4 billion parameter model, so it runs quite fast compared to the others, and then you can see we've got the results back there.
[38:54] So, that's custom model serving. That's something that we've released out in beta recently, so go through and try that out as well if you want to try exotic models. In terms of the system, like we went through a lot of theory today, a lot of
[39:06] concepts, a lot of architecture. These are the numbers. So, we're processing about 5 trillion tokens per day on top of our platform. We're getting over 5 million batch runs that are coming through with our batch scheduler as
[39:19] well. We manage over 10K plus GPUs, and you can see there's a whole bunch of different organizations that rely on Databricks for their model serving needs.
[39:31] So, thanks for your time again. Hope that gives you a bit of an idea about what we're doing in the back in order to make LLMs efficient on top of the Databricks platform. And hope you're enjoying your summit.

Learn more about the Databricks Data and AI platform.