Running LLM Inference at Scale: Production Databricks Model Serving for Superhuman

Summary

Superhuman (formerly Grammarly) migrated production LLM inference for their grammar correction feature to Databricks Model Serving, handling peak loads of 200,000 requests per second with a four-nines availability target.
The team validated the migration through shadow traffic testing, A/B testing with quality metrics, and cost modeling before rolling out the fine-tuned LLaMA model to 80 percent of production traffic.
The production rollout demonstrated that LLaMA-based inference on Databricks is cost-competitive with a legacy deep learning stack, while managed infrastructure eliminated the operational challenge of provisioning large-scale GPU capacity.

Running LLM Inference at Scale: Production Databricks Model Serving for Superhuman

Serving LLMs in production requires solving throughput, latency, correctness and cost simultaneously. At Superhuman, we built a production system serving a fine-tuned LLaMA model for grammar error correction to millions of users on Databricks Model Serving. Peak load reaches 200K requests per second with strict latency requirements and demanding cost constraints.
This talk covers the production architecture, experimentation with A/B testing, validation checks, and cost modeling. We share the challenges we faced, latency spikes, quality regressions, and how client-side metrics revealed root causes. Shadow traffic testing prepared us for production scale. We now serve 80% of the red underlines feature on Databricks achieving four nines availability, demonstrating that LLaMA inference is cost-competitive with our legacy deep learning stack.
🤝

Chapters

FAQs

What is Superhuman's red underlines feature and how does Databricks power it?

The red underlines feature is Superhuman's grammar error correction capability that highlights mistakes as users type, powered by a fine-tuned LLaMA model served through Databricks Model Serving. This feature handles millions of users at peak loads reaching 200,000 requests per second, requiring strict latency and availability guarantees.

How did Superhuman validate the migration to Databricks Model Serving before full rollout?

The team used shadow traffic testing to send production-scale requests to the new system without affecting live users, verifying that it could handle peak load and identifying latency issues in advance. They also ran A/B tests comparing quality metrics between the LLaMA model and the legacy deep learning stack, with cost modeling to confirm the economics of the migration.

What latency and availability requirements did the Databricks deployment need to meet?

Superhuman's ML production team targets four-nines availability—99.99 percent uptime—as their baseline, with strict latency requirements for the grammar correction feature because delays are immediately visible to users as they type. These requirements drove the shadow traffic and load testing phases before the production rollout.

Why did Superhuman choose Databricks Model Serving over maintaining their own vLLM infrastructure?

Superhuman previously ran LLM inference on an internal vLLM deployment, but provisioning and maintaining GPU capacity at the required scale had become a significant operational challenge. Databricks Model Serving provided managed, scalable GPU infrastructure, allowing the ML production team to focus on model quality and cost optimization rather than infrastructure management.

Full transcript

[00:07] Hello everyone. My name is Mike and this is the session on the production inference, how we do the LLMs in production. So yeah, Mike, I'm coming from Superhuman, formerly known as Grammarly.
[00:23] And I'm coming here from our Berlin hub. We prepared this presentation together with my colleague Kristoff. Unfortunately, he had last last second circumstances that prevented him from
[00:40] coming. I will try to cover for both of us. Let's me run a quick poll. I will do more of those during my session.
[00:55] Who is running ML models in production here? Oh, quite a bit of people. Who runs them on CPUs? A bit. GPUs? And what about the scale of the
[01:12] the inference? Is it tens of GPUs, like tens? No. Hundreds? Hundreds? Thousands? Like a few thousands? I can learn a thing or two from you. But yeah,
[01:28] as you may know, getting GPUs, provisioning them, is a bit of a challenge in the past few years. And here we will cover how Databricks helped us in in this situation.
[01:45] So Let me talk about our team. I'm leading the ML production team. We are doing inference. And um we own a few dozens of uh ML services at
[02:03] this point, mostly LLMs. Uh but they also include some uh smaller deep learning models or classical classifiers. But for the LLMs, uh for you to understand the scale, this can reach uh
[02:19] 200,000 requests per second. So, this is our peak load. Uh and uh this includes uh the service that I will be covering today. More on that later.
[02:37] This is the focus of our group. Um so, we provide infrastructure for the teams. Um we provide the frameworks, uh the tooling so that uh once a uh researcher or ML engineer has a model trained, they do not have to reinvent the wheel
[02:54] to figure out how to serve it in production. They just use whatever we provide. We cover on-call and incident support so that they don't have to wake up at night. We cover uh them here as well, and we also
[03:11] make sure that this doesn't happen too often by maintaining um uh good stability of the system. So, we make sure that uh our platform is scalable and can handle whatever load
[03:27] that is required by uh the business and uh that we have the necessary compute capacity. Our availability target is uh 99.99, so four nines.
[03:42] We try to do better, but this is our baseline expectation. We provide observability, so think of metrics, dashboards, and alerts. So, um this is baked in into the
[03:58] platform. And um importantly, uh we make sure that we are very cost-efficient. So, uh those features, those LLM calls, they are powering the product. And um
[04:15] if we are not cost-efficient here, uh the business may not be profitable. So, we make sure that we are the best, and it's not possible to get better practically.
[04:30] Um before getting into the technical details, uh let's check what our product is doing. But uh before doing that, another quick poll. Uh who has used uh Grammarly before? Uh quite a bit of people. Uh who has used Coda Docs?
[04:49] Uh a bit. And Superhuman Mail? Also a bit. Okay. Uh So, when I started um my journey at the company 7 years ago, I joined Grammarly.
[05:05] But uh last year, we acquired two other companies, Coda and Superhuman Mail. And uh by the end of the year, we also did a rebrand. Now, we are called Superhuman. And this represents our shift from a single product writing
[05:23] assistant to a um um multi-product productivity suite that includes Grammarly, Coda, and Mail under a single umbrella.
[05:38] And the feature that we will be covering actually today is the crown jewel of Grammarly, the red underlines. So, I think if you used the Grammarly, you should know what it is about. But uh yeah, you may be typing in an email client or you are typing a
[05:56] Slack message. And if there is a misspelling or some grammar error, we can flag this with a red underline and a suggestion how to improve that. And
[06:11] we have quite a bit of users of our product product and this uh uh sets some expectations for the back end that is powering this feature. So,
[06:27] if you are writing, you expect the suggestions to show up immediately. You cannot tolerate high latency. So, we need to provide the result within less than 1 second, practically maybe half a second
[06:42] or even a bit less. Um users uh are using this particular feature quite heavily. It appears everywhere where you write. You don't request that. Uh the load is pretty high.
[07:02] Also, we need to make sure that the quality of the suggestions are really high. So, we do not produce any strange suggestions that will erode the trust in our product.
[07:17] And once again, because of high volume, we need to be really cost-efficient here. Around 4 years ago, the researchers had uh
[07:34] a challenge. LLMs were um starting off and we had to reimagine our uh grammar error correction pipeline. So, when I joined the company, um
[07:52] there are there were many suggestions powered by static rules, like linguistic rules on uh grammar trees. And at some point there were more than thousand of such rules. But as I joined
[08:07] we were building more and more deep learning powered features. And at some point there were several backends that were powered by uh smaller language models, maybe size of um
[08:23] 50 million parameters. And uh over the time there were more and more of them. And uh we had to maintain each of those systems. We had to understand how to aggregate different opinions by
[08:39] different models into a single coherent result. And with LLMs there was an assumption that maybe we don't need all of that and we can just replace uh this uh combination with essentially just a single LLM and a little bit of uh
[08:57] post-processing. And yeah, also we didn't know whether this will be even cost-efficient because uh the models that we used before were much smaller, so we could serve them
[09:14] um much cheaply. And uh the latency was also a concern, so the previous uh um systems uh they could respond uh probably faster than before.
[09:30] But um the researchers uh started with Llama as the first solid model that uh was performing quite well in their benchmarks. The model size was 1 billion or larger.
[09:48] And over the time um we saw that everybody just is centered around the OpenAI compatible API that most of the serving frameworks are supporting. So, it became much
[10:04] easier to evaluate them and use the best that is available. And the optimizations like floating point eight or speculative decoding in the end drove
[10:20] the cost per of inference down and made the resulting solution comparable in price versus the old solution. And still it was orders of magnitude cheaper than using a frontier LLM which,
[10:37] as you know, can be quite expensive. Here is an example. This is Databricks UI that we can use to poke around with like a singular request. So, you see here
[10:54] a sentence that is space tokenized and there is a little joint token at the end. It is a special token that our researchers are using to tell the model, "Okay, here was the input. Please follow with the completion of the corrected
[11:12] version for this sentence." And it does it so we can use this to just have a quick check in Databricks.
[11:28] So, when we started serving Llama, we used our internal serving. But over the time there were some signs of limitations with this approach and we started evaluating um third parties that can help here and
[11:44] Databricks was included. So, what were the pros of using Databricks here? Uh before we had separate system one to train model, one to store the model and
[12:02] like a solution to serve. It was disjoint and like many of the tools were custom built and it was not like a single unified platform. With Databricks we can get a single place
[12:18] that can follow from data to model and to serving the end-to-end workflow. So, sensitive data stays at the same place and
[12:33] we can ensure the model versioning and data lineage and and governance. And regarding the challenges that we were starting to have with our in-house serving um
[12:49] availability has been a concern for us. Historically we were running in AWS US East 1. It was a bit hard for us to go to multi-region and within US East 1 we were running out of compute capacity for
[13:05] the necessary GPUs. And it was um sometimes we had the capacity but few times we did not and it was becoming more risky for us. And with the using
[13:21] Databricks we could also rely on their team to uh, the model. So, we did our optimizations, but, uh, probably that was not the core thing that we wanted to work on.
[13:41] So, when we were assessing, um, Databricks, uh, we, uh, we wanted to to check the following. So, the output should be, uh, consistent with what our model was providing internally,
[13:57] because users trust what we show them. Uh, if we do a strange suggestion that, uh, can harm that. Uh, the cost had to roughly match what we were having internally. Like, we could not have like
[14:14] 10 times or 50 times the cost. And, um, the, uh, KPIs that I mentioned before should also have been satisfied. So, the latency is critical for us. Uh, the scale, like, we have a certain scale
[14:30] in pattern that I will show a bit later. Um, like, um, the Databricks, uh, solution, uh, should have supported that scaling pattern. And, uh, support, uh,
[14:45] before we were doing on-call for that, uh, uh, serving with, uh, using Databricks, the expectation is that if there is anything, on the serving side, either they handle that or if we see that we need to
[15:01] escalate, then we can do that easily. Now, let me show the architecture of the, grammar correction, uh, processing pipeline. So, our clients, uh, first connect to
[15:19] the entry point, the common API. In practice, those are web socket connections and as you are typing, it mirrors the state of the text in your text box to the kind of internal state
[15:34] inside copy. But as you, for example, are typing and put full stop or have pause in typing, that converts that into HTTP calls to the grammar service to
[15:51] perform the actual check of this portion that you have typed so far. And inside this grammar service, we had to implement a switch. So, depending on the
[16:10] membership in the experiment of I can the gate, we use static experimentation platform to handle that. So, depending on that fork, we either go to the internal VLAM based serving solution or we go to Databricks. But to go to
[16:26] Databricks, we also have another component here. It's called LLM proxy. It's our internal know-how secret sauce that we have been using quite long ago to access third-party frontier models and it has all the
[16:43] necessary tracking that we use in-house. But now Databricks also has an offering here. Just keep our LLM proxy because of maybe historical reasons. But practically, we had to implement OAuth
[17:01] because it's I think a bit different from the other LLM providers. So, we have like long-lived secret and we need to exchange it for a token that I think is good for 1 hour.
[17:17] So, the this had to be implemented inside our LLM proxy. So, once we had the integration in place, we had to do some evaluations
[17:33] before we could start running data with the actual users. So, we set up an AB test. That was our ultimate goal, but the goal was kind of a bit strange versus our other experiments. Usually, other experiments
[17:50] want to improve certain metric, certain business metric like retention or uninstall rates. But here we just wanted to show that no matter what we use either internal or Databricks, there are no statistically
[18:07] significant changes to the business metrics. So, that was our ultimate goal that proves that we can be safely starting to use Databricks.
[18:23] But before that, we ran some more down-to-earth verifications. The easiest is the exact output match. Even if we run against the same endpoint, we don't get 100%. This is due to
[18:39] batching and numerical rounding errors. But here you see that if we were comparing our LLM-based solution versus Databricks inference, the output match was around 94%.
[18:57] Still, that was quite acceptable because once we started checking the output differences, they were mostly like another option of another valid correction for the same sentence that may be corrected like differently.
[19:15] But still once again the ultimate verification was the AB test. There were other things that are applicable in to the
[19:30] grammar task in particular. So there is an M2 score that can help us calculate the precision and recall for the given evaluation set. And those are some example numbers, not necessarily what we had
[19:49] during evaluation, but this is an easy way to get just some number. So to understand how to make apples-to-apples comparison, we had to do the cost modeling. And
[20:06] the well-known metric here is cost per million tokens. So you run traffic. You may not necessarily have to run 100% of your traffic, but usually it has
[20:21] to be sufficient enough to make a reliable numbers. So for example, if you run 1 or 5%, the number may be not not optimal. But once you have 10 or 30%, I think 30% is a very nice
[20:39] representative number, then you can have something that will resemble the actual production usage. So you just have this snapshot running for a week, for example, and you normalize by the
[20:54] total number of input and output tokens. In our case, the number of input and output tokens roughly is the same. And on the graph you also see our uh weekly usage pattern. Here, I think
[21:11] Friday was a public holiday in the US, so it may highly also depend on what happens in the US. US is the biggest uh uh user of our product. But, uh when there is a business uh day
[21:27] in the US, we have a high usage, and when it's night, we have low usage. And this uh can also pose some challenges to the auto scaling. So, as you see, we can scale quite high like from fairly low numbers to pretty high. And
[21:44] uh the highest uh slope is uh going from Monday uh sorry, from Sunday to Monday. But, yeah, once again, uh those are some example numbers. Uh practically, they were better. But, uh once again, you can
[22:01] compare that to the numbers that Frontier LLM uh cost uh and that can be dollars or maybe tens of dollars, and here we are talking about cents. So, this is about cost efficiency.
[22:19] Uh as you know, nobody is perfect, not even Databricks. We had a few hiccups uh along the way, but I must note that every time we had uh some challenge uh popping up, we had uh the Databricks team ready to go to uh help with that,
[22:36] and I think the turnaround uh was really good uh here. So, you can check this latency graph uh and think what what's what's wrong about it?
[22:53] So, as I said, uh we have the latency requirement of 1 second, ideally better. And uh here, we are mostly under second, but there are some outliers. But also in general, it's very shaky.
[23:10] So, it feels like here we were running at the edge of what the computer could offer. So, we were probably utilizing the GPU to the maximum extent, and I think sometimes the queue was building
[23:26] up. And uh we noticed that uh because we had the the client-side metrics in LLM proxy, this custom component that we had, and we had a very detailed 10-second aggregation metric that we could slice
[23:43] and dice and compare uh versus our internal baselines. And we raised that to Databricks, and I think in the end it was related to auto scaling. I think practically it has to deal with
[23:59] concurrency-based auto scaling. Under the hood, we use RPS-based auto scaling because all the requests have um similar distribution of token lengths, and by using RPS uh
[24:15] we can leave a bit of breathing room so that we don't run at the edge of what the GPU can offer. It's a bit of a um wasted compute, but
[24:30] once the auto scaling change was implemented, as you see, the latency stabilized, and it lowered quite a bit. And I think, yeah, it was very much in line with what we had internally.
[24:54] Here is a challenge, find the four differences. I think one of the things has been highlighted already. But yeah, you can take a look here. And uh yeah, I can hint uh
[25:10] the output in the middle column is kind of a valid version that we could have. Uh but uh it's a bit strange. We usually never have uh this uh produced by our internal model.
[25:26] And then when we started checking more thoroughly, we uh noticed a bit of off-by-one change in prop tokens and total tokens. And notably uh one once we checked this further,
[25:42] uh we understood that if we insert the beginning of text uh token artificially, then everything becomes back to normal. So, what happened here? Uh before launching
[25:57] production traffic, we obviously checked everything and everything was fine then. Uh and uh inside the tokenizer configuration, uh uh we I think there is uh just a built-in uh
[26:12] uh instruction that we need to prepend this token. It just happens. You don't have to specify it manually, but it always is there. And uh apparently there was a a regression uh in the model container that uh
[26:30] effectively removed this uh addition of beginning of text token. The model still was working, but um our researchers uh one morning were checking the uh the metrics in the experiment, and they noticed that the numbers of suggestions
[26:46] were off. And then we rerun our quality checks, and we realized that it's very different. And yeah, once we understood that it's beginning of text token, we escalated that and this regression was fixed.
[27:02] But yeah, the learnings here were it's very important to have the AB um testing and a system like StatSig or another experimentation uh framework to track
[27:18] such deviations and also to run periodic quality checks. Another thing that popped up was due to scaling. Let me first explain what is shadow
[27:34] traffic. So, before running with like actual user traffic and returning the responses to the users, we set up our uh grammar service in such a way that it would send the request both our internal
[27:52] VLLM and to Databricks. But, the response that we were getting from Databricks, it was essentially discarded. So, if there was an error or there was some other output, we didn't care at this point. But, this helped us
[28:10] to understand whether the system can handle the load properly. And once again, we had our gate there. We started small, like 1 5% and we as we bumped this further, we
[28:28] noticed that at some point we were starting to get strange spikes and latency increases. And um um They were not showing up on the
[28:43] Databricks dashboards. So, the inference was fine and it turned out that it was authentication subsystem on Databricks side that was probably hitting its limits, and the auth containers had to be fixed and scaled
[29:00] properly so that they can keep up with the load on the inference. So, the learning here was once again, it's really important to have the client-side metrics and alerts on them, but also just the fact that
[29:16] sometimes it's not the inference, but some infrastructure that is in between, like authentication or network load balancers, things like that. So, you need to identify at which point
[29:32] an issue occurs. So, regarding on-call, even though we didn't have any major outages during our testing, we still had to simulate something.
[29:49] Another quick poll, who has been on-call in the last 2 weeks? Okay. A bit of people, but not as many as expected. But yeah, we don't want to be paged at 3:00 a.m. and
[30:05] not really understand how to escalate an issue, for example, if it's on Databricks side, or run into a permission issue that you cannot create a ticket, or once you create, you get confused how to route this properly to the team. So, we had to create like a
[30:23] mock outage or mock event, so it didn't really happen, but we just created a ticket to Databricks, and we checked whether it was handled properly. And for the first time
[30:38] that we did it, we realized that there is no such option as like inference. We had to create more generic Spark ticket and it took a bit of time to get forward that to the right team. But that is now fixed. Now if you are creating a support
[30:55] ticket, there is a proper inference uh option. So let's now check what is the outcome of our partnership. Here is the real production rollout timeline. As you see,
[31:11] it took a bit and it was squeezed between two production freezes on one on Thanksgiving, one on a winter break. And it was also not a big bang approach. We also started scaling and then we continued scaling after the new year.
[31:33] Now there is also as a result of this cooperation, a blog post that has been up maybe for a month or so and the engineers from Databricks uncover some of the secrets that were even surprises for us. We didn't know what
[31:49] was going on behind the scenes for the optimizations, but here is quite a technical blog post that uncovers that. And now uh
[32:05] we were able to push this 200 RPS plus to the foundation model provision throughput on Databricks. And that represents roughly 80% of our red
[32:21] underlines feature. We keep a little bit on our side and flexibility to increase or decrease this percentage depending on the operational situation. And the availability has consistently been four nines or above.
[32:38] And uh with this uh example, uh we see that now the uh inference offering uh by Databricks is quite solid, and it can handle quite a bit of traffic, and I hope that this was partially due to our
[32:55] project as well. So, yeah. Now, time for Q&A. Thank you, everyone. Uh you can I can keep this slide if you want to connect. Uh And also, don't forget to provide feedback.
[33:12] But, uh yeah. Now, Q&A. We have some microphones. Uh please line up uh uh so that everybody can hear you better. You may use this one or this one. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.