Skip to main content

Stateful Apps for Thousands of Users: Horizontal Scaling, Session Affinity, and Sharding

Summary

  • This video explains how Databricks Apps achieves horizontal scaling across multiple application instances to support workloads growing from tens to tens of thousands of concurrent users, a capability recently released in beta, using a load balancer that distributes traffic across replicas.
  • Session affinity—sticky routing that keeps each user's context and in-memory state attached to the correct replica—and rolling deployments with zero-downtime updates are key architectural components for maintaining stateful behavior at scale without sacrificing developer experience.
  • The talk distinguishes between hard state that must be persisted externally and soft state that can be gracefully rebuilt, and provides a practical checklist for writing apps that recover from unknown session state.

Stateful Apps for Thousands of Users: Horizontal Scaling, Session Affinity, and Sharding

Watch: Stateful Apps for Thousands of Users: Horizontal Scaling, Session Affinity, and Sharding
Databricks Apps make it easy to stand up a powerful internal app, but most teams don’t have a clear mental model for what happens when usage jumps from a handful of users to hundreds or thousands. In this advanced, under-the-hood session, we’ll walk through how Databricks Apps can remain stateful at scale: horizontal scaling across multiple app instances, session affinity to keep user context and in-memory state attached to the right replica, and sharding to distribute traffic and load safely across your fleet. By the end, you’ll have a practical architecture playbook and checklist you can apply to take an existing Databricks App—from LLM agents to analytics UIs—and harden it for real production scale without giving up the fast developer experience you already enjoy.
Talk By: Andre Furlan Bueno, Sr. Staff Software Engineer, Databricks ;

Chapters

FAQs

How does horizontal scaling work in Databricks Apps?

Horizontal scaling in Databricks Apps adds multiple instances of the same application rather than simply increasing the size of a single instance, which enables high availability and the ability to serve tens of thousands of concurrent users. The platform's load balancer distributes incoming requests across replicas and selects the appropriate instance for new connections. This capability was recently released in beta and represents a significant step in the product's maturity for enterprise-scale workloads.

What is session affinity and why is it important for stateful apps?

Session affinity, also called sticky routing, ensures that a user's requests are consistently directed to the same application replica that holds their session data and in-memory state. Without it, a user's context could be lost whenever their request is routed to a different replica with no knowledge of previous interactions. Databricks Apps implements session affinity to keep stateful user experiences consistent even as traffic is distributed across multiple instances.

What is the difference between hard state and soft state in app scaling?

Hard state refers to data that cannot be reconstructed if lost—such as user-submitted form data or transaction records—and must be persisted in external storage. Soft state refers to data that can be gracefully rebuilt or re-fetched if a session is lost, such as cached query results or UI preferences. This video recommends designing apps to minimize hard state and build graceful recovery paths for soft state so that horizontal scaling and rolling deployments do not break the user experience.

How do rolling deployments enable zero-downtime updates in Databricks Apps?

Rolling deployments update application replicas one at a time rather than taking the entire app offline, so users continue to be served by instances that are not yet being updated. New traffic is gradually shifted to updated replicas only after they are confirmed healthy, avoiding the need for a separate rollback mechanism. This strategy is particularly valuable for production apps where any downtime would impact end users during a deployment.

Full transcript

[00:08] Welcome to talk about data bricks apps and scaling data bricks apps. My name is Andre. I'm the overall tech lead for data bricks apps. I build the product. It's my third time a day I was talking about it and this time I decided to talk
[00:20] about something that I think is shows the maturity of how the product's coming which is like how to scale and uh see a lot of customers having trouble with that. So I'm going to talk a little bit about this. Uh we did we invest a lot of
[00:34] engineering effort into it and is a bit of a technical talk um packed of little tips. So, hope you guys enjoy. Um,
[00:53] so the the beginning of it is like you create an app, you think everyone no one's going to use it, but people actually start using it. And now you have an app that is in production in quote and uh and it goes from 10 to 20 to 50
[01:07] to 100 users. And you go, there's customers now with 10,000 users in the in their app. How many of you have this problem right now? Quite a few. That's nice. Um yeah, so
[01:20] this talks for you. Um the the reality is uh we had for a little bit of time like just bigger instances but if you want high
[01:33] availability and you want to go for it to tens of thousands of users you need horizontal scaling um and that is what we're going to focus today. But uh it is something we just
[01:45] released in beta I think two weeks ago. It's very limited, but we're we're getting there. And uh I'm very excited about the tech that we built for enabling uh your apps to scale and tens
[01:58] of thousands I mean concurrent. Um,
[02:23] so man. Yeah. So today we're going to talk about like u how to reason about app scalability. uh what are stateful apps and why I named the talk stateful apps and the difference between soft and hard state
[02:35] that's very important nowadays to understand that uh the technical challenge horizontal scaling and how we solved it and just at the end a practical checklist of what to think about in different use cases that I see
[02:47] applications today um so how to reason about app scalability um there's There's two main ways to think about scaling. You can either make it
[03:01] the app bigger or you can add more instances of your app. Uh there are use case for each one of them. So if you want to add more u have your app have
[03:13] more CPUs and more memory and whatnot that's good but you there's some applications. So you do this if your app is using doing some heavy computation. uh if you really need that CPU.
[03:26] Uh the other reason why is you want to keep it simple because at the end of this talk you hopefully understand that as soon as you add more instances things become more complex. You kind of need to think that state is going to be across a
[03:39] bunch of machines and it's going to be different than how you work in your laptop. Uh and the other thing is you use frameworks that kind of assume a single machine. Streamlit gradio for example, they kind of assume that you're
[03:51] going to vertical scale. They do not work very well horizontal scaling. Uh and when to not use them, please don't use if you're just launching a Python and no JS process because they're going to use only a single vCPU and you're
[04:04] going to waste a lot of money. Um, this horizontal scaling if I can get this thing to go.
[04:23] That's better. Horizontal scaling you use when your app is mostly orchestration IO. So if your app is receiving a request going to SQL or going to an LLM and coming back or the database then you probably want to use horizontal scaling. uh you are using
[04:36] some standard Python Node.js frameworks that are don't use the concept of workers uh because you're going to be constrained by that single CPU a single thread that NodeJS has or Python has. Uh
[04:48] the other reasons that you want the user is very spiky. Um you want autoscaling if you want autoscaling is you want horizontal scaling. Vertical scaling for autoscaling very few frameworks
[05:01] actually supported in Python and Ojs. you need something a bit more like Go or Java to be able to do that. And then the other reason is you need high availability. The reasons is like I don't know if you guys are Azure but
[05:13] Azure causes me a lot of headache and every so often they things fail and when things fail if you have only one instance there's nothing that I can do but go again and get another one for you. But if you have
[05:26] multiple it means that you have some reliability in there and that's very very important for mission critical applications. uh you should not use this if you have just a few users but you uh have high
[05:38] computation apps uh some ML people are doing this and you like it just becomes very complex to reason about horizontal scale if you're doing high computation uh and if you at the end of this talk
[05:51] you don't understand what I'm talking about you probably shouldn't be doing this either because you're going to be confused uh the first thing and very very important is that the majority of customers that I see actually don't need
[06:04] to pay more money to scale their app. The bottleneck is not the app. The bottleneck is somewhere else. Okay. Uh if your app can an app whatever framework can easily do a thousand
[06:16] requests per second. Uh and I'm pretty sure most apps have like a hundreds of milliseconds 999 of latency inside the app itself. Node.js is actually pretty fast. Python can be very fast. a fast
[06:29] API uh application, but your SQL can only can only take 10 requests per second. It doesn't matter how many your how much your app is doing. And SQL quer is actually very expensive and slow.
[06:43] Like if the pin and of your queries in your app is 5 seconds, what's the point of adding more compute in your app? It's just going to add to the queue. The same for LLM. LLM is even worse. uh because normally the request response and the
[06:56] streaming of the tokens is actually pretty expensive and you have to pay a lot of money to get a lot of requests per second. But so always do this first. Think about like what is the bottleneck and uh if
[07:09] you want to understand better we have this feature to export open telemetry data from your app trace or instrument your app so you actually understand why your app is not scaling. Um
[07:21] majority of apps today in data bricks apps do not actually need horizontal scaling. Um you can just fix it quickly. Um and the reality is an optimized app
[07:34] can serve thousands of concurrent users. So now to optimize an app is time and knowledge and sometimes you just don't have that. So sometimes you just throw
[07:46] money at the problem like I do and data bricks app itself. Uh but it's important to like to not to think about scaling as like a system problem. We need to think end to end instead of thinking oh how
[07:58] we're going to scale the app compute. Maybe the app compute's not what we need to scale. Okay. So the next now the meat of the problem um state for apps and the
[08:10] difference between soft and hard state. Uh this is I think one of the interesting things about Brid data bricks apps is that um what what we find when we understood about
[08:23] how to build and scale apps are very different in the world where you have the majority of apps are AI agents analytics applications or workflows. This kind of apps
[08:35] behave very different from a little CRUD application like a to-do app is going to go to the database. is going to return and it's very fast very like standard to how to scale but these guys they hold a
[08:47] lot of contacts in memory AI agents hold chat history to output intermediate plans you're like keeping all that stuff in memory uh very you you kind of only store the checkpoints uh analytics like
[09:02] if you have to go to the secret house again to get the filters the selected rows and everything that the user has selected you're going to spend a lot of time doing that all the time so you always keep something in memory as well and sometimes people even cache result
[09:15] query results although you have to be careful about that uh workflows you always like executing the middle of something there's a lot of inrogress work that in progress work is very hard to keep it only in the database and
[09:28] store in every instance so that is soft state and like I put the number here but I'm pretty sure there's even bigger uh like the majority of apps vast majority of apps that bricks apps today have soft
[09:41] state and if you get rid of the soft state the app breaks. So it's very that's that's kind of what it is. Um and again like soft state in memory kind of the cache in progress and
[09:55] hard state is what we store in lake base when you really the source of truth. Um
[10:07] uh and I put a little last year I presented this at any summit but this is kind of a picture of like if you're just doing a crowd application you're going from the web app to lake base the database and coming back it's very fast it's like milliseconds everything is
[10:19] deterministic everything is good uh you can make it stateless the old two 12 factor applications work here the reality of our apps today in data bricks are very different. They're
[10:31] running for minutes. Everything's streaming asynchronous. There's web sockets everywhere. You're going through a lot of lambs. You're going to a lot of models. It's actually very expensive to redo everything that you did again. So,
[10:43] it's is a more complex architecture compared to this one. Uh but that's the power of data bricks and data bricks apps. Um and uh
[10:55] yeah and that's that's why we spend so much time working on a solution for this because how I I spent a lot of time thinking like how do I allow you guys to scale your application in this world? Um
[11:11] so what are the technical challenge that we had and what do we build? Um I'm going very fast. It's good. Um the first problem um until
[11:23] very recently databicks apps would rebuild every time again if you wanted to start your app again. So every start you rebuild your app. Uh one of the key variants of
[11:36] horizontal scaling is that every single instance needs to be the same. If the instance is different, you're going to have a really hard time figuring out why your app is not behaving the way you want. So we had to make this fix this and the
[11:51] other is that we need to fix the other problem which is like sometimes you build in dev and you go and deploy in prod and there's something that changed you didn't pin a dependency or some environment was off and your deployment
[12:04] fails and now your app is down as doesn't work for production applications. So what we did and that's what's coming with the horizontal scaling product feature now is that uh now every build
[12:19] we actually create optimized container layers and store in a registry and the build itself has an ID and is stored on every single and used for every single replica. It also means that if I am, for
[12:33] example, if I need to update the OS and I need to rotate your app, the exact same code will always run and you're not going to rebuild. Uh it took a lot of time for us to do
[12:45] that to make it fast, but we got it. And the second is that we always build in one instance, see if it works, and we're going to add health checks and everything for apps. So if the app
[12:57] starts successfully and the deployment successfully only then we switch traffic and start the roll out to all the other instances. So it's a blue green deployment per instance
[13:09] uh of your of your app and that is huge because it it means that a few other things will come up. So for example, if you have a problem with your app and you want to roll back, I have the build. I
[13:21] can easily implement a no roll back my deployment and you can have the deployment everywhere. Uh and you can see as the the rollout progresses progresses, you can see like what is the
[13:33] actual app build that is used in each instance uh and is all in the logs and everything. So that's that's very important. It took us a long time to build this. Um, so blue green deployments and stable
[13:46] builds. The other problem is load balancing. Um, load balancing is kind of the the meat of this talk, but it's like a very hard thing. The most the hardest thing in my
[13:59] job right now is networking. And uh there are typically two ways to do this. One is you that what we call the roundroin. each request to land in a random
[14:12] instance. So if you have someone in your your client in the browser and every time they click a button and there's a request to the server is going to go to potentially a different instance of the server. So you cannot trust that the
[14:27] whatever is in the browser in the user session is in a specific instance is everywhere. Uh it works very well if you have fully stateless applications and you and you really want to scale auto
[14:41] scale very fast and down because it's simple and the load is actually very well balanced also works very well if each request has kind of the same load.
[14:54] If your load is killed per is skewed per types of requests, you might have an imbalanced but it's fine. The other one is sticky um which is like his name is saying you
[15:07] kind of like as soon as you get a session you get a little cookie and that cookie uh or an ID of whatever it is like redirects to the same replica every time on that session and that's it. It
[15:20] never changes. you have to create a new session to go somewhere else. Uh the problem with sticky sessions is that it works very well because if you network fails and you have a web socket or something and you redirect it goes back.
[15:33] So you kind of always know it's going to come back to the same place. But at scale it will break because you cannot load like if you add a new instance all your load in your servers don't migrate
[15:46] to the new one. They just stay there. Uh so which means that autoscaling becomes really tough in with sticky sessions because what's the point? You have to have a lot of in andout sessions to be able to load and most data applications
[15:59] are not necessarily like that. So that's kind of what it is for load balancing. Um Slack wants to update. Seriously,
[16:41] So, ro round robin. Like I was saying, uh if I just wanted to do round robin, it would be actually quite simple. But uh the way I saw apps created
[16:54] if you have just round robin like especially the context like a lot of agents specifically uh the way the loops are designed, they're designed to continue executing on that loop and go to the LLM and come back. Very few of them are actually
[17:07] designed in a stateless model. they don't go to the database and restore it every time and it's actually very chatty if you do that and it will cause troubles with the database. uh care like
[17:20] the other thing is that a lot of the times especially with u some some frameworks we're using websockets and secs uh you you're streaming results back and something fails or the user
[17:33] close the browser and start again or whatever uh you kind of like lost lose context once you lose context it can go to a different place and it means that you can actually get for example re-execute a query query was like almost
[17:46] done and it was just streaming the results of that thing and now you're you're done. Uh the other thing is that it's very hard to manage longunning queries uh that to completion in a server that
[17:58] you can update um if you don't keep that information there and continuously pull if you have whose server which replica is going to query to know that who's going to pull the SQL warehouse you know
[18:10] if the core is done takes a it's very hard to do that if you don't have uh soft state uh the other one is ownership Debugging is really hard if you don't
[18:24] know which replica because you kind of had to stitch for a whole user workflow. You have to stitch data and logs from each one of those replicas together to understand. So you need to understand about trace ids and you need to know
[18:36] about request ids and empir trace ID and you need to like start to to think fully distributed. So it's a it's kind of a kind of complicated. Uh so what we build databix open sourced
[18:50] this thing called dicer. I don't know if you guys saw this. This is their official uh official picture. And I kept it here because it looks complicated.
[19:02] Uh but the whole idea here is that you're not doing sticky session or you're not doing roundroin. You're actually sharding your web ser instances. And the difference with sharding is that it's load aware and it
[19:14] can be rebalanced on the fly. So you don't always assume that the request that the user session will go to the same instance but you can almost always assume that it go to the same
[19:27] instance and that is a huge plus because it's soft state you can rebuild but if you minimize the number of times a session goes from one replica to the other as you autocale your app and loads
[19:39] gets uh distributed it really helps a mental model on how to build applications. The other thing is that uh because now you can keep things in memory and you can design your application with more state in memory you don't need to think about uh going
[19:53] to external distributed cache coming back serialization cost going to the database all the time come back different connection pools and all that stuff that it gets very complicated at scale um actually unit catalog and SQL
[20:08] today use this and one of the main reasons why we could scale the services is because of daer Uh we had a huge problem with caching distributed caching data bricks uh and after we introduced c daicer it kind of
[20:22] went away uh is is it was huge for us internally and now with we spent like a few months trying to get this technology out for data bricks apps and every data
[20:34] bricks apps that has horses scaling enabled uses this today which is really cool. Um, so, so that's the idea. The idea is that you're sharding and, uh,
[20:49] I have a little simulation that I want to play. Let me see if I can find this.
[21:07] Does anyone can see this? No, no one can see this. Yeah,
[21:27] Cool. So, I wanted to show a little bit how this works in actual in practice. So if you have round robin um you have a single you have two instances is let's say that you start to have more sessions you start to have load in your
[21:40] application there's load there's caches how many cache miss you have cache hits so the more things you cache the more hits you have uh and then you add a new instance your load immediately gets redistributed
[21:54] but you also have more cache misses because your app is going everywhere your requests are going everywhere. Uh and then if you add a bunch of more, you kind of have starts to have a lot of
[22:07] cash misses and your lo but your load is still distributed. Um and you can say oh if I get skewed
[22:25] some the instances even though it's killed it's kind of like it's going to continue to be balanced. Um okay so now that's the idea about round robin stick key
[22:40] you start with a replica with a one replica and start adding load you get your load let's say that now you need to scale up your app let's say you turn on all the scaling you add a new instance
[22:53] nothing moves there's no redirection there's nothing that happens uh only when you start to add new they were going be distributed randomly across the instances, but your your instance the
[23:05] replica that had issues before continue to have issue. You can't really uh balance your application and if you add yet another instance now say I'll just add more more
[23:18] replicas it doesn't solve any of the issues of the problems. you can get new user, new sessions, it's going to work, but maybe some new sessions going to actually end up in the replica one and that's going to cause even more
[23:30] problems. Uh, and if you have skewed uh traffic with some some traffic is is more to other replicas, then you pretty much killed your first replica
[23:43] because all the rep the the requests the s the heavy sessions are in one and they don't migrate. That's uh that's the problem with sticky sessions.
[23:55] Uh charted. That's what we did.
[24:10] If you add a new replica, we will redistribute some of the load automatically. Now, this simulation is a little off. happens in reality if the load is up to like I think 60% we're actually not
[24:23] going to change anything we only change when we when it's needed and when the replica actually needs the load distribution because we're minimizing the model sessions that go between replicas. Um
[24:36] and if you have skewed and you start to get load in one replica you see that we start to redistribute more to the other one. So we really try to me to balance the load across
[24:49] replicas. Uh and if you add some more, no, we try to do that and the cash hits and cache misses are pretty low because you kind of always always go into the
[25:01] same replica. So that's kind of that's the idea. And um yeah, we spent a lot of time doing this and very happy about about it. like there's no one else, no other platform
[25:14] has this and I think everyone's having trouble at scale with AI applications today and I don't know if you've seen how like how other like anthropic and all the others are actually hosting their own uh harnesses and what not and
[25:27] that's becoming really complex because of this problem of like soft state in progress work things that are very expensive to rebuild and redo uh and no
[25:39] one has everyone's using round brown robin or sticky. And now we have sharded.
[26:11] Uh so a few things I want to go um on specific types of applications AI agents like while your the loop is running you can keep most of the conversation context in memory you don't need to go to the database restore every time
[26:23] there's a new request uh it really helps the performance and how you reason about what's happening because the way it worked in your laptop is going to be the way it worked up deployed. Uh but you cannot
[26:38] just rely on it. Never rely on soft state. You always have to have checkpoint state stored in the database. So every time you have reach a critical milestone of say okay from here on I
[26:50] don't want to repeat anything before you store that in the database because as you scale up or down or anything you're going to have possibility that your session is going to go to a different replica and the worst thing can happen
[27:04] is that you get a new session a replica replica says I don't know what this is about let me read from the database you can start from there so it's it's really nice uh that way uh Now
[27:28] please be careful to not mix user data within your app. uh that is yeah that is really important and I've seen people not actually doing uh creating the cache
[27:42] key per user ID and that can be a huge problem. Um a user also may have different sessions depending on how you architect your application. So you might want to have a session ID plus a use ID
[27:54] at the cache key. Um so be very careful with that. The other part is that every time an app goes down, rescales, does anything, anything that I am want to do with the app, there's a graceful
[28:06] shutdown. You have to listen to the sig term, there's a 30 seconds for you to do whatever is needed. state save the state and uh all the replicas they will pick up that session and we'll continue from
[28:18] the save state in the database
[28:30] workflows uh a lot of people are building workflows most you can also think that almost all agents can be thought of as a workflow but sometimes you want to do something different which is like I want to go here and then call this other thing and then come back start this and go to
[28:43] these other to to Slack send this email to whatever like there's in business there's a lot of business workflows and people build a lot of apps for that so what do we do intermediate cache result you cache intermediate results you don't
[28:56] if a user is getting the in progress you probably showing what's going on in the workflow in the UI you don't need to go to the database every time the user is getting that information and your client can just keep polling or you can keep streaming and it's it's very uh it's
[29:11] very fast and cheap to to keep that in memory. So that's okay to have that in soft state. uh but as soon as you complete the same thing for AI agent complete something and you say okay from here on I I don't want to repeat this
[29:24] that I want to save it record in the database keep it there and uh
[29:38] and write the app in a way that if a session comes and doesn't know anything about that workflow you can always restore from the database. Uh the other thing about workflow specifically is that um
[29:51] if for some reason you need to restart and redo some step, you don't want the stab to be have different results. You want the steps to be a potent because you're going to call something else outside and if you call something
[30:04] else outside with a different result, then you you kind of like missed. Uh so think about potency. think about like what how can I do to not do things twice to not repeat not send two emails to
[30:17] send two messages save two things it's very important for for uh workflows and the other one is like you can because we have sharded you can trust uh serverside events serverside events every so often
[30:31] they can disconnect and there's a very easy way to reconnect the browser to the to the cl to the replica to the app and you know that in their bricks apps that reconnection always go to the same instance and get access to the same soft
[30:44] state. Uh it really helps the the mental model of how to build application this way.
[31:00] Uh analytics a lot of apps are just dashboards fancy dashboards. Uh but uh if you think about dashboard there's a lot of like for example a user picks
[31:13] you know a a filter or pick something all of that you don't need to like really be redoing all that queries you know what what are all the for example what are all the the regions in my in my
[31:26] my data like that query needs to come back and you might have to be to keep that in memory and some of that state is actually can be used across sessions because it's just like it's the same region for everyone. So you don't need
[31:39] to keep quering the same thing all the time. It makes a huge difference in real dashboards. Uh the other some in dashboards specifically hard state is more like user configuration. What can
[31:51] the user did the user set up? Maybe they picked oh I prefer dark dark or I prefer to always have this filter applied in my dashboard. That's the stuff to save in the database. Uh number three here is like is
[32:05] aspirational but ideally you should not set data caches in the a memory that doesn't have that doesn't change. You can set its but
[32:19] ideally you shouldn't even have we should listen to changes in the data. Um that's something that I'm working with the SQL team to see if we can do that. But uh uh if you have things coming,
[32:31] data changing or permissions changing or something like that, you need to react to those changes and if you kept that data in memory is going to have you're not going to be uh up to date. Um so be
[32:44] careful about this. I don't have a good solution except from DTLS, but be careful. Uh the other one is that like a lot of a lot of exploration dashboards are actually you get something and you
[32:56] kind of like just slicing and dicing on a single data set. Most of the times it's very it's kind of dumb to go to the SQL house every time to every slice and dice. So you can do that in memory too.
[33:08] Uh especially for smaller data sets. Uh that's what everyone does next. So kind of analytics is kind of like this and you can see that like I see this every day and um if you start to
[33:23] think about I'm designing my application and everything has to go to the to the back end and that has to read from the database for everything that I ever need in that server everything is stateless complete stateless it becomes really
[33:37] hard to design this it's a lot of database queries and come back and then you're going to have problems in database how you going to scale data basically going to be the bottleneck. Uh that's what we were having in unit catalog.
[33:53] So this is kind of what I had. Um now open to questions you guys.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.