Rethinking Database Cost with Lakebase Autoscaling

Summary

Traditional OLTP databases charge for provisioned compute whether it is used or not, forcing organizations to overprovision for peak demand and pay for idle capacity—a pattern illustrated by Hallmark's seasonal spike problem where Mother's Day and Father's Day traffic required over 100% of baseline capacity for just two days per year.
Lakebase's serverless autoscaling model provisions a minimum baseline and scales compute up to match demand spikes and down to zero when idle, with storage and compute billed separately so organizations pay only for what they actually use.
Features like database branching with copy-on-write pointers enable CI/CD workflows using production data without incurring additional storage costs, and peak usage analysis shows most databases rarely exceed 10% of provisioned capacity under the traditional model.

Rethinking Database Cost with Lakebase Autoscaling

Traditional databases charge you whether you use them or not. Lakebase's serverless autoscaling changes that, scaling compute up to match traffic spikes and down to zero when idle so you only pay for what you actually use. In this lightning talk, you'll see how autoscaling works in practice and what the cost implications look like across workloads from development environments to production applications.
Talk By: Mike Jerome, Lakebase, Databricks ;

Chapters

FAQs

What is the Hallmark seasonal spike problem described in this video?

Hallmark's production database traffic peaks sharply on Mother's Day and Father's Day—two single days that together have about the same traffic volume as the entire Christmas and Thanksgiving period combined, but concentrated in a very short window. This forced Hallmark to provision more than 100% of their baseline capacity for a spike that occurs on just two days, leaving large amounts of paid-for compute sitting idle for the rest of the year.

How does Lakebase autoscaling work and what does it cost?

Lakebase allows organizations to set a minimum baseline compute and a maximum, then automatically scales up to match traffic spikes and down to zero when there are no queries. Compute and storage are billed separately, so when compute scales to zero, only storage costs accrue—eliminating the waste of paying for idle provisioned resources.

What is database branching in Lakebase and how does it help with CI/CD?

Database branching creates a copy of a database using copy-on-write pointers rather than duplicating the actual data, so a branch that points to production data can be created without incurring additional storage costs. This allows CI/CD pipelines to test against production-equivalent data without the expense of maintaining separate full copies of large databases.

How does Lakebase compare to traditionally provisioned databases on cost?

This video presents a cost comparison showing that most databases rarely exceed 10% of their provisioned capacity, meaning organizations are effectively paying for 90% idle compute under the traditional model. By scaling to zero when idle and charging compute and storage separately, Lakebase's cost model aligns spending with actual usage rather than worst-case provisioning.

Full transcript

[00:07] Good morning, everyone. Welcome to my talk. Today, we're going to be talking about uh the pricing of databases. So, if you're used to working with traditional OLTP databases, you're
[00:20] probably used to seeing something like this, where you provision the database, it has a certain amount of CPU, and it has certain amount of RAM that we'll call 100% on the charts.
[00:32] And then you have your usage, which hopefully is always under 100%. And you can see the green line here. Um it's going up and down during the day. Um but you can see, you know, roughly
[00:44] uh half of this chart is green and and half of this chart is red. So, I'm going to tell you a a true story. I used to work for Hallmark greetings cards. Um we were selling greetings
[00:56] cards online. And when I joined the business, this is what I expected uh the database to look like. So, you can imagine like it's January, there's like a little kind of like peak through the year, and then
[01:08] as you hit Thanksgiving and you hit Christmas, you see like this spike up, and not everyone sends their Christmas cards on exactly the same day. So, that's the shape that I expected.
[01:20] Um I was quite surprised when I arrived. This is actually what Hallmark's traffic looks like. So, there's a couple of things to notice here. There's two big spikes. That's Mother's Day and Father's Day. They
[01:33] happen on one day, and their spike is about the same area under the curve of the whole of Christmas and Thanksgiving, but it happens in a really, really short period of time. Um and because the spikes are so high,
[01:46] we actually had to go for more than 100% capacity. So, if that original uh system was, say, 100 gigs of RAM, we actually needed to provision 150 gigs of RAM as an example. And what
[01:59] you can see here is there's actually like a a ton of time where there's a red area. And red is effectively resources that are available to you, but you're not using.
[02:14] And so when you start to think about other database environments, so that was the production that we were looking at. This is what a UAT environment might look like. So it's far, far less busy. Um you can see people are coming in in
[02:26] the morning, they grab their coffee. Seems like peace peak of the testing here is happening in the afternoon. But there's even more red on this environment. So if this was a staging environment, ideally you'd want exactly the same specs as production so that you
[02:40] could check that workloads and indexes, etc., were working exactly as you'd expected. And then you have CICD type environments. Again, you're provisioning that full-size box, and you're probably
[02:52] doing like bursts of tests as you do your CICD, and then that box is doing absolutely nothing. Um so it's a very different shape, but there's still a lot of red on the screen versus green. So that's uh resources that are available
[03:05] to you that you're not using. If we then think about something like a high availability production environment, you're going to have that same load on the main. And if all's going well, you
[03:17] have two followers, and they're largely dormant, just kind of waiting to take over in the event of a catastrophe. And again, when you look at those three, there's a lot of red on the screen. So there's a lot of uh
[03:29] resources there that are not being used. When I've been saying resources available, what I really mean is what you pay for. So you were paying for all of that red,
[03:41] even though you weren't using that red. So the same graph again, I've just changed the label on the access. Really, it's what you pay for is the red line, and your usage is the green line, and
[03:55] that is how a traditional provision database works. With Lake Base auto scaling, it looks more like this. So, we will provision the minimum
[04:07] resources that you need to to deal with your current workload. So, we'll scale up and scale down. And when you hear auto scaling, I think a lot of people do think about the scaling up as
[04:19] being the main benefit, but really the the trick with auto scaling is is scaling down. And like I said, we will give you the minimum resources that you need without going too small. So, that's the shape of how it looks when you're on
[04:31] Lake Base auto scaling. So, the charts I've shown you so far are kind of idealized charts, but this is actually a real-world example of how that would work. So, this is taken off
[04:44] some customer data. What you can see here is there's a baseline. So, this customer has actually set their minimum baseline to be one compute units. So, you never see it fall below that. But you can see
[04:57] as you go through the day, the the traffic's spiking up and down. There's one big spike towards the end of the day. We've been cautious, and we've we've provisioned a lot more than necessary because we've seen that like increase in
[05:09] load, and we want to make sure it's not going to continue. And then as quickly as we can, we can bring it down. So, this is actually pretty close to the to the idealized chart. This is really what it looks like. And then I kind of have a
[05:22] very similar example. This is real-world auto scaling with a customer over 1 week. So, you can really see the the peaks and the troughs there. And I think the thing that's interesting about this example is it's really clear where
[05:35] Saturday and Sunday are. The two lower lumps at the end of the chart there. So, again over the weekend, you have a lot of provision systems where you'd see you'd see more red, more resources that you're paying for that you're not using.
[05:54] So, I was also working with a customer who is an airline at Databricks. And I did some work with them to look at their provision databases state over a 7-day period so we could actually understand what their usage was. They
[06:06] were really interested in understanding if they moved to lake base auto scaling, will it actually save them money? So, we we ran diagnostics over 7 days on their existing database and
[06:18] the facts were actually astounding and also typical. So, we found that 59% of their instances had no activity whatsoever in the 7-day period. So, they were sitting there fully provisioned,
[06:31] being paid for, and they hadn't run a single query. And this is sounds surprising, but actually it's really typical. There's lots of databases that have just sitting around not being used.
[06:43] A project that was started and then abandoned, an old CI/CD environment, a development environment. And I guess what I want to uh really kind of show the benefits of here with auto scaling is we can scale to
[06:55] zero. So, in any of those environments where there's absolutely no traffic, we scale to zero. There's no compute cost whatsoever. So, 59% of this customer's instances would have no compute cost at
[07:07] all in that 7-day period. And I think the other thing which like underscores my my Hallmark example is that when we looked at their peak usage, very, very few of them were ever actually going over 10% in usage. Only
[07:21] about 9% of them. So, it's really common to kind of work out what you think your peak was going to be and then you put your finger in the air and you add a few more gigs of RAM or an extra core or two just to kind of be safe. Kind of reminds
[07:34] me of when I'm doing my washing, I always put like an extra chunk of washing powder in the washing machine and hope that the clothes are going to come out cheaper. Um and so we can't save 100% on the compute costs, but over-provisioning is
[07:47] so common that I am very confident that auto scaling in most cases can reduce cost. So, I also want to talk about storage. So, we separate the storage and the
[08:00] compute with auto scaling. When you think about a traditional database, and I'm going to use
[08:12] have like three copies. So, you you assume you copy your compute and your RAM. You're also going to want to have large disks, particularly if you're doing something like load testing, you're going to want fairly decent copy
[08:25] of the production database to check performance characteristics. So, you can't just rely on seed data. Uh, you also have the challenges of of keeping your seed data up-to-date. And with auto scaling, we we have branching,
[08:37] which makes that problem easier. Although, it's not necessarily related to pricing. Um, and then the other thing is high availability. So, in a high availability world, again, you're paying if you've got two hot standbys, you're paying for
[08:50] three lots of storage in a traditional setup. With Light Base auto scaling, we have separated the storage from the compute, and that gives us a ton of
[09:02] advantages. So, the first thing is if you create a branch, it doesn't actually increase your storage usage at all. So, imagine the environment where you have a production environment, and then you want to do
[09:15] uh a change, maybe you want to like do a database migration. You can branch production, and we'll for round numbers, we'll say production's 1 TB. You branch production, you then have a full copy of your
[09:27] production data, and you can run tests on that in isolation. And all you pay for in that branch, that test branch that you're using, is the changes. So, um this this happens because when we
[09:40] when we do branching, we have something called copy on writes. So, we we create a pointer to the original data rather than actually duplicating the data. And then, we just measure the differences along along the time. So, if you're doing something like CI/CD, it's
[09:53] amazing. You can get a copy of production. You can run your CI/CD across actual real copy of the data in isolation. Um and then the great thing is you just throw it away. So, your storage costs would just be likely a few
[10:07] meg of changes during the CI/CD run. And the same with the compute costs, it was just while you had the CI/CD run. And then you throw it away. And the next time you want to do CI/CD, you create a new branch. And you're just paying for
[10:19] that like ephemeral environments. Um Also, with high availability, the way that we charge for storage with Lake Base Auto Scaling is you pay once. Um
[10:31] so, in the background, we are replicating the data around to deal with the a failover situation. But we're not passing that usage charge on to you. So, you can think of like having like three computes are reading from the same
[10:43] storage layer. You're not paying storage for every single HA instance that you have running. Even if you have six instances running.
[10:57] So, to kind of give a picture to that. If you imagine a a typical provisioned environment on the left-hand side here. And we're going to go with 1 TB of storage for for round numbers. You end up having provisions
[11:09] 3 TB for your production and your two high availability nodes. Then you have a staging environment and you'll have a a TB provision there. You have a TB provision for your CI/CD. So,
[11:21] you're up to 5 TB in a traditional system. Even though you only have 1 TB of data. And in contrast, with Lake Base Auto Scaling, you have the 1 TB in production, so the usage is the same
[11:34] there. The two HAs, they don't consume any more storage as far as you're concerned. And then in this example here, I've created a staging branch, and you can see I've kind of half filled the square
[11:46] to say this when we've been in in staging, we've we've made some changes, we've added some rows. And so what you're paying for there is just the changing just the changes. And then in this example, I've branched staging off and
[11:58] and created two CI/CD environments as well. Um and again, you're just paying for the changes. So the the usage is 1 terabyte plus the fraction of changes.
[12:12] So I wanted to just give you some tips if you are working with like base auto scaling on how to keep the cost down or the usage down. So the first thing is whenever you start a new project, um by default, scale to
[12:25] zero is enabled, and I would leave that on. And so many new projects, they get abandoned or they don't get used, and you don't kind of don't have to worry because if they're not being used or they're just in development, developer's
[12:38] gone uh on vacation for a week, it scales to zero, and there's no charge. Um so that's a great default, but as you start to move into production, you have some kind of baseline, and um I when I
[12:50] showed the the real example earlier where the customer had set the baseline to 1 CU. There might be cases where you don't want to go all the way to zero, um even if technically there's no usage. And the
[13:03] reason why is when you do go to zero, you do lose your cache in RAM. And so you can set a baseline, so you can say actually I just never want to drop below 1 CU, or if it's a very heavy site, maybe there's another number that
[13:16] you want to set as a minimum threshold. And if you do disable scale to zero and you set that minimum, we'll actually reduce the cost by 25% of the computes below the minimum. So again, if you're
[13:29] on a really big app and you know it's never dropping below 10 CU, then disable scale to zero, sit them in set them in to 10 CU, and everything below that will receive a 25% discount on the computes. And then
[13:42] whenever that app needs to scale above that minimum, you would pay the standard auto scaling prices. So that's a good tip when you've actually gone into production and you're happy to have an established baseline. Um the other tip I want to want to call
[13:55] out is create Postgres indexes. So there are other talks here about AI that can help you create the right index, but they make a huge huge difference to the amount of compute that you need to consume to serve your queries. So it's
[14:08] really worth keeping an eye on them and they can drastically reduce the computes. That allows us to scale down and ultimately reduce the costs. Um and the last thing is if you are an
[14:20] environment where your you have high availability, you're paying once for the storage, but because we're keeping those two other nodes hot, you are paying for the compute on those other nodes. And that means if we fall over, like everything's
[14:33] in RAM, it's ready to go. But they're actually available to you to connect to directly. So if you have read-only queries that you don't want to affect the main production system, you can run those against your hot standbys in HA.
[14:46] You're already paying for them. Potentially there's compute there and you can take advantage of this. So that's it. That's that's all my talk. I'm very happy to take questions at the
[14:59] end if anyone's got any questions. I'll do it in person cuz I think it's people can't hear the the question being shouted out, but I'll I'll be here on the on the stand at the end. Uh and then I have one one last thing. If you're
[15:11] using Lake Base, we would love your feedback. And it's a paid study. So this is not about the conference. This is about Lake Base itself. We We a paid user study and we'd love to
[15:23] get your feedback. So, thank you. That's all.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.