Sponsored by: Keebo | Teach Your Warehouse to Drive Itself

Summary

Keebo uses reinforcement learning to continuously optimize Databricks SQL warehouses in real time, controlling three levers—warehouse size, idle time, and cluster size—within customer-defined guardrails to maximize performance while reducing cost.
A mid-market healthcare technology company achieved a seventy-five percent cost reduction in two weeks with zero performance impact after deploying Keebo on their Databricks SQL environment as usage and costs were growing faster than headcount.
Keebo's outcome-based pricing model charges one-third of verified savings, using a documented baseline, actual spend, and SLA compliance metrics to prove the value of each optimization cycle rather than relying on projections.

Chapters

FAQs

How does Keebo optimize Databricks SQL warehouses?

Keebo uses a closed Q-learning feedback loop—observe, act, measure, update—to learn a warehouse workload's unique patterns and continuously adjust three levers: warehouse size, idle time, and cluster size. All actions operate within customer-defined boundaries and performance SLAs, ensuring Keebo never degrades query performance while optimizing cost.

What results did the healthcare technology customer achieve with Keebo?

A mid-market healthcare technology company that had seen data infrastructure costs grow faster than usage or headcount deployed Keebo and achieved a seventy-five percent cost reduction within two weeks. The savings were delivered with zero performance impact while the company continued to onboard new use cases and applications.

Why do data teams struggle to control Databricks SQL costs on their own?

Cloud data platforms like Databricks have dramatically lowered the adoption barrier, causing more users, applications, and complex pipelines to query data, while operating costs grow faster than expected. Data teams typically fall into three ineffective patterns: reactive firefighting, building internal cost-tracking tooling, or deferring action until contract renewal time.

How does Keebo's pricing model work?

Keebo uses outcome-based pricing, charging one-third of verified savings. Savings are calculated using a documented baseline cost, actual spend after optimization, and SLA compliance metrics, making the value delivered measurable and auditable rather than based on projected estimates.

Full transcript

[00:09] I want to start with a story. Um the story is about a customer of ours, a midm market healthcare technology company. For them, data is not just about reporting. It powers their dashboards, internal apps, and a growing number of AI and analytics workloads. So
[00:23] the data team did exactly what they were supposed to do. They made data easier to access, right? So more users came on more applications started depending on data bricks SQL. So from the business
[00:35] size business side this was a complete success but then the success turned into a new problem. So usage was growing well cost was growing faster headcount was not growing and on top of this the
[00:49] business still expected them to onboard more use cases more applications obviously within the same budget. So turns out they're not alone. Actually, this is not just a one-off problem. What platforms like data
[01:02] bricks, data clouds like the likes of data bricks and and and snowflake in particular have done is that they've done an amazing job lowering the adoption barrier uh to accessing data and allowing everyone pretty much across
[01:14] the organization to tap into data and innovate very quickly. Uh so as a result, more users and applications are tapping into data. They're querying more data sources and they're creating uh more complex pipelines. But obviously
[01:27] all of this means that the cost of data infrastructure is going through the roof. So in a way data clouds have turned a capex problem into an opex problem. They've lowered the adoption
[01:39] barrier but they've significantly increased uh the operating costs. So when dealing with this kind of situations I've seen data teams typically fall into one of three camps. Uh the firefighting camps, engineers
[01:53] spend a couple of sprints trying to clean things up before things get out of hand again. The dashboarding camp where there's always a team who's perpetually building internal tooling trying to figure out where cost and where money is
[02:05] going. And uh finally the wait until renewal time camp, right? Uh by the way, just a quick show of hands. How many people have been in the first camp? How many people have been in the second camp? Don't worry, this is a judgment
[02:18] free zone. What about the third camp?
[02:30] marketing team decided it was too honest for the slide, so I went with the with the plight version. But the bottom line is this. None of these approaches scale in the long run. So, fortunately,
[02:43] the story we started with had a happy ending. Two weeks later, that same customer was able to use KBO uh to cut their cost, their data bricks bill by um almost 75% as you can see from the
[02:55] charts with zero performance impact. So that is the result. But the more important question here is how do you get an AI agent to do this safely and this is going to be what the rest of this talk is all about. My name is
[03:08] Bzaman Muzafrey. I'm a co-founder of KOAI. I'm also a professor at University of Michigan Ann Arbor. Um, I've spent the last 20 plus years of my career at the intersection of AI and database systems. Um, and a lot of what myself
[03:23] and my PhD students have focused on over the years comes down to one question. How do we use AI to make data infrastructure faster, cheaper, and more autonomous?
[03:35] So, how do you get how do you actually get uh to build an AI optimization agent that seems trusted enough to put it in production? First, I'm going to talk
[03:47] about the training data. Then, I'll talk about what are the actions that the agent should be allowed to take and then how the agent can actually learn from feedback from the environment. And then finally, we'll talk about how to
[03:59] validate the savings that the agent is actually delivering. Please consider this a road map of the next 15 minutes of our time together. All right. Uh first training data. So one of the
[04:12] design decisions we made from early days was that for enterprise teams to be willing to adopt a solution like this, we shouldn't require any access to sensitive data. In other words, we
[04:24] restricted the agent to purely learn from uh performance telemetry metadata. essentially things like how many queries have arrived, what times they arrived, how long they took, how much they cost, how many buy, stay, scan, things of that
[04:36] sort. And interestingly enough, turns out you don't even need to see the query text. All you need to know is whether the query that's running right now is something I've seen or I've seen similar to in the past or it's a completely new
[04:50] query. And you can actually do that by using a class of hash functions and and carefully designing your hash values without ever seeing the plain query text. So the first design decision here is that our training data is restricted
[05:04] to nonsensitive performance telemetry metadata only. All right. So that's our training data. What about the actions? What are the uh the second design decision is what are the actions that
[05:16] the reinforcement learning agent that we're going to be talking about is allowed to take. Um as someone who spent two decades in this space, I can tell you a reinforcement learning agent is as useful as the actions that you allow it
[05:28] to take. Ideally, you want your actions to be impactful, measurable, and safe to bound. They need to be impactful enough to move the needle in terms of cost and performance.
[05:41] You want them to be measurable in the sense that you want to be able to see what happened after you took those actions. And finally, you want them to be safe to bond so that customers can define a range they're comfortable with
[05:53] for each of those actions. So they can actually delegate it to the to the agent. Now when it comes to data brick SQL, those criteria essentially bring us to three natural knobs. The warehouse
[06:05] size, how much compute should we use? the idle time, when should we suspend our resources versus let it continue and stay warm for the next queries and cluster size, essentially how much
[06:18] concurrency should be available. But the thing to keep in mind here is that for all of these knobs, the customer is still in charge of controlling which of those actions they want to allow to the the agent to take
[06:31] on a for each of their warehouses and what is the range they're comfortable with the agent take uh taking actions. For example, this particular warehouse. I'm okay with the agents moving it between a small to large. So the customer is still in control, right? So
[06:44] this is not about AI can do anything. This is about autonomy within customer defined boundaries. So let's take that uh idle time as a as
[06:56] an example, right? Uh on the surface that feels like a very simple question, right? If no one is using the warehouse, just suspend it. But the bottom line is that if you do
[07:08] this, if you have a situations where the queries come back soon, suspending a warehouse too early can actually incur startup cost, call cache problems and latency and queuing and all sorts of
[07:20] other risks, right? So the problem is not really what is the action that's going to save us the most money in this minute but rather what are the actions that are going to give us the lowest total cost
[07:32] over time while staying within the users provided performance SLAs and this is exactly the kind of decisions that reinforcement learning is perfect at making
[07:45] so now we know what's our training data it's metadata only we talked about what are the actions in the context of data bricks SQL that you know we want the agent to be able to take and how the customer is still in control of which of
[07:57] those they want to turn on for which warehouse and what is the allowed range. Now let's talk about how you can allow the agent to learn safely from feedback. Here's the closed loop. You observe the
[08:10] state. You choose an action from the allowed list of actions based on that state. And then you act on that action. Then you measure what happened after you took
[08:22] that action and then accordingly update your policy. If the action led to savings without hurting performance, the system learns from that. And if the action caused performance problems, SLA
[08:34] violations, queuing, latency, all of that, the system learns from that as well. So essentially every action becomes feedback for the next action. Now this is the Q-learning version of
[08:48] this closed loop. In essence, the Q here is the value of taking a particular action A in a particular state of the system S. And
[09:01] the job of the Q-learning algorithm is to learn the estimated value of taking every action for every particular state over time. So the way this does it is that whenever it takes an action a it
[09:14] measures the outcome of that in form of a reward function and then uses that reward and that feedback loop to update its estimate. Okay. So now the important part here in this equation
[09:27] there's a lot of there's a lot of math here but the important part to pay attention to is the reward here doesn't need to be just how much money we save. The reward could be a combination of the savings latency queuing SLA penalties
[09:39] all of that. And that's exactly how your learning is not just going to lead to what's going to be the cheapest. It's going to be how am I going to get the lowest cost to deliver whatever performance goal that the user had over
[09:52] time. Right? So what's the lowest cost of delivering the goals that the uh user has specified here? Now when we talk about autonomous optimization, I want to be very clear
[10:05] here. Autonomous does not mean unbounded. The user still gets to define their own performance guard rules and performance SLAs. They get to choose whatever metric they care about. Think average latency, 99
[10:20] percentile latency, Q time, you know, what have you. You get to also define a threshold and define um an evaluation window for it. And then given those,
[10:32] KBO is actually going to start learning. So Ko's job is not come to come in and define for you what good performance means for you. Every customer is different. The customers get to define what they consider good performance. And
[10:45] Ko's job is to figure out what's the lowest cost possible to deliver the goals that the customer uh defines over time while respecting their performance needs. In other words, the human defines
[10:57] the goal, the AI figures out how to deliver it safely. Okay, so that's the closed loop and the guardrails. But the final thing that we need to uh address here is that okay, if we claim savings, how do we verify the
[11:11] savings? How do we know that hey, it wasn't just that maybe half of my team was on PTO this week. Maybe I have a really bad intern who writes terrible SQL queries and he was sick that day. Like how do we know that the cost that
[11:23] went down was because of the actions that the agent took? And this is really important because if you cannot verifiably attribute the savings to the actions of the agent, number one, you cannot trust its learning and number
[11:36] two, you cannot use it as a basis of pricing, which is what I'll I'll come back to in a second. So for your savings to be verifiable, you need three things. a baseline what would have happened if
[11:49] you didn't have kbo the actual usage what did happen when you had kbo and finally the SLA checks as a result of those actions did we stay within the SLA boundaries that the user
[12:01] specified or not if you don't have any of these three then you cannot claim savings so here's a concrete example let's say for one of your warehouses your one of your reporting warehouses on data brick SQL at 2 am the AI agent
[12:14] determined two things Number one, this warehouse is underutilized. Number two, downsizing it is not going to violate your SLAs. So went ahead and reduced your warehouse, which which you had set it to be large by default,
[12:27] reduces it to medium at 2 a.m. Let's say 12 minutes later, the agent predicts or observes that cues are about to build up. So it automatically reverts you back to your default size, which was a large.
[12:40] So that area that I've marked in yellow is objectively arguably how much money you saved because of the direct action of the agent. In other words, what you've saved is 12 divided by 60. The
[12:53] agent saved you 12 divided by 60 times the rate difference, the hourly cost difference of your default large, which is what your baseline would have been, and the uh medium.
[13:05] Okay. So when this works, it's not just about lowering your data bricks bill. It changes how data teams operate. Instead of spending spend after spend with
[13:17] manual optimizations, they get all this time back to work on productive work. And instead of optimization becoming a once a year uh exercise, now your data teams and they your platform teams have a constant assurance of efficiency.
[13:32] More importantly here, instead of just taking some vendor's claims um on blind faith, you get to actually use those verified savings as a basis for pricing.
[13:44] So you take out the risk from the pricing. So for example, Chebo in our example, we charge customers one-third of whatever we save them. So if we save them 3 DBUs, we keep one DBU, we pass two DBUs back to the customer. If we save them zero, we charge zero. It's
[13:57] never happened. But you get the idea, right? So a solution like this you would not actually need to um need a budget. KBO creates its own budget and gives you
[14:09] uh gives you some as well. So I mean don't get me wrong lowering your bill is important but I think the bigger win here is that optimization becomes something that the platform does continuously instead of something that
[14:21] your engineers have to keep coming back to manually. So by now hopefully you've seen that to create an AI based optimization agent
[14:35] that teams and enterprise teams would trust enough to put in production, it's not just about your model tuning. You need to think very carefully about three dimensions. Number one, you need to remove the security risk. This is why
[14:48] for example, we don't access any customer data. So by definition, it's a no-brainer from a security perspective. Number two, you're going to have to address the governance risk. The customer should still be in charge of
[15:01] what actions, what range are allowed for which warehouse. And they get to dictate their own performance goals instead of the AI deciding what's acceptable performance. And finally, and quite honestly, just as
[15:13] importantly, if it's if if not more so, you also need to come up with a solution with with with which incurs no pricing risk. So to do that you need to have a strategy for verifiable saving that
[15:25] helps your agent learn faster and more reliably and more safely but it also reduces the risk and aligns the customer and the vendors um incentives. So if any of you guys are dealing with data brick
[15:38] SQL cost performance manual tuning would love to chat with you. I was just informed by the organizers you guys cannot ask questions because there's no mic. Um, but I'll be over here happy to chat if you have any additional
[15:50] questions. We also have a booth uh booth uh 615 over there. Uh you guys can go meet the team. But would love to chat with you and thank you so much for uh joining the call.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.