Skip to main content

Cost Governance at Scale @ Databricks

Summary

  • Databricks' internal Data Platform team manages one of the world's largest Databricks environments and uses System Tables, automated tagging policies, and budget tracking to attribute cloud costs across hundreds of internal teams.
  • The cost governance platform is built on four ingredients: integrating billing, usage, audit, and HR data; automated workload ownership mapping; Genie, Apps, and alerts as the interface layer; and AI agents for proactive optimization.
  • AI agents now auto-generate cost-reduction pull requests and perform reactive root cause analysis on unexpected spend spikes, turning cost management from a manual fire drill into an automated, data-driven process.

Cost Governance at Scale @ Databricks

Watch: Cost Governance at Scale @ Databricks
“With great compute power comes great financial responsibility.” As the internal Data Platform team, we manage one of the largest Databricks environments in the world, which means we also manage one of the largest cloud bills. this video unveils our internal FinOps practices. We will demonstrate how we use System Tables, budgets and tagging policies to attribute costs to hundreds of internal teams, identify waste and enforce accountability. Learn how to implement the same governance controls that we use to keep our finance team happy while empowering 10,000+ employees to innovate.
Talk By: Arpan Ghosh, Engineering Manager, Databricks ; Bogdan Kyryliuk, Software Engineer, Databricks ;

Chapters

FAQs

What are Databricks System Tables and how are they used for cost governance?

System Tables in Databricks capture billing, usage, and audit data across the platform, providing a unified foundation for understanding who is spending what on which workloads. Databricks' internal Data Platform team joins this data with HR and team ownership information to attribute costs precisely to hundreds of internal teams.

How does Databricks attribute cloud costs to individual teams?

The cost platform uses automated workload ownership mapping and tagging policies to associate every Databricks job, SQL query, and compute resource with the team that owns it. When a cost spike occurs, the platform can immediately identify the responsible workload and team without manual investigation.

How are AI agents used for cost optimization on Databricks?

Proactive optimization agents analyze usage patterns and automatically generate pull requests with specific cost-reduction recommendations, such as right-sizing jobs or switching to serverless compute. Reactive debugging agents can also perform root cause analysis on unexpected spend spikes without requiring engineers to investigate manually.

What does a practical FinOps program on Databricks look like?

The Databricks internal FinOps approach combines four ingredients: integrated billing and usage data, automated ownership mapping, an interface layer using Genie and Apps for self-service cost queries, and AI agents for automated optimization. The result is a system where engineers get cost visibility without disrupting their roadmap and finance gets reliable attribution without manual reconciliation.

Full transcript

[00:11] Picture this data bricks a year ago. The company is in hyperrowth. Headcount is doubling. And the cloud bill is growing even faster. The data platform team is heads down, sprinting towards a launch, getting
[00:24] ready for data and AI summit, the same conference we're sitting at right now. And somewhere in the midst of all of that, the conversation you're about to see is happening. Not once, every week.
[00:39] Bogden, I'm from finance. We have a huge emergency. The AI workspace bill has doubled 100K in just the last week. Are you aware of this? 100,000? Are you sure
[00:51] it is us? Did you talk to a data platform? >> I did. They said it's your workspace that's causing the spike. They send me to you. >> There are five different teams running their workloads right there. How do you know it spiked?
[01:04] >> All I know I have a spreadsheet with your name on it. The CFO needs a remediation by the end of the week. >> By Friday, I have a feature to launch. I cannot drop everything. >> Bogdan, it's 100k adding up week over
[01:18] week. We have to do something. >> I know, but even if I look, I can only see one job at a time. I don't have access to the usage or billing data. This is locked down to
[01:30] admins. I don't even know who admin is. How do I debug this? Look, I don't care. Figure it out. This is a cost emergency. We're bleeding money. This takes priority over everything else.
[01:44] >> I guess I'll talk to my manager and delay a feature launch. I have no idea how long it's going to take. >> Well, then you better get started. >> I will. >> Thank you. Sound familiar?
[01:57] It's a classic standoff. Finance owns the bill, platform owns the infrastructure, and users who are supposed to own the spend have no clue.
[02:09] The net result, the road map suffers. Wouldn't it be great if we could just point an AI agent to go off and root cause that 100k overage without interrupting anyone?
[02:22] That's exactly where we want to take you. But before that, a quick poll. How many of you have been in a similar cost conversation recently?
[02:34] Finance showing up in your inbox asking about an unexplained bill or suddenly asking you to shave 30% out of the blue. Most of the room.
[02:46] And when that conversation happened, how many of you knew where to start looking? where to start answering those questions when the data is split in six different places and you have no idea how to even
[02:58] get to it or join it. >> All right, not alone.
[03:11] your team get completely thrown off your road map? days lost, sprint slipped, launch delayed, a fire drill that wasn't even yours to begin with.
[03:24] I think every one in this room has been in this situation. This isn't just a data bricks problem. It's a universal pattern. We understand this pain very intimately
[03:36] because it's our day job to fix it. Hi, I'm Arpon. I'm an engineering leader at Data Bricks. My team owns the company's internal cost platform. This
[03:48] is Bogdan. He's the technical lead on that team. Our platform manages cost for the biggest data bricks deployment in the world. A year ago, we were feeling every bit of
[04:01] that pain you all just acknowledged in that poll. We lost multiple quarters of feature work because we had to divert teams to chase down cost efficiency mandates from
[04:14] the top. We had multiple cost incidents and investigations that distracted engineers from their actual work. And each of these incidents took anywhere from two to six weeks to resolve.
[04:28] Not to mention the hit on our user experience when we were forced to place arbitrary limits on usage just to get a handle on cost. Let's break down the underlying problem
[04:40] together. Revisiting that conversation, finance at Bogdan's door, an unexplained bill, the AI workspace. When such a bill shows up, there's three
[04:53] simple questions that follow. What caused this overage? Who owns it? And why did it happen? Sounds simple enough, but let's start
[05:05] digging in and see how we actually go about answering these questions. First, the bill itself. These are the dollars. This typically lives in some sort of cloud cost management SAS. Think
[05:18] cloudability. Or worse yet, it's a spreadsheet that finance updates at the first of every month.
[05:30] Now you need to find out your units. How are you built? Your SKs, DBUs, gigabytes, compute hours, all of that is locked away in your cloud console. Think AWS cost explorer.
[05:44] And finally, which workload drove this overage, which query, which dashboard, which job? All of that information is locked in data bricks in system tables.
[05:58] So maybe at this point you're starting to build a picture of what actually drove this overage. Now we get into the who. Which user actually owns this workload
[06:10] or better yet which service principle represents that user that is running this workload. That answer is also in data bricks in a different table audit logs. But a user on their own is of no good to
[06:23] finance. They need to map them to a team to a cost center and that is trapped in workday. Yet another system owned by a different team with a different login.
[06:37] So maybe we're starting to see which workload caused this overage and who you need to chase down. The final question, why did this happen? What truly caused it? That answer, if
[06:51] you're lucky, is trapped in your Git commit history, your CI logs, but more realistically in a Slack conversation from 3 weeks ago.
[07:06] We just saw six different systems owned by six different teams, six data silos, and no join key between them. And even if you figure out how to navigate these silos, the data is often stale or just plain missing.
[07:22] This is exactly what we mean when we say cost is a data problem. It is not a finance problem. It is not a process problem. It is a pure data integration problem disguised as a finance question.
[07:38] And mind you, that was just one incident. one workspace, one week. Imagine having to do all of this all over again next week when you have a new
[07:50] workspace, a new workload. You run into the same data silos and the same data gaps. Rinse and repeat for every unexplained bill you get, and you're describing
[08:02] someone's job, not an incident anymore. So the data is clearly fragmented but underneath that even the people dealing with cost are fragmented.
[08:15] You have three different parties with three completely different incentives. On one end there's finance. The thing they really care about is predictability.
[08:27] They need to walk into that board meeting and say we spent exactly how much we forecasted. They want to see the cost numbers move only in one direction and that's down.
[08:40] On the other end, you have customers, users of your platform. Think your product engineers, data scientists, MLAs. All they care about is shipping features
[08:52] fast. They want their queries to run faster. They want bigger clusters to train the latest and greatest models. And in doing all of this, the last thing on their mind is cost. because they
[09:04] think it's someone else's problem. And in between that, you have the platform teams being squeezed from either side. Finance wants it cheaper. Customers want it faster.
[09:19] These misaligned incentives create two very real walls separating these three teams because no one has a complete picture of cost and no one is talking the same language.
[09:37] Now imagine this fragmentation at scale. We support more than 10,000 employees running more than 100,000 workloads across hundreds of regions and all three clouds. At that scale, cost really adds
[09:50] up fast. So we did what platform teams are great at. Cost is a data problem. So we built a data platform to solve it. Bogdan
[10:03] architected that platform and he's going to walk you through how we built it using data bricks. Take it away Bogna. Thank you Arpon.
[10:18] Cost is a data problem and you just heard why. Now let me show you how we solved it. It took four foundational ingredients. Number one is data. things you need is a detailed data on every dollar spent, usage, attribution,
[10:30] ownership. Number two is processing. Raw billing usage logs are not enough. We need to join, clean, dduplicate, clean all of the data and publish govern tables that both finance and engineers agree upon. Three, instrumentation.
[10:44] Every workload tied to the team and a budget. Because if data exists, but without this, no one can act on it. And number four, observability. Right view to the right audience in the right time.
[10:56] Dashboards to read apps to act alerts to page and reach cost owner when usage crosses the threshold and agents to automate the rest.
[11:08] And best part, if you're on data bricks, you have all of those four ingredients built in already. No need to build a custom data platform. Let me show you how we put all of them together to construct a single cost
[11:20] platform. We have sources on the top, our cloud bills, system tables, audit logs. We have governed cost pipeline and the data in the middle. This pipeline turns raw usage into governed cost tables with contracts built on top of
[11:33] data bricks. And we have interfaces on the bottom. Dashboards, apps, and agents and data flows down from sources through the pipeline to interfaces. We use them to build a reliable, easy to operate
[11:46] cost platform. Layer number one sources or a data layer. Remember cost is a data problem and we have data right here. Data bricks gives you a system tables managed delta
[11:59] tables that autoop populate usage billing audit compute metadata all of it. No need to build a pipeline. No need to maintain a schema. Just queries the data. Let me tell you a story. In my prior job I had to instrument spark hive
[12:12] and presto clients by hand. I wrote dozens of PRs, changed hundreds of files and I and wrote all of this metadata into SQL comments and job names. It was brittle, unreliable, but I had no better
[12:24] option. And anyone could easily bypass my implementation. Now raw instrumentation data is already available in the system tables for you. And if you think from first principles,
[12:37] cost is simple. It is usage times by unit price. Most of the data lives in the system tables that you need to solve this equation. Those are billing usage gives us platform activity measured in
[12:49] DBUS. List prices gives us ADBU unit price. Access audit helps attribution and compute tables connect our usage back to underlying cloud resources and show us utilization.
[13:03] And for us, system tables cover roughly 80% of the what we need. The rest comes from cloud uh bill cloud bills and storage inventory. But if you own serverless workspace, you are covered
[13:15] completely by 100%. So cost is a data problem and data is already here. If you want to learn more about system tables, feel free to scan a QR code. It'll take you to the documentation.
[13:35] Now how do we actually get to the cost governed tables? The foundation is our cost pipeline. Let's talk about inputs. Billing is the most critical one. However, let's take a closer look at the cloud billing SKs. They are AWS or Azure
[13:47] instances, storage, data transfer, net gateway, cross a traffic, DBUS, cloud specific charges, hundreds of them. Asking every product team to forecast along all those dimensions is impossible. They do not want to
[14:00] understand all this complexity. They want to know three things. What drove the bill? who owns it and what action to take. So as a platform we abstracted away all
[14:12] of this complexity making an important architectural decision for computing network we use a single unit data platform DBU instead of exposing network charges separately like net gateway or cross transfer we allocate network
[14:26] proportionally to the compute usage to our customers it's not a perfect attribution but it's easy to understand and easy to predict for storage we we just use gigabytes we expose we do not
[14:38] expose billing details that like that are like storage tier or API traffic. We leave those details to the platform to handle. Our customers just care about two units DBUS and gigabytes. It's a simple mental model for all of them to
[14:51] use. We also made a second important architectural decision. Our tables are split into two. One of them is usage and unit price. Usage for internal customers
[15:03] and unit price is for platforms. Product teams know what they want to run. How often do they want to run and how much capacity is they need? They're responsible for managing and forecasting their own usage. Platform owns the unit
[15:16] price and responsible is in making cheaper the platform cheaper and more efficient to run through serverless photon butter defaults and efficiency work. And all of this make conversation much much healthier. Instead of arguing
[15:29] over one giant bill, we can separate two questions. Did the usage go up or did a platform became more expensive to run? To build usage and unit cost tables, we pipeline joins system tables, cloud
[15:41] bills, tags, teams metadata. This results in a trusted data that finance and engineers both agree upon. As you have heard, ownership is a key
[15:53] part of the pipeline. To get ownership, we need contracts. Data engineers in this room know a shift left concept, right? Catch problems at a definition time, not at a runtime. A schema issue
[16:05] caught at a code review is cheap. Same issue caught at a runtime at night is expensive. We apply exactly same principle to cost. Every workload declares its cost contract, a team order before it even runs. A goal here is to
[16:18] shift responsibility to the left to the users who create the jobs. As you can see in the slide, this is how it looks in practice. A real data bricks job from our platform that carries two ownership signals. team tag and service principle.
[16:33] Explicit owner tag is a simple key value label you can attach to a job cluster warehouse table or a model. In our case that points to an immutable team UID that in turns maps into a team
[16:46] definition, a name, a budget, a learning rules and preferred communication channels. And if for some reason tag is missing or stale or gone, we fall back to a second signal. Service principles that ran the
[17:00] workload. We have built a service that provisions them and keeps ownership mapping always up to date. I have seen hundreds of tickets not acted upon because they were assigned to an owner. Getting ownership matters a lot. Teams
[17:13] change their names, services move, but the platform in our case can resolve each workload to the current accountable team. And the best part, you can easily ask Genie who owns this workload and you
[17:25] can answer right away in the UI. And now we have data flowing contracts um that enforce ownership as well. But it's not just enough. The platform also
[17:38] needs to shape how workloads run. This is the next layer. Defaults, policies, infrastructure as code. Defaults for us make an efficient pass an easy pass. Policy catch drift enforcement keeps
[17:51] everything up to date for humans. Every framework and template we publish ships with a smart defaults. When a team uses declarative asset bundles or a terapform to create a project, it starts with serverless photo optimization and system
[18:05] compression enabled. New teams inherit the deficiency for free. They have to have to work to make things inefficient. And for agents, we invest heavily into our AI instructions and plugins and also
[18:17] high quality references. Agents copy what they see. If our codebase shows an efficient pattern, agent generates a code that is efficient to run. The platform shapes what a proposes at
[18:29] scale. Defaults cover most of the workloads, but we still need enforcement. For that, we use infrastructure code and CI checks that catch violations before we hit production. As a result, over 90% of our
[18:42] workloads have clear ownership and follow best practices. And when we defaulted and migrated our workloads to serverless, it saved us roughly 80% in DBU cost. And our jobs became 12% more
[18:54] efficient, faster to run. So find a pattern once and code it as a default or a policy and every new workload benefits automatically. This is our platform leverage
[19:07] and interfaces. This is our our cost platform becomes useful to everybody else. It takes three parts. Genie, apps and dashboards. Genie is the most important for us. It allows us to get
[19:19] access to data using nature language. If finance asks why did my bill go up this week, they get a transferred answer without writing SQL. And Genie can answer because it reads from the same govern tables that powers everything
[19:32] else. Data bricks apps give us an action layer. We use our cost app for forecasts and budget requests. Databicks handles hosting, authentication, and infrastructure. We just write the code.
[19:45] Fun fact, last internal app I had to write, it took me just 20 minutes with cloud and data bricks. It's easy now. And last dashboards, they give our teams visibility into the spend forecasts and
[19:58] optimization opportunities. We have the same govern data and different interfaces. Dashboards help teams to read apps to act and Genie lets them ask questions.
[20:11] All right, we have built a platform, but like any platform, it costs something to operate. We have to monitor pipelines, keep the apps up and running, support hundreds of teams with their questions and overages. Quick show of hands, who
[20:25] loves debugging data pipelines in the weekend? Anybody? Not a fan. I'm not either. Operating cost is where many platform teams burn out. We do not. We let Genie and agents take the toil.
[20:40] We have built two primary agents to solve this problem. To make useful on their own, we gave them exactly the same context that engineer would have. They can they can access cost tables, code repositories, Jira tickets, all the
[20:52] documentation in Slack. Here is code a cost debug agent in practice. It's a regular Monday morning 9:00 a.m. Cost ticket lands. Workload double overnight.
[21:04] In the morning, an uncle opens a ticket and already has an answer. Agent pulled the usage data found oversized job. Wrote a hypothesis and suggested a fix. Engineer opens the ticket, updates the
[21:16] config, closes the ticket. Trash goes from days to minutes. And because we use data bricks genie, this is not just for engineers. Finance asks, "Why did my bill go up this week?" Gen turns it into
[21:29] SQL against our govern tables, returns a chart and explanation. No engineer in the loop. Team gets their answer in minutes. Remember two to six weeks number from the wall that it took to investigate and solve a ticket. It's
[21:43] hours now and sometimes minutes. And to show you the scale, we have seen almost 2,000 junior requests in last quarter. Can you imagine? It would have been 30 to 40 Slack pings a day for my
[21:55] team. Without Genie, those questions will just go back to us one by one by one. And this is why data matters. Cost is a data problem. And Genie thrives on governed data.
[22:07] Cost debug agent is reactive. But we also wrote an optimization engine that is proactive. It scans system tables continuously for antiatterns. Classic jobs that should be on serverless oversized clusters, unoptimized tables
[22:20] or idle endpoints. When it finds waste, it doesn't just send an alert. It creates a PR. It acts as a tireless background engineer trimming the waste across hundreds of thousands of workloads.
[22:36] And now let's take a step back. He remembers the wall from the beginning. Finance on one side looking at the bill engineering on other side building the product and platform was really stuck in the middle translating between those two. Now wall starts to come down. We
[22:50] have three works, three roles working on a shared platform each doing the jobs they actually signed up for. Finance is no longer chasing engineers. They set goals and controls and review forecasts.
[23:02] When they have question, they ask Genie directly. Platform is no longer in the middle. We focus on driving unit price down, setting efficient defaults and having strategic conversations about features versus cost and product teams
[23:15] are no longer flying blind. They own their numbers, their workloads, their forecasts, and everyone looks at the same dashboards, queries, same tables, and trusts the same data. Wall is gone,
[23:27] and teams are finally getting along. 20%. This is what all of our platform work added up to. First half of 2026 entire
[23:39] cost platform every pipeline every dashboard every alert every support question was kept by 20% of my time not five engineers not two 20%.
[23:51] And remember numbers from the beginning many quarters of feature work paused dozen incidents in a year two to six weeks to resolve a single ticket that was before after the platform it is just 20%. Each role is doing the job they
[24:04] signed up for. Teams forecast and usage platform ships features and improves unit cost. Finance sets guidance and controls. Hard part goes to data bricks and agents.
[24:23] And to be honest, we didn't do everything right. Our first big mistake was chasing perfect attribution. We wanted every dollar tied to a specific team. First 80% was really easy. Last 20% took far more engineering times than
[24:36] the ever returns in savings. Some costs I just shared. PI scanning, network egress, shared infrastructure. The lesson was 100% accuracy is not always the goal. Attribute what matters
[24:49] directly models. The rest is a shared platform text. Simple unit economics are much easier for teams to understand, easier to finance, for reason about, and easier for agents to use as well.
[25:05] The second mistake we had was top down cost mandate. Finance would come to us and ask platform go and hit the number. We had to pause the feature work, hunted down inefficiencies, deleted unused jobs and we sometimes broke things, made mistakes. Finance was not happy. We
[25:19] didn't move fast enough. Platform was burnt out because we paused our feature work. Product teams felt disrupted. So we changed the model. Product teams owns their usage. Finance set guidance and
[25:31] controls and platform makes an efficient pass default. They change the conversation from go cut spend to here is the data here is an owner and here is an action.
[25:46] And at this point you might be thinking this is all useful but how do I actually lower my bill? Right? Fair question. Arpon please come back to us and show some tips.
[26:01] Thanks, Maggden. You're absolutely right. No cost talk is complete without talking about savings. I'm going to give you three things you can do this week while you're out here at the summit to
[26:13] lower your data bricks costs. Number one, make serverless the default for your next job. Not only is it easier to use, it is also
[26:26] more cost- effective. Here's a great example. We had a team that was stuck using classic compute maybe at 30% utilization. They switched to serverless and the bill
[26:40] came meaningfully down. Same workload, lower toil, lower cost. The platform now owns the entire compute life cycle.
[26:53] Right sizing instances, terminating them when idle. If you want to know more about migrating to serverless, our team's giving a talk later. Scan that QR code. Definitely check it out.
[27:11] Number two, turn on predictive optimization today. Here's another great example. data platform owned this massive streaming pipeline which is basically ingesting all logs from data bricks. We
[27:25] were running separate custom optimize and vacuum jobs off to the side but they still couldn't keep up. We s we turned on predictive optimization and then the platform took care of all the table
[27:38] maintenance lowering our cloud storage bill but also more importantly freeing up the time of those engineers who were babysitting these storage maintenance jobs.
[27:51] Once again, same workload, lower toil, lower cost. Number three, rightsize your instances to the actual load profile of your
[28:04] compute. If you're using classic compute, chances are it's been a minute since you've looked at the instance configuration of your clusters and warehouses. There's money waiting there to be saved.
[28:18] Agents are great at this kind of optimization. Scan that QR code. That's the exact Genie prompt we use to rightsize our instances for our internal data platform.
[28:30] Scan it, point Genie at your own workspace, and start saving.
[28:42] But these are just three short-term tactics to save on costs. The real savings comes when your team invests in building a cost platform. It's the leverage created and the time
[28:54] you give back to your engineers to actually focus on feature work. Once all the data is in place, once everyone's looking at the same dashboards and once the agents are
[29:07] handling all the cost toil, your world completely changes and it looks something like this.
[29:19] Don't worry, our agents are still hard at work. Remember the conversation at the start, me shouting at Bogdan about his overspending workspace? That conversation is still happening, but between our agents.
[29:31] Finance's cost governance agent detected an overage in the middle of the night. It filed a Jira ticket. Bogdan's cost debug agent picked up that Jira ticket, did a root cause analysis,
[29:45] and submitted a PR, all before on call even woke up. Now all that's left is to approve the PR. Weeks of toil and investigation shrunk
[29:57] down to minutes of confirmation. It took us a year to get here, but you can get there quicker. All the tools you need to build your own cost platform is already in your
[30:11] account. system tables, Unity catalog, LakeFlow, apps, Genie. You already have access to all of these. You paid for them. You just need to compose them.
[30:25] And it really boils down to acknowledging two things. Cost is a data plat data problem and data bricks is the data platform to solve it.
[30:41] Thanks for coming. Here's all the resources we covered during our talk. We have our serverless migration talk which you should definitely attend. U our prompt to Genie for right sizing our instances and getting started with system tables. You have all the data in
[30:54] your hands. Um thanks again. We'll be hanging out here and in the hallway. Um stop by. We'd love to hear your version of the AI workspace cost story. Um yeah, happy to chat. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.