Cost Management Foundations: The First 100 Days Checklist
Summary
- Cost onboarding foundations: This video provides a foundational checklist for the first 100 days of Databricks adoption, ensuring proper setup of workspaces, compute, and tagging strategies to measure ROI effectively.
- Policy-driven cost management: Learn how to establish tagging and policy frameworks early that enforce best practices and prevent budget exhaustion from uncontrolled or unmonitored cloud spending.
- Future-proof budget planning: Discover how proper initial setup enables accurate cost forecasting, chargeback capabilities, and data-driven budget allocation rather than monolithic IT cost pools.
Cost Management Foundations: The First 100 Days Checklist

Talk by Piyush Singh, Staff Product Manager, Databricks ; Sadhana Bala, Product Manager, Databricks
In this video you'll learn how to onboard to Databricks in a way that ensures you can effectively measure and manage ROI of using Databricks down the line. We will show you how to set-up workspaces and compute, decide on a tagging strategy and how to utilize policies to enforce best practices and make future you a happy camper.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
Why is cost management important during Databricks onboarding?
This video emphasizes that early cost management setup prevents budget exhaustion, enables accurate forecasting of future platform spend, facilitates chargeback to business units, and provides data-driven guidance for achieving cost savings targets.
What are the key components of the first 100 days checklist?
This video covers setting up workspaces and compute infrastructure, deciding on a tagging strategy, establishing policies to enforce best practices, and creating the foundation for ongoing cost observability and management.
How does tagging strategy impact long-term cost management?
This video explains that a well-designed tagging strategy enables accurate cost allocation to business units and teams, making it possible to perform chargebacks and hold departments accountable for their Databricks spending.
What challenges do cost-unaware organizations face?
This video describes how organizations that don't implement proper cost practices early face budget exhaustion, inability to forecast spending, monolithic IT costs that can't be allocated, and uncertainty about where to achieve cost savings.
Full transcript
All right, it's 1:50 on the dots. Let's let's get started. Thanks for joining everyone. Hope you had a chance to grab lunch before this. Feel recharged for, you know, halfway through the summit. Before we dig in, just a quick forward-looking statement that this presentation is forformational purposes only. Some of the content you see here may be subject to change. And a reminder that at the end of the session, um, at the end of every session, you'll get a survey.
It helps us get your feedback on how you thought this one went. Today, we'll be covering cost management foundations, the first 100 day checklist. This will be a foundational course, so we'll start cover the basics and work our way up there. I'm a product manager here at Data Bricks. I focus on overall cost observability, cost management across product lines. And I'm P Singh. I'm also a product manager here at Data Bricks.
I focus on our compute platform which is where uh most of your spending happens. So we're going to talk about kind of how to manage costs across many many different dimensions today. Awesome. Well, I probably don't have to convince you about this why think about cost management early on. There's a reason you're in this room. But between myself and P, we've talked to hundreds of customers over the years and we have a good sense of what are the common pitfalls on the early days of using the platform that can really hinder customers from having maximum success on data bricks.
You know the the key takeaway here is all the best practices we'll talk about today if you set these up early on you'll reap the benefits in the future. So uh do play pay close attention to those. So cost of unaware organizations what we've seen I guess in our data as we speak with customers couple of pain points that they feel early on when they don't get it right from the onset often they'll find themselves at risk of exhausting internal budgets earlier than expected they had no good datadriven way of forecasting their future spend on the platform some of them when it comes to quarterly budget planning realize there's actually a large monolithic shared of IT costs that they can further allocate down and that makes it almost impossible to think about chargebacks in the future and others when they're sort of given a mandate from the top down to achieve x percentage percentage of savings on their data bricks platform it's unclear where to start in the first place.
It's almost too much fat to trim to begin with. But customers that have figured out their story around cost management management on data bricks see the opposite happen early on quarterly planning or whatever your budget securing cadence is. They're able to anticipate and forecast growth both from individual use cases coming on or new product lines they're adopting. They're able to support the primitives of chargeback.
So if they're building products on data bricks, get a good sense of what are the overall unit economics of products and services they're building on top of the platform. And also as they're evaluating new technologies, serverless model serving as example, they're able to constantly adopt and share those technologies among their user base because they're confident they can still stay in budget. So in this session, we will structure our presentations through three pillars here that really are a confluence of effective cost management.
From an organizational standpoint, we will talk through how should cost reporting mirror your internal organization's P&L or operating model from the first day and also how centralized should cost governance be. We'll talk through a couple options we've seen in how customers have adopted their governance model on a technical technological standpoint. We will talk through how if you're in an admin hat, how you actually evaluate different products and technologies to choose from.
What are the right use cases that make sense for one technology versus another on the platform? And finally, from a business standpoint, we will talk through tools and processes that you can put in place to both on a continuous basis and on a more periodic basis keep a pulse on your cost. But first, what are costs on data bricks? Again, we want to run a beginner session here and talk through what the components of cost are.
Now, this is not to scale, but here's if you're looking at your data bricks bill. Here's an illustration of the types of cost lines you might be seeing. Likely, most of your costs are coming from our sort of traditional compute products, warehouse, clusters, jobs. At a high level, what drives those costs? Well, it's a mix of variables. It's overall the types of instances that you're selecting to create your cluster on.
It's the size of your cluster, the number of nodes that might be available for your cluster to scale. horizontally or your warehouse as well. And then finally, it's the total runtime. It's the uptime of your warehouse, your cluster. Uh we it's built on a per second basis. So those are the variables that are proportional to your overall compute costs. But as you saw in the keynote today and as you'll hear more tomorrow, we are launching different types of products on data bricks and those come with different types of billing models as well.
For example, there's another line of products that's the second category you're seeing here. endpoints whether it's for model serving, vector search, data bricks apps, the the way costs are the types of costs that are proportional to those types of products actually more about provision capacity. You want to have a certain level of reliability and availability of these endpoints or these applications. So that's actually more of a mix of variables like processing power, the number of vectors that are available for your vector search endpoint to serve or pay per token if you're using that form of model serving endpoint.
And kind of going down again, you know, these are sort of maybe smaller line items you're seeing on your bill, but just so you're aware of if you're using any of the default storage product offerings at data bricks, those costs will be proportional to the overall storage costs as well as the number of API transactions that you might be hitting against those storage buckets. If you're using networking, the cost there will be proportional to the number of gigabytes that are transferred or if you're using something like private link, even the number of hours that the active endpoint is available.
Of course, this is just the data bricks side of the story. What you'll also be paying costs on is directly to your cloud service provider. If you're using non-serverless products, there you'll be paying your underlying cloud infrastructure costs. So for example, EC2 networking costs that are actually being provisioned for your uh clusters to run as well as storage. But the beauty with serverless, which is what we'll talk about in a in a second, is it makes cost reporting a lot simpler.
The types of costs you'll see are really just the big pillars on the on the left hand side for you. And one thing we're excited to announce is we've invested heavily in this space in this last year, both for a cost observability standpoint as well as a cost management standpoint. Spoke with a customer who accurately cited that he's seen more progress in cost management in the last 12 months in the last four years. And we have a lot of cool new features that we're excited to announce today and in the future.
With that, I'll hand it to P to talk about the organizational considerations. Thanks. Okay, so when we start thinking about costs and best practices, what comes to mind? The first thing is making sure you really understand what the uh organizational boundaries are within data bricks so that you can set them up correctly um and so you don't have to run migrations in the future. Broadly, there's three layers at which you can kind of subdivide your resources and use them to allocate and track costs.
The first one is the account level. Most customers only have a single account. Uh but you might have multiple accounts if you really want strong isolation between different lines of business within your within your company. Uh if you're running in multiple regions or on multiple clouds, you will get entirely separate billing information for these different accounts. But you can of course join it together. Um but typically we see most customers are running just a single account.
Now within the account you can have multiple workspaces. Workspaces are another isolation boundary uh but they're not quite as strong as accounts, right? So um we find a lot of our customers will set up distinct workspaces for different levels uh different stages of their CI/CD right. So like if you have dev test uh staging and prod uh workspace you can kind of isolate um your resources that way. Another way you can also subdivide your costs um through workspace isolation is by assigning different teams to different workspaces.
Inside of the workspace, you can actually also set up kind of organizational boundaries, right? Um, so when you create resources like clusters or other types of compute, warehouses, other things like that, um, you can actually decide which users get access. So you might want to group your users based on uh, and create clusters for specific groups of users based on, you know, which cost centers they roll up to, which projects uh, they they are working on.
This can kind of help you subdivide your costs and understand um you know which project or which team is consuming how much. Now these two different constructs accounts and workspaces they're managed by two different personas. The accounts uh are kind of managed by account administrators. These are super users within datab bricks. They have broad privileges within the platform and they get access to lots of uh billing information that isn't available to other users.
So they can use tools like budgets to track and alert on spending. Um they can use our account usage dashboard in the account console. So this is a dashboard that actually helps you visualize your cost. We'll have an example of that in just uh in just a bit. uh and they can actually broadly define um the roles and privileges other users have when it comes to how they can consume billing information um or if they're able to create resources uh that spend you know create spent within data bricks workspace administrators are one level down they are admins for a single workspace and so they govern access to that workspace for users they can create things like cluster policies to kind of design guard rails for spending within a single workspace uh and They can even manage compute creation for users in a workspace.
So um we'll just get to this in one second actually. Um you can kind of see there's there's different ways that our customers tend to manage how users can use data bricks, right? So typically this is kind of the progression that we see from our customers as they mature over time. So when you first start using data bricks um you might have just a small team of users with broad privileges within the platform. They can kind of do and create anything, spend an arbitrary amount.
Maybe you're just trying to kick the tires on data bricks, understand what it's all about. Um, and then, uh, as you kind of mature over time, you might add more and more users, right? U, but you don't want all of those users really to have broad privileges within within data bricks. You don't want them to, you know, create an arbitrary amount of spend for you to manage. So, what we typically see is the um, the new users will be managed by some central platform team.
So um that central team will basically create and manage resources for those users and assign them to clusters that they've created for them. Uh but they won't let them actually create compute within data bricks because that can if they do it incorrectly um you know cost a lot of money. Eventually though we find a lot of our users or a lot of our customers will will scale even beyond that point where um having a central team administer all of your users if it's hundreds or thousands of users is no longer tenable.
And in that case we see that um they will actually delegate responsibilities for managing cost to kind of more trusted users within their organization. These might be your senior data engineers, you know, your uh DevOps engineers um or just team leads, right? And so a central platform team will basically delegate those responsibilities through tools like cluster policies. Um, and so these trusted individuals can actually create compute, but subject to guard rails that are defined by your central admin team.
So your account and workspace admins can create these policies which allow you to delegate uh, you know, access to creating compute to more trusted users, but you can kind of constrain them in terms of how much they can spend. And then those users can create compute for your your um you know users that you don't really trust to create uh clusters and warehouses uh and give them access to those those resources. Now when you think about aggregating your costs, reporting on your costs, um you're probably going to want to subdivide your costs.
Like S mentioned, you don't want to get one big monolithic bill, right? You want to understand who is responsible for consuming what whether it's a specific cost center, project team, what have you. And so the way data bricks is designed to manage this is through tags. Now what are tags? Tags are basically keys and values, just strings that you can apply to nearly every resource within databicks. And these tags um they basically will be propagated to your billing records both on the data bricks side and on your cloud provider side.
So what you can do is after you get your billing records now that you have these keys and values you can aggregate your costs by those dimensions that you defined. It can be any custom grouping that you'd like. So for example, if you want to see how much a particular project is spending, as long as all the resources in data bricks for that project are tagged to so that they uh you know belong to that uh that resource that um that project, you'll actually be able to aggregate your cost after the fact and see exactly how much that project is costing you in data bricks.
What you really want to avoid is that top bar there where nothing is tagged and you kind of just get one big one big bill and you can't tell where it's coming from. And so you can kind of see right we've uh already started exposing this this tagging functionality across multiple interfaces in the product and the expectation is that every product within data bricks will support tagging and on top of that you can also as admins enforce tagging uh so that your users don't have to input arbitrary tags.
You can actually require them to supply tags and fix tags that they can supply through tools like cluster policies and budget policies for surplus. Great. So those are some organizational considerations and now let's look at some technical uh considerations when it comes to thinking about how you manage your costs. So broadly this is a very complex topic and we won't get into all the details here but the thing to keep in mind is obviously there's always a trade-off between price and performance.
So you need to really think about what your business business needs are when it comes to performance and where you're willing to trade off um you know better performance for price or perhaps you know where you're willing to accept less performance for a cheaper cost. Now the dimensions of performance that are you know important to keep in mind are things like how fault tolerant does your uh workload need to be. So for example if you use spot versus on demand instances you're trading off uh fault tolerance for cost.
Spot will be cheaper, but you might suffer spot evictions, which can kind of increase the length of your the runtime of your workload. You need to consider throughput and concurrency. How many users are going to be using your DBSQL warehouse? You should think about what your SLAs's are, your execution time for your workloads, how long should they run, how much compute do you want to throw at these workloads in order to run them faster.
And then of course things like scalability, right? How many users are going to be executing workloads at the same time? Um, and then how quickly do you want users to be able to, you know, start running their notebooks or executing their DBSQL queries. Let's look at a few examples of how these trade-offs work in practice. So in this first scenario here, you can imagine, you know, uh a company is basically going to um see a big spike during the holiday period and they know their business analysts are going to be running lots of DBSQL queries to kind of um do some demand forecasting to help with um you know supply chain analytics during the holiday season.
So what will we need to consider when it comes to DBSQL in order to make sure that these users are set up for success? Broadly, there's two things, right? So if you think about what's going to happen is a lot more users are going to be running queries at the same time. And so when you think about DBS SQL that would mean you want to increase the concurrency that's supported in your warehouses or in your in your DBSQL endpoints.
So there's a specific control that lets you scale the number of clusters created behind that endpoint that will let you basically increase the number of users the amount of concurrency that you have available uh for any particular warehouse. On top of that, you can imagine during the holiday period when everyone's under time pressure, they might be frustrated to spend a lot of time waiting for their warehouse to come up.
And so you might want to do something like switch your warehouse to a serverless DB SQL warehouse where the startup latency is very low. Alternatively, you can increase the autotermination threshold. So that way when you know someone steps away for a coffee and then comes back, they don't lose all of their work and all their cache data uh because the warehouse was terminated. In the second example, you could imagine um you know a company is working on an Intel pipeline and it has very strict SLAs, right?
There's some downstream dependency. Maybe you're delivering data to a customer and you need to get it there within a certain amount of time otherwise you'll be penalized. So what are some considerations there? Well, of course, fault tolerance and reliability. If you're suffering from, you know, reliability issues, you probably won't be able to deliver your data on time. And so that comes back to things like spot versus on demand instances.
We use spot by default to help you save costs, but if you have a really strict SLA, it might not might not make sense to use spot instances. You also want to think about execution time. So you want to size your cluster correctly or your warehouse correctly in order to actually be able to execute this workload quickly enough, right? So if you have a really strict SLA, say 10 minutes, you might need to pick a larger size for your warehouse or for your cluster in order to execute your workload in time.
And then lastly, um you need to think about startup latency. Again using 10 minutes as an example, uh if you need to complete a workload and ship data somewhere uh in 10 minutes and it takes you eight minutes to run the workload, you can't spend 5 minutes starting up your cluster. And so you have lots of tools available there. So for example, you could switch to using serverless jobs instead of normal jobs. You can use an instance pool to accelerate the launches of your instances from dropping that from 5 minutes to two minutes.
And so there are many knobs you can use for each of these to basically trade off the price that you're going to pay for the performance needs that you require based on your business requirements. Thanks be on section three now. How do you actually operationalize some of what we put into practice? The customers that are most disciplined about cost management. We see we see them take sort of three approaches in terms of cadence on how how often they're looking at their costs.
On a weekly basis, you'll often have, especially in a more federated model, team leads, department heads do a weekly audit over overall spend trends. For example, am I seeing any anomalous spikes in my spend this past week? Has there been a significant growth rate in one product line versus another this week versus last? On a monthly basis, they may pull in more executives, overall management, their finance team to understand pacing.
Maybe you've signed a commit with data bricks. Maybe there are internal budget targets you're trying to hit. What is your actual pacing on spend against these targets? And then on a continuous basis, of course, you shouldn't want you don't want to manually be checking on a weekly or monthly basis, but on a continuous basis, how can you be confident that you'll be alerted if anomalous spend is de causing you to deviate from your budget target?
There are three tools and we'll get into detail on on each one of these that allow you to do this continuous weekly and monthly type review process. System billing tables is your is your gateway into granular and low latency cost data. We'll get into that in a second. on top of that using the AIBI usage dashboard as a reporting interface to do that type of executive and operational cost review and then on a continuous basis relying on a tool like budgets to ensure that you will be alerted if the pacing against your budget is accelerated compared to where you expected it to be.
So we'll start with that first one here and this is just an example of a the company they're in the crypto space they use exactly this model as well. Their executives look at the dashboard on a monthly basis, but because the volume of crypto data can be volatile on a weekly basis, they have individuals checking their spend targets as well. Out of curiosity, by a show of hands, who is familiar with the system billing tables?
It's a good portion, maybe 70% of the crowd. The way to think about cost observability is there there are different sort of interfaces for it potentially but perhaps you're looking at your Azure bill, GCP bill, AWS bill to get a high level sense of how much you've spent on data bricks on a certain period of time. But if you want to get low latency, high granularity usage breakdown, that's where system billing tables will come into play.
In general, system tables are delta tables that will give you a window into all the operational activity across your platform. billing tables as one specific instance or schema of those system tables. Okay, so what can you find in these system billing tables? If you click into it, there are a lot of useful fields here that will make your cost reporting easy and effective. For example, every spend dollar you can try to you can tie to a billing origin product.
So you'll have a sense of which particular product lines that spend came from and also which particular skew if you're getting into a second level of detail. Also what particular objects billable objects did the spend come from? Was there a particular warehouse job ID warehouse ID as I mentioned that will that information will be present in the usage metadata field. And then with identity metadata this will actually allow you to attribute cost to the principle level whether that's a user principle or a service principle that might be uh leveraged for sort of a kind of runsummit job for example.
And then as P mentioned those tags that you apply through the UI, through an API, through a policy, those t tags actually propagate down to the system billing tables as well. That way you can group cost based off your own business taxonomy. So what did we do last year? We built a AIBI usage dashboard on top of the system tables and that's available in the account console today. Is this something folks are familiar with by a show of hands?
Great. Right. It's a good good number. Um, so this is available today in your in the account console. You can you can import a dashboard, spread it as widely as you want within your organization. We have some customers that actually make the dashboard accessible to all users in their in their workspace because they want everyone to be empowered to see the costs that they generate. The dashboard will give you a high level overview of overall usage.
Again, for a monthly or weekly cadence, this might sufface. If you're looking to do chargebacks, how do you actually tie those costs? based off a tag key value pair. And if you're just looking to understand what are the top drivers for an optimization exercise, the dashboard will give you that lens as well. But what we've heard for you is that you all want more than just a highle reporting interface of course. So we've spoken with a lot of you who have used the version one of the dashboard and we're excited to be introducing usage dashboard 2.
0 postsummit as well. What this is is on the same system table but also pulling in other system tables that have been newly released as well. This is a more powerful powerful and actionable dashboard that lets you do key tasks like chargebacks and showbacks tagging compliance and uh down to the level of organizational accountability. Who are the actual individuals I need to nudge to optimize their cost. This is a quick preview of the usage dashboard.
You'll see forecast capabilities are available through the AI forecast SQL function. An ability to break down your cost by product, an ability to rank your costs by top objects. This is a little fast, but these slides will be available here. I'm looking using budget as a primary tag, and I'm looking to filter my costs by the budget tag. you'll see a large portion is untagged. But when I click into what is tagged, I now can actually charge back against for example the opex and runtime uh budget key.
And if I actually want to drill into what particular objects have this budget tag or missing it, I can actually use the this top objects tab to drill down to the specific object ID. Here I'm actually identifying a couple of objects who owns them and I can click into a SQL warehouse from that interface and pull up the particular warehouse as well. We'll have more resources available on the dashboard 2. 0 but uh in a second we'll show you a survey.
If you're interested in getting a preview of this view, let us know and we'd be happy to share it on top of the usage dashboard for a more continuous asynchronous monitoring experience. Last year we launched budgets. This is again available in the account console. What this allows you to do is to create a budget either at the account level, workspace level or on a combination of tags. Define a particular budget number.
Perhaps this is your commit divided by 12 for example. Share that out with a couple of individuals and then you can be confident that you will receive an alert via email if you've actually surpassed that budget. Again, we've talked with many of you who have used this tool and we're excited to announce a couple of enhancements coming to the to the budget tool where you'll see richer visualization to allow you to more quickly identify the particular spend drivers.
Um, more configurable thresholds so you can get earlier alerts if that's what you desire and an improved tag search experience so you have an easier time parsing through all the tags in your system and understanding what to apply a budget on. We'll we'll leave you with best practice number six here. You know, we've talked about performance, we've talked about cost, but there's a third dimension that is historically harder to try to wrap your hands around on on any data platform, data bricks and beyond.
This is how do you actually measure the business value of the assets that you're creating? As an admin, you should be constantly asking yourself, are these dashboards that I've built up and are refreshing on a daily basis, are they actually powering critical business insights? Are these jobs that are also running on a frequent basis? Are they automating manual processes that save us a lot of time and effort? And finally, are these AI experiments, the models serving endpoints that I know my data scientists have provisioned, are there are they running useful PC's that we believe will uh, you know, eventually make its way into production at some point.
How do you get answers to these questions? Well, we're working on a number of tables and logs that allow you to get those types of insights. For example, using the audit logs, there's a system access audit table. You can actually answer questions like, you know, how often was this Genie space updated or how often was this notebook accessed as well? These are event logs that are available for numerous services that give you that sense of utilization.
Using the query history table, you can actually understand what are the number of unique users that have executed a query by interacting with the dashboard. Are there actually a lot of users interacting with this dashboard in the first place? Data lineage, which is a very special feature of QC, will give you a sense of lineage between jobs and tables. Is this job that's running on a daily basis? Are the downstream tables to this job actually also high value?
And just as another example for model serving endpoints, which again are typically provisioned um or paper token, what is the actual like payload of requests against this endpoint? These are all types of questions that you can answer using some of the resources on the right hand side and that allows you to scratch this final surface of what is the actual value and ROI I'm getting out of my data bricks activity. So with that we we conclude um before we wrap up we would actually love to take a couple of seconds of your time and have you snap out your phone take a um sort of picture of this QR code.
What you'll find here is actually a sense of where we're investing our road map in on cost management and we would love to for you to vote and help us identify what we do next. Uh so I'll leave this on for a couple seconds to get to ensure that you get those surveys loaded. And beyond that, here's a recap of the best practices. But we because this is a foundational session, we want to ensure that we have time for Q&A.
So we will actually transition over to that now.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.