FinOps at Scale: Best Practices for Cost-Efficient Growth on Databricks
Summary
- FinOps Framework: This video covers cloud financial operations combining cost observability, spend monitoring, and chargeback models on the Databricks Data and AI platform.
- Cost Culture: Learn how to develop organizational cost efficiency through tools like Usage Dashboard v2.0, enabling accountability between engineering, finance and business teams.
- Scale Responsibly: Ensure end-users remain both cost-efficient and productive while staying accountable to budgets as platform usage grows across departments and use cases.
FinOps at Scale: Best Practices for Cost-Efficient Growth on Databricks

Talk by Cody Davis, Lead Solutions Architect, Databricks ; Sadhana Bala, Product Manager, Databricks
You’ve seen your usage grow on Databricks, across departments, use cases, product lines and users. What can you do to ensure your end-users (data practitioners) of the platform remain cost-efficient and productive, while staying accountable to your budget? We’ll discuss spend monitoring, chargeback models and developing a culture of cost efficiency by using Databricks tools.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00Introduction0:38FinOps overview and framework8:50Inform: Cost observability13:37Usage dashboard v2.0 demo21:58Optimize: Improving efficiency31:30Operate: Continuous management
FAQs
What is FinOps and why does it matter for Databricks?
This video defines FinOps (cloud financial operations) as a framework and cultural practice maximizing business value of cloud technology through data-driven decision-making and collaboration. As Databricks usage grows across teams, FinOps practices ensure cost efficiency and shared financial accountability.
What tools does Databricks provide for cost management?
This video demonstrates the Usage Dashboard v2.0 for cost observability and monitoring. The platform provides visibility into spending patterns, enabling organizations to understand consumption across departments and implement chargeback models that promote cost-conscious behavior.
How do chargeback models work in this video?
This video discusses implementing chargeback approaches to allocate Databricks costs to individual teams or departments. This creates financial accountability and encourages teams to optimize their usage, balance performance needs with cost efficiency, and make informed decisions about resource allocation.
How can organizations maintain productivity while controlling costs?
This video covers developing a culture of cost efficiency through clear visibility and shared accountability between engineering, finance and business teams. By using Databricks' observability tools and establishing cost awareness practices, organizations can ensure platform users remain productive while staying within budget.
Full transcript
0:00 Introduction
Welcome everyone. Thanks for taking time. I know it's lunch time, so please enjoy your food. If you're eating your lunch, if you've had lunch before, hopefully enjoy that as well. Uh, as we get started, just a quick forward-looking statement here. This presentation is forformational purposes only, but we're excited to show you some really cool stuff pretty soon. Uh and like this session and every other session coming up, you will get surveys in your app.
So ensure that you fill those out so we can provide the best experience for you guys going forward as well. Fin at scale, best practices for costefficient growth on data bricks. I'm product manager here on data bricks focused on cost management and cost observability. Hi everyone. Uh I lead a team of solutions architects in our digital native business and get to collaborate with our lovely product team on observability, warehousing and a couple other areas.
0:38 FinOps overview and framework
Definitely excited to talk to you all. All right, so just to start out of curiosity uh last week was anyone in San Diego at the BOPS conference by a show of hands? Okay, we have one who I know well our product leader as well. A lot of people want to do two conferences back to back. Yeah, San Diego had better weather than SAF. But for those of you who don't know, the PHOPS Foundation hosts a conference every year for PHOPS practitioners.
We actually learned at that conference that there is a huge population of PHOPS practitioners globally. I think they quoted 85,000 individuals. So PHOPS is something that you think about every day. We think about every day and we want to cover best practices in that field in this session. For those of you who don't know what is PHOPS, cloud financial operations, it's both a framework and a cultural practice that maximizes the business value of cloud technology.
Here we're talking about course data bricks in a way that's driven by data in decision-m and the end outcome is shared financial accountability through collaboration between both engineering, finance and business teams. So this year was sort of data brick's inaugural presence at the PNOPS conference and it's something we care deeply about and are excited to dive into this realm today. If you're a uh you know if you're an account admin here on data bricks maybe you've seen a step change in your usage on the platform through a curve like this and this may not necessarily be a bad thing.
Maybe what you're seeing is more users joining the platform especially as we on as you onboard different types of products. Maybe you're seeing more data flow through the platform as you're hooking up more ingestion connectors as well. More experiments, especially in the AI space. Your teams might be prototyping proof of concepts on the platform and more migrations, especially if you're trying to centralize potentially data bricks as your primary jobs orchestrator as an example.
So, we've heard from many of you and customers like you that while that growth is is exciting and you're seeing a lot of value on your platform, there are scaling pains that you're facing. And what are some of these? One of them might be sort of incomplete cost allocation. You're having a hard time tying all your dollars back to particular teams and that's making it hard to do fully loaded chargebacks. Maybe the growing admin burden to monitor hundreds of workspace is just becoming manually impossible to do. or with the new frontier of AI, understanding how to do PHOPS for AI is a puzzle and that's something we're figuring out as well.
Maybe you're seeing an influx on sort of lower trust or less experienced users that may not be steeped in the cost optimization best practices, but they're able to do powerful things on the platform. And maybe at the end of the day, uh, you know, if someone's asking you what is the total ROI you're getting on data bricks and which areas are you having positive unit economics, maybe you're having a hard time telling that story.
So let's tackle that those types of pain heads on through the PHOPS framework. If you've seen this before, this is the classic framework from the PHOPS foundation. We'll break our session into three pillars here. So to start in form, this is around making we want to make sure you understand what data sources and report and reports you should look at to understand your cost and usage data deeply. We'll then talk about optimization beyond observability.
How can you actually take actions to ensure that you're bite-sizing your spend and getting the most performance for the least amount of dollars? And finally, we're talking about PHOPS at scale here. So, how do you actually imbue PHOPS throughout the veins of your organization? So, PHOPS is everyone's job and not just something the central admin team has to manage. So, what does that look like more practically? uh typically we see uh within organizations to create this feedback loop of FinOps operations all the way from the top down.
So this usually starts with an executive having control of a given budget wanting to just know understand and be able to manage how are we doing on our budgets, how are we doing on our vendor contracts, how are we trending overall up or down and where do we need to focus uh to optimize and manage our spend that may be running away. Those executives may then pass that question of where we need to focus uh identify one to three initiatives and say hey uh I need you guys to be able to understand this area of usage and ask deeper questions to understand what's driving that usage what's contributing to it and is it correct.
So they'll typically pass those questions down to the P&L leaders, typically middle management director level that own specific business areas, use cases, and uh teams that own that usage to diagnose and find and manage those cost drivers to ask questions like for this specific team or environment, what is driving this cost? Where are things efficient? And where is spend not being managed well? How do I fix that? Those middle uh middle level managers that own these use cases and own uh the sort of outcomes uh for what is driving the data platform can then work with the engineering teams that actually build on top of data bricks to start to automate optimize and manage that usage finding the root cause figuring out how to fix it and then how do we make sure this usage stays managed.
So this feedback loop going from top down and then changing what the engineers do dayto-day uh to bubble up a better operational framework from the bottoms up is what we want to showcase today with the given tools that we have and what what we have in the product and what we have in the broader ecosystem to complete this feedback loop. So uh staying true to data bricks' commitment to open source we have a full spectrum of solutions that allow you to do this end to end feedback loop of observability.
First level is obviously self-built self-built. You can use all of the data bricks logs the APIs and the system tables uh to build your own custom solutions. So lots lots of customers like to do this. Uh the benefit is that you can customize this to your heart's desire to do whatever you want. The downside to that is not everyone has the time, energy and money to reinvent the wheel. Uh these are these observability problems are all the same problems that we all in this room have.
We would mostly rather uh you know one person solve a problem and then share that with the broader community. So that leads us to a really rich ecosystem of field and communitybuilt uh dashboards, frameworks and assets that are built by SME and subject matter experts uh and then shared with everyone in the broader ecosystem. This helps validate, test metrics, but also keep innovation speed pretty high and keep the actual logic and frameworks open so that if you do want to customize something, you're customizing some of these tools based on battle tested, you know, hardened logic and not something you may be sort of building from scratch.
A couple great examples of these what we'll show are uh tools like our dashboards like a uh warehouse advisor dashboard to manage warehouse performance, workflows advisor dashboards to optimize jobs, etc. Even further than that, last year we released our account usage dashboard. Uh that's a native tool dashboard in the product and then the cost per query dashboard on data bricks SQL. This represents sort of a higher level of support from data bricks.
Uh these are tools built by data bricks people um where the logic is validated, battle tested and even supported by data bricks but it's still open, still customizable if you want to extend those tools. And then of course lastly we offer native product integrations such as the system tables themselves, monitoring frameworks as well as budgets that Sodna will talk about as well. So we have a full gradient and suite of products that you can use in concert together to build your own observability solution for your platform.
So we'll showcase a couple demos here uh in both of these areas as well as just customer stories. Awesome. Start with pillar one in form. Again, this is all about observability. How do you get a good sense of what you're spending on, where you're spending that, and how that spend is pacing over time? The key takeaway here for you is cost observability starts with system tables. Hopefully, that's something you're familiar with already.
8:50 Inform: Cost observability
If not, system tables are delta tables that store all the operational activity across your account. In particular, the system billing schema provides high granularity and low latency visibility into all of the billable usage on your platform. And when you double click into that into that table, you'll see there are a couple of key fields that you want to be aware of as you're building your cost reports. So some of these fields, billing, origin product, that will give you the ability to tie every dollar back to the originating product or skew that the spend uh sort of emanates from. uh usage metadata that will give the ability to tie the dollar back to the particular object ID, the job ID, maybe the warehouse ID, where that spend emits from.
Identity metadata, that field will give you the ability to actually tie the dollars back to the responsible principle, whether it's a user principle or service principle. And then of course, Finop's best friend tags. The custom tags field will give you the ability to slice and dice and organize your spend based off tags that you apply based on your business taxonomy. We announced this last week at the Pinpops conference, but we're excited to announce a private preview of the billing usage data in the focus schema.
The focus schema is a common data model that will make it possible for you to harmonize your data bricks billing logs with other logs and other types of vendors. you may manage as your fidops practice. spoke with this but last year we launched and this is available today in the account console an AIBI usage dashboard. This is built on top of the system tables I just mentioned and this provides you with a highle interface that allows you to do a couple things.
Get an overview of overall usage across your across workspaces across SKUs. use tags as a top level filter and identify what the top billable objects might be. Maybe what are your top 10 warehouses that contribute to your spend? But Cody has a lot to share on what's coming up in this realm. Absolutely. So just curious before we dive into the new version of the dashboard. Who has actually used this initial V1 usage dashboard so far?
Roughly half. Okay, cool. Um great distribution. So uh in this initial version uh we realized that it's a really great start to be able to identify, manage and see usage uh from a high level as well as do some basic tag filtering and top spending. Perfect start but doesn't quite get us that full feedback loop of going from top down executive budgeting down to IC level um operations and back up. Uh so we talked with customers and really identified three key areas to complete that feedback.
So in this new usage dashboard 2. 0 that we'll be releasing soon and showing today uh in a demo here in a minute um we extend version one uh to start with high level spend and then drill into that. So we want to go deeper to be able to uh track key actions in the dashboard. Identify key drivers and objects as well as uh figuring out who owns specific objects that's driving runaway spend or maybe unmanaged usage. connect that down to the individual contributor and then bubble that back up to create that feedback loop.
So with this new framework, you're able to utilize tags to do chargebacks and P&L management. So you can create a policy layer to make sure all usage is tagged by team or use case or environment or budget or however you want to manage your spend with the way that fits your business. And then you can additionally find usage that is not compliant to those tags. So, this this is what I like to call ghost usage, but this is just unmanaged usage where you don't know what it's for.
Um, you don't know if it's driving value. You want to be able to find and fix that and manage that very quickly. And this is what's going to ultimately create the feedback loop to start to take action from top- down exec reporting to finding uh untagged usage, managing that and then creating actions such as policy alerts, setting proactive budgets that Sodom will talk about, and then digging into specific engineering uh tactical next steps that you could take like right sizing a SQL warehouse or making a job smaller or just tagging metadata so you know uh where this usage is going if you want to tag it to your revenue generating use case that you want to manage.
So let's take this high level framework and apply it to the actual new usage dashboard. So I will go over here. All right. So this is V2 uh of the usage dashboard. Uh the first thing you'll notice is uh we now are in we now have tabs in AIB AIBI dashboards. So this really helps open up uh that flow to go from high level spend down to individual detail level. So let's say I'm an executive at data bicks. Uh theoretically I want to be able to manage my spend.
13:37 Usage dashboard v2.0 demo
I can see my top level spend is this much over a 14week period. Um I can even forecast spend in this new version. So we utilize AI forecast natively to be able to forecast usage uh to what I to what I want to you know forecast. Let's say I want to see next week's usage or next month's usage. I can even subscribe to this dashboard um and trigger alerts on this as an executive to see maybe hey if forecast usage goes above a certain amount I want to get an alert.
I want to know I want to proactively get ahead of that before it becomes actual realized usage. So you can definitely do that here and see that all in one trend. So as an executive at database I can clearly see SQL model serving vector search clean rooms all of these are top drivers of spend and all of these have been increasing pretty recently. So this big chunk of green and pink usage is is sort of uh kind of scaring me a little bit as an executive.
I want to see what's going on with this because my model serving SQL and especially my vector search usage has started small and very recently increased week overw week. And in the dashboard, if you uh manage your business, you want to see monthly spend, daily spend, um you can dynamically toggle that with this new global filter that lets you see the entire dashboard by the time frame that you want. So here we're looking at weekly and our weekly spend is increasing quite a bit.
And we're seeing it's coming all from, you know, really this top big workspace here, but it does trickle down into other other workspaces throughout. So, I want to be able to dig into who owns that, who do I need to go to, and who what initiatives do I need to create um to manage and either make sure that spend is reasonable and planned for or turn it off and make sure it doesn't happen again in a non-managed way. And this is a very quick as an executive top level summary where I can see quickly period over period uh what my increases are.
So uh I can see very quickly that SQL usage has been increasing quite a bit as well as model serving and vector search. So I will then pass that over down maybe to a couple direct reports to identify and diagnose the root cause of that. So let's actually do that. We'll go into this new deeper dive usage analysis tab. As maybe an engineering manager or engineering director that owns uh maybe a portfolio of teams or a portfolio of business use cases.
Um, I can then search all of my usage by any tag filter. So, this was in usage v1. Um, but this extends that functionality even further to be able to do really two key actions here. Um, to be able to track usage that is managed, categorized by team, use case, environment, whatever it may be. And also very quickly find and detect and manage my unttracked usage. So you can see very quickly this is a demo environment so unsurprising but we have a huge problem of not tagged or usage that isn't tagged by team.
So I do not know who owns this not tagged usage and I really want to be able to quickly find out what that usage is, who owns it so that I can properly categorize it and fix it. And then as I start to do this um with my engineers and uh and IC's that work under me in these various teams, I can track my match rate over time. So we want to see this improve. So as you uh begin to inject this into your operations, you can track the improvement compliance to your tagging policies over time.
And you can of course save this dashboard to basically snapshot all the tags that you use. So here I can scroll down even further. I have a similar view where I can track my usage over time and by workspace or any other uh grouping that I want um to be able to look at that for my specific tagging year. So I can see 5% of my usage is tagged. We have a huge problem of a massive explosion of usage of not tagged usage. So I want to diagnose, find and fix that.
And we have a team, our our data platform team. Uh it looks like they have some brand new usage. Um but it's exploded very quickly just in the last week. And I can quickly see that. And I want to be able to go to that team and make sure they're budgeting, planning, and optimizing the usage. Maybe they're onboarding a new use case and they need to rightsize things. I can quickly get ahead of that here. Let's say you don't have a really mature tagging process yet.
Uh you just want to know who's using what tags and, you know, sort of get a get ahead on things um uh on a system that already exists. We added this tag popularity section that just lets you figure out uh very quickly what tags are being used. So this common uh common feature that customers have asked for just saying you know what are my use tags I don't know where to start. Uh this shows you that you know budget policy serving endpoint team are great great places to start and you can continue to add from there.
So now that I can manage usage uh by tag to basically form fit my ops framework to my organization. I can then drill even deeper and go down that feedback loop. And let's say we can look at our team and find these specific what we call billable objects. Maybe that's a SQL warehouse, a job cluster. Um I can look for usage that is tagged or isn't tagged. But I can very quickly just go to this heat map and see that SQL and vector search is a problem.
There's a lot of spend here. It's actively been used in the last 30 days. Um, I want to be able to scroll down here and see my top drivers of that spend. So, here my top driver of spend that is not tagged, not managed is a SQL warehouse. So, I can scroll over and see all the information about it. Um, I can see that my next top driver of spend is a properly managed SQL warehouse. So, that's good. Um, but this is a big problem.
So, what do I do with that? How do I find out who owns it? What do I do to fix it? Well, if I have permissions to that SQL warehouse, I can actually just go in um straight from the dashboard directly click into this warehouse. I can quickly see who owns it, who created it. I can even just tag that usage directly here. So, I can fix my untagged usage, you know, and categorize it into a data platform use case. So, now I've gone through this entire feedback loop in a matter of three minutes.
Um and that used to take you know teams quite a bit longer and you used to have to ruffle through that and it would be pretty difficult. So this is a really really helpful uh feature functionality that customers have have asked for quite a bit here. So this uh is a great end to end starting point of that end to end feedback loop. Um a couple additional things here we have plenty of new preview features that are toggable here.
Um, the most popular asked in example is the ability to either use list prices in this or actually use our new preview account prices table that includes your actual contracted rates as well as you can actually override and simulate your own discounts and or your own prices um in this override spot spot right here as well. So if you if you're like working on a new deal um with data bricks and you want to figure out what spent might be do some budgeting simulation um you can override and say you know new SQL discount.
1 whatever that may be. So lots of really great functionality there. Um but yeah so now we can then go from high level executive overview you know figure out forecasted usage to a couple initiatives that I need to focus on uh where engineering leaders can then drill into that manage usage by team and then the specific engineers and IC's can then find specific assets that need to be fixed and managed and this sets us up for you know an end toend feedback loop for that sort of section one of the inform phase of things.
So, now that we know what our usage is, who owns it, who to go yell at, uh, let's talk about what to do next to fix things. Oh, thanks. Yeah, we're excited to get that in your hands soon. So, stay tuned for updates there. Yes. So, section two, optimize. Uh, now you know where you need to figure out, now you know where to rightsize your spend. The question is, how do you do it? what are the particular objects that you want to have more control over.
21:58 Optimize: Improving efficiency
We'll talk through sort of three examples of you know real life dashboards that let you do this across three different product lines. But just to start uh it's important to remember that optimization we often talk about cost optimization but really you can't think of cost without performance on the um other side of the house and it's not always a trade-off between price and performance of course but even as fops practitioners you want to be aware of the different technical requirements of your workloads the workloads that you're looking to optimize and those technical requirements are often driven by the business use case of that workloads.
For example, uh can this workload be tolerant to fault and disruptions? If yes, then there's a lot of different types of instances you may want to use like spot instances to manage costs. If no, then you may have less flexibility there. What is the type of concurrency requirements on your workload? Like how many simultaneous users might you expect to hit the same application or the warehouse? And as another example, are there sort of tight SLAs and SLOs's around the overall execution time endtoend for your workload?
If yes, they may you may have less flexibility there, but if not, uh there that kind of gives you more options for optimization levers. The most disciplined customers take a top- down approach to cost and performance optimization. Often the lowhanging fruit here is with waste reduction. Really, this is identifying objects that could be de decommissioned. they could the refresh frequency of these objects could be reduced as well because the business value just isn't justifying the cost that these workloads are emitting.
Okay, maybe you prune those objects and what you're left with is really critical business use cases on the platform. The next step is then right sizing and this kind of goes back to the previous slide. How do you actually align the compute and the instance choices that you're selecting in your configuration to best align with the technical requirements based off the use case? Once that's done, of course, looking at the data and code layer is important as well with data based off the access and the query patterns on on the data tables you're looking to optimize.
Are there more efficient partitions and clustering techniques that you can use? And with code, how do you best structure that code so it's efficient on your engine? As a bonus uh tip, do try the optimize method on the data bricks assistant that will actually give you prepackaged optimized code based off the query that you've written. And this isn't just theory. We we've seen customers do this successfully through a datadriven PHOps lens.
This is a real world case study of a customer that was able to achieve actually millions in annual savings through leveraging system tables for optimization identification and prioritization. How are they able to do that? Well, like I said, waste reduction usually is the is the lowhanging fruit here. They actually realized hundreds and thousands of spend came from legacy pipelines that were running post a migration that were no longer actually useful in terms of the data it was outputting.
They also saw some pockets of optimization from turning off photon photon in non-production environments. You know photon's really great for rapid fast performance execution but maybe in a non-production environment you don't really need that type of SLA or SLO requirement. They also saw opportunities for cost reduction from improving auto termination rules on a cluster. They had clusters that had over hundreds of minutes of auto termination as a policy.
So these questions were actually up even though they weren't running workloads or or were of val of value. And sort of last and this is actually a very sort of favorite cost-saving technique that we often try to sort of teach our customers is for your job workloads migrating them from all-purpose clusters to jobs compute to take advantage of both better pricing on jobs but also so jobs compute scale up and down based off actual workload needs.
Awesome. So in the demo we showed how to find usage that may be large overspending. Um I want to cover now three examples uh that as an SA leader I am talking to customers all the time about these three examples on once you find usage that may be high what do you do? What do you do about that? How do you technically solve that problem? And the top three problem areas and solutions that we have uh that are open source and we'll provide at the end of this presentation are jobs.
So, how do we rightsize and manage all of our jobs across our ecosystem? We may have thousands, tens of thousands of jobs. How do we manage that at scale? The second one is SQarehouse sizing, performance management. The third one is really table health and storage uh storage management as well. So speaking to example one, um you can utilize system tables and we have an open source dashboard built by our field engineering experts uh that utilize these system tables out of the box to do things like calculate cost per job.
Uh manage job SLAs's and runtimes. So find your jobs that are kind of above your SLAs's that you need to find and fix. Um look at cost per job, cost per owner of a job. So, going back to that who owns it uh scenario, how do I know who's spending what on the platform? Who do I go to rightsize spend? Most interestingly, um nearly all customers want to know two things. What are their top jobs that are maybe under or oversized?
So, both can contribute to overspend. If your jobs are undersized, then your compute can be running much much longer um than than it would need to if uh if you rightsize your compute. This is typically very unintuitive for customers. They think if my compute is smaller then I'm spending less money. But uh that is not that is not always the case. So it's good to see both sides. Um the other side of that coin that's that's equally if not more popular is how do I find my jobs at scale that are not utilizing all the compute they have.
You know it's very easy uh because data bricks is easy to use. you can create a cluster and you know run a job, put it in production and not actually realize you may need like half the compute that you actually deployed or even less. So this dashboard is a out of the box solution accelerator built by our field expert team um just to help find that burrito analysis of your top over and undersized jobs and uh another example of top cost by data bricks runtime.
That's a really common question um for customers that have old run times that are kind of aging out. um those often cause performance problems over time as data increases. If you don't keep up with the run times of DVR, you know, maybe you have some runtimes that are three, four years old. Um that can actually end up costing you more than helping you and it may be worth the migration effort to the newer run times. So this helps identify and fix some of those key issues.
Example number two, my personal favorite, uh is SQL warehousing. So SQL warehousing uh relatively newer on data bricks. By that I mean four years or so. um but now uh has really gained traction all throughout the company all throughout the industry and we've spent the last couple years really building uh open and robust tools to manage warehouse performance. So you can utilize newer system tables such as the compute the warehouses system table query history and warehouse scaling system table to manage warehouse performance.
So actually measure SLA for one or many warehouses uh and then uh identify scaling trends. So, does your warehouse need to scale more for concurrency or does it need a larger uh compute size? So, that question although it seems simple is often hard to answer without a top-down view of what your workload shape looks like. And this warehouse advisor dashboard provides that out of the box. Lastly, um doing that sort of top down approach that has mentioned.
It's very important in warehousing because otherwise you could just be endlessly chasing queries that look expensive that don't actually help fix your warehouse level problems. Um but we do have a section that allows you to quickly click into the common issues um such as high queuing time, high result fetch time, disk spillage um to find and fix bad queries very very quickly from a top- down lens. So very popular dashboard.
Um and then the third example is table optimization and storage. So I'm sure uh most of you here have probably a good chunk of tables that you don't even know if anyone uses them yet jobs are writing to them. Um how much is that cost? Uh we have open tools where customers have either built themselves or extended our uh subject matter expert builds dashboards to identify stale tables. Maybe there's jobs writing to tables, but they haven't been read from in over six months.
Well, we can delete those. That's wasting storage. That's wasting ETL capability. Maybe we want to actually charge back the storage of those tables, calculate cost per access. Well, we can utilize those lineage tables in data bricks as well as our other system tables uh to do some of that analysis. Um and so these are two examples of dashboards um that we that we have available open to be able to do those things. And those are really the three most popular use cases that I see.
Getting to that optimized section of finding that bloated spend and then fixing that um in the actual product itself and all these are open solutions that you can extend yourself as well. Okay, wrapping it up. This this is our third pillar operate. So again going back to the framework this is actually about great you know you know this data is available. There are dashboards you can use to optimize the spend. But as your as the user base on your platform is growing, how do you actually build this continuous sort of feedback mechanism loop for fin?
31:30 Operate: Continuous management
And what are tools that you can use to actually automate that? We'll speak to one of our one of the favorites and by show of hands is has anyone used the budget tool on data bricks before? It's in the account console. So maybe maybe a fewer. It's tucked away in the account console. They're working on that as well. But budgets allow you to define and share alerts when spend has actually exceeded a predefined budget threshold that you set.
Those alerts can be shared with downstream teams, with your management team, whoever uh is relevant. And those budgets can be set both at the account level, workspace level, or at a combination of tabs. And based on your feedback, we're excited to announce a couple of enhancements coming to the budget tool. One will be richer visualizations that will allow you to more quickly identify what the actual drivers of the budget override are, more configurable thresholds to define custom alerts and get alerts earlier based based on what your what what level of noise you find acceptable and a improved tag search experience.
You know, we know customers love using tags to identify how to define a budget and improving that experience to show you what tags are most popular and high usage will allow you to accomplish that better. Taking a step back, one of the themes we we hear often is idea of shifting left on on PinpOPS. And what does that mean? You know, traditionally the personas that you think of as being responsible for Pinpops are individuals like this, maybe finance management, IT asset management, especially for SAS licenses.
But really on data bricks, as you know, the spend generation starts far upstream of that maybe BI analysts to ML engineers depending on what products you're using. And the idea of shifting left on PINOPS is empowering those individuals to have access to that data and ensure that they comply with those cost governance from the get-go as well. How do you do that? On the transparency front, we talked about budget alerts as a way to accomplish that.
But also the system tables that I mentioned, those are designed to be as widely shared or not shared as as you desire. They're using the UC catalog and permissions there. You can share those tables widely. We have examples of customers that make those cost dashboards like demoed accessible to all of their downstream users for transparency sake as well as automation. We have cost governance policies, compute policies, serverless budget policies as well.
Those can be those can be encoded in Terraform. So even upon the time of infrastructure provisioning those policies are applied. You can autocreate budgets with the count APIs. Every time you create a new workspace a budget is autocreated as well. and then using data bricks asset bundles to ensure that as you're deploying jobs and pipelines across environments you're always using the same cost optimized template. So kind of zooming out on that same thread um in in the field I generally see a a gradient and spectrum of kind of two different customer types uh that try to implement the sort of observability governance and finance management framework.
The first one is sort of the monolithic approach. So customers will create a core data platform team uh that governs and tightly controls all billable objects that go into data bricks. So if a user, a data scientist, analyst, data engineer wants to create a job or a warehouse or a cluster, they then have to file a ticket, go through some rigid process to do that. Um and then you know get that cluster created by the data platform team.
Uh this is really great because it uh has generally 100% categorization uh out of the box. So out of the box you have instant visibility into spend but this can very quickly bottleneck your teams and it requires sort of more technical expertise more process overhead by not just your core data platform team but all of your downstream teams that have to know the system required to comply with it. What we see more recently especially as the modern data stack and the data mesh concepts um gain in popularity uh is this sort of federated approach where you still have a data platform team uh or a PHOPS team that's responsible for uh managing spin but instead of uh gating everything uh this team creates a single policy layer maybe it's very thin layer just a couple standard set of tags to use that all billable objects need and then they federate found the how to the to the remaining teams.
So all the different data teams can use whatever process they want to create, manage, deploy clusters, jobs, and as long as they have the appropriate tags, then that usage is great. And then all the data platform team needs to do is create a monitoring and alerting system on top of that then either deletes the billable objects that are not compliant or um this is the most popular option is to create an observability framework that identifies untagged usage or mistagged usage that we just showed in the dashboard.
You can simply schedule that as an alert, automate that in data bricks SQL natively and maybe give some of those users, owners of that usage a week, a couple days to fix that and get it into policy or maybe justify the spend. Um, so kind of building that framework up in a more federated way uh lets these data teams that are downstream run quite a bit faster and not feel like they have to like walk on glass to every time they want to add a new use case, but you still get clear visibility.
So the downside to that is you may not always get immediate compliance, but you have a very seamless way to find and fix uh usage that isn't tagged properly. And there's kind of a whole suite in between. Um but this federated approach has been more popular in the recent uh recent uh trends here and going to that operate level. So we've found out how to find fixed usage, optimize spend. How do we actually automate that?
So we actually have a ton of solution accelerators built by our field teams and our customers um to do things like setting up alerts on forecast and spin. Uh so that dashboard that we reviewed, you can actually just pull out those queries, create a data brick SQL alert and schedule an alert on any condition. So if you want to uh identify when forecasted spend increases maybe more than 20% compared to last week, all you have to do is copy uh that existing account usage query, schedule it as an alert, and you're good to go.
You can automatically get those alerts. Uh same thing for automatically notifying owners of untagged or mistagged usage. That tag detail table that we looked at the end, um you can just make that an alert. You can create entire monitoring system out of that and then even do that for team level contributions. Uh and then lastly for performance. So that warehouse advisor dashboard, those jobs dashboards, all of these you can build alerting systems to identify uh when jobs or queries are running longer than an SLA or a warehouse is scaled up for too long.
All of these things can be easily pulled out of those dashboards and automated via alerts. We can provide some cool examples of some of those especially the SLA monitoring is a very popular solution built by our field. And hopefully you found this helpful. Um that is the end of our talk. But thank you so much for joining. Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.