Easy Ways to Optimize Your Databricks Costs

Summary

• Usage observability: Organizations can understand Databricks usage patterns using AI/BI tools and System Tables to identify cost optimization opportunities and track spending effectively.
• Cost organization and control: Tagging strategies combined with policy enforcement enable organizations to organize costs by business unit and set budgets that align with operational needs.
• Strategic budget management: System Tables and cost monitoring dashboards help teams identify cost spikes, forecast spending, and allocate budgets across projects and business units efficiently.

Easy Ways to Optimize Your Databricks Costs

Talk by Yassine Essawabi, Senior Resident Solutions Architect, Databricks ; Youssef Mrini, Solutions Architect, Databricks
In this video, we will explore effective strategies for optimizing costs on the Databricks platform, a leading solution for handling large-scale data workloads. Databricks, known for its open and unified approach, offers several tools and methodologies to ensure users can maximize their return on investment (ROI) while managing expenses efficiently. Key points: Understanding usage with AI/BI tools Organizing costs with tagging Setting up budgets Leveraging System Tables By the end of this video, you will have a comprehensive understanding of how to leverage Databricks' built-in tools for cost optimization, ensuring that their data and AI projects not only deliver value but do so in a cost-effective manner. this video is ideal for data engineers, financial analysts, and decision-makers looking to enhance their organization’s efficiency and financial performance through strategic cost management on Databricks.
Intelligent analytics for everyone: https://www.databricks.com/product/business-intelligence
BI meets AI: https://www.databricks.com/resources/ebook/business-intelligence-meets-ai
Explore the only unified platform for agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction
• 1:00 Observability
• 1:30 Agenda
• 2:42 Customer User Journey
• 3:49 Three Steps
• 4:24 Tags and Policies
• 10:56 System Tables
• 16:37 Budget Alerts
• 26:32 Recap
• 27:04 Demo

FAQs

How can organizations monitor Databricks spending?

Organizations can use AI/BI tools and System Tables to understand usage patterns and track costs. These observability tools provide visibility into spending by workflow, pipeline, and endpoint.

What is the role of tagging in cost optimization?

Tags help organize costs by business unit, project, and resource type. Combined with policy enforcement, tagging enables organizations to track spending at a granular level and allocate budgets appropriately.

How do budget alerts help with cost management?

Budget alerts notify teams when spending approaches defined thresholds. This proactive approach helps identify unexpected cost spikes and enables timely corrective action.

What System Tables are useful for cost monitoring?

System Tables provide detailed information about resource utilization, spending patterns, and account usage. This data enables teams to create dashboards that monitor costs, track efficiency, and forecast future spending.

Full transcript

0:00 Introduction

So good morning all. We're super happy to have you at an 8 a. m. session. So thank you so much for being here. I'm a solution architect based in Paris and I've been working for database for the last couple years. Yeah. Hi everyone. Thanks for being here and thanks for coming to this very early session. Um I'm super excited to be here to be presenting with you. I'm part of the professional services team at data bricks in Singapore.
Uh so I've been with data bricks for five years now and I've been helping customers design and deploy their lakehouse uh platform and this covers topics from governance, security, automation. But across all of these uh engagements, there was one topic that was often uh underplanned or overlooked. It's observability. So observability allows you to monitor the behavior of your system and also track all the activities and like identify suspicious activities and authorized access.

1:00 Observability

Monitor the cost spikes that you might have across your workflows, your pipelines, your endpoints and also make sure the resource utiliz utilization efficiency across all of your environments. So that's why today we will be talking uh we'll be following this agenda where we start by talking about best practices for observability. How to set up properly define and configure your tags. How to enforce these tags using databicks policies.

1:30 Agenda

We will cover the different types of policies. And next we will walk you through all the different types of databris system tables that provide you all the necessary information to monitor your databicks platform. And next we will see uh deep dive uh uh with using different detailed examples of dashboards to for example monitor account usage uh through our dashboards, monitor lakeflow usage and also monitor SQL uh database SQL usage and at the end we will be as a next step we will be showing you how you can leverage databicks demos in order to kickstart your monitoring. and your observability journey by using some pre-built dashboards that we have available in the datab bricks uh demo center.

2:42 Customer User Journey

All right. So, uh before we start, I just want to go to quickly go through the observability COG customer user journey that we can break into three different components. So, we have PHOPS. FHOPS allow you uh to monitor the overall spend, drill down by business unit, by project, forecast the spend for the upcoming quarters, upcoming months and also define the budgets for different business units, right? Then we have DevOps.
So DevOps allows you to monitor the health of your ML models, the health of your data sets, monitor the reliability of your pipelines, your workflows on data bricks. And next we have sec ops. This is where you want to monitor data access like who's accessing what if there is like if you can identify any unauthorized access your data through your model endpoints or through your uh any assets within unity catalog and throughout this whole presentation we will be following these three steps.

3:49 Three Steps

First we have setup, how to properly configure tags and policies. Then monitor, how to leverage system tables, build uh your queries and your dashboards for your observability. And the last step is how to manage and leverage these dashboards by setting alerts, sending uh notification to the right stakeholder in case of a cost spike, in case of an author unauthorized access, etc. Okay, so starting with observability best practices, we will be talking mainly about two things, tags and policies.

4:24 Tags and Policies

So what's a datab bricks uh tag? A databicks tag is a key uh value pair that you can set at the level of each databicks resource. For example, at the workflow level, uh compute like clusters, endpoints, any type of resource, you can define the tags. But the definition of tags will depend on your organization like how uh like what level of granularity you want to have at your organization, right? But we usually see this these type of and recommend these type of tags.
So you want to set the generic keys like the business units, the project, the environment as well. And if you want more granularity then you can add things like the builder for example if it's a specific team that develop some uh ETL pipelines and using certain compute if it's a partner and other uh other tags like the version the role the products etc. Now once we have the tags defined and created within data bricks, we want to find a way to actually enforce these tags.
Make sure that whenever uh user a group of users or service principle are creating certain resources, they are making sure that they add these tags to these resources. And this is where uh policies uh become very handy when you deploy and when you configure your platform and databicks uh policy we have two types. We can start with cluster policies. So cluster policies allow you to enforce the right configuration in your organization.
For example, you want to enforce the version, the run runtime version to be to always be the latest version when creating a cluster. You want to enforce, for example, the usage of auto termination. You want to enforce the type of workers or the spark version that needs to be that must be used by uh by your team. But in our context and more importantly, cluster policies can allow you to enforce the usage of tags, right?
So whenever a user a group of user a service principle CI/CD pipeline terraform try to deploy certain resources if the tag is not defined that resource will not be deployed. So if I go through my CI/CD pipeline and try to deploy some resource it will fail and I need to first define the tags before deploying these resources. And here you can see the an example of where I define the cluster policy and in permissions you will be able to assign this policy to a user or a group of users.
So whenever they try to do anything they need to follow the recommendations that you try to enforce with this policy. And here you can see uh different examples of the different levels where you can use uh data bricks policies. So for example in clusters uh in the bottom left you can see the definition of tags that I'm adding to my uh cluster when I create the cluster. The one in the middle you see a database workflow again it defines the tag for example the team it's the engineering team and the last one is the SQL warehouse or SQL endpoints again we see how the tags are defined when we create the SQL warehouse.
Now once you define the cluster policy you define what are the tags and then you grant as you can see on the bottom right you grant this uh policy not grant you assign this policy to a certain group of users. And for this one, it's uh not just the cluster policy. It's more the budget policy where you can define the budgets and the tags for this budget and then you assign this budget policy to a user or a group of users.
So for this specific one it's uh we covered at the beginning cluster policies for classic compute or classic clusters and this one it's more for serverless when you start using serverless on database. So when you are using serverless they you're not creating a specific cluster you're just defining the budget policy for serverless and you assign it to your different user or different groups of users. So if I go to the next slide as a user when I go to my notebook and I try to use uh uh try to use serverless you can see on the right side I can pick which budget policy I want to use and it has the tags predefined for that budget policy and the idea here is that all your users or your teams are leveraging serverless but they have the t tags assigned or attached to the these users.
So whatever they do, you can use this to track their consumption and their usage for by specific uh teams or by specific business units. And similarly, if you're not using notebooks and using jobs with serverless, you can see that here you define the budget policy and the budget policy tags is set to engineering. The same logic would apply for uh for server for serving endpoints. Again, you will define the policy and the tags when you're creating the serving endpoints.
So when the user are using this, you can track and drill down the consumption and you usage by this uh specific team. All right. So now that we have defined and configured our tags and also defined the policies, we will see how we can use database uh system tables in order to build our dashboards and to moni like start up the our observability journey and define the different dashboards uh that we want. But before starting what's a databicks system table?

10:56 System Tables

So system table is a list of tables provided automatically by data bricks that contains operational data about your databicks platform. So whatever is happening in your database account, your database workspaces, it's automatically tracked in system tables that you can find in your platform. And we have different types of system tables that we will go through and then when we start configuring the dashboards, we will see we will show you how you can leverage these tables to create your dashboards.
So first one we have audit. I will try to break them down by categories. So first category we have audit access and lineage. In this one uh you can you have things like column lineage and table lineage that tracks and gives you a record for each read and write operation on your columns or on your tables. You have things like audits that tracks literally every activity happening within your workspace or within your accounts.
For example, if someone tries to create a cluster, any any operation happening in your platform will be tracked in the uh audit table. On the other side, we have billing. So, billing allows you to track the consumption of uh or how much you are using the platform. for example, how much certain workflows or certain clusters are consuming or certain endpoints are consuming. And since we defined our tags, I can use this data and I can filter by the tag and I have the usage or consumption by uh business unit or by project.
Next one we have compute. So in compute we have all the different computes uh provided by data bricks clusters uh warehouses for instance uh clusters and warehouses it's a slowly changing dimension uh uh data that keeps track of the different configuration changes within your uh clusters or endpoints and this can be really useful to monitor for example configuration drifts if you see that someone changed some certain configuration ation or you want to roll back to a previous configuration of your clusters.
Next one we have lake flow. So lake flow it's a the com combination of database jobs and DT or delta live tables. And similarly here we have two things. We have the timeline. The timeline allows you to track the duration of your pipelines. And this is super useful because you can see if there there's any spike in the duration or run of your workflows. And then you have jobs, job tasks and pipelines. Again, it's CDC data that tracks the configuration of your jobs of your uh of your tasks so that you can keep monitoring if there is any configuration uh drift.
Marketplace uh we have two things in marketplace. the impression like how people are u accessing trying to access your or viewing your listings on marketplace and the second one is the access events. So any requests or approved requests will be automatically tracked in this uh marketplace system table. uh predictive uh optimization uh predictive optimization is done automatically by data bricks and here you will find the list of uh operation that data bricks did for example vacuum compaction clustering so you can monitor what's happening uh what what kind of optimization are happening to your uh tables and query history it's straightforward it's the list of all the queries that you uh that the team or your users executed using SQL endpoints.
So you can keep track and again you can use them later to see the longest running queries. You can drill down by tags, by business unit, by project or by user. Next we have serving. Uh so serving you have the list of all the endpoints or uh the models that you are serving and their configuration and also you can see the usage and the consumption of these endpoints and similarly you can break it down by business unit or by users to see how much they are using uh these endpoints.
This one is new uh and in in short it's the equivalent of what you see in the UI of MLflow. All the runs all the experimentations experiments that you do on MLflow you can find them directly in a system table provided by data bricks and you can use it later in your for example in your MLO pipeline uh to go through all the different runs and different experiments. All right, thank you for all your hard work. As you can see, he enables he did enable system tables.

16:37 Budget Alerts

He did the tagging which is super super critical for our observability journey. So just to summarize, we have tags that can be applied through cluster policies for classical compute and we have serverless budget policies for all the serverless compute. then every single compute or that will be created after those uh those uh those rules will be automatically tagged. So this is very very important. Then we will leverage system tables to deploy dashboard that will help you track all your costs and let's start first before going through the dashboard going to talk about something called budget alerts.
This one is very critical. So basically what you can do you can define the name of the alert. It can be for example your uh like uh I don't know the business unit or maybe a specific use case called for I don't know data engineering you can specify the amount of the budget maybe $3,000 per month specify one workspace or multiple workspaces and then based on the tags that you see defined maybe project equal to data engineering I will just go like below and define the email that will receive the alert in case we are above the budget basically can be maybe the finance team or the phenops phob team they will automatically receive those emails if you are above the threshold then you can also track if you are on track or like below so this is super super critical and this is the example as you can see I set up the budget of $35,000 like remaining 20 20,000 of course I spent 14,000 you can see the percentage and of course you can also track the current usage um in case I am above the consumption will receive the emails.
And the next one is the usage dashboard. This dashboard is very very important. You can uh you can import it if you are like the account if you have access to the accounts console or you can just upload the JSON code that will deploy this dashboard for you. And this one is super critical because you will be able to identify like the uh the cost by skew by workspace the cost by day. So all the things that you should be aware of will be deployed for you automatically.
Let me let me show you how it looks like. But first you need to go to the account console for those who have access to it. Click on usage then consumption and click on set up the dashboard. This will automatically set up the dashboard for you or you can just deploy it in import this uh dashboard in your workspace. And once you're done, you're going to have this dashboard. As you can see, I can view uh like the aggregation by week.
I can view by building project, by products, by skew, by workspace. So, I can basically track the daily consumption. For example, I know that in SQL like from like in one year I spent almost 1 million. I can see the variation whether like I have an increase of consumption or maybe a decrease by by skew. If you have tags set up, you can also filter by tag by project. And this is this is brand new, but the version I'm I'm showing you right now, this is the current version, but tomorrow uh, the PM of uh budget, like PM of money team, will announce the V2 uh which is like a brand new uh brand new experience with a better UI.
As you can see uh you will have like the total usage the aggregation by product the daily costs and like the the unit the skew the workspace. So we have a lot of things and of course this is just an example but you can go beyond those dashboard just by leveraging system tables and you can add specific metrics that you want to see on on your dashboard and experience. So this is very very important. We give you the baseline and then it's up to you to modify to tailor it to your own experience which means you no you don't no longer need to reinvent the wheel just customize it and for the second dashboard it's called lakeflow dashboard and this one I find it's more critical than the than the first one and this one provide you with a comprehensive monitoring of all your jobs jobs and pipeline experience and it can really help you optimize your costs understand the health of your workflows and who's driving the most the most usage and let's go to let's go and discover the dashboards will be much more much easier in the meantime so as I mentioned this one will help you identify the most frequent job the most under like maybe under uh underused resource which means you can like resize those type of clusters and also track the most contrib contributors or the jobs that fails the most.
Um, oops. Yeah, here we go. So, as you can see, I can track the cost for like the last uh the last 13 days. I can see the aggregation of the cost. But let me show you the most important one. So, you can see, of course, like the rate of success, failure, skipped, cancelled, cancelled jobs, and then you can see the most important part, which is the most expensive jobs for the last 13 days. So you can see this one costed $3,755 with four runs.
So basically you can just aggregate and see the most expensive jobs. Maybe those are the ones that you need to optimize or maybe switch them to serverless in case you're using classic classic compute. And that's not everything because sometimes the job can be very cheap, but a specific run did was like super expensive, which means like the basic run like is maybe $5, but one specific run did it cost $300. So you can see those those uh those runs over here.
So I can see that this this run ID cost $1,483. So I can with the run ID, I can open the job UI and understand what went wrong. Maybe I had a spike in the data. Maybe I don't know someone pushed the wrong join. So this is really this is really really important. And of course uh I can also see the variation of the changes. So for example, one job did that cost $7 and the next month started costing maybe 400. So it's really easy to go and say okay something went wrong here or then try to understand how you can solve this issue or maybe you just had more data and this is good for you which means you're bringing more AOE a to your business and we have also job cluster analysis and this is for the optimization so you can see like the jobs with the more or the most potential savings.
So I know that this dashboard delta to delt this I mean job delta to delta uh deserve to have a look at it and maybe uh go optimize it. And you can also see like the jobs with the lowest CPU usage which means I can resize this this cluster and the one with the most highest average of memory which means I need to resize it and increase the size else I may face the out of out of memory errors. Of course, keep in mind that some of those metrics uh won't be available if you're using serverless because data bricks optimize uh your workflow the workflows on on your behalf.
So you have a lot of interesting things and this also one of this also very important in case you're using some outdated DBRs and make sure to always use the latest one and mainly the LTS sometimes can optimize your jobs and you can benefit from you can benefit from it and maybe if forgot to do his job uh you can have a look at the antag jobs which can help you like furthermore aggregate the aggregate the cost and we have a lot of interesting things like the jobs that are using allpurpose clusters or what we call the interactive ones.
So it is worth like switching to job clusters because they're 30% cheaper. Um let's go to the next one. So this one is tailored for data analyst. It's going to help you track the cost of every single query you're running in uh your class. And you will have I think there is a talk a deep dive talk about it. But what you can do you can just you can either like have a look at the cost for all the for all the types or you can just say okay let me know like the cost for example for all the queries that I did run on AIBI dashboards or Genie or the one I did run on Tableau PowerBI or any any tool using uh this the SQL warehouses.
Of course, you will able to see the number of queries I did run the user the user count and I can also see the aggregations by source type. So, as you can see here, like AIBI dashboard costed cost almost $4,000, SQL queries $800 and you can also see like the like which one costed the most like Riot here did run a lot of a lot of uh queries and then you can like either aggregate the cost by day or by week or by month.
Of course, it's up to you, but the dashboard is already deployed. And then you have the most important one. You can see like the most expensive queries like and the query how much it did cost. And you can basically click on the statement ID. It will take you to this to the uh query profiler where we can try to understand how you can improve uh this this query maybe optimize optimize it further more to reduce uh to reduce the cost.
So this is this is a very very important dashboard that you need uh you would need to use and then we have debit demos that will help you kickstart your journey within database. All right. So uh hopefully you learned something new. You're excited to start using system tables to kickstart your observability and monitoring uh in your organization. Right now, you want to go back to your work and you don't want to build all of this from scratch.

26:32 Recap

Maybe if you have something pre-built, something that you can leverage from this session. This is where you can go to database demo uh demo center and use the DB demo examples that we have especially this one. So if you follow this you install DB demos in your notebook and you select the system tables uh demo what it does it will automatically install all the dashboards that we have pre-built for the for monitoring and I will walk you through some uh very cool examples some detailed example that will help you a lot uh to kickstart your uh observability journey.

27:04 Demo

So first one uh in this one we can uh see the list of workspaces and we can track the consumption by each workspace uh per day right. So you can see for each day what are the top consuming workspaces. So if in your design you are uh building one workspace per business units or one workspace per business domain here you can see what which business unit or which business domain is consuming the most and the next one what we can see is for a specific workspace.
Now you had a overview of the consumption through workspaces. If you focus on one specific workspace now you can see the consumption of warehouses the top warehouses within that workspace. So in this one we have the list of uh warehouses for each day we have the distribution of how much each of these warehouses are uh is consuming. So we started by workspace and then we went uh we started by uh monitoring all the workspaces.
Then we focus on a specific workspace to see all the different warehouses. Now we can do the same thing for users to have the top consuming users uh using databicks warehouses right and here I can even filter it by the query source. So for instance it's ad hoc queries AI BI dashboards alerts genius space workflow and then you can further drill down to f focus on a specific type of query and see the consumption and usage for that type of query.
Next one is we is uh will allow you to monitor the costs per endpoint or per per model serving. And here we can see the different types of model serving like found foundation models, CPU models, GPU models and again you can the next step can be to filter by the tag that we created earlier. So to you can focus on a specific team or a specific uh use case or business unit. All right. And again now if we focus on a specific workspace I can see the daily consumption by uh model serving endpoints.
And on the right side you can see again the different types of model endpoints CPU model feature foundation model and so on. The next one, we want to track the weekly consumption uh in your workspaces and also track the diff the health and reliability of your work uh workflows or jobs like how much jobs failed uh running succeeded and use all of this data to monitor the health and performance of your uh of your workflows.
And of course this data you can then probably use it use all the history of your data to actually forecast the spending for the upcoming months to help you define the budgets for each business unit. All right. So we focus on endpoints at the beginning. We can do the exact same thing by focusing on workflows. So here I have the daily consumption or the daily usage by workflow. We can focus for instance on specific business units in the production environment or production workspace and we can track the job execution and get the dollar or the uh consumption value for each workflow.
And of course, if you see any spikes, you you should set up alerts to notify the stake the right stakeholders and go troubleshoot to see if it's a normal behavior, if it's just an unexpected spike, or if you need to change and optimize your work. All right, and that's it. All right, thanks.

Learn more about the Databricks Data and AI platform.