Enterprise Cost Management for Data Warehousing with Databricks SQL

Summary

• Cost Visibility and Attribution: Databricks SQL provides detailed insights into spending with attribution to internal projects, enabling organizations to understand Total Cost of Ownership.
• Proactive Cost Controls: Learn how to set up proactive controls and monitoring to continuously optimize Databricks SQL spend at warehouse, user, query, and table levels.
• Cost Governance Framework: This video shares best practices for managing complex Databricks SQL deployments with multiple workspaces, warehouses, and user groups across organizational teams.

Enterprise Cost Management for Data Warehousing with Databricks SQL

Talk by Joo Ho Yeo, Product Manager, Databricks ; Patrick Yang, Staff Software Engineer, Databricks
this video shows you how to gain visibility into your Databricks SQL spend and ensure cost efficiency. Learn about the latest features to gain detailed insights into Databricks SQL expenses so you can easily monitor and control your costs. Find out how you can enable attribution to internal projects, understand the Total Cost of Ownership, set up proactive controls and find ways to continually optimize your spend.
Here's more to explore:
The best data warehouse is a lakehouse: https://www.databricks.com/product/databricks-sql
How to migrate legacy data warehouses: https://www.databricks.com/resources/ebook/migrate-your-legacy-data-warehouse-databricks
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why is Databricks SQL cost governance important?

As organizations scale their Databricks SQL deployments with multiple workspaces, warehouses, and users, cost governance becomes complex. Understanding spending patterns, attributing costs to business units, and optimizing usage helps control total cost of ownership and maximize ROI.

What cost visibility features does Databricks SQL provide?

Databricks SQL offers detailed insights into expenses with attribution to internal projects, enabling organizations to track spending at account, workspace, warehouse, user, and query levels. This granular visibility helps identify cost drivers and optimization opportunities.

How can organizations set up proactive cost controls?

Proactive controls include monitoring warehouse utilization, tracking user and query-level spending, optimizing table design and queries, and setting budget alerts. These controls help organizations prevent cost overruns and continuously improve spend efficiency across their deployment.

What optimization strategies are covered for Databricks SQL?

The video covers optimization at multiple levels: warehouse optimization for resource efficiency, user behavior analysis for consumption patterns, query optimization for performance per dollar, and table optimization through features like clustering. These strategies help maximize value from Databricks SQL investment.

Full transcript

Welcome everyone. Uh today we are here to talk about cost management with data bricks SQL. We'll share some tips and best practices on monitoring and optimizing your DBSQL build as well as demos to show you how to actually do it and take it into effect um in your uh deployments. product manager on the DBS SQL warehouse management and monitoring team. Joined here with who is a staff software engineer and one of the tech leads on our team.
Uh before we start, just some housekeeping notes. Some things we will share today here are forward-looking and may be subject to change. And at the end of uh our presentation, we'll have a survey for you to fill out. Before we dive into the how, let's first discuss why. Why does DBSQL cost governance matter? And to answer that question, let's start with what your deployment may have looked like on day one. You maybe you had one workspace with a starter warehouse, couple analysts are running notebooks on it, notebooks to query some information.
The value is limited. It's a great start. Uh but you had very simple cost governance when the bill went up. uh it's very easy to understand why why it went up and what the next steps uh to take are but today your deployment looks a little bit different probably something like this on your right you have multiple workspaces you have multiple warehouses you have your marketing team is using Tableau dashboards for one warehouse you have another uh set of users uh querying using SQL editor um perhaps on the same warehouse or a different warehouse there's There's a lot of value that we are helping you deliver to your organization.
But cost governance becomes very complex here. You can't just look at your bill to get all the answers for questions like who used what, uh how much, and why did it just why did my warehouse bill just go up by 20%. So, we're here today to guide you to simplify cost governance so you can continue on focusing on bringing value to your organization with DBSQL. We've split today's sessions into two parts. First, we'll help you understand your build at each level of granularity from the account level all the way down to the usage level.
We'll also cover how to how you can do this proactively and continuously um you know throughout your day. Then we'll show you how to optimize your DBS SQL usage, how to optimize warehouses, users, queries, and tables so you can get the best performance on every dollar you spend with us. And at the end of each session, will uh do a quick demo on how you can do this today. So before optimizing or scaling your usage, you have to first understand what your bill is today to set the baseline.
And the bill should be more than just a number. It should give you insights on how you're using DBSQL today. And we want to help you slice, attribute, and monitor spend at every level to help you fully grasp your DBSQL usage and be able to evaluate it. So let's begin. But the first question is where should we even start? We're simplifying this for you and suggesting that you start from the top at the account level. Follow the guide on the lefth hand side as we walk through cost monitoring step by step.
So we recommend starting at the account level to get a bird's eye view of all your warehouse spend across your workspaces. Find where the anomalies are and where it's worth your time to doubleclick and investigate further. You have two key tools here. First on the lefth hand side the account usage dashboard. This gives you a high-level visual breakdown by workspace and uh warehouse and you can set this up very easily in the account console.
On the right you have system tables. If you want more control or need to customize reporting, our billing system tables let you write SQL queries to analyze spend by any grouping of warehouses that you want. Covering the basics here. Hope all of you know this but the best practice for cost governance is to apply custom tags to your warehouses. So for example, if the marketing team has four warehouses, tag these warehouses with key team and value marketing and then these tags will propagate to the system table and the dashboard that we just covered.
And you'll be able to monitor all those costs together. So you can do things like showbacks, chargebacks, or just understand how each team's costs are trending over time. So now you've identified that the cost of this specific warehouse is going up. Are we done? No. You still need to understand what's driving. Is it? Is it PowerBI? Or is it a specific AIBI dashboard? But then you may ask, isn't billing at the warehouse level?
How can I get from warehouse level costs down to the specific user or dashboard's cost? And that's right. Data bricks charges for warehouse uptime. So in the billing system table, you'll find, you know, a row which has the billing information for that warehouse. But the good news is that with the other system tables that have query level information like the query history table, you can approximate the query attribution um very simply uh by using these tables.
For example, if a query took 2%age of the warehouse's compute time, it will be attributed with 2%age of its costs. And we've made this even simpler for you uh with a DBSQL cost materialized view that you can find in our data bricks labs GitHub repo. And we'll walk through a demo of how to use that. So now we've taken costs from the warehouse level down to the query level. But it's very unlikely that a single query is driving up all of your warehouse costs.
Uh so how do we get to the cost drivers? How do we get to the root of the drivers of cost? Again, is it? Is it PowerBI? How do we get to that? And this is where the DBSQL cost dashboard comes in, which is also available um in in uh data bricks labs uh GitHub repo. And this gives you the three views that you care about. How do I get my top users, top tools, and top data bricks assets. Now you can understand who, what, why are your costs changed and also use this dashboard as a starting point to manage the costs.
We'll do a quick demo on that shortly. So great. Uh, now I want to do this proactively and continuously because I don't want to be uh staring at my screen 247. And we got some solutions for you here as well. On your left hand side is budgets or out-of-box solution. So if the team's monthly BI workload should be $1,000, you can set a budget uh using the workspace and tags uh that match to that team's BI workloads to monitor progress and be notified when that $1,000 budget has been uh exceeded.
On the right hand side is our alerts product and you can use this for custom notifications. So, tying this back to the uh cost per query uh materialized view that we discussed, uh you can set it up such that the alert will notify you when any user's usage exceeds $100. We'll cover this in our demo as well. All right. All right. Thanks, Ju. So, yeah, I'm going to walk through uh a couple of the items that just presented.
Um so for starters before we get started um as mentioned the some of the tooling that we just presented available to you like the materialized view for co uh granular uh query costs as well as you know the dashboard that uses that materialized view are available currently um in the data bicks labs um repo. Um so this is currently because it's in private preview. So um you know you'll have to access the repo and essentially mount them yourself.
Um but it's pretty straightforward to import um you know this dashboard and recreate this materialized view. Um, so yeah, if you need to do so, you know, it's in the data bricks lab sandbox repo. It's pretty easy to find on Google as well. So yeah, why don't we jump into a usage example. So I've already gone ahead and imported um both the materialized view which remember it takes both the um the billing usage uh system table as well as the query history system table and combines them to build a cost per query model.
So I've taken that materializ imported it into my um data bricks account. I've also, you know, imported the dashboard that um JO was just mentioning, which uses that materialized view. Um and this dashboard is really cool. It allows you to kind of do this sort of slicing and dicing uh of the of the data that's present in the materialized view. Um based on a couple of different dimensions, right? Um so, for example, you know, we can kind of slice by like cost per user.
Um we can slice by source type. Um we can also slice by, you know, uh object source ID. So, source type is going to be like what tool it could be. So you know you're seeing some data bricks native tools like AIBI dashboards and SQL queries but also like you know external tools such as like PowerBI or Tableau. Um and then the source object ID is going to represent you know if it's an internal um you know data bricks um asset uh you know what uh what's the ID of the asset right um and then also it gives you kind of this breakdown based on all the queries um that it sees in this materialized view like the attributed cost per query right um so I'm going to quickly walk you through an example of like let's say I want to do a a little drill down and note this is an AIBI dashboard so you get the full power of kind of like the features that the AIBI team is working on to kind of like make it a lot easier to investigate um these sort of um uh items.
Um so for example, maybe I want to take a look at this user. Oh, look, it's, right? So like let's take a look at, you know, what been running on top of this uh workspace, right? So if I click this, you'll notice that, you know, all the AIBI goodness is happening. It's fil cross filtering across all the other visualizations. Um and he's mostly been running AIBI dashboards, right? In particular, this dashboard, right? So this dashboard is kind of interesting.
So let's kind of filter through through that to take a a closer look. So you know by doing that um you'll notice that uh all of the uh queries that um you know are in the statement level detail portion of the dashboard have been filtered to just that dashboard. Um so we can take a look at the individual queries maybe see if there's some hot queries or something like that from this dashboard. See what's going on here.
Um but for now for the purpose of this exercise what I'm going to do is I'm going to scroll over to here which um you know populates that query source ID which again is representing the ID of the dashboard that executed this query. Right? So this is a this a hot link. So like I can press this link. You know what I'll I'll be brought over to you know an actual dashboard definition. Um and here you know I can take a look at some of the dashboard.
Maybe maybe some parts of the dashboard loading slow or you know maybe it's like a really aggressive schedule. So in this case you know I'm going to happen to take a look at the schedule button here. So I notice that hey put an hourly schedule on this dashboard. That's a little wasteful right? So, why don't we go ahead and edit this dashboard uh schedule to reset this to maybe once a day. I don't think JO needs to like, you know, get hourly freshness on his dashboard, right?
Um, so I can save this. Um, you know, if I want to be really aggressive, I can even go in and pause it, but I'm going to be a bit nice. I keep his schedule for that there. So, um, but yeah, this is just like one quick example of like a walkthrough of like what you can what the power of this like materialized view and this um this new example dashboard can do, right? It allows you to do all these sorts of like investigations and drill drill downs much easier.
All right, so the next thing I wanted to show off um is going to be um at the account level. So we talked about budgets um and the new budgeting tools that we provide you at the account level. Um so if you're familiar um with the data bricks account um UI, you might be familiar with this page. This is the account home, right? This is where you go if you want to manage your warehouses or sorry your workspaces, excuse me.
Um so for this one, we're going to take a look at the usage tab, right? Um so if we go into usage um and we go into budgets um you'll see that um you know you can provision these budgets essentially as mentioned um by going to this button here um and then you know setting for example like a specific amount you want this budget to represent right this is going to represent the for like the limit per month of the spend that you want to like uh monitor here so let's say I want to do it for a thousand let's call this like my new demo budget uh and then okay normally like you can select which workspaces are available here.
Um, so let me maybe just do a really quick refresh. Uh, it's because I didn't log in. Sorry about that. Yeah. So, let me just redo that really quickly. So, demo budget. Uh, and then, you know, I can select which workspaces I want this budget to apply. So, you know, maybe I'll just have it for all the workspaces, you know, and then, you know, as as mentioned, a best practice is to always tag your uh compute resources to make sure that you're only um you you might want to your budget to a specific, you know, team or something like that.
So, we can do something like team marketing. So, we're just targeting the marketing teams um warehouses. Um and then we can also add an email here. So um you know this this email will just be used when the budget is breached to kind of send a notification um to that email to say hey like you know um by the way you should take a look at this budget and we just went past the limits so so maybe I'll send it to myself right okay so created the budget um you know it takes a bit of time to populate so in the interest of time I have a couple of examples here um that I've set up previously so when the budget's like fully formed um you'll notice that you get this really cool like you know burndown visualization essentially um that you can track for like how how quickly are we burning through our budget on a daily basis.
Um you know nice metrics of like okay we have this month remaining you know of um you know this much spent and the the run rate. Um and just to show you an example of what uh an exceeded budget email might look like. I have this example in my inbox, right? So, you'll see that, you know, we'll send you an email that says like, hey, you know, based on this this budget name um and then like based on this budget amount, you know, we just exceeded it by this percent.
Um, as as well as like a quick detail of like, hey, like this it was because this budget was, you know, exceeded by like this workspace, right? Um so all right so the last thing I wanted to show off was you know uh kind of the fact that you know by providing you the um the details like in system tables as well as our like you know m materialized view that uses the existing system tables um you know we're we're doing this very consciously right because we we we're putting this on top of data bricks right which in my humble opinion is like the best data analytics platform in the market right um so what that means is that you can use all the power of data bricks to kind of customize um your um your budget analysis uh if you will, right?
Um so, for example, in this case, you know, I'm going to show off a bit about the new um alerts v2 um beta that we're just launching. Um but you can use the alerts v2 to kind of write queries on top of this existing data. Remember, it's just data inside of your lakehouse, right? Um so, right now I have this um you know, this uh alert set up. Uh if you take a look at the query here um what this query is basically saying is that um you know if any users uh like query uh like usage uh dollars estimation is above 10 for the last 30 days I want to send an alert right I want to send a notification to an email or a p uh you know a slack or something like that right um so so you know we have this configured and yeah just wanted to show you this quick example of just like one way that like you can use the power of like the existing data bricks platform to you know like do budget analysis right um And you know with the rest of the tools in the platform like notebooks or dashboards or etc. you know let your imagination go wild with however you want to you know customize this behavior.
So all right uh back to you uh Ju. All right. Um so we also have some additional features coming soon uh to make DBSQL monitoring experience even more powerful. First thing is schedules. So um as walked us through in the demo, it is possible to infer some of these scheduled workloads from the cost dashboard that we have available. Um but uh we uh it's it's still hard to see all this in one place and also see uh data including like hey is this schedule at the hourly level like who owns this uh schedule etc.
Uh next quarter we'll build we're building out system tables for all firstparty schedules um that will let you do this and you will be able to attribute class to each schedule and navigate uh to the parent asset like the notebook or the job uh to manage it. Next is query label. So sometimes the dimensions that we provide you like user ID or uh spec like client application is not sufficient for your cost monitoring needs.
Maybe you need to understand hey like which queries are coming from uh spe a specific Tableau dashboard or a specific project and do cost monitoring at that level. And with query labels uh which we are building out you'll be able to do this. You'll be able to apply uh optional query labels to each query execution um both from SQL language and directly from these external tools so that you can get the cost attribution at those levels.
Cool. So now we have some tools now. Now now hopefully you know some tools to understand your bill. Uh but that's only the first step. This section will be on how you can optimize uh for better ROI. Uh so that you know uh so that so what are the steps to make your deployment even more effective. So where do we start here and here are the four levers that you can uh play with for optimization. warehouse, user, query, and table.
And it really depends where your deployment is most unoptimized, but we'll cover all these four bases. Let's start with the warehouse level because the settings here that you set at the warehouse level can make or break your bill. I'm sure this is obvious, but the warehouse settings contains levers to control your spend. You can set your hourly spending limit with the t-shirt size and max cluster count. So, for example, if you're using an XL warehouse uh with two max clusters, 2 * 80 160 DBUs per hour will be the max uh spend limit for your warehouse.
You can also use autostop so that you can reduce warehouse uptime when there are no queries running. And lastly we have uh serverless which is our uh skew of recommendation for both best price and performance. So now that we understand some of these levers how can we use them to optimize warehouses. So first of all serverless SQL warehouse um uh first of all start with serverless SQL warehouse. With serverless SQL warehouse um you can get up to two times the performance um uh compared to classic and pro.
So serverless has all these features on the left side that lead to cost optimizations on the right side with faster startup times within the magnitude of seconds compared to minutes uh for other for classic and pro. Uh you can set a more aggressive autostop which will reduce uh warehouse uptime. We also have intelligence workflow management. Uh so using AI we are directing queries to the optimal clusters uh for query performance and this you get automatically improved utilization which leads to cost benefits as well.
Lastly we have faster autoscaling. So instead of creating multiple warehouses you can consider merging warehouses and let us do the horizontal scaling for you. With this you'll get a reduced uh TCO and high um with autoscale. Another step is to look at your current uh warehouse to understand where there are opportunities for optimization. So as you all know we have our monitoring UI as well as system tables uh to provide all these bits and pieces you need to evaluate your warehouse uh performance and where there are room for optimization.
For example, you can understand where is my warehouse uh when is my warehouse uh experiencing repeated queuing? Are my clusters reaching uh the max cluster limit and what is the P95 latency of my workloads and how is this changing week over week? So with this information, you can choose the right size for your warehouse. Just to note that everything about right sizing is a trade-off between performance and cost. If you're experiencing that, you know, the workloads are slow and they're not meeting the SLAs's of your organization, consider increasing your t-shirt size and max cluster count.
If you're constantly meeting the upper limit of your budget, then con consider reducing t-shirt size and setting a more aggressive autostop. Also, monitor for how your P90 latencies are changing so you know that the performance degradation is manageable. Lastly, if you if you are experiencing that P99 queries like a handful of long unoptimized queries are degrading your performance, instead of trying to change warehouse uh configurations, your best bet will be to optimize these queries directly and we'll cover that in the next section.
So, a common driver of warehouse costs are these P99 queries. Uh these longunning queries keep warehouses up and also hog resources causing noisy neighbor problems uh for the other workloads on your warehouse. So how do we solve for these long runaway and unoptimized queries? The first option is to use statement timeouts. The statement timeout will kill queries that go beyond a certain threshold and is a great way to uh manage these runaway queries that do more damage than good uh for your organization.
And today we have these timeouts at the workspace level. The issue is that depending on the workload type, you may expect different amounts of time for the query to compute. So for example, uh for a BI workload, you might expect it to take 10 minutes versus a ETL workload that may uh you may expect to uh take an hour. But today we only have this uh single uh lever that you can control. The good news is that this quarter we'll be previewing uh warehouse statement timeouts, which means that you can configure these timeouts at the warehouse level.
So you can specify, hey, for this warehouse that I use for my BI workload, set a 10-minute timeout and then for the ETL warehouse, 2-hour timeout. The other option to address these queries is to optimize them. And there's two options here. So the first one is on your left. You can ask assistant to help you optimize your query. You can do this before or after an execution and will do a demo uh shortly on how to do that.
The second is uh the more advanced step of using query profile to find bottlenecks. So recently we did a revamp of this uh of this UI and now we have the new top operators panel surfaces that new panel that surfaces the uh the bottlenecks and the hotspots in your query right away. Um so you can jump in and uh debug there. So now that we've optimized warehouses and queries, um the job is still not done. Uh many of you have multiple warehouses and governing how users use them correctly is a core part of the complexity of cost governance.
Are users accidentally choosing to run small interactive queries on large warehouses? provisioned for critical ETL workloads. Are these queries keeping large warehouses up and making cost isolation difficult? To address that issue, uh we've started to preview default warehouses. With the new default warehouse setting, admins can configure the warehouse users will be pre-selected to uh for their interactive SQL workloads.
When a user creates a query in the SQL editor, uh the default warehouse that you choose will be selected um automatically. Users can also override this and set for example if they have a warehouse provision for their team or their specific use case um which is likely for the power users. So uh with these two uh to levers uh we're providing flexibility and control at the same time. All right. What if you want to go more?
What do queries also have in common? Tables. Can tables be optimized? There's two steps here as well. The first step on your left side is to use manage tables to get predictive optimization. So instead of having to manually run commands to optimize your table, we'll do it for you and intelligently uh when the ROI is right for you. And through automated statistics and automatic liquid clustering uh customers can get up to 20 times query performance improvement uh with predictive optimization.
The second option is for more manual optimizations. Using system tables you can identify which tables are the common source of long queries. So we have lineage system tables that show for each query which tables were read and we also have the query history system tables that show hey which queries are the long queries. By using these two tables together you can identify which tables are creating uh are are the common source of long queries.
And one thing that you can do is that if you identify that uh joins between several tables are the root cause of your long queries, you can consider creating a materialized view so you can cache this expensive part rather than having to rerun this time after time. All right, back to for the demo. Cool. Thanks. So, yeah, I'm going to show off kind of a subset of what we just talked about or what Ju just talked about. So um first off I do want to show off our new like default warehouses um you know mechanism.
Um so you know back to the data bricks uh workspace. If you go to the settings panel um and if you go to uh the compute section of the workspace admin you now have this new setting here um called default warehouse right and what this will do is you know as an admin you can set this warehouse um to whatever you feel most comfortable with. Um generally what we'd recommend is something like a 2x small or an extra small right?
Um because what this is going to uh be used for is essentially when a user goes to open a new asset like they want to do some ad hoc querying that like that warehouse is going to be populated automatically right we're not forcing them to use that warehouse but you know if they're not thinking about it you know it's best to put it on something that you know doesn't impact the rest of your workloads right um so now that we've set it to you know WHM team you know if I go to the SQL editor and I create a new query um you'll notice that hey it's all populated WHM team you know previously if you used the product before you might know that it does this like it used to do this alphabetically.
So it's nice that you can kind of select which one you want users to kind of run their ad hoc workloads on. So that's that's pretty nice pretty nice quick optimization. So all right um so moving over to um kind of like the warehouse monitoring aspect. Um so you know here we're we're at the uh the SQL warehouses list page. Um if you've used data bricks SQL or you've tried to manage data bricks SQL before you've probably seen this before.
Um, so we're going to hop into one of these warehouses. Um, and uh, you know, this is the panel that you use to manage your warehouse. Um, and if you've looked at, you know, the execution stats for your warehouse before, you might have seen the monitoring tab before. Um, the the thing I want to highlight here is that we've done the team has done a really great job over the last year kind of like adding a lot of new features to this monitoring tab.
So, it's way more powerful than it was last year when we were presenting this. Um, so, you know, we have charts that kind of represent the peak query count. So this is meaning like you know at any given time how many queries are running on the warehouse um you the completed query count so basically you know in a given time window how many queries were completed as well as a number of running clusters at any given time right and I' I do like to show off that like you know we do we did some really cool work to like for example um make the cursor the crosshair like um you know follow when you're like scanning through um so for example it makes it really easy to compare like hey you know when this when this peak of ceued queries was running you know what was the cluster looking like or when the number cluster count looking like?
What was the completed query count looking like? You can compare these numbers really quickly. Um, as well as this really cool, you know, like if you do a cross drag, um, you know, it will like cross drag across all of them, but also, you know, like cross uh filter which queries you want to see in the history table underneath, right? Um, so if you wanted to say look at this exact peak, you could like filter just that peak and see like, hey, which queries were that were on that peak and which queries were completed during that time frame, right?
Um, so just to give you a couple of quick examples of what you might look for in this um in these charts like um so for example as as mentioned earlier like if you're seeing a lot of query queuing you know if this yellow bar is really high and you care a lot about you know latency um then you might want to increase the cluster count right that might be a signal to you that like hey I just don't have enough clusters in this warehouse to fulfill the load that the load spikes that I'm seeing right now right um and then some other things like um you know we can also confirm that like hey the some like uh data bricks like autostop behavior is working correctly, right?
Like after like quickly after the completed uh query count went to zero, we can see that um data bricks um you know ramp down the cluster to scaled scale the warehouse down to zero, right? Um so we can confirm that like hey all this is operating correctly, right? It just gives you the confidence to like know that like hey you know all these features that we're talking about up here actually work, right? Cool. All right.
Um and then the last thing I want to show off is going to be kind of the query level optimization techniques. um that are available to you. Um so the first one is kind of cool. Uh it's uh basically uh kind of like an optimize my query button, right? So I've run this query here. Um and now we have this button in the bottom right on the SQL editor that you can just press like and it'll give you some recommendations, right?
Um so we're using the data bricks assistant, you know, AI tooling to kind of take a look at your query profile and see like, hey, what is going on? Is there something I can change here? You know, um uh that can make my query uh like faster or like more optimal. Um so you know it'll give you some recommendations here. So uh for example one common thing that we see a lot of people uh you know run into and this is actually frequently a problem that I run into um when I'm quering data bricks internally is that I'm not adding uh a filter column like a wear clause for uh a table that is partitioned by some certain column right it's a really good recommendation to kind of do that espec so you're not scanning the entire you know two years worth of data you're scanning the week that you need to.
Um so we see the assistant say hey you know it's a good idea to filter data early right um so um you know it'll give you basically an example of an optimized query um and we can kind of take this and you can kind of show the diff say like hey these are the kind of the things that changed about the um the query text and then you can also just insert this into your um uh your SQL editor and then rerun it as you see fit right so for the purpose of time I'm not going to rerun it I'm running a bit low on time here but I did want to show you like if you're you know like if you want to go a bit deeper right if you're um if you're interested in seeing like the internal mechanics of like what's going on in your query execution.
Um, you know, uh, you can still investigate the query profile itself, right? Um, so actually let me let me see a show of hands. How many people have taken a look at like the query profile like kind of like the tree view before? Oh, actually a surprisingly large number. Awesome. So, um, yeah. So, I think you'll be really pleased to see that we've the team did a complete overhaul on the uh the UI and like you know the UX of the query profile experience.
Um, I'm really proud of what the team has done because I think it's just much more informative. It really tells you what's going on much quicker, right? Um, so, you know, here's just a really simple like TPCH query, you know, that I ran to, um, you know, give me an representative example. Uh, but now, you know, we we kind of rearranged some of the statistics here. Um, made it a little bit more easier to understand. Um, but the real cool part is if you open the query profile, you know, we'll we'll highlight essentially the operators um that are really important.
So, for reference, this is the physical plan. This is essentially what you know um the uh SQL warehouse is like executing in terms of like um you know the physical plan of the query um and we'll highlight like hotspots essentially um but if you're not really a big fan of like scanning through like large graphs you know we we we basically sorted everything for you so you could press this top operators button and then it tells you like hey you might want to take a look at this inner join if you're looking at optimizing the performance or this query specifically you might want to take a look at this specific inner join right um so as you can see it's just a much easier experience.
I think it looks much more beautiful than um our previous query profile experience. Um and it's much more intuitive to use as well. Um so for example, in this case, you know, one one usage case might be like um hey, like I know that this inner join is a common join that I do across a lot of my queries. Um and it it seems to be taking a lot of time in this case. So what I can do is essentially say, hey, I'm going to write a materialized view to kind of like optimize this specific join, right?
Um you know, maybe you don't want to make a materialized view specifically for this query. it might overfit your use case but um you know like set um if you know that this is a common case across multiple queries then it might be a good way to optimize just say I'm just going to like um you know cache this specific join right cool all right that's all I wanted to show back to Ju All right we're almost out of time uh but so thanks and I hope everyone in this room has learned a couple of things from this talk to make cost governance easier for you um we've given you the tools and tips uh so that can monitor and optimize DBCL costs that match your organization's needs.
You can continue scaling with us. Uh here's a quick recap of all the different features that we covered. Um don't think we'll have time to to rehash these, but on the left you have how to understand your bill and on the right um how to optimize them. And on the bottom are previews that are coming. So if any of these are of interest to you, please reach out to us through your account teams and we'll h we're more than happy to uh enroll you in these.
Lastly, uh these are all the other sessions that you may want to uh visit. Uh the first one on DBSQL, what's new, second on specifically uh improving performance uh for faster queries. Uh as well as ones around predictive optimization and cost management as well. Um, so please check these out as well. Awesome. Thank you everyone and uh please don't forget to fill out these uh surveys.

Learn more about the Databricks Data and AI platform.