Enterprise Cost Management for Data Warehousing with Databricks Lakehouse
Summary
- This video presents a refreshed enterprise cost governance playbook for Databricks SQL environments, addressing how agentic workloads add new cost and complexity as AI agents generate SQL, create dashboards, and consume compute without human oversight.
- Two governance pillars are outlined: visibility through cost monitoring by warehouse, user, and tag with drill-down attribution, and control through budget policies, query limits, concurrency controls, and automated storage optimization.
- AI-powered query diagnosis and plan analysis tools are demonstrated alongside Genie integration for natural-language cost questions, with a vision toward fully autonomous performance management on the Databricks Data and AI platform.
Enterprise Cost Management for Data Warehousing with Databricks Lakehouse

this video shows you how to gain visibility into your Databricks Lakehouse spend and ensure cost efficiency. Learn about the latest features for AI-driven insights into Lakehouse expenses so you can easily monitor and control your costs. Find out how to enable cost attribution to internal projects, improve performance and cost with agentic workload optimizations, and take advantage of intelligent compute and storage capabilities that do the tuning for you.
Talk By: Jiabin Hu, Staff Software Engineer, Databricks ; Joo Ho Yeo, Product Manager, Databricks
Chapters
00:00Introduction01:46Context: agentic workloads add new cost complexity03:28Pillar 1: monitoring spend by warehouse, user, and tag05:15Cost trend drill-down and attribution best practices08:43Pillar 2: governance with budget policies and query limits12:14Budget alerts and tagging strategy15:13Query timeout and concurrency controls17:29Storage optimization: managed tables and auto-compaction19:09Serverless SQL: automatic provisioning vs. manual warehouses24:25AI-powered query diagnosis and plan analysis27:30Vision: AI-driven autonomous performance management30:30Query optimization examples and cost savings36:47Genie integration for natural-language cost questions
FAQs
Why are agentic workloads making Databricks cost management more complex?
AI agents generate SQL queries, create dashboards, and consume compute automatically and at scale, meaning cost spikes can originate from non-human sources that are harder to attribute and control. The traditional model of tracking spend by team is insufficient when agents acting on behalf of business users generate a large and growing portion of warehouse activity.
How do budget policies and query limits work in Databricks SQL?
Budget policies allow administrators to set spending thresholds at the warehouse or project level, with alerts triggered when costs approach defined limits. Query timeouts and concurrency controls provide a secondary governance layer by automatically terminating queries that exceed acceptable run times or prevent overload when too many queries run simultaneously.
What is AI-powered query diagnosis on Databricks and what does it surface?
AI-powered query diagnosis analyzes execution plans and identifies patterns causing performance problems or unnecessary cost, such as missing statistics, suboptimal join orders, or inefficient scan patterns. The feature surfaces actionable recommendations so administrators and data engineers can resolve issues without manual query plan analysis.
How can Genie be used for cost monitoring and optimization?
Genie integration allows administrators to ask natural-language questions about Databricks spending, such as which users or warehouses are driving the most cost over a given period. This makes cost visibility accessible to non-technical stakeholders who can then work with data teams on targeted optimizations.
Full transcript
[00:08] Hey everyone, uh happy uh afternoon. Hope you're having a great conference. Uh my name is Juo and I'm a product manager on the DBSQL team. Uh joined here with one of our tech leads, Jabin. >> Hello. >> So over the next 40 minutes, we're going
[00:21] to tackle a topic that is absolutely top of mind for every data leader and admin right now. monitoring and optimizing across your data bricks SQL workloads at enterprise scale. Things have gotten a
[00:33] little bit crazy with agents writing SQL code, generating dashboards, etc. And we want to refresh our playbook to help you in this new era. If thinking about this
[00:45] gives you a headache, you're probably in the right talk, but also please check that you're in the right session. Uh through this talk, we'll walk through a couple practical frameworks, do some live demos, and by the end of this
[00:57] session, you'll have a clear playbook for getting your SQL costs under control while getting the best performance as well. And at the end, we'll also cover how to use Genie to do this easier than
[01:09] ever. Jabin and I will also stick around right over here uh for Q&A after the talk. So, please stay till the end of the session. So before we jump in just here just some
[01:21] standard forward-looking statement. Uh we're going to be sharing some of the features earlier on. Some some of these are in preview and some of these uh we are building uh as we speak. So please remember that these are subject to
[01:34] change. Also uh housekeeping note, please remember to complete your session surveys uh through the data bricks events app. Uh we actually read all these feedback uh and it helps with our
[01:46] job security if you give us great reviews here. So please leave good reviews for us so we can take a look at them later. All right. Uh let's set the context
[01:58] here. Why does enterprise governance matter more today than it ever has? So winding back, think about day one of your data bricks journey. You likely started with a single warehouse, one
[02:11] team, couple users that were running some simple SQL workloads. When the bill went up, it was easy to know uh why it went up and you could just go tap your
[02:23] co-orker and ask what was going up. Today, the P picture is vastly different. You have multiple workspaces, dozens of warehouses, hundreds of users. Now you have even more users that aren't
[02:37] even human. Um, agents are generating incredible value for businesses, enabling non-technical users to get value out of data. But they also have added a lot more admin burden from the
[02:52] sheer number of uh users that have been going up to the volume as well as more unoptimized queries coming in. So as a result, admins are feeling very
[03:04] overwhelmed. Um, so on the left side is how you would have felt if you attended last year's session. Uh, feeling very chill as an admin. But today you're on the right side feeling like things are
[03:16] on fire. Good news is we're here to help you regain complete control and confidence. Um, and we've developed this uh playbook built on three pillars. So the first
[03:28] pillar is monitoring. Knowing exactly where your money is going and being able to attribute every single dollar to a specific use case. The second pillar uh
[03:40] is optimize. So to understand where to start optimizing and how to optimize uh depending on what uh matters most to you. Is it performance or is it is it
[03:52] cost? And then lastly uh the last pillar is genie. uh let us help you with monitoring optimizing for you with Genie which is your personal admin agent.
[04:08] All right, we'll start off with the first section. The core focus here is straightforward. Figuring out exactly where the money is going starting from the account level down to each specific use case.
[04:22] For many people the question the first question you may have is where do I even start looking? Should I start at the warehouse where uh which is where the uh which is what I am build for? Should I
[04:34] start at the account which is the highest level or should I start at individual queries which is where uh compute resources are actually consumed.
[04:47] So our recommendation here is to start at the highest level so you can get a holistic sense of how costs are changing. For this we built uh this UI on the left side which Jabin will uh demo um called cost governance tab which
[05:01] is governance in the governance hub of the account console. So admins can start by viewing the accounts bill for any product um for any time period and then view uh the change uh compared to the
[05:15] previous time period. Uh from there you can drill down uh to few cost trends per specific warehouse or per tag. Um our best practice here is to use a uh unified set of tags for uh for
[05:29] warehouses that belong to the same teams uh so that warehouses that share tags can be grouped and monitored together. And lastly you can also view cost trends at the warehouse level.
[05:41] Okay. So now uh now you know how to find hey like is a specific warehouse cost going up. Um but that's not the end like now what do you do? How can I understand
[05:53] what drove that specific warehouse cost? So while your bill shows a single row for your warehouse, what actually drove
[06:05] your consumption is probably some user like Jabin or a tool like PowerBI or uh specific notebooks or dashboards that are uh running recurringly.
[06:22] And to help you answer this question of attributing uh costs down to individual use cases, we built this attributed usage system table which is currently in private preview. As you can see here, the system table attributes warehouse
[06:34] DBUs directly to individual query executions. And from there you can group by metadata such as user tool or data bricks asset like a notebook or AIBI
[06:46] dashboard um and aggregate cost and uh understand trends on those dimensions. Okay. So now you have some bases covered
[06:58] but there's still other questions that you may have. For example, how do you know what a specific project like a marketing project costed? or if you have multiple DBT models, multiple PowerBI
[07:11] dashboards, how do you know each specific uh dashboard's contribution to your warehouse costs?
[07:23] And that's exactly why we built query tags, which is a feature now in public preview. Query tags allow you to apply multiple custom tags for each SQL execution uh from any interface. So the
[07:35] on the right this is from PowerBI but you can apply it from Tableau you can apply from your custom apps through our uh statement execution API and um on data bricks UI as well. Um so to this you can add uh business contexts like
[07:49] cost center name your own customer's name your custom app name and even better uh from those partner tools like PowerBI and DBT we've worked with them so that they automatically pass their uh
[08:02] their model names as tags. So these tags flow natively into your system tables. Um and as we showed in the attribute usage table uh the query tags show up there as well. So you can
[08:15] uh group costs by those dimensions. All right. So now we can understand uh who actually drove your warehouse costs. Was it Mark a specific AIBI dashboard uh
[08:30] PowerBI a specific PowerBI report or any custom uh projects?
[08:43] Oh second. All right. The last section here um is uh proactive monitorings to ensure you never surprised by your bill again. So we have budgets and alerts uh which
[08:55] alerts are in GA. So budgets are allow you to set warehouse level spend caps with automated notifications um and is our builtin feature for cost governance across the databicks platform. It's also
[09:08] integrated in the cost governance tab that I just showed. And then we also have alerts for custom monitoring. So you can set alerts and combine them with system tables like the attributed usage
[09:22] system table. So you can monitor, for example, hey, when a specific DBT model costs exceeds $100 a month, uh, please notify me.
[09:37] All right. Um, so that's enough talking and now Jabbin will demo these features for us. >> Okay, thank you JO. Let me switch to the demo screen and uh see how we can use these new features to monitor your cost
[09:50] in real action. I'll start at the grand level of monitoring your cost at the account. This is the new cost page in the account console's governance hub. As
[10:03] you can see here, the whole spending of your databicks account is shown on a single page. You can also drill down by product here. And in this session, I'm going to focus on the SQL cost by
[10:15] clicking into spend by product and select SQL warehouses. As you can see, this is a list of all of your warehouses in the account with
[10:29] their spending metrics and also trend over the last 30 days. You can do sorting on arbitrary columns here to really identify the warehouse that you want attention.
[10:42] Okay, great. So, this is warehouse level of cost. What if you want to see sub warehouse level granular cost? This is now made possible with the new
[10:56] attributed usage system table. So in this notebook, I have a sample query on the new system table selecting everything and highlighting a few uh important columns. I'm focusing on a
[11:10] single warehouse and let's see the results. Okay, so this is the query results and every single row here is a warehouse
[11:22] query that you have issued and with the statement ID and with this active usage quantity column that shows the attributed DBU per query data for this
[11:36] particular query. Along with that there are a few important useful metadata. for example, the warehouse ID, client application,
[11:48] or the user who triggered this query. And with this information, you can now drill down your warehouse cost based on the dimension you need. For example, in
[12:01] the second query here, I am grouping the cost by user. So with this it can tell me on this particular warehouse what are the distribution of cost among different
[12:14] users between me and okay so with these building blocks you can build useful visualizations uh like dashboard
[12:26] like this. So in this example, I'm joining the attributed usage table which has the DBU data with the list prices table to
[12:38] convert the DBU into real dollars. And again I'm breaking down that into daily spending on the user basis. As you can see here on this particular
[12:50] day, the whole spending for the uh for the warehouse is $30 and distributed among these two users. I can also slice and dice by other dimensions like cost center between PM
[13:04] and an on the same warehouse. And if these pre-curated dashboard is not fully answer your specific question, you can always ask Genie.
[13:19] So here I have asked Genie a targeted question of how much is my spending outside business hours. So, as you can see, Genie is uh
[13:32] accessing the same data that powers the dashboard, but in a much more targeted and customized fashion in order to generate this nice report that summarizes my spending of about $62
[13:44] plus this nice visualization. Okay, so going back a little bit to the to the second chart that's that breaks down the cost by cost center. You might
[13:57] wonder how is that possible? Okay, this is made possible by tagging the DBSQL queries with a new query tax feature. I will quickly show how.
[14:11] This is the syntax of setting query tax. there are key value pairs applied at that session and effective for all the remaining queries of the session.
[14:23] So let's run this. I should expect both of these queries that h uh to have this cost center and tag as yeah this the second query should
[14:36] also have the same tag. I can also set multiple texts on a single line and change an existing tag to a different value. I'm changing the
[14:49] cost center to PM and adding the project unicorn tag. As you can see now the two new tags get applied. These tags will show not only
[15:01] in UI but also in API and the system tables we have just shown. And uh this is just an example of setting tags in first party database UI
[15:13] but also you can set query tags in your favorite third party business uh BI applications. For example, I have PowerBI here. If I try to connect
[15:28] to data bricks from PowerBI, I have this new box of query tags that I can insert any query tag here. And uh
[15:40] like Dr mentioned, not only we can do this uh customer tags, but also with PowerBI and DBT, they automatically populate some important metadata as query text to us. For example,
[15:55] dbt here sends us the dbt model names with key dbt model name and the real model name as the value. With this you can trace back your third party queries
[16:09] to its origin. Hopefully with this your monitoring journey is much easier. Okay, back to you Juulo. Cool. Awesome. Thank you uh Javin for the demo. Um, so now
[16:22] that we can trace each dollar from the account level down to individual queries and we've learned how to aggregate that um, for the business context that makes sense for you. Let's move on to the more
[16:34] exciting section. So now that you know where your costs are going, how do we start optimizing and where is the um, and how can I get best cost and performance based on my business needs?
[16:51] Okay, so same question like where do we even begin here? So when you look at optimization you have to choose a direction. Are you optimizing to cut costs and stay under budget or are you optimizing to deliver
[17:03] better fast performance uh to your business users? The good news here is that you don't have to choose just yet. Uh before you choose between cost and performance, you
[17:15] should always start with the table stakes features. These are the win-win data bricks uh features that we provide out of the box to optimize for both cost and performance.
[17:29] Our key feature here is manage tables. Manage tables are an absolute must. Through predictive optimization and automatic liquid clustering, the platform automatically and intelligently
[17:41] runs commands like optimize, vacuum, analyze um based on your historical workloads. So you can get up to 20 times faster in um query performance. There's a lot of talks the the uh this week to
[17:54] go further in depth on the technology that's backing this, but in short, data bricks has invested a lot in AI to improve performance and costs. We use AI to maintain clustering keys, statistics
[18:06] that help skip as much data reading as possible uh during each of the query execution that leads to the increase in performance. And the cost comes down because if you can run your queries
[18:18] faster, you don't have to have your uh warehouses up uh for for a long time. And the best part of this is that you get all of this with zero maintenance. You turn on manage tables and then we
[18:30] take it on from there. The second feature is serverless SQL. Serverless SQL is also a no-brainer. You get near instant startup times, rapid autoscaling to handle spikes, and
[18:43] incredibly aggressive auto stop. So you don't have to pay when there's no more queries coming in. Serverless handles intelligent workload management for you as well. Uh which means that you can get the highest utilization possible with no
[18:57] maintenance. as well. Customers have enjoyed up uh beyond 50% cost reduction, even higher performance um improvements and the majority of our customers are on
[19:09] serverless SQL today and growing. Okay, so now that we've covered the table stakes, the next set of actions depends on whether you want to optimize for cost or performance.
[19:23] If you are optimizing for cost, warehouse configurations is your first line of defense and where you should start. Set your hourly spend limit. This is just multiplying the DBU per hour
[19:36] rate that you select with your t-shirt size by the max cluster that you also can configure um in this um in this setup.
[19:48] The next uh thing to consider is setting a auto stop that matches your uh your needs. And we often see customers not setting aggressive enough autostops. Um so yeah, take a take a look and please
[20:01] set up uh auto stops uh for your warehouse. um if you want to optimize for cost. Lastly, uh we've introduced warehouse level statement timeouts. Uh which means you can set uh shorter
[20:14] timeout for your BI warehouses and longer timeout for your ETL warehouses. The objective being to prevent runaway cost, but also um without disrupting your production uh workflows.
[20:34] Okay, so now you've uh set up the cost your warehouses to be cost optimized. Even then, if users aren't using the warehouses correctly, you can run into surprise costs. And this is shown in the diagram on the left. If users have
[20:46] access to different warehouses, they may be accidentally choosing to wake up extra large and like 4 XL, 5XL warehouses that are meant for specific ETL warehouses when all they're trying
[20:59] to do is get the first rows of a table. To address this issue, we built a default warehouse setting uh which admins can configure a specific designated warehouse that will be
[21:11] automatically chosen uh when users are writing exploratory queries. And as a result on the right side, you can see that the designated warehouse is the only one that it's uh woken up. And of
[21:25] course, that leads to cost savings on the other hand. So we we talked about um optimizing for cost. If you want to optimize for performance, uh good news
[21:39] is we also have powerful features here. And here's the basic framework based on what you what your requirements are. If you want to get the best per query performance and you're running heavy
[21:52] workloads, you should increase your warehouse t-shirt size. Um, and kind of an extension to that, we've uh recently uh introduced our largest t-shirt size
[22:04] 5XL, which as the name implies is twice the capacity of 4XL, uh, which has helped a lot of customers that are doing terabyte scale ETL with strict time
[22:16] requirements. If you want high concurrency, the knob that you want to increase is the max cluster count. So that as we covered um we can handle the autoscaling during
[22:29] bursts of query traffic. Lastly, if you want even higher, even lower um latency like subsecond or sub 100 millisecond latency with crazy
[22:42] concurrency requirements, try out Lakehouse Realtime, which is in beta um and announced during the keynotes today. Just on that note, um
[22:54] hopefully you heard about this from uh in the keynotes in Reynolds presentation. he can do it better than me. So please rewatch it if you have not uh watched it yet. But in short um this
[23:06] is a massive shift in our industry. Uh traditionally to achieve subsecond latency uh enterprises had to copy data out of their governed lakehouse into specialized external databases. Now you
[23:19] can get subund millisecond latency speeds directly on your secure open lakehouse with no additional admin overhead. uh we think this is going to be a gamecher for many customers and the
[23:31] industry. So please let us know if you are interested in learning more about this. Okay, this brings us to our last section on Genie. So Genie is many things, but
[23:45] it's also your admin agent that will help you with all of the monitoring and optimization that we just covered. Think of Genie as your shield um to the admin overhead agents have created for
[23:58] you. So our long-term vision is uh for Genie to help you everywhere and uh anywhere. Um and we'll we'll talk about that, but we'll start from the lowest level. We'll
[24:11] start with queries um which are really the the atoms of your SQL deployment. Uh, previously understanding query metric, previously understanding like getting to why is my query running slow
[24:25] um took a lot of knowledge basically a PhD in data bricks to parse the the the visual on on our left side and this is kind of what has been um in the industry. You need to understand
[24:38] operators um the the tree graph that connects those operators and specific metrics on them. We've simplified all of this um to what is on the right side uh
[24:50] which is a a feature um that we that is in beta called query performance insights. Genie translates all of this into plain English summary uh diagnosing why a query was slow. Um there's this is
[25:03] just one example. There's about 20 types of insights that Genie looks for and surfaces and we're continuing to build that out when you need to go deep. Genie also helps you um in that uh query profile
[25:18] surface. Um as you can see here uh it highlights specifically which operator is the the bottleneck causing the slowdown and the specific metrics that are affected. It's kind of small here
[25:31] but um Jabin will cover some of this in the demo. But Genie just doesn't stop at just the diagnosis. It also helps you fix the
[25:43] issue. Uh with Genieode um integration and beta uh you can get a one-click query rewrite. Genie code recommends the correlated uh the corrected SQL text. Um
[25:56] also asks you for input if you need to provide them for example if you need to add like a join key or filter. Um it will ask you for that um and runs the new SQL directly for you. Um after the
[26:09] SQL is run um you get a new diagnosis from Genie on your updated query um and you can decide to continue optimizing further or stop there.
[26:21] So this entire feature um is in is in beta now um so please give it a shot um and let us know if you have any feedback here.
[26:35] So now we talked about like this the lowest level of like query executions and how Genie helps you there. Um and that's that's not it. Uh Genie also um helps you provide um the high ROI recommendations across your account or
[26:48] across your workspace that can improve not just one individual query uh but performance across the board. Um so the product on the right side uh is called performance tab which is also in
[27:01] governance hub. Uh Genie continuously analyzes your deployment u to identify unoptimized tables uh warehouses that could benefit from right sizing and query patterns um that uh that continue
[27:16] to show up uh uh providing um kind of uh hurting your performance.
[27:30] And the long-term vision um is that uh Genie will be able to help you everywhere and and um for anything from anywhere. Uh so as I mentioned um th this is the long-term vision uh that
[27:43] data uh Genie can help you with optimizing not just individual queries but notebooks, tables, dashboards and everything. And so this um on the
[27:55] right side is actually something that uh we are building out right now is like using Genie to optimize an entire notebook and not specific cells or or executions and finding long poles of
[28:07] where performance bottlenecks are.
[28:19] Cool. We're back to Jabbin for a demo. Okay. Thank you Drew. So let's see how we can use these tools to manage and optimize the performance. In first part of this demo I will show how we can use
[28:34] Genie to automatically optimize the query by leveraging these new performance insights. Here I have prepared a simple query that pulls the data from two tables with a
[28:46] simple filter. I have disabled the query cache because I really wanted to execute on SQL warehouse rather than returning previous results. So let's run it.
[28:59] Normally this would take a few seconds to come back. Let's see. Okay, five seconds this time. And uh as you can see here, Genie is busy finding opportunities to optimize.
[29:12] After it's done, this optimize button pops up. Let's click that. to kick off an optimized session on the side. While Genie is busy at work, let's look at the
[29:24] query execution status. As you can see, it has found one performance insight that says we are doing a drawing without a drawing key. And if we zoom into the query profile,
[29:38] it actually highlights the exact part of the query plan that can be optimized. again saying the same thing that drawing without drawing key results in an exploding drawing.
[29:50] So let's go back to Genie. Genie has finished and it is leveraging this performance insight combined with the query text and the under uh the underlying table layout to recommend us
[30:04] adding this drawing key on the date column. Okay, let's accept that and run again.
[30:17] As you can see, it comes back much faster in about 1 second. You might say, okay, this is only five to one second. It may not be a big deal, but imagine this is powering one of your most popular dashboard. And uh this kind of
[30:30] optimization can really save your cost and make it faster. Okay, so exploring join is just one of the examples of many performance insights that we have a available.
[30:44] We have insights about query optimization also optimizing your data layout as well as your compute resources and this list is growing. Okay, so this part is as if you are the
[30:59] data scientist optimizing your query. But what if you are admin want to optimize your account workload at scale? So I will switch gears to back to the
[31:12] account console for the performance page in the governance hub in account console. This is a central place to manage and uh monitor the performance of your SQL
[31:25] warehousing workload at scale. As you can see at the top there are few key performance indicators of your workload. P90 latency change once per months query
[31:39] volume change and the query failure rate uh failure rate change and your spending change. On the right hand side we surface some insights or recommendations discovered
[31:52] by Genie. For example, we have discovered that there are few SQL warehouses that have a high queueing rate and suggest you to change the max cluster count. In addition, we look at
[32:07] the popular tables that your SQL warehouse queries and find out if their layout can be optimized by turning on either auto liquid clustering or
[32:19] predictive optimization. Okay, going back to the main part of this UI, you can also drill down your performance metric based uh by wareh by workspace or by warehouse. For example,
[32:34] let's go into the warehouse view. This is the all the warehouses in this account with their key performance indicators. P90 latency and the change per volume
[32:47] and the change plus whole set of other important performance metrics. You can do sorting on these columns and uh hopefully this will let you
[32:59] quickly identify the problematic warehouses that worth your attention. You can also do filtering here. Let's go to my favorite warehouse and see its details.
[33:13] On the right hand side, you can see there are few tabs starting from the insights tab. Again, it surfaces the areas that can be improved. For example, shortening the queue by increasing the
[33:25] max cluster count or improve the table layout. On the computer tab, you can get an idea of the workload breakdown on this warehouse by application or by users.
[33:41] And uh on this table tab, it tells you what are the tables being queried by this warehouse and whether they are managed or external. if they're managed, whether they are taking full
[33:55] advantage of these automatic optimization features. Finally, on the queries tab, it shows you a breakdown of the type of workload on this warehouse, whether they are
[34:08] doing read, doing write or doing other streaming workloads. Okay. So if this pre-curated UI is not fully answering your specific question
[34:23] always ask Genie here in this example I have previously asked hey Genie in my account what are the SQL warehouses that have over 1,000 queries
[34:37] and latency uh over 10 seconds is accessing the same data that powers the UI but issuing customized queries to my target question and give me this nice
[34:52] summary of such 18 warehouses that meets my standard as well as the detailed list of these 18 warehouses for me to act upon. Okay, so hopefully with this the
[35:05] performance management and optimization story is much easier. Thank you and back to you.
[35:17] All right. Uh thank you Javin. So as we conclude um here are the three key takeaways uh to bring back to your teams. So first monitor exactly trace trace where each warehouse dollar is
[35:29] going starting from the account down to specific queries. use our attributed usage system table and query tags so you can aggregate and granularly moni granularly monitor at any dimension um
[35:43] that you would like. Then use our budgets and alerts features to do this monitoring proactively. For optimizations, start with the table
[35:55] stakes features. Use manage tables and serverless SQL so you can get both best performance and cost. Then if you want to optimize for cost, use warehouse settings as well as default warehouse.
[36:08] And then if you want to optimize for performance for higher per query perf, use larger t-shirt size for higher concurrency uh increase the max cluster count. And
[36:21] if you want subund millisecond high concurrency workloads, try out our new real-time compute. Lastly, Genie is your personal admin agent. Query performance insights helps
[36:35] you understand in plain English why a query was slow. And you can take action by clicking on the optimize button which will be in in the authoring services
[36:47] like notebook SQ editor as well as in the query profile. Lastly, performance governance hub, which Javin just showed us, helps you identify high ROI performance improvements across your account and
[37:01] workspace. Just the last thing to call out here is we're actively building out this experience. So, if there are, you know, missing features in terms of how you would like Genie to better support your admin um admin workflows, uh please let
[37:16] us know um via your account teams or in the survey today. So hopefully after this talk you're feeling a little bit better and and chilling on like on the right side uh
[37:29] with the help of Genie.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.