What’s New for FinOps on Databricks

Summary

This video shows how to operationalize FinOps on Databricks across three phases—Inform, Optimize, and Control—covering how to enrich workloads with tags, explore cost and performance through dashboards and Genie, and implement proactive alerting across compute types.
New capabilities demonstrated include the Governance Hub for drilling from high-level spend to specific cost drivers, tagging rate alerting for untagged high-spend compute, and compute optimization with cost-efficient cluster recommendations.
Budget policies and per-user spend enforcement provide the control layer, and the video covers serverless cost controls including throttling and access management to give FinOps teams full accountability over Databricks spending.

What’s New for FinOps on Databricks

As your Databricks adoption scales, FinOps teams need full visibility that connects performance, usage, and spend, and tools that turn insight into action. In this video, we’ll show how to operationalize FinOps on Databricks . You’ll see how to enrich workloads with tags, explore cost and performance through dashboards and Genie, and implement proactive alerting across compute types. We’ll also unveil new FinOps-focused features & tools that your team will want to be aware of.
Talk By: Sadhana Bala, Sr. Product Manager, Databricks ; Youssef Mrini, Sr DevRel, Databricks ;

Chapters

FAQs

What are the three phases of FinOps on Databricks?

The three phases are Inform, Optimize, and Control. Inform covers gaining visibility into how billing flows from granular usage through system tables to dashboards. Optimize focuses on identifying inefficient workloads and applying cost-efficient compute recommendations. Control involves setting budget policies, per-user spend limits, and serverless throttling to enforce financial accountability.

How does the Governance Hub help with Databricks cost management?

The Governance Hub provides a unified dashboard that lets FinOps teams drill down from a high-level view of total spend into the specific workloads, users, or compute types driving costs. This video demonstrates how teams can use it to identify failing and inefficient workloads and take action without needing to write custom SQL against system tables.

What is tagging rate alerting and why is it important for FinOps?

Tagging rate alerting notifies FinOps teams when high-spend compute resources—such as clusters or Lakebase instances—lack the required cost attribution tags. Without tags, it becomes difficult to assign costs to specific teams, projects, or business units, undermining the explainability and accountability that most FinOps practitioners identify as their primary challenge.

How do budget policies work on the Databricks Data and AI platform?

Budget policies allow administrators to set spending limits at the user or team level, enforcing financial guardrails before costs become a problem rather than discovering overruns in audit reports. This video includes a live demo of per-user spend enforcement and also covers serverless cost controls such as access management and throttling for further granularity.

Full transcript

[00:07] Hi everyone, thanks for coming for a session like Yousef said two hours before the big party and the chain smakers go on. So we'll try to be a good um intro to them. My name is Saba. I'm a product manager at data bricks. Um my sole focus is actually on phops and cost
[00:21] management. So thinking about this experience from the data through all the uh controls and UIs that we serve. Yousef. Uh yeah, my name is Yousef Mar as you can see and I'm a senior deell and that's it.
[00:33] >> Okay. Um so I guess and also there's a safe harbor announcement slide we're missing but just keep in mind that everything we're discussing today. So a lot of this is forward-looking and we're excited to
[00:45] ship it but there's no specific commit commitment on timelines that we're announcing here. So you have started some of the poll but just to kind of get the audience warmed up and also to give us some data sampling. How many of you all raise your hands if you consider
[00:58] yourself a reporter on data bricks? So your job is tracking and spending. Okay, maybe actually about a half. Um how many of you are sort of the optimizer on data bricks? You're about a half as well. Um
[01:12] how many of you actually own the budgets? So it rolls it rolls up to your head if something is spiking maybe a third. Um and then I think we asked this question but how many of you are administrators? So you're actually setting control, setting policies. Okay.
[01:26] And this question is a little bit more interesting. What keeps you up at night? What brings you to the room today? Um how many of you um is the issue predictability on data bricks in terms of your cost?
[01:41] About a third, maybe a little less. How many of you it's efficiency? Like are you actually getting the most value for your cost? Explanability. A lot more hands went up on that one. And then accountability, making sure
[01:54] that your end developers and teams are awesome. Okay, so my big takeaways there is um explanability is a big one and we're hoping that some of the tools we show you today will actually address that headon.
[02:06] Um what is PHOPS? I won't spend too long on this definition. It's like u you know the Phops Foundation. The conference was actually last week. I don't know if any of you were at the conference last week.
[02:18] um maybe one or two, but it's operational framework and also a cultural practice, operational discipline about maximizing the value of your lakehouse. Um also ensuring that your decision-m and your anomalies are
[02:31] being caught in a timely fashion and you're also creating accountability between admins and yourself and the individual job owners, cluster owners, model developers.
[02:45] Uh you know what's new in PHOPS? There's also a whole new series of challenges that kind of on the product team we're seeing our customers face with on a daily basis. Number one goes without saying AI spend is the number one PHOPS priority right now. I think maybe two
[02:57] years ago according to the Phops Foundation a third of organizations were managing AI spend. That number has tripled in over two years and your token costs are difficult to reason about. They also rack up quickly. Um so that's
[03:10] really the the number one priority we hear unanimously. Second off is the statistics actually shocking to hear, but organizations that are actually matching u sorry managing a hund00 million in spend average only 10 PHOps
[03:23] practitioners that manage and optimize the whole budget. Um so we're hearing a real need to be able to delegate and federate cost awareness, cost visibility away from the sort of magical account
[03:35] admin that a lot of people don't have permissions to down to the end users and operators on data bricks. um three and four are similar. You're you saw this morning and yesterday how Genie is becoming a primary authoring surface for
[03:48] creating pipelines, dashboards, queries, you name it. So you actually need your governance to scale with pace of how quickly workloads are being generated on the platform. And you're also opening up the uh bars to doors to data bricks to a
[04:01] lot more business users, inexperienced users um where optimization infrastructure may not be their expertise. So our product roadmap and what we've developed this last year is really tailored to address these four challenges head-on.
[04:15] Um we'll talk about three phases and we'll be peppering in this session with a lot of live demos. So may the demo gods be with us. Uh we'll start with making sure you have a good grasp on detailed and granular visibility in your
[04:28] costs on data bricks. Um obviously the bug doesn't stop there. Tools and techniques you can use to optimize your spend. um and a lot of new features on the control bucket. How can you actually ensure predictability set an upper cap
[04:41] on your spend? So we'll start with inform. Just want to start with a very sort of foundational 101. How does cost reporting? How does billing work on data bricks? So um part
[04:53] of my team actually owns the sort of massing billing platform where we're processing billions of dollars a year on data bricks. It starts very granular. So every product line on the platform whether it's a job SQL query or an
[05:05] endpoint we are a we're um aggregating usage on actually an second minute hourly basis. Data bricks actually has fairly low latency when it comes to usage tracking um especially compared to
[05:17] somewhere our peers on the cloud on the cloud end where you're sort of seeing your AWS curve reports refresh every 24 hours. We actually try to bring down latency to at least on average an hourly basis across your product line. So then
[05:30] that data is aggregated. We enrich it with a bunch of metadata and it shows up in two places. So the first place it shows up and this depends on how you purchase data bricks whether it's through marketplace or direct with your
[05:42] cloud is for some of you you actually will see the data bricks costs directly in your cloud's billing feed. So your AWS cur reports Azure cost management exports GCP billing exports. But you're probably noticing there that the costs
[05:55] there are fairly highly aggregated. It's usually at the skew level maybe a bit more granular on Azure we have a lot more granularity on costs but really the granular cost and usage tracking you'll directly access in data bricks itself.
[06:08] So that's through um sort of our internal data store that we expose many of you will be familiar with system tables but also directly we serve this data in UIs and controls in the data bricks platforms. So two places you're
[06:21] seeing costs, but to get really low latency and granular costs, uh we want you to be familiar with all of the offerings we have on data bricks so you can do it native to our platform. Cost tracking is not new on data bricks.
[06:33] The system tables have been around for sort of two or three years. We've released dashboards. We actually spoke about the version two of the usage dashboard last year. But we cons consistently hear a couple of um
[06:45] customer problems from you all, right? One is the usability of system tables. It's super granular, low latency, but that schema is getting broad and it's not the most usable when you're actually trying to get your answers quickly.
[06:57] Number two is discoverability. Um, I won't spend too much time there, but accessibility is a big one as well. So, the dashboards and the system tables out of the box are generally accessible to account admins who can then set up their
[07:10] own sort of um permission sharing model to filter data or share the data with their end users or workspace admins. And a lot of the cost reporting was also at the account admin. So you'll see in our demo shortly, we're bringing this down
[07:22] to the workspace level and also introducing permissions in the future so your end users can see their costs as well if that's allowable in your organization. And finally, it's intelligence. Um, you know, it's it's
[07:34] interesting as we're building POPS tooling uh in the current age. We're still a couple steps behind some of our peers in the cloud. they've been around for much longer, but we're trying to take an agentic and AI first experience to building PHOPS. Um, so really giving
[07:47] you that intelligence in your pocket as soon as you wake up in the morning. So reacting to that, what I'm about to demo is the new governance hub that we're excited to announce a couple sessions have touched on this already.
[07:59] I'll be focusing on the cost governance hub which is now going to be your single intelligent command center for you to get a single pane view of all of your data bricks costs across workspaces teams and products um all in a snappy
[08:13] and uh polished manner. Yes. So yes cost reporting is going to become a firstass administrative experience that's native to the product. uh you'll see aentic integrations there
[08:25] with Genie and uh out of the box cost recommendations. I I'll give you a a demo in a second. One thing I want you to note in the demo is where we've really optimized the experience on the governance hub is to get you from the
[08:38] high level to the specific cost drivers in just a couple of clicks. So in the governance hub, you'll be able to explore how much are you spending this quarter? Uh okay, it's the ML platform workspace. Which team? Seems like it's
[08:50] the model training team. What product? In this case, it's it's SQL. Let's get a step deeper. What warehouse in SQL? Okay, we've identified it's a specific ad hoc warehouse that's maybe not
[09:02] actually really being used in an ad hoc manner if it's costing that much. Um, and actually down to the query and user level. So, just keep this in mind. This overall user journey of going from high level to low level is what we want you
[09:15] to experience native in the governance hub. So without further ado, um let's introduce you all to the to the governance hub. And can everyone see this? Okay. Do I
[09:27] need to zoom in a bit? Especially in the back row folks seeing it. Okay. Okay. Awesome. So this is it will be in the account console, but that doesn't mean it's only accessible to account admins. >>
[09:40] account admins will be able to view costs at the at the account level of course but if you're a specific workspace admin when you come in here um you will see for example if I'm an admin of the E2 workspace which is a staging
[09:52] workspace internally you'll be able to come in here and view the costs of the workspace you're an admin of so accessible to a lot more users and we're also working on building a PinOps or billing admin role so those users can
[10:04] get access to this experience as well. Um, so let's set the stage. Say I'm a platform admin managing a team at the account level. It's a Monday morning. I'm waking up and I'm here to do my weekly audit review of my cost. Now, I'd
[10:17] rather be watching the FIFA World Cup game happening. So, I'm hoping this will be seamless and easy. And let's see if the data bricks product team did a good job building this out. Um, at high level, it looks I can see my spend last
[10:29] 30 days. There is singledigit growth in cost. Nothing I'm too worried about. Um, no major spikes I'm seeing. What does catch my eye is that about a quarter of my costs are actually um tagged and and
[10:41] that's going to be a problem when we actually start charge back charging back and um show showing back our cost to our end users. But I'm going to drill a bit deeper here. So I actually want to get a sense of like what are the some of the
[10:53] biggest cost drivers um in my account. So I'm going to go ahead and group by workspace here. And I can see uh you know I I I already knew this before, but our biggest cost driver culprit is the
[11:06] dog food account um in data bricks. So I'm going to take that a level deeper and try to drill into where they're spending. So in dog food, we actually have a tagging policy where we have teams um tag all their costs with a
[11:20] budget ID. So I'm going to let that load. Now I can actually see um again each of these values are individual development teams that are by policy tagging their workloads. Um I see there's a a runtime development team
[11:34] that is contributing a far fair share of cost. I'm going to drill down a step further and now I actually want to um see what products they're spending spending on. So, I'm going to group by
[11:47] product. And I see, okay, this is uh it makes sense. A lot of our costs are from automated scheduled pipelines and jobs. Jobs are the biggest driver cost um for this team. I'm going to take this a step
[11:59] further and actually try to get down to the specific job level. So, here I can see there are three jobs that are about contributing $5,000 each um the last 30 days. And as an admin, my job usually
[12:14] stops around here where my next step is going to be to identify the specific job owner. Um, so service principle in this case and take this a step further to understand what's driving the cost. If I click into the job here, this actually
[12:27] opens up the job in my workspace. And interestingly enough, the job here, you sort of see something interesting here. It's uh a lot of failed runs in the last couple of days. So, this is going to be a candidate for me to take home and add
[12:39] to my weekly audit log, weekly cost review log with my team. Um, I try to ask Genie a little bit on the side why this job is expensive. Looks like it's running on for an um sort of power
[12:51] powerful instances, really long task durations, but it's obvious to me that this is a failure that I would want to investigate further. So, there you have it. Um, maybe a little bit slower because I was demoing it. In less than a minute, you're able to get from the high
[13:04] level down to specific culprits. um workload wise that you want to investigate further. What's interesting is our granularity of cost has gotten really good this past year. So say I actually want to take a
[13:18] look at some of the other product lines. Um I'm interested in diving deep into my SQL costs. So I click in here, I filter for SQL and I see that the biggest cost driver here is a shared SQL warehouse.
[13:30] We actually have a lot of simultaneous dashboards and jobs hitting this warehouse. And I want to quickly ask Genie um to understand and and tell me what the drivers of those costs are. So then I click that I hop over into um
[13:44] Genie one experience and despite my typos um I ask Genie what is actually driving up cost on this warehouse and what's supporting this on the back end is actually a new private preview we
[13:56] just launched of a system table that now attributes your warehouse costs down to the query statement level. Um, and that's a private preview. If you're interested, uh, reach out to your account teams and you can have that enabled. I'm quickly sort of seeing that
[14:08] in the past seven days, those warehouses being driven by five users that I think are going to get a call from me right after this demo. And um, you know, their specific applications, their data bricks
[14:20] jobs. Looks like there's a lot of automation that's actually invoking this dashboard in the run of it. So those are just a couple ways I'm making use of the of the governance hub. you know, filter by tag, get down to the granular
[14:33] workload level. One other area that I'd like to bring attention to is we are shipping a series of out of the box cost recommendations to really ensure that some of your best practices optimization
[14:45] guidance is not something you have to manage and configure but comes um directly sort of into your inbox or into your experience. So in this case, one of the recommendations, as you can see, this is a problematic one given our
[14:57] tagging rate is flagging high usage, high spend compute resources that are untagged. Uh I go in here, I review the resources, and um shockingly there's a
[15:09] fairly expensive cluster that today is not being attributed to any team. So this is falling through the cracks in my chargeback model. Um, if I actually click into this cluster,
[15:22] this is indeed untagged. And yeah, you can sort of see here there is it is missing a tag. I'll probably reach out to Danny here um to have him add a tag. You'll see the recommendations get a lot more agentic automated um and richer in
[15:35] quantity um over the year as well. So, let me hop back to my demo. the the cost experience the cost governance hub that I just showed you. Uh we also have experiences for data governance. If
[15:47] you're a data steward and care about data um so data quality and um data hygiene, AI governance as well. If you're looking at your Unity AI gateway costs and you understand which models
[15:59] and which providers you're spending tokens on and and we just launched a performance monitoring experience where you can actually get a sense of which workloads are driving up latency um and developer experience in your account.
[16:22] uh the Genie one we're we're still integrating Genie1 to be part of the um account console experience like Genie code is in the workspace level but yeah right here I'm clicking into the ask Genie flow uh it's popping me in with some context into a genie one
[16:47] on a per user basis. We haven't added that view in yet. Um
[17:01] >> workspace admins will already be enabled for this governance view, but I can I can take more questions at the end of this session as well. Okay, back to our presentation. I know Yousef has some great stuff to share as well. So I want to make sure I don't take too much time. Um you know one one thing that was
[17:14] really critical u in that experience is ensuring you have a great tagging framework. So our you know one of the bars for us is to make sure every product line has some interface that you can tag whether it's API or UI. So
[17:28] you'll see that you can actually tag a lakebased project you can tag an app. Um we support the ability for you to apply tags by policy. So these are serverless usage policies for your serverless cost and then your um uh compute policies for
[17:42] clusters and and classic compute. And what's what's coming soon but we're working on is it's not just a matter if the object is tagged or not but you actually applying the right tag that um fits your chargeback model. So we're
[17:54] looking at ways that we can actually bring govern tags into the uh cost tracking experience. This is how you can uh use govern tags on UC objects. how can they actually use govern tags to ensure that there's no misspelling of cost center when a user is trying to
[18:06] attribute cost there. Um we spent a lot of time last year our PHOPS talk on the usage dashboard version 2.0 that's not going away. We're actually still working on building
[18:18] better BI tooling. Um so if your team is looking to more deeply customize maybe add their own views to these dashboards. The usage dashboard continues to be accessible in the account console and uh
[18:30] we also have some reference and we're building skills natively for you to create your own cost genie agents that will help you optimize and understand your spend as well. So the QR code there goes to a GitHub repo that can get you
[18:42] started with a example of a Genie agent. Last thing I'll um end on here is uh you know the number one of the highest priority things we hear from customers is great data bricks costs is it's great
[18:55] to track but how do I actually track my Azure AWS GCP costs alongside uh so we actually have launched an open- source reference implementation that several of our customers are using where you can actually automatically pull in AWS cur
[19:09] Azure cost management export files into data bricks join that with your system table and actually get total cost of ownership of your cluster. So for a cluster, not just the DBU usage, but the
[19:21] underlying VM cost, networking costs as well. Um so the QR code is up there for you to um see see how some of our customers are doing it today. So with that, um over to you, Yousef, how do we
[19:33] go beyond just visibility? >> Awesome. Thank you, Sadena. Well, she raised the bar too hard for me, so please don't hold me accountable.
[19:46] I think this is pretty much important to like I got many many qu many customers asking how am I supposed to optimize my workloads. Well, I mean you have two options. You can either let the system automate everything for you and this will make sure that anything you're
[19:59] doing is working on the right way to minimize to minimize the compute and you have many options. First of all, something we should recommend you and it's normally it's enabled. It's called
[20:11] predictive optimization. So basically, you no longer need to do anything related to the data management. And by data management, I'm talking about the vacuuming. I'm pretty sure I have a lot of engineers who forgot to run vacuum by
[20:24] like dropping all the old files, which means you're saving on the storage cost. the optimization by compacting small files and having fast workloads which means you're going to reduce your compute cost the auto liquid like you
[20:37] can switch easily uh columns and make sure you avoid rewriting the table each time you need to switch from like part like partitioning by country and I don't know adding a new column by city for example you have also the analyze so
[20:50] this is a little bit technical but let's see it's like a highway which means going to also improve the uh the efficiency of your queries thus reduce the cost and of course we have bunch of interesting things like such as query
[21:03] performance insight which is mainly whenever you run a workflow you will have some suggestions saying hey maybe you should filter on this specific column maybe you should uh switch to I don't know shuffle uh uh broadcast join
[21:16] for example so those are some recommendation can can be very very interesting and of course you have also serverless and this is something very interesting because pretty sure like whenever you provision the compute
[21:28] you're almost over always overprovisioning. While with serverless, you just create like choose the option serverless and if you have a spike of workloads, it will go up or it will go down based on Jesus. And this will make
[21:40] sure that you're always running the fastest and the most performant and let's say cost efficient computes. And on the other side, if you want to let's say deep dive a bit, well, we have a new tables for you. The first one, I think
[21:53] so many people were asking for it. It's called the attributed usage. Sadana showed you previously how you can track the cost per SQL warehouse. That's great. But what if I you could understand like how much each query did
[22:06] did cost. This is even deeper which means you can go and for example select the most expensive queries and try to optimize them using the query insights. This is just amazing like we're really making things easier for you. We have
[22:20] also the job run timeline. This is where you can see the duration of the jobs and again you no long you don't need to also when it comes to optimizing the workloads think about the times oh Slack want to install something
[22:33] slack >> think about live demo effect that's you see this is real this is true uh like you can see how long the workflows are running and you no longer need to go and optimize every single job just think
[22:46] about the the word the the one that needs your attention and that's how you can you'll be able to find And then the last one, well, I'm pretty sure all heard about token maxing. This is a way to find out who used the most
[22:58] most tokens and understand whether it brings some error to your company or not. And this is a continuous let's say pipeline, which means you should understand how things are working and also let the system optimize things for
[23:11] you. And as you can see, Sadhana's team has been doing an amazing job. They released so many new system tables, so many updates like in July, September. I mean, I can't keep track of all those,
[23:23] but you will find them. Which means you can now either build your own tables if you want your your own dashboards to track anything on the platform or you can also use the pre-built our out
[23:35] ofthe-box dashboards. So, those are the two dashboards that I do recommend. Well, mainly LakeFlow D- jobs dashboard. And basically this is the this is an amazing dashboard that will help you track like the cost per workflow and the
[23:50] most for example find find out the most expensive jobs the most expensive runs the jobs that are failing the most so you can go and optimize or fix them the like the clusters that are overprovisioned so you can maybe either
[24:02] switch to serverless or uh like reduce the size. So you have bunch of interesting analytic analysis can be done there and then you have the DBSQL warehouse advisor and this is really a
[24:15] little bit technical but it shows how you can improve your SQL warehouse by understanding like the like some technical stuff like spill disk spill memory to make sure that you're choosing really the right size and the right
[24:27] number of workers when the right number of uh warehouses when it comes to to running your workloads. And of course we talked about the optimization but we need to make sure
[24:39] that we are also able to control the cost. And this is something important. Our our CEO mentioned it this morning and Keith like all these speakers were talking only about control. And this is something very important when it comes
[24:51] to uh using data bricks and I'm pretty sure like many many many people ask like questions okay I want to make sure that above certain threshold I want to cut my
[25:03] uh like stop the workloads well we've been running like many we've been talking with many customers run some surveys and this things that in the platform things work differently for example if we take the example of the
[25:16] production jobs do I really want to stop the production because it goes like it went above the threshold because I don't know you didn't think about this spike that will happen. So this should be a little bit different. We need to make
[25:28] sure that your works production workloads keeps running but then we can of course send you an alert or or define some specific SLA. Same thing will go for example for the AI model serving you know that with the with this with with
[25:41] tokens things can escalate quickly which means you need the the different behavior. need to make sure that you're able to stop the usage beyond specific threshold else your cost will go higher than ever for serverless notebooks like
[25:55] those are used by many many developers. You don't you don't want to you're not thinking about like giving or attributing like a usage to a developer but you need to make sure that your the entire team can share a specific size of
[26:08] workloads that they are not go above I don't know and a mediums medium cluster or or or a large depending of course on how they are doing things and of course we have SQL warehouse well I mean not all the SQL not all the queries deserve
[26:21] a large large SQL warehouse some of them deserve just a small one so those are really different things so that's why they cannot be addressed in the same way and this is very very important and that's why we have three ways to control
[26:35] the cost. So the first one when we're going to deep dive on it, we're going to show you demos. You have first of all like for the serverless notebooks this is how you can enable them and grant permissions to to developers to use them
[26:48] and this of course applies to notebooks, jobs and spark declarity pipeline. We have also let's say the rate limiting and this is how you can give let's say a t-shirt size to a job. It cannot go higher for example than M or L or S
[27:02] depending on the depending on the on the workloads. Same thing applies to serverless notebooks. And then we have budget based control and this is mainly related to AI gateway which means after
[27:14] a certain threshold you will stop or either send an alerts or stop the usage and this applies to Genie and uh the u the AI gateway. And we're going to start our first demo with how you can deploy
[27:28] your first Unity AI gateway to make sure that you can control the cost of Genie, but also your LLMs. So as you can see over here, I have my I
[27:41] have my uh my GPT 5.5 that I have. So what I will do, I'll come here and I'll specify a tag. So for example, I'm going to serve this model to my developers. For example, let's suppose it's codeex.
[27:54] I'm serving Codex to the developers. So obviously uh LLM dev and the team going to be the engineering.
[28:10] So I'm going to save of course I'm going to add the tag and then I'll move to the budget. So over here as SA showed I'm going to click on the budget going to create select like the name of the budget. Let's call it LLMdev.
[28:23] I'm going to select the workspace, the Unity AI gateway. Then I'll select the tags. So remember I put the LLM dev and then engineering. I'm going to draw engineering. I'm going to save it.
[29:06] >> Okay, I did select another another tag but again I just created this one that's why I didn't it didn't propagate yet. So basically over here I chose a tag and then I have something important called share threshold. So basically over here I will specify the budget that I will
[29:20] attribute to this team. So let's say I'm going to attribute a budget of $5,000 and then I have two options. It's either when I I when I am above that threshold, it's either I'm going to send an alert
[29:33] or I will just stop like block the usage. Then I'm going to save it. And then I can define I can define a per user uh per user threshold. So basically here
[29:45] I'm going to specify for example that I'm going to give $100 to every user. >> You're doing a shared threshold. >> H >> you're doing a
[30:11] a specific user for example suppose this one is a power user I'll select sadhana
[30:24] and because she does vibe coding she she deserve $200. You see I'm too kind with her. And then I'll be able to save this uh to save this to to save this specific budget. Of course, once it's once it's
[30:36] saved and then like people start using the platform, I can then click on that specific uh this is one I already created. I can view the threshold I have. So over here I have threshold of $10,000 per user. Then I can click on
[30:49] per user spend. And as you can see over here can see the user that are that did exhaust the usage and over here I didn't block the usage. It was just send an alert but I could have stopped the users above the threshold I would have
[31:02] mentioned I would have specified and once I have this specific information I can click and then maybe this user now deserve to be a power user. So what I will do I will just override it and put
[31:14] now the new threshold that should be for this user uh maybe $2,000 dollars and I'll I'll save in it. So this way I'll be able to understand what was the usage, what was the AOE and
[31:28] does this user deserve to have a higher uh level or not. And the same thing would apply would apply to Genie. So basically in the Unity AI gateway I could I could have shown you that I have
[31:40] the possibility to select here Unity AI gateway and then you can choose Genie suggested tags. You select a genie and then you can grant permissions. How many user like for example I will give a budget of 10 $10. Then above this you
[31:55] will I will either send the others or block genie uh usage. >> And a quick thing on Genie as you all know July 6th is when the Genie uh usage goes live. The Genie budgets are live now in public preview. They're going to
[32:07] G before then as well. So you can actually proactively start setting up those caps in your account ahead of Genie billing. So basically that was the and and something important the $10 or the 150
[32:20] DBUs that are given are not included in this in this dashboard which means anything you going to add that would be the in addition. So if you have $10 and you put a limit 15 then you have $25 per user for on for to use geni code or
[32:35] genie and in addition we have serverless cost controls and I think this one is pretty uh is pretty important basically as we mentioned first so we did call
[32:48] something called seess entitlement I think it's the yeah sorry serverless entitlement so first you have now the opportunity or the possibility to grant the serverless usage to a group
[33:02] of people and to disable it for others. So that was the first step and as a second step we're adding the serverless cost control where we could choose a rate limit. I'm I will for example give
[33:14] the the opportunity to say this job can go can like use a certain like medium serverless cluster or an L. So first you grant permissions and then you you will cap this the specific workload and this
[33:27] is really the next this is the next big thing that's how it's separated from the entitlement from serverless notebooks and serverless jobs because they work in a different way. We don't want to share or give the same rate limiting and
[33:40] that's what we're going to show you. you will see it's very easy when it comes to the entitlements are as as simple as giving the uh uh giving the permissions when it comes to uh what is it uh
[33:52] >> I think it's that last tab >> we're showing you we're literally building this right now so we wanted to show you what we're building >> so I'm going to create uh I'm going to create a compute and let's call it dev
[34:05] and then I'll choose whether it's uh s m or l and then I can easily grant the permission that who can use or manage. So the person who going to manage will
[34:17] have the possibility to of course increase the size of the uh the workloads and the one can use is the one who's been benefiting from this permission to make sure that this user cannot go higher than the uh than the M
[34:30] and then you will be able to create this and this is tightly coupled as I mentioned like the serverless entitlements where we will be able to grant user permission or revoke the permission when it comes to using uh to using uh serverless
[34:49] And so we delivered this session. We want you we want your help. If you can share with us your feedback, that would be great. >> And the feedback is we read every single piece of it. Actually, last year's feedback is what inspired the governance
[35:01] hub and some of our budget features. So just to show you how uh we really take this to heart. So >> yeah, and just keep in mind, make sure to use tags. Those are tags are very important. That's the only way for you
[35:14] to understand how like the the granularity of your cost. You have what Sad Hannah show like demos the control you will be able to drill down till the like lowest lowest one and of course the
[35:28] predictive optimization that will help you optimize your cost and this is an iterative cycle plus the possibility to to rate limits and put some guard rails make sure that you're always in you're always controlling your cost and you're
[35:41] being proactive not reactive. >> Awesome. Thanks everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.