Databricks FinOps Genie — Cost Observability Meets Optimization Insights

Summary

The FinOps Genie solution combines a Databricks-native AI/BI dashboard, a Genie Space, and a FinOps agent to turn usage and billing data into actionable cost intelligence, proven at scale across 1,000-plus workspaces at a large bank.
A Genie Space lets users query their FinOps data in natural language to get explanations and optimization recommendations, moving FinOps from a reactive, visibility-only exercise to proactive cost action.
The architecture uses Databricks system tables as the data foundation and includes a knowledge agent for policy-aware cost Q&A, plus forecasting and anomaly detection to surface optimization opportunities automatically.

Databricks FinOps Genie — Cost Observability Meets Optimization Insights

We will showcase Databricks FinOps Genie, a cost observability and optimization capability that combines a Databricks‑native FinOps dashboard, Genie space and FinOps agent, proven at scale across 1,000+ workspaces in a large bank. Learn how to turn Databricks usage and billing data into a lightweight, reusable cost intelligence layer that surfaces key drivers of spend, anomalies and optimization opportunities through a single dashboard. See how a Genie space lets users “talk” to their FinOps data in natural language and receive detailed explanations and recommendations. Take home patterns for using a FinOps agent and recommendation model to right‑size clusters, address anomalous workloads and enable teams on Databricks FinOps best practices, so you can apply the same approach in your own estate.
Talk By: Shaurya Rana, Sr. Solutions Architect, Databricks ;

Chapters

FAQs

What is the Databricks FinOps Genie?

FinOps Genie is a custom cost observability and optimization solution—built on Databricks capabilities, not a standalone product—that combines an AI/BI dashboard, a Genie Space, and a FinOps agent. It was deployed at scale across 1,000-plus workspaces at a large bank to surface cost drivers, anomalies, and optimization opportunities.

How does the Genie Space work in the FinOps solution?

The Genie Space lets users ask questions about their FinOps data in natural language and receive detailed explanations and actionable recommendations. A knowledge agent makes the Genie policy-aware, so responses reflect the organization's own FinOps rules and governance context.

What data sources does the FinOps Genie use?

The solution is built on Databricks system tables, which contain detailed usage and billing data across workspaces and compute types. This lightweight, reusable cost intelligence layer surfaces key spend drivers without requiring a separate data pipeline or external cost tool.

How does the FinOps Genie move from reactive to proactive cost management?

Instead of building dashboards that only report what happened, the solution uses a FinOps agent and recommendation model to proactively right-size clusters and flag anomalous workloads before costs escalate. Forecasting and anomaly detection capabilities surface optimization opportunities automatically.

Full transcript

[00:09] Hi everyone. Uh good day. Am I loud and audible to everyone in the room? Great. Um very good afternoon everyone. Uh my name is Shia Rana. I am a solution architect uh at data bricks. Work for
[00:21] our financial customers, financial services customers more in the AMIA region. And I know it's uh a session in the afternoon but uh if PHOPS and costs keep you awake then I think uh you will
[00:34] uh be quite awake during the session. Uh the primary plan today is you know to talk about an implementation of PHOPS at scale which we did at a large bank and I'll talk you through what were the key
[00:46] design decisions we took uh when driving for PHOPS and optimization at scale. Uh so let's start with a quick uh show of hands right uh how many of you own or help maintain like a observability or a
[00:59] phops dashboard in your data estate. Great. So all of you are kind of phops practitioners. Okay. And how many of you then think uh you actually have really good visibility into everything your
[01:12] teams are using specifically from a datab bricks perspective? Wow. various some some very confident people which is amazing. Uh and then okay I already just had one hand on
[01:25] visibility. I was expecting more but then uh how many of you think that these dashboards or the observability solutions are leading to actions happening like people proactively taking action based on these PHOPS dashboards
[01:37] etc which you maintain a few hands. A few hands uh that is good. So that is what we're going to talk about today, right? How to make the PHOPS observability, PHOPS dashboarding, the
[01:49] mostly reactive trend into proactive, right? How do we move in the PHOPS life cycle to being proactive when it comes to cost optimization and observability? Uh so quick uh touch on the agenda. I
[02:01] think all of you are PHOPS practitioners. So we won't go deep into what is PHOPS, what is observability. Uh and then we'll go into what data bricks gives you today from a Phops perspective. And as you might know Phops has inform, optimize and operate when
[02:14] you think about the PHOPS key principles. And then we'll get into the interesting bit which was how did we design for PHOPS at a global bank. Our three top moves I've distilled it into three top moves which we did and then
[02:26] we'll do the demo right and I think uh the catch was everybody looked at this phops genie right? So what is this phops genie? Um quick quick caveat right this phob genie is not a data bricks product phops genie. It was something we built
[02:39] which you can build for yourself right and that's going to be the takeaway how you can build your own phops genie and not just phops it can be for anything else too but primarily how that phops genie was built for eliciting action
[02:51] from your multiple teams and of course what was the impact we got out of it and where this goes next uh we'll try to keep questions at the end I'll try to leave time at the end for questions and we'll also take some takeaways from here
[03:03] I have tried to open source a lot of the things which we built uh with this rank for you to then take away and try it out yourself. Okay. Uh quickly going into observability, right? So observability
[03:15] as you all might know is the way you understand a system and anticipate and prevent problems. One big important part about observability is that observability is different from monitoring. Right? Monitoring is somebody trying to monitor what you are
[03:28] doing. Right? So you are building a dashboard trying to understand what is happening under that. Observability is when a system makes itself observable. Right? For that you need the right telemetry coming out of your system since day one right and that is what is
[03:42] step one to do any kind of phops right you need good observability data bricks as a platform gives you close to end toend observability and we can go into that in the Q&A what is something which you don't think uh is not observable on
[03:54] the data bricks platform but it starts with step one which is observability next is then the phops piece right so this is the definition I'll not go into it but phpops the driving factor for It
[04:06] is financial accountability through collaboration. And typically uh when you are a data platform architect or a data platform owner or even if you are owning a business unit or a delivery team, it
[04:18] is this collaboration that needs to happen between engineering some kind of finance function right who are responsible maybe IT function and then the business teams or your data scientists or your end users who are
[04:30] building on your platform. uh and then there are multiple you know personas which come in uh I've highlighted phops because I'm we are going to talk today about phpops but this kind of uh paradigm or this genie which I'll show you can be extrapolated
[04:43] then for other personas too right uh you can have security operations where people want to detect threats etc so for example if you saw the keynote with lakewatch lakewatch has its own genie which is for sec ops personas you can
[04:55] have devops which is devops for infra devops for data and then you can also have devops for AI right and this is now being called AI ops or LLM ops, right? And then each of these personas have different user journeys or the questions
[05:07] they're trying to answer. We'll try to answer the ones specifically for FinOps. How much are we spending? Which projects are costing us the most? How much should we budget for the next year? Are there any anomalies uh when it comes to our
[05:19] spend? And most importantly, right, how can we maximize ROI on our data estate? So that's high level on what we're going to go into today. And this starts with the PHOPS life cycle, right? Um, how many of you was
[05:33] there anybody over here who was at the Phops summit last week in uh San Diego? Anybody? I know nobody does two conferences back to back, but uh there's a Phops summit and there's a Phops Foundation summit which happens in San at San Diego. Uh which is where you know
[05:46] there's a big community of PHOPS practitioners, right? uh and typically sometimes phops is a separate function data might be a separate function or it might be lo defined right and phops life cycle consists of these three levels
[05:58] right step one is inform getting end toend visibility and getting end to end allocation of what are you using when it comes to your databicks estate we'll stick to datab bricks mostly today right and that's where tagging strategy
[06:10] mapping strategy comes in right so this is the first step but this is the hardest step quite frankly right because when I'll show you in the demo mapping tagging Enforcing these uh best practices is step one. Then comes the
[06:23] optimize piece, right? Uh you go into optimizing primarily in the cloud world your utilization, right? because we are in a opex model not like a capex model which you're used to typically in the on-prem systems and within data bricks
[06:36] there are three primary levers right uh data bricks charges you prim primarily for compute right so the clusters the CPU etc right as you know storage and compute is decoupled so you pay for the
[06:48] compute but then databicks can also help you optimize the storage right even though the storage will sit on your S3 ADLS Google storage whatever it might be but databicks does has capabilities to help you optimize the storage used by
[07:00] your compute. And then one thing which people do not realize in many cases is the network cost too, right? There can be things like egress cost, there can be things like firewall rules, there can be things like uh uh private links etc.
[07:13] Right? Those are two costs which you need to be aware of when you are thinking of phops perspective and once you have the visibility you have started optimizing is when you come into the operate cadence right how to build this
[07:25] practice uh the phops practice where you can continuously monitor and report you can enforce this governance and you can automate a lot of this cost optimization and also one most important thing is make it self-service right because a lot
[07:39] of your teams might have a centralized data platform where a centralized data team decides or infrastructure team or a devops team decides you know each cluster how does it get provisioned in that case it is comparatively straightforward to do phops right
[07:52] there's a single but bottleneck or uh single owner for your phob strategy but then when it comes to a federated model right think of things like data products data mesh kind of aology then you need
[08:04] to make it self-service all right and that is where you need to think of a great operating principle for phenops okay uh coming to And what does data bricks gives you for fops right um data
[08:16] bricks has multiple capabilities available to you out of the box in these three levels right the a few which I like to call out is in the inform phase uh I don't know if you paid attention but we launched this thing called the
[08:28] governance hub right it is now in I think private preview that is an account level governance hub which will have three levers cost AI and data governance uh that is the product uh uh direction
[08:41] when it comes to finops or even data ops. All right. And all of this is underpinned by system tables. So how many of you are aware of what system tables are? Okay. Most of the room. Yeah. So that is
[08:54] a single source of truth or a system of record or the telemetry data which makes datab bricks observable. We'll go into that because my whole demo this whole solution is built on system tables. Separate to that as you might see I have broken it down by field and product. We
[09:07] have a lot of solutions which the product provides you out of the box. But then field as in field engineering which I'm a part of builds a lot of accelerators which you can use and plug and play and use them for your fin use cases. Some good examples being like the
[09:19] workflow advisor dashboard for your jobs. Uh there's a table health dashboard which is now being integrated into the governance hub. Uh and then you also have this uh SQL warehouse advisor uh which is again becoming a part of the
[09:32] product. But these help you for specific SKUs you use from data bricks and how to use them effectively. uh I I'll try to share this deck in the GitHub repo so you guys can get links to all of these because these are all
[09:44] hyperlinks to each of these uh resources. Okay. And just a quick touch on system tables, right? Those of you who do not know what system tables are, right? So system tables are data bricks provide provided manage tables in delta format.
[09:57] So open source format about a customer's data bricks instance. So everything you build on data bricks the telemetry data of that is available on system tables. Right? If you ever look at the database
[10:09] UI, you will see a system catalog within that you know they will have multiple schemas and as you can see over here there is billing, there is compute, there is query serving a lot of different tables, right? Uh data bricks provides these to you for free and there
[10:22] is a automatic retention you get for these system tables and those you can use then to query or build these dashboards uh out of the box. Okay, that was a primer on what all is
[10:34] PHOPS but let's get to what we did right. So what we did was we turned cost data, right? All of this data, all of this dashboards, etc. into cost behavior at a large bank with 1,000 plus
[10:46] workspaces. Originally, I was going to get somebody from that bank. We decided to co-present, but they couldn't make it. Uh otherwise, it would have been great to hear from themselves, but I I what I did was I took a replica of what they did, and I'll show you that in a
[10:58] demo, too. But how did we build for action, right? We didn't build for we initially of course built for uh you know monitoring purposes but then how did we change that into actionable insights or actions that can be done by
[11:11] teams. So this was the use uh scenario right 1,00 plus workspaces data best practices everywhere there was no action and the paradox was like this right the data was obvious you could see that there was
[11:23] 200k plus of visible waste in 30 days alone and nobody acted right there were dozens of dashboards not not being actionable nobody knows who has to basically take an action on this phops
[11:35] recommendation there were shareepoint sites with best practices there were power dashboards with all the data but uh you know getting the development teams or getting the multiple teams in this fragmented uh architecture to to
[11:49] take some action on it was not really happening and guidance right uh guidance is usually you know it has to come with clear action actionable next steps otherwise guidance is mostly taken as takeaway and action doesn't uh happen
[12:01] after it this is an example of what they had right there were multiple PowerBI dashboards as you see there is no der of observability and data right I've redacted a lot of things but you see multiple pains in a PowerBI dashboard
[12:13] showing you spend analysis, jobs analysis, what is allpurpose compute, query level analysis, a lot of data out there. It was very information dense. People had all the data, right? But the action wasn't there.
[12:26] So what we decided was we will stop publishing more data or best practices and we'll design for action and attention, right? How can we use AI? How can we use generally design thinking, right? To get get people to action on
[12:38] all of this uh observability data. So top three design moves, right? Number one, information design from a dense everything dashboard to a layered view, right? Starting from an exact summary to actionable to-do and then giving people
[12:51] the capability to still drill down, right? Because there are people who will challenge your recommendations. There are people who will say that, you know, okay, this is not relevant for my team. We still need x amount of compute or y amount of networking bandwidth etc.
[13:05] Uh then of course you know one thing which was very interesting uh what they did was is this concept of a leaderboard or a comparison right when you have humans in the mix uh comparison or leaderboards really drive action right so uh you know we created a leaderboard
[13:18] when it comes to things like optimization uh there was a mandatory phops council setup and a phops review setup where if you are on the top uh list or or you call the naughty list of being too wasteful uh you had to take
[13:31] some necessary mandatory actions right and this was top- down setup at scale, right? And that helped at least from a human perspective to get uh you know get to fixing things when it comes to your data or AI estate. And then of course
[13:44] all of this was substantiated by Genie, right? Genie was the place where if anybody has any questions or any doubts about what they have to do with these actionable recommendations, Genie provided you all the information and we added one thing to this which was a
[13:56] policy aware knowledge agent and I'll go into that to what this knowledge agent was. So starting with the information design as you can see this is u a screenshot of the dashboard we built it starts with a high level what is the
[14:08] total spend and then the next uh if you see below there is a leaderboard what I've done is I've taken a replica of what we built there and literally put it in our demo workspace and this is real consumption but I've just created team
[14:20] names for reduction purposes but you see you know then there's a leaderboard under it under it which shows you efficiency scores for the multiple teams and that clearly defines this human uh cycle psychology that okay our team is
[14:32] lowest when it comes to on the on the leaderboard. Right? After doing this information this uh design step uh and this was the exact summary, we went into actionable to-do straight away. Right? This is where the
[14:44] optimization piece comes in. Right? What to fix and who has to do this, right? So it was not like you know here are 20 different signals of optimization you can do based on clear dollars saved without affecting performance is where
[14:57] we gave this actionable to-do right and that is where you know we uh tried and tested multiple recommendations so that we do not go out with a recommendation which affects a team's performance right so we start with low hanging fruits and
[15:10] I'll go a bit into detail on that and that is what you know the number one paradox or number one issue we hit was How do you balance for cost and performance? Specifically in data bricks, right? You have to think about
[15:22] things like fault tolerance. You need to think about throughput versus concurrency. How much is the execution time? Will this scale? If you tell somebody to go to a single node cluster, they start hitting issues and then they start saying, "Oh, this was what we got
[15:34] from the PHOPS review and uh our production jobs are failing." People do not want to touch production code if it works, right? Howsoever much it may cost. So, you need to be careful when you build these recommendations, right? And of course, things like startup
[15:46] latency, etc. And because of that we started with this kind of pyramid of what are the recommendations you should start with right a lot of people think you know let's start optimizing data optimizing code right let's get our developers to
[15:59] write really good code right that is the last thing you should go after what we realized in the case of this bank with 1500 workspaces was half of the benefit was by just reducing waste right things like idle compute things like right
[16:12] sizing infrastructure which I have second over here and then later you know you can go into deep dives into optimizing data and code. Just FYI, I was walking around the expo hall yesterday and I think I saw 10 different
[16:24] vendors who have solutions which help you with optimizing data and code, right? Which are great quite frankly they are great but now with AI writing most of your code a lot of it is already optimized, right? And you can of course
[16:36] leverage them but I would still say start with the lowhanging fruit and then you can go into detail and a lot of the solutions uh third party solutions do a good job at even calling these obvious signals. So you know this was a case where they decided to build it from
[16:48] scratch internally but there are a lot of good solutions out there too in the partner ecosystem we have and then a little slight touch on the compare and compete piece right so as you can see there's the team leaderboard
[17:00] dashboard but then also there's a genie right so there's somebody asking a question that what what are the teams uh with the lowest efficiency score and what do they need to carry out as optimization actions and you see you get
[17:12] the teams and clear actions by literal deadlines given out to them. Right? So that was the compare and compete piece. So what is this uh phops genie and what did we build exactly? Right? What are the underlying things we built? We built
[17:25] a simple AIB dashboard did some information design so that it is layered from an exact summary to do to drill down. Then a curated genie space. So as you might be aware AIB dashboards give you genie out of the box where you can
[17:37] have conversations with the dashboards itself. But we also created a separate genie space which could be then served or embedded as API for anybody in the bank to view it because this is tele this is telemetry data anybody should be able to view it and everybody should be
[17:50] able to see this leaderboard right and finally we integrated this with the agent bricks knowledge agent this this capability just went I think it's in private preview still but a few months back and I'll show you in the demo where
[18:02] what we added to this was because Genie gives you structured data right all the phob data you get it from Genie agent bricks gives you unstructured data. So there were pages in SharePoint with Spark optimization best practices, VM
[18:14] right sizing uh uh best practices or company policy like you know if you want to use serverless you need to work with private link right or if you want there is a mandatory autostop you need to have all of these policies were written on
[18:26] SharePoint pages which nobody was reading all of this was integrated into Genie. So if somebody comes with a question to the genie that okay why am I uh you know spending x amount what can I do it will pick up from those pages and give you the recommendation grounded on
[18:39] the company policy. So yeah this is just a repeat of the examples we did with the genie as you can see over here some very interesting questions being asked that what are the unused tables uh which we can
[18:51] decommission right so as I talked about a lot of waste right people have built tables or you can call it technical debt this uh bank was using data he's using database for eight years now right so they started in the spark days so a lot
[19:03] of technical debt was lying around right and what a system tables can give you is what is a read pattern based using things like lineage it can tell you that if this table is not being used at Right? Uh you can start decommissioning those. Right? Uh those are the kind of
[19:16] insights which Genie can provide you. Right? Can I move my workloads to serverless compute? Uh if it is better from a cost performance perspective, but then you know you need to know the policy. What is the bank level policy to move to serverless compute? Those are
[19:28] the kind of questions this Genie plus knowledge agent combo can help you answer. uh and yeah this was uh one very important discussion we had that LLMs know all about like all the spark
[19:40] documentation is on the internet all the datab bricks documentation is on the internet genie is already trained on it right so why not just the underlying lm which genie uses is good enough which in um in in most cases is genie code 2 you
[19:54] know is trained on all the datab bricks documentation it knows how to optimize code it knows how to optimize pipelines but you might have your own organization specific policy Right? You might not be able to just take those recommendations
[20:06] and action on them. That is where you can put in your own guardrails when it comes to these optimization recommendations. That's why we added this knowledge agent along with the genie.
[20:18] Okay. So these are the kind of questions. I think it'll be better to show you them in the demo. And let's just go into the demo then. I'll just change my view here. So yeah uh this is the dashboard we
[20:31] created. This is of course a replica of what was created uh in the bank. What they did was they created an app where this dashboard was embedded as an iframe with the genie as an iframe embedded as part of it. Right? Because they did not want anybody to come into the database
[20:43] UI. The phops persona doesn't want to log into a database workspace. Right? The lead of a delivery platform wants to come into a separate app where they get this leaderboard. they get the exact summary and if developers want it in
[20:55] their own uh workspace uh that was also an option given to them to literally take the uh the JSON uh file for this dashboard and run it on their own uh workspace. So as you can see over here starts with an executive summary of you
[21:08] know what has the total spend etc. This is real data by the way for databicks's own workspaces right and you can start seeing over here these are the optimization leaderboard right so what we've done over here is we have given global filters here by line of business
[21:21] this is where the number one step right the tagging piece so you can still see some are nulls that means they're not properly tagged and the subdivision and then within that the environment right some people just care about fixing production because dev doesn't have
[21:34] anything or sometimes it's the opposite you want to fix do not touch production but just make changes in dev or test right So all those tagging is what drives this dashboard and then uh comes the interesting bit where you can start asking questions right so you can see
[21:46] things over here like average efficiency score right so uh AI dashboards give you the capability to talk to your dashboard directly so this is that embedded genie we're talking about right and I've already asked a few questions conscious
[21:58] of you know time you can see over here uh how is this efficiency score calculated what is group technologies efficiency score right group technology is a team within this right and it tells you you know what is the formula right
[22:12] and what what what is the efficiency score how does it calcul is it calculated and what are the different efficiency scores by lines of business right and there's a target which has been provided and based on that you understand that are the teams meeting
[22:24] the target right and then there's another uh visual in this um dashboard which is talking about things like spot instances right so what are spot instances a lot of you uh you know a lot of people are like what are these spot instances why is this popping on the
[22:37] dashboard All these kind of FAQs which you have is where Genie can help answer it right and this is where Genie has common knowledge about things like what are spot instances why is the spot instance uh usage currently 30%. So for
[22:49] example, this bank set a policy saying that spot instances are mandatory in dev right spot instances are comparatively discounted VMs you get uh when you are setting up uh infrastructure uh on data bricks right and this gives you all that
[23:01] detail so this is step one the dashboard uh with the view right I'll just shut down the uh dash uh genie and these are the todos so you see these are the clear todos and those three buckets I told you
[23:13] right reduce waste right size infra and then optimize data and code Right. And what we seeing over here, the biggest bucket is reduce waste. Low hanging fruits which you can straight away fix. And this is real data. The only thing
[23:25] I've uh you know uh redacted over here is I've made up team names. But this is real data. Right? This is uh for one of our field engineering meta store which we are hitting over here. Right? And as you can see over here, clear
[23:37] recommendations provided to teams. What is the estimated dollar savings per month? What is the severity? What is the utilization? What do they need to do? Right? uh now still this might not be complete information. They'll be like oh I'm showing up over here but what do I
[23:50] need to do? That is where you can go into the standalone genie for more uh deep understanding of what are the next steps you need to carry out. Uh so that is high level the dashboard and then if still people want to drill
[24:02] down right they want to go into details of what is the all-purpose compute spend what are the job spend what is the SQL warehouse spend you already get all of this data quite frankly uh those multiple links I had we already database
[24:14] provides you all of these very good dashboards for SQL warehouse for jobs you can just plug them in into this dashboard that's what we did too right that uh that is not the hard part giving all the telemetry to your end users there are plug-and-play dashboards available right it is about mapping it
[24:27] to your teams and providing it as actionable recommendations and uh some things which are interesting which is also coming as part of product are things around data level recommendations. Right? So things like unused tables or uh tables which can
[24:40] benefit from automated automation which optimization which data bricks provides things like vacuuming things like uh uh predictive optimization right a lot of them are provided out of the box for you from data bricks and teams are just not
[24:52] aware of these latest and greatest. So a lot of times the multiple teams looked at the central team for this best practice guidance and they were the ones working with us as the account teams. We provided them that best practice. Now it
[25:04] become self-service. Right? The team can go to this genie get all this guidance self-service and start actioning it. Right? And if they think that this action is not elicited for their team they should not do it. They can of course there's a phops review where you
[25:16] can discuss and decide on what are the actions you going to take. And then this is the standalone genie space. Right? I already asked a few questions to this genie space to show you u how this works. Right? So you saw
[25:28] the dashboard and the dashboard had an embedded genie. Uh this is a standalone genie space. What is the difference in this? Right? The difference over here if I go into the configure tab right uh is if you see the data for this uh space
[25:41] right you have multiple system tables derived views. But over here I think I dropped it. Let me just add it again. Uh basically the biggest differentiator in this is let me add it again. I think it
[25:54] got dropped while I was refreshing it. This is the bit where you can add a knowledge assistant into a genie. Right. So what is a knowledge assistant? A knowledge assistant is a part of agent bricks. It's like a rag in a box. Dump into it uh hundreds of sharepoint files.
[26:08] dump into it um some Google Docs where your company policy is written or PDFs or wherever right and you just provide that as a rag in a box which then Genie can refer to while taking decisions both
[26:21] on structured data and this semiructured or unstructured data. So uh when I ran this I think this was integrated I think it got dropped but now you can see this is a separate knowledge a uh knowledge agent se sitting next to all the tables
[26:33] which are coming from system tables and again similar questions then what are the top optimization recommendations by team right uh all the details provided to you right uh I'll go into more detailed question I'm from XY Z team how
[26:46] to action them are we violating any PHOPS policies right those are the kind of uh uh when it comes to the leaderboard or you know the policies being alerts alert alerted on. So the teams were alerted if they were violating a phops policy and this went
[26:58] as a uh automated email to the software component manager is what they call it right. So that is where then they came to the genie to ask okay what are the policies we're violating how do we fix them. So you see this this is where they
[27:10] start getting a clear uh description you know what is the efficiency score this requires mandatory review the auto termination policy is within 15 minutes this is the stuff which genie normally cannot give you right this is written in
[27:22] your policy which you're defining at a central level uh there's a separate uh policy for warehouse uh there is a spot instance policy there is a DBR version policy right all of
[27:34] that has been applied so maybe a DBR is going end of life those kind of very bespoke policies come as part of it right and these are the then step by step given to them how do you fix all of this right uh I'll
[27:46] skip through this a lot of detail here the best part is I am going to share the source code for this dashboard you can set it up yourselves too and then things like what are the top unused access uh assets right so how many unused tables
[27:58] do we have near zero utilization warehouses near zero utilization clusters we call these zombie warehouses some data scientist team spinning up a lot of things so there are two ways Right? This is reactive right now. This
[28:10] can become proactive too. Right? So when I said the operate piece, this is the optimize piece. In the operate piece, you won't just allow. You'll have policies where this cannot be allowed. But this this kind of happens, right? When you give people the capability to spin up their own clusters, you will see
[28:23] this kind of waste popping up. And that is where you need to cut down on it and then build CL policies to not even allow such thing happening again. But this helps you get a good view of what is going on in your data state. So that is high level uh you know what what this
[28:36] Genie provides you. Uh I I have more details in the GitHub where you know what are the typical questions and what are the typical algorithms we used under the hood for this. Uh okay back to the
[28:48] talk track. Uh let's go into slideshow mode. Conscious of time here. So what was the impact right? Uh about 20% less wasteful compute and this we
[29:01] are talking you know at millions of dollars being spent. Uh what was the impact right? This allowed them to free up resources, free up infrastructure for doing you know for their engineers or
[29:14] the developers to get more business value to the bank. Right? So this waste was uh visible but it wasn't actionable across these 10,00 workspaces. Now teams started self-correcting to climb the
[29:26] leaderboard. Right? So uh it's sometimes it's the carrot, sometimes it's a stick as they say but basically team started selfcorrecting on their phobices. uh most importantly the actions were now self-service. This was a very high level
[29:39] view. You know, we wanted to understand that what are the places where we found the maximum value and interestingly this is of course u specific to this customer. So it can be different for every organization how mature they are
[29:51] when did they start adopting data bricks how how their teams are how well versed their teams are with data bricks for example. But in their case 72% of the benefit was by shutting down of legacy infrastructure or pipelines. Right?
[30:04] There were so much uh things they had built with time which nobody was maintaining and it wasn't being used right thousands of tables sitting there which are not being used and they populated with pipelines running every day. Maybe that pipeline could be
[30:16] instead of a daily batch it can be a monthly batch. That was the biggest thing they found out which was quite surprising for me too. Then comes the typical things like you know leveraging spot leveraging serverless uh for non-pro or cluster right sizing you know
[30:29] VM right sizing or in serverless case the different uh the autoscale setting up autoscale there very big one you know if you are aware of what is all-purpose compute what are job clusters then migrating from these all-purpose
[30:41] clusters to job clusters as you might know all-purpose clusters serve a different purpose versus job clusters and simple things like improving auto termination rules right this was then you know made as a policy as you saw
[30:53] over there right but because this is a highly federated bank the central PHOPS admin doesn't have workspace level admin access so they cannot enforce this so this still has to be then you know put it in this dashboard and put it in the
[31:05] genie so that the workspace admins work and put these policies at scale right so those are the top uh pieces which they found out which is really saving them millions uh in cost across their data and AI estate
[31:18] what where this goes next and there are a few key things which this doesn't show and which you might be thinking of already, right? Forecasting, right? That is number one. Based on this, we need to decide on our budget for data bricks or generally our entire cloud estate,
[31:31] right? And forecasting is an art, right? You can do a very simple thing like put an AI query over there and get a forecast, but every customer has its own uh bespoke seasonality or peak and trrow when it comes to forecasting. So you can
[31:44] add some kind of forecasting model to this to then forecast for the next few months, right? Anomaly and spike detection, we haven't done much in that space. That is something which is available to you by the uh in the product provider dashboard though uh a
[31:57] lot of things still need to be uh you know chagged and things like chargeback. So there are a lot of use cases where you want to do charge back. There's a central team you can have some chargeback that can be built of course as part of the policy uh in things like
[32:10] discount and contracted price modeling. This is not integrated into this. Uh our product team is working on making that available out of the box in system tables. So very soon you might not even have to do that model that in otherwise you can just put a blanket multiplier
[32:23] for whatever discount uh you have for data bricks. And finally right all of this converging this is what is coming from product by the way is converging it into genie code skills all the optimization recommendations instead of
[32:36] them still being you know they are actionable but some a human needs to do it. What if all of this can be done itself right? So if you see this slide the northstar is agentic workflows which opto autotimize based on this data or
[32:48] based on the trends or based on the guidance provided that would be the northstar right now. How these teams are do working with this is they take the recommendation put it into genie code and genie code does all the recommendations which are provided by this dashboard. Next step would be which
[33:01] is coming with our governance hub is taking all these recommendations and Genie code picks them up gets a human in the loop to review these changes and all the optimizations are done out of the box. That would be the northstar. But
[33:13] takeaways would be, you know, design for attention, not design for all the data, everything all at once, right? How do we get the attention? How do we make it actionable? Make it targeted and self-service. And in our case, what
[33:25] worked is making it a human game because it's the humans who need to take action on all of this data. So what I've done is uh just today I have put a lot of this into a GitHub repo, an open GitHub repo. um I had to
[33:39] uh I couldn't um make our database GitHub repo public due to some policies. So literally I've just dumped it into a open GitHub repo. What I'll do is I'll update this GitHub repo with the deck for today and of all the additions we
[33:52] keep making and very soon I think we'll work with product to you know see how to make this better. But this is the V0.1 I would say of what we uh built uh with this bank. So uh this does have the
[34:05] system tables AIBI and Genie built in. The knowledge agent is in private preview right now. So you might have to get access to the private preview via your account team so that you can integrate the knowledge agent also right but it has the optional example of a
[34:18] knowledge agent what can be there in the knowledge agent but typically you would want to point it to your own company knowledge base when it comes to optimization. Uh great. Um separate to this just after
[34:30] this session there is this session which is what's new for Phops on data bricks if you're still interested to hear more that's done by our product team what is there for phops out of the product out of the box right this this is a very
[34:42] specific uh implementation we did at a customer but you can hear about what's the product direction and again uh you can take a picture of this maybe of all the different sessions we we have around uh PHOPS happening at the conference a lot of them might have already happened
[34:55] but you can always watch them on demand uh they all get published on our website. Uh this is just a mandatory survey. You can skip it. Okay, I think five minutes left. Uh if you like the talk, you can
[35:08] of course uh fill the survey. I will open it up for questions then.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.