Implementing GreenOps in Databricks: A Practical Guide for Regulated Environments

Summary

• GreenOps optimization: ABN AMRO Bank uses Databricks system tables to monitor resource usage and reduce waste across 500+ workspaces.
• System tables power: Automated dashboards analyze millions of job runs daily to surface optimization recommendations for engineers and managers.
• Practical implementation: The bank shares real pitfalls and learnings from deploying GreenOps in regulated banking environments at enterprise scale.

Implementing GreenOps in Databricks: A Practical Guide for Regulated Environments

Talk by Dave Ruijter, Solution Architect, Blue Rocket (ABN Amro)
Join us on a technical journey into GreenOps at ABN AMRO Bank using Databricks system tables. We'll explore security, implementation challenges and best-practice verification, with practical examples and actionable reports. Discover how to optimize resource usage, ensure compliance and maintain agility. We'll discuss best practices, potential pitfalls and the nuanced 'it depends' scenarios, offering a comprehensive guide for intermediate to advanced practitioners.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is GreenOps and why does it matter for data platforms?

GreenOps uses data observability to monitor resource usage, identify waste, and recommend optimizations. This video shows how to build automated dashboards from Databricks system tables that reveal cost savings opportunities across thousands of jobs.

How can I monitor waste in a large Databricks environment?

System tables track all jobs, clusters, and workspace activity. You can build PowerBI dashboards that analyze this metadata to identify failed jobs, underutilized clusters, and recommend specific optimizations to engineers and managers.

What challenges should I expect when implementing GreenOps?

Security, implementation complexity, and the 'it depends' nature of optimization trade-offs are key considerations. This video discusses real pitfalls ABN AMRO encountered in a regulated banking environment and how to navigate them.

Can I implement GreenOps in a regulated industry?

Yes. The Databricks Data and AI platform provides system tables and observability features that work within regulated environments. Proper access controls and auditing ensure compliance while enabling cost optimization.

Full transcript

Good morning everyone. So before we uh really start just a quick imagine exercise. Imagine you're in a a company where you have over 500 workspaces, more than 3,000 users on those workspaces creating their data workloads and more than a 100,000 jobs running daily on average and the weekend a little bit lower. Um, in your imagination, based on your experiences, would that estate be perfectly clean and organized or do you see some waste, some failures, some um underutilized clusters?
Um definitely um if you look around in our company's estate, you'll see some things you you can optimized optimize and that's what this session is about. Uh it's about me telling you how we uh try to optimize our data estate. Little statement that everything I say you can't take for granted and I don't promise you anything blah blah blah blah blah. Uh don't forget to complete your survey after the session or right away whatever.
Um so let's have a quick teaser about the solution um we have built to help us to monitor our data bricks landscape. This is a PowerBI dashboard. On the left side, you'll see uh a navigation uh hierarchy and the datab bricks monitor is now where we are in and you'll see immediately like more than 10 pages in the databris monitor. We have many things we can monitor with our solution. Uh, and one example here is the recommendations page.
And this sort of spoon feeds a couple of recommendations that me and my buddy uh have um designed to help the users in the organization, maybe management, maybe engineers to just know what they can optimize. We'll dive into these uh pages later on a little bit more. the agenda of the session. Um I'm going to introduce my my company Eenomro a little bit and what we do with data bricks and then I'll show you what we have built, how we have built it.
Uh we use system tables for the solution. So we'll talk about that and um I think maybe the most important thing is the challenges the pitfalls the learnings that we have yeah perceived um I want to show share that with you to make your journey as smooth as possible. So, quick introduction. Uh, Abidomo Bank, it's one of the three major banks in the Netherlands. It's a global bank. Some people know it, uh, some people don't.
25,000 employees, quite big. A main office in Amsterdam in the Netherlands, but also, um, yeah, good presence in all over Europe. What do we do with data bricks? We've been using it for a while now. Um, we have quite substantial number of workspaces, over 500 actively used. We we actually have 1,200 workspaces, but those include the ones that are about to be decommissioned, etc. Um 3,000 active monthly users. That's also a lot.
Um many of them use it daily and weekly. And our platform is the platform within the bank for data workloads. Uh that's why it's also used that much. And we started using Unity Catalog about two years ago. um in a like whenever it was available we started to embrace it and now it's the standard to use for managing our data and we have a close collaboration with data bricks a couple of people in the room as well thank you um yeah so that really helps to be on top of all the things that is u happening and as I said um as you can imagine if your data bricks state is this large um you can look around and there's not not a lot.
Um you don't have to look around a lot to quickly find something to optimize. So me and my buddy in in the Netherlands uh my teammate, he's not here, but um uh yeah, we we looked at that and we couldn't be a bystander of that. So we wanted to change that. Uh we saw clusters running 247 and we wanted to do something about that. Like it it it touched our hearts. We don't really care about about the cost of the platform.
Um, but we do care about the waste, the footprint of the environment. A lot of energy is consumed by the stuff that we're running in data bricks. So, we want to optimize that. Don't have waste. So, what do we what did we do? We educated, we rolled out policies and tagging, mandatory training. Um but in the end we did um feel like we were just doing it manually everywhere. We didn't have the scale yet. So um to support our work we created a a little monitoring solution.
Uh we started small deployed it to a couple of workspaces and the people using that workspace they had lots of success. They reduced terabytes of data. uh they could monitor their field jobs a little bit better and uh yeah it reduced the footprint of the we optimized the footprint but we struggled to scale it up. Uh we couldn't deploy this decentral solution to all of the hundreds of workspaces that would be undoable.
We were not the admins of those workspaces. So yeah, that's wasn't sort of it was a dead end for us to really um move move further and at the same time the organization stepped up their sustainability goals. It's it's becoming really important for the bank and um they they give it more priority. They they knock on the door and say hey what what does data bricks do? How like the people managing data bricks within the bank?
How can we improve that? Uh so you know those two things was a a driving force for us to put our feet on the table and think uh what can we improve uh what would we best to focus on and in the end we uh we decided to make our solution more lean more green and more agile and those things um are sort of the fundament under what we do. It's a bit a bit blah blah marketing, but it really is um the green thing is really important for us, but also the lean.
Um we think we need to reduce everything uh to only the stuff that we really need the really use. Um, and the agile thing is if the platform is using the latest technology, the latest runtimes, you're you're able to adjust really quickly to what data bricks offers to what your company says, okay, now we need to do genai. Well, fine. We can just really quickly adopt that by being agile and being on top of the things. Um, and that has impact on the green side of things as well. the later runtimes, they are more optimized, faster with the same cost.
So why not use them, right? All right. And then um to scale up to our impact, we needed more awareness, more knowledge sharing. We needed the people to be self-sufficient. Um it's no longer and having a meeting with people. they can just look at their report and study it, use it, um, and become more self-sufficient and they do the stuff themselves now. Um, and with that new solution, we also introduced sort of a better language to talk about um, greenups within the bank.
Um, that's it's a bit vague and indirect, but management and engineering, they they don't share the same language. So if you talk about a cluster, maybe an engineer understands it, but if you go deeper, sometimes they just disconnect. And with our with our report, they they both share the same definitions and language. So that really helps for both of them to talk the same language, and that helps to you make it make it even more easy to optimize the platform.
Those are all nice goals, but um how do we do that? Well, we need to give the people insights into the things they can improve. Simple as that. So, um, we wanted to table spoon them table spoon feed them these insights about cluster utilization out of termination, job successes, runtimes that are on really old runtimes, etc. And um, for us to also give the information to the right people, we needed the organizational mapping to be part of that as well.
With only a workspace name, um, the management they don't know what that is, right? They see a gooid or something, maybe a name. But um we also wanted to have the proper domain and departure mapping and ideally we would have footprint metrics, sustainability metrics. That was really a challenge as well. Data bricks doesn't uh supply that by default. you only have the DBUS, maybe some some some dollar metrics, but things about energy consumption, it's um something you need to do yourself.
So that is also a bit about why not use the out of the box insights. Um today I learned that tomorrow a lot of things will be announced and today many new things are shipped by data bricks but still this this it's still focused on the operations. They did they do a better job now with these improved dashboards with identifying trends etc. But we need um footprint metrics and that's still not there. And also we need to tie it to our own goals, our own regulated context context and we need the like the global scale of within our uh uh company of course and some of the dashboards are still local.
They're workspace level dashboards. So yeah, that doesn't really work for us. So we build our own solution. The picture is a bit simplified and I also focus today on the system table as a major major source. We do have other things in our monitor that we read from other places as well, but let's focus today on on this picture. We have a Unity catalog view layer and a PowerBI solution on top of that with a semantic model reports bundled into an app that we make available within PowerBI.
PowerBI within ABN umbro is the BI platform the reporting platform. Maybe in the future that might change a bit but people might do more in data bricks. I'm looking forward to that to experiment with that but for now uh PowerBI is like the place where also the management lives they they know how PowerBI works. So for us to distribute this uh the PowerBI app was the best solution. What do you need to get something like this really working?
Um a couple of data bricks workspaces and they need to be Unity catalog enabled. If you don't have that yet, at least get one where you can develop a solution like this. Uh because you need data brick system table access and that is a requirement. You need a unity catalog uh enabled workspace to be able to access the system tables. I'll talk about it later a bit, but um if you have just one workspace and maybe 10 or 100 more, the uh those will show up in your system tables as well, not only the Unity catalog enabled workspaces.
So that's uh that's nice. And for this solution, you of course need also PowerBI workspaces. Then let's let's show more of our solution. Um, as I said, we have dozens of pages, but I want to show you uh just a couple of examples. Um, and later on maybe I can demo more one-on-one. So, this is the consumption trend page. Uh, this helps as a starting point. It looks nice. Management likes it, but it only shows uh the data in a in like a account level um way.
It's good for us to monitor what's the bigger average, the bigger trend. Um, and it's it's pretty stable. Um, you see some dips in the weekends. That's really interesting to see. Uh, over the holiday period in January, December, we also saw major dip. Um, but then we recovered and it's it's the same again. And uh, what's really uh, yeah, what people use a lot is the table. um I had to offiscate a lot of the data but you get the idea every row in that table is a domain and uh you can see the consumption per domain so the domain owners are really interested they don't want to be or maybe they do want to be the top um uh consuming domain and uh PowerBI by its nature has interactive click clickability so if you click for instance on the left side to on SQL the whole dashboard would focus on that so you can understand quickly who in the company is using SQL warehouses um and who's maybe not yet.
Um then this recommendation page, I already discussed it a little bit, but it's the thing in the middle with the colors that that plot, that's also really interesting for the users. They really like to compare themselves to their peers. They don't want to be at the top uh the bottom left corner. That's where they have the largest spend, the biggest footprint and the largest um potential of savings. Our recommendations identify the savings that's quantifiable and uh the more you have the more you are in the bottom left corner.
So people ideally move to the purple or the blue uh section to have an optimized way of using data bricks. And that also really clicked with management. They they they now they are um engaged and the engineers the proper engineers the good ones they they already were engine engaged they they were actively using they were knocking on my door saying hey you you report this but can you explain what what what can I do but management was like yeah okay you have this report whatever but now here with this this visualizations they are really really engaged so that helped us lower on this page we have the three recommendations Um first one is um uses us usage of spot for your machines.
Um those are machines that are unused Azure capacity that you can just um take uh when it's available at a deep discount. It's only 10% of the VM cost. So that's really um um yeah a good thing to use. Um there are some some it depends scenarios like you can't use it for every scenario but for many you can. So if you click on that um recommendation you'll uh dive into the details and that's also really important to take away from the session.
We need both the top level management metrics but also the details need to be one click away. you need to be able to look at okay what cluster is that is it why do you report that as a recommendation to save um and optimize um these these these rows in these tables are individual clusters and uh if you click on the name you'll even jump to the data bricks workspace to then maybe change it directly uh management won't do that maybe they are not part of the workspace even but the engineers like me we can so they boom quickly we can reduce the um the size of the cluster or change it to a spot cluster.
Going back to this page, uh the second recommendation is about preventing failed workloads. Um failed workloads don't deliver any any value. They they they just compute something, boom, they fail. Uh but yet they induce cost and footprint. So we need to limit that. And if you click on that recommendation, you'll end up at this page where you get an overview of all the field jobs over your scope. Uh if a domain owner is looking, you can they can slice it on their domain workspace on their workspace etc.
You get the idea. And this uh gives a trend and also they can slice at the top right. You see the status. So if you only want to look at the timeouts or the the failed jobs, they click and the rest of the page will filter to that selection. It's really valuable for people to identify the field workloads. And what we think is something we recommend to improve is anything lower than 75% success rate. It's a bit arbitrary.
Maybe 80 90 is is the new level. But yeah, we think 75 is is okay for now. We'll be more strict in the future. I guess if people are like all at the 75 or higher, then we we set the bar a little bit high a bit higher. But now there's many many many jobs failing. Um even my own jobs I have the decentral solution is still running and on some workspaces it's failing 100% of the time and I just didn't notice. I get emails but I get a lot of emails so it helps a lot.
All right so that's this page. Um another example is this page that offers a consumption per job. Uh you can look at all the job and its runs in the selected period. This is one month or three months. It's at the top left corner and um there's many metrics to order and select and filter uh to identify improvements. And if you click on one job, the graph at the bottom will show the runs of that job. So here you see a couple of quick job runs, but then also a couple really long um durations.
Maybe that's okay and expected. Maybe that there was a peak in data, but maybe not. Maybe something else happened. And the longer your job runs, the more computed costs. So yeah, the maybe you should look at this. And the good thing about this report is that it it's really quick to identify the outliers. If you go to each workspace, each job, yeah, it takes a lot longer. You don't do it now. It's just spoonfed into your Yeah. into your your monitor basically.
And then this one also really uh use useful um it's about cluster utilization compute utilization. Uh all the clusters in your organization as I said we have 100,000 jobs per day. That's a lot of clusters. Um orange means underutilization. Uh you can reduce the cluster definition. Um those metrics are also available in the database workspace. But who clicks on every works uh job and then goes to the metrics page? Nobody does.
So um you only do it when you're like, "Hey, this job failed or why is it taking longer than normal?" Then you go to the metrics. Otherwise, you don't. So when it's running fine, it's fast, right? Yeah. But it's fast because it's like really over um oversized. So, um, this example, there were some occasions where the utilization was reaching 100% really shortly, but the rest it didn't do anything. It's like 5% CPU. So, you should you should look at that as a as an owner of this cluster.
Another example, the same page, but this uh shows a cluster that's properly sized, but if you look at the um the right side, you have the driver and the worker nuts. Look at the driver. That's orange. It doesn't do a lot. So with this cluster definition probably can we can reduce the driver uh configuration. The workers are probably properly scaled but the driver can be tweaked and uh the the top table also immediately shows the size of the the driver and the worker.
So you can immediately see oh okay 96 cores that's maybe a bit much. If it's already at eight, well, let it be. Yeah, there's other things to focus on that's that have more impact. Okay, that's the quick tour of of our pages. We have many more, but um yeah, I think u let's focus on how we built this. Um where do we get the data? Data system tables as I said, they contain the operational data of all the workspaces in your account.
Uh as I said, not the not only the non uh uh uh not only the Unity catalog workspaces but also the non-Unity catalog workspaces and they're stored in your um region but in the Azure datab bricks storage account. So um be mindful that when you need access to the tables you'll need to uh cover the networking part as well. Um, there's many tables, 25 and counting. I think this this week they're adding more. So, uh, it's sometimes hard to keep track.
The documentation is quite good. Helps you with the ED like this. But, um, yeah, you just need to start working with it. Uh, a couple of notes on it. you might want to consider enabling schema evolution in what you're what you're building on top of the schemas because um new columns can be added just sort of randomly uh not randomly but at any time that's what I mean and um yeah you want to you want to be able to consume those at least not fill your your views or your jobs on top of this uh it's not real time so if if you're looking for something like that um this isn't tables is not the place to get that data.
Uh if if you have like a daily refresh and you think, okay, the lag is is a bit too much, maybe refresh a couple times a day because data is added throughout the day to these tables. And lastly, there is a free retention period for this data. As I said, it's in their storage account at data bricks. They pay for it. U within that free period, it's costless. But um the retention period is is a year for most of the tables.
Sometimes the tables are 30 or 90 days. So look at that. If you need more, yeah, reach out to your account team and then they can probably extend it maybe with a cost or not. I don't know. And to use these system tables, you need three things in place. It's important. This can be bit complex as well. So, I wanted to give you this. Um, an account admin needs to enable most of the schemas and that's a little command that they need to run once per schema account admin.
So, if you're not, then you can't do this yourself. You need to uh find that friend in your organization. And as I said, you need to grant network access to the storage account. Um, this can be a hassle in your company. I don't know within ABNR, it's a regulatory environment. it's a bank firewall opening up to the public. It's it's a big deal. So there's a process to do that. We needed to document that and then um we got it in place.
And lastly, um they are a catalog the system tables just as any other Unity catalog. So you need to grant access to be able to read from it. Uh that's maybe the simplest part to do. All right. Um before we dive into the last part about the challenges, the pitfalls, the learnings, I want to briefly touch a very important topic uh security um it's until now I I said just share everything share everything give insights uh and it's it's good this transparency gives you trust and accountability within the users of the bank and in your organization.
It enables collaboration and learnings from each other. But some data is sensitive. Um most of the data is metadata. But as I say, it's just data as well, right? So, uh look at the company policies you have for sharing data. If it's PII, uh the audit table might contain or does contain PII data. So, yeah, look at that. Verify what you can do within the rules within your company. Datab bricks also says with that black comment that uh only access admins have access and if you want to share certain tables they advise to have some role level security or views dynamic views that when when department A is looking that they only see department A's data and that's that's important thing we do that as well in in our solution.
All right, the challenges. We struggled with some things. It was a learning curve. It was nice. Uh but one of them is the volume of our data. We have hundreds of millions of records in some of these system tables. That's because of our data state. Maybe in your environment it's a little lower and then you don't have this this struggle. But PowerBI semantic model that we have um we constantly need to look at that like this new table that we want to add um how many records is it is PowerBI can handle hundreds of millions of records that's fine but the system tables also have a lot of columns and uh they're usually also like strings so um that's easier to uh not not that easy to get into power yet.
Um the next thing is the metrics. Um by default you have US dollars and we live in the Netherlands. We we need euros. So we have a euro conversion and we need um energy consumptions in kilowatt hours. We need uh CO2 emissions. Um that was really hard to set up. Um we have our calculation now. Is it perfect? No. um we have some data over the years but Microsoft and also data bricks they don't share a lot about this so we had to just search around use some some research papers and then uh yeah we now have something but uh it would be really nice if if if this was made easier so yeah let reach out if you if you also struggle with this then we can help and then as I said earlier tagging and mapping is important uh domains uh teams environments production versus development.
We need that. Um, and the only thing you can really do that is via tagging of the of the of the objects. So, make that mandatory have some policies for that. Then, uh, this becomes um, yeah quite easy to do. Then the pitfalls um, and me were both engineers originating in from the data warehousing BI. So we always think more data is more better. Give us data. But there's a lot of things to to find in the system tables.
So um yeah, sometimes you just need to put the brake on and and think about your users. Um they don't need 20 slicers to work with filters, whatever that maybe I would use that, but uh they they need a simple dashboard just um it works out of the box. That's uh that's your aim. And um we were also struggling. We do it better now to connect with our audience. We were in our cubicle just developing the report, add more data, more sources, more pages.
But um yeah, you also need to just talk to them and say, "Hey, we have this page, use it." And then yeah, the imp then you make the impact, right? So that's uh that's something we sometimes need to focus on a little bit more. And we have made a Powia report because um that's where our users are. The management is there. Um think about your your situation. Maybe you have Tableau or other reporting tools. Uh it's a pitfall to just use your your comfortable zone.
Maybe if you're a Datrix engineer, you want to make a Datrix dashboard, but that's maybe not what your user wants to have. And we overestimated the knowledge of our users. We still do. It's a pitfall. We when we say, "Hey, just use spot VMs." They don't know. Maybe some of them do. But many of them are like, "What spot VMs?" Yeah, that's a checkbox. I never touch that. Um, so don't don't overestimate what they know. And uh add some some links to pages where they can learn what you're talking about.
And the learnings that I want to give that like what we over time learned. Keep it simple. The kiss principle is really important. In the in the beginning we had a whole gold uh silver gold medallion architecture for our solution. That was overkill. A couple of views were fine. And now when there's a new column we just we have we have schema evolution. So it automatically is added in the power semantic model. And that's really really nice for us. it makes it a lot easier to maintain.
Um, we do lean on the retention of the system tables. We don't copy the data ourselves yet. Um, so that's something that we're going to focus on soon. I talked at Khan about that. Like maybe we need to um, uh, extend the retention or copy it, have a separate process to copy some some data. Um, but u, yeah, still keep it simple. Uh I think I talked about to reach the top you need the bottom as well. It's it's like both ways.
Um and over time the system tables got known more and more and more. There's now lineage in there. A lot of users want to use the system tables within the bank. Um and over time we sort of became the go-to guys for that. But um yeah, sometimes it's it's hard. we'd like to educate but also we are not the gatekeepers of that data other people in the bank are. So we we forward them to that. Um and lastly the recommendations for us was really like a level up.
Before that we just had the other pages with all the details. You could just find everything in there. But like the aggregation on top of that where do you need to look at what is really what I can do? That's the um the recommendations page that we started six months ago and now we're adding and adding more to that because that's that's that's what goes around what gets screenshotted and sent to to other uh managers like uh you need to focus on that.
It says you can save 50,000 euros. Go do that. And otherwise uh yeah before that it wasn't like crystal clear. All right. Um to conclude, I really want to urge you to just take the first steps. Just go build something small. Um every action you do counts. Every cluster you you you optimize and you don't have to do it alone. Uh reach out to me. Um happy to help. Uh even I can do some consultancy to build it together with you.
And we have some time for Q&A. Uh but yeah, really start today, reach out to me and we can shape tomorrow. Thank you. Thank you.

Learn more about the Databricks Data and AI platform.