FinOps: Automated Unity Catalog Cost Observability, Data Isolation and Governance Framework
Summary
- Centralized governance: Westat implemented a centralized Databricks platform supporting hundreds of research projects with strict data boundaries.
- FinOps framework: Leverage Infrastructure as Code, cost tagging, and Unity Catalog for automated cost observability and budget management.
- Research scalability: Enable isolated project environments with row-level security, regulatory compliance, and self-service cost analytics.
FinOps: Automated Unity Catalog Cost Observability, Data Isolation and Governance Framework

Talk by Brian Sokol, VP, Data Architecture and Engineering, Westat ; Dylan Ford, Databricks Practice Lead, Aimpoint Digital
Westat, a leader in data-driven research for 60 years+, has implemented a centralized Databricks platform to support hundreds of research projects for government, foundations, and private clients. This initiative modernizes Westat’s technical infrastructure while maintaining rigorous statistical standards and streamlining data science. The platform enables isolated project environments with strict data boundaries, centralized oversight, and regulatory compliance. It allows project-specific customization of compute and analytics, and delivers scalable computing for complex analyses. Key features include config-driven Infrastructure as Code (IaC) with Terragrunt, custom tagging and AWS cost integration for ROI tracking, budget policies with alerts for proactive cost management, and a centralized dashboard with row-level security for self-service cost analytics. This unified approach provides full financial visibility and governance while empowering data teams to deliver value.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
What is Westat's data platform architecture?
This video describes how Westat, a 60-year-old data research leader, built a centralized Databricks platform to support hundreds of research projects for government, foundations, and private clients. The architecture provides isolated project environments with strict data boundaries while maintaining centralized oversight and regulatory compliance.
How does FinOps improve cost management on Databricks?
This video explains that Westat implemented config-driven Infrastructure as Code with Terragrunt, custom tagging, and AWS cost integration for ROI tracking. This FinOps framework provides full financial visibility, automated cost observability, and proactive budget management through alerts and policies.
What governance features does Westat use from Unity Catalog?
This video demonstrates how Unity Catalog enables Westat to enforce consistent data governance, maintain row-level security for sensitive research data, and ensure regulatory compliance across all projects. The platform provides centralized metadata management and lineage tracking for audit and discovery.
How does this approach serve Westat's research community?
The platform empowers research teams with self-service cost analytics through a centralized dashboard while maintaining strict data isolation and security. This enables researchers to focus on their work while the platform handles infrastructure management, cost tracking, and compliance.
Full transcript
Okay, hopefully everyone can hear me. Yeah, nice. Um, so today we'll be talking about um how Apoint Digital helped Westat achieve success on data bricks using Phox essentially. with me is from Westat. Um, as I mentioned. I lead the data bricks practice for Apoint Digital. I'm also a data bicks MVP. I've been with Data Bricks, uh, I've been with Apoint for about 3 years, growing the data bricks practice, um, rolling out projects, building accelerate accelerators, that sort of thing.
Um, prior to Aimoint, I worked with Disney um, for about four years rolling out um, data bricks there for their theatrical and parks divisions. Aimoint Digital is a market leader in analytics. We cover everything from data engineering um infrastructure, operations research, mathematical optimization and we have a very strong um generative AI and machine learning practice. Uh we were fortunate enough to win the data bricks partner of the year backtoback.
So that um 2024 and last night awarded 2025. Um, we also have a very strong culture. So, you can see here we um we have a um high employee retention rate and people love working for Aimoint. So, always hiring as well. So, look us up. Hi, So, uh with Westat. Um I'm the VP of data architecture and data engineering. Um, Westat does public policy research primarily for the US federal government. Um, that includes uh, a lot of the HHS world, so CDC and the CMS and as well NIH as well as Social Security Administration, Department of Labor, Department of Education and so forth.
I've been in the kind of intersecting space between uh, data and technology on the one hand and public policy on the other. Sort of always working in my career. um in that uh nexus including working for a long time on homelessness data as well as social security disability insurance and I spent uh during the co years a lot of doing a lot of work on public health. Westat has been in the data collection and data analysis uh business for over 60 years.
Um and through that we've built many we've done many many contracts uh again primarily for the federal government. Um and we have therefore a very strong uh culture in our client work on um data and data analysis. We have hundreds of statisticians, lots of PhD economists and epidemiologists and all these kinds of things. So we've been working kind of old school um data, you know, data collection and data analysis for many years.
As I like to say, we've been doing data products before it was cool. We just called it delivering data. Um so but now we wanted to kind of up our game and get into the newer um the newer spaces. As an example of some of our projects just so you get a sense of the differences and the scale of some of the things that we do. We have one big project called NAPE which you might is national uh assessment of educational progress.
You might have heard of it. It's been in the news as the nation's report card where we actually go out and we we do testing of approximately 250,000 students in the fourth, 8th and 12th grades. And this has been going on since 1969. We've been colle uh this data has been collected. So we can see over the years how students have progressed. It's also the only thing that we have as a country to be able to compare uh educational progress across across states.
So every state has their own testing, but there's no way to really uh determine uh you know how Massachusetts is doing against Oregon or something like that except for through through this project. So there's a lot you can understand a lot of data collection effort in there. Um but also a lot of sensitivity. We're dealing with student records because we also want to collect their demographics and their and that's and other things about them and the socio economic status.
So it's very sensitive data. Similarly in another well in another case we have the uh MEPS project which is the medit med medical expenditure uh panel study which basically is the uh source of information uh on medical expenditures throughout the United States. So we do lots of data collection from households on where they spent their money and we're hoping we just bid recently bidding data bricks on the medical provider component.
Um so again this is kind of a national sample of all Medicare expenditures and just a third just to I'm just trying to give you a flavor of the different types of data is the cancer tracking cancer testing support unit which basically helps uh researchers that are doing uh cancer treatment studies to get enroll people into their into their studies and roll out protocols and do and do the research that they need to do and we kind of provide centralized support for that.
So all these are different types of things. Next slide please. Um that we do and there are about hundred other projects that we're we're doing at Westat in any in in any given year. All some much smaller with just a few people doing the research one-time analysis some with streaming data with data pipelines and large uh analytics and machine learning and all these kinds of things. So what we needed when we decided that we wanted to do kind of create one single enterprise platform um we had we had a bunch of challenges.
Up until then we've basically created um custom uh environments for each project based kind of on their specific uh highly tailored needs. And a lot of the stuff was on prem because they developed over time and they've kind of all built on them on themselves in a kind of incremental uh way. But we wanted to say now we're going to be doing doing using one platform to do all of our work. So one key challenge was data segregation.
Now I hear a lot at these conferences about breaking down silos which in some ways you know we need to do but most of all in our case actually our challenge was to make sure our data was segregated because we have all these restrictions that coming from our contracts and just due to the sensitivity of the data we needed to make sure that our education data was not mixed in with our our health health data. Our clients had requirements that they couldn't be intermingled.
But it's a bit of a challenge because we also have people working our developers and our data scientists and data analysts work across all these things. Maybe in a day they might be working on four or five different projects. So we needed tightly controlled governance in order to maintain that segregation. And also it's not so simple as that because at the same time there's also a lot of data sources public use data sources like census data or geography units things like that are that we don't want to have downloaded and you know each time for every new project that we wanted to be able to share and develop on top of.
So that's was one key challenge that we were looking at. The other key challenge is that we have all these pro projects and once we get them the government wants us to start working on them yesterday, right? We get it and then you know we can't say oh it's going to take us you know 6 months to set up our our infrastructure. We needed almost a push button way to start up new projects which meant um you know figuring out what data sources we need and connecting to them.
We needed to code them in order to get to kind of track them from the the work that we're doing into which project it was doing for so we can build it later. We needed to be able to um code the where the data where the data is going to be stored all these different things all at once as and of course setting up all the different users and their roles. So we needed to be able to do that in basically less than a day for each new project so we can get get up and running.
And thirdly, I kind of already alluded to this, but we needed to be able to track cost. So if you have people going from working on one one project and then toggling to do another project for an hour because they, you know, got interrupted and needed something immediately and then going back, we needed to know exactly which cost which compute within with in our platform was being attributed to which project. And similarly at the data on the data storage side we're we deploy um on AWS we needed to also associate the same costs um for the data storage with the uh compute costs and people needed to be able to monitor tightly and kind of have complete observability into those how much cost we're spending on a monthly basis in order to know whether you know whether we're staying in budget and take action if we need to.
So all these uh kind of key challenges, you know, were a lot as we started to bring up the platform. So we did turn to our partner and he's going to explain is now going to explain a little bit how we got it done. Thanks. Thanks. So a lot of the challenges that was talking about can be solved by PHOps. Um so what is PHOPS? PHOps is essentially applying DevOps principles like automation, observability, streamlined processes.
Um, so applying those to the way that you operate to maximize value, um, and increase accountability across your engineering, uh, your finance teams and your business users. uh the Phops Foundation basically built a a framework that um enables uh PHOPS and so they break this down into three different categories uh operate inform and optimize. So I'll walk through each of these and how we enable Phops on um uh on data bricks for Westat.
So the operate um portion of the framework is essentially um reducing processes through automation um and building out your data culture um so that you have like a shared responsibility in um tracking costs. Um the inform section is basically ensuring that you're tracking all of the consumption. All of the activities that are um being undertaken on data bricks are being tracked so that you can um assign them to um workloads.
Um and optimize is essentially um just reducing um minimizing wasted spend and m like maximizing performance. Um so the way that we built out the operate portion of PinOps is essentially using Terraform. So we um uh built a um essentially the um the two ways that we could have solved this problem for Westat was either build a workspace for each um project which would have been an enormous overhead. Um and the other other way we could have done it is through Unity catalog.
So essentially through Terraform we're building a Unity catalog as a a sort of a micro environment. So um each project is essentially equivalent to a catalog. So with Terraform we we're deploying a catalog. We're deploying um AWS S3 buckets. We're adding tags. We're um creating different groups. So there's a a project admin group. There's a developer group. Um, we're pre-creating compute. Um, and all of this is enabled with Terraform, which is essentially just infrastructure as code.
Um, one of the challenges with Terraform though is that it's very easy to get software bloat essentially as you build out each module. Um, you end up repeating a lot of the code in Terraform. And so what Terror Grunt did does is essentially just um enable you to build a template of Terraform code and you then use Teragrant to define the configuration that is then applied to um to that Terraform template. And so it essentially just reduces a lot of the um the overhead, a lot of the code that you um have to define. um the inform category.
Um essentially now that we have all of that um uh metadata and all of that consumption data tracked at a granular level, we're able to um uh build an observability dashboard. So with the dashboard um Westat's able to uh look at each project individually. Um we use um system tables. Um so system tables are some system tables are enabled by default on a data bricks workspace. Um I'd recommend if you're not using all of the system tables that you look at um how to enable all of them.
There's a simple script that you can run that just enables all of your system tables. It gives you a lot of um insights into um what's running on your data bricks platform. Um yeah uh and then the final section is uh optimize with data bricks. Um so we use a lot of out of the box features. Um, data bricks um provides a lot of um great functionality. So we um we default to like serverless compute. We use manage tables.
Um there's a lot of great features in using manage tables. You get um like optimize commands run automatically. Um, another thing that we've built into this cost tracking dashboard is um, budgets. So each project has a budget that's um, uh, the project owner can set. So they're able to track the costs at a granular level um, against the budget. Um so uh they're able to identify if the consumption patterns that they're seeing um match what the goals of the project are.
Um and so all of this is through a like a self-service analytics dashboard that we've built for Westat. Um so now that we've deployed all of this platform, um Westat's already seeing their um customer data being secured. uh they're already starting to charge back to their clients. Um and we're already seeing innovation be accelerated on the platform. Uh I've got a QR code for a blog that we wrote on on Phops. Encourage you to check that out.
Um we will be writing a technical series on everything that you saw today. We'll be um writing more blogs on all of that. So, um, encourage you to, um, to look out for those and then. So, now what we're able to do is go to our clients and we've kind of branded our own pro, you know, our own product as West Litics. That's Westat and Analytics obviously. Um and we kind of can sell that both that we can do everything that they need if you look at the center in terms of in terms of their data pipelines that those are built in and we have kind of the most powerful you know tool on the market to do data acquisition engineering all the analysis that we've always done as well as AI and actually had a little bit of an argument with the graphics people because I wanted to say that AI was going to across the spectrum because we also can do AI as part of using AI as part of the ingestion and engineering.
But in any case, as well as data dissemination, which can be delivering data data products or it can be bringing people onto our platform. Um, we're creating a workbench so that analysts who are our clients can go in and do and do their own analyses. Um, so we could do that on the project level, but we also have these enterprise functionality. And what you get when you get West Litics is you get Westat's expertise and our quality and our our kind of reputation of being in this market for 60 years.
And we're built on data bricks which provides a central way to bring the scalability and the security and the and the governance. Um as well as of course the topic of this conversation which was you know the cost consciousness of being able to monitor for cost which was sort of the what this federal government is certainly very interested in. Um so that's what we have for you today except to remind you to sign up for to take the survey and feedback and we're here for any questions.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.