Operating Databricks at Scale: Govern, Build, Deploy, Optimize at FanDuel

Summary

FanDuel implemented a four-phase platform engineering lifecycle — Govern, Build, Deploy, and Optimize — to manage Databricks adoption across hundreds of users with diverse skill levels in a SOX-regulated sports gaming environment.
The Govern phase establishes infrastructure as code in Terraform with Unity Catalog and medallion layers plus cluster policies; the Build phase introduces standardized PySpark migration frameworks; and the Deploy phase implements code-first governance through Project Stargate and Databricks Asset Bundles.
The Optimize phase drives efficiency through systematic tagging, compute isolation, hardware selection including AWS Graviton, and storage strategies, creating a paved road where every platform decision scales across the entire engineering organization.

Operating Databricks at Scale: Govern, Build, Deploy, Optimize at FanDuel

As organizations scale Databricks adoption across hundreds of users, technical coordination, cost control, and governance become critical. FanDuel built a platform engineering operating model around a Govern, Build, Deploy, Optimize lifecycle that enables self-service teams while maintaining security, compliance, and cost efficiency across the platform.
Learn how FanDuel implemented infrastructure as code in Terraform, standardized PySpark migration frameworks, deployed code-first governance with Project Stargate and Databricks Asset Bundles, and optimized costs through tagging, compute isolation, hardware selection including AWS Graviton, and storage strategies. Includes real strategies for enforcement, team adoption, and building a paved road for platform self-service.
🤝

Chapters

FAQs

What is FanDuel's Govern, Build, Deploy, Optimize framework?

FanDuel's Govern, Build, Deploy, Optimize framework is a four-phase operating model for managing the Databricks Data and AI platform at enterprise scale, ensuring that every governance decision, development standard, deployment process, and cost optimization strategy applies consistently across hundreds of users. The framework was developed during FanDuel's migration from Redshift to Databricks while operating under SOX compliance requirements as a subsidiary of public company Flutter Entertainment.

How does FanDuel use Terraform for Databricks governance?

FanDuel implements infrastructure as code using Terraform to provision and manage all Databricks resources, ensuring that the platform configuration is version-controlled, reproducible, and auditable. This approach forms the foundation of the Govern phase, along with Unity Catalog for data governance, medallion architecture layers, and cluster policies that control compute costs.

What is Project Stargate at FanDuel?

Project Stargate is FanDuel's initiative to move Databricks governance from UI-driven configurations to code-first management, implementing everything as code through Databricks Asset Bundles. This approach enables consistent deployment practices across engineering teams and integrates with the broader Deploy phase of the Govern, Build, Deploy, Optimize framework.

How does FanDuel optimize compute costs on Databricks?

FanDuel's Optimize phase uses systematic resource tagging for cost attribution, compute isolation to separate workloads, and hardware selection strategies including AWS Graviton instances for better price-performance. Combined with storage optimization strategies, these practices ensure that cost efficiency is built into the platform by default rather than managed reactively.

Full transcript

[00:07] All right. So, my name is Eric Benson Pierre. I'm the director of core data platforms at FanDuel. Hi everyone. I'm Prabhjot Singh. All right. And we're going to get started with Databricks at scale, lessons from FanDuel operating, migrating, and governing at enterprise scale.
[00:25] All right. So, who are we? So, like I said, uh my name is Eric Benson Pierre, director of core data infrastructure. Um I lead along with many other people, of course, uh FanDuel's data platform strategy and modernization, uh drive platform standardization, and responsible for enterprise-wide uh Databricks adoption. And hey everyone. I'm Prabhjot. I go by
[00:41] Prabh. I lead the FanDuel data tooling team. And my main uh work has been architecting the orchestration, ingestion, and transformation frameworks, which we'll get into later on, and then also standardizing the deployment processes across data engineering. Cool.
[01:00] All right. So, the problem. Operating at scale. So, one of the things about Databricks is you can do anything, right? You can do everything in the UI, you have many different compute options. Um you have many different users, right? So, you have uh data engineers, software engineers, analysts, uh ML engineers, and they all
[01:16] have differing uh levels of skill. And you have to account for that whenever you're um working on the platform. Right? And in addition to that, FanDuel is in kind of in a unique place where it's sports gaming, um highly regulated, um SOX controls
[01:32] Flutter, the parent company for FanDuel is uh a public company, so we're subject to uh Sarbanes-Oxley uh regulations. Um we we also have uh budget constraints, right? Every company here, every business has budget constraints. You can't spend as much as you want on the platform.
[01:48] And we're in the middle we were we were, at least when we developed this framework, uh we were in the middle of a migration from Redshift over to uh Databricks. All right. So, this is our framework, right? Uh govern, build, deploy, and optimize. And whenever
[02:06] you're managing a platform and which is myself and Prab and a few others that are in the room, uh whenever you're managing a platform at for a large organization, you have to do it at scale. Every single thing that you do on the platform has to scale. You can't just look at an individual job. You can't just target one individual team.
[02:23] You have hundreds, some of you might have thousands of users and you have to do things in a way that um is efficient and makes and maximizes your impact, right? So, just a little bit about each one and then we'll go ahead and and uh turn it over to Prab. So, govern, um
[02:39] make sure that you are uh considering you're deploying everything as infrastructure as code, right? Build, make sure you're standardizing your migration uh frameworks or just your your code in general. Uh deploy, uh make sure that you are thinking about how you
[02:54] are bringing your code from development all the way up to production and optimize. Uh consider how you are holding the teams accountable on the platform and ensuring that they're using the compute and storage uh in an efficient way.
[03:12] Thanks, Eric. So, we started with governance because platform adoption was growing very fast at FanDuel, but platform governance was lacking behind and one small team couldn't account for all the different use cases we had. We had machine learning workloads, we had operational workloads, we had data
[03:27] engineering pipelines, and once that starts happening, it can kind of get out of sync once ClickOps gets gets involved. And as I as it says here, ClickOps ClickOps doesn't survive scale. We get a lot of configuration drift. There's no audit trail of any issues or any changes
[03:45] that were done to the platform. Every team kind of gets their own inconsistent setup. There's security gaps. And most importantly, there's no cost attribution. So, we don't know who is running what on Databricks and what they're doing, how much it costs, and if we're getting value out of the usage that they have.
[04:02] So, the one principle we have is infrastructure as control, infrastructure as code is the only control plane. So, we want everything to be in Terraform, clusters, catalogs, uh any grants, identities, everything is in Terraform. If it does If it exists on
[04:18] Databricks in our platform, it's already been in code, so it's been it's gone through pull requests, it's been reviewed. There's an audit log, so if there's a SOCS audit, any other audit, we can go back to get get history, look at everything that was done, and we know who made the change, why they made the change, cuz everything is tied to
[04:34] a support request, a Jira ticket that we did for a feature request, or some maintenance that we had to do. And across all the different environments, it's reproducible. So, in the case of a disaster recovery, we lose some assets, some old Terraform state gets applied, we always know what the true source
[04:50] is in the code, and we can just always reapply that. And within that repo, it's not just about putting stuff in Terraform, right? That's like writing a resource in Terraform, it's easy, but what's more important is the dependency management. We needed to have the account and identity defined before we could go and
[05:08] give them grants. We needed to have the S3 buckets defined before we could set up the catalogs and schemas that are tied to it. And in addition to that, setting up self-service starter kits, so teams can also go and self-service and not have to rely on one small platform team to do all the work
[05:23] for them. And the same thing applies across all the different workspaces, so we can have our different dev, QA, prod environments without needing to really have this uh different configurations for each one.
[05:38] And on the data side, Unity Catalog becomes the enforcement point. Uh everything we have is in managed tables, so that way we are able to leverage all of the Databricks optimizations and we have special organization for each of the different data sets that we have. So, we have the
[05:54] medallion layers which are data engineering owned, they're trusted data, those are the single source of truth for FanDuel, um but teams are also able to build their own products or on data products in their project space and they can experiment in their sandbox space which is limited to just their own team so
[06:09] that they're not able to really put stuff into production without going through the proper guard rails as in CI/CD which we'll get into the deploy phase and any restricted data that has to be a little bit more governed is restricted so only certain groups are able to access it and all of that is
[06:26] managed through Unity Catalog so everything is least privilege by default read only and writes really only given to admins or service principles in production to be able to to be able to change production data.
[06:41] And in this case after all of this is set up, the guardrails travel with the workload. So, people aren't able to just come up with a new idea and go with it because they're not able to do it. So, with cluster policies we have cost attributed already to with the tags uh only secure cluster mode so no one's
[06:57] going in a uh one of the no shared isolation modes. There's compute caps with how much how many DBUs they can spend so no one's running a cluster overnight that is not really being used but it's just active with 10 nodes running forever. Um there's
[07:12] identity groups that are synced with Octa so it's very easy to get access. You don't need someone to go in and log log you into the application. You just go put in a request through our forms and it all gets automated. And for CI/CD, no long-lived tokens that are stored in repos or Git tokens.
[07:29] We enable OIDC federation and that just connects through GitHub and is able to or or your any version control provider and you're able to have a secure connection into Databricks to deploy your assets. And moving on to build this is where the next challenge shows up. So, we had the platform governed,
[07:44] but we didn't want everyone to go and build their own different process. We don't want to reinvent the wheel. And that the governance gave us the consistency, but we didn't have the velocity until we really built a standardized migration framework. And that's what we'll cover in the build
[08:00] phase. So, we had governed platform, but we were still reinventing the wheel. Every team, you know, would be building their own pipelines. They'd be inconsistent quality. The observability wouldn't be accounted for, and the migration would be slow. So, we wanted to get ahead of that when we were doing our migration
[08:16] from Redshift to Databricks. And the build framework that we came up with was a modular PySpark library to follow the simple framework of ETL. So, read something, transform something, and then write it. And a lot of our sources are S3, JDBC,
[08:34] Kafka, or SFTP, or REST. So, in order to make that easy for everyone to to use, we built a PySpark library with those modular components that they can just import, whether they're doing it in a custom PySpark code, or a templated entry point that we've already set up in
[08:49] the CLI. You can go run through those without needing to, you know, everyone writing spark.read.parquet or spark.read.csv. You just have a standardized place to do it, and you don't have to repeat the code every single place. And it makes the library version
[09:05] controlled, so we can always make upgrades, and everyone can adopt to a standard framework. And the payoff for that was a standardized migration. So, before everyone had their own migration pattern, custom code, took took a long time, inconsistent quality. And then
[09:21] after that, we had repeatable entry points, templates. Instead of months, it could take weeks to build a pipeline, or even less, um especially with the agentic support. Everything was consistent, observable, and tagged from day one. So, we were able to build in a a of those platform guard rails
[09:37] from the beginning. And over to deploy, I'll hand it over to Eric, but now we had workloads running in a consistent way, and the next problem showed up when we had to get those into a production environment, and over to you.
[09:55] Thank you, Rob. All right, so deploy. All right, the deployment problem. So, one of the issues that we faced and then and I mentioned this in the beginning, um on Databricks you could almost do anything, right? You could just do it in the UI, you could create a notebook, and it's running, and it's uh reading and writing data just like that, right? Um and you
[10:13] have many different teams doing things in a many different ways, and then that could be okay if you have uh analysts who want to do things one way uh versus data uh data engineers or ML engineers, that's totally fine. Um but you run the risk of having analysts producing uh
[10:30] production-grade uh maybe they're not production-grade, but they're production assets. And you really, really want to watch out for um analysts who may not be thinking through uh what they're doing, and they're just trying to get it uh get a table out or get a a report out, and kind of bringing
[10:47] them into the fold of like how to do things properly uh on a platform like Databricks, right? And Yeah, let's keep going. So, uh in comes Project Stargate. So, this was my brainchild. I kind of wrote
[11:02] it up and started shopping it around, and it turned out this was something that the organization really, really needed, right? So, this is uh going back to Rob's point, everything as code, right? We don't do the UI thing or we're moving away from the UI thing and into um a world where we're doing everything
[11:18] as code, and it's one governed path for deployment of all assets uh on the platform, right? So, uh DABs or declarative automation bundles is what they call them now. Um you can really deploy anything on the platform using this framework. I highly encourage
[11:33] it for any of the platform administrators to really think about how to get this integrated into your workflows for your teams. And you're you're going to really get the visibility into who's deploying what and ensuring that they're doing it the right way.
[11:49] What it brings for the organization is one standard path for the deployment of assets, version, reviewed, repeatable. And then in addition to that on the UI, you're not really going to get the security visibility that you get in in GitHub. Right? So in GitHub, when you're doing
[12:05] the CI CI CI CD checks, you're going to be able to whatever your security provider is, you're going to be able to put that on your PRs and you're going to get a lot of value out of that in the long term. All right, so this is just kind of how it works. It's very easy for
[12:21] us. It's easy, right? It's a software development life cycle. Hey, you're not break No nothing groundbreaking. But if you have analysts, right? Or analytics engineers, people who really aren't used to it, and they're maybe just used to maybe doing things in let's say the Google BigQuery, right? You're just
[12:38] writing SQL queries and that's all good. But on Databricks, like I've been saying, you can do so many different things and it's important to bring them into a framework where you can take them kind of into a a long-term sustainable path. Otherwise, you're going to have cost overruns, you're going to have sloppy code, you're going to have
[12:55] assets that aren't really trustworthy or reliable, and you can really get a lot of value out of pushing them in this direction. All right, so one size does not fit all. So for the FanDuel team here,
[13:10] everybody especially on the platform side, what we found was analysts did not like this. They did not like And that's an understatement. Yeah, they did not like it. They did not like YAML. They did not like dabs. They
[13:27] didn't They didn't get it. Like, why? Why do we have to do this? And it I would say it took a few months, really, and escalations and complaints and quite a bit, um, to kind of get them in the right direction. That said, um, once we, uh, really invested in the in
[13:43] the training, um, teams were like, "Okay, I get it. If I'm going to produce assets that the entire business is going to rely on, then it needs to follow a certain standard." And that's really what I'm trying to get across here. I'm not saying that this is like groundbreaking in the sense of like, "Hey, this is a
[13:59] software development life life cycle. We know what that is. Everybody here more or less knows." But, how do you apply that to your entire organization? Right? And then how do you build trustworthy assets, um, that could be, um, QA'd, developed, QA'd, and then deployed and relied on
[14:14] across the organization? Um, natural fit, obviously, data, ML, software engineers, they're like, "Hey, this is great. We were using Terraform before. Now we have DABS and now we have a full uh, a better workflow or more, uh, more familiar, uh, workflow to to go, uh, to use. And then obviously
[14:30] testing and uh, reproducibility and promotion is pretty straightforward for for engineers, right? All right, what it enabled. So, this is pretty easy. Uh, reliability, reliability, resilience, uh, security and governance, and
[14:47] operability, and developer experience. Uh, like I said, like for us as engineers, it's it's pretty straightforward. This is what we want to do, and if you're not doing it as as an engineer, I I highly encourage you to, uh, think about it and get out of the UI and get into the to the infrastructure as code across all of your,
[15:04] uh, assets. All right, optimize. All right, so, uh, what you want to do with this framework as a platform administrator is make sure you're building in efficiency as a default. Um, and, you know, we face this at at our organization a fair amount where we're
[15:21] kind of chasing down this team or that team to stop spending money on whatever it is. Um but what you really want to do is like do this all in the beginning. Do it all in the beginning so that you're not spending uh you know, your precious time chasing down teams around the uh organization and you can really spend
[15:37] your time doing the more advanced things, right? If AI, whatever it is, you want to spend your time on the um on the high-value tasks, right? So, it's not chasing lower bills, restricting users, uh reviewing every workload by hand. It doesn't work. It doesn't scale. It's not going to work across thousands
[15:52] and thousands of workloads. Uh what it is is transparency into usage and costs, uh accountability by owner, and then intelligent defaults uh that guide good choices.
[16:09] Okay. Everything gets tagged, right? So, we had the uh tag-pocalypse. Uh was that like last year? And basically, um even prior to the uh migration, we were on Databricks, but it's mostly um machine learning, data science, and some analytical workloads. And those weren't
[16:25] tagged, right? And so, what we did is we, you know, within the data organization, we said anybody who's using the platform, anybody who's running anything on the platform, it has to be tagged. There's just no way around it. So, we created automation. It goes through all the jobs, relatively straightforward. If there's no tag,
[16:41] there's a there's a set of governed tags. Um if it's not tagged, it gets paused. And so, obviously, people complained. They were like, "Why is my job paused?" or whatever it is. It took them maybe 2 weeks to figure it out and it that was it. And that's the standard going forward. So, I would really, really encourage you um
[16:58] as uh platform administrators or just on the platform, um not to worry about the backlash, maybe. Um you have a standard, let's meet it, and then you'll find that the organization adopts it um pretty quickly.
[17:17] Also, it's worth investing if depending on the size of your org uh in a FinOps team or at least one or two engineers to kind of look into anomalies or spending on on the platform. All right, right compute for the workload. I like the way we did it here at
[17:32] FanDuel. Every team gets their own dedicated compute. Some organizations they share compute across different different teams and they're just like, "Hey, it's all in one one boat." We found that every team regardless of size having their own compute works well for us. Some teams use a lot and they
[17:47] really need to be isolated off. Some teams are relatively small, but that's perfectly fine. Just isolate them. The cost differences aren't it doesn't add up to much in anything anything significant. And then the the sizing of their compute based on their size and workloads you can be a little bit more
[18:02] fine-grained in what your what they get, right? And and because you have tagging you can also attribute it back to them and you get that return on investment in your um in your compute. Match the hardware. So, we generally use jobs clusters as
[18:18] opposed to serverless. We did a deep dive into all the different instance types towards the end of last year. So, compute compute memory and storage optimized and then also serverless as well and just looking into like what's best and what works best for what workload.
[18:35] Consider what your discounts are whatever it is on your account and you'll find whether it's on AWS side or on the Databricks side and I think you'll find what works best for you. Another recommendation is AWS Graviton. This is actually AWS's proprietary
[18:52] chip let's say or instance type that is fair amount cheaper. I think it's about 15% cheaper than all the other Intel and AMD. So, I would definitely recommend if you're using if you're not using if you're not using serverless, I recommend looking into
[19:08] using Graviton for your uh types. And it works well. I think the limitations are relatively small. I think it's like some GPS uh stuff that it doesn't work with. All right. Um so, we talked about uh we talked about compute. Let's talk a
[19:24] little bit about storage. Uh storage often gets uh put it in the back cuz it's relatively cheap. Um but just some easy things here that you can do uh to keep your storage cost down. Uh well-configured uh S3 life cycle uh policies uh take the time, take a sprint, maybe two,
[19:41] and decide how you're going to um how you're going to archive your data or what the tiers are for the archival of your data. And that'll be money well saved. Um another thing that we did was in the lower environments, um tables that have don't have any activity are actually
[19:56] deleted after 30 days. So, it's 30 days, it check the automation checks, and then it will actually change the name of the table so that person has some time to kind of like, "Oh hey, that my table disappeared." And then it'll actually be uh deleted after that. So,
[20:11] this is going to keep your uh dev and QA relatively clean. Uh prod, obviously you don't want to do this in prod. Um but in dev and QA, this will really keep it clean and keep your storage costs for, you know, stuff that you're really not even using. Uh liquid clustering, this is easy. Just make sure it's uh on
[20:27] pretty much everything unless you have a a very unique workload uh table that you're really trying to um customize. Uh next up for us, uh we're going to take a look at uh auto uh time to live and just see if that works for us. It's so I won't say it's newer, but it's not a new feature.
[20:43] Um and basically just putting uh retention policies on the at the record level and seeing if that's something that um works for us. It depends, you know, 10 years. You know, who knows what we'll be doing in 10 years, but uh maybe a year is enough for you guys and that might make a a significant difference.
[21:05] All right, takeaways. All right. So, where are we trying to get with this this framework, right? Uh the destination is self-service. So, ideally, and we're still on this journey ourselves, is to actually have teams basically come onto the platform and get the resources that they need without having to come to us as platform
[21:21] administrators, right? So, I'm a new team. Hey, I want to get all my compute. Uh and maybe I need some documentation as well and like what what I need to do, especially in the in the age of AI, I have a skill, I could just fill it out, submit the PR to the Terraform repo, and
[21:37] I'm good to go. And we could do that in less than a day, right? There's some permissioning and you know, onboarding and all that stuff, but that's the the destination that we're trying to get to. Um the paved road is basically everything that we've described today. It's set up, right? Every user that's onboarded onto the platform has a uh
[21:54] has a way forward and they know what to do on the platform without having to come back over and over and often it's complaints, but this is the way to to resolve that is to kind of get ahead of it and make sure that there's a a road forward or path forward. And platform as an enabler, I'm not sure about anybody in this room, but often myself and
[22:11] Prabget quite a bit of complaints or escalations about why they why a certain user can't do this or do that. Um we want to make sure that those are removed. It makes us look better, but it also um makes uh teams move faster on the platform and do what they need to do to to get business results.
[22:31] All right. Uh yeah, so just wrapping it up. Um just some things to take away. Uh governance before growth. Please do this before you migrate onto the platform or before you have a large uh kind of critical mass of users. Uh standardize the 80%. Um you don't have to standardize everything, right? But if
[22:47] you could get those 80% of workloads, I think you'll be in a good position. Uh make production boring. Everything as code. Just stick by that, right? And then compound the uh the returns, right? So, uh make efficiency the default, automate
[23:02] accountability, and enable self-service. All right. That's it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.