Sponsored By: Qubika | Building a Modern Data Foundation on Databricks: Lessons Learned From FanDuel

Summary

Qubika migrated FanDuel's core data workloads from Amazon Redshift to the Databricks Data and AI platform, overcoming petabyte-scale performance constraints, governance complexity, and the challenge of unifying batch and real-time streaming across more than 50 source systems.
Three major engineering challenges were addressed: reducing 20-minute engineer feedback loops to 5 minutes with notebook-based validation, rebuilding governance and metadata in Unity Catalog, and achieving zero data loss at Super Bowl demand spikes using AutoLoader, Photon, and CloudWatch triggers.
Qubika built an AI Migration and Governance Agent — powered by Lakebase as its persistent memory layer and Genie for querying progress — that automates DDL translation and Unity Catalog artifact generation, delivering 60–80% less manual effort and 2x faster domain onboarding.

Chapters

FAQs

Why did FanDuel migrate from Amazon Redshift to Databricks?

FanDuel faced performance and scalability constraints in Redshift that could not handle demand spikes from major sporting events like the Super Bowl, alongside a governance nightmare managing metadata through AWS Glue crawlers. The team also needed a unified platform that could handle both batch and streaming data and serve as a strong foundation for AI and analytics at petabyte scale.

What were the biggest engineering challenges in the FanDuel migration to Databricks?

Three challenges stood out: 20-minute feedback loops that slowed migration engineers (reduced to 5 minutes using notebook-based validation on all-purpose clusters), the complexity of rebuilding governance and metadata in Unity Catalog across dozens of source systems, and maintaining streaming reliability with zero data loss at Super Bowl scale using AutoLoader, Photon, and CloudWatch triggers.

How does Qubika's AI Migration and Governance Agent work?

The agent is built natively on the Databricks Data and AI platform and uses Lakebase as its persistent memory layer to manage long-running migration workflows and reason over complex schema dependencies. It automates governance translation tasks including DDL translation, Unity Catalog artifact generation, and Terraform output — work that previously required significant manual engineering effort.

What results did the FanDuel migration deliver?

The migration achieved 60–80% less manual effort and 2x faster domain onboarding compared to traditional migration approaches. The key lesson shared in this video is that governance is often the overlooked component of data migrations, and treating it as a first-class deliverable from the start — rather than an afterthought — is critical to long-term success.

Full transcript

[00:08] First of all, thank you all to be here. It's the third time in a row presenting on the Databricks Summit. It's a real pleasure for me to be in this place presenting about what we did on Kubica. For those that you know don't me, I'm
[00:21] Sebastian Diaz, SVP of data and AI on Kubica. I'm mostly in charge of all the offering and sales solutions on Databricks. And I have the great opportunity here to present what we do
[00:34] with FanDuel, uh one of our biggest clients on Kubica. What we did for them on the migration process and what we learned from there. So, first of all, I would love to start
[00:48] giving you some context. Uh I really like it to understand what was our situation when we started. So, as you could see, uh FanDuel it's a really fun uh uh business case. They
[01:01] have a petabyte petabyte uh uh data scale. They have more than 50 sources, real-time data, highly regulated environment because on gaming,
[01:14] obviously, you have to be having an audit trail to be able to execute it. Massive scales, so think about when you have the Super Bowl or big uh sports events, they have a crazy spike on the
[01:29] demand and data services need to keep going. They could not fall. And especially, uh they have consumers from everyone uh about analytics, data scientist, AI,
[01:43] everyone is consuming the data that we were migrating. And one of the biggest uh target that we have was the latency. We shouldn't be having worse latency,
[01:56] for example, in those moments. So, as it says, reliability at scale isn't a feature, it's a product itself on FanDuel.
[02:09] So, why they were migrating from other sources? What were the problem from other sources? For the meeting, I'm going to tell you what was the other source. It was Redshift. And they were having issues from performance issue,
[02:21] performance and scalability constraint, Redshift was not able to address all the needs, especially on these peaks of demand. Governance was crazy. I don't know if
[02:33] anyone tested, but in AWS, you have Glue crawlers working to make all the metadata and governance layer. It was a nightmare to maintain. It was difficult to unify the batch
[02:47] processing and the streaming data. And lastly, they wanted a strong foundation and AI analytics platform to keep working. So, this was the reason why they didn't they wanted to migrate
[03:01] to another place, and Databricks was the tool selected for it. So, as you can see, and and I always love this because it sounds so simple, right? I think if anyone work on data on
[03:13] working on migration, this is how it looks, right? You have your legacy data warehouse, you have your migration engine, and your medallion architecture.
[03:25] But in the reality, there was some issues that we discovered migrating that were having a lot of concern about it. So, the first one and one of the most difficult, it was challenge number one,
[03:39] the slow feedback loops. So, let's say that they were migrating, our data engineers were working on migrating, and they were discovering that each time they want to
[03:51] uh a schema change, they wanted to um be able to understand if what they were doing was okay or no. There was some constraints around the clustering, the
[04:03] providing of the service of the of the migration, the heavy CICD pipeline they were having, and the impact on the productivity if they mess out. So, it was around 20 minutes per
[04:15] iteration, so per testing, to actually check if what they were doing makes sense or not. So, think that they wanted to add a new column. It takes them 20 minutes to test if actually that is
[04:29] going to break things or not. And that was crazy, especially on a company like Fan Duel, they were moving a lot of data really fast. This was taking a lot of it. And most of the problem was provisioning
[04:43] of the cluster, Java execution, and CICD. So, one of the things that we did to able to solve that was accelerating and validating around Databricks.
[04:57] So, we established all-purpose clusters to be able to be able to always provide services when they needed to a rapid inner loop of feedback and testing. We do all the testing on notebook base.
[05:11] In that way, they were not needed to use third-party tools to validate their data scheme or data testing uh quality. So, they were able to do it with a notebook. Within the notebook. And also, most of
[05:25] the uh parameters were auto-generated from the DAGs. With all of that, they were able to define and validate in this in only 5 minutes. So, we were able to reduce from
[05:38] 20-minute feed feedback loops to only 5 minutes. This made that each time they needed to make a change, the whole process to actually evaluate and push it was much faster.
[05:52] The other challenge that we discover, and I always love this one because for me is a forgotten child on a migration, governance. Like when all of us think about moving
[06:04] from one platform to another, we think about, "Okay, I need to move the data and need to move the tables and that's it." Well, actually the governance is one of the biggest and
[06:16] important thing. And when you are moving from another workspace or another tool, sorry, into Databricks, you are going to have some difficulties. Why? Because the how the
[06:29] the governance and catalogs are managed by other tools, like for example in AWS, are really different with Unity Catalog. So, something that we needed to do to be able to work better on this migration
[06:42] was to solve the governance and metadata complexity to move from one place to another one. And the other was the validation and trust. This one was our biggest. As I
[06:54] mentioned, this was this could not fail. We could not could not move from AWS to Databricks and have fail on the on the data because they were needed to production.
[07:06] So, validating the information, have clear data quality uh rules to be able to establish that what I had on my legacy platform and what I have on Databricks it's the same,
[07:19] that I do didn't lose anything, and that I'm able to completely rely on what I have on Databricks. So, all these checks and especially especially the regulatory compliance checks were needed to be
[07:31] implemented and that one was a big challenge in an industry like uh Fanduel. This one was another interesting fact. Do you remember when I mentioned about
[07:45] the peak, for example, in a Super Bowl? In those cases where we're receiving a streaming data, the amount of information that we receive and the aggregations and process
[07:57] that we we were doing, we have no room to lose data down that process. And especially on a big demand, it was crazy. So, what we end up doing, we end up doing an streaming merge optimization
[08:11] combining autoloader, predicate pruning, file compression, and trigger tuning. We were able to save all the information in those time without augmenting the
[08:24] the pricing and the billing on those moments and without losing information. And how we do that? So, mainly what we did it was all the information that was a streaming was landing on S3 bucket. We
[08:36] were having triggers built on CloudWatch that when the data lands, there was a trigger in Databricks where that they ingest only the information that just land. In that way, Databricks didn't
[08:49] have the necessity to read the whole catalog of S3 to actually detect what he should be getting. In that way, we were able to to consume that information without
[09:01] increasing heavily the cost. When the data lands on Databricks, what we end up doing was doing a microbatch. And after that, merging the information in windows and
[09:15] validating on the moment, so we are were able to compare that we were not losing data. With all of that and leveraging on the Photon engine, we were able to grant that demand and SLA
[09:28] for the time where Super Bowl or NBA finals or and so on. So, with all of that, what we get? Obviously, it was an amazing road with
[09:41] Fanjol doing the migration. We learned many things and mostly we build accelerators because what we say is this is going to happen again. So,
[09:53] if we are able to identify the patterns, the recurring issues in all of this, right? It was all what we mentioned, the scaling constraint, and we did take that there was an
[10:05] opportunity to create an agent because this pattern repeat across multiple companies. And why we made an agent that was able to help us on that
[10:17] uh governance and migration process? Because the agent was able to be flexible enough to adapt in any migration that we have. Because yes, they are repeated patterns,
[10:31] but it each company, each industry has their own caveat that we need to be able to adapt. So, the agent that we build to help us to migrate, it was able to, first of all, take the governance from the old
[10:45] platform and translate it to Unity Catalog. This one was a big help. We didn't have the the the obligation of having data data experts, Redshift experts to actually do that migration. We were able
[10:59] to do it with Databricks experts that leverage on AI to able to translate it to our Unity Catalog language. We were able to generate on real time validation and reports that they were
[11:11] checking that what we are migrating makes sense or not. We were able to generate artifacts like, for example, Terraform or DABS. So, let's say that I'm going to put you a scenario. I want to migrate a whole
[11:24] domain from Redshift to Databricks, and from that domain, I want to be able to identify how is the schema, how is the catalog, how is the role users, and so on. With all of these, you are able
[11:37] to put all that information into the agent, and he's going to generate you the Terraform and the Databricks assets bundles to reproduce all that into Databricks.
[11:49] And because we also leverage on Lake Base to build this agent, we were able to save the state. And why we did this? So, in a highly regulated industry like Fanduel, we were on need of actually
[12:03] identifying why we take each one of the decisions. Why I'm not want to migrating in this way or in another one, and we save all that into Lake Base, and in that way we were able to take the
[12:15] opportunity, take that information, query with using Genie, and understand what is the actual uh progress on the on the migration.
[12:30] This is how it looks, the architecture of the agent. As you can see, you have your legacy, Redshift, Snowflake, whatever you want. We use connectors that mainly uh do a crawler on the metadata and normalize
[12:42] the DDL information. And after that, the agent is in charge of translating and validating. And all the things that he does, he save it on the Lake Base.
[12:54] And after that, he produce all the assets into Unity Catalog assets. So, it was a transformation built by an agent. This is the architecture. You are going to find many
[13:08] uh common resource on this type of solutions. So, we have uh the Lakehouse platform things that we mentioned. We have our legacy warehouse. We have the Unity Catalog assets that we generate.
[13:21] We have a Databricks apps to have a front end for this agent. We have the MLflow to be able to serve the models, but also to evaluate how things are coming. And we have data brick layers to
[13:33] serve as a memory management chart for the chatbots and also for the state. Look at this. This is how we increased after Fanduel, how we were able to produce much better. So, mostly we
[13:47] reduced the manual effort from 60 to 80% after Fanduel. We were able to onboard new domains two times faster than before having the AI agents.
[13:59] We were able to lower the migration risk. And also, we were able to maintain consistency across the whole that migration the domains. So, before this accelerator, it take us
[14:12] 120 hour a month to migrate a domain. After this, it's only takes a one week. This reduce a lot the defects on the governance layer. And for me, without a
[14:26] good governance, you are going to tend to fail in a migration. Doesn't matter that the data is in that in data bricks, if you don't have a good governance layer, you are going to fail on it.
[14:39] Key takeaways that we take from here. In a migration, complexity is underestimated. Especially for the governance of domains. There's also many business
[14:53] models and patch inside the the legacy data platform that you tend to forget about it. You need to shrink the inner loop. Many times, the migration takes time
[15:07] because the feedback loop is so big that the teams take so much time to able to validate of if what they are doing makes sense or not. And what they end up doing, they forget about the feedback
[15:19] loop and they commit errors. So, that's the reason that you need to be able to shrink the inner loop. Asians helps a lot on this. Why they help? Because migration has repetitive
[15:31] patterns. And you are able to use Asians to helps and support on that. And lastly, Databricks is an amazing tool for foundations. Because not only you are able to
[15:43] create and move the data there, you are able to have the same Asians that are helping you on that migration. With all of this, I would love to thank you. I would love to
[15:55] invite you to our booth. We are on the 514, I think, or 541. 514. Well, we are able to talk more about
[16:07] this. We build more things around this. For example, right now, we have a whole AI dev kit to migrate from one place to another using Cloud. And it's really interesting, especially if you are in the middle of migration or you are doing
[16:21] governance layer schema change, this kit is going to help you a lot to accelerate it. We are reducing 75% on development time using Cloud working with Databricks. And
[16:35] also, we are able to reduce cost, optimize Databricks applications, and lake based databases. So, with all of this, thank you again for giving me this opportunity and have a nice event. Thank
[16:47] you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.