200 Spark Jobs to 3: Migrating a Governed Lakehouse with Spark Declarative Pipelines

Summary

Flutter SEA migrated 1.5 terabytes of daily streaming data from Oracle Cloud to Databricks, consolidating 200 Spark jobs into 3 Spark Declarative Pipelines and achieving a 40% cost reduction with 96% bronze-to-silver performance improvement.
A Triforce team model combining Flutter engineering, Databricks Professional Services, and Reply enabled rapid delivery while managing migration risk for a live 24/7 platform serving Sisal, Snai, and PokerStars.
The use of dependency graphs and connected components to model pipeline relationships, combined with liquid clustering and Photon optimization, resolved a production SLA crisis that emerged during migration.

200 Spark Jobs to 3: Migrating a Governed Lakehouse with Spark Declarative Pipelines

Migrating 1.5 terabytes of daily streaming data from Oracle Cloud to Databricks is complex, but Flutter SEA accomplished it while maintaining 24/7 reliability for three major betting brands: Sisal, Snai, and PokerStars. this video covers the end-to-end migration journey, including architectural decision-making, team structure, and the critical production moments that tested the system. Hear about achieving 40% cost reduction, 96% bronze-to-silver performance improvements, and consolidating 200 Spark jobs into just 3 Spark Declarative Pipelines.
Discover how a Triforce team model combining Flutter engineering, Databricks Professional Services, and Reply enabled rapid delivery while managing risk. Learn architectural choices like singleplex vs multiplex pipelines, the breakthrough use of dependency graphs and connected components, and how liquid clustering and Photon optimization resolved production SLA misses. Real-time use cases including recommender systems, smart search with semantic tolerance, churn prediction, and fraud detection now run on unified governance with Unity Catalog and Delta Lake.

Chapters

FAQs

Why did Flutter SEA migrate from Oracle Cloud to Databricks if the platform was already working?

Although the Oracle Cloud platform functioned reliably, it was slowly becoming a strategic constraint. The legacy architecture lacked the flexibility and ecosystem to support new AI use cases like recommender systems, smart search, churn prediction, and fraud detection that the business needed to build next.

What is the Triforce team model Flutter SEA used for the migration?

The Triforce model combined three parties: Flutter SEA's internal engineering team providing domain and platform knowledge, Databricks Professional Services providing migration expertise and architecture guidance, and Reply as a systems integrator handling delivery execution. This structure enabled rapid delivery while distributing risk appropriately.

How did Flutter SEA consolidate 200 Spark jobs into 3 Spark Declarative Pipelines?

The team used dependency graphs and a connected components algorithm to model the relationships between the 200 existing jobs and identify logical groupings. This analysis revealed how individual jobs could be collapsed into three coherent Spark Declarative Pipelines without losing the data flow logic.

What happened during Flutter SEA's production SLA crisis and how was it resolved?

After migration, the silver layer began missing SLAs, triggering a cross-team war room response. The root cause was resolved through a combination of liquid clustering to optimize data layout and Photon optimization to accelerate query execution, restoring SLA compliance without requiring architectural changes.

Full transcript

[00:07] Hi everyone. Awesome. So, let me just reiterate that this topic is not about AI agents, agent agents orchestrating other agents or agents orchestrating humans.
[00:22] So, first of all, thank you so much everybody for working in in the wrong room and being too polite to leave. This is a story of something that worked. Flutter SAE had a real-time streaming
[00:39] platform implemented in in Oracle Cloud and it worked just fine. So, why would you rip off something that worked? Please hold on this question because in the next 30-ish minutes, we will try to answer. Together with me,
[00:55] there is Arianna, our senior engineer in in tech leading the reply that helped us to do this successful delivery. And Andrea Santurbano, a former deployed engineer in Databricks that helped us with the migration.
[01:11] This is a story of 1.5 TB per day, 200 Spark jobs that became three and a very tense week in production in production. Let's dive in. Okay, so here's what we're going to talk about in the next half hour. First, I'll
[01:27] introduce you to Flutter SCA, the brands, the players, the scale, because none of these make sense if you don't actually understand where the player is coming from. Then we have the old platform, the one on Oracle Cloud, because we worked on the platform for
[01:42] years, so we'll tell you exactly what it was like, what worked, uh and was low what was slowly bringing us down. Then I'll show you free products on the new platform, the lakehouse on Databricks. Finally, Andrea will introduce you to
[01:57] the team model that made the migration possible, the actual migration itself, including the week where everything almost went wrong. Finally, I'll come back to you to show you what's next for Flutter SCA. So, let's start with Flutter.
[02:14] Before every pipeline, before every architectural diagrams, we have to talk about the business. Because behind every technical decision, you have to understand what does a betting platform need from its data. So, who is Flutter SCA? Flutter SCA,
[02:31] Southern Europe and Africa, is the regional business unit of Flutter Entertainment. If the name doesn't ring a bell, maybe the the brands will, because it is a group behind Fanduel, Paddy Power. But in Southern Europe, it runs three important brands: Sisal, Snai, and
[02:47] PokerStars. Sisal was founded in 1945. It is one of the market leaders in Italy's for lotteries, sports betting, online gaming. It has thousands of physical shops all around Italy and also
[03:02] a digital platform. So, every ticket sold in one of those shops is an event coming into our platform. Then we have Snai. Snai it's one of the most important betting brands in Italy. Its history runs in sports, especially horse racing.
[03:18] It has a lot of shops, but also now a strong push into the digital. And finally, PokerStars, the one we all know, I think. The onli- the original online poker room, the most famous poker brand in the world with a strong global identity and also a
[03:34] loyal player base. But what all of these have in common? So, for example, a lottery ticket in a in a sold in a tobacco shop in Naples, a live odds change on a horse race, a poker hand played from another continent. All of these is coming into
[03:51] the same platform in streaming. Uh brands they still live together because each brand has its own identity, its own customers, its own products. All of them share the same technology,
[04:07] the same services, and the same data model with four principles. Platform first, meaning the the platform is built once and then reused for all the play brands, sorry. Then we have brand led. Platform serves
[04:24] the brand, not the other way around. Then data and AI driven, meaning insight-based decision making, not instinct. And finally, governed by design. Security, compliance, control built in from the start, not patched on
[04:40] later. And remember this last one, governed by design, it is the one promise the old platform could never keep. So, spoiler alert. I want to show you these numbers first because they are
[04:55] totally worth it. 40% of this year reduction compared to the old OCI platform. 96% performance boost from the bronze to silver layer.
[05:11] And the last one is my favorite. 200 Spark jobs to just three Spark declarative pipelines. Let me reiterate it. 200 to three. But also, what is at the bottom of the
[05:27] slide is what makes a team dream uh have good dreams during the night, sleeps well. Observability, we created dashboards in order to basically monitor everything, cost, uh
[05:42] cluster utilization, uh and alerts when SLAs are not covered. Stability, no more all manual Oracle restarts and auto scaling provided out of the box. Then, thanks to Reply, they
[05:57] built a standard framework that where all the Flutter data products live. And in the end, it scaled. So, the same offset were used from Sisal to Znai,
[06:14] PokerStars, and they plan to extend to more other brands. Okay, so let's go back to the old platform, the one on Oracle Cloud. Because to understand why we moved, you have to first understand what we were moving away from.
[06:30] So, this is the old platform, the one on Oracle Cloud, and to be honest, it worked. Each one of these boxes did its job. Because this is not a story about a broken platform, it's a story about a platform with its ceiling. So, look at the boxes. Ingestion,
[06:45] custom-built flows on OCI data flow. The governance, built but without any catalog or lineage in place. If you wanted to know where a column came from, you have to ask the person who built the pipeline. Observability, again, custom because
[07:02] there weren't any tools that supported streaming tables. Dashboarding, a super set layer glued on on top of our OCI platform. The AI use cases were developed inside OCI data science and
[07:17] then integrated into the pipelines by end. And finally, the one that ties it all together, manual configuration. Because all the components were configured by us to talk to each other component. There's one word that keeps repeating in this slide, and it's custom. Custom
[07:34] means powerful, but custom also means a tax. You pay it every day, on every change, on every new hire, and on every incident at 2:00 a.m. Uh we have three important things about this platform. The volume, it's 1.5 terabytes of data
[07:52] per day streaming in. And the platform could support it. It's just that for each new use case, it was a negotiation with the architecture. The bottleneck, the one we lived every day. Everything was relying on custom
[08:08] code, workaround. So, onboarding a new hire took months because the knowledge wasn't in any catalog or any standard, it was in people's heads. The debugging was painful, and innovation, it went to the back of the queue behind keeping the lights on.
[08:25] And finally, fragmented ownership. Two very important products, the recommender and the smart search, were externally owned. So, Flattr SIT team didn't own them. When something broke, it wasn't clear who owned the
[08:40] fix. So, at the end, it was a platform that worked, but it was at its limit. So, the goals were clear and bigger than the platform could handle. They were to reduce costs, remove the friction,
[08:56] take the latency from minutes to seconds, bring the AI use cases in house, and finally stop accumulating technical debts. Now, before I show you the new architecture, the one on Databricks and
[09:11] the Lakehouse, I need to talk about three products in production right now. The first one, the recommender platform, and the wording is deliberate. This is not just a recommender, it's a recommendation platform. It's one of the old originally owned
[09:29] externally owned product, sorry, and now Flattr SIT teams owns the model, it owns the optimization logic, the algorithm, and the decision rules. The recommendations are optimized for GGR, gross gaming revenue, with brand
[09:45] customizations on top of it. Uh so, what does it do? It tells a player what to play next. So, continue where you left off. You like this category? Try this game. So, personalized player recommendation
[10:01] based on what the players are actually playing. And the architecture is simple. Databricks computes the players' features with streaming jobs and the clarity pipelines. The serving layer is updated in real time, so the recommendations are fresh, not as of
[10:17] last night's batch. And the most important part, everything is reproducible. So, the architecture is done once for all the brands, Sisal, Snai, PokerStars. So, build one and then deployed three times with only the
[10:33] customizations on top of it. The next one, the smart search, and the tagline says it all. So, find the right bet with the wrong keyword. Because smart search on uh on sports betting is brutally hard
[10:48] because the catalog is live, so the events appear, the odds change, matches end. Players type fast on their phones with slang or typos. So, Juve, for example, has to find Juventus. Or a misspelled player name
[11:04] has to find this right bet match. But the smart search handles it all with uh tolerance to the typos and to partial matches, with the also semantic search on sports-related synonyms. Everything
[11:19] is fully integrated into the betting platform, and the response times are under 2 seconds. Again, it's a simple architecture. With Databricks streaming jobs, the indexes in OpenSearch are in real time. So, the events appear and
[11:36] the odds change and you can see them in the platform almost in real time as soon as they happen. Um so, it's a fault-tolerant search on live data and it's in production right now. It's serving players right now.
[11:51] And finally, the one that ties it all together, yes, the platform serves the business users. We have real-time dashboards, we have SQL Python notebooks, immediate feedback for explorations and operations. And these alone would have justified the
[12:06] migration. But what the most important part is what runs on top of the data. So, responsible gaming for the monitoring the player behavior. And because in this industry, this is not a feature, it's a duty.
[12:21] Then we have churn prediction. Then we have next best model. And also fraud detection, obviously. And the live decision support for the traders, the one that are in charge of adjusting the odds, for example.
[12:38] In fact, with the jobs and with Spark declarative pipelines, the events, but not raw events, AI-generated features flows into the marketing system and then through ADI endpoints, they flow directly into the the back office tools.
[12:54] So, the recommendation for the traders are available for them in real time. Um Uh so, remember the govern-by-design? This is what buys you. Because you have the data all in one place.
[13:11] Uh it is governed, it has data quality inside, so AI stops being only a use case. It is a production line. But now, how did we migrate 1.5 terabytes of data per day without disrupting the business? So, let me hand
[13:28] it over to Andrea. Thank you, Ariana. So, when our customer moves to Databricks, they are not just buying software. They are buying a promise that it will work. So, for the next few minutes,
[13:45] let me pull back the curtain behind how the engagement model worked. First, what is FD? It's basically what was before the Databricks Professional Services. Now, we are
[14:00] forward deployed engineer and is a set of three different skills with one shared outcome. Oops, sorry. The customer success. First of all, we have the forward deployed engineer. He's the builder, the
[14:16] one that sits with the customer and and ends for him high-priority business workflows, transforming them, implementing them into MVPs. Then, we also risk the delivery with hands-on architecture and prototyping
[14:33] and technical execution. We just We don't do slides. We actually deliver products. Then, the delivery solution architect, the embedded technical advisor who is who sits with the customer and is there for long-term operation. He's the one
[14:50] that basically inherits what we built once we left and uh he makes sure that what we built continues to scale, to work and scale. Third, the technical project manager, our single point of contact windows
[15:06] workstreams, does escalation for risk, uh highlights risk, does escalation, and translate engineering reality into leadership language in order to report that at QBR.
[15:24] Now, look at the bottom of the slide. So, this is the honest part. What we are we are experts, trusted advisor, and force multipliers. And we also do delivery assurance, like in this case.
[15:41] What we are not, we are not free, but there are of course investment opportunities from Databricks. Uh we are not an system integrator or a partner replacement. We work as in this successful engagement with a Reply. We
[15:57] work with the partner in order to ensure that there is a customer success. We are not staffed with augmentation and back box builders because everything we deliver, we deliver for the customer.
[16:13] Now, this is what the success looked like in this engagement. We had Flutter and we call it a a Triforce. It's a reference to the Legend of Zelda video game. And we have Flutter. It's the business
[16:30] domain ownership and is that engineering team drove the delivery. They own also the production accountability and they are the design authority on the platform direction. Then there was us, Databricks.
[16:47] Uh me as a senior forward deployed engineer, our delivery solution architect, and our solution architect. And last but not least, we had Reply, our implementation partner. Uh they are an so engineering capacity
[17:02] and the knowledge is co-built with us and they are there for long-term operations. Now, let's talk about what the team actually delivered. This is the act of the talk. This is the
[17:18] customer story. So, as we said the older implementation, the legacy implementation in uh OCI worked. And let me tell you why they they did they they basically
[17:34] found out that the platform was not was costly and was hard to scale. So, that's that's was the main decision about why they decided to migrate the uh the database to Databricks. This is not a sanitized story because it's the real one and
[17:52] we will discuss the moment where everything was starting to fall apart and we how we fix it. So, every migration start with a big decision with a with a fork. Uh you have to decide how to approach the a
[18:08] migration, okay? And you have two option. We have these two option that we put on the table uh to flatter and we decided together. On the oops, sorry.
[18:24] On the left-hand side, you have the front to back. So, you start migrating the layers that are most close to the business. So, this means that you have an early business value because you can onboard you can start onboarding business on the platform very fast.
[18:40] And this of course encourage uh adoption. But on the other hand on the other hand, there is a complex backend uh clean up later uh because uh you probably in this
[18:55] approach you probably use lift and shift. So, you have then to rework uh the things that you uh imported that you migrated. And the interfaces between the new and old system may need temporary workarounds.
[19:10] Then on the right-hand side, there is the back to front the back to front. It allows you to build strong frame foundation, so adopt all the modern technologies that are available in Databricks.
[19:25] And it needs, of course, less rework after you did the migration. On the cons, you know, uh it's basically uh uh has lower visible results to the business because you need to rebuild everything from scratch, so from the
[19:42] bronze to the gold layer, and it has, of course, longer time to value. Now, how do you plan this kind of migration? And this is where it gets engineering beautiful.
[19:58] Thanks to the great work that Reply did, uh they collected all the uh dependencies uh between uh Kafka topics, silver, bronze, silver, and gold tables, and we build this dependency graph.
[20:15] Then, we computed uh a very simple graph algorithm, uh the uh weekly connected components, that is for isolating isolating uh uh the subgraphs in uh in a bigger graph. And each we identified 37 37 uh contained self
[20:35] uh self-contained units, and each one became an independent uh uh uh migration uh deliverable with clear ownership and it enabled also uh uh parallel execution, so we can we had basically assigned different subgraphs
[20:50] to different teams. And this replace a single monolith uh uh migration with a portfolio of small shippable items delivering quick wins.
[21:06] So, this is basically the difference uh between a migration that you fear and a migration that you can manage. Then, we had another choice to do, and we also discussed with the customer what fits best for him for them.
[21:27] On the left-hand side, we have singleplex, which means that for each Kafka topic, you have one single bronze table, and it has strong isolation and per stream customization. It's easier to debug because if something fails, you know to which pipeline I have to look
[21:45] into. And it's best when your topics have different SLAs or compliance need. On the other hand, when you need to scale it, as you saw,
[22:00] they had like 200 Spark jobs. Not of all of them were Kafka topics, but they were Kafka topics, of course. We had like 80 Kafka topics or something. On the other end side, we had the
[22:18] multiplex architecture. Basically, you have multiple topics that ingest just one bronze table, and it's basically
[22:35] simpler because it simplifies your architecture because if you need to add new more sources, you don't need to change your code. And it's best when you have different topics that have similar semantics or similar SLA.
[22:52] In our particular case, we opted for multiplex with a a slightly different variant. It's called staged with stage two bronze table. This means that we added here another layer of
[23:07] bronze table after this one that basically mimics the silver tables and then we use this bronze table to ingest the silver ones. Now, yeah, I want to be honest with you.
[23:25] The The time migration uh it's, you know, it's not the right one The right one was more like an end-to-end platform uplift uh because we covered several topics, foundations, migration and pipelines, and quality and
[23:41] operation. For the foundation, we supported Reply and Flutter on the workspace design from the security network perspective. Uh you see we designed the the catalogs and the
[23:57] storage for performance. Also, we provide uh the governance models uh via ABAC or RBAC and also define the cluster policies, roles, and responsibility in order to govern uh the infrastructure. Then the migration pipelines,
[24:14] uh we discuss all of that. We also We also found when we uh when I was reversing engineering the old code, I found uh um a small bug uh in the uh
[24:29] in the Oracle GoldenGate uh partial update in Delta merge. So, there was some data that that was coming from GoldenGate in the the Kafka topic and the Delta merge was uh handling it not properly.
[24:45] And last but not least, uh quality and operations. So, we discussed about how to handle data quality with expectation but also with the quarantine pattern. Uh how to implement Databricks asset bundles,
[25:01] we built dashboard for implementing job observability, SDP observability and also FinOps for managing the cost of every single spark declarative pipelines. So now I I don't want
[25:16] to go through all of this kind of thing. Instead, let me zoom in into one problem that basically cost us our SLAs.
[25:32] So, every honest migration has a moment where things go wrong and this is ours. So, we did some load test of course in pre-production and everything was good. But then we put the pipeline in
[25:47] production and we observed that the silver layer was missing the 5 minutes SLA. It was like something like 50 minutes to fill the silver layer. So looking at workloads, we observed
[26:04] that auto liquid clustering was not persisting effective keys aligned to the match predicates and dynamic file pruning was not kicking in in the right parts. So basically the merge operation was rewriting more than
[26:20] needed. More files than needed. And what we did is to build a cross team war room. So with us Databricks, we brought in the product
[26:36] engineering. So the people who built liquid clustering, the query optimizer and photo optimizer as well. We also did clearly Sorry. Daily syncs with clear owners per
[26:51] work stream and then we did the fixes. So, we forced the broadcast hash join where appropriate. So, we increased the broadcast hash join sides in order to handle more more data.
[27:09] And this reduced the SLAs to from 50 to was 15, right? Yeah. So, and uh it was not enough. So, we decided we decided that to enable Photon,
[27:24] which enabled the dynamic file pruning. And we also resized the Photon cluster to a very minimal cluster, one driver and just one worker. And then it worked. And we were able to
[27:40] prove that it worked because of the observability dashboard that we built before. So, now I want to pause here a moment because I want to name names because behind, you know,
[27:56] every single architecture diagram or Spark declarative pipeline implementation, there are actually humans that stayed late at the night. So, thank you so much to Francesco and his team. Francesco was supposed to be here on the stage, but he had personal
[28:13] issues and he could come in. Thank you so much to Matt, Sam, Matthew, and Bart. And of course, thank you so much to Arianna and the Reply team to make this delivery successful. Now,
[28:28] this is not an error. I want to reiterate. I want you to leave with three big messages. So, what the customer get is a huge TCO of
[28:47] in terms of cost and uh also simplified by a lot the architecture migrating these 200 Spark jobs to just three Spark declarative pipelines. And our FD supported the uh uh
[29:02] the successful delivered uh uh done by Reply. Now, let me hand over to Arianna for the conclusion. Okay. So, you've heard the whole story. The decisions, the migration, the crisis, and also the comeback. Now, let
[29:20] me close with a part I find the most telling because uh um we need to understand what Flutter is doing next. So, the best evidence that our migration is successful is not only the cost reduction, but it's also what is available after the migration.
[29:37] The first thing, scale to other brands in other countries. The same assets, the same platform will be extended in the future uh to other brands and other countries because the architecture was made exactly for this. Then we have a new Databricks upcoming,
[29:52] a new app for search of the betting tickets. It's currently in development and is targeted for Q1 2027. It's a user-facing app directly inside the lakehouse. Then we have new AI use cases inside the same platform because
[30:07] the cost of new AI use cases is just a fraction of the first use case. And finally, the symbolic one, an assessment to decommission the data warehouse entirely consolidating everything into Databricks because the old platform is not just
[30:23] being left behind, it's being switched off completely. So, that's it. And if you have any question, we're happy to take them.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.