Redesigning Kaizen's Cloud Data Lake for the Future

Summary

• Data Landing Zones architecture: This video showcases how Kaizen Gaming redesigned its cloud data lake with a modular Data Landing Zones framework that decouples resources and enables independent scaling and cost accountability.
• Cost transparency and FinOps: Learn how automation, Delta Sharing, and DBx Asset Bundles simplified infrastructure migration while enhancing cost visibility and enabling accurate cost allocation across teams and markets.
• Large-scale data operations: Discover how a redesigned architecture handles 1PB+ of data with 350M daily transactions, enabling faster insights and reduced onboarding time for new analytics initiatives.

Redesigning Kaizen's Cloud Data Lake for the Future

Talk by Nikolaos Michail, Data Platform Team Lead, Kaizen Gaming ; Triantafyllos Tsakmakis, Senior SRE/DevOps Engineer, Kaizen Gaming
At Kaizen Gaming, data drives our decision-making, but rapid growth exposed inefficiencies in our legacy cloud setup — escalating costs, delayed insights and scalability limits. Operating in 18 countries with 350M daily transactions (1PB+), shared quotas and limited cost transparency hindered efficiency. To address this, we redesigned our cloud architecture with Data Landing Zones, a modular framework that decouples resources, enabling independent scaling and cost accountability. Automation streamlined infrastructure, reduced overhead and enhanced FinOps visibility, while Unity Catalog ensured governance and security. Migration challenges included maintaining stability, managing costs and minimizing latency. A phased approach, Delta Sharing, and DBx Asset Bundles simplified transitions. The result: faster insights, improved cost control and reduced onboarding time, fostering innovation and efficiency. We share our transformation, offering insights for modern cloud optimization.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What prompted Kaizen Gaming to redesign its data architecture?

Rapid growth exposed inefficiencies in their legacy cloud setup: escalating costs, delayed insights, scalability limits, shared quotas, and limited cost transparency hindered efficiency for a company with 18 markets and 1PB+ of data.

How does the Data Landing Zones framework solve these problems?

Data Landing Zones is a modular framework that decouples resources, enabling independent scaling and cost accountability. This approach reduced overhead, enhanced FinOps visibility, and improved efficiency across the organization.

What role did Unity Catalog play in this transformation?

This video shows how Unity Catalog ensured governance and security throughout the migration, providing unified access control and compliance across the new architecture while maintaining data quality standards.

What scale does Kaizen Gaming's data platform now support?

The redesigned architecture successfully handles 1PB+ of stored data with 350M daily transactions, over 400,000 transactions per minute during peak hours, supporting 13 million active users across 18 markets.

Full transcript

Welcome everybody. I'm M here ismatis and today our topic is uh redesigning cloud data lake for the future. So some introductions uh I'm the guy on the left. I'm the data platform team lead at Kaisen Gaming and I'm an SR DevOps engineer at Kaizen Gaming working with a data team. So uh our agenda for today uh first of all some introduction about Kaisen gaming uh about data in particular about the data platform and challenges that we face and a lot of the challenges uh stem from our architecture.
Uh so we're going to see our old architecture what problems this caused. Uh we will have some light theory here about cloud hopefully you don't get bored and uh how we propose the new architecture to solve this uh challenges. And the last part is about the migration process. So uh new architecture means always a migration. So what how we approach this uh project to distinct phases, how to migrate our infrastructure, our code and our data and some conclusions and lesson learned.
So first things first uh Kaisen gaming we're a game tech company. Uh we operate in online gaming and sports betting. Uh and currently we are over uh 18 markets uh globally on the Batano brand. Um and the goal is to be in 26 markets over 2026. Uh we have 13 million active users. Uh which uh and in our site we we serve over one and a half million gaming events per year. So this means a lot of uh business data generated day.
So and this data end up in our data lake. Uh so currently in our data lake we have over two pabytes of information. Uh we have a lakehouse architecture. So uh the storage part is in Azure data in Microsoft Azour and for the compute we use mostly data bricks and some other metrics for our velocity. Uh daily we have over 400 million transactions across all markets and on big service hour uh we have over 400,000 uh transactions per minute.
Uh also what's impressive here uh it's the scale that uh the rate of the growth. So in just 10 years we went from 27 uh million transactions per year to 121 billion. Uh you can see this trend that's almost exponential and we expect this trend to continue in the future. Uh so scalability issues are really important for us. So uh a lot of people also work uh in the data department. Uh so currently uh the top level management views uh being data driven as a core strategic capability.
It's labeled under data DNA and we have over 150 people uh working on data BI and AI teams in various roles and those people serve uh over 85 departments internally and uh the footprint inside the company is over 10,000 uh end users internally and to tackle this complexity we use some ideas from data mess. So we have uh teams focused on specific domains uh business domains and we have uh these teams to deliver value directly to stakeholders and the platform teams support this team uh by providing tools and capabilities.
So what are the challenges for the data platform? Uh okay, you shouldn't see this. Uh first of all, our architecture spans a lot of years. Uh so um some decisions that were made in earlier um in the past were done with a different intention. So it was most important for us to deliver value to stakeholders quickly uh versus optimizing. Uh and this meant that accumulating we accumulated a lot of technical debt. Um so some decisions that we made in the past uh was good enough up until now but do not fit our scale or our ambition anymore.
So what are these decisions? So uh we have uh some shared resources like uh in our old architecture uh storage is a shared resource. So this means that different teams uh different projects um different environments even uh compete for the same quotas and the same limitations. Um so this problematic also you can't have true environment isolation and having your your dev your your stage and your production. It's difficult to have software development life cycle uh principles. uh also it's difficult to attribute costs between projects and to understand where your course are coming and governance of course is uh is trickier when you have uh shared resources.
Another pain point for us uh was everything was set up manually through Azour portal. Uh so this meant change management was difficult and also it was errorrone. So uh we had an opportunity to combat this uh technical debt. Um a an initiative was launched across tech department to migrate to a single uh cloud account and this was our opportunity for us to focus on the problems that really matter for us. Uh so first of all scalability uh we want a data platform that can scale uh for an increasing volume of data and increasing people working on the same data without any performance degradation.
Uh also security is important for us to uh to have a strong governance model to be u compliant with regulations etc. And the last parts are all of course about money. So we want to be able to attribute cost to specific teams to specific projects and use this information to be efficient to eliminate waste in our process in our pipelines and to uh our eventual goal is to reduce operational expenses in cloud. Under the hood, we had some other uh goals as well.
Uh we want to establish some domain boundaries. So uh each team should be able to eventually work uh independently inside their domain. So uh not having coupling between teams. And the ultimate goal um in the future is data democization. So this is the notion that business can have access to uh quality timely information um without any barriers. So, so the will guide us uh through the cloud part. Thank you. Uh so let's have a look at our uh infrastructure about data bricks.
Uh but first of all, why datab bricks? Uh no, can you hear us? Can you hear us? It's okay now. No, no, it's not. Yes, it is. The words are painted there. So, uh why datab bricks? Well, being in a datab bricks conference, I will not elaborate on that one. But, uh where can we find datab bricks? Where is datab bricks hosted? Datab bricks is provided as a service on all big uh three cloud providers AWS, Azure, GCP. and selecting one or the other does not really affect the actual uh implementation.
Uh but uh where does one start if uh he wants to build infrastructure in the cloud and framework is always helpful in such cases and in this case cloud adoption framework provides documentation and tooling to help you build a reference cloud architecture. Our provider of choice was Azure and their implementation of this framework is based on terraform the factor infrastructure as code tool. The key element in all cases of uh uh this framework deployment are landing zones which are containers for uh cloud resources. of interest in our case are the data landing zones which host the workspaces and the data themselves as well as the data management landing zones which host the unity catalog and metas store resources.
Uh in Azure cloud this uh uh landing zone landing zone is implemented as a subscription but uh what exactly is a subscription? Fear not because some definitions are coming. Uh so these are some terminology and hierarchy on the uh cloud of Azure. Uh first of all we have a tenant which is an identity and access management service. Then we have subscriptions which are used to isolate quotas and billing of the resources deployed.
Then we have uh resource groups which are uh groups of logically related cloud assets. And then we have the resources themselves like workspaces, stoages, virtual machines, networks and so on. But since we are talking about a cloud redesign, we need to know from where we started. So as mentioned earlier, we had a single asure tenant with a a common SH subscription for all our resources and this has caused a quot overlapping issues in multiple occasions.
Uh also we had a a single storage with a common configuration for all our data regardless of their type and peculiarities and as also mentioned adding insult to injury most of our resources uh were deployed manually through the ASU portal itself. So for all these reasons, PHOPS like cost transparency and uh cost attribution became very difficult as well as scaling. So taking a step ahead we would u uh implement a new architecture but we had to decide to decide about many things.
Uh we would implement the cloud adoption framework mentioned earlier. So we would create various landing zones that were used to separate business lines of business departments and environments. Uh this meant that we would create multiple workspaces but uh we needed some capacity planning so that because there is a a soft limit set by datab bricks on the uh number of workspaces in each account which is about 50 and we need to be careful not not to cross that limit.
Also we need to uh take care about the sizing of the networks in each case because too small networks meant that we would not be able to spin up enough cluster nodes for the clusters uh but for the databicks workspaces but too large networks meant that we would waste a lot of IP addresses something that is really important in the very big organizations or very big setups. Uh also we we should take care of the synergy of resources and uh uh review the way that we would break up uh the resources the work the workspaces the uh vaults the stoages to the various teams and environments.
And also having uh witnessed uh the expanding costs of uh having a single storage uh we would create multiple stoages with uh different configurations for redundancy and replication depending on the data they would host in each case. And we needed to take into consideration all these not only for the current situation but also keeping in mind the constant uh and huge growth of Kaizen as mentioned earlier so that we should remain future proof.
So for the for the new architecture having security in mind we totally separated the production uh environments for the production workloads from all other workloads and we created one as tenant specifically for production and another Azure tenant for all other environments like development staging and so on. Uh the teams would have workspaces in multiple landing zones. Uh uh sorry uh and in each in each uh tenant we would have a data management landing zone to host uh Unity uh catalog resources and uh also uh multiple landing zones uh that are used to separate environments and departments from one another.
And uh in each one we would have resource groups for the teams themselves. And uh for the teams belonging to each environment and department and where needed we would have a common resource group for the resources that are shared among all the teams of this specific environment and department. In each of these resource group, the teams would have their uh datab bricks workspace, their key volt to host their secrets, their secrets and one or more storageages to host their data and uh with proper configuration depending on the data in each case.
So as mentioned earlier, we we duplicated this we replicated this uh tenant setup twice. one for production and one for non-production workloads and the teams would have uh workspaces in multiple landing zones that would mean uh different uh environments in each case. Uh there was a development environment where teams could develop their uh jobs by directly connecting to the workspaces and then they would uh commit their code to a git repository.
Then after that a staging environment existed where that was used for that's used for end to end testing and the deployment there happens with uh through pipelines CI/CD pipelines by using datab bricks asset bundles in this in this stage production data can be uh consumed if needed for the testing and uh this is available by utilizing deltas. somewhere down there below the titles. And uh the delta shares being read only would help us ensure that there would be no data disasters during the testing phases.
And uh also uh for the final stage we had a production environment where through an uh merge request and approval procedure a deployment happens to the to the production uh the production environment. So what have we achieved uh with uh this new setup? We have achieved uh better scalability due to uh bound due to setting boundaries and limits and isolating resources to have uh different quotas and separate quotas using different landing zones which are subscriptions that isolate resources as we saw. uh we have improved phenops because of uh we we have cost transparency cost attribution each team its environment is on a different landing zone so we these things are separated and are easier to find where its cost belongs and we have cost optimization because of resource flexibility we have multiple stoages instead of one and we have we can configure it we can configure the stoages as needed for each data type and uh then of course we have better governance because uh multiple workspaces allow us to uh configure better and access control.
Also isolation between uh for sensitive data having a different production than other environments and uh of course instead of uh deploying manually we use uh terraform and infrastructure as code. So we have resource resource standardization and consistency and also change management and all this leading leading us uh a little closer to our goal of achieving data democratization. So now will talk about the process of migration.
Thank you. Hello again. Can you hear me? Yes. So migration a famous quote nothing can be certain except death and taxes. I will add to this data migrations as well. So a quick poll here. Uh please raise your hand if you ever been a part of a migration project. Okay, more than once for me. So you know the pains. Um so our approach here is to split this project to three distinct phases before, during and after. And each phase has distinct uh goals.
So before migration we needed to um design our security model which is tied as well to our data governance uh part. We need to understand our limitations because we have developed a lot of services uh that expect to work uh in a single uh environment setup and also uh before starting implementing is the best time to define conventions standardize namings etc. Once you start implementing this is very hard to do. Uh for the actual migrations we need to decide about a migration pattern.
So um we saw earlier that we have a lot of stakeholders and uh those stakeholders expect their KPIs the reports etc. Uh it's not an option to say okay now uh we freeze everything and one month later everything is migrated and you will have your reports again. So we needed to find an uh to find an incremental approach and also we need to develop our own tooling to help us uh uh do this migration and uh the last part is of course creating the infrastructure that we saw earlier uh deploying our applications and migrating our data.
After migrations we want to uh check uh our performance. So we don't want any performance degradation going to this new setup and we want to start using our cost insights to reduce operational expenses. So this is about cost optimization here. So about security uh first thing first uh we need to understand what are the main uh administrative responsibilities uh based on data bricks uh best practices um we separate these to three distinct administrative roles account administrator uh who create account level resources like workspaces and metas stores meta administrators uh that create securables like schemas and catalog and delegate access and workspace administrators that have access in workspace and can toggle on toggle off features create clusters etc.
Uh so having these uh roles in mind uh we tied to specific groups we used this notion code group based access control or GBA and uh for our production environments we exclusive exclusively used service principle uh this is basically service accounts in Azure IM h so all data access all jobs or infrastructure as a code pipelines run using the service principles. Uh the next part was creating actual infrastructure. So what we did here uh we used uh what is called terapform modules.
Um so modules are basically uh components terapform that have some input have some output and do some magic in the middle. Uh so you need to understand only their interface. You don't need to know what happens inside. uh this is a nice u case of abstraction and also it helps to tackle the complexity of this uh the kind of things especially when you're dealing with change. So for example we have a module that creates cataloges.
So let's say you want to add a specific tag in all cataloges. You only need to go in the module that creates the cataloges and do one atomic change there and everything is propagated. And uh also we had some discussions about who should uh does what. Uh on the top part we have this possibility of the infrastructure team. We have a separate team that create usually your cloud resources. Uh so they created the subscription the resource group everything that's top level in the cloud. uh in the database account level uh we contribute both and at the bottom part uh inside the workspace to create cluster cluster policies etc uh we contribute as a data platform team but our end goal uh is to educate our users to do this kind of uh stuff themselves so this is kind of self-s infrastructure um so about the migration uh we used a lot of data and bundles a quick poll again how many of you do you use just bound in your flows.
Okay, some hands here. Uh so basically this is a feature of datab bricks uh that's a wpper in terapform. Uh so you can define everything in some files. So for example you see here an example file that have different configuration for each environment. Uh so all our jobs refer to these files and you can uh define centrally some configurations. So for example I have highlighted here the spark version means this is the data bricks uh runtime.
Uh the policy is the policy for all job for all our jobs in this environment and the service principle name. So this is defined centrally. Uh so for example migrate as well to a new uh datab bricks runtime. This uh you change simply this line here. uh so this is very useful for us to dealing with this complexity of multiple environments and this streamlined a lot our CI/CD process uh we also used uh a lot of uh in the in the old architecture we use database file system which is tied to each workspace uh we migrated volumes so this is a um this is across workspaces and has access control it helped us a lot uh for uh storing our our packages our jars our wheels our configuration files So uh the most important part uh is of course data u so we had some limitations uh we had a lot of table so this is not a typo we actually have over uh 20,000 tables and views um so um this is not manageable by single team we can't have a single team migrate all these uh pipelines and uh schemas uh what we wanted to do is to uh help uh different teams to migrate at their own pace.
So uh and during this migration the most important part is to have our commitments to stakeholders. So this should not disrupt our daily uh operations and also uh in the old architecture we exclusively use external tables. Uh now we want to migrate managed because most of new database features are coming to manage. So eventually if you want to opt in uh we want to be on man tables. So uh our approach here our solution is um and users to um don't understand to be invisible for them the migration.
Um so what we did we will see an example for this uh our tables were in both meta stores in all the new architecture but the data actually reside in one uh and we hide this complexity using delta sharing. Uh our uh goal here is to keep downstream dependencies for our end users stable so they don't need to change their queries and uh their notebooks etc. We tested this uh solution for latency for eress etc. because we migrate to the same cloud region, this is not a factor for us.
But if you migrate to a different region, this is something you need to take into account. And this is of course led us to adopt a divide and culture approach. Uh so all teams should help in this project. Uh so I have an example here. Um so on the left side we have two tables. Uh on the bottom we have a table named catalog. table and we want to migrate this and on top in the old setup we have a dependency so we have table_dev uh in the new tenant uh we automate uh this uh we have a delta share so this is off uh offered for it only through the share table but we create automatically a view that goes to the share and uh creates this table catalog table as a selection as a view from the share so this means that our data are available for read in both in both architectures.
So let's say you want to migrate the bottom table. Uh we have this tooling that hides all this complexity. So uh we use the share to do one clone. So we create this table uh you can actually uh pause here and do some testing uh test your applications etc. And if you are okay and you want to continue your migration, you continue and uh you do one last deep clone to sync your uh your data. Uh you drop the view and you rename the temp table to uh the actual name.
So what we achieved up until now we migrated actual uh the data physically from the old setup to the new. What we need to do now is to change our dependencies. So uh we use the same tooling to go back to the old tenant. We use the symmetric delta delta share here uh and we create uh this view catalog table. Now in the old tenant is a view that selects from share and the old table is marked for uh for deletion. So it's renamed to table. archive and you can uh eliminate it uh when you feel comfortable with this.
Uh so what's important during this process uh is that dependency not change. So in the old setup I was always expecting a catalog. And this was always the case. So my downstream user did not change his query. Uh and of course we use our tooling to do this at scale. So we migrate whole schemas etc. U so our progress uh currently we have created infrastructure for over 25 teams. So uh each team has multiple environments and we have done a lot of workshops.
So uh we do pairing sessions. We we need to educate our end users how to use the setup because uh this is the first time for them and we have created the migration tooling. About the data migration part we are uh currently in the middle of this uh migration. It's a really big project and after the migration we want to focus on cost optimization. So we need to reduce our operational expenses. Uh we want also to address um the catalog.
So we have a a catalog with data from different domains and we want to split it to domain catalog. So a domain can be a specific product or customer etc. And our ultimate goal is uh then users to uh use this uh modules and this stuff to self serve the infrastructure. So I want to create a new cluster policy. I just uh change this repo here. So uh some lessons learned some problems that we faced. Uh it's difficult to do governance on a distributed environment.
So it's important to have uh spend some time to understand all the complexities of this before starting implementing. Uh also we had a lot of team coupling. So it's this is not ideal. So you need to be aware of this coupling between teams. So I expect my workflow to end. So another uh team should start their workflows. Uh something that was challenging for us that we different uh we had different maturity between teams.
So some of the teams were completely new to data bricks. Uh so we need to on board them. Uh we invested a lot of time on uh doing pairing sessions to write documentation to do video and this kind of stuff. And also when you need to mobilize a lot of people it's difficult. So even though we say this is migration is uh at their each team can do it at their own pace we you need some kind of coordination. So it's it's nice to have this commitment and also um some problems for example uh we had some implicit dependencies.
So we migrated a table and someone was reading this table which was append only as uh using uh structure streaming. Uh so we messed up their checkpoints. So we started to explicitly check for these kind of things. So uh that's it for us. At the left side you see a QR code. You can go there and see some presentation slides uh and the text for this whole presentation. This is the development team. I want a quick shout out to them.
So uh thank you for your time. This is basically for us. Okay.

Learn more about the Databricks Data and AI platform.