Embracing Unity Catalog and Empowering Innovation With Genie Room
Summary
- Large-scale migration: Bagelcode migrated 10,000+ tables from Hive Metastore to Unity Catalog over five months, redesigning ETL pipelines with Delta Lake and optimizing partition logs.
- Governance elevation: The migration improved governance, simplified management and unlocked Unity Catalog's advanced features including fine-grain access control and metadata lineage.
- Genie Room innovation: Post-migration integration of Genie Room with Slack enables natural language queries and lineage-powered tools, accelerating decision-making and operational efficiency.
Embracing Unity Catalog and Empowering Innovation With Genie Room

Talk by Junghoon Lee, Data Engineer, Bagelcode ; Soochang Chung, DataOps Team Manager, Bagelcode
Bagelcode, a leader in the social casino industry, has utilized Databricks since 2018 and manages over 10,000 tables via Hive Metastore. In 2024, we embarked on a transformative journey to resolve inefficiencies and unlock new capabilities. Over five months, we redesigned ETL pipelines with Delta Lake, optimized partitioned table logs and executed a seamless migration with minimal disruption. This effort improved governance, simplified management and unlocked Unity Catalog’s advanced features. Post-migration, we integrated the Genie Room with Slack to enable natural language queries, accelerating decision-making and operational efficiency. Additionally, a lineage-powered internal tool allowed us to quickly identify and resolve issues like backfill needs or data contamination. Unity Catalog has revolutionized our data ecosystem, elevating governance and innovation. Join us to learn how Bagelcode unlocked its data’s full potential and discover strategies for your own transformation.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
Why did Bagelcode migrate to Unity Catalog?
As Bagelcode grew from 2018 to 2024, managing 10,000+ tables via Hive Metastore became inefficient. Unity Catalog improved governance, simplified management and unlocked advanced features.
How long was the migration?
The migration took five months of careful planning and execution, redesigning ETL pipelines with Delta Lake and optimizing partition logs while maintaining minimal disruption.
How does Genie Room enhance the platform?
Genie Room integration with Slack enables natural language queries, allowing non-technical users to interrogate data and generate insights without SQL knowledge.
What is Bagelcode's scale?
Bagelcode is a rapidly growing social casino gaming company with 55M users worldwide, known for top-performing titles like Club Vegas, processing 1TB of new data daily.
Full transcript
Thank you for coming today, especially at the last session of a long day. It's really nice to meet you all. a data engineer at Bagel Code, mobile gaming company based in South Korea. Today I'll be sharing how we embraced Unity catalog not just as technical upgrade but as an opportunity to rethink our data architecture governance and ownership. And later in the session my colleague Such Chang Chong will introduce how we empower innovation with new features in Unity Catalog including GD.
We'll start with our story, why we decide to move to Unity Catalog and the challenges that pushed us to make that decision. Then we'll cover how we planned, how we executed, and what we learned along the way. And finally, we'll show how that foundation led to something new. Bagel Code is a rapidly growing global mobile gaming company and our flagship title Club Vegas is one of the top performers in the social casino genre.
Uh we are ranked in top 15 globally in the social casino market and we've been featured in Google Play's top 100 grossing games in the US. Today we have over 55 million users worldwide which is where Unity catalog steps in. Our journey as a publisher really began in 2017 when Club Vegas became a breakout heads in the global market. Building on that success we launched a series of casino games expanding our footprints across different markets.
And in recent years, we've been exploring new experiences like puzzle idle and simulation etc. broadening both uh our portfolio and player base. Today we've published over 30 titles and we're continuing to grow as a global multi- genre game publisher. Since 2018, datab bricks has powered every layer of our data stack. That's over eight years of building at scale on the same platform. Today we load around one terabyte of new data every day and process more than 12 terabyte daily across pipelines.
We manage over 10,000 tables powering everything from ETL to machine learning to operational analytics. Uh in short, datab bricks is the backbone of our data platform and Unity catalog was a natural next step for us. As our games and teams grew, our data stack more complicated and harder to manage. We started to lose visibility into table image, making harder to track how data flowed or what depended on what. We started to lose visibility into table image.
Uh sorry uh at the same time we were handling more and more ad queries requests uh which often required time and manual context to answer. uh all of our tables were managed in hive metas store which means we couldn't take advantage of nor capabilities because in 2021 data bricks announced unity catalog introducing gamechanging features like table lineage and advanced data governance system tables for audit logs and identity based access control but we didn't migrate right away because we has been on data bricks for eight years.
We uh so we had a lot of legacy built around hypetaur because migrating means unwinding tightly coupled pipelines while keeping our service stable. And at the time that just wasn't realistic. But as Unity catalog matured and new features kept coming, we eventually decided it was time. These are what that transaction actually looked like. Before you see our old steps, a complex warehouse with limited visibility, manual PII masking, non asset park tables and data requests handled manually.
After now with Unity catalog, a centralized warehouse and built-in uh table lineage audit logs, identity based masking and native support for delta lakes and we are now able to uh delegate queries to intelligent agents, not to data engineers. Migrating to Unity catalog wasn't just a lift and shift. We had to rethink how we upgrade, organize and store our data. Uh we started by looking closely at the migration tooling provided by data bricks, specifically UCX.
Uh although we couldn't use UCX for fully automated migration, it is still played a key role throughout our journey. One reason we didn't adopt automation was because we wanted to migrate uh gradually starting with low traffic, low impact tables and jobs while keeping our production services stable and uninterrupted. Sorry. In the beginning, UCX helped us build a clear migration plan and timeline. And as we progressed, it let us track uh our status in percentage and identify which jobs and tables still need to be migrated from start to finish.
UCX was uh consistently useful tool that kept us organized. As we planned the migration, we realized it was also a chance to redesign our catalog structure. Our hive metaur setup had grown with us since the early startup days and over time it had collected a lot of small inefficiency legacy patterns that no longer made sense. So we started thinking more strategically. We considered adopting the metalon architecture and also explored a studio-based structure organizing catalogs by game studio.
Uh metalon architecture gives us uh clean data layering and reusability while studio based catalogs offer stronger ownership and security by isolating data protein. Uh after raining our options, we decide to go with a studio-based catalog structure because this gave us several clear advantages. First, it enabled stronger data isolation and security by separating access at the studio level. Second, it brought clear ownership.
Each team knows exactly which data they manage and control. And finally, it helped us operate more efficiently by reducing cruise team friction and making our workflows cleaner. Once we finalized our catalog structure, the next step was migrating the actual tables. At the time most of our tables were in parket format but we quickly realized that migrating them because without acid compliance writing to a parket table often means other ETL jobs where users couldn't read from it uh causing pipeline failures and access delays.
So we decide to convert pocket tables we can to delta lake during migration and updated our uh workflows. Here's an example of how we managed the park tables in our legacy workflow using Scola. When a table was being overridden, it couldn't be read at the same time. simply didn't allow it. Our jobs would throw a refresh table alert when this happened. So at least we knew what went wrong. Uh technically we could call refresh table before every read, but that added complexity and felt more like a band-aid than a real solution.
Instead, we override records using save as table method, which gives us a much cleaner and more stable workflow. This unlocked powerful features like asset transactions and table history and lineage. Ideally uh we have we would have converted all pocket tables to delta before starting the migration but uh extending the overall migration timeline wasn't something we wanted. So instead we converted only the tables that were safe and easy to handle early on.
For the rest we decided to complete the catalog migration first and then convert them to delta afterward. One of the biggest advantage of using Delta Lake was that the same delta table could be accessed from both hype metasur and unity catalog as long as they pointed to the same underlying storage. This flexibility was crucial because minimizing data downtime was critical for our business. It allowed us to keep both systems running in parallel and write new records to shared storage, keeping tables fully synchronized throughout the switch over.
They gave our teams and external partners a safe buffer period to adapt to the new system without disruption. Once we finalized our catalog and catalog design and migrate eligible to Delta Lake, it was time to actually execute the migration. And we knew from the start to avoid risk, we needed to move carefully and precisely. So we took a step-by-step approach. uh after fully investigation uh and experience we began migrating all tables into Unity catalog to ensure a safe migration.
We adopted a phase approach. We started with tables that had no downstream dependencies followed by uh middle layer tables and finally migrated the source tables such as row data uh event logs operational t databases. However, things didn't go as smoothly as we had hoped. To be safe, uh, we set up a weekend migration test scores starting with low impact tables, then moving on to critical ones. Uh, but many of those high impact tables hadn't been converted to delta yet.
Uh, they were still pocket and that's when the problem started. Pipeline slowed down. Jobs jobs timed out and then slle our slle exploded. datab bricks job failure or alerts just kept firing like a fire alarm we couldn't shut off. This issue came from a key difference between hive metas store and unity catalog and in hive metas store partition information is stored internally using the metas store repair table command.
So when a select query runs, hype reads from that pre-tored metadata and speeds things up with partition pruning. It's fast because it already knows where to look. But Unity catalog works differently. It doesn't store partition metadata by default. So when you run a select query on a hive style partition table in Unity catalog, it scans all all partitions paths recursively on every query. That added extra stages slowed down our pipelines and many many cases cause timeouts uh especially on the wide tables and deep partitions.
Now here's an example of how partition metadata is structured. uh it closely follows the format of delta lake delta log. The metadata captures key details like comet time stamp, user information, table creation parameters and partition columns. It also explicitly records each partition path and its values giving us a clear and well organized view of how the table is updated over time. Uh for teams that like us uh still needed to run party tables on Unity catalog data bricks offered a solution partition metadata logging option.
It works like hive meta store by storing partition information inside unity catalog root storage. Uh once enabled queries use that metadata to prune partitions efficiently. Just like before, all we had to do was add a spark config when creating or reading the table and suddenly things just got f again. Uh migrating to unity catalog in initially increased our S3 get and list operations because of high style pocket tables.
But after enabling partition metadata logging option, the request volume dropped back to 34% eliminating pipeline timeouts and improving performance. The real cost improvement came when we fully converted to uh those tables to delta and then integrated autoloader to optimize indition and in the end our total S3 cost dropped to just 10% of what it was before migration. Here's what our migration journey looked like from start to finish.
Uh we began planning in March 2024 and spent a few months just prepare uh preparing from catalog design to workflow validation like updating our ETL code to make it fully compatible with Unity catalogs constraints. And from late June to mid July, we we began migrating leave most tables applying the partition metadata logging options I mentioned earlier uh to keep pipeline performance stable. At this stage, most of our effort was focused on replacing hardcoded catalog variables in ETL scripts and monitoring for functional regressions.
We were still operating in dual metasur setup with high metas store and unity catalog both in use. Then in mid July we moved on to core production tables. Uh that's where ran to issues. Uh some of our legacies called UDFs start failing due to unresolved uh library dependencies. uh unfortunately we uh we couldn't pinpoint it the reason but uh we decided to refactor all uh scolio udfs to SQL functions which is not only solved problem but also gave us a noticeable performance boost.
Uh finally in late July we migrate the highest impact business tables. This handled by our weekend migration task force who executed the switch over the great care. After days of stable monitoring, we officially shut down hype meta store in August. While that marked the practical end of the migration, we spent the following weeks converting the remaining party tables under uni color to delta bringing everything up to modern symbol arts.
Uh to talk about what came next and how we built on this foundation, hand it. Great. Good evening. Uh I'm the data off team manager at build code and I'm also a data engineer. Thank you for your interest in our section uh session and thanks to Jung for introducing our journey of EC migration. Now that we have seen the whole tough migration process, you may be wondering if it is really worth migrating to Unity catalog.
So I'm here to share some of our innovations after the migration. uh starting with natural language query powered by Genie followed by tracing table image with streamllet using effortless PII masking and finally auto logs with lelers for data governance. So let's begin. Our first innovation was using AIB genie with our Slack agent. Databicks AIB genie is a tool that was in the keynote that is a tool to engage with data through natural language only supporting unique catalog tables.
So how is this beneficiary to us? Even new users with technical backgrounds often struggle with SQL and some of the users aren't capable of accessing certain data either by the lack of skills or the lack of computer uh permissions. But in a datadriven company like us, these users still need an easy way to query data. Our challenge was simple. How can we make data queries easier for everyone? Nontechnical users and even new team members needed a simple way to query data without struggling struggling with complex tools.
The solution we thought was uh leveraging our ex existing Slack agent to create a smooth and intuitive experience and thanks to data bricks genie we made it happen. a fast, cost-effective solution that just works. Okay, before we go into a deep dive, uh I'll show you a de demo of our AI agent to give you a glimpse about how our AI agent works. Okay, so our AI agent is built on a slack bot. So to use it, you have to mention the slack bot and you type in the query command and ask for certain data such as uh what is the ratio of users who entered chapter 2 in PNC.
Uh FYI, PNC is an acronym for one of our games and you can see the database loading sign that shows that the AI agent is processing the data. And this is where Genie API comes in. Genie gets the request in natural language and then sends a response with the requested data. Finally, the response is formatted as plain text or even as a downloadable CSV file and sent back to the user. Yeah, you can see it can download it.
And if you see the thread on the right hand side, uh you can see that the agent returns the analysis of the request followed by the actual SQL query to get the data and the result data in the thread. So how can you get such results with Genie? I'll take you through some steps needed to set up Genie spaces. The first step is to restore tables for the genie space. Theoretically, you can resto register up to 50 tables or more, but I recommend you to keep it compact even less than 10 or five.
After registering tables, you can get multi-dimensional analysis and insights. Uh users can query these tables directly, ensuring a seamless and efficient data exploration experience. After registering tables to a genie space, we provided an instruction prompt. This included specific rules for data queries like specifying alternate game names, acronyms, or even ensuring call names to follow certain conventions such as snake cases.
We also define key business terms such as inapp purchase and re add revenue to help the system process queries accurately align aligning with our business needs. This instruction may differ on the data you're processing and the application you're going to make. So this is the complete architecture of our genie agent. So when a user Okay. Okay. So when a user sends a message to Slack, a Slack event is sent to a lambda function through the API gateway.
Then the lambda processes the message parsing whether it needs to refer to our documents or send the query to data bricks using Genie. If the user requested data is similar as the example before then it sends a request to an LLM to decide which genie space to send the request. In our spa uh in our case we divided the gen spaces by game studios as by our unity catalogs structure. When the Genie API sends a response back, uh the Lambda processes and formats the data and sends it back to Slack.
Additionally, we also collect user queries and the corresponding SQL in order to better understand the user's needs and improve the system in the future. So, here's what Genie and our AI agent has helped us with. In the past, uh, business analysts or and users would have to talk to a data analyst and request data and the data analyst would process the request and provide the requested data. But there's no knowing exactly when they will get the results and sometimes people would hesitate to just uh to send a request to another person.
Now that we have Genie, things became different. The users, all they have to do is just type in their request in Slack, wait a few seconds, then boom, you get the results instantly. Okay, let's move on to our second innovation. Discovering data lineage. Data is usually processed through multiple steps starting from the source tables to bronze, silver, gold layers and mark tables. If you want to see a certain table uh where it gets the data from, you would have to go through your 100 lines of code for the source.
Thanks to unique catalog, you can easily discover your source and also the downstreams of your data like this in the screenshot. This feature is invaluable for every data scientist and engineer. It also trace the original data to check data consistency, troubleshoot issues and also update ETL when needed. This uh this feature was very handy for us when there was an issue with our ETL and needed back fill. We leveraged this feature into a streamlined app powered by data bricks app feature which enables you to deploy apps easily on data bricks.
We made an app that shows all of the upstreams and downstreams and all the descendants of the table you select. If a worker play fails, we can easily check the data consistency across all its descendants and decides what need to be backfilled. Okay, our next innovation was PII masking. As we introduced uh in the earlier slides, we are a gaming company that service globally all around the world and process user data. So that means we need to follow certain regulations such as GDPR.
So when handling personal identifiable information PII, the data needs to be masked. Before you need catalog, you would have to create new views with the necessary columns masked, which is not that technically difficult, but it will mean managing even more views, which isn't great. However, with Uni catalog's PII column masking feature, managing data governance has become much simpler and more efficient. We can easily create a function to mask specific columns or rows as needed.
It also lets us assign governors at a group level, giving us more flexibility and control. For example, internal users can access all columns by default while external users using tools like superset or data grid that we provided can only see the mass version of the tables. Okay. Finally, our last innovation was managing database workflows. We've been uh using data bricks for over eight years and we have over 800 workflows.
Most of them orchestrated by Apache Airflow. But sometimes users unintentionally disrupt the workflows like assigning overly large clusters or even deleting those orchestrate jobs. To handle this, we started using the database audio logs feature to keep things under control. Audio logs automatically tracks all actions in a table, making it easier to monitor and manage. You can see the user email, event time, action name, job ID and etc. uh all in a table and you can find this in the systems catalog.
So what we did is we implemented Slack alerts to notify us in near real time whenever jobs are created, deleted or updated. The alerts include the total number of jobs, the number of airflow trigger jobs, uh the job ID, job name, the owner, and the cluster configurations. Uh you can see from the left uh the green one indicates the jobs that have been created, the red ones for the jobs that have been deleted, and yellow for the jobs that have been updated.
Additionally, to avoid accumulating outdated jobs, we monitor inactive jobs, too. We created a weekly alert to flag jobs that haven't run in the last 60 days, providing their job ID, job name, and owner for cleanup. By getting all of these updates to our most used collaboration platform tool, the visibility to critical operations have greatly improved. To wrap up, after migrating to Unity catalog, we could empower the following innovations.
We enhance security, clear accountability and ownership, and streamlined operations for improved efficiency. We gain complete visibility into table lineage allowing seamless tracking or table creation and dependencies which strengthens data governance and transparency. By using Genie, we created a smarter, faster and more convenient experience enabling swift datadriven decision- making. Uh I think the slides have changed but uh thanks you thank you all for listening to such a late time before the data afterparty and I want to give special thanks to our other contributors on UC migration data engineer uh Syunha our manager Hanjin Jang and all the solution architects of data bricks Korea for their dedication.
We hope you take a lot of insights from our presentations and don't forget the surveys. Thank you.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.