Cost-Effective Data Architecture and AI Practice With Databricks at FunPlus

Summary

• Cost Optimization: FunPlus reduced data platform expenses through architecture optimization on Databricks.
• Scale and Performance: The company handles over one million daily active users and 5TB of incremental data per day with Databricks.
• Gaming Use Cases: Databricks powers operational decisions, KPI monitoring, and critical metrics like DAU, retention, and revenue for game operations.

Cost-Effective Data Architecture and AI Practice With Databricks at FunPlus

Talk by Chao Chen, Data Director, FunPlus
FunPlus's journey to building a cost-effective and efficient data platform with Databricks: exploring how FunPlus leveraged Databricks to tackle key challenges, enhance data engineering and ML efficiency, and showcasing best practices and their impact on game development and operations.
Here's more to explore:
The best data warehouse is a lakehouse: https://www.databricks.com/product/databricks-sql
How to migrate legacy data warehouses: https://www.databricks.com/resources/ebook/migrate-your-legacy-data-warehouse-databricks
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does FunPlus use Databricks for game operations?

FunPlus leverages Databricks Data and AI platform to monitor key performance metrics like daily active users, retention rates, and payment revenue. This enables the data team to support operational decision-making across their portfolio of popular game titles.

What scale does FunPlus operate at with Databricks?

FunPlus manages over one million daily active users and ingests more than five terabytes of incremental data daily. The platform supports numerous BI dashboards and data API calls for business teams across multiple game titles.

What are the main benefits of FunPlus's Databricks architecture?

The cost-effective architecture allows FunPlus to handle massive workloads efficiently while supporting critical gaming analytics, including player lifecycle analysis and real-time operational metrics that drive business decisions.

Which gaming titles does FunPlus support with Databricks?

FunPlus uses Databricks Data and AI platform to support popular titles including State of Survival, Kings of Avalon, Guns of Glory, Sea of Conquest, Stone Shop, and DC Dark Legion across millions of players worldwide.

Full transcript

Okay. Hello everyone. Uh thanks the AIS team to uh for bring us together to explore the latest innovation on data and AI and I am so exciting be one of the speakers to share our practice on data bricks uh on behalf of my team and uh let me introduce myself briefly. I work as a product director at far plus game and I focus on the infrastructure system design and building such as data platform, data service, data product and marketing tools, some operation systems and AIGC platforms to support our game successful publishing and operations.
Um, prior to front plus I I ever work at N games and Tensson games. Yes. And let's back to the topic of the session. I will represent my team to share our practice about cost effective data architecture and AI practice on data bricks and I will break down the topic into four parts. The first part is a brief overview of fun plus and the second part I will introduce uh our data architecture our use cases and challenges and the third part I will accept expend more on some best practice and benefits of using data bricks and the last part I will talk something about our next steps and future plans.
Okay. So what is fun plus? Yeah, FP plus is a game developer and publisher company that uh founded in 2010 and we headquartered in Switzerland and our main development team is in China and we have launched a new a number of some popular titles like state of survival, kings of abalone, guns of glory and sea of conquest, stone shop and DC dark legion. So uh is someone here is our player? Okay. Well, yes. Our mission is aimed to create a world-class gaming experience to our players all over the world.
Yes. So, as a g as a data platform in a gaming company, data plays uh critical role in our daily workflow. So, in this part, I will introduce our data platform architecture, our major use cases and some characteristic and challenges of building a data platform in a gaming company. Let's take a look at our data scale. We handle massive workload daily over one million daily active users and over five terabyte incremental data per day.
And we support a large number of BI dashboards and data API calls for our business team and uh as a game as a gaming company there are some typical use cases that data drives value. The first one is operational decision support. We support like core KBI monitoring. We check some key metrics of our product like DAU, new users, uh payment revenue and retentions and some life cycle value etc. And we also provide detailed analytics for our game server ecosystem and our gameplay engagement.
And the second part is marketing and advertising optimization. We provide evaluation and analytics for uh advertising performance and creative materials performance and cross media quality. And we support player behavior insights by analyzing our users um progress in the game and their preferences on the game plays, their in-game cordicature and their in-game social interaction. And we have built a personalized recommendation service by analyzing and machine learning each single users uh behaviors and prevances.
And we recommend the most suitable uh task items and events to our players to make them more engagement. Yeah. And other use cases like realtime monitoring and abnormal situation alert, fraud detection and analytics for our gaming community engagement. uh take a look at our previous architecture before. The key point is uh due to some historical reasons different uh business team in our company are using different um data warehouse provider.
Yes. And we're resulting high development cost on data integration and data construction. And it's challenging for us to build a um unified marketplace to uh share the data across team. And it's also challenging to the stability and the usability uh of some upper layer data applications such like BI dashboards, a whole queries and some data services. And apart from the large data volume and diverse data sources, we are also facing frequent game version updates which requires more agile and real-time data feedback.
And in practice, different uh data requirements from our different game generally vary a lot. We has SLG games and RPG games, puzzle games where we have to balance what kind of data should be built as a common layer and what kind should be talented for special needs and in the same time we have to address the rising cost. Yes. And all of these are big challenges. So that's why we decide to migrate our data warehouse to a unify an AI power and userfriendly uh data platform. uh and we have uh during the migration we would like to reshape our data workforce and thanks data bricks team they send a sp special specialist team to assist during the whole migration that's truly important so in this part I will introduce how far pass leverage data bricks to address key challenges and improve our uh data engineering and machine learning efficiency in terms of the enhancement of data engineering there are some there are some key points By leveraging the uh Unity catalog, we achieve a centralized metadata management and complete data lineage checking.
Uh this truly important for uh enhancing our data discovery and understanding and by using the federate query capability of datab bricks. We are able to assess multiple data sources in a unified circle environment and we also use the AI power query assistance for improving our productivity. Another significant benefit is about the data collaboration. That means we can now easily share our work space uh including notus queries dashboards cross team and I will introduce uh two specific pattering.
So as we know a dashboard relies on multiple tables which are populated by workflow jobs and uh data processing code. So when there are issues happen on a dashboard something like delays or errors troubleshooting requires layer bylayer investigation is quite time consuming right and to solve this pinpoint we leverage the data lineage data to generate upstreaming and downstreaming dependency information automatically across dashboards tables workflows and calls with clickable links.
That means when there is a issue happen, we are able to see the underlying table immediately and check back to the workflow and the code. For example, on every single dashboard in our BI system, the relative d the relative workflows and the um uh the table uh the dependency information is generated automatically with clickable links and displaying the tool tip. So when we click the link it goes to the t table catalog page or the workflow page.
Yes. And in the same way on the job details page and table catalog page the relative workflow and dashboards information are generated um automatically with clickable links and displaying the desript description area. It sounds simple right? But it truly reduce the troubleshooting time from hours to minutes. Our cross team members don't have to grab someone to figure out the upstreaming and downstream information on dashboard.
They can check them by themsel. Another playlist is about uh integrating uh databicks genie. Yeah, as we know database genie is conversational AI chatbot that define for quering data using natural language within a specific business domain and query scenario. And a genie consists of data table scope and instructions and some query samples. And we believe data bricks genie provide another choice for our data team to deliver data.
Before that in the past I think dashboards are the most widely used pattern for data delivery. But now we our data team can create and deliver a genie to our uh business team allowing them to ask questions by themsel to query data. Yes. And we believe a practical and user-friendly uh conversational chatbot should follow the standards like requirement confirmation. That means when there are unclear questions, the chatbot should clarify with the users and multi-turn conversation.
That means the cho maintains the context for follow-up questions. And uh we think the chatboard should provide the process of the reasoning and the thinking uh for the questions and allowing our users to readit and re-execute the circle. And all these capabilities, all these are the capabilities that data bricks genie already has. And our practice to integrated genie uh is like instead of creating a single genie to handle all the queries uh which may result in low accuracy because so many tables and so many overlap definition of metrics and we do this like we create multiple expert genie and each one focus on a specific business domain and each one is fine-tuned by its creator for in ensuring the high accuracy and in our in-house HouseBI system.
We provide a UniFi uh chat interface so our users can ask questions on the UniFi interface and the questions are looted to a uh specific the most proper expert geni to handle. Uh we also integrate Genie into our enterprise IM software as a lowboard in a chat group. So our team members can ask data questions anytime. So we believe this approach ensures higher accuracy and better performance. Uh once again designer is like having a data analysis available 24 hours a day.
Okay. So in terms of the uh data cost uh after m migrating to data bricks we achieve uh 40% total cost reduction by some effective cost optimization and governance. Here I would like to share some strategy. The first strategy is time to life management. So in practice we create and drop a lot of tables daily. So where we um have to um design a policy that uh uh provide a more reasonable clean up strategy. So we design a tire management strategy to assign tables into different levels like level one to level n based on their popularity and lineage and each level has its own alert mechanism and low data retention periods after table are dropped.
And the second strategy is about selecting uh com compression algorithm. So after a series of tests we collaborate with databicks team to make the Z standard algorithm compatible and this resulting more than 30% improvement in compression radio and cost saving and the third strategy is about uh compute resource control. So besides serverless sometimes we use job cluster when we have to like assess some internal network data and uh instead of allowing our business team to create resources by themsel we design a policy that recommends the most suitable computing resource templates for different types of data applications and this reduce some unreasonable use of resources.
And the fourth strategy is about data partition. We use liquid to set cluster which is more flexible and uh it can optimize the partition uh automatically to reduce the data scan volume. And the last strategy but is also very important is about data driving cost tracking. We collect and track all the cost data and we have in time cost alert and looting uh cost audit including some auto notification of high cost jobs and queries and send to the owners and we have detailed cost daily comeback for monitoring and alerts uh based on some cost data on project level on personal level on dashboard level.
The above is about the uh cost government and in terms of the uh improvement on machine learning efficiency. You know actually in gaming industry machine learning has so many applications. Uh by leveraging the ML technology we can build our personalized recommendation system and player turn prediction and user profiling uh which help us to enhance our use user experience. Especially in implementing personalized gaming experience we aim to build a refined user life cycle operations mechanism.
So essentially we think uh recommendation system is aimed to uh deliver the right content at the right timing for the right channel to the right person and all of this requires a foundational infrastructure that for data cleansing and feature engineering and model training model evaluation and in pre uh solution data should be export from the data warehouse to S3 and then load it to the decision maker to uh for data cleansing, model training, model evaluation and other model uh training done the models are set back to the S3 and when model inference we have to load to the decision maker and eventually the prediction results are set back to the uh data warehouse.
While the current solution the whole progress the whole process and the whole workflows can be complete on a unified data platform uh loading the law data directly and perform the feature engineering model training model evaluation inferences uh model inference and we have version control and uses u monitoring. This truly enhanced the uh efficiency of model training um mod uh machine learning development and we believe this kind of one-stop machine learning development experience help us to build a more smooth and effusion development workflow.
All the data and models are process breathing data bricks use spark no extra import or export and it enables linkage tracking for data and models. So make it easy for troubleshooting right and here I will introduce a specific practice about building high performance in game sentiment analytics on data bricks. So uh you know uh our users in game chat data and sentiment data uh is crucial for product optimization and VIP players retention.
It help us it help us to understand our users uh voice their emotions and their experience. uh help us to collect some feedback of new features and uh monitor the in-game social ecosystem and all of this require machine learning and AI workflow. So this is a simple overview of our in-game chat data processing workflow in databris. uh we extract and filter the chat log uh using database spark and do the np feature extraction and after that we channel a model to cluster different languages and different topics and then we uh use large language model to extract and analyze uh some insights and topics from each cluster and after that we are able to build a comprehensive uh dashboard for visual for visualizing the social engagement and emotion data.
And besides, we provide a flexible BI tools for our business team to query and filter chat data by all kinds of dimensions. And the result shows the topics of the uh users and AI sentiment summary report that generate by LLM. And in addition, we build a AI sentiment in game realtime monitoring that uh help us to detect the term risk of our VIP users. That means when uh a VIP user uh indicates a intent to quit the game, say something like I I don't want to play the game anymore or I want to quit the game.
So AI recognize it and notify our human CS team. So our human sales team can get connection to the VIP users and our in-game AICS bot will send a reassuring message to the VIP users. Yes. Okay. And uh the uh what I share above is about our platics of data engineering, machine learning and cross governance on data bricks. And um we are continue to explore more potential value of data and AI in the future. So here I I will share some potential ideas.
The first idea is about data and AI enhanced dashboards. So as we know uh dashboard uh contains um a lot of data charts, data tables and data matrix and we believe tables actually uh dashboards are actually the organization of data and analytics framework. So we aim to transform dashboards from a simple visualization to a powerful insights. For example, we can uh leverage the large language model capabilities to pass the dashboards and the chart metadata to a clear mind map to help our uh business team to understand the content of the table content of the dashboard and large language model can automatically generate some key discoveries based on the dashboard and allowing our users to get quick insights and also users are able to select multiple charts and ask them questions. large language model will analyze it and give some insights.
Yes. So we believe dashboards should no longer be simple data description but will be a intelligent insight tailor and another idea is about data and AI enhance marketing. Yes, as we know marketing is crucial for game success and create a excellent creative materials for marketing is important part of it. So we plan to collect uh integrate massive creative data. Those data uh includes our own data and third party data and this kind of data uh generally consists of some unstructural data like video and image assess right and some structural data such as uh performance metrics for example CTR CVR impressions and by uh by leveraging the capabilities of multimodel understanding capability we are able to analyze the features and the patterns of creative like the visual elements, the hook, the uh creative tactics and some game plays, some narratives from the asset and after that we are able to channel a model to analyze the association between the creative pattern and their performance.
So the so the model is able to uh help us to predict the performance of our creative. Uh we believe it will be the marketing intelligence engine that help our creative team to generate uh uning ideas about creative and optimize our marketing strategies and we we are also plan to build a semantic search and reasoning analytics for our creative assessment uh via some better vet search technology or embedding technology.
So comparing to the keyword search, we believe semantic search can deeply understand the intent of behind the natural language queries and matching the most relevant assess for our marketing teams have them to get quickly insight uh from the assess and we believe all of this should uh can be complete uh based on data bricks capabilities uh via delta lake mro and model serving weather Yes, that's my sharing today. And in summary, at fun plus based on data bricks, we we've not only achieve our costsaving goal, but also improve our data engineering and machine learning efficiency.
And in the future, we are continue to explore more data in AI applications. Yes, thank you for listening.

Learn more about the Databricks Data and AI platform.