Building Scalable Anti-Cheat ML Systems for Live Games with Databricks

Summary

KRAFTON built a near-real-time cheat detection system for PUBG using Databricks, combining structured streaming, MLflow, online feature stores, and Databricks Asset Bundles to take ML detection from research code to production at scale.
The pipeline processes multilingual community feedback using Vector Search and LLM validation, then applies LLM-as-Judge evaluation with ensemble judging and step-by-step analysis to measure topic extraction quality reliably.
KRAFTON navigated key production tradeoffs — latency versus cost and consistency versus scale — and used horizontal scaling for official matches to handle peak concurrency demands in a live game with millions of players.

Building Scalable Anti-Cheat ML Systems for Live Games with Databricks

Real-time anti-cheat systems are critical for maintaining player trust and game service health. KRAFTON shares how they built scalable ML systems for PUBG anti-cheat by combining two signals: player feedback from communities across multiple languages and in-game behavioral patterns. The key challenge was turning noisy, multilingual player voices into consistent operational signals that drive enforcement decisions.
This talk covers the complete pipeline architecture. Learn how to extract consistent topics from community feedback using Vector Search, LLM validation, and LLM-as-Judge evaluation to measure quality reliably. Then see how KRAFTON built a near-real-time cheat detection system on Databricks, optimizing from research code to production using structured streaming, MLflow, online feature stores, and asset bundles. Discover the real-world tradeoffs: latency vs cost, consistency vs scale, and how to debug distributed system bottlenecks in deeply nested data structures.

Chapters

FAQs

How does KRAFTON detect cheating in PUBG in near real time?

KRAFTON built a near-real-time cheat detection system on Databricks using structured streaming, online feature stores, and MLflow to process in-game behavioral data and flag suspicious activity. The system combines gameplay pattern analysis with multilingual player community feedback to generate consistent enforcement signals that drive ban decisions.

What is LLM-as-Judge evaluation and how does KRAFTON use it?

LLM-as-Judge is an evaluation method where a language model assesses the quality of another model's outputs — in this case measuring how consistently topics are extracted from multilingual player feedback. KRAFTON uses ensemble judging and step-by-step analysis to make this evaluation reliable and measurable at production scale.

How does KRAFTON use Databricks Vector Search to process player community feedback?

KRAFTON uses Databricks Vector Search to extract consistent topics from community feedback posted across multiple languages and platforms, converting diverse player voices into structured operational signals. These signals are then validated with LLM techniques before being used to drive cheat detection and enforcement decisions.

What are the main challenges of building production ML for live games at scale?

Building production ML for live games requires managing tradeoffs between latency and cost, and between consistency and scale, particularly as player counts and match volumes grow. KRAFTON specifically addressed distributed system bottlenecks caused by deeply nested data structures and used horizontal scaling to handle peak load during official matches.

Full transcript

[00:08] Hello everyone. Uh thank you for joining our session today. I know this is right around the lunch time. So, thank you especially for spending this part of your day with us. My name is Hyojun, a data analyst at Krafton, working with the game and
[00:24] community data. I'm joined by my colleague Joey, an AI served engineer who builds ML pipelines with real gameplay data. And now that we are actually standing here, Joey,
[00:41] I'm starting to feel the pressure. How about you? Uh same here. I did not realize this venue it feels so big until we got here. Right. But I guess that's a good reminder for today. What do you mean?
[00:56] Um whether it's a presentation or a live game service, uh things get harder when the scale gets bigger. Exactly. Um today we'll share how Krafton uses AI and data to handle this
[01:12] kind of scale in live games. Sounds good. Then let's get started. Uh before we get into the details, for those who are not familiar with Krafton, uh Krafton is a global game company from
[01:27] Korea. Uh we are known for games such as PUBG Battlegrounds and Subnautica and Elyon. And today we'll focus on PUBG.
[01:44] Uh PUBG is a battle royale game where up to 100 players land, start with almost nothing, find weapons and items, and fight until only one player or one team is left. So, in PUBG, winning is not easy.
[02:05] Uh this picture shows some of the awards that Crafton received uh in 2017. PUBG had more than 3 million concurrent players and also set seven Guinness World Records. So, yes, the game was big. And when a
[02:23] game when a game become this big and competitive, one serious problem always follows. And that problem is cheating.
[02:38] To understand why this matters, let's look at it from the player's point of view. Imagine you have survived almost the entire match. You started with nothing, find the weapons, and made it all the way to the final moment.
[02:54] Now, you hide behind the wall and wait for the right timing. But somehow, the enemy already knows exactly where you are. And sometimes, their bullets even penetrate the wall.
[03:14] In another match, once again, you spend several minutes gathering weapons, armor, and supplies. You finally feel ready for the fight. Now, you're crawling in the grass, trying to stay hidden until the right moment.
[03:30] Then, out of nowhere, a car comes flying toward you. You have no time to react. You are eliminated before you you even understand what happened. Um if this happens once, you feel
[03:45] frustrated. If it happens again and again, you start to lose trust. And when players lose trust, they eventually leave the game.
[04:02] That is why anti-cheat is not just a technical problem. It's a trust problem and a service health problem. Once the players start to doubt fairness, even normal losses can feel suspicious. So, our The question is
[04:17] how do we understand understand this problem and how do we respond to it? To understand the cheating problems, we look at two types of signals. And the first is gameplay data, which
[04:33] showed what suspicious players actually do inside the game. The second is a player voice, which showed what players say outside the game. And today, my focus is on the second signal, player voice.
[04:53] When players feel they have been cheated, communities react very fast. Players say things like, "How did the player shoot me behind the wall?" "There are too many cheaters." "Is the craft doing anything?" This feedback comes from many countries
[05:10] across many platforms and in many languages. But the hard part is not just volume or translation. Player voice is full of slang, jokes, and sarcasm.
[05:29] For example, a simple laugh can be a real laugh, but it can also mean sarcasm, blame, or frustration. So, the same text can carry different meanings depending on context. If we miss the context, we may miss the real issue.
[05:45] Our goal was clear. We wanted to turn those player voice into clear signal that our team could actually use. Our first approach was simple. We every hour we asked an LLM to read
[06:02] community posts and messages, extract important topics, and summarize the trend. At first, the output looked good. The summaries were natural and the extracted topics were looked reasonable.
[06:18] But when we looked at the output over time, we found a serious problem. The LLM gave a different names to the same issue over time.
[06:34] For the same cheating issue, it might call it wall tracking in 1 hour, ESP suspicion in another, and cheating frustration later. For a human, these may look like the same issue. But for a system, they become become
[06:51] different topic names. So, the same issue was split into many different topics. As a result, the trend became messy, the dashboard became hard to read, and the real problem became harder to track.
[07:08] This was our first big lesson. LLM output can look good at first, but if the output is not consistent, it it is hard to use in real operations.
[07:25] So, we changed our approach. We decided not to treat every new topic as a completely new issue. Instead, we built a topic dictionary to keep a stable set of topics. When a new topic came in, a vector search helped us was to similar
[07:42] topics from the dictionary. It gave us a It gave us short list of possible matches. But, we did not want to rely on similarity alone because, as we saw earlier, in community data, different words can
[07:57] can point to the same issue, and similar words can sometimes mean different things. That is why we added an LLM validation layer. It checked whether the new topic and the existing topic were actually about the
[08:15] same issue. If the issue was already covered, we linked it to the existing topic. If it was truly new, we added to the dictionary. This way, the dictionary stayed stable,
[08:30] but it could still grow when a truly new issue appeared. The full pipeline looked like this. Let me show what this means in practice. A player may write, "I was hiding behind the wall, but the
[08:46] enemy still knew exactly where I was." The LLM uh might turn this into a topic like suspicious wall tracking. But, our pipeline doesn't stop there. It checks the topic against the topic
[09:02] dictionary. If this issue is already covered by an existing topic such as ESP suspicion, we'll link the post to that topic instead of creating a new one. So, in the dashboard,
[09:18] this feedback becomes part of one clear trend, not another broken topic. And this whole pipeline runs on Databricks.
[09:34] With AI query, calling LLMs became as simple as writing SQL or using built-in function without managing API keys, building separate building separate API call logic, or integrating everything back into the pipeline. This made it much easier to apply LLM
[09:52] reasoning to our internal data. And with Databricks Vector Search Index, we could enable Vector Search directly on top of our existing Delta tables without setting up a separate vector
[10:07] database and manually syncing data to it. Together, they allowed us to spend less time on setup work setup work and more time on prompt engineering, output review, and the actual analysis.
[10:26] This is what it looked like in the dashboard. At the top, we could track community sentiment and volume over time over time. Uh when we selected a time point, we saw the main topics from that moment. And when we selected a main topic such
[10:43] as anti-cheat, we could go deeper into subtopics and the summaries. Now, the dashboard was easier to read and we could track community issues more clearly.
[10:58] At this point, we thought we were done, but we were not. We had a few hard questions. How do we know it is actually good enough? How can you measure quality without manually reviewing every output?
[11:15] And when a new LLM comes out, how do we know if we should use that model? At that moment, I I felt like the work was getting bigger and bigger, but we couldn't stop there.
[11:34] But to answer these questions, we used the LLM as a judge as a part of our evaluation pipeline. We already had two key artifacts from the pipeline, the input community text and the model output. We passed them both to a judge model
[11:49] along with evaluation criteria. The judge returned a score and a short reason. This gave us a scalable way to measure quality without manually reviewing every output. The next challenge was making this
[12:06] measurement reliable. If we relied on only one judge model, the score could be influenced by the model's evaluation bias. To reduce that risk, we used judge
[12:21] models from three different providers. Together, they worked as an evaluation ensemble. Database models model starting supported the latest models from different providers,
[12:37] which helped us test and compare them in the same environment. This gave us a more balanced view of quality. From there, we need to go one level deeper.
[12:53] If the final output was bad, was it from topic extraction, topic comparison, or summarization? That is why we evaluated each step separately.
[13:10] For extraction, we checked whether the model captured the real issue from the community text. It was not enough for the topic to simply sound good. It had to match the player's message and keep the important context.
[13:27] For a comparison, the question was different. Are these two topics are actually the same issue or do they just sound similar? In this step, we have to be careful in both directions. We could merge different issues into one
[13:45] or split the same issue into many names. For each step, we scored the LLM response from zero to five. This helped us compare models by task,
[14:00] not just by overall quality. When we looked at the scores step by step, we found something interesting. The newest or most expensive model was not always the best choice for every
[14:18] step. Some models were strong at extraction and some models gave were stable at validation. And some models gave a good enough quality at a lower cost.
[14:33] In our evaluation setup, we found a way to reduce LLM cost by about 45% compared with our first baseline pipeline while keeping the quality level we needed.
[14:49] But the value was not only cost reduction. The bigger value was explainability. In real work, at some point, someone will ask us "Why did you choose this model? Why not the newest
[15:05] one? Why did you build a pipeline pipeline this way?" And usually, that someone is our boss. We cannot just say, "Well, it was new or everyone is talking about it." Uh with
[15:21] this we can explain our choices with evidence. Uh, so far I talked about player voice outside the game. Let me wrap up with two key takeaways. Uh, first noisy player voice can become
[15:39] a clear and consistent signal when we combine vector search with the LLM validation. Second AI systems also need measurable evaluation criteria. If we cannot measure quality, we cannot
[15:56] improve it with confidence. Uh, player voice helps us understand where players feel the problem. But, listening to players is not enough. To actually catch cheaters
[16:11] we need to look at what happens inside the game. So, with that I'll hand it over to Joey.
[16:27] Uh, thank you Hyojun and um Okay, let's start it. Uh, in Hyojun's presentation we've seen how AI can help us understand player behavior and identify emerging trends at scale. And building on that, I'd like to share another example of how we are applying
[16:44] AI at PUBG on Databricks. This time, instead of understanding player behavior we're using AI to protect the gameplay experience itself.
[17:01] Specifically, I'd like to introduce how we built an ESP cheat detection ML pipeline. PUBG's anti-cheat system is composed of multiple layers. And each layer is designed with different goals and responsibilities.
[17:16] Among them, this particular pipeline was developed to detect and sanction cheaters uh sanction cheaters uh as quickly as possible. The reason simple, the longer detection takes, the more very uh matches cheaters
[17:34] can continue winning even after they have already been identified. Ultimately, our goal was to build a low latency, large-scale serving system. In PUBG, the two most common match types are competitive matches and official
[17:51] matches. Competitive matches are ranked matches with a much higher level of competition. While official matches are the largest scale standard gameplay mode with significantly larger player and match volume.
[18:07] In terms of inference workload, official matches generated more than seven times the workload of competitive matches. As with most ML projects, having a working model doesn't automatically mean
[18:22] the system is production ready. At the time, the research pipeline was mostly local Pandas-based feature engineering combined with ad hoc Python code containing some Spark queries. The model also relied not only on
[18:39] current match logs, but also on historical features like long-term player behavior patterns. And those historical features were being computed from Delta table based on data uh data marts on Databricks.
[18:54] So, at that point, we had local Pandas processing, Spark-based feature uh historical feature processing, and real-time serving requirements all at the same time. This project started around late 2024 to
[19:10] early 2025. Back then, agent-based workflows on Databricks were nowhere near as mature as they are today. We had two months and a team of three people. Still, honestly, our first goal was
[19:27] pretty simple, to build something that was actually operational in production, even with very limited resources. Honestly, I don't think we could have delivered the working pipeline on time without Databricks.
[19:44] Our internal data platform was already built on top of Databricks, and things like workflow orchestration, workspace-based collaboration, and monitoring and dashboards, and integration with Delta table-based
[20:00] data marts were already in place. Databricks allowed us to spend much less time on infrastructure provisioning, and focus more on the actual feature engineering and serving problems.
[20:19] We first started by building an hourly batch pipeline using Databricks workflows. We separated the feature engineering logic into Pandas-based jobs and Spark-based jobs, and optimized each part differently depending on the workload
[20:35] characteristics. Originally, even running inference on a very small number of matches could take two to three hours. But, after the refactoring and optimization work, we were able to
[20:50] process all competitive and auction matches within about an hour. After validating the model to some extent through the hourly batch pipeline, we moved on to building a near real-time
[21:07] pipeline. And because all the batch processing simply was not enough. Even after detection, there was still a delay before actual bans were applied, which meant cheaters could continue playing and ruining
[21:22] several more matches in the meantime. So, we had to redesign the entire pipeline for real-time processing, including data ingestion, feature engineering, and inference.
[21:37] We suffered from related teams. We were able to receive gameplay log ingestion events in real-time using Kinesis and SQS, and build a near-real-time inference pipeline based on structured streaming.
[21:56] During this process, we integrated the existing Pandas-based feature logic into structured streaming using applying Pandas. At this point, some of you might wonder, "Why didn't you just rewrite all the feature logic in native PySpark?"
[22:13] We considered that as well, but when we looked closely at the actual feature workflows, many of them involved nested audit also and session-level sequential computation. Technically, these operations could be expressed in PySpark,
[22:30] uh but doing so it require fairly complex higher-order expressions, which would significantly increase code complexity. We also felt that a full Spark native rewrite wouldn't fundamentally change the nature of the computation process
[22:47] itself. In the end, we were still iterating through nested sequences inside each row and performing filtering and aggregation. So, instead of forcing everything into native Spark, we decided to integrate
[23:04] the already validated Pandas logic into structured streaming using applying Pandas. And for computing matches that approach was sufficient to achieve near real-time latency.
[23:23] Uh but once real-time serving actually started we began paying for our past decisions. Lesson points it really felt like our past selves has started sending invoices to our future selves. And unfortunately, future us had no
[23:40] choice but to pay them. Uh from that point on, we were dealing with three major problems. Cost reduction, operational stability, and scaling the system for real-time serving at offshore match scale.
[24:03] So, the first problem was cost. The initial streaming jobs had a very large memory footprint due to the combination of applying Pandas based feature processing and large pre-computed feature joints. As a result, the pipeline required both high memory
[24:19] node types and large cluster sizes. This made the operation cost quite high. So, with technical guidance and POC support from the Databricks Korea SA team, we optimized the system in three different ways.
[24:37] And eventually, we could reduce the overall infrastructure cost to about 1/10 of the original setup. First, we adopted Databricks online feature store.
[24:52] Originally, the streaming pipeline performed inner joins against the entire pre-computed feature tables. And that approach resulted in high memory usage, expensive joins, and increased latency.
[25:08] So, we redesigned the system around online feature lookups. Only where only the required keys were fetched on demand, while keeping training and serving features consistent. As a result, we were able to
[25:24] significantly reduce memory usage and latency, while also improving traffic training serving feature consistency. Next, we migrate the feature training and inference workflow to MLflow and
[25:40] serving endpoints. With MLflow, we were able to manage the things like model version tracking and reproducible deployments much more reliably. At the same time, we moved inference into serving
[25:55] endpoints and exported exported as an API-based serving layer. That gave us several benefits. Decoupling streaming compute from model serving, reducing the memory footprint required
[26:12] for model loading and inference, and improving overall serving stability.
[26:28] During operation, we also ran into intermittent OOMs and segmentation faults that didn't clearly show up in cluster metrics. At first, we thought this was simply a resource issue, but after a lot of debugging and with help from Databricks
[26:44] support, we eventually found the real cause. When using apply in pandas or two pandas, Spark data gets converted into arrow format before being passed to Python workers. And our gameplay match logs contain
[27:01] deeply nested structures. The problem was that converting those nested arrays into arrow format required much more memory than we initially expected. And in some attic cases, the conversion itself became unstable.
[27:19] So, the issue was not simply that the cluster was too small, but the nested pair of structure itself was part of the problem. As a result, just increasing cluster memory or tuning spark configs was not
[27:35] enough. Instead, we reduced both memory usage and instability by flattening nested pairs and filtering out unnecessary columns.
[27:52] Operational stability was another challenge we had to deal with. The researchers were continuously improving the model and developing new features at a very fast pace. But ironically, that speed itself started creating consistency issues
[28:07] during deployment. And which uh this reduced our serving stability. We realized that one of the main problems was that feature development was not consistently using the same code
[28:23] versions as the serving pipeline. So, to solve this, we created and distributed a shared utility library, which allowed both researching and serving pipelines to use the same standardized feature logic.
[28:45] Well, the final major challenge scalability. Eventually, we started evaluating expansion from computed matches to official matches, but official matches generated roughly seven times the inference workloads of
[29:02] computed matches. At first, we assumed we could solve the problem simply by scaling up the cluster size, but in practice, that was not enough. The problem was the and not just the total amount of compute.
[29:23] As you move toward official matches scale, several parts of the workload stopped scaling cleanly within a single streaming jobs. As key cardinality as key cardinality increased, Spark stopped treating the workload like
[29:39] selective point lookups and started choosing much broader scan and join strategies. And offline Panda processing also introduced task local bottlenecks because large nested payloads had to be
[29:56] converted and processed inside individual Python workers. As a result, even when cluster resources increased linearly, latency did not.
[30:14] So, rather than continuing with a scale-up approach, we redesigned the architecture around scale-out. We built a broker job that fan down gave a fan down game play log events across multiple structured streaming inference jobs.
[30:30] This allowed workload isolation, horizontal scaling, and independent job management. But, of course, once we introduce multiple streaming jobs, operational complexity also started
[30:47] increasing. So, to keep the system manageable, we standardized deployment and job management using Databricks asset bundles. And because we had already significantly
[31:02] reduced infrastructure costs through the earlier optimization work, we were able to scale out the system while staying within roughly the same cost range as the original pipe pipeline before optimization.
[31:18] In our technical POC, we were able to verify near real-time processing even at official match scale. And finally, we made it to the key takeaways.
[31:34] There are three major things I personally learned from this project. First, even on Databricks, building and operating a real-time production ML system still requires a deep understanding of
[31:50] distributed systems and workload behavior. Even now, in a world where generating code has become much easier, engineers still need to think carefully about what actually creates bottlenecks,
[32:07] what tradeoffs are acceptable, and which parts of the workload truly benefit from distributed processing. And second, if you want to operate production systems reliably,
[32:23] it really helps to be nice to the people around you. From real-time ingestion to serving stabilization, we received tremendous help from our researchers and related
[32:38] and the database Korea team. And actually some of the people who helped make this project possible are sitting in the audience today. So, that makes me a little more nervous as well.
[32:56] And finally, technical data's charge interest. Once real-time serving started, we kept running into the cost of distance we had made under heavy time pressure. But, despite all of those challenges, we
[33:14] learned a tremendous amount throughout the journey and hopefully some of those lessons were useful to you as well. I hope this talk resonated a little at least a little with people working on similar real-time ML pipelines or
[33:31] large-scale serving systems. So, that brings us to the end of our session. We're running a little short on time, maybe. So, we'll skip the Q&A for now. So,
[33:46] here are Hyojeong's and my contact details. I have also included my colleagues, Junkyu and Si Young, who co-developed this anti-cheat pipeline with me. If you have any questions, please feel free to reach out to any of us. But, if
[34:03] possible, please direct most of your questions to Si Young and Junkyu, not me. They've been enjoying the summit while I ended up giving the presentations, so I think this is only fair.
[34:19] So, thank you everyone for listening and enjoy the rest of the summit. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.