Scaling Real-Time Fraud Detection With Databricks: Lessons From DraftKings

Summary

Real-Time Fraud Detection Pipeline: DraftKings integrates real-time streaming, machine learning, and rule-based detection within a PySpark framework.
Feature Transformation at Scale: Leveraging PySpark-based feature transformations enables rapid model training and real-time inference across historical and live data.
Shadow Mode Testing: Testing models and rules in live environments before deployment reduces risk while optimizing for sub-second latency requirements.

Scaling Real-Time Fraud Detection With Databricks: Lessons From DraftKings

Talk by Greg Von Pless, Principal Data Science Engineer, DraftKings ; Monika Hristova, Principal Software Engineers, Draftkings
At DraftKings, ensuring secure, fair gaming requires detecting fraud in real time with both speed and precision. In this talk, we’ll share how Databricks powers our fraud detection pipeline, integrating real-time streaming, machine learning and rule-based detection within a PySpark framework. Our system enables rapid model training, real-time inference and seamless feature transformation across historical and live data. We use shadow mode to test models and rules in live environments before deployment. Collaborating with Databricks, we push online feature store performance and enhance real-time PySpark capabilities. We'll cover PySpark-based feature transformations, real-time inference, scaling challenges and our migration from a homegrown system to Databricks. this video is for data engineers and ML practitioners optimizing real-time AI workloads, featuring a deep dive, code snippets and lessons from building and scaling fraud detection.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is DraftKings?

DraftKings is a company in fantasy sports, online sports betting, iGaming, and other verticals that prioritizes safe and secure platform experiences.

What fraud challenges does DraftKings address?

During major sporting events with high user activity, DraftKings addresses challenges including payment fraud, account takeover, and other security threats.

What database powers online feature serving for real-time fraud detection?

DraftKings uses Databricks feature store to power real-time fraud detection. Online feature tables co-located with ML serving infrastructure enable sub-millisecond feature lookups, critical for making fraud decisions within acceptable latency.

What is real-time machine learning?

Real-time machine learning makes predictions on streaming data with sub-second latency. For fraud detection, models must score transactions instantly using up-to-date features, balancing model accuracy with serving latency.

Which platforms support real-time feature computation and online serving?

Databricks supports real-time ML through Feature Store for online feature serving and Model Serving for low-latency prediction. Together they enable applications like fraud detection to make informed decisions instantly at scale.

Full transcript

`0:00` Introduction and context

So, uh, and I are going to be presenting about our journey in migrating a, uh, sort of homegrown custom, uh, streaming feature engineering system to a data bricks based one. Um, and we've organized around a few different kinds of lessons we've learned along the way that we wanted to share. Um, yeah. Hello. I'm principal software engineer at DraftKings. Uh, my areas of expertise are distributed 30 time system built on top of patch flank, spark and data bricks.
And my um current focus at DraftKings uh is our internal fraud detection system and different streaming initiatives. Uh. I'm a principal data science engineer. Been with Giraffe Kings for five and a half years, I think. Um and yeah, I do the stuff that it says on there. Okay. So, um, for anybody that hasn't heard of us, DraftKings is, uh, um, fantasy sports, online sports book, um, I gaming and more, uh, company, number of verticals.
Um, and for the scope of this presentation, we're going to be talking about the some of the work that we do to try to ensure that we're providing a safe and secure um, platform for our users to engage with our products. Um, and so yeah, we've got a lot of people doing a lot of stuff. uh especially, you know, being a sports betting platform when um when we have major sporting events happening, um there's a you know, a lot of user actions happening and we want to be able to um you know try to mitigate some of these kinds of problems, payment fraud, account takeover, uh things like that.
Uh and for the scope of this talk, we're specifically focusing on um introducing machine learning models to um kind of like critical flows. So, you know, if things take too long, it actually slows the user experience and we don't want to have that happen. Um you know, outside of that, there's a whole lot of different things that DraftKings invests in to try to make sure we have the safest platform we can. Um to provide a little context before we go through all the lessons that we're going to talk through, we're going to give an overview of the what the homegrown system kind of looked like and then what we've evolved it into.

`2:42` Architecture evolution

Um so I get to talk about the old stuff and then gets to talk about the new stuff. So, uh, when we first looked at building the, uh, the machine learning part of all of this, it was something that, you know, we kind of had different teams working on different parts of it. Um, and so the system we ended up with reflects that a little bit in the sense that, you know, the data science was data scientists, I can talk, I swear. um went off and you know built their model training pipelines and engineered their features and all this stuff and ended up with models they trained that went into our data bricks registry and then separately our engineering team built out the stream processing and you know live feature computation all that stuff.
Um the outcome was was fine. It worked. Did the thing we were able to you know introduce a machine learning model live in in one of these flows. Um but what we ended up with highlighted on the slide here is um there's points in the system where we kind of had to manually synchronize and make sure things were do being done the right way or else we could introduce the dreaded offline online skew and then suddenly the performance metrics we got with offline testing didn't match exactly to what we were seeing live and we had to keep going back and figure out why and lots of iterations.
Yeah. So we decided to build and we decided to move towards a new architecture which was built on uh data bricks. Um the architecture ensured that each event that we actually get is uh processed consistently and whenever we need we can basically rerun the whole flow and reproduce everything. Um what was interesting for me was that usually when you onboard projects and systems on data bricks you expect all the new cool technology and your teams are going to be quite uh motivated about how they would work on that platform.
But what it turns out is that we also needed to adjust some of the processes that we have in order to um in a way make the team teams work better on the platform. So here I have outlined a few of the most important points that we covered and I'll just give a brief explanation and then we are going to talk through those uh in the next slides. So first and most important thing we wanted to address the problem that just mentioned regarding the data skew and the different pipelines.
So we decided to unify the process and introduce one single pipeline that does the same transformation um both for the batch and for the streaming data. Next um just because we in a way merged the work between the data scientists and the engineers we saw that we actually had to address that problem and we um merged their backlog as well. So both teams had to work towards the same goal and uh yeah achieve the work on the same platform.
So uh the next thing that we did we wanted to be able to use that uh unified pipeline in a way that it both populate the offline feature store which is used for training and validation and at the same time the online feature store which is used during the serving phase. That way we reduce the features skew and yeah we have we have achieved basically zero uh zero uh features Q. Next in order to achieve all of that we needed to ensure that each event that we actually uh get in Kafka is actually persistence into the data delta lake uh the way it was originally produced.
So basically no changes in its schema. So whenever we needed to, we could replay and back view and come up with new features that uh we can we can use. All of this resulted in a system that was built in for replace back test and it was really easy to basically um back test challenger models against the champion one on production. And this way we were always sure that whatever we had on production is it's the best model that we uh we had and as usually happens when you do such a big change a new frictions pop up and you have to work towards those.
Uh one of it was it turned out it's not enough to merge the backlog of the data science and engineering teams. going to talk about that one in a minute. And also we needed to refactor parts of our invent instrumentation in a way that we had to ensure that all the streams carried the right context the context of the domains uh that we had. Um Rick. Cool. All right. Lesson one save the right data. Um if the lesson sounds really obvious it's just because we've done such a good analysis.

`8:04` Data capture strategy

Um, so really what this is about is um figuring out the data that needs to get captured so you have the right training history uh can be more complicated than it seems up front. Uh, and so what we started with was a bunch of existing microser systems that um, you know, some of them talk to each other through like synchronous REST calls, some of them are talking to each other by sending events on uh, some kind of event queue. um you know they have their own dedicated databases and um in order to get the data that we were using for a lot of our machine learning work and data science work and things like that um our data engineering team make sure to sort of I like to say scavenge the data I think some people would call it liberate the data um but you know get the data out every everybody's databases through uh you know change capture um and for a lot of use cases like this provides us everything that we need.
You know the systems capture all of the important transactions and things in their database and then we're able to build off of that. Um but for the kinds of use cases we're talking about here, some of the features that we needed to develop were um you know kind of like they have to be fresh, they have to be low latency and they are you know related to events that just happened somewhere in the system. And so then what we run into is uh sometimes the services weren't writing the events to their databases as soon as they happened.
And so we didn't know in the historical data when did this event actually get created. So now we have to start subscribing to events that they're publishing and harvest those. Uh and then we run into, you know, between what's being written to the databases and what's being published to event cues. Sometimes that didn't actually have every single thing that happened in the system. Um there were some things that happened that we had never had a business reason to record before.
So it just didn't get saved. But then it turned out when we're trying to reconstruct what was happening to present really good training data to the model, we needed to know that stuff. Um so now we have to start capturing like when do these serviceto-service calls happen. Uh and so yeah, the summary here is essentially when you're dealing with high freshness, low latency, and like short time windows, um it's not enough to know kind of like what each service did, but you need to know exactly how the system as a whole processed it because delays and other things inherent to the distributed system impact what did anything look like at exactly the time the model got called.
Um, the answer to this is to try to like push your engineering teams to embrace more streaming first patterns and get ahead of things and try to get them to publish and record important events. Um, so that you'll have that data before you need it. And then when you find out that you don't have it, um, it never hurts to have a really cool data engineering wizard around who can go, we had a guy that went and scrambled through all the data and managed to like recreate history through, I don't know, magic.
It was cool. So yeah, next lesson is how we built those unified pipelines or the unified training and inference pipelines. Um so in order to achieve this we wanted to follow the capo capa style which is like it has proven uh to work. So for us in that case the batch is just you know historic stream. So basically we can we can reprocess it as it should be uh processed if it was uh part of the event event stream. Next we depended a lot on the unified spark API.

`11:42` Benefits of unified pipelines

So um as mentioned having two different pipelines it's really hard to implement and keep in line uh between the teams but actually having this unified uh spark API it's really easy to basically make the teams work together they share the same code reviews the same code so it was just easier to uh to help each other next with the fastback fields and the replay testing we were able to uh we were able to uh do fast uh back tests and uh let's say new and emerging uh fraud patterns that we we thought we we seen.
We could really fast basically reprocess them and um have them surfaced and decide how to proceed from that point on. And last but not least, the developer velocity. Imagine that having two different teams working on absolutely the same requirements but the one team working on the uh u on the data that's uh processed as batch and another one that's actually processing everything in real time and using totally different framework.
It turned out it's really hard to actually align those teams to prove that uh the code bases that they had actually produced the same results to taste it twice uh to have a process where you can actually be sure that the results are absolutely aligned and they don't drift too much part. So yeah developer velocity was um uh we definitely saw improvement there. and why it matters. First of all, the hardest to debug nastiest problem in the ML systems uh feature skew, we basically right now we have it.
Yeah. Uh it dropped to zero for us. Uh next those fast experimentation loops that I mentioned. Yeah, it's really f easy for us to just run the same pipeline or come up with a new pipeline with new uh theory and just prove it by reprocessing everything and see the results. Next because we uh keep full history. So every event that we need for our fraud system, every transformation that the pipelines did, every model version, every evaluation that the model did, everything gets snapshotted into the delta lake.
So we have all the data and we can learn from it. We can improve. So basically we can always analyze or reprocess everything and be able to see where our thresholds needs to be adjusted. uh and uh this gives a lot of confidence for our internal teams because imagine our operation teams just a slight adjustment might affect them and now we actually have a proof why we did that. So yeah, it's really cool that we can do that and we can have such a system.

`15:12` Governance and data mesh

But as it usually happens, it's really nice to have unified pi unified pipelines, but they are not a silver bullet. And if you don't have the right conditions in uh your company, having a unified pipeline won't help you solve any problem. So if you don't keep the complete uh row history and if you don't have um data mesh mindset it's not going to work for you. So here I have outlined a few risks that we identified at draft kings and how we mitigated them.
First G already hint about that one but missing or trimmed row events. So what happens no history uh features drift apart it's not possible to do back testing. it's not possible to replay. So what you should do the best case scenario keep the row data tables forever and if this is not possible at least as long as your model needs to look back if not yeah you have to backill from database snapshots or object store logs or other external sources but that process is really slow and messy it's it's not ideal next a lot of people and a lot of teams I have seen they want to keep only the aggregated um data in the landing tables.
It sounds tidy but it's um it's it's not okay just because you stop the innovation of features. You can't compute new features, you can uh compute uh uh you can't reprocess. So don't do it. If you want to let's say um if you want to be sure that you can actually reprocess and have the same functionality for your system just launch immutable append only source events before you do any batch ET or any kind of transformation on top of the data.
Next, it sounds really nice to have a centralized team who actually can implement all of your transformation pipelines. But what we noticed is that when we keep that um ownership in a central team that this is not the only uh tasks that they have to work on, uh it causes delays for the domain teams and um it's really hard to keep up with what actually the business wants. So what we actually did is we embraced a little bit the data mesh mindset.
So we pushed the ownership of the row and created tables together with the contracts and SLAs to the domain teams and then the centralized team actually started implementing schemagnostic pipelines that land every message as is in the lake and um we use it from there. Then the uh downstream curation modeling and contractual governance we delegate as appropriate. It could be the centralized team or our uh domain teams.
Next I think um we've heard a lot of lectures about that one or presentations um yesterday with governmentness and lineage. If you can't trust your data just nothing works. So it's hard to prove that what you did is actually working. it's performing the way that you expect some audits may fail. So for us to actually enforce u better governance and lineage we had to introduce the right um tooling in order to know that uh we have that one.
So for us it was the unity catalogs, the schema registry for the streams and automatic lineage um capture which we used to trace down every model evaluation down to the to the stream that actually yeah helped helped it. All right, talk a little bit more about teams and backlogs. Um yeah, so another uh I guess pain point we had run into especially developing the first system that we talked about um was not necessarily aligning the backlogs between all the people involved in trying to build the end toend streaming system.

`18:58` Cross-functional collaboration

Um, and what we end up with is, you know, we've got the data science team over here and they've got this is one thing they're working on and but it's not the only thing they're working on. And then they start developing features and then they need those features to get put into the streaming pipeline. But the people building the streaming pipeline supporting that, you know, that model building effort is not the only thing they're working on.
And so there's a lot of, you know, waiting that ends up happening when you're trying to get everything synced. And then there's a lot of status meetings where somebody says, "What's going on with that?" Oh, we're blocked by them. And then, you know, when managers hear the word blocked, their like hair goes on fire and everyone freaks out. Um, and so as we were building this new system, something that we we went out of our way to do was um make sure that everybody involved in sort of uh implementing all the stream processing and the sort of feature publishing to the online feature store.
Everybody working in developing what features we need and training the models. We're working off shared set of priorities, shared backlog so that as we needed to hand work back and forth, code reviews, validations, all this stuff. Uh it was a lot easier to make sure that you know progress didn't stall. Um and then the other thing that happened here was um going to talk about it a little bit more, but I'm going to steal some thunder. um our you know streaming engineering team actually went out of their way to go learn Pispark so that they could write all of their transformations in code that was going to be easier for our data scientists to read and review and edit um which was really cool.

`21:08` Migrating to Apache Spark

So yes um as you can imagine uh it took us a few iteration until we got to the point where we are right now and the original MVP started with data science having their own codes as mentioned and engineering starting working on this absolutely new and amazing um project where they had to implement real-time transformations. Originally we picked uh Microsoft Orleans which is um an actor-based umn net framework and um we were targeting the freshness of the features.
So in other words, we wanted to be uh optimized for latency and our goal was to achieve subsecond for our state stateful computations and we did. But then we decided we decided thinking what's next how we want to build our platform looking at the at the system holistically and we realized that just having some features optimized for latency is not enough and we wanted to evaluate more dimensions. I have listed uh the most important uh ones for us and by looking at those we realized that probably um yeah we might need to uh migrate to something else.
So stateful computation and latency optimization basically with release we had a subsecond while with um Apache Spark we achieve the same stateful transformation um at about two two and a half seconds. It's lower but it was still it is still within our ouras but then when we come and start discussing started discussing the replay and back testing and we realize that it's really hard to implement it with top uh sorry with Microsoft Orleans uh it ended up with a lot of custom code and um hard to test.
So once we actually decided to fully migrate to Apache Spark it was quite easier. Spark supports uh uh native functionality like watermarks or time travel with the delta table. So it was something that came out of the box for us. The next one features skew as mentioned before two code bases two teams uh twice testing the same thing and on top of it we had to come up with a data skew monitoring process to ensure that the features never drift apart which is quite a heavy process and it's it's not fun.
So with spark we run the code to a single pipeline uh which start processing first historical data and then it continues in real time. zero data skew. Easy as that. Next, the ecosystem and ML tooling. As you know, um as you may know, um Microsoft Orleans is really good framework, but it just it doesn't has that much of a support for the uh ML systems. file with Spark and data bricks. We have all the functionality that our data scientist needed like MLflow support uh feature store uh hooks everything that any a data scientist would need and then the unified batch and streaming API.
It's really cool because now we can process uh with the same API data that we have in S3 buckets or Kafka and you don't need to change the code. It's the same uh data frame uh operations. Um and la last but not least, I feel like has something for the. NET language, but data scientists don't like. NET. So, it was easier to migrate um all of our engineers to uh Pispark rather than doing vice versa. And as it usually happens, you know, developers, they are always hyped about any technology and we were quite um excited about Spark, but once we actually started digging and working with the framework, we figured out it's not perfect.

`25:00` Managing framework limitations

It just uh it just does the work for us. So I have listed a few of the um pain points that we have and some of the fixes that are headed our way. Uh so one interesting thing was the watermark s on idle streams. So as you may know the global mark watermark only progresses whenever you actually have events that arrive and even though this is not the case at draft king because we have quite a lot events in our streams sometimes it may happen that some event there is silence for a bit.
What uh this results in um window that never gets evicted because the watermark doesn't progress and basically your uh state lingers indefinitely until another event arrives. It's not a big deal but yeah sometimes sometimes happens. The next one is the microbatch latency floor. uh with all our tests and um our production systems we just noticed that achieving a subsecond transformation uh with the current state of the framework it's just not it's just not possible but the cool thing is that um Apache Spark already provides the realtime mode and u I expect that ceiling to disappeared there yesterday there was amazing session about that one uh next one no state schema evolution Um imagine just um doing a simple change like uh widening the value type of um in a stateful operation.
Try that deploy production and you will get query fails with the state schema not compatible. uh our workound right now we usually wipe the checkpoint out and then we start from the beginning and yeah we have a workaround but with the new version of spark this is already addressed and uh there is schema evolution. Uh next what we noticed is that there is no parity of the current spark version between the uh scala API and the pi spark.
And for example people who actually use spark with scawa they may know that there is a way to actually uh provide a userdefined initial state uh in the kawa api by using uh flat map group with state or map group with state. with Python this is not not an option. So what we actually did did is or usually do is we always start processing the data from the beginning then we have the state and we continue in real time and this is by the way also addressed in the new Spark version.
So we are quite excited. And uh last but not least uh up to now there were no uh timers per per key which is uh something that we heavily depend on in our implementations. We have a workar around with um state uh group uh timeouts which works but it's not in ideal and the code is a little bit complicated but with the new um transform it state API we already have timers and we are quite excited about it. We already tested it and yeah it's amazing.
Okay, my last lesson is the tradeoffs between having a system that's optimized for latency and a system optimized for cost on data bricks and what we've learned. So initially uh we were optimizing our system for latency and we tried to have the same um with the new database/spark implementation and we were able to achieve uh a few seconds end to end feature freshness for both stateful and stateless uh pipelines and we were quite heavily utilizing uh futon clusters uh which we used to optimize our persisting of data into the delta lake and we managed to achieve with those uh uh jobs around the second for that persistence but what actually happens is that it turned out futon is quite expensive for system that are built uh to run 24/7 and system with our traffic.

`28:39` Optimizing latency and cost

So we had a new goal. Let's reduce uh 50% the cost by 50% and still try to keep the latency inside um our SLAs. So after three weeks of optimizations we actually did it. we achieved uh we got our DBUS down by 78% the uh underlying cloud infra uh was down by 70 and we still managed to keep our SLAs's but we increased the latency by three times and what we actually did first of all we disabled for uh all the jobs that you used it which reduced the uh the cost by three uh three times and increase the latency two and a half times.
So you see it works. It's just expensive. Then we started right sizing our instances cuz before uh we were using beefy um compute optimized clusters which were giving amazing performance and then we started right sizing them. We uh did a lot of horizontal scaling instead of vertically. We also uh downsized our driver nodes. So we tried to keep utilization no more than 80% because if you keep it more uh if you have it more it actually increases the latency additionally.
And after doing all of this, what we noticed is that for the stateless uh jobs, there were uh there was no uh latency bump uh no no change in the latency while for the stateful computations we saw um yeah between half a second and a second latency bump bump and maybe you don't know or you know I don't know uh but um having simple queries um just adds additional DBU cost uh to your bill. So what we did is we um merged simple queries into a single pipeline and this additionally saved uh save yeah from the cost and I know that recipe is not exciting but what we actually did first of all uh what we actually recommend is first of all ensure that your code is well optimized for us this wasn't the case because uh we involved our data bricks team uh since the moment we started started implementing the pipeline.
So our code was well optimized since the very beginning. But when you actually start testing, ensure that you start uh small and only scale when you see that you don't have the performance that you expect. Use futon only when it really accelerates your um aggregations and we actually need it when you actually need it. uh tag and surface uh every cause that you have on data bricks and do regular uh tuneups whenever the data especially when the data volumes change.
Uh upgrade to spark 4. This is part of our uh backlog already because spark for your time mode uh aims for subsecond transformations and always cooperate with your data bricks team because they definitely know how the things work and they're always happy to help us and I think this is this is

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.