Real-Time Fraud Detection with Databricks Feature Store

Summary

Coinbase built a real-time fraud detection platform on Databricks Feature Store, achieving 100–200ms average feature latency and a 95% P99 improvement over legacy batch systems to protect thousands of transactions per second.
Databricks Real-Time Mode on Structured Streaming powers streaming features with declarative APIs, delivering a 16x freshness improvement while a features-as-code approach with Git and CI/CD integration improves developer velocity.
AI agents accelerated migration from Coinbase's legacy feature system to the unified Databricks platform, reducing development cycles from weeks to hours, and Lakebase provides workload isolation for reliable production serving.

Real-Time Fraud Detection with Databricks Feature Store

Real-time fraud detection requires features served with low latency and high consistency. At Coinbase, processing thousands of transactions per second demands sub-100-millisecond freshness to block attacks before funds move. this video reveals how Coinbase scaled its fraud detection platform using Databricks Feature Store, achieving 100-200ms average feature latency and 95% P99 improvement over legacy batch systems.
Learn how Databricks Real-Time Mode on Structured Streaming powers streaming features with declarative APIs for features-as-code. Discover how AI agents accelerated migration from Coinbase's legacy system to a unified platform, reducing development cycles from weeks to hours. Explore workload isolation in Lakebase for reliable production serving, and understand the architectural principles, freshness, consistency, and cost efficiency, that support fraud prevention at scale.
🤝

Chapters

FAQs

What is a feature store and how does it fit into a fraud detection system?

A feature store is the bridge between raw data producers and ML model consumers, ensuring the same feature calculations are used for both model training and real-time inference. In fraud detection, it serves precomputed features — such as transaction velocity and location signals — to the model at inference time with sub-200ms latency.

How did Coinbase achieve sub-200ms feature latency for real-time fraud detection?

Coinbase used Databricks Real-Time Mode on Structured Streaming to serve features with low latency, replacing legacy batch systems that could not meet freshness requirements. Lakebase provides workload isolation so production serving is not affected by analytical query load.

Why is training-serving consistency important for machine learning fraud models?

Training-serving skew occurs when features are computed differently at training time versus inference time, degrading model accuracy in production. Databricks Feature Store ensures consistency by using the same feature definitions for both offline training and online serving.

How did AI agents help Coinbase migrate its fraud detection platform?

AI agents accelerated migration from Coinbase's legacy system to the unified Databricks platform by automating parts of the development process, reducing migration cycles from weeks to hours. This allowed the team to move faster while maintaining production reliability.

Full transcript

[00:09] Hello everyone. Welcome. My name is Daniel. I'm a platform engineer from Coinbase. Today I'm joined by my teammate Austin and Michael. Our talk is titled with right features, right time. In the next 40 minutes, we're going to help you to
[00:25] go through the model model rising the real-time fraud feature serving on Databricks. Inside the fraud detection domain, having a good model is not sufficient. Getting the right feature at the executive the decision is a mate is also
[00:40] crucial. So the goal of for this session is show you the share the reference architecture and also the playbook for the migrations so you can bring back to your team. Regardless you are work for the fintech
[00:56] domain or any other ML environments with a high traffic volume, low latency. Uh before start, I want to learn how many of you have the appearance with the feature store or I have the chance to
[01:11] use feature store in your company. Okay, I saw some people had such appearance. So we probably want to have some 101 in the following session to have a quickly go through the what's a
[01:26] related concepts in this related to this session. So before start into the fraud specific domain, I want to firstly try to place the feature store into the the enter system of the ML platform.
[01:43] So feature store is a crucial model of the whole systems. If you look at this a diagram, we place the feature store in the center of the diagram. So on the left, there are plenty of the raw data could be like the streaming input or offline tables or could be some
[02:00] API request response. On the right is a consumer who needs a feature to to populate or empower their business on their model or rule engines. The feature store is the bridge between them. They build a bridge between the people who build the feature and the
[02:15] people who like MLE or data set data scientist and use this feature to empower their business requirements. So, also the feature store has the the two type of use cases. One is the offline use cases. We usually call it
[02:31] like for preparing the the feature for the training purpose for the data scientist or ML engineer. Also, there is a online use case. It's for the reference time inference time when we try to fetching the features. When we call the live model, we get the
[02:47] executive feature we use for the data training. So, feature store builds a bridge to ensure that all these detail technology challenges can be handled smooth can be handled smoothly when and also seamlessly when
[03:06] they build the bridge between the producer who generate data and the consumer who consumes the data for their business requirements. Now, let's talk about the fraud specific use case. So, in the fraud domain, usually at the
[03:23] Coinbase current platform, every seconds we got thousands of trade flow through the Coinbase platform. But, most of them are legitimate. But, mixed into the traffic are the fast-moving and sophisticated fraud attempts. This
[03:39] traffic we try to detect or this activity we try to detect try to block them from the fraud leave the platform and protect our customers. So, inside that, the inside the fraud domain, the speed is very important.
[03:54] Freshness is very crucial when we try to get the right features. So, you can imagine that it's not just uh related to the feature quality, it's also related to the bring a big difference between when you are can block the use block the attacks at the
[04:10] time which is happening versus you only suppose the problem when the fund already leave the platform and uh attack is already happened. So, that means only a one a milliseconds of improvement on the feature freshness
[04:27] will bring a big impact on the performance of models. So, fraud doesn't really wait for the batch features. So, that means we not just we have to rely heavily on some streaming feature and real-time features that is to be
[04:43] calculated by the platform in the real time. So, some model maybe like a regular um the ML platform models, they can they are acceptable to for the minute refresh the features. But, for fraud,
[04:59] it's not unacceptable. So, the entire entire session is focused on how we get the right features at the right time.
[05:15] Okay, let's look at the way we started at the Coinbase. On the left is there are two systems we were we built before. So, one is the called Tecton, which is a third-party vendor platform currently being sunset. And also, it has over 20,000 features
[05:31] cover the batch streaming and real-time features, as well as we have uh more than 10 of the ML teams are currently supported by that. It also provides strong SLA which is a very important for us,
[05:46] especially when we have some um needs on the uh the response time as well as stability of the systems. On the bottom is a system we built by the Coinbase app. Primarily is a built a customized system we built for the batch
[06:03] features and a small amount of a Flink pipelines, which is generates a streaming features. The bigger challenge of these two systems is like we have to maintain the two set of tools as well as we have to maintain two interfaces and when we
[06:18] define the features. And uh this also drag the MLP when the ML platform team who need to uh have a maintenance two system as well as for MLE they have to learn where is a right platform they can define the features. At the at the beginning of the system we
[06:36] also have to consolidate and hydrate the data together before serve to the end users. That's all the efforts we have to maintain if we keep the two systems run together. So the destination is one single platform can handle all the complexity
[06:51] of the work uh in the one systems. So we decide to migrate to our solutions to Databricks Feature Store, which give us a unified interface and simplified like the DSL that you can define the feature in one language and it can
[07:07] quickly generate data by empower is real time uh by by language is a real time mode Spark streaming. Uh later we will cover that if you not familiar with that. And it also provide additional effort like the lineage from Unity
[07:23] Catalog and a very good isolation on the data model serving. That's a very important for us, especially if you when you build a platform, you have a lot of use cases on your platform, you you may have some very important or critical business
[07:39] critical uh features you don't want to be impacted from by the non-critical features. Isolation will be the key to that. Okay, before we jump into the concrete architecture,
[07:54] I want to cover a bit about the what the right features and right time actually requires. So, we have a three, uh, important principles during the when we architect or make the technical decision
[08:10] when we pick the feature store or build a feature store. So, first one is a fresh. The features reflect have been happened not just a minute ago. So, that means we can have the capacity uh, capability to uh, parsing and the transformation
[08:26] transformate the data into the feature we need. The second is a consistency. So, for the whole system, especially for the ML domain, uh, we have to ensure the the data you use in the training stage exactly the same as the data you use for
[08:42] the, uh, serving. If you see that the inconsistency happens, that may cause the the model degrade silently. The third one is a practicable. So, this is a to ensure we have more confidence when we really start to operate this feature platform or feature store when
[08:59] we serve the different part of use cases. Even these use cases have a different requirement on the may have a different, uh, environment like a different QPS, different type of features, different combination of features. We want to work practicable
[09:14] and P99 latency and also the cost, uh, consultant from here. So, why we call the feature store is really a harder problem is here. So, feature store have to fulfill all the different needs. Sometimes we call
[09:29] we we call them pull against each other. For example, it must have keep the low latency as the high QPS. And also it have to support uh large amount back feature back field jobs in parallel. It also have to handle like the fresh data
[09:46] well without in the without having the uh cost exploding. At same time, it must uh fulfill the provide a reliable solution for different type of use cases from the different teams. So, all this thing combined together make the feature store is really hard
[10:03] problem to resolve. So, today when we talk about how a good feature store, we are talking about having all the things together do the right. Okay, I'm going to have the the whole session in break them to in the four
[10:19] parts. The four first one is a real time. So, how would we leverage real time mode? Later the uh Paco will introduce that to ensure we have a fresh data or fresh features. The second is a write features. That's
[10:35] introduce how would I leverage this declarative API uh provided by data frame data data bricks feature store, which allows us to define the features in one place and instead of building our own pipeline to maintain manage and then maintain the
[10:51] feature generations. The third one is how we leverage all the AI to help us to quickly drive the migration between two separate systems like as I I talked before, you see that there are two legacy systems. How we can work this diverge definition of the
[11:08] feature into a single feature definition at the data bricks. The last one is how would we test and verify that's the uh the isolation of the serving can help us to protect our core business and ensure that we have very confident when
[11:25] we launch the uh different type of the models without impacting the existing critical ones. So, then I'm going to hand over to Paco to start off part one about real time and ease game.
[11:44] Thanks, Daniel. Um, my name is Austin. I am going to talk about the right time part of the right features right time. Um, and our experience migrating our features over to a Spark's new real-time mode and what we got from that. I'm going to walk it back a little bit though and talk about streaming features
[12:02] a little bit more. Um, a lot of feature platforms use batch features and those are pretty easy to do. Every once in a while you query a table and you get you can aggregate information about a user. That doesn't really work when we're
[12:18] looking at transactions that are processing very fast. And so we have streaming features that compute values continuously. For example, logins over the last 5 minutes. You can also look at transactions over the last day, behavioral patterns over whatever time
[12:35] window you need. And for this purpose, freshness is defined as the time between an event happening. So, for example, a user logging in and the service processing that, emitting the event. And it showing up as a servable feature
[12:51] for our models. There are two architectures to do streaming features. The old way is micro batch and then the newer way that I'm going to talk about is real-time mode and it helps to think about it as an airport trying to travel between
[13:06] terminals. Micro batch mode is a lot like a shuttle that goes between terminals. People line up ready to go to the next terminal and then once the shuttle arrives, they all go and they go to their next destination.
[13:22] Whereas real-time mode is a lot more like a moving walkway. As soon as you're there at the walkway, you can get on it and get to the next terminal. Microbatch is a lot simpler and it's kind of the default for a lot of feature platforms including Spark structured
[13:38] streaming. So, you might think that if you have batches happening every 200 milliseconds, that would be pretty good because then you're processing data every 200 milliseconds, but for freshness, it actually ends up being closer to 300 or
[13:54] 400 because what a batch is processing is data that happened during the last batch. Real-time mode is a new mode in Spark where you can process data. It spins up continuous jobs to process jobs and process data in parallel. So, as soon as
[14:11] the event happens, it goes through multiple steps to be written. So, here's what we get from that. We're able to do stream tiling which allows us to create tiles. So, again, looking at something over the last hour,
[14:27] rather than either writing a single aggregate value or multiple like a value for every single login, we can create tiles so that looking at an hour, we have 10-minute tiles, you only have six to process when
[14:42] you're looking at a model. We also are able to get under that sub-200 millisecond freshness which is something that we really struggled with at Coinbase when using the microbatch architecture. We were ended up having to throw a lot of resources at features where freshness
[14:58] really mattered and it became very expensive. But, with real-time mode, we could turn it on and because it's processing the data instantly, we get very great freshness. And the other benefit here is we can do all of this in the same Spark engine.
[15:14] So, we've used Flink before to process features and it is great at doing that in in real time, but being able to use Spark is great because it's the same platform, the same tool that we use for our batch pipelines, for analytics. A lot more people are going to be
[15:30] familiar with it. And so this allowed us to create low freshness pipelines for all of our features, all of our streaming features at Coinbase using the declarative framework that Michael is going to talk about later and applying
[15:45] AI to that declarative framework. Here's what it looked like in practice once we applied it. On average, our features took about 630 milliseconds to have take an event, process it, and write it to the sync.
[16:02] In a lot of cases, it was worse than that, but with real-time mode, we saw a range from 20 to 60 milliseconds up to a 16x improvement. The other thing that we were looking at is kind of the tail end of that, users that have a lot of data or the data is
[16:18] very complex, and it's taking a lot longer to process their data. Our P99 improved, so the top 1% of users processing data, that improved by 95% on the features we were testing
[16:33] real-time mode on, which is those very important freshness features. So we're able to get that data to the models and catch the fraud before funds funds move.
[16:48] And this is what the data looks like for that. So the chart on the left here is our our averages, and I I will note that the metrics are a little bit different, so it isn't quite comparing apples to apples, but it does match our experience, where the left the gray bar there is all of the features we migrated
[17:05] to real-time mode, the processing time on average, and then on the right side, where we have a median for features being processed in real time mode. And so, we do have kind of a median
[17:21] there of 637 milliseconds, but up kind of towards the P75 there is greater than a second. And then in the worst case, it was taking longer than a second for most of our features to to be processed even in
[17:36] a streaming architecture. With real time mode, even those ones that are processing a ton of data, very complex, um it's it's processed in usually under a second.
[17:52] And this is what it looks like at the transaction level. We got a lot of sophisticated attackers who move very quickly, and they will also um introduce a lot of that data to try to confuse our systems, backlog them, so that they can bypass our checks. So, a lot of it all of this is all
[18:09] happening under a second. A use an attacker might log in to an account that they're trying to take over. And from there, we're sent we're making requests for behavioral risk data, etc. for to internal platforms and also some vendors who can send us information, for
[18:26] example, about an IP address. Once we get that, it's really important that we ingest it quickly because attackers move so quickly and can start that withdrawal within a second. Without real time mode, it was kind of a gamble whether or not we would have the
[18:41] data in time. But with real time mode, we saw a lot better better results and um more a much better signal for fraud. We also saw improved consistency. So, it one thing that's important when
[18:57] training a model is that the offline store, the data that you're using for the model, matches what's going to happen in production. A lot of times that wasn't the case because there was no accounting for that extra latency for processing. And so, when we were when we migrated
[19:14] the features using that declarative framework, we were able to do so in place and cut our our training serving skew where those values didn't match in half. Some models saw up to 71%.
[19:29] And this allowed us to make better real-time decisions. The main thing I think, if there's one thing you take away from this, is that there is no trade-off here. We didn't spend more money to enable real-time mode on our infrastructure. In fact,
[19:47] like I mentioned, we were throwing a lot of resources to get the batch processing to go faster. When real-time mode was enabled for us, we actually cut our streaming infrastructure costs across our features by 60%. That allowed us to be a lot more
[20:03] efficient in how we write features because we're no longer tinkering with those configs to try to get the um get the features working.
[20:20] It also improved our config surface. We have fewer con- configurations, fewer pipelines that are built by hand to process at a specific target time. We now have, I think it's eight total configs that MLEs can pick from rather than copying and pasting. It's very
[20:36] clear what they do and what they're for. And they can use the right config. So, our MLEs are a lot more productive and I'm also more productive as a platform engineer because I'm not getting paged about things not working like they're expecting.
[20:55] Uh so, I'm going to pass it off to Michael, who's going to talk about the future of the feature platform. Thanks, Paco. Uh hi, everyone. Uh my name is Michael. I am a machine learning platform engineer here at Coinbase.
[21:10] So, as Paco had and Alistair had mentioned, we have a lot of challenges here to fight for fraud use cases here. And then my section is going to be more on like the future of the feature store platform and then what kind of benefits we are able to get out of it and also how we
[21:27] are able to leverage on the AI to help us to speed up the the process of development as well as the migration. So, so the first thing I want to mention here is the is a new concept. It's called a declarative features. So, what is it? Um
[21:45] So, this is a new way for us to define features in a way that all the feature definitions they basically exist in the forms of code. So, the traditional way which is what we call as imperative. Um So, say if you're engineer,
[22:00] you have to build everything from scratch. So, you have to be responsible for so many different components. For example, you have to write a Spark SQL query. You have to schedule it and also you have to manage back for your own. If something goes wrong, you have to like you have to do your own debugging
[22:17] process and then you have to manage everything on your own and connecting all pieces dots together. But with the new way of defining it, what we're calling as a essentially like new way of defining the features, we basically allow users to
[22:33] to make the definitions in the format of code. So, you are able to make all these entity definition, source definitions and aggregation definitions all in one place just like how you define maybe like in a like API interface. So, it also creates a really good
[22:50] isolation in terms of how you do the separation. So, you get to separate the the infrastructure logic from the business logic. So, the platform really help us to handle like all these like uh like the internal processes, like the materialization, as well as backfill,
[23:06] and also it really helps you uh like for example for machine learning engineers, they could just focus on uh how the definition gets completed, and then they don't have to worry about like the rest uh of as of how the like the infrastructure gets uh constructed and how it operates.
[23:21] Um yeah, there's another benefit. Uh for example, uh when when you're defining declarative features as code, uh it comes with all these very mature features in Git. Uh for example, you get the code review process, you get the CI/CD pipeline definitions, and also you get all the all these versioning, uh
[23:37] roll out, roll back, and also histories. Say something goes wrong, you can just very uh easily identify the issues from the historical commits uh without having to worry about just like to uh so you essentially dig into holes to look at like all these issues.
[23:56] Okay. Um so, here's a definition uh uh like an example of a declarative feature API. So, as you can see here, uh on the right-hand side, uh there's a code definition. Uh this is one of the user features example. So, it has a source definition, uh essentially where the feature lives at, and also uh it has
[24:12] entity, which is basically a joining key uh for like the like the models you can consume. And also has a definition of the features. Um it can have like the volume of the like the spending for this customer. Uh it could be like the the uh you can also specify the aggregation as sum of the
[24:28] amount. And also you can get the max timestamp as aggregation uh to get the last logging status uh of this customer. So, uh there's a couple benefits uh I already mentioned. So, the first thing uh like first the like feature definition lives in source control uh config repo, and also you can apply and
[24:45] apply workflow with CI/CD validations. You can also uh apply code review, versioning, and also support rapid development in CD pipeline.
[25:00] Okay, I want to introduce a new concept here. Uh it's called a feature velocity formula. Um this is what we always believe in. So, there are two components. The first component is is what we call declarative APIs that uh which I just introduced. The second component is what we call AI agent.
[25:15] Uh with these two combines, this really help us to increase our developer velocity and also help us to reduce the amount of work that we have to do uh from weeks down to hours. Uh with the help of AI agents, there um there are many benefits.
[25:30] So, uh for example, when we're doing the feature migration from the legacy platform onto the new one, uh we're able to leverage on AI's help uh to help us to identify some common patterns uh of all the features and then to help us to rewrite like these features from old
[25:46] platform onto the new one. And also, uh we can help Like with the help of agents, we can also uh construct like all the like all these autonomous um like like sanity checks and also all these like uh validations and and and
[26:02] also the parity checks uh like on top of that. Hey, um another very useful uh feature of uh using AI to help us uh is that we're able to leverage on AI's help uh to to come up with a migration uh
[26:19] strategy for us. So, for example, uh we want to migrate 51 feature views uh as part of our migration and then we can leverage on AI's decision making uh to help us to categorize like all these features um in terms of like the level of
[26:35] complexity. So, for the simpler feature migrations, uh we can just uh rewrite them very easily as a sparse SQL one-liner. And then with the medium complexity, uh AI can help us to do the transformation and help us to reshape the data uh from a source uh into like a stage
[26:51] eight stage table and then we can have the downstream uh for the platform to consume. And for the more complicated ones, we can leverage on the Python. So we can keep the like the Python logic to be separable and also testable
[27:06] before we have these consumed for the downstream like feature platform to consume. So here is is the migration modes that we currently support in a vision. So we have two modes. One is complete migration and then the
[27:22] other one is a hybrid approach. So the complete migration essentially for a set of features that currently it's being used by by the production model. We can say like say if we already have sufficient amount
[27:37] of confidence, we have a clear path to do the migration, we can just do a full migration to the new platform. And then if if there's there's a set of features that we don't have full confidence, we we can just keep them as is in a legacy platform. And then for the new
[27:53] platform, we can migrate some of the features partially in into the new one and then we can have the hybrid approach to have both legacy feature and also like the new features to be serving production model at the same time. So uh So the milestone
[28:09] So this model is called the transfer risk model internally. This is the first end-to-end model that we fully onboard to feature store for Databricks. And then there's one blocker that we have which is essentially the lifetime window feature and then it is identified
[28:26] as the number one remaining obstacles for us to do the full migration process. Okay, next I'm going to talk about Lake Base. So Lake Base is the infrastructural layer that supports the features online
[28:42] serving. So it has a couple advantages. So the first thing has low latency where able to get the online serving latency down to 50 milliseconds for the real time feature retrieval. This is extremely important for fraud use cases especially models needs to
[28:58] make informed decisions quickly. So the second part is auto scaling. So within Lake base it has the building auto scaling groups that helps to fight against traffic spikes. So it is not uncommon to see traffic spikes
[29:14] especially for fraud use cases. So the third thing here is the workload isolation. So for the production serve models we have different serving path so some models they're more important than others
[29:29] and in that case we need to ensure that these workloads they're being isolated say if the less critical path is being impacted having degradation we want to make sure that the less critical path is not interfering with the critical one. So workload isolation extremely important
[29:46] in this case. Okay I'm going to talk a little bit more about the reference architecture here. So essentially if you look at the top row this is how the data gets aggregated. So on the left hand side the
[30:02] real source of truth is coming from the stream data. And then the stream data gets into the aggregation engine and we call internally as the real time mode. And then once the aggregation completes that gets persisted inside of like the Lake base instance and then
[30:18] having the data available basically have an endpoint and then this endpoint is going to support all the online inference use cases. And there's another source which is the Lake house data and then that also supports for example some other features like the batch feature
[30:34] like online store compaction and also some exploration analysis used for expectation use cases for example. Okay. Um there are a couple key takeaways. Um so, the first thing is the real-time
[30:50] performance. Uh we are able to get some significant improvements out of like the like the Databricks feature store and also the real-time mode. Internally, we're able to get the sub 100 ms streaming freshness. Um it is very impressive. So, second, uh
[31:05] as I just mentioned, uh the workload isolation is very important. And then, we want to keep the mission critical fraud serving uh to be safe and also stable uh without like like any like the the partial degradation the partial degradation like having like the uh
[31:21] the effect on a full like the system. So, third thing is the develop is a is a delivery velocity. So, I just introduced you with the declared APIs and also AI agents help uh to help us to move faster and then to reduce the workload from weeks down to hours.
[31:38] Uh so, all in all, um having uh having a Databricks feature store uh for Coinbase is not just only a technological improvement, but also uh really help us to fight against fraud more efficiently and also help us to
[31:53] operate more at scale. Okay. Uh this is my session.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.