Real-Time ML Features for Payment Fraud Detection at Billion-Event Scale

Summary

Checkout.com processes 9 billion payments annually and makes fraud block-or-pass decisions within a 100–200 millisecond window, using real-time ML features built with Spark Structured Streaming and sliding windows on Databricks to detect attacks as they unfold.
The three-pipeline architecture — training, serving, and monitoring — maintains perfect parity between offline and online features, preventing data leakage and ensuring that models trained on historical data behave correctly in real-time production environments.
Moving from hourly batch feature refresh to real-time streaming features enabled Checkout.com to detect 20% more fraud and unlock an eight-figure revenue opportunity, with TTL-based state management and event-time watermarking key to production stability.

Real-Time ML Features for Payment Fraud Detection at Billion-Event Scale

Payment processors like Checkout.com face an extreme machine learning challenge: detecting and blocking fraudulent transactions within 100 to 200 milliseconds. Traditional batch feature pipelines, refreshing hourly, leave massive blind spots during fraud attacks. Checkout processes 9 billion payments annually, and every second of delay in fraud detection represents millions in potential losses. this video explores how to build real-time feature pipelines that keep models trained with accurate, point-in-time correct data.
Learn how Checkout.com engineered real-time features using Spark Structured Streaming, sliding windows, and transformWithState on Databricks. Dinis Peixoto and Matt Slack detail the three-pipeline architecture (training, serving, and monitoring), show how to maintain perfect parity between offline and online features, and discuss techniques like TTL-based state management and event-time watermarking. See how sessionization and real-time mode move beyond sliding windows for even faster fraud detection, and discover why this approach identified 20% more fraud and unlocked an eight-figure revenue opportunity.
🤝

Chapters

FAQs

How does Checkout.com detect payment fraud in under 200 milliseconds?

For each of the 9 billion payments processed annually, Checkout.com runs it through multiple ML models within a 100–200 millisecond decision window, using real-time features computed via Spark Structured Streaming on Databricks. These features capture behavioral patterns from recent transactions, enabling models to detect fraud attack patterns as they develop rather than hours later with batch pipelines.

What is feature parity and why is it critical for fraud ML at Checkout.com?

Feature parity means that the features computed during model training exactly match those computed in real-time serving, ensuring the model encounters the same data distribution it was trained on. Checkout.com treats parity alongside freshness as one of two keys to success, using a dedicated monitoring pipeline to detect and alert on any drift between offline and online feature values.

What is transformWithState in Spark Structured Streaming?

transformWithState is a Spark Structured Streaming API that gives engineers full, custom control over state management logic beyond what standard aggregation operators provide. Checkout.com uses it to implement complex session-based feature computation that goes beyond sliding windows, enabling more nuanced fraud pattern detection with finer control over state lifecycle and TTL management.

What business impact did real-time ML features deliver for Checkout.com?

Moving from hourly batch feature refresh to real-time streaming features enabled Checkout.com to identify 20% more fraud across their network. This improvement translated into an eight-figure revenue opportunity, protecting merchants and their customers across the $300 billion in annual payment volume that Checkout.com processes.

Full transcript

[00:08] Hi everyone. So today we'll bring you a session about real-time machine learning features for payment fraud and what we learned from shipping them at checkout.com. Uh and we have billion event scale. So yeah, quick round of introductions. I'm Dish. I'm a machine learning engineer at Checkout. I worked at the
[00:23] intersection of um machine learning systems for fraud and our machine learning platform. and I'm together with Matt from data bricks which spe special specializes in low latency streaming pipelines at scale and has been working with us for the last six months. Before getting into the session, I want to u
[00:39] present what checkout does. So basically we are the payments infrastructure for some of the largest brands in the world. Some of which you might be familiar with like eBay, uh Uber and Spotify. And basically we try to solve some of the common challenges that you have in
[00:54] payments such as improving um acceptance rates which basically means ensuring that the payments are accepted but also prevent trying to prevent fraud for our merchants trying to protect our merchants and their customers. We processed over 9 billion payments last
[01:09] year which amounts to $300 billion and we've seen across our network over 1 billion cards. So you get an idea of the scale at which we operate. Um for every single payment that we process, we run the payment through multiple machine learning models. Some of them are used to assess the risk of the models, the
[01:26] risk of the payment. So basically like whether the payment is fraud or not. And this is going to be the focus for today's session. Okay. So payment fraud happens really really quickly. Some of our merchants are constantly targeted by uh fraud attacks and during an attack they can
[01:42] have thousands of payments every single second. And for each one of those payments, we have to decide in a 100 to 200 milliseconds window what to do with that payment, whether to block it or not in case we think it's fraud. If we get the decision right, we are able to flag over $100 million every single year uh
[01:59] in fraud. And for this, we need to build real-time machine learning features so that our models can use to detect this type of attacks. And this is this type of features is what we are going to focus on today. Let's go over an example of a fraud attack. Uh, obviously like
[02:15] this is not a real example, just to show like how it works. Um, and some of the the things that we see in our day-to-day. So, it's 2 a.m. and a fraudster has a bunch of cards, stolen cards, and a script. The first step is basically to run those cards over smaller merchants with just tiny like
[02:31] charges to see which cards work versus the cards that don't. And the second step is taking the cards that actually work and run them through bigger merchants and try to buy anything that is resellable for a fraud for a machine learning model for fraud. What's really useful in this
[02:46] case is how many cards we are seeing from this fraudster based on the IP that they are using. And obviously like for every single payment, we have to decide really really quickly what to do uh with this feature. And we can't really run a query on a database that tells us how many cards were used for a given IP
[03:03] because we only have 100 milliseconds to decide whether to do this, whether to block the payment or not. And we use hundreds of features in our machine learning models. So you can see in the chart on the right that one option to do this is to run um jobs in the background that keep updating those features. And
[03:19] the top row is if we had like a datab bricks batch job running every hour. And you can see that it wouldn't be really that useful because every like throughout the entire attack the feature value would have been zero and instead would would be really useful for the model was basically to refresh that
[03:34] feature value every second or every few seconds. And this is what we can achieve with real-time features or a streaming data bricks a streaming job running on data bricks. We already had a feature store at checkout and it was actually like quite we had a lot of features and we had we
[03:51] were covering a lot of scale. So we had over 500 feature definitions, five billion feature values being served. We have extremely high throughput because we see a lot of payments and for each payment we might look up multiple features. So we had over 1,00 1,000 kpo and the latency was there as well. It
[04:07] was extremely quick. Basically every single lookup is a key value lookup which happens in under um 10 milliseconds. So this shows that serving the features was not necessarily our bottleneck. Our bottleneck was computing them upstream and making sure that they are fresh enough so that we can um u
[04:24] identify the fraud attacks that we were seeing earlier. So shipping real-time features is actually two problems. I think freshness is the first one that always comes to mind. Basically like how fresh the features actually are. Uh the second one which is actually what mo most teams
[04:41] forget is that the features that we are serving in production need to match exactly what the model was seeing when we train the model. And we actually decided to focus on parity first. And we went from for sliding windows which almost give us parity for free while we know that there's only so much that we
[04:57] can do in terms of freshness with sliding windows. But this is our this was our first approach to try to deliver value with real-time features in production. Let's go over the end to end process of shipping a new feature with sliding windows. So basically a data scientist comes up with an idea and we can use the
[05:14] same example as before which is the distinct cards that we've seen for a given IP in the last five minutes. Then we created some sort of like DSL that they can use to define the feature. So they use a decorator feature job. They write a Python function and in that Python function they need to provide what's the source of events that they
[05:30] are going to use for the feature. In this case it's a Kafka topic with payment events. Then what's the entity that we are aggregating values for which is the customer IP the type of window that we are using which is a sliding window with size five minutes that slides every 30 seconds and then the aggregations or features that we want to
[05:46] compute within those windows which in this case would be just the count distinct of cards that we've seen for this IP after the feature definition they can run a bunch of research so they backfield the the historical values point in time they attach a notebook run a few experiments training a model
[06:03] evaluating the model and trying to iterate to find which features are best for the model. Once they are happy with the feature, they can ship the feature to production and we automatically provision all the infrastructure that they need and we actually provision three pipelines for them in production which we will see next. Provisioning the
[06:19] infrastructure is actually done with data bricks assent bundles. So everything is automated and the only thing that they have to do is to write a bunch of Python code and we get the infrastructure automatically provisioned for them. The three pipelines that we provision for the data scientists are so the first
[06:34] one is where everything starts training a model. So we have a pipeline that keeps computing every single day the point in time correct values for the features that they just defined. Then we have a pipeline for serving which is basically what's going to keep updating the values once we take the model to production. And the last one is a
[06:50] pipeline to make sure that the features that the model was trained with are also the the same features that the model is now seeing once taken to production. Let's focus and deep dive into the training pipeline or rather the the pipeline that is used for the training features.
[07:06] So basically the most important thing in the offline feature computation as we call it is to make sure that the feature definition that the data scientist wrote is going to be is is going to match what we have in the offline feature computation. So basically that the features are computed with point in time correct values that they can use to
[07:23] train their models. So for instance like for a payment that happened at 214 our offline feature computation job needs to ensure that the feature value that we have for this payment is the same that the online feature computation pipeline would share with the model when we run inference in
[07:39] production. This is the end to end architecture of the offline feature computation. So basically it all starts with a a source of events. In this case, we have an event source as a delta table with historical payment events. And then we have a feature engineering pipeline
[07:55] which runs in batch every single day. It's basically like a datab bricks workflow. And then we use the feature engineering client provided by datab bricks to create an offline feature table in the feature store. So it's also a delta table with historical feature data. It belongs to the unity catalog and it's automatically registered in the
[08:11] feature store as well. What this means is that then consumers can attach their notebooks uh to a cluster and read the features from the feature store and use this to do their research or our production training pipelines can apply the same approach to get the features from the feature store and do model retraining automatically.
[08:28] Once we have the offline features in the feature store, training a model becomes really easy thanks to the tooling provided by the provided by data bricks as well. So we use their client to provide it like a um a data frame of payment events and a label which in this case is whether the payment is fraud or
[08:44] not. We create a feature lookup based on the feature tables that we have in the offline feature store and we can we are able to create a training data set based on this. Once after creating the training data set we can just train a model and once we are happy with the result we promote the model to the model registry and we are ready to serve the
[08:59] model. So now that we covered the training part and we have a model ready to deploy, we need to focus on the serving part. So basically now we need features that the model is going to see in production and we can use those features to decide whether the payment is fraudulent or
[09:15] not. And while for training we need historical um features for to train the model for inference we just need the latest feature values. So for instance for the fraudster IP we only need to know how many cards you used they used in the last five minutes. In production we use sliding windows. So
[09:32] basically as you can see in the chart every row uh in the in the diagram is basically like a new window which spans over the the time of like five minutes and every 30 seconds we slide the window. So we end up with a new window. As soon as we get new events which are the dots in the in the diagram they fit
[09:48] basically uh the window that is open at a given moment in time. So you can see that in the first window the count would be or the number of events in the window is two while in the last one is actually five. And we do this for every single entity that add um that basically has an open window. So we see a lot of due to
[10:05] our scale we see millions of IPs in a short amount of time. So what we see is actually like millions of um um entities different entities times the amount of windows that we can have open in a given moment in time which in this case is 10. And the reason why it's 10 is because
[10:20] basically like in five minutes divided by 30 seconds we get 30 10 active with windows. The things that are also worth mentioning here is that we always keep a watermark to account for late events. Uh and every time a window closes, we write
[10:35] the the the feature value into our feature store. So this is basically translated into a new feature in our feature store. This is how the end to end feature computation works for online. So it all starts again with a event stream. But in this case instead of a delta table with
[10:52] historical events we need what's happening in a given moment in time what's happening right now. So for that we consume the events from a Kafka topic and then we have the feature engineering pipeline running as a sparkly clarative pipeline computing the sliding windows and we use Rox DB as a state store as
[11:07] well and finally we write this into an online feature table in Delta because delta is not appropriate for inference because we can't really do single digit lookups in delta. We have a downstream pipeline which consumes from a streaming delta table takes the
[11:23] features and publishes it to a dynamob table. This is actually that we are working on replacing. We want the feature engineering pipeline to write directly to Dynamo so that we don't have the step in the middle which is adding a bit of um where we are losing a bit of freshness.
[11:42] Okay. So we now have a training pipeline, we have a serving pipeline. The only thing left to cover is a monitoring pipeline to make sure that the two are providing the same results. When it comes to monitoring, we basically focus on two different things. The first one is parity to make sure that what the model is seeing is the
[11:58] same thing in production is the same thing that the model was trained with. And if the if it's not, you can see in the chart um below on the left side, it's basically like we expect most payments to have um the same features in production and in the training data set. And we do this through feature logging. So basically like for every single
[12:14] payment that happens we log the features that were used for that payment and then we compute the same payment in uh our training data sets and we check whether the features match or not. And you can see that if the the match percentage um starts decreasing over time. We know that there is a bug the there's skew
[12:30] between training and serving and we need to create an incident because probably there's something wrong with the model. The model is not being used for what it was trained to do. And then we can also have drift which is it has nothing to do with training serving skill but the distribution of the features themselves are changing. So for instance for the
[12:46] feature that we are talking about about distinct cards used by an IP in the last five minutes we can have an average of like five and then out of nowhere we see that the value is incre the average is increasing to 20 or 30 and in that case it means that there's something in the world that is changing and our models
[13:02] probably need to be retrained because they are not accounting for the new fraud patterns that we might have in production right now. Okay. So what did we learn from taking all of this to production? So the first one is the reason why we picked parity
[13:18] in the first place. So essentially we had data scientists creating their own backfield logic to backfield the features and in this case they were for instance counting the number of declines for a customer in the last five minutes and they were interested in the charge request of at that happens at time three at time t and you can see that there's
[13:34] actually like three payments that happened before that time that were declined. So everything sounds fine right? Well, actually not. A decline might happen a few seconds or even minutes after the payment was requested. And in this case, actually like only one of those three payments was declined
[13:51] before the the charge request at time t happened. So we had leakage because the model was accessing a decline event that only happened after the payment that we are taking into account. So in this case, the real value should be one instead of three. And the model was learning from something that we wouldn't have access to in production. With the
[14:07] new decorator that we created, basically the training and serving pipelines are set up automatically and we ensure that there's no leakage because we always monitor training serving skew and we the features should always match between online and offline.
[14:24] Something else that we also realize is that the features are only as fresh as the last write to the store. And if we are using sliding windows and writing every 30 seconds if we don't get any events for a given IP or something like that we stop writing into Dynamob then what happens is that in the example that you are seeing there the last right
[14:41] wrote five as the the let's say the count but then we never uplate the value so we end up with a stale feature because there's no more windows for that given entity and in order to solve this we introduce the DTL so basically every single row carries an expired at which
[14:56] is basically going to expire the record once the record no longer makes any sense. And because we use Dynamo DV, the AWS doesn't really ensure that the expire at is going to be the record is going to be deleted right away. Sometimes it might take a few minutes or even a few hours. So we also look up in
[15:12] the lookup layer check whether the record was expired or not. And as a rule of thumb, we try to ensure that the TTL that comes with a row basically is higher than the slide interval. And the reason for this is that if it was lower, we would probably end up with new values in between windows. So if the TTL was 15
[15:29] seconds, there's another 15 seconds until the next window. So we would end up with new values, which is not ideal. And probably the biggest learning for us was understanding where the freshness budget um or where we are actually spending the freshness. So the first
[15:44] thing that comes to mind is the slide interval. So if the windows are only sliding every 30 seconds, that's where we are spending most of the time. And that's true, but there's a lot of other things that actually come into play as well. One of them is the computation of the features themselves, which can take up to three seconds. Then the watermark or how long we wait until we have until
[16:01] we for late events. And last but not least, the thing that surprised us the most was actually publishing to Dynamob because every 30 seconds we are publishing millions of rows to Dynamo DB. This can actually take a while and it becomes a compromise of the right capacity units that we provision on
[16:17] Dynamo and how fresh we want the features to be published. One thing that happened for instance in production was that the rights were queuing because we don't we didn't have enough capacities in provisioned in Dynamo and we had no idea about that. So Dynamo was threat throttling the rights were still ending
[16:33] up there but we were losing a lot of freshness because of that. In order to solve this, we introduced basically monitors around every single step in a pipeline and we now monitor the end to end freshness as well. And basically like it becomes just like sliding windows provisioning capacity in Dynamo becomes a compromise of costs and
[16:49] freshness. So we can provision really high right capacity units but then we are spending a lot of money to ensure that the features are are published rather quickly. Where did this get us? So after p after taking the model to production with
[17:05] sliding windows built um on top of the new platform, we actually were able to identify 20% more fraud which contributed to a 10 million more fraud identifier every single year. And because our fraud models are not only used for fraud but also other uh payment optimizations that actually where we can
[17:22] ensure that the payment is accepted, we actually are looking at eight figure revenue opportunity for us which was pretty cool. But this was sliding windows and sliding windows can only get us so far. So the next step is to go under the second uh freshness um to to
[17:38] ensure that we are able to detect and act even faster than we are with sliding windows. And this is where data bricks help us pushing for even more freshness.
[17:54] Yes. So as we've covered like we're limited with the freshness to this model of the sliding window. So, if you're familiar with sliding windows in structured streaming, you can specify a time interval and then the events as they arrive, they're bucketed into those time windows and that time window slides by whatever time interval you want. So,
[18:11] that's got some limitations on it. For example, if you get uh certain events pattern of events arriving, as we'll see on the next slide, then you might miss potentially a fraud event. Obviously, the benefit of sliding windows is that
[18:27] it's quite easy to have parity between your online and offline features because it's easy. It's a very simple computation to compute the offline features and then they'll match the online features. But the problem you've got is if you've got a very spiky if someone's doing fraud attack in like a
[18:45] few seconds or something like that and your slide window is maybe 30 seconds or something like that then you could have a fraud event which is happening over the boundary of the slide and therefore you miss actually that this fraud event is happening because half the fraud
[19:01] attacks happen in one window and then by the time it's slide the window is moved on to the next slide then the other events are in there. So you miss the action that's happening. So let's see how this looks in practice.
[19:16] So yeah, of course, so the the guy who's doing the fraud isn't nicely planning his fraud attack. So it all lands in one sliding window and so you pick it. So there could be a scenario particularly if the sliding windows if the slide is quite large um where you miss a fraud
[19:33] attack or you fail to detect because obviously you're measuring a threshold of distinct IP addresses or device fingerprints or whatever feature you're whatever um entity you're using. So you could miss that because the the guy who's doing the fraud is not going to be
[19:48] so kind as to make sure all his attacks land in the same sliding window. So how do we avoid that? So the way to avoid that is to move to what we call session windows. So the idea with a session window is that as the you see a
[20:05] new IP address, you effectively start the session and then as you see new events arriving for that IP address, you're going to just grow the session. So that session is going to grow. Potentially you might see the five, you know, five payments within that window
[20:21] as you can see. sorry, six payments on the left hand side. So you'd actually capture the attack burst in one session. And so in this case, the benefit here is you've captured the full extent of that fraud attack within one session.
[20:39] If there's then a gap, so if it was a normal payment, obviously you'd see a transaction and then you'd see a gap and then another transaction. So you can then expire that session because it doesn't look like fraud. It's normal behavior. With the session windows, you're then going to capture that fraudulent activity within one session.
[21:00] So, for those of you who are familiar with structured streaming, you'll probably know that there is a built-in session window. And so, that offers some benefits as we've talked about over the sliding approach. So, in the sliding approach, every payment event will go into its
[21:15] separate window depending on um how often you're sliding. So if you're sliding once every 30 seconds, um, so if you're sliding 30 seconds and you and you're moving the slide every five seconds, then that event is going to be in six buckets because it's going
[21:33] to go in the every time it slides, it's going to move into the next bucket until eventually it falls out of the window. So that creates a huge number of windows. So it's not very efficient either. So the advantage with the sessionbased approach is actually we manage that
[21:48] session window and all those events effectively are handled together. So we're not exploding the state which happens in the left hand side because each of those windows on the left hand side is going to have to be stored and maintained in in the state of spark structured streaming.
[22:04] But there's a few downsides with the built-in session window. Some quite major downsides. One is you don't get control over when events are emitted because of course if you see a fraudulent transaction start to happen a pattern of fraudulent
[22:20] events you probably want to send send an event as soon as you trigger the point that you think it's fraudulent. So say you get five transactions coming in and maybe on the fifth one you go okay now we're going to alert because this has passed the threshold of distinct I distinct transactions for this IP
[22:37] address. So now we're going to flag that and we're going to write that to the feature store. But you can't do that with the built-in event. You have to wait for the session window to close. So that's a bit of a pain. Another problem with the session window is that you can only define um uh you can only define it
[22:55] for one um one time period. So for in this example, we want to have multiple time periods. We want to be looking back over 30 seconds, over 5 seconds to see the amount of distinct events. Well, how many distinct transactions are there in the last 30 seconds? How many distinct
[23:11] transactions are there in the last 5 seconds? They become separate features in the feature store. So, we want to be able to manage that ourselves within the um the state management. So, how do we do that? So, we use this
[23:27] feature called transform with state. And this is an API that's been around for a while. It was originally called apply in apply with state in pandas. There's been numerous iterations of it and now it's kind of settled on this name of transform with state. And what transform
[23:43] with state allows you to do is fully customize how you handle new events arriving. So when you see a new transaction arriving in the input stream, you've got full control over what you do with that and you've got full control over how the state is
[23:58] managed. So for example, you could maintain a list of um for a given IP address, you can maintain a list of the time stamps when that IP address has been used. Um or you can do it for for example for a given device. You can see
[24:15] how many IP addresses are being used for that device. So you've got full control over the state and how that works. And also you've got control over when you emit an event. So like I say, you can look at those scenarios where
[24:31] perhaps you see five transactions arriving. Okay, that's tipped the threshold. We've seen five transactions in the period of two seconds or something like that. Okay, so let's emit an event because we want to update the feature store. This looks like this is fraudulent.
[24:51] The downsides with this approach, there's always downsides. The downsides with this approach is we've now moved away from the very kind of comfortable sliding window approach which made it e easier to maintain our online and offline store. So we've got to think hard about how we calculate the offline
[25:06] store because now the the size of that session window is not fixed time boundaries. It's based on when the events arrived. And if we're doing like distinct counts over the period of 30 days or a year. we're looking back potentially 30 days in transactions for a given IP address then that's
[25:23] potentially a lot to calculate. So let's dive into transform with state. So the benefits here like I say is it gives you full control over the state we store. So we can maintain that distinct
[25:38] list of time stamps used for a card, distinct list of IP addresses. We can also maintain a count of how many times we've seen transactions to this IP address. And it supports both event time and processing time timers. And that's quite
[25:54] important because we want to use the event time. So in this case, the event time is the time stamp the transaction happened. The processing time is when the event arrives. Obviously if there's any lag in the data coming from our upstream um system like CFKA for example
[26:09] our pub subsystem then the processing time is going to be inaccurate. So we want to use the event time to calculate because you might see if you see any lag in the stream then that's going to potentially miss a fraudulent event because the events might be strung out when actually when they really happen
[26:25] the transaction time they were actually close together. you get full control like I say of how you process the input rows and also how you handle timers. So timers are quite important because we can use timers to say for a given IP address if nothing's happened for a
[26:42] period of time. We can then say okay we'll downgrade this we no longer think this is a fraudulent transaction potentially at that point we can also say uh because we've seen not seen any transactions for 30 seconds for this IP address. We can then uh fire an event
[26:58] off to say now um clear the value in the feature store because we now no longer think this is a fraudulent um IP address. And the same code can run in stream and in batch. Now, like I say, because the
[27:16] paradigm is no longer moving from sliding windows with fixed time intervals to like a session based approach, it can be quite expensive to do it in batch because you have to replay the events in order. But the benefits are, as we can see, is we're more likely to capture those fraudulent
[27:32] events than we were with the window with the sliding window based approach. So here's some examples of the type of state you can store in transform with state. So value state is just a single counter. So in this case in an example
[27:48] of the fraud use case, we'd store things like um how potentially like some kind of velocity counter like just a count of uh transactions that we've seen maybe the last time we saw you know the first time we saw a transaction within this session. So you can kind of do then
[28:05] creates uh features which are based on like averages um of how many transactions we're seeing per second that kind of thing. Um for distinct counts obviously we might want to count like how many devices we've seen for an IP address if the if the fraudster is um
[28:23] uh you know is trying different devices for the same IP address or vice versa. Or we might want to see often fraudulent behavior detects things like do we see uh the geoloccation moving around much faster than we'd see an actual customer moving around. For example, have they moved a 100 miles between uh two
[28:41] transactions which are very close together. So you can do those kind of things. We can store in the list state and potentially also the map state. We can store that kind of information which allows us to compute every time a new event arrives. We compute for example things like what's the um like what's
[28:58] the number of distinct devices we can do things like geo geoloccation those kind of things. So it's very flexible in what state you can store with the sliding windows and the built-in session window operator. You just don't have this flexibility over how you can store the state.
[29:18] So here's what happens when an event arrives. So the new event arrives and we'll immediately look at that event obviously decode it from CFKA and then we make a decision based on that. We're going to update the state. So for example if a row comes in um we're going to obviously
[29:36] log the time stamp because we want to count uh keep a count of distinct events. we might want to store like the um the IP address or the device fingerprint for the card and then we can do a count of distinct on the fly as soon as soon as that event happens. we
[29:51] can then make that decision over is this fraudulent or not and then we can fire off the uh if we decide it's fra fraudulent we can fire off an event which we can then push into in this case Dynamob
[30:11] at the same time when we see a uh anytime we see a new event we can decide what we're going to do about the timer so if it's a a new transaction we might say okay we'll extend the timer or We might you know do some other behavior like say after a period of time we want to check whether this record now needs to be evicted from session. So for
[30:28] example if we fire a set a timer to be 30 seconds in the future then if there's no events for 30 seconds for that um card fingerprint we can then evict that record from state.
[30:48] So this is kind of the future real time feature pipeline that we're trying to get towards. So end to end events come in from CFKA they're in protobuff so we des serialize with them and then we push them into transform with state and this is doing the processing at scale of all these events
[31:05] and at the moment they're being written to Dynamo potentially they're well delta first then to Dynamo as Dennis talks about we're very soon going to be writing them direct to Dynamo skipping that Delta offline step and we're also exploring writing them to lakebase
[31:25] Like I say, you can use this same approach with offline features. Yes, there's potentially performance implications. There's way you can you can optimize around that by chunking the data into by looking at like the gaps in the data. So for a given IP address, there's no data for um say 10 minutes, then you can you
[31:42] can calculate the offline features more efficiently. If you try and obviously replay all the data historically through transform with state, it's going to be very slow. So you have to be quite clever about how you do the offline features. But fortunately, it's the same API. So it is the same API that you use
[31:58] to compute both. Okay, one thing we still need to do is so Dennis talked about creating this DSL which allows the data scientist to use like a common language for creating the um the features for for the sliding
[32:14] windows. we need to take that DSL and and apply it to this transform with state approach as well. So the data scientists can use the same language for creating um features driven off this sessionization based approach
[32:34] that so another added bonus of using transform with state is that it's supported in real time mode. So what's real-time mode if you're not familiar with it? Well, essentially, historically, Spark structured streaming has always been based around this microbatch-based approach, which effectively means you've got all these
[32:52] executives consuming data in stream in parallel, and then every so often they all have to stop, agree on where they're up to in the processing, and then start again as you can see in the top diagram there. So they all every so often have to stop and there's a few problems with
[33:07] that is the one obviously that every time it stops it it adds latency but also if there's any skew in the data for example for a given merchant or for a given um uh card fingerprint or IP address there's suddenly a lot more transactions than there would be for another one then you get skew and one of
[33:23] these tasks can then hold up the other ones so all the other ones are sat there waiting which means you could miss a fraudulent event for one merchant because there's an extra 5x transactions for another merchant. Actually, this approach is sat there chugging through all those transactions
[33:38] for this merchant while it's missing fraudulent events on the other one. So, that's the top approach is very heavily uh can be heavily affected by skew. With real-time mode, you don't have that problem because all the executives are free to basically consume these
[33:54] transaction events as fast as possible. And that means we can also get the latency down. So with structured streaming as Dennis showed earlier on that um that freshness slide is that there's always this processing element with you've got the sliding windows of
[34:10] 30 seconds and then you've got proc calculating the feature which takes one or two seconds and then writing it to Dynamo which maybe takes seven seconds. So with real time mode we can really compress the time it takes to calculate the events down to potentially like one
[34:27] or 200 milliseconds. And with certainization, we're also compressing that sliding window. So that sliding window is now reduced. So with this with this approach, ideally we can get the freshness down to once we do the the right to Dynamo, we can get this uh down to potentially under five seconds.
[34:46] And so this is where we're going. So the sliding windows is in production and we're looking at implementing the sessionization features. And once we've got that stable, then we're doing the real-time mode. So that's where we're going in in in the future. And we've been working on these for the last few
[35:02] months to to PC these. Okay. Thank you very much everyone. I hope that was useful and interesting.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.