Ultra Low Latency Streaming for Real-Time Feature Engineering

Summary

Traditional feature engineering requires maintaining two separate systems — Spark for batch and near-real-time features plus custom bespoke pipelines for sub-second freshness — creating logic duplication, training-serving skew, and infrastructure complexity.
Real-Time Mode in Apache Spark Structured Streaming eliminates per-batch overhead to deliver millisecond-level feature freshness, while Lakebase (Neon-powered) serves as the low-latency online feature store backend for real-time serving.
Coinbase achieved an 80% P99 latency reduction with sub-100ms results using Real-Time Mode, and DraftKings has migrated from Aerospike to Databricks to unify their offline and online feature pipelines in a single architecture.

Ultra Low Latency Streaming for Real-Time Feature Engineering

Feature freshness is critical for fraud-detection, ad tech, real-time recommendations, dynamic pricing, etc. Historically, ML engineers have had to maintain two separate systems for feature engineering, leveraging Spark and Spark Structure Streaming for batch/near real-time features, while building bespoke pipelines when they needed sub-second feature freshness. In this video, we'll learn about Real-Time Mode in Apache Spark Structured Streaming, the engine behind the Databricks Feature Store, and how companies like DraftKings are using it to power real-time features with millisecond-level freshness. You'll learn how to pair Real-Time Mode with Lakebase to build a state-of-the-art online feature store and leave with a practical blueprint for a unified feature engineering platform that eliminates logic spread across different systems and shifts the focus from infrastructure orchestration to semantic feature modeling.
Talk By: Abhay Bothra, Sr. Staff Software Engineer, Databricks ; Jerry Peng, Staff Software Engineer, Databricks ;

Chapters

FAQs

What is the two-system feature engineering problem?

Most ML teams maintain separate systems for offline batch features used in model training and online real-time features used for low-latency inference, which forces duplication of business logic and often causes training-serving skew. Real-Time Mode on Databricks aims to unify these two systems in a single Spark-based pipeline.

What is Real-Time Mode in Apache Spark Structured Streaming?

Real-Time Mode is a streaming execution mode that eliminates the per-batch startup and commit overhead of standard micro-batch processing, enabling Spark to deliver feature freshness at millisecond latency. It is available on both classic and serverless Spark Declarative Pipelines and can be enabled via a configuration flag.

What results did Coinbase achieve with Databricks Real-Time Mode?

Coinbase achieved an 80% reduction in P99 latency with sub-100ms end-to-end feature delivery using Real-Time Mode on Databricks. This level of feature freshness is critical for fraud detection and other financial use cases where stale features can lead to missed detections or false positives.

How is Lakebase used as an online feature store?

Lakebase is a Neon-powered PostgreSQL-compatible database on the Databricks Data and AI platform that serves as the low-latency online feature store for real-time model serving. In the unified feature pipeline blueprint covered in this video, Real-Time Mode writes features directly to Lakebase, giving the inference layer sub-millisecond read access without separate feature store infrastructure.

Full transcript

[00:07] Hi everyone, thanks for joining myself. I'm Jerry and my co-orker by here in our talk on ultra low latency streaming for real time feature engineering.
[00:19] To be honest, I'm super excited to kind of talk about this, you know, topic because it really materially affects my life and I think it kind of the subject also probably affects the lives of everyone sitting, you know, here. Things
[00:33] from, you know, doing fraud detection for your financial transactions to even predicting the delays of a flight you might be on are all related to basically the subject we're going to talk about.
[00:46] and how well those systems work is also related to you know what we're going to talk about today. So I'm super excited to kind of talk about how data bricks and what my team has been doing to basically help on that front.
[01:06] So what is the agenda of our talk? First I'll give an overview of what feature engineering is. Be a quick overview in case not everyone's famili you know familiar with the subject. you know talk about things like why what are features why feature freshness matters and the
[01:19] idea of this two system problem that we have typically see in you know feature store and feature engineering then we'll talk about real-time mode this is something that we recently kind of have uh developed you know recent as in the
[01:33] last couple of years developed at data bricks to really allow structure streaming and spark to deliver ultra low latencies my colleague here will then talk about how can you actually build real time
[01:47] feature pipelines on data bricks yourself using some of the things that we offer like SDB lakebase and so on and so forth then we'll also talk about if you don't want to do it yourself you want to do it in a more declarative fashion how can you use data bricks
[02:00] feature store which is another product that data bricks offers to basically do the same thing we'll then go over some customer stories some customer study case studies about you know the impact of kind of this work the material impact
[02:13] of this work on you know actual customer use cases and lastly we'll talk about what's coming next what are you kind of new releases and so on and so forth all right let's get to it
[02:27] what is feature engineering so what is a feature right you can think about you know any sort of you know system this could be a financial
[02:39] transaction system you know transactions are coming in it could be sensor data, it could be clicks from your website, it can be any of these kind of raw events, right? And what you're really wanting to do in terms of computing a feature for
[02:52] your feature pipeline is you're computing likely some sort of aggregates. You know, what are these values over some period of time? For example, what is the number of transactions made by some person over
[03:05] the last five minutes? Maybe it's also where they're located, right? Where did these transactions happen? because that's important for fraud detection. Can also be other things, right? Like my example of airport airline delay. What
[03:19] are maybe some of the sensor data? How how how full or how empty are basically the fuel gauges for my airplane, right? All of these basically come in as raw
[03:31] events and you have a feature pipeline, right? That basically computes these aggregations, these windows, and you write them to a store, whether it's online or offline. And I'll go over the difference in a little bit for basically
[03:43] these features just to go over briefly go over some of the modern use cases. I've also kind of touched up upon the classical one fraud detection you know credit card not you
[03:56] know present or there's any sort of basically kind of fraudulent um you know transactions detected multiple transactions and largely different geographic locations things like that intech you know real-time bidding you
[04:09] know um these and other things like recommendation personalization dynamic pricing all these use cases really require features to be computed with
[04:21] really sub-second latencies or there'll be financial consequences um as a result of basically missing the of more stale features.
[04:35] So typically for feature engineering you'll see kind of this setup you have your offline and your online feature stores. Your offline feature store what is it used for? It's used for training. You know you're usually using some sort of batch compute. Spark is already used,
[04:48] data bricks already used often in this kind of area and you're doing point in time correct kind of lookups for you know uh you're you're for these features. Oh, I want to basically get the these features the computed values
[05:01] of these features at this point in time. I want to join that with basically labels and then I want to train my model based off of you know that. And what is really important in those situations is not necessarily latency. It's more like
[05:13] throughput you know and cost. And then there's a on and then there is an online path. This is where you're actually serving kind of results. And we call this the kind of the online feature store is the is basically the store in
[05:27] which you have you need to compute very fresh features and you have input basically you have events coming in and you want to use those basic computed features and you want to apply your model to compute a result. This is where
[05:40] you actually say for example do your fraud detection. You have transactions coming in. You have your trained model. You want to look up basically the latest features for that and then you want to basically compute a result. This is where latency really matters in terms of
[05:54] not only serving but also in terms of the freshness of your features because you know it needs to match as accurately as you can to what is computed in your offline store.
[06:12] And you know this is just to kind of iterate there's usually you know there's two features types of feature stores right offline online but there's also two sets of pipelines typically in this classical case is you have your on if you have your offline model training flow right you load your historical data
[06:26] maybe use spark to basically compute these features you write them to an offline feature store it could be you know basically you know any sort of data table iceberg table delta table and then you uh train you know using basically
[06:38] that data set you train your model with you know the appropriate labels and then you have your online serving flow which is for example incoming transactions. You get a stream of these things. You compute basically based uh features your
[06:50] latest features uh based off of uh basically those uh those incoming events. You apply your model to basically those latest features and you actually want to make a decision or a result or in the fraud detection case
[07:02] you know whether there is fraudulent or not. So, but typically there's kind of two, you know, paths and a lot of times, especially in the past, like two separate technology stacks are used. One could be spark, one could be a lot of
[07:15] times people use maybe storm or flink in the past, things like that to uh to for your online kind of serving flow. Let me go over some challenges with feature engineering.
[07:27] The the most kind of um obvious one or the common one is this concept of online and offline consistency or skew between basically training time and serving time features that causes
[07:39] models to be you know to fail in production or to be just less accurate in production. Right? So what does that mean? It's basically uh what it really kind of boils down to I think is two kind of important points is differences
[07:53] and how the features are computed between your online and offline pipelines. Right? Like I mentioned in the previous diagram, you have two stacks essentially, right? One for your online you your offline stack and then
[08:05] you have another for your online kind of flow. You might have to implement those how you compute these those features twice and because of that sometimes inherent sometimes unintentionally you introduce basically differences minor
[08:18] differences in how you basically compute those features you know because there's different APIs for different systems or you know engineers people make mistakes right and another thing that affects
[08:30] kind of latency especially affects accuracy for your uh especially for online inference is stale features that are computed for your online store because when you train your model,
[08:42] you're doing these point in time correct lookups, right? You're saying, "Okay, seven days of I want to compute these features at this point in time, right? And I want to and these are the correct labels for basically those features. I
[08:55] want my model to basically match them. If I if I actually have stale features computed during my online path, it my offline path doesn't kind of take care or take into account this latency,
[09:07] right? It's really in a in a and and it's really modeling the the fact that there is actually no latency, right, in my offline training path. So if I actually have latency in my online path,
[09:19] it actually decreases accuracy. So stale features is what we see is what we hear from customers and users. you know, if there is latency in basically how we're computing our online features, that actually hurts their accuracy because
[09:31] it's not basically accounted for in their offline training. >>
[09:43] this talk is headed is and and a way to kind of solve some of the issues that we've talked about is you know like I mentioned before for these type of workflows there's two stacks that
[09:55] happen can we solve some of these problems if we basically do some sort of platform consolidation and we unify them right the reason why you want to use two stacks is because things like you Flink
[10:07] or custom cost for cassu or storm in the past they can deliver basically very low very high freshness right very low latency so you're using that for your online path and you're using spark for your offline path because you know
[10:20] everybody uses it and has really good throughput and it works very well for those batch use cases but can you have basically a stack that you can serve both of them on spark and on data bricks right so you know you solve the stale uh
[10:35] feature problem, right? And you al what you also what you can also solve is the inconsistencies that might happen when you have two implementations to compute your same to to compute those features.
[10:53] So this is a quote from actually one of the folks that um one of the from Coinbase, one of the companies that started using real time uh mode because of they had a data freshness kind of issue. they have a feature freshness issue. And by basically saying I I'll
[11:06] read the code here. By leveraging real-time mode and spark structure streaming, we're able to achieve an 80% reduction in end toend latencies, hitting sub 100 milliseconds in our P99
[11:18] and streamlining our real-time ML strategy at massive scale. The performance allows us to compute over 250 ML features all powered by basically a unified Spark engine. So basically
[11:31] this is a customer and a user of data bricks that was basically using spark to compute both their uh bash and compute their both their offline store feed
[11:43] offline features as well as their online features using spark but they were running into basically the state of freshness kind of issue and they switched over to something I'll talk about real-time mode and they got huge latency boosts and actually it improved
[11:55] kind of their accuracy we'll discuss a little bit more about the exact use case later on in my talk, but this kind of sets the stage for me to kind of introduce real time mode. This is something that we've kind of developed
[12:07] at data bricks in the last couple of years to really augment the capabilities of you know Apache Spark streaming. So let me kind of talk about that for a little bit. So what is real-time mode? So real time
[12:19] mode is a spark structure streaming execution mode that provides latencies in the milliseconds. Right? You simply just have to flip a switch you know there's no code changes you
[12:31] know required all the APIs are the same and you will be able to you know allow your structure streaming existing structure streaming queries to process data with end to-end latencies in the milliseconds and recently we actually g
[12:44] this feature
[12:57] some context right so we have this existing what we call the existing execution modes which we call the the microbatch mode right it's already used widely for ETL use cases and you know for use cases that are in the if you
[13:09] have latency requirements or SLAs's in the order of seconds or minutes it already works very well for that a lot of people are using data bricks using basically you know spark to basically spark structure streaming to do that already what you really struggle with
[13:22] kind of with structure streaming is bringing that latency to be under a second, right? Especially it's very relevant for these kind of feature engineering use cases that want to have as low as high freshness as low latency
[13:34] as possible. And this is why where we built real-time mode. And this is the gap kind of real-time mode fills is being able to process data using the same APIs but with ultra low latency.
[13:48] Let me this is not a talk about how kind of RTM works. I gave a talk on that last year. But let me just kind of give some high level ideas of how it works, right? Um really the like idea is let's run
[14:00] microbatches for longer. Instead of basically having the idea of processing data in terms of ranges, let's let's kind of flip that on its side, right? Let's just do checkpoint intervals instead of checkpointing the amount of
[14:13] data we've processed each batch, right? So like let's run for batches for longer have long running batches and we can put checkpoint uh basically at the end of those batches and the other thing we did is run stages concurrently right if you
[14:27] know how spark works spark runs basically one stage at a time those are basically kind of boundaries you have to finish one stage and then you run the next stage that's also a very big uh
[14:39] part of why it's hard to get the latency down to in the milliseconds so the other thing we did is let's run all stages you concurrently and have data be immediately uh ready to be processed downstream once you know it's basically
[14:52] completely processed upstream right so those are the higher level ideas like I mentioned before in real time mode instead of basically checkpointing or saying every single
[15:04] kind of these micro batches okay like let's basically process this one and then checkpoint we're going to say we're going to run for x amount of time you know basically it becomes a checkpoint interval we're going to checkpoint and then we're going going to start again. So this is what really minimizes that
[15:18] per batch kind of overhead and gets us that low latency. And the other thing I mentioned is stage scheduling is instead of basically running one stage at a time like that's how it's done currently in Spark, we're going to run all stages at
[15:32] the time at at a time. And what we're going to do is we created this new shuffle implementation. And it's kind of an incremental shuffle, but we call it the streaming shuffle is that when the stages are running concurrently, the the upstream stage is able to send
[15:45] data immediately down stage downstream to be consumed and processed. So that everything your whole pipeline is your whole like query is running in like a pipeline fashion. Once this data is
[15:57] processed in your mapper stage, you send it down immediately, you know, in in more or less a pointto-point kind of fashion to your reducer and you're processing that data immediately. So all of these concepts coming together, we're
[16:09] able to get, you know, really low latency. And one of the other super cool things I think about all this is, you know, we're not telling you this is a new kind of API. You don't have to change your code.
[16:21] All you really need to do is specify that you want to use real-time mode. And then you can use it by simply changing the trigger right there. One line change, you're able to now run in a
[16:33] different kind of execution mode, get that low latency without, you know, majorly changing kind of this, you know, the query, the structure streaming application that you probably spend a lot of time writing.
[16:48] And here are some like uh you know benchmarks. I won't go to them in detail, but essentially what I want you to take away from this is with real-time mode, you're going to get an order of magnitude, you know, boost in your latency, if not more, by basically
[17:00] using, you know, real-time mode. One of the other cool things um that recently we released is RTM on Spark declarative pipelines. Um, Spark
[17:12] declarative pipelines is basically another way for you to implement kind of your incremental processing, your stream processing in a declarative fashion instead of kind of a more imperative way using the dataf frame API. So recently
[17:25] we added kind RTM support on both classic and serverless for STP. So you can take advantage of all the cool things STP has to offer like versionless, you know, basically zero time down zero
[17:38] downtime maintenance and things like that as well as autoscaling uh with RTM and whatnot by again simply just like enabling it with a simple couple lines of configs.
[17:55] Yep, I talked about that as well. Um now I would invite my uh colleague up here, Abbe. um he's an expert on feature you know engineering and he'll talk about kind of the next topic is how to actually build you know feature engineering pipelines with kind of the
[18:08] stuff you know we've uh I' I've covered so take it over thank you J if I talk too fast it's because I've had a lot of caffeine just ask me to slow
[18:22] down uh show of hands how many people here have tried to build feature engineering platforms at their respective companies. Okay, that's a lot of people. And how many here are data practitioners or data
[18:35] science practitioners who are training models but need these platforms to be present for their day-to-day modeling work and uh machine learning life cycle. Okay, this is the right audience. You're
[18:47] in the right room. And I'm here to tell you that if you have one takeaway today, it's that data now has all the pieces necessary for you to build your own feature engineering platform. if you so desire. But there's also a managed
[19:00] offering that I'll only talk about very briefly, but there's other talks at the conference that will go into a lot more depth uh to those. Uh one more call out is uh RTM is really a game changanger. Realtime mode that
[19:13] Jerry talked about. Uh it's a huge innovation in Spark and if you if you don't know about what it is, I'd really encourage you all to try it out. It's super easy to use as Jerry mentioned and uh there are some wins you can get just
[19:25] very very easily right so how do you build your own uh feature platform feature pipelines on data bricks so this is just a skeleton of what a feature engineing platform needs to do uh and we
[19:37] talking only about like the the online pipelines we're not talking about the offline feature engineering part here but you have a streaming query that's reading from some streaming source let's say Kafka Kinesis event hub so forth and it's doing some computations Mostly it's
[19:51] simple enrichments, aggregations and writing it out into some online feature store. Traditionally what I have seen is tends to be some radius Dynamo uh there's issues about those that I'll
[20:03] talk about and then you have a feature serving endpoint that's going to go and look up that feature store and uh you ideally want latencies that are sub 50 milliseconds if not lower.
[20:15] Now uh as I talked about uh in the previous slide, one big component in real-time features tends to be aggregations. So you are looking for uh values like hey like how many transactions did this user have in the
[20:27] last 10 minutes, right? Uh and similar aggregations that are looking at essentially things that happened between now and sometime in the past, right? And for these sort of aggregations, I'm going to argue that tumbling and
[20:40] sliding windows which have historically been present in Spark are not sufficient. The main reason is that they are they are
[20:52] increment and uh what you desire is something that looks back from right now to some point in time right and uh rolling windows will either give you like a stale value or uh or a partial
[21:04] aggregation which is not what we're looking for and there's an issue with sliding as well which I'll go get into in the next slide. What we have developed is a native rolling window operator in streaming that's going to look up that's going to do exactly the aggregation between now and now minus
[21:16] some duration and emit the final value into some sync. It's going to discard anything that's not in the current window as well as when when when your time wall clock moves forward it's going to expire events from the tail of the
[21:29] window so that the window is kept up to date and this will happen even if there are no new events coming up coming into the system it's going to keep them fresh. So let me try to use the pointer.
[21:41] Can I use the pointer? Okay, doesn't work. So tumbling windows as as as we can see in the visualization are essentially uh have fixed boundaries and you either get the window with the blue and the green dot or the one with the
[21:54] yellow dot and both of them are not good for the t equal to 12 at which I want to do a lookup. You can have sliding windows with very short slide durations, but the problem is that then you're like you're computing a lot of windows that
[22:06] are not really necessary for your use case and you're creating uh your streaming query is going to be slower. You're going to create a lot of load on your online store. Ideally, what you're looking for is a system that computes only the latest rolling window and
[22:18] discards all events that are no long that that are not relevant for the window and keep sliding the window forward as your wall clock is moving on. Right? And this is exactly what the new rolling window operator gives us.
[22:32] Okay, now we have rolling windows and we have realtime mode. What the the only missing piece here is now an online store and a couple of years ago I would have said that hey post P postcrist is not the platform for to use as an online store but the amazing team that we
[22:44] acquired from neon and the engineering that we're continuing to do at data bricks is really pushing the boundaries here and there are several reasons why a lakebased architecture that we have is actually the right one for an online feature store. The big argument I'm
[22:57] going to make in the favor is is cost. As anybody who's tried to build their own online feature store using Reddius and Dynamo knows that these systems can get extremely costly and the big reason that happens is because they were
[23:09] architected for a read heavy workload. So you have write once read 10 times and the cost actually maps out. Whereas feature engineing pipelines tend to have a very reverse sort of characteristic but they're very very right heavy and
[23:21] you want a system that doesn't price you or doesn't penalize you for writing too much. a post the big downside that postquest also used to have was it it would not scale very well for very high throughput rights and that's a problem
[23:33] that the that uh the lakebased team at database has solved uh we put out a blog recently about how we are doing that to support very high throughput rightes into posgress the disagregated storage and compute
[23:46] architecture really allows us to reduce cost for the customers obviously you can choose uh to to have like larger caches that gives you lower latency but there's a lot of room to play there it can autoscale to zero when you have very low
[23:58] low uh low volume reads and the great thing is that it's all governed by unity catalog so you have a s simple like a single interface where you govern your online and offline stores right now you want your feature
[24:12] engineing pipelines that you're building in data bricks to write to lakebase historically the option would have been you you use uh the for each sync in u in structured streaming and then you do a
[24:24] whole bunch of stuff yourself you're doing connection pooling you're doing your own batching rate limiting failure handling and so on you figure out like you're hand rolling your SQL all sorts of security issues can come from there uh you're dealing with authentication
[24:36] credentials and so on right and u it it the performance ultimately is going to depend on how good you are at dealing with uh these knobs I'm happy to announce that we have recently developed
[24:48] a native JDBC streaming sync in uh structured streaming and that's also available in structured streaming and SDP that does all of this for you and all you have to do is just set your sync to be postgress in uh in jobs and uh
[25:02] there's an even simpler API for STP and we take care of all of this for you. We do batching, we do connection pulling, retries, back pressure, so on. Uh we automatically allow you to do upserts. We very soon we'll be launching conditional upserts. Uh we do primary
[25:15] key detection. O is managed through Unity catalog. You don't have to worry about anything and uh it's extremely simple API and the performance is great, right? We we are able to get like sub 20 sub 10 millisecond reads from lakeb like
[25:28] we said pretty high scale. And this is how the end toend architecture now looks like when you if you build this on data bricks. You have your RTM pipeline running on SDP that's using uh rolling window operations tight
[25:41] to end streaming sync that happens to be leg based where you can do upserts uh and do very low latency reads. Everything is end to end exactly once and uh this this is an architecture that you can go and build today. Now who's
[25:53] excited about building this themselves? I have an even better better uh answer for you which is don't build it. We we have packaged all of this into the data bricks feature store and that's essentially the project project I have
[26:06] been working on uh since I've joined data bricks uh a year ago to essentially develop these these pieces but then package them into the databicks feature store which is as simple as this you write your feature definition you have a
[26:18] feature engine client and this feature definition simply specifies what's my source what's my column what's my entity key key and what aggregation I want to do and you say materialize features
[26:30] And in the background, this takes care of all the magic for you. Provisioning the lakebased instance like setting up the STP pipeline with the right code. So you don't have to worry about getting the correct features in the sync uh and their operations, metrics and so on.
[26:44] Which means uh just to get into a little bit more detail when you do materialize features, it's going to actually set up three separate pipelines for you. It's going to set up one offline pipeline for creating a replica of your real-time
[26:56] streaming data into delta which is going to be useful for training use cases and back fills. It
[27:08] or your online store. This is using the batch uh the batch engine. And then the third one is going to be the real-time pipeline that's running continuously reading from Kafka writing to lake base and using RTM. Right?
[27:27] So in one in just one declarative API you get something that's declarative managed governed end to end in one platform. So you no longer need to maintain a separate flink pipeline for your online features, a Dynamob cluster or Dynamob uh endpoint for your online
[27:40] features or a radius cluster or so on and everything is in one place, right? Um managed on the datab bricks umbrella. So that's uh that's about feature engineering and how you can do it
[27:53] yourself or use the databicks feature store. Now uh we're going to talk about two customers who essentially used the databix platform to get great and for feature engineering use cases and got great results in terms of what freshness
[28:05] they were getting and how simple life was uh for them. Right? So who has heard of Coinbase or not heard of Coinbase? Maybe that's a good question. Okay, everybody. Uh so Coinbase is uh a block
[28:17] uh cryptocurrency trading platform and uh obviously as you can imagine they have uh use cases like fraud detection, risk uh personalization uh that they want to do for their customers all of
[28:29] which they want to make as real time as possible because they want to offer great customer experience. Now they were running into severe issues when uh last year at DIS I had a meeting with their team and they were running into severe issues with because their
[28:41] models were their features were just not as fresh as they wanted them to be and as a result they are seeing that their model is not performing very well and that's coming from the online offline skew and they are pushing like their team is now tuning knobs and spark and
[28:55] clusters to figure out how to get the most juice out of the box and despite all of this effort they're only able to get just around 850 900 millconds of uh tail latency.
[29:07] Over the last year, we have worked with their team to get their features to be subund milliseconds in in freshness. And as a result, they have done measurements which show that there are there's 99% consistency between their online and
[29:19] offline features. And they have done all of this while reducing their cost by almost 50%. Which is a huge win for them. And they they just recently published a case study with us talking about all the wins that they got and how simple it was for
[29:32] them because all they had to do was change the trigger type from microbatch mode processing time trigger to realtime mode and that's it. Right. And so a simple agents were able to help them do the authoring and they were able to do a
[29:44] roll out extremely quickly once they did all the validation themselves. and they like they were really going to migrate start migrating some of their critical pipelines from Spark to another system which which we saved them by just
[29:57] uh off like with the with RTM as the offering right another customer is make my trip which is uh the India's largest uh online travel agency uh think of it them like Expedia and they offer uh essentially
[30:12] solutions for c consumers and B2B for uh corporate travel and similarly similar to uh Coinbase. They have use cases for personalization uh and other uh they also have agent use
[30:24] cases which is awesome right and their architecture essentially look like uh look like this where uh one context is that for B2C use cases they want to have super real-time freshness whereas for B2B use cases they were more they were
[30:37] comfortable with slightly higher uh lag and as a result the solution the team was moving towards was to have two separate pipelines one of them running on spark for the B2B use case and the other use like running on flink for for the BT B2C use case for very low
[30:50] freshness. But thankfully again to RTM they were able to centralize on a single platform which does the the the computation of their features for both B2C and B2B use cases. Looking into the
[31:02] architecture, they are essentially running uh they're using aerospike as a system where they store last end visits for a user and they were using uh the the spark for each operator for doing that computation and writing it to uh to
[31:15] aerospike. And similar to Coinbase, they were really struggling with their performance. there's like over 1 minute of 1 minute in P99 latency u and really like
[31:29] stainless stainless issues and moving towards a direction where they maintain separate systems uh which obviously increases the risk of skew that's Jerry talked about if you have separate systems maintaining online like online offline features just like that
[31:41] if you have the same feature being maintained for B2C use case in one system and B2B use case in another system has the same risks obviously with the with the ops overhead of uh maintaining these two systems. Now with real time they're able to get like a
[31:54] sub50 mcond p99 p50 and sub uh half a second of p99 which is almost uh over order of magnitude better for uh for their use case and they able to get a clickthrough rate uplift of 7% which is
[32:07] a huge win for their business. This is direct business impact uh not just technical uh validation and they're able to do all of this on a single spark engine. they just take a union of their B2B data, B2C data, push them into the
[32:19] same pipeline and write it out into their aerospike right now looking ahead. So the we've talked a lot about uh real-time mode the the things we have developed with the streaming sync uh rolling window
[32:32] operations. So let's look ahead where we're going with uh streaming and fe like supporting uh feature engineering as a database platform. So the top section is all stuff that we have done. We have done RTM unstructured streaming
[32:45] is now GA rolling window operator is private preview uh although it's already powering the data bricks feature store if you use that stream stream joins are uh present in RTM uh we just released those RTM and SDP is private is public
[32:58] preview and uh writing to postgress is also uh private preview in both the platforms uh on SDP and public preview on streaming and the databicks feature store is uh is going to be public preview shortly as well right now the
[33:11] team is really like one of the big problems we have heard heard from customers is u hey can we have autoscaling right and we have heard that message loud and clear so we are bringing autoscaling to SDP RTM uh on
[33:23] running on SDP the stateless is is going to be private preview like public preview very soon and uh stateful will come uh next quarter uh the native sync
[33:35] is going to be uh public preview on SDP soon and uh we're also working on enabling for each sync in STP in case you still want to do a DIY Y feature store where you don't want to write to lake base for some reason but write to stuff like radius or dynamo.
[33:49] That's it. U Jerry maybe you can come up here and we can take any questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.