Stream into the Future: Mastering Stream processing for All Workloads

Summary

Apache Spark Structured Streaming has grown 3x in usage over the past 18 to 24 months, with over 5,000 active customers using it for use cases spanning fraud detection, IoT, real-time personalization, BI dashboards, and CDC ingestion.
This video covers key Spark streaming innovations including a new real-time mode delivering sub-100-millisecond latency without a separate engine, Spark Declarative Pipelines for simplifying production operations, and a Lakebase connector for streaming into OLTP workloads.
Three customer case studies demonstrate real-world performance: MakeMyTrip achieving 44ms P50 latency for real-time personalization, an AI IDE deduplicating terabyte telemetry logs at ingest, and Coinbase achieving 100ms P99 latency for fraud detection feature consistency.

Stream into the Future: Mastering Stream processing for All Workloads

The data world isn't slowing down—it’s streaming faster than ever. And if you're still thinking batch-first, it's time to flip the script. Join us for an overview of new features in Apache Spark™ Structured Streaming. Learn how it is enabling AI workloads for Cursor, machine learning workflows in Coinbase, and complex real-time requirements of MakeMyTrip. If you are an advanced user, this video is for you: we will showcase advanced features like real-time mode, transform with state, and streaming from hybrid sources. If you like simplicity, this video is also for you: you will discover how Spark Declarative Pipelines’s constructs can simplify stream processing.
Talk By: Indrajit Roy, Director of Engineering, Databricks ; Navneeth Nair, Staff Product Manager, Databricks

Chapters

FAQs

What is real-time mode in Apache Spark Structured Streaming?

Real-time mode is a new execution capability in Apache Spark Structured Streaming that delivers sub-100-millisecond latency without requiring organizations to adopt a separate streaming engine. It extends the existing microbatch architecture to support much lower latency profiles, allowing teams to use a single engine across latency requirements from milliseconds to minutes.

What are Spark Declarative Pipelines and how do they simplify stream processing?

Spark Declarative Pipelines (SDP) is a framework for defining streaming and batch data pipelines declaratively, removing much of the operational toil involved in managing production streaming pipelines. This video discusses for-each-batch support in SDP and how it enables simpler, more maintainable streaming workflows compared to imperative programming approaches.

How are real customers using Apache Spark Structured Streaming at scale?

Three customer examples are highlighted in this video: MakeMyTrip uses Spark streaming for real-time travel personalization achieving 44ms P50 latency; an AI code editor uses it to deduplicate terabyte-scale telemetry logs at the point of ingest; and Coinbase uses it for fraud detection feature consistency at 100ms P99 latency.

What Spark Structured Streaming features are on the roadmap?

The roadmap discussed in this video includes autoscaling for real-time mode, improvements to stream join capabilities, and replay improvements for recovery scenarios. The Lakebase connector for streaming data into OLTP workloads is also presented as a new ecosystem capability.

Full transcript

[00:08] Hi everyone, thanks for coming out on a Thursday for this talk. My name is Navnit and I'm a product manager here at data bricks. I work on stream data processing. I'm joined by my colleague here Indrajit today and we're going to do a real deep dive into the streaming stack at data bricks. We are going to
[00:21] talk about four things in this session over the next 40 odd minutes. Uh the first one is really the business case for streaming and why we think everyone should be doing this. Um second we'll talk about how Spark streaming has
[00:33] really innovated here and some of the recent innovations in Spark streaming. We'll then talk about three real customer stories. Uh these are real customers building their streaming workloads and the business value it is driving for them. And finally we'll do a
[00:46] short demo uh using some of the latest features in Spark streaming. Let's dive right in. Okay, so why does streaming matter? Gone of the days where like your data
[00:59] used to be this static stockpile that used to kind of run once a day in a batch manner in at midnight. Most customers don't really tolerate their data being processed a minute late or even a few seconds late for that matter.
[01:12] And there's real reason for this because real-time decision- making on a continuous stream of data can drive real business value for them. And this business value, it's not a niche use
[01:24] case or something niche for which you need to use streaming. It's it's actually pretty broad-based. Fraud and risk, personalization, IoT, anomaly detection. These are some workloads on which you would use
[01:36] streaming to do realtime decision- making where you're trying to make decisions in milliseconds. There are also use cases such as log and event analytics, real time uh BI
[01:48] dashboards, CDC injection. And these are use cases where you're trying to do stream processing but high throughput stream processing. And um you don't need to take my word for it. We actually have telemetry on our platform which shows
[02:01] that structured streaming is growing massively. In fact, our usage of structured streaming has grown 3x in the last 18 to 24 months. And almost every uh lakeflow pipeline or pipeline that's
[02:13] built on data bricks has a streaming component to it these days. A bunch of customers use streaming. Um these are these are some of the logos that you can see. Overall, we have 5,000
[02:25] plus customers who are actively using Spark streaming. And uh like I said, it's broad-based across industries and verticals from travel, financial services, AI native, dev tooling, all of
[02:37] all of these customers end up using Spark streaming. So the real question is why? Like why has Spark streaming become the default? And there are two reasons for this. one uh Spark is the only engine where you
[02:50] can kind of build your entire spectrum of use cases um using that one single engine from ultra low latency all the way to high throughput ETL. You can use the same Spark APIs to address each of
[03:04] these use cases. You do not need additional multiple specialized engines depending on the latency profile or the cost profile that you're doing. It can all be served using one single Spark streaming engine. And the second is
[03:16] expressiveness. So we have invested a ton of effort in making streaming itself highly expressive, intuitive to build. And we have in we have invested in the spark declarative pipelines engine which
[03:29] is which most customers use for two reasons. One, it's really easy once you once you start using it to build your streaming workloads with it. And most importantly, it really takes away the operational toil associated with
[03:41] maintaining and operating streaming workloads. And all of that is basically automated and handled by uh by Spark. Now how does this happen?
[03:54] some of the recent launches. Uh I'm going to invite Indrajit to come and talk through this section. Switching to two. So folks, I'm Indrajit. I lead engineering for streaming. Um maybe a
[04:07] show of hands. How many of you are using streaming or have orgs which use? Okay. So pretty much all of you how many of you used it for the last 10 years let's say okay last five years
[04:20] okay I'll be curious those who used last 10 years what is the technology that you used 10 years ago >> streams
[04:34] uh there was one more hand if >> we use Kafka >> Kafka a lot but after Kafka was there anything you used as a pipeline uh for processing thing. >> Um, >> no. Custom Python script. >> Okay. Okay. Yeah. I recently met a
[04:48] customer I think a couple of days ago um who said they have been using storm from 1015 years ago and they have nobody in their engineering who actually knows what to do with it because if something goes bad you can guess what happens.
[05:01] They don't even want to touch that code. Um, okay. So what I want to talk about today is a little bit about spark but also a little bit historical context on how spark streaming evolved and that
[05:13] will give you incentive to think about why you should use spark streaming and STP and also the new features that we are coming up with. So there are two parts of the story really. The first part is more
[05:25] architectural like how did we actually end up in a very different architecture uh than common streaming platforms. So I would say this is more like how we which is data bricks and spark original
[05:37] creators they picked a choice which is very different from conventional wisdom and the second one is really about hey how do you express it? So the growth that namid was showing us 3x growth how does that show up? So those are the two
[05:50] components. So the first part is conventional wisdom is if you're doing streaming let's say if you want to do ETL workload you pick a different engine let's say you use spark some other engine and then if you want to do low
[06:02] latency workload you pick a slightly different engine so there are two engines that you need to do uh for your streaming workloads and if you look at spark spark went a very different direction which is it looked at
[06:14] microbatch processing and the crux of microbatch processing is you look at data the stream that is coming in. You pick a small epoch of data. Then you process it. You take the next epoch of
[06:26] data. You process it. And then you take the third epoch of data and you process it. Does this look familiar? Isn't this almost like batch? Because batch you would just say I'm going to run this
[06:38] through the day. So I take the full data that comes through the day. I process it. This looks very familiar, right? This looks almost like batch. But truly, it still provides you streaming. And there are two key benefits of using this
[06:50] kind of architecture. The first benefit is what I said. It really looks very close to batch. And that means that anybody who is familiar with batch processing can actually jump in and
[07:02] change to streaming very quickly. And here's a code example. So the left side of the code if you see is somebody reading from Kafka and doing batch processing. The right side is basically the code with spark structured
[07:15] streaming. The only changes that you need is the keyword stream. All that the keyword stream is saying is my data is unbounded. It's not just fixed. It's just coming in and in. But the amount of
[07:28] changes you need is almost minimal, which means that a lot of users who are used to batch processing can just very easily learn how to do stream processing. So that's the first benefit.
[07:40] The second benefit is even more critical which is when you use an architecture like microbatch processing you can actually get really good performance because when you are looking at processing batches or like epochs of
[07:54] data you can work on pieces of data together and amortize your overheads. So this gives you many benefits. For example, you can do vectorization and I'll touch upon it later on where you want to go and look at a bunch of rows
[08:07] and then or a bunch of columns and make changes together. So that gives you lots of performance benefits. But people really want like even higher throughput. People want even lower latency and of course everybody wants
[08:19] things to be very expressive because you have very complex logic that you want to express in this world. And so we have done over these last few years a lots of innovations which are available now for
[08:31] any user to use right now. So one example of that is high throughput. Uh spark even like spark streaming even like 8 n years ago was one of the best in terms of high throughput. But in the
[08:45] last few years what has happened is people are writing more and more code. There's lots more data being generated. So people want the answers in the same amount of time at the maybe the same or lower cost. So you need higher
[08:57] throughput and one of the things that we built was microbatch pipelining. What it does is if you look at today's like the premicrobatch pipelining work spark would pick up data. It would do three
[09:11] things. It would first go and say let's plan how do you execute? Then it does the execution then it commits. That's for one epoch of data. Then it repeats the same thing for the next epoch of data and it will continue to do that.
[09:24] One of the key insights from our engineers was like look you can pipeline a bunch of these things because while you are doing your planning you can start planning and or maybe you are planning and executing the first epoch
[09:38] you can start doing the same thing for the next epoch. Now this is not true for all use cases because there might be dependencies and you cannot do this pipelining but for a large number of use cases this works magically and what this
[09:52] means is you are not spending time waiting you are actually utilizing your resources a lot more and we have seen in production that this can give you two to 3x performance boost and this is called microbatch pipelining it is available to
[10:06] all of you on serverless if you're using streaming but We went even further because this is one way to get performance but what about using vectorization which means that hey when you process data can you
[10:19] process all of them together and use some of the hardware extensions. So we have a photon engine which is natively written in C++ and what it does it it gives you vectorized column oriented
[10:31] processing and all of this is again available to you without any code change both for microbatch pipelining as well as photon you may just choose to use them and if you do that you have no code changes
[10:44] required but you get a lot of performance boost when we did some benchmarks on just photon here mostly these are stateless streaming programs we have seen that in many cases you get up to 3x performance
[10:57] boost just by enabling fodon. So I do recommend you to try it out especially in the serverless world. This solves the throughput problem which is you have huge amount of data and you
[11:10] want to process them quickly with low cost. But what about latency? Like if you go into the cases like fraud detection and maybe like real time IoT, you want really low latency. And we
[11:23] haven't really touched upon low latency. In fact, that's where I started the talk and said like most people assume that you need two different engines. If you have ETL workload, you might have a latency requirement of seconds to
[11:35] minutes. And in that world, Spark is great because you use microbatch mode, you get seconds latency. And that has been true for a long time. We have been really good at that space. But when you get to hundreds of milliseconds latency,
[11:49] can you actually deliver? And that was the open question for us for a while. From the last one one and a half years, we have been spending a huge amount of time to get that latency under subsecond. And we have this new project
[12:03] called real-time mode, which is now GA and available to all of you, which gives you subund millisecond latency for most workloads. In fact, in certain workloads for our customers, I think Namit will touch on some of them get even like tens
[12:17] of milliseconds low latency. The key insight over here which I'm going to talk later is you can do a lot of stuff where we were waiting for processing those places where you're waiting are
[12:30] all gone. And one of the key insights over here is that you no longer need to go to a second engine such as Flink because even those cases where Flink used to be good, we can be equally good
[12:43] or better. And this is some of those examples which shows that our performance and latency are pretty much comparable in some cases better than Flink. Um so in terms of just general insights
[12:56] uh I'm not going to deep dive on the like what are the architectural changes in real time mode because there was a talk before and there's lots of blogs also out there recently on real time mode from our team but what we did was
[13:09] we ended up having more concurrent stages. So in previous version of microbatch mode where you you might want a phase to finish then you start off the other one those phases are gone. you just have concurrent executors which are
[13:21] continuely continuously processing the data. The second thing that we have built is a streaming shuffle. So we don't have to really wait for going to disk and so on. Most of this is completely in memory
[13:34] now. And we also make sure that when you are now creating these epochs of data, each element in that data can keep flowing through the system without having to wait. And that gives you really low latency.
[13:47] And all of this is available without major code changes. In fact, in most cases, you just need one line of change, which is that red color line or reddish orange line which says it's a realtime
[13:59] trigger. As long as you do that, you should be able to see your latency go down to hundreds of milliseconds. Okay, that was one section about throughput and latency which I'm hope that I'm able to convince you that if
[14:13] you use Spark streaming, you can both get really low latency, hundreds of milliseconds and in the workloads that you care about throughput, you can get really high throughput as well. But what about expressiveness? Because many of
[14:27] you probably have really difficult business logic which needs to be expressed. Uh the first question I ask you like how many of you spend time writing streaming programs versus just babysitting it? Like what is your uh
[14:41] feeling about you spend more time babysitting or you spend more time writing code? Any answers? Babysitting. Who is spending more time writing code?
[14:55] Nobody. A lot of people are doing neither of them. But I'm hope that your team probably is working on these things. Our intuition and our experience with working with clients is that people really spend time babysitting because
[15:08] having something in production is really hard. Writing code has become so much simpler now. And most of the time that is spent is is reliability. Is your pipeline up? Is it running? If there's a problem, what do you do with it? Right?
[15:21] These kind of things. Or let's say if you have a pipeline that you're writing, how do you even incrementalize it? Right? These kind of questions. So a couple of years ago we started this project called Spark declarative
[15:33] pipelines and the key goal of that was to provide reliable multifflows which means first it is easy to express difficult business problems and the second is if you use it we take care of
[15:47] pipeline management and even cases like how things are up and running how binary rollouts happen and these kind of things especially when you're using it in serverless mode. This example is just to show you like how much easier it is to
[16:00] write business logic. Now the left hand side is basically doing a normal CDC kind of thing where you have changes upstream let's say your upstream database and so on and you want to just incorporate it in your downstream. So
[16:13] deletes and insertions very simple that itself requires you to write a lot of code even in spark structure streaming. Um it is much simpler than if you had to handw write stuff. So spark structure streaming is much better than that. But
[16:25] you can see how much cleaner it is in spark declarative pipelines because we provide you higher level primitives such as autocdc. So there you're just saying hey do an auto CDC change data capture
[16:38] just read the data and merge it into your downstream. So that's all you're specifying over here. Okay. Um but I think where the future is going is that hopefully if you have good
[16:52] abstractions you will not be writing in this framework in this language at least what you writing it is English basically prompting and the way it works is if you now use genie code you should be able to
[17:06] spend most of your night time specifying your intent and what you expect the outcome to be output you should not be spending too much time actually going into the details of what is the constructs that have been given by
[17:19] streaming systems. Today if you use Genie the right hand side picture shows you you can just prompt and say that hey I want to build a pipeline which takes some data set airlines data set or taxi data set and I want to just clean it up.
[17:32] These kind of questions can just generally be translated directly into spark declarative pipelines. Uh and even if you're using notebooks it can be used there. So overall the future what that
[17:44] we see is most of you should just be using genie code. The underlying technologies such as spark declarative pipelines and streaming will just be seamless underneath all of this and you'll get both performance and throughput and debugging becomes easier
[17:58] overall. Okay. So if I were to give you some takeaways, these are my three takeaways. The first takeaway is Spark itself is a single engine for both high throughput and low latency. So if you have
[18:10] workloads which require high throughput, please use Spark. If you care about hundreds of millisecond or even lower, use Spark. You will see that the same infrastructure will give you both of
[18:22] these u will solve both of these use cases for you. Uh the second takeaway is Spark declarative pipelines. It is the single system that can give you reliable multiflows. It will make streaming
[18:35] really easy for you and reliable. So consider using that wherever you have a chance. And the final thing is I hope from now onwards this is an AI world. You are going to just jump onto Genie
[18:47] code because that's where most of the coding will happen in the future both for you and for your team members. There are lots of things that we have been building in the last few uh quarters. I want to just go through some
[19:00] of them. This is more detailed view than the higher level overview I gave. Uh the first set of work is realtime mode. It is GA as of this DAS conference which means all the talk all the mention that
[19:12] I did about hundreds of milliseconds is available to anybody who wants to go in production right now. The second one is we spent a lot of time on how people can express complex business logic. So we
[19:24] build newer features such as transform with state. This is again GA and as well as for each batch which is one of the most loved and most used products or features within streaming is available
[19:36] in spark declarative pipelines. So you can use it to write to multiple different syncs and so on very easily. Finally, we are broadening the ecosystem of sources and syncs, which means as of
[19:49] today, you can start writing to lakebase, one of our uh newer products that we have just recently released as well. You can basically start reading and writing to Kafka STP. So all of this is available right now. But there are
[20:03] many more things we are working on for the future. Again, many folks want even lower TCO and we have newer things we are working on such as autoscaling for RTM, real-time mode. It should be coming
[20:15] up soon in the next couple of quarters. Similarly, we are widely increasing the net of operators and keywords that you can use in real time mode. People have been asking us about stream join. People
[20:28] have been asking about hey can we do custom aggregations? Some of these will show up in the next quarter itself. uh and we will have more better integration with zerobus which is our new ingestion
[20:40] layer. Uh there was a deep dive talk I think yesterday on it. Uh so you should be able to check out some of those replays. And finally we are spending a lot of time to simplify CUJS. An example is
[20:52] let's say you have a pipeline and something bad happened and now you want to rewind and replay. That's a very common thing people ask about. We have just got into um beta for time travel
[21:05] and soon we'll have newer features like fan now demultiplexing. So demultiplexing means let's say you have a data coming in and now need to spray it to multiple tables based on certain criteria. So many of these features are
[21:19] things we are working on right now. This should be there in next couple of quarters to be used by you all. Okay, with that I'm going to move give the mic back to Navnit who can talk about some of their customer use cases.
[21:39] Okay. Okay, let's change gears a bit. I'm going to talk about three real customers who are in production. Um these are customers who are using our streaming capabilities both through structured streaming as well as Spark declarative pipelines. Um the first one
[21:52] is uh make my trip. So make my trip is Asia's largest travel aggregator. So uh the scale of users is almost like multiple hundred millions active users
[22:04] every day. Uh who come to their websites, their mobile apps to essentially book travel, hotels, airlines etc. For them personalizing the experience on their application is really important
[22:17] and uh they want it to be realtime personalization. So for example, if you go to that app and you search for a hotel or a flight, they have real-time context on you as a user and they know
[22:29] your search history and then they personalize the experience based on your search history. This is not only if you click on their UI, but if you're using their agent chat bots, even the agents need that real-time context to be able to serve the experience for you. And uh
[22:43] they do this using Spark. So they use the real-time mode feature which Indrajit spoke about. Um so for them since it's it's more a question of like hey how do I kind of optimize and get subsecond latency what they do is
[22:56] basically they they curate the context with the help of spark realtime mode and then serve it onto their applications and they're able to do this in uh 50 milliseconds. So their p50 is 50
[23:09] milliseconds is how they kind of serve this real-time context in their applications. Let's look a bit into the architecture. So there are there are two so they have two businesses. They have a direct to consumer business and then they have
[23:21] B2B. So uh they're corporates. You're booking travel through your corporate programs. These are two different click streams. The both these click streams come through two different message bus uh topics Kafka uh and they basically
[23:35] reach Spark. What Spark does is it takes each and every event stream and then it enriches that event with the the search history or the context of that user. that context is basically stored they
[23:48] use aerospike as their um as their almost like their state store to store this sort of context. So Spark does a read from aerospike, enriches the event and writes to Reddius and their UI basically reads from Reddius
[24:00] essentially. And so what are their results? The results were 44 milliseconds P50 latency. So sub50 uh their P9 so this is like 99 percentile of events were basically being served in
[24:12] 500 millisecond. And there was real business impact with this use case. uh they actually measured the click-through rates on their applications before and after uh they started using Spark real-time mode and they saw a 7% jump.
[24:25] So it's basically directly uh impacting their business results and their revenue. There's a there's a blog that we have co-authored with make my trip on the data website which goes into even more detail goes through their code
[24:38] examples. Highly recommend anyone who has similar use cases to look through this. There are a bunch of learnings they had around how they kind of battle tested the system and made ready for production. Definitely worth looking
[24:50] into. Okay, the next company is uh this is one of the largest AI native dev tooling company. So they offer idees but this is a AI native IDE and this is a use case
[25:04] where they need streaming but it's not an ultra low latency use case but more something where they need to optimize for high throughput and the the problem is simple the problem is that they have
[25:16] uh telemetry logs. So every time you use their IDE, there is telemetry that is captured. Whether you accept the AI suggestion that is made, where do you click, what kind of quick feedback are you giving, they want to capture all of these data points because this is how
[25:29] they drive their product decisions. This is how they drive their pricing decisions, all of the analytics that you want to do to make your product better. Now, each of these log tables is huge. Like it's multi- terabytes. That's it's
[25:42] one of the biggest data sets that we have kind of like had to process. And their core problem was that when they were kind of streaming this data in and analyzing it, they they there are a lot of duplicates in this data for a m for
[25:55] multiple reasons. The same data telemetry point is captured multiple times. And this caused problems for them because if they were to just do a dump copy into the bronze layer, they would
[26:07] their storage costs basically just go up and they would have to run these expensive cleanup jobs to kind of ddup that from the bronze to silver so that they're able to make actually good product decisions with that data and it
[26:20] was just too expensive from both a storage as well as a compute perspective. And the way they kind of approached the solution was that they basically did ddup at ingest time so that the bronze
[26:32] layer itself is completely dduped and you don't need to do basically a separate cleanup job and you're kind of controlling storage costs. And so how did they implement this? So they have basically all of their log telemetry
[26:46] logs land up in S3. They use autoloader. So autoloader picks up basically their telemetry logs and in their injection job they use for each batch which is an operator that we have. It's a really powerful operator. They do the ddup with
[26:59] in the for each batch. So they run an insert only merge and what it does is it does ddoop before it kind of writes into the bronze layer and then the bronze layer is basically now a simple appendon
[27:11] thing which any downstream consumer can read and and process and they're doing this at massive scale. So multi- terabyte lock tables ingested. It's all dduped and and basically all the consumers are able to just simply read
[27:24] because it's append only over there. The third customer I want to kind of talk about is Coinbase. So Coinbase is the biggest crypto exchange in the world
[27:36] and for them fraud detection is one of their biggest biggest business problems. So they were losing like millions of dollars on just like fraudulent crypto transactions and they need to detect fraud again as fast as possible. Their
[27:51] SLAs were like subund milliseconds was what they were targeting and they found that they were just not able to get this in typical streaming systems. They tried to first build this using traditional spark microbatch mode and they realized
[28:03] that there were while their models were being trained on historical data which was point in time correct in real time for real-time inference when they used to start computing these features there was a delay because the the feature
[28:15] computation used to take a few seconds and that led to offline online feature inconsistency which impacted the model accuracy and basically they were not detecting fraud as fast as possible. So the problem for them was hey how do I
[28:28] ensure that during real-time inferencing I'm able to calculate features and use them with the lowest possible latency because that has a direct tie to my AI model accuracy
[28:40] and they basically now run all of their feature calculations through Spark real-time mode. So over 250 ML features are basically calculated for every transaction and these are being used by
[28:52] their downstream fraud detection models to kind of detect fraud and this is done all on Spark and they basically use real-time mode for it. They're able to do 100 millconds as their P99 latency
[29:06] and they actually measured their online offline consistency and it is improved by basically 99%. So it went from like 65% baseline all the way to like 98% overall. So how do they do this?
[29:20] Basically they have they have user events. They have transactions coming in. It gets piped through a bus which is Kafka. Then
[29:33] windowed aggregations. So they basically use Spark to calculate and then they write to their feature store. Then their model basically reads from feature store, scores it and then gives a prediction. And it was super easy for
[29:46] them. The the key insight for them was that hey, they were experts in Spark, but they didn't want to use a separate engine like Flink because that would involve like kind of translating code and using a different engine to kind of
[29:58] calculate. They were just able to because real-time mode is the same engine, the same APIs. It was just a trigger type change. It was like one line code chain that they essentially did to start basically moving all of these features to real-time mode. And um
[30:12] this these details are completely hidden away from their engineers because their engineers sorry by from their ML engineers because ML engineers know how to write batch code. So they used to describe their business logic. They continue to do it in the same way but
[30:24] they have automated CI/CD pipelines that basically know that hey these need to be run under realtime mode and then they basically automate the entire feature creation process.
[30:37] Okay, so we have around 10 minutes left. I want to use the last 10 minutes for a quick demo. So this demo is I'm going to demo a real-time application which is an air traffic control application. So here's a
[30:50] context. Flights across the world emit data. This is data around there like their position, their height, what angle they're flying at, what velocity they're flying at. This data is basically being
[31:03] captured like it's it's radio signals that are captured by commercial networks, hobbyists and they put this in this data source called the open sky network. Anybody has access to this and it basically tracks every flight around
[31:15] the world. What we did is we kind of pulled this Open Sky network data and we put it through a message bus in this case Kafka and had Spark read it. And what Spark does is it basically now is
[31:28] able to track the location of every flight around the world. And we did some really interesting analytics on like hey how do we look at uh whether a flight is going up going down is it landed is it grounded um is it are where are the
[31:43] congestion points across the world where there are a lot of flights. So a lot of stateful stuff that we kind of do to help help people kind of monitor the traffic of flights. We write that to a really low latency sync which is
[31:56] databicks lakebased which is a postgress sync and we use data bricks apps to kind of visualize this entire thing over here. Let's dive right in. So this is the end UI of the application
[32:09] that I just described. So this is a brakes application. It what you see over here these are all like flights across the world. So I've zoomed in into Europe actually and each of them represents one flight that is currently flying all over
[32:22] Europe. I'll click on one of these flights to see basically the data around each of these flights. So you can see this is a flight um it it is currently it was end route to its location and
[32:34] just started descending. This was like real time. It just went from end route to descending. And you can see data points around that flight. It's at 32,000 3,200 ft. What angle it is flying at, what velocity it is flying at. And all of this is being like basically
[32:47] calculated by the Spark engine and shown here in 500 milliseconds and it changes like you can see 350 494. So it's like subsecond you're able to capture the data of that flight and where it is going and you can basically track any
[33:00] flight in the world and I can zone into this one zone which is basically the Amsterdam airport around that. As you can see there are a bunch of flights some of them are in route some of them are in route but have started descending. Others are the green ones
[33:13] are the ones that are just taking off from there and then the gray ones are the ones that are grounded over there and you can see that hey there are a large number of flights. We can even track the number of flights that are
[33:25] there in that area and you can see so this is the Amsterdam airport location and there are around 20 32 flights over here and we have classified this as a congested area. So this is an area where there are large number of flights. There
[33:37] are other areas where there are zero flights, two flights which are which are not as congested. And we're doing all of this analytics in real time in subsecond. It's just basically changing. You can see it changing. There's a leaderboard in terms of what speed
[33:49] different flights are going at. And this is being updated literally subsecond right now. How do I do this? I've actually built this using this is actually built by one of our field specialists uh our amazing
[34:01] field specialists. And so what he does is he reads from event hub. So we have Kafka and event hub. Uh what it's re doing is it's basically taking all of the open sky network data and it's enriching it. So what enrichment is it
[34:15] doing? So one of the enrichments it is doing is so you can see it's reading from Kafka. This is your typical dot readstream Kafka and it enriches by looking at the altitude the velocity rates whether this is actually on
[34:28] ground. So if if it's less than 500 ft
[34:40] cruising at a cruising altitude. And if it's less than 500, it's basically descending. And so it's doing this sort of basic enrichment to figure out, hey, what stage it is in. Then you basically take the per aircraft
[34:54] positions and then you're writing to lakebase, which is the OOLTP sync that I was talking about. And the way you do that is basically we have a new native connector that we have launched. This is a JDBC based connector which which spark
[35:07] users to write to lakebase. All you do is provide those three options the data brick instance the lakebase instance name table and you just say hey I want to write this real time. You specify
[35:19] real time by saying a specific trigger called dot trigger real time and uh five minutes over here which I'm highlighting is basically the uh the checkpointing frequency of the engine but essentially you're writing to lakebase and that is
[35:32] what is basically getting populated over here. So all of this data is being written to lakebase and this this UI is just visualizing on top of of lakebase. So you can see whether it's nro the enriched information as well as the
[35:44] positional data that was read by the engine and happening in like 200 to 400 seconds. Next you saw the congestion stuff that was happening. How did I calculate the congestion? That's a really stateful process. So what I do is
[35:57] basically I define uh I define different zones like the Paris airport, the Amsterdam airport and I say I set some thresholds saying anything above the more than two flights is where what I consider as congested. So for each of
[36:09] these areas I need to kind of track two things. Uh I I set up two classes. One is called the zone tracking processor. So what it does is whenever a flight enters or exits it emits an event. It's like a there's a plus one when when a
[36:23] flight enters into the the zone coordinates and then a minus one event that is emitted when a flight leaves the leaves those coordinates. And this is basically updated every time a data point comes in from the flight or read
[36:35] into the open sky network. And then there's another class which is called the zone count aggregator. What it does is it basically keeps a count of the number of flights in the in that location boundary and the list of ids of
[36:48] each of those flights. And basically what it does is it maintains a count and it basically checks the count against the thresholds that I defined on top and if it crosses the threshold then I basically create an alert and I take
[37:02] these two classes and I basically chain them together. So there is like this is the first thing where I do the per craft per aircraft zone tracking whether it's entering into a zone or not and then second where the per count aggregations
[37:14] are happening and it's emitting an alert over there and this is this is all like standard structured streaming code you guys can this this notebook is in fact publicly available and there's a video on it so you can look into it but
[37:28] essentially once I do the I chain the two classes I again write those write that information to lakebase it's the same JDBC VC streaming connector the stream uh lakebased connection options and I write it in real time mode so it
[37:41] knows that hey the right needs to happen with ultra low latency and that's how basically this block which is the zone congestion is being calculated and displayed and again all of this is happening in subsecond so I know it's
[37:54] getting updated as you can see like the count is changing in each of these locations as as things come in yeah and basically this is a live application that um like airports across
[38:07] the world are kind of looking at right now. This is how they basically look at air traffic control. This is what the the control tower essentially looks at in each of these places. And all of this was basically built using spark streaming writing to lake base and then
[38:21] a UI layer which is an application just leading from that and and displaying it. Cool. Um yeah, I think we have probably one minute left so we did well on time.
[38:33] Uh that's all we had in terms of content. We wanted to share. The key takeaways I wanted to kind of leave you with is that look streaming is no longer niche. It's growing. Everyone's using it. Like 3x growth in the last 18
[38:46] months. Almost every pipeline now has a streaming component. It's being used across workloads. The reason people are using Spark streaming for it is because as Indrajit mentioned, it's the one engine where you can do both low latency
[38:59] stuff as well as high throughput uh use cases. So you don't need to kind of have expertise across different engines and it's really expressible and we have kind of invested in ensuring that the
[39:11] babysitting that people used to do on streaming workloads is being kind of automated away and we taking care of it and there are bunch of new features that we are continuing to ship both to improve latency to improve cost profile
[39:24] to improve expressiveness. So please uh go ahead and start building on it. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.