Delivering Sub-Second Latency for Operational Workloads on Databricks

Summary

Sub-Second Latency Achievement: Project Lightspeed's Real-Time Mode in Apache Spark Structured Streaming delivers consistent p99 latencies under 300ms for operational workloads.
Stateless and Stateful Query Support: Real-Time Mode handles a wide range of streaming queries—both simple and complex stateful operations—with predictable low-latency performance.
Operational Use Case Requirements: Understanding latency SLAs and requirements enables organizations to build real-time systems that support mission-critical business operations.

Delivering Sub-Second Latency for Operational Workloads on Databricks

Talk by Jerry Peng, Staff Software Engineer, Databricks ; Karthikeyan Ramasamy, Head of Streaming, Databricks
As enterprise streaming adoption accelerates, more teams are turning to real-time processing to support operational workloads that require sub-second response times. To address this need, Databricks introduced Project Lightspeed in 2022, which recently delivered Real-Time Mode in Apache Spark™ Structured Streaming. This new mode achieves consistent p99 latencies under 300ms for a wide range of stateless and stateful streaming queries. In this video, we’ll define what constitutes an operational use case, outline typical latency requirements and walk through how to meet those SLAs using Real-Time Mode in Structured Streaming.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is Real-Time Mode and how does it achieve sub-second latency?

Real-Time Mode is a feature in Apache Spark Structured Streaming from Project Lightspeed that optimizes streaming performance for operational workloads. It delivers consistent p99 latencies under 300ms, enabling organizations to meet strict SLA requirements for real-time processing.

What types of streaming queries does Real-Time Mode support?

Real-Time Mode supports both stateless and stateful streaming queries, including complex operations like aggregations, joins, and windowed computations. This broad support enables diverse real-time use cases while maintaining predictable sub-second latency.

What real-time data processing solutions?

The Databricks Data and AI platform provides comprehensive real-time data processing solutions through Apache Spark Structured Streaming and Real-Time Mode. This video demonstrates how organizations achieve consistent p99 latencies under 300ms for operational workloads, supporting diverse use cases from fraud detection to personalized recommendations.

For low-latency operational workloads, is micro-batch or continuous streaming better?

Micro-batch processing with Apache Spark offers fault tolerance, cost efficiency, and predictable behavior for operational workloads. This video explains that Real-Time Mode optimizes micro-batch architecture to achieve sub-second latency, providing a unified engine for both batch and streaming without sacrificing reliability.

How can I support batch, streaming, and real-time marketing use cases on one platform?

The Databricks Data and AI platform unifies batch, streaming, and real-time analytics on a single architecture. This video shows how Real-Time Mode in Structured Streaming enables organizations to support multiple workload types—analytical, streaming, and operational—with consistent, predictable performance.

Full transcript

`0:00` Introduction and agenda

We start? Okay. Okay. So before I get to the title of that, my talk is a forward-looking statement which is uh standard for all our talks. Anyway, the title of our talk is uh delivering subsecond latency for operation workloads on data bricks and we have been working on this for a couple of years and uh today we are showing showcasing some of the aspects that we have done so far. Um so the talk is divided into four uh distinct parts.
First I will give you a streaming overview and adoption of it at data bricks and followed by what type of use cases that data bricks pipelines run. Then uh we introduce realtime mode spark realtime mode what is it and what are the big ideas there and also we give a bunch of use cases that we tested with customers and how much latency that they were experiencing before in the previous system versus how much the latency that they are getting in the RTM system.
So this uh that will be uh uh depicted on the final customer validation section. So let me give a streaming overview. So the world operates in real time. So what about your decisions? In fact like uh the speed of business has increased nowadays but uh even then data is analyzed at uh every day or every week but that is not uh competitive enough. So we need to do it uh right now and in real time. So organizations are also also started having more streaming data every year.

`1:25` Streaming overview

This information is most valuable when it's used in real time. So the value of the data when you uh acquire the data right away is the highest and it exponentially decays as the time goes on. So, so the moment you capture insights and use the data that has high is value. So, with the new opportunities for real-time data. So since every organization generates a vast amount of real-time data for example uh bank generate a lot of transaction records from point of sale and uh airline reservations and all the call center records and the third party data also is provided by weather news feeds and market data and real-time traffic data and uh then the interactions when you click on interactions on the web and mobile phones those clicks are captured as web clicks and a social post, social media post, emails and instant messaging.
Similarly, you have IoT broad IoT sort of industry where sensors are sending data always. So that includes uh um sensors uh sensors in the trees that you're monitoring and then environment that you're monitoring in uh geoloccation tag machine logs which is generated from uh uh machines uh computers that are doing your applications or even microservices. Uh those are machine logs and mobile devices also generate logs.
So this generates a lot of opportunities for real time in terms of applications. What are the interesting application that you can write that includes fraud detection and personalized offers when you're looking at the click data and uh vaccine distributions when you want to do real time and also engine in-game analytics connected cars uh and vehicle maintenance. So there's a bunch of applications that you can do. Now when you generate all these real time data, how are you going to process it?

`3:56` Stream processing basics

So stream processing essentally captures the data from uh uh databases or uh CDC apps or uh CDC or apps or collection agents where all the data that I mentioned in the earlier slice is collected and it is staged in uh two different uh listing systems. One is uh S3 or buckets where you realize the data in terms of S3 files. Whereas another one is where the radar lances in a buffer called Kafka or pulsar those kind of uh message bus systems and from there uh stream processing engine takes the data processes it and aggregates it into whatever the uh aggregations that needed and writes it back into AS3 systems or even Kafka or whatever it is and which in turn uh serves analytical applications for dashboarding purposes or it can be used for operational applications.
So operational applications is what we will be talking today. So streaming with spark um so we have advanced stream processing spark on many accounts. So the one key thing that you have to uh note is uh spark is a unified engine that process both for batch and streaming because the API is compatible for both and it's also fall tolerant because uh we fail we use a microbatch model in the Apache Spark and when the microbatch fails uh we roll back the microbatch to the last uh processing uh checkpointing which is essentially the previous the end of the previous microbatch and each microbatch also is planned which means uh all the data that is going into each microbatch is already known and uh since uh it's already known when you replay the batch you get the same behavior of the uh fall failed batch.

`5:07` Streaming with Apache Spark

So which is fault tolerance is good and also the advantage of fall tolerance is since you are doing a microbatch model uh which we we call it as a pessimistic failure model. So we can do that in uh uh what do you call Amazon uh chip instances as well where there's a lot of failures. So but uh when the failure occurs it since it's at the microbatch level we can rewind it and pick up the new instance as well. So, so streaming in spark also has low latency because in the microbatch low latency in the few second range but that considers low latency for a lot of applications as you can see and is also cost effective.
You can trade off the latency and the cost by dialing down on the slots executed slots. So if the slots are uh lot of slots are available then all the slots can run simultaneously which means your latency will be low and then low number of slots is there uh what are the tasks that can schedule it get scheduled but if there are tasks that are waiting for it then latence will increase. So it provides a nice uh uh trade-off between cost and how many gradients you want to get and easy to use and all the powerful operators like uh stream static join stream join and all the various things are available and advanced stateful processing also is included.
So where you can do the newly released an API called transform with state which allows you to do a stateful processing in Apache Spark and increasing um so I've been amazed about the growth of streaming in data bricks when I joined data bricks we are doing around 10 or 15 million jobs so now it has gone up to more than 30 million so which is uh growing at 60% job per yet. So this is mindboggling especially considering the fact that the number of jobs that are running is uh so huge and streaming customers in every industries uh ranging from software tech, retail, media and entertainment and financial services, public sector, transportation and healthcare and life centers.
So so streaming is applied across the board. there's not only one industry is using where they realizes real time but uh it's applied across all over the place. So what is what are the streaming use cases and a broad uh categories and a broad consensus. So I I divide the streaming workload into three different categories. One is analytical workloads, another one is machine learning workloads, another one is operation workloads.

`8:36` Workload categories

Traditionally database has been good at running analytical workloads and machine learning using the microbatch performance. So what is the analytical workloads does? Analytical workloads analyze the streaming data for instant insert insights and fast edition. So once the data comes in uh it's aggregated or uh on a hourly basis or a minute basis whatever the aggregation needs to be there and stored in delta and then it those tables can be queried to power dashboards that can generate uh insights on a hourly basis of product sales or whatever it is. machine learning train models on uh the freshest data using spark and score that model in real time that also can be uh used in spark.
So if you look at the flow of uh data flow of these uh use cases, the data is lands up in Kafka or Kinesis or some message buses or even a cloud storage like S3. data is ingested into Delta Lake um as using streaming itself and um further computations are done at the using streaming jobs that take the data from the delta lake compute it and uh store it back into delta lake itself. So once uh so that is where the metallian architecture that you're aware of it right to go bronze table silver table and gold table comes into play and uh once uh those tables are uh returned back into delta using streaming jobs then you can serve the analytics out of it.
So which can you can power the dashboards or you can even score the model in real time in machine learning. So, so this satisfies two use cases, analytical use cases and machine learning use cases. Now, let us look at operational use cases. How many active login do have per streaming account and what are my operators of an industrial equipment? Do I need to send a promotion to users struggling to purchase who processing currently?
So business critical operational use cases like these require consistently low latency even though low the definition of low latency varies from customer to customer but uh we'll give a range. So data flow on these uh works um on these operation workloads uh indicate that the data available in Kafka has to be processed Kafka or Amazon Kinesis or any message store process the data and you have to write it back to some of these message stores or operational databases like OLTP or Dynamo DV.
The reason why we have to do that is if you write into delta delta is based on s3 and s3 has a lot of uh latency built in because uh each write into uh s3 can take around uh minimum 1 second because two 500 millconds that you write one is to write the log as well as write the actual data. So, so that's why the low latency operational use cases go from uh fast uh message buses to fast stores that include message buses themselves or RD OB databases or ran DBs or key value stores and data bricks has been shown to the best place for streaming and ATL and injection having conquered the analytical and machine learning space.
What can you do about the low latency? So we have been doing a microbatch mode which is planning on streaming and injection and what are you going to do about the operational use cases that's introducing the real time mode will talk about the real time mode. Sounds good. Thank you. I think one thing uh failed to mention is introducing us. That is. Hopefully that's obvious. Um, thanks everyone. We are both from data bricks and we both have worked in the area of stream processing for a very long time.

`12:38` Introducing real-time mode

We virtually worked on every stream processing system, every messaging system out there and that's why like we are super excited to really introduce what um, data bricks is doing next in this space. you know what our team, the stream engine team is doing next. Real time mode. I think some of you were at my talk yesterday where we went on a deep dive into exactly how it works. The emphasis of this talk is a little different.
We're going to be mostly going over the use cases that some of our, you know, customers at data bricks have been using real-time mode for, but I'll provide like an overview of kind of how it works so it becomes a little bit more concrete and it's not just, you know, vaporware and we're just, you know, inventing a bunch of stuff here. So, I'll just go over a little bit kind of how it works. And also, um, we talked about this yesterday, realtime mode.
This is going to be open source. So, you guys will see the code in open source. So that's going to be out there as well and we're going to contribute basically this feature to open source structure streaming as well. So let me first kind of talk about you know what is real time mode right? So the real goal of real-time mode can be kind of summarized in one sentence. The top goal is real mode provides latency in the order of milliseconds.

`14:19` Performance and capabilities

End to end processing latency in the order of milliseconds. And one of the other special things that we've designed is you can simply flip a switch and to basically enable real-time mode. You don't need to, you know, change any code. It should just be simply flipping a switch and you can basically enable real-time mode and your existing structure streaming queries. And if you were there for my demo yesterday, for my talk yesterday, I gave a demo where basically I did that.
I had structure streaming query. I changed basically the trigger. If you're familiar with structure streaming, I changed the config and then immediately the latencies I was getting from the query the pipeline turned from seconds to double digit milliseconds. And that was really cool. And another thing to say is at data bricks for real-time mode, we're currently in gated public preview. So basically we are actively working with customers and basically supporting them in production.
Let's look at some numbers. These are actually old numbers that we ran about a year ago. Um the numbers actually now are a lot better. Um that we've ran the numbers again. keynote tomorrow, he will actually show the real the more latest numbers and they're actually, you know, uh, you know, a bit better than this. But this kind of these numbers still kind of prove the point of real-time mode that we're basically able to, you know, significantly lower the latency over the existing execution modes.
What I mean by existing execution modes, if you're familiar with structure streaming, we're talking about trigger processing, trigger processing time. you know that's basically the kind of mode that's used by default and you would get the lowest latency right trigger available now that's not really a latency discussion there that's more of a you know throughput kind of mode or kind of periodically running a batch incremental batch up but as you can see we're basically achieving two orders of magnitude lower in latency from basically you know tens of seconds to seconds to double to low triple digit milliseconds to double digit milliseconds in latency right So, it's a significantly lower in terms of latency when you use real-time mode.
You know, it's not just low latency, it's also scalable. We've been running, we are running and we've been running basically also, you know, uh testing, benchmarking real-time mode at scale. And also our customers that we've been working with have been basically testing it with their real production workloads. And here we demonstrate that you know real-time mode is scalable. It can scale. It can process you know not just data in low latency but also high volumes of data as well.
And going back to kind of what you know Caric discussed in this slide right we have this latency spectrum the existing execution modes right we call it microbatch mode but I don't even really like that term. it's, you know, let's just say existing execution modes, right? Data bricks pretty much have like, you know, is doing very well or have, you know, I would like to say, I'm a little biased, but we pretty much have won kind of that area in terms of streaming ingestion and ETL into basically the lakehouse.
The thing that we need to tackle is these lower level lower latency use cases, right? These ultra low latency use cases that are on the rise now, right? As Cartha explained, everything becomes more and more real time, right? If you if you will and everybody wants results, things done quicker and quicker, right? So, this is where basically real-time mode serves, you know, basically its purpose to serve those kind of workloads in that basically area with those latency targets.
So, let me kind of talk about how real time mode works. You know the big ideas are basically you know running longer batches right there's a lot of overhead in the existing kind of you know how we run you know microbatches and the existing execution mode. So basically you know let's run longer batches you know tasks are more longived and let's also do concurrent stage execution right where basically all stages run at the same time.

`18:27` How real-time mode works

So here is basically uh kind of a depiction of that before kind of when we were running and processing time or the current kind of execution modes. You know batches are typically basically based off of how much data is available at that moment. We're going to process it. Um and then we're basically going to check for the next available data, process it. The batch sizes or batch ranges are mostly based off of basically data size, right?
And there's a lot of basically overheads. We discussed some of that in my talk uh I gave yesterday with my co-orker Suing. So we really you know after you know realizing these things we you know basically you know thought about it differently right let's let's redesign this or re you know kind of rewrite the mechanism so that batches are actually not in terms of how much data but in terms of time that they execute for a fixed amount of time regardless if there's data or not basically this batch and all of the tasks the whole basically DAG of your query will be running so that when data comes in we are able to process it super fast.
So one of the other things we've done here is instead of basically uh cutting or checkpointing the range of the batch before we actually process it. You know if that's a detail of how structure streaming works. Now we basically since we're running on fixed uh length durations for batches we will run the batch say for five minutes right executing you know all the tpples processing all the messages that come in wherever we've processed up to we will checkpoint and then we will start again and then we would run for another you know set amount of time and then we will checkpoint so on and so forth.
The next thing I talked about is running all the stages concurrently right concurrent scheduling. So currently in structure streaming one stage if you think about your query right it can be multiple stages right your map stage or reduce stage or even further reduce stages that you may have right currently regard in spark and you know spark and in structure streaming stages run sequentially one stage at a time they're clearly kind of not so conducive to latency if you could if you if you think about it so one of the other things we've is let's run all the stages concurrently, right?
So basically mappers can send data to reducers immediately once they've done processed it. So if you realize you know if we do this what else is needed the other thing is needed is a new type of shuffle because the current shuffle the sort mergebased shuffle that you know is used in batch queries and also in streaming queries in uh in Spark will not work right what we really need is a push pushbased shuffle I know if you're familiar with Spark there's already a push pushbased shuffle but this is different this is more pointto-point point mappers will send basically data directly downstream to reducers upstream will send downstream data immediately when it's basically available.
So that's also basically how we can achieve low latency as well is you know concurrent stages and also with the streaming shuffle that can push data out immediately. So now you know let's talk about kind of validations right the use cases some of our customers have um used real-time mode for right and what have they been doing. Of course I can't really say the exact names of these customers but we can you know talk about their use cases and kind of the general area where they you know worked in where they're you know part of.

`22:15` Customer validation

So uh the first use case you know we can talk about is um a use case by an OTT provider and what they want to do is track new devices right pretty common cases. So they want to see like what is the first device from basically you know a user that have come in like they want to keep basically track of that. So they have you know uh you know a pipeline that uses you know that has that data coming in from uh Kinesis and then we're basically reading that um uh with a query running in real time mode and also it uses the operator transform a state to keep track of what is the first device this user basically uh has used to uh watch the content of this OTT provider right first basically tracking new devices tracking basically the first device seen for this user and then they're writing the results to Reddus where basically they can um serve those results as well.
So that's basically one of the use cases that we worked with um this customer to implement and uh here are some numbers that they've you know basically ran and gave to us. MBM is the microbatch mode, the existing execution mode, processing time, basically the query running and processing time trigger. And the yellow there, the bars, if you can even see it's so low like there's a there's a huge contrast, right? Like is running the query running in real time mode, right?
And you can see the big it's a huge difference. Sorry, hopefully the numbers are clear, but it's a huge difference, right? Orders of magnitude difference and basically latency. And they're really basically happy with that. Another use case here um use case number two is with a very large bank that if I say the name everybody would know but their use case very neo standard fraud detection of credit card transactions right not not so not so not so hard to imagine so they u they wanted to obviously test this with real-time mode because you know as caric explained for things like that latency really matters right you don't want to catch you know these fraudulent you know transactions when it's already over, right?
You want to catch it as soon as you can. So, they kind of uh PCed on two kind of different um variants of this on real-time mode. One is, you know, reading from Kinesis using transform with state and then they use this operator for each to write the data to uh Dynamo DB, right? They wanted to basically um uh serve the results to Dynamo DB. But they also tried something else, right? They u basically the same uh workflow same workflow but they wrote the results to lakebase right I think that was also kind of announced in the keynote today net lakebase is basically a postgress uh offering at data bricks right and we have some results here for basically using the existing execution modes on Dynabo DB the latency was in the order of, you know, tens of seconds, right?
You got 14 seconds, you know, tens of seconds. With real-time mode on Dynamob, we're able to lower that, you know, quite a bit. But if you're using lakebase, the latency can even be further decreased, right? And now we're talking about P99s and what 151 milliseconds, right? That's what we got. And that's really cool. So, here's another use case. The customer here is a large travel site. And what do they want to do? They want to basically keep track of the last 10 searches of a user, you know, basically on both their, you know, website version and also the mobile version of um of their service.
So they just want to keep the last 10 searches. So when basically the customer comes back, they will know what the last 10 basically things uh that you know they've searched, right? So they have this pipeline where they have two sources you know from Kafka which one is data from you know um their mobile app and one from their website they're unioning it and they're using for each to uh to write to the results to uh to basically uh get results from you know the current results from aerospike and also write the new results to aerospike and they're using basically aerospike to serve basically uh their results and you Same story with the latency.
With the existing execution modes, they're able to only get latency in the order of seconds. With real time, um they're able to, you know, decrease that even further. And there was, you know, also a concern with maybe cost with real time. So there are basically also we've created tools or strategies that they can overs subscribe course or you know in case of say reading from Kafka single task or single course can read from multiple Kafka partitions to basically lower the cost you know um of running their query in real time mode as well.
But yeah those um you know those are the numbers for basically you know that use case. Um and we have uh another use case. This is from a large large uh uh food delivery service. So basically what they wanted to do was you know update their feature store right another basically common use case for that requires low latency. So this is also basically in their pipeline. They're reading from Kafka. They're using for each um to read and write to Reddus.
Reddus is basically their feature store. They're using that as their future store. They're basically, you know, reading and writing and updating that. And they're also basically then sending out some results downstream to Kafka. And this was an interesting case. The reason why was this customer was actually pretty what is the right term here? like brave maybe um because before they were using this mode called continuous mode if people are familiar with that this was introduced in 2017 and it was like a kind of a toy experimental mode that you know the goals are similar to RTM but it wasn't really fully fleshed out like it supported basically stateless queries and it offered low latency and it was never really supported by data bricks it was just something like you know this is something we could do in the future but never was really fleshed out but they decided to production with it and you know that's great great for them but like when they heard that oh you know we have real-time mode and we're going to support it you know seriously and we have a path forward to the future for both stateless stateful and everything like basically going to be full functioned they were really excited to get onto this new mode realtime mode and it turns out they actually get better latency a little better but I think the most benefit for them is like they're no longer on this experimental thing that you know is not really getting any updates or no one's really supporting it.
They're moved over to real-time mode which you know data bricks is seriously committed you know uh committed on. So there is um another another use case for basically a midsize financial product company. So what they wanted to do is they didn't give us a whole lot of information on this one but they what they wanted to do is use real-time mode to transform data into the required format downstream for fraud detection.
They didn't do the fraud detection in basically um the query itself but they wanted to basically get the data into the correct format so they can do fraud detection downstream. So you know another fraud detection use case was a Kafka to a UDF to transform their data custom code that they've written and also basically right to the sync uh a Kafka skink as well here. Um I don't think we had a comparison with the existing execution modes but we had some numbers with when they ran it for uh real-time mode.
So you know those are the numbers pretty low P you know P99 50 milliseconds right that's the latencies they were achieving caric there was another I I wanted to make sure I cover this there was another use case where they did the flink comparison right which one was that first one so I wanted to make sure I want to talk about this because it's actually really cool remember those are the numbers we got right P99's for basically RTM was 152 milliseconds.
This customer device first thing this OTT customer right they actually ran this on Flink and we get a lot of questions about well RTM versus Flink right like this was a case where our latency actually came in better than Flink Flink had the customer told us about uh they were hitting latencies about 200 to 300 milliseconds and real-time mode was able to bring that down lower to 152. So that was you know another basically validation point for real-time mode you know as well and we were super excited to hear that story.
So some of the other use cases that you know we've heard customers you know uh been talking to customers about one is you know um kind of a multipplexing use case partitioning a stream into you know multiple streams right um based off of you know whatever logic they want to split a single data stream into multiple streams. So you know downstream there can be you know different types of consumers downstream. Another is data anonym anonymization.
And another is data encryption, you know, of incoming events. And then another I think is quite kind of popular is online model serving. They want to do, you know, online inference and, you know, basically serve results in, you know, real time and low latency. So I guess, you know, I I hope hopefully you guys have this question. How do we get started with real-time mode, right? How do I, you know, start testing it? Is this guy that's telling you all this stuff, is it real?

`32:33` Getting started

Is it not real? Right? You guys want to hopefully get, you know, try it out for yourself? How do I get started? One, let me Okay, well, let me first go over kind of what is supported, right? Like I mentioned, um, it's currently in public preview. It's available on DBR16. 4. The environments we're supporting it is classic, basically kind of no isolation, and DLT classic as well. The languages we're supporting Scola and Python execution modes update mode currently is what we're supporting sources um all of basically the common message buses Kafka msk eventhub and kinesis as well and the syncs we support um are basically Kafka and the for each sync that I mentioned uh before and if you want to kind of start testing uh real time mode.
I would um suggest you reach out to your account team and then they can provide basically an onboarding you know guide that we can you know give you and you can basically start using real-time mode or testing it out yourself immediately or you can you know send us an email. I'll put our email addresses here. feel free to contact us and we can you know basically you know get you started on that you know path as well or you just want to tell us your you know use case discuss what you kind of you know your thoughts are or you know should I use real time mode for this feel free to basically send caric and I you know a message um yep that's all uh that's all I have thank you for you know listening in and I guess we can maybe do some Q&A thank you everyone

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.