Speed at Scale on the Lakehouse with Lakehouse//RT

Summary

This video challenges the assumption that a separate real-time serving layer is necessary for customer-facing dashboards and embedded analytics, showing how Databricks Lakehouse//RT enables sub-second response times for thousands of concurrent users directly from the lakehouse.
The Raiden engine architecture is explained in the context of eliminating the complexity, governance risk, and infrastructure overhead that separate analytics stacks introduce, supported by real-world customer case studies and benchmark results.
Lakehouse//RT is positioned as a complete replacement for dedicated serving layers, with pricing parity with DBSQL and a straightforward path to consolidating infrastructure while improving data freshness and reducing operational burden.

Speed at Scale on the Lakehouse with Lakehouse//RT

Data leaders frequently face a challenging architectural decision: when deploying customer-facing dashboards or embedding analytics into applications, they often spin up a separate real-time serving layer to handle low latency, high concurrency workloads. While this can address performance, it introduces significant complexity, governance risks, and infrastructure overhead. this video challenges the necessity of that serving layer. We will explore how organizations can consolidate their infrastructure by serving high-concurrency end-user analytics directly from the lakehouse with Lakehouse//RT, through real-world customer case studies and benchmarks.
Talk By: Himanshu Raja, Sr Director, Product Management , Databricks ; Matt Holzapfel, Sr Product Adoption Architect, Databricks ;

Chapters

FAQs

What trade-offs does a separate real-time serving layer introduce?

Adding a separate serving layer requires duplicating data, maintaining separate governance policies, managing additional infrastructure, and accepting latency in how quickly updates propagate from the source lakehouse to the serving layer. The video argues that these costs in storage, operational overhead, and data freshness are unnecessary when Lakehouse//RT can serve the same workloads directly.

What performance does Databricks Lakehouse//RT deliver for embedded analytics?

Lakehouse//RT delivers sub-second query response times for thousands of concurrent users, as demonstrated in customer case studies presented in this video. The Raiden engine is designed specifically for the high-concurrency, low-latency query patterns of embedded analytics and customer-facing dashboards.

What are the embedded analytics and customer 360 use cases discussed?

The video covers customer 360 views where business users need a comprehensive, near-real-time picture of individual customers, and embedded analytics scenarios where product dashboards must feel as fast as native application screens. In both cases, the argument is that Lakehouse//RT eliminates the need for a dedicated serving layer.

How does Lakehouse//RT pricing compare to DBSQL?

According to this video, Lakehouse//RT is priced at parity with DBSQL, meaning organizations do not pay a premium for the real-time capabilities compared to their existing serverless SQL investment. The session also covers how to get started and what the migration path looks like for teams currently using a separate serving layer.

Full transcript

[00:09] We all know that well most of us know going fast can be easy but going fast at scale is really the hard part. And these next two presenters are going to show us how you're able to do that with the lakehouse. So with us this morning or this afternoon or this evening depending
[00:22] on where you're joining us from is Must is um pardon me Hamanchu Raja and he's actually a product manager and he leads our data warehousing our engines and our
[00:34] storage. Now alongside him is Matt Holto and he's an adoption architect and he actually helps take all of that goodness and take that speed to real customers. So with us today we have the one who
[00:46] built the engine with the one who puts it all to work. Please join me in welcoming Himu and Matt in speed at scale on the lakehouse with Lakehouse real time.
[01:06] All right. Um should we go on ahead and get started? I know people are still filing in but you want to go on ahead and get started. Yep. All right. Awesome. Thank you everyone for for joining. Um that was an awesome keynote. Thanks for uh coming here to to follow
[01:18] up on on it. Um as as mentioned, I'm Matt Holesafle, product adoption architect. Um but before I was at uh data bricks. Um I led product at a
[01:31] master data management um company. Um and our product was built on top of a legacy um data warehouse and um we always had this problem where um people
[01:45] would use the product in order to do very very largecale data transformations. That's why we we used a data warehouse in order to to help with with that. Um but because they were doing all these transformations, they were seeing hey my data is much much
[01:58] cleaner now as a result of of what the software did. Um a lot of them wanted to actually visualize that data. They wanted analytics that they could view and understand um so that they could see
[02:11] how has my data improved as a result of the transformations um and data quality routines um that that were done um on top of it. Um but that created a problem for me and and that that's the problem that I want Honu to to help me with um
[02:25] as as part of this session which is um my users want more analytics. They help the product sell. They create a great um product experience but they can make the the UI feel slow. Um and just to bring
[02:38] that home um let's let's see what that actually means. And so um going into I mean this is very much very similar to to the product um that um I built um and
[02:50] here um we were giving customers a a single view of of their customers. And when people would go and and demo it um we would start with with a view like this and then a customer would always say hey this this seems interesting. Can you drill in to a couple of those
[03:04] customers? Um and so we would drill in and then it would start to hang. And this went like well how was your weekend? um what what are you planning to to do? Oh, there it is. All right. Now, we can continue the the demo. And
[03:16] my sales engineers, they would always come up and they'd say, you know, the the lag times on the UI is really making the product harder to sell. And we'd look at the latency and the time required and it's only a couple of seconds. And so, you know, how big of a
[03:29] problem can can it be? It's just it's just a couple of sec. Okay, there we go. Now, we have another one. And so we hear about a query that takes just a couple of seconds, but when we're putting it into an customerf facing product, when
[03:43] the enduser experience is a dashboard or an analytic that takes just a couple of seconds, it makes the product feel really clunky. Um, and so I really need
[03:55] help with this because that's how I sell my product. I show people these dashboards, they understand the value that my product's delivering, and then they they want to uh they want to buy it. Um, and so if I had a magic wand, if and and today is my birthday, actually,
[04:08] and so this is a little bit of me asking um Hamachu for a Thank you. And that is not part of the act. That is true. Um, and so, you know, for my birthday, what
[04:20] I really want is I want my warehouse that's already in place, that architecture that I already have, make the queries run fast, sub-second query performance, because if it's longer than that, as you saw, people like, is this thing broken? and uh my my product it's
[04:34] going to take off like any product manager whatever you build you expect to be super successful and so I need to plan for thousands of concurrent users and I'd really really really not have uh prefer to not have to go to engineering
[04:47] and say hey go look at a bunch of new vendors qualify a bunch of different parts of the stack and try to stand up something new because I know that's going to take weeks months maybe even um years and I'm trying to ship fast I'm
[04:59] trying to get this thing to market because Um, I I know my product is is awesome. Um, so Himachu, what what are my options here? >> Yeah, thanks Matt. So, first of all, happy birthday. I literally had no idea it was his birthday. I still don't know
[05:11] if he's lying or he's he's I actually don't know. I hope it is. Um, okay. So, yeah, Matt, I have three options for you. Thanks for for these requirements. By the way, just to orient folks, you
[05:23] know, like so yeah, Matt is like playing the role of like your classic customer, you know, like he has these requirements and I'm playing me, which is the product manager. So I'm going to like, okay, Matt, you have these three requirements. Thanks a lot. These are actually well, very well written out requirements to
[05:37] help build a product. And I have three options for you. So the first option is let's pretend this is not a problem, right? Let's do what we all do, which is run your you have a warehouse today and
[05:49] why don't you just run your analytics, you run your dashboard, your BI tool or your analytical application on top of the warehouse. Uh there are pros and cons of every approach. So let me go with the pros first. A, you literally
[06:03] don't have to do anything. It's there today. You are already set up. All your permissions are set up. So just go and run your dashboards on top of the warehouse. The cons are you're not going to get the
[06:17] subsecond performance. You're going to get the performance that you saw one to two seconds, 5 seconds, you know, it doesn't matter like it's it's going to be in that range. It's going to be noticeable by the human eye. So when you are looking at the dashboard or you are
[06:30] looking at the application you will notice that there is a lag. The second thing is it's not going to serve thousands of users at the same time. So yeah if you are doing a demo and the d
[06:42] the dashboard has let's say 10 queries or 15 queries it's probably going to be fine. But if you're actually in a real world scenario where you have hundreds of users or thousands of users hitting
[06:54] your application uh analytical application or your users logging onto the dashboard at say 8:00 a.m. in the morning at the open of business, you're going to suffer from latencies as well.
[07:06] So that's the option one and those are the trade-off. It's the least amount of effort but you're not going to have the latency and concurrency. Do you like this option, Matt? uh our
[07:18] product's going to the moon. So I I need something that can handle concurrency. >> Okay. So I have another option for you. Second option. You're probably familiar with this. This is you extract your data
[07:30] from the warehouse and you put it into the BI tool itself. So you take the data that you care most about and it cannot be very big, but you can put it in the BI tool itself and then you're going to
[07:43] have really really good performance. So the pros of this approach is that yeah you're going to get really good performance. You can potentially serve thousands of queries. Uh concurrency can be really really high. But then the con
[07:55] of this approach is you cannot reuse what you have. Now you have to have an additional pipeline that is extracting this data and putting it into the BI tool of your choice. And then secondly and probably the biggest con is now you
[08:09] have stale data. So the the the the freshness of the data would depend on your extraction pipeline. You can go and extract every five minutes, every 10 minutes. Most customers would probably
[08:22] do an hour or daily batch jobs. So that's the trade-off. >> What are your thoughts, Matt? Do you like this option? >> Stale data is kind of a a deal breaker. As a a data product, we need to be giving fresh data to our customers.
[08:36] >> I see. Okay. Maybe option three will get lucky. So option three is let's go and build a separate site stack. I'm going to give you a brand new product. It's a completely specialized warehouse
[08:50] and it's going to give you subsecond latencies. It'll also give you thousands of users uh that you need. But the big con is of course you cannot reuse what
[09:02] you have. You have to go and ingest data in this specialized format uh from your lake. uh and you have to have separate governance now because these are two separate systems. So you have to copy
[09:14] whatever you have in your lakehouse for governance and you have to apply it on the specialized system as well and it may it may be possible that these systems don't have the same governance
[09:26] capabilities. So you may have for example row and access level control on your lakehouse but you may not have it on the specialized warehouse. >> Does this work? >> That's a that's a dealbreaker for my users. >> Okay, I think we are done. Let's so
[09:39] sorry I think I I don't want your business. Um no. Okay. So
[09:54] >> So let me give you the fourth option Matt. This is the secret fourth option which is lakehouse RD. It's on your lake. You can do any analytics that you want and you don't have to make any
[10:06] compromises. sub-second performance, thousands of concurrent users, you don't have to do any data movement. These are your tables in the Delta or iceberg that are there today. Unity catalog works on
[10:18] top of it and it's only a single copy of data. How does that option sound? >> Well, I want to kind of see this because I I I really don't believe that you can do this. And so you're you're saying
[10:30] that if I just change my engine >> to Lakehouse RT, you're saying you're not making a copy. You're not like what are you doing? >> It's it's just a new compute type. You
[10:43] go and choose Lakehouse RT like you do you would do today as a SQL warehouse and you just try it out. Tell me uh what you think. >> I mean, you don't even feel the engine working anymore. like this is what I
[10:56] expect an app to to be like whenever I work with an app and kind of on the consumer side like they're snappy like this this is great awesome okay okay so
[11:08] okay yeah so just to level set again so lakehouse RT is a new warehouse type that we are introducing it really is as easy as just changing your compute type into the platform any application that
[11:20] is running it's available through SQL execution API so you can access the data through SQL execution API and then the driver support will also come uh OBBC JDBC drivers. Um and uh it's millisecond
[11:34] performance at a massive scale. So uh the principle our goal is tens of milliseconds performance and the scale is you know it's however big you want it to be. It's your existing delta and iceberg tables. So com reduced cost uh
[11:48] simplified system and unified governance. So again, Unity catalog the table is in the lake itself and you just don't have to run any extraction pipelines, nothing.
[12:00] Um, how did we do this? There are couple of key pieces. The first is it's a it's a completely new native engine built from scratch. So there is no JVM overhead. It directly integrates with
[12:12] Photon. So there is very little uh difference between the so when when this new engine that Renault introduced called Raiden talks to Photon it does it really really efficiently. It takes
[12:24] advantage of the new hardware that has that cloud providers have built over the years especially like all the SIMD advancements and then the bigger pipes between the the instances and the object storage layers. So it does all of that
[12:37] and then it caches everything. It caches from data to metadata to query plans to provide you that subsecond latency uh performance. Um
[12:49] the the challenge though is now you you've created even even more problems for me. And so I had the first problem where I showed my users a dashboard and then they wanted a drill down. And now
[13:01] that they have all of these super snappy analytics, they want to actually see real time data. And so it's not it's no longer just enough for my users to be able to query um at very low latency and
[13:16] have it feel very snappy. They actually need the data to be fresh. They want to see data that has just been created over the past two 3 seconds. And so is that a problem that that you can help me solve?
[13:29] >> You are a very demanding customer, Matt. So okay. Yeah, you're right. You're right. I have solved your serving problem. uh but now yes I have introduced another or I did not solve
[13:41] all the problem fully right so this is what the stack looks like for most of you folks which who truly want the end to end uh latency where you have some kind of a streaming pipeline running
[13:53] which is you know generally kafka or flank right and it's like running at a very low latency and now I have lakehouse RD on the serving side but the challenge is that at least right now
[14:05] these two are not additive in Most systems you know 100 millisecond of Kafka plus you cannot just say oh 200 let's say 100 millcond on lakehouse it will be oh end to end time is 200 millisecond the challenge here is these
[14:17] pipelines are often very brittle especially streaming pipelines are often very brittle they're pretty complex and then you know there is like uh something happens backfill happens you have to go and rerun the entire streaming pipeline
[14:29] so so that's like the challenge that remains and let me just tell you what is available today and then I'll also talk about our vision. Um so with zero bus and lakehouse RT you can actually have
[14:43] end toend latency uh of milliseconds instead of you know uh seconds or minutes. So the whole idea is with you use zerobus to ingest the data. Uh you write it out in your manage tables with
[14:56] the predictive optimization and liquid clustering. We can be in uh in the background uh making these these uh tables optimized and then lakehouse RT can query them in subsecond. So each piece feeds the next and then the
[15:10] pipeline the end toend latency time is kept extremely low. So uh let me say let me tell you what's available today and then I'll talk about the vision as well. So this is the piece that I just showed. If you look at the lakehouse
[15:22] architecture. So here you have your data producers on your left side could be your operational apps your OLTP database right and then you're moving data from zero with zerobus and ingest um uh uh
[15:35] and then with lakehouse RT you can directly now query that data uh and and you can serve it in milliseconds. So that's the piece that we are doing with uh lakehouse RT and zerobus ingest. The
[15:48] thing that's not there today is in most cases you probably need to do some ETL. You want to do some transformation from the source side to the destination side. Right? So that's the the where the real
[16:00] time mode uh from Spark declarative pipeline will come in and this is where you can also do the transformation. So these green arrows right now especially these arrows they today this piece is not there and this is our our plan to
[16:13] make this as well so that you can actually do end to end from ingestion to transformation to querying within millisecond. Uh so super excited for this should be coming uh very soon
[16:25] that's our vision to come and give you end to end millisecond latency. So maybe Matt, let's see how at least the first piece that's already there today, which is zero bus to lakehouse. How does that look? Maybe you want to give it a spin.
[16:37] >> Yeah. Yeah, that that'd be great. And um you know what one of the things that uh Oh, there just um popped up um instantly. And with with Lakehouse RT in place as part of of our architecture,
[16:51] it's really just changing like what's what's possible within within our app. And so now we're going to shift away from just being a purely batch master data management customer 360 product to
[17:03] something that's actually real time. All of our our customers are saying, "Hey, everything is real time in our business. We need things operating in in real time." And so I want to give them a new feature that actually shows um for their
[17:16] customers and um for the customer 360s that um that they have um what are all the events that are um coming in and what is actively changing with the the
[17:29] data. And so when we use zerobus um for um that very low latency um high throughput ingestion in conjunction with
[17:41] lakehouse RT as our serving engine then it can take my app and what I'm positioning with my customers from this static view of their data which is very
[17:53] performant and it is great that now we could serve that up in near real time but now this is a completely completely new market for me and a completely new set of opportunities that my customers can can go after. And so here we're
[18:06] seeing these transa these um events coming in in near real time. And this is updating um every couple of seconds as these these new events come in. And so people um are able to actually um
[18:19] understand what's happening within within their business. This is great for use cases like observability where we have uh customers that work within manufacturing and a problem that they need to solve is they have call center
[18:32] agents who they get a call um about something that's happening um with with a car. They need to see live telemetry of what were the events that took place with that car over the past 10 seconds,
[18:44] 20 seconds. And if they're doing live troubleshooting, they want to be able to see is anything changing. Hey, open the door. We want to see what kind of signals are we getting. With this combination of zerobus plus real-time
[18:56] lakehouse, those customer service agents, they could have a view just like this where that new those new events are are coming in. And as we um outlined, you don't need to build a brand new architecture in order to to to do that.
[19:09] And so we can put this experience in the hands of um those call center agents um right right away. Um, and so those were just a few of the examples of the use cases um that that
[19:22] have been um unlocked as a result of not only the real-time lakehouse but also having um zerobus um as part of the stack because it's not always just about
[19:34] read latency. There are a lot of use cases and I think one of the best use cases for this are those um real-time analytics. when we're working with apps now, we expect all of those dashboards, everything that we're doing with data to
[19:47] feel very snappy. In Lakehouse RT, it solves that. Um, but there's a whole other class of use cases where endtoend um latency matters even more. Things like observability and and monitoring um
[19:59] or operational monitoring. All of these are unlocked now on top of the lakehouse as a result of these these innovations. Um, and so I mentioned one of the the manufacturer um, one of their use cases
[20:13] around a customer service agent where they're trying to understand what's going on with a car. They need that near real-time view of what's going on with the car so they could help their customer um, in that very very stressful
[20:25] situation. Another very stressful situation is a security event. um and Cisco um their um security researchers and threat analysts when there's an
[20:37] event they need to go digging through an absolutely enormous corpus of data pabytes of information about not only the event but all of the information
[20:50] surrounding the event. um because they're trying to understand, okay, if this security event happened, what happened? Why did it happen? How do we prevent it in the future? And if they're doing all of those kind of needle and
[21:02] haststack lookups and their dashboards um or their analytics or their queries are how I showed you at the very beginning where they're waiting two seconds and then they're waiting three seconds and then another security researcher gets gets online and now
[21:15] they're waiting five seconds. It's really really hard for them to be able to respond quickly and effectively. Um and before um uh real-time lakehouse was a real um uh thing. They were using a
[21:29] separate stack. Um they they had followed one of the um option um in this case it was option number two where they had a separate serving layer. They had an extra pipeline. They're using data bricks for hey if we need to just do um
[21:43] very long batch um historic analysis we want to look across 365 um days of information and um do um uh kind of historic analysis data bricks is is
[21:55] great for that. Um but for our security researchers use case where they need that data in near real time we have a separate serving layer. We got them into the uh real-time lakehouse um private preview, ran the exact queries that they
[22:11] were being uh that were being used by their um security researchers. 3 second lookups went down to 460 milliseconds. That's something that you
[22:23] feel as an enduser in terms of productivity because now when you're looking up a threat, it's not about waiting for the machine to work and you're not worried about the system blowing up on you and then your your day being ruined. You're able to to just
[22:36] continue to to look for these signals and be able to ultimately solve the problem. Um, and you saw in the keynote there are dozens of other stories just like this where customers are working with these dashboards. They're two
[22:49] seconds, three seconds. Um, maybe even more. And then with the real-time lakehouse, we're getting that to subsecond. >> All right. Um, I thought it was my job, Matt, to convince you to use Lakehouse RT.
[23:02] Seems like I did pretty well. All right. So, what's the catch? Um, you know, there must be there must be a catch. Truly, there is no catch. There is one copy of your data. There is no migration needed. It's a simple engine selector.
[23:17] uh and actually the so if if you're using DBS SQL it's actually priced same as DBSQL so uh truly there is no catch we really recommend you to go and try out lakehouse RT um and yeah as I said
[23:31] it's literally as simple as you open up your dashboard or whatever you have and you just select the new compute type uh once you create it uh cool that's it that's it from us there is a talk later
[23:44] about actually the engineers who built this new engine, the Raiden engine that Reynold talked about. And the talk is uh actually in Moscone South. Yeah. At 1:50 p.m. to 2:30. I highly recommend. You're
[23:57] going to get some fun insights on like the story that Ali was talking about that why they called it Raiden. And then they're going to do a deep dive of the engineering details of how they actually build this. But we we decided this session should be about uh how you're
[24:09] going to use it and what the use cases are. Uh and that's it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.