Skip to main content

Under the Hood of Lakehouse//RT

Summary

  • This video provides a deep technical exploration of Lakehouse//RT (Raiden), Databricks' purpose-built real-time warehouse engine that delivers millisecond query latency at 6,000 to 12,000 queries per second without requiring a separate serving layer.
  • The engineering team explains the architectural design choices, including reliance on open Delta and Iceberg formats with no proprietary data, multi-connection gigabyte-per-second ingest throughput, and auto-scaling ingestion paths supporting multiple input formats.
  • Predict Optimization and Liquid Clustering best practices are covered as the automatic tuning mechanisms that optimize every read and write path for real-time workloads on the Databricks Data and AI platform.

Under the Hood of Lakehouse//RT

Watch: Under the Hood of Lakehouse//RT
Historically, achieving sub-second query latencies across massive concurrent workloads required moving data out of the core lakehouse. this video deep-dives into Lakehouse//RT, a new real-time warehouse purpose-built to eliminate this architectural compromise. We will deliver a deep technical exploration of Lakehouse//RT’s underlying architecture, demonstrating how it maintains ultra-low latency without sacrificing resource efficiency. this video will analyze the specific engineering mechanisms that allow the engine to handle thousands of concurrent queries simultaneously.
Talk By: Jeremy Lewallen, Product Manager, Databricks ; Mostafa Mokhtar, Principal Software Engineer, Databricks ; Nong Li, Engineer, Databricks ;

Chapters

FAQs

What is Databricks Lakehouse//RT and what problem does it solve?

Lakehouse//RT is a real-time warehouse engine purpose-built to eliminate the need for a separate serving layer when handling sub-second query latency at high concurrency. Historically, achieving millisecond response times required duplicating data into a specialized serving system, introducing extra storage cost, governance complexity, and data velocity lag.

What data formats does Lakehouse//RT support and why does that matter?

Lakehouse//RT is built on open Delta and Iceberg formats with no proprietary file formats, meaning data stored for real-time queries is the same data used throughout the rest of the lakehouse. This eliminates the duplication tax—additional storage, separate governance, and slower update cycles—associated with traditional real-time serving layers.

What benchmark results are demonstrated for the Raiden engine in this video?

Live benchmark demos show the Raiden warehouse handling 6,000 to 12,000 queries per second with response times in the 16 to 32 millisecond range even as concurrent dashboard load scales up. The video also presents comparative benchmark results showing Raiden outperforming alternative approaches across multiple scale factors.

What is Predict Optimization and how does it help real-time workloads?

Predict Optimization is an automated tuning capability on the Databricks Data and AI platform that continuously analyzes read and write patterns to apply the right optimization strategies, such as Liquid Clustering, without manual intervention. For real-time workloads, it ensures storage layout stays optimized for the query patterns that matter most.

Full transcript

[00:07] All right. Hey everybody. Uh my name is Jeremy. I'm a product manager on Lakehouse Real Time. I'm joined by Mustafa and Nong who in the front row over here. They're going to join me on the stage in just a second. Uh they have stage fright, so they wanted to sit on
[00:19] the side. Just kidding. Um so, okay, cool. So, uh you guys saw the great keynote this morning. Um and we're going to go through um a demo. Oh, I'm going to show you some more of the demo that we had earlier. We had a lot of
[00:32] technical difficulties getting this set up and so I'm going to try my best to pull the demo up for you. Does that say demo? >> Scale demo. Okay. All right. So, bear
[00:46] with me. The for some reason there's a a problem with the way the mirroring was happening. Okay. So, earlier today you saw the demo. Um, and I want to kind of walk you through the demo one more time, but maybe a little bit more in depth.
[00:59] Um, just to kind of show you the power of what this new engine can do. And then we're going to go through today not only um, a little bit about Lakehouse RT. The guys who build it are here. Um, Nong in
[01:11] the front row here, he is the actual the technical lead for the whole project. Um, and Mustafa is the technical lead for all of performance. So, they are the key players of making this system run. And uh what we're going to do is we're
[01:24] not going to today we're not just going to talk about just Lakehouse RT. We're actually going to show you how to get real time performance end to end across data bricks. So we're going to cover everything from zero bus to predictive optimization to lakehouse real time.
[01:36] Show you all the best practices, everything you need to know there. And then we're also going to do the level two 300 uh by Nong uh on lakehouse real time. But first let's go back to the demo. So, um, the what they showed
[01:48] earlier today, if I can, this is really weird. It's like covering up the bottom part. Um, so what they showed earlier today was uh when you we
[02:00] built this simulator, they can actually show it hits the the uh Raiden warehouse, the lakehouse realtime warehouse. And you can see that when you click the button uh and you start the queries, you actually get this very instant response time. I think it's 16
[02:13] milliseconds for eight queries. What they didn't show earlier was that as it scales up, I think it's there. There you go. There's your 10. There's your 10 dashboards. The response is similar. You
[02:27] know, you're at 32 milliseconds. You're still super fast at 80 QPS. You couldn't see this on the screen, but as it's getting larger, you can see it's still uh keeping the latency flat. The
[02:40] P95 is 34 milliseconds. QPS is the top end would have been about 800. It's around 700 here. And then of course if you get to the thousand mark to see how well this engine performs.
[02:53] That's your thousand QPS or 6,000 QPS roughly. Uh the this is a total of about 8,000 queries that are uh issued. You can see the top band here is around 40
[03:06] milliseconds at the P95. Now let me switch back.
[03:23] Thank you. Um let me see here. Okay. So um let me start by saying that that was a version that we have where it just demonstrates the power of uh lakehouse real time and that's just pure speed at high concurrency. But there is that's
[03:35] usually not just the um uh type of workload that you're dealing with. Usually there's data that's coming in. You need that data to be fresh. And we have a whole system that makes that
[03:47] work. We have a demo that's internally that we also wanted to show. Um but we ended up going with the faster speed one today. Um but the the other versions show that hey, as the data can come in that zero bus can land that data in your
[04:00] lakehouse within about 5 seconds. um predict optimization will kick in, liquid clustering kicks in and then you're able to read that data out and that's the kind of endto-end workflow we're going to cover here today. So
[04:12] let's start with talking about the world before. So historically uh real- time serving meant that you had to have this second layer. We've done about a hundred customer interviews where we talked about uh we talked to customers who had
[04:25] their lakehouse, they had it all set up nicely, they had Unity catalog governing their data end to end. um they have their jobs there, their BI, their streaming ETL. Um everything is set up
[04:37] great, but then somebody shows up with uh some real-time use cases, use cases that are usually within a few seconds of time, and they suddenly have to start copying the data out into this second serving layer, uh the second real-time
[04:51] layer that we have today. They've got to keep this duplicate copy. They've got to keep this data in sync. But this is usually a huge tax for these customers. The first thing that they complain about is that they have to duplicate the data.
[05:03] They have a second copy as I just mentioned this double the storage, double the cost. That suddenly breaks governance. Their Unity catalog auditability that was there, their permissions, their lineage, it's all gone now. Now they've got two
[05:16] places where they have to do it, where they have to set it up. And then thirdly is that the number one thing we hear is that they lose velocity. They want to focus on product, not plumbing. And so what ends up happening is is that now
[05:28] they become responsible for the debugging, the scaling when the system goes down, they're on call. And that's not the business that they want to be in. They want to be in the business of shipping product. And so
[05:41] we're going to cover today this before world where you had that and now this new world, this after world of where you can do this all on one real time system. And this is the the combination I've been talking to you about of zerobust
[05:54] predict optimization and lakehouse real time. So the world t the the world now is this new engine. It's a millisecond performance at massive scale and it's directly on your lake. It has no lock
[06:06] in. It's on delta and iceberg. It's no proprietary formats. You're not trapped in with your data. Uh there's no data duplication. There's no extra pipelines. It's serving in
[06:18] place. Um, and then lastly is that it's governed end to end. So it's the same permissions as your Unity catalog. Everything's all in one place. You don't have to move it anywhere. And Mustafa is going to cover our
[06:31] benchmark results. And I'll be back to talk about the zero busend pipeline. That's this one right here. >> Okay. Hey everybody. Uh, my name is Mustafa. I'm very excited to be sharing
[06:43] some of the great performance results from Raiden today. uh basically uh real time lakehouse realtime lakehouse is a fairly exciting project was was built from the ground up for performance with
[06:55] with no excuses. So uh as you can see on the screen here uh with draining we're able to deliver queries like under 10 milliseconds. Previously that was not possible with the older systems. We're able to hit concurrency uh up to 12,000
[07:08] queries per second. Uh but actually this number is a bit old because like the team like a couple of days ago like were able to make some optimizations and bring this up to 16,000s queries per second. That's several orders of magnitude faster than other fastest
[07:20] system out there. And uh yeah, so let's kind of uh dive into it. So first off, we're going to talk about a benchmark called ClickBench. It's an open benchmark for analytical databases. Uh it's built by a company I'll refer to
[07:33] our friends in in yellow. Um and it's basically uh meant to measure latency against click stream uh data and u and web analytics. It kind of primarily used
[07:46] to basically measure performance of scans, aggregations and source. It doesn't have joins but it's still like a fairly interesting benchmark and it was kind of like the benchmark was kind of was meant to be show like who's who when it comes to like low latency and analytics. And as you guys see here uh
[07:59] we're faster than our friends in yellow and much faster than our friends in blue as well. Our friends in blue they have two different offerings. uh one called Gen 2, the other one called interactive. And just kind of one thing to kind of keep in mind, uh the uh real-time
[08:13] lakehouse results are against open file formats. They're against Delta, uh they're against iceberg. You're able to get this per this performance without having to ingest your data in a proprietary uh file system. Uh next, what I'm showing here is TPCH.
[08:27] Uh TPCH is one of the industry standard benchmarks. So we didn't we didn't want to just like make uh real-time lakehouse fast for simple scans and ads right there. There's more there is more to that. So TCH is one of the industry
[08:39] standard benchmarks. We looked at different scale factors small, medium, large and X large. And as you can see here, basically Raiden outperforms uh the competition in in every one of these. Now the last industry standard
[08:52] benchmark is TCDS. Uh TPCDS is a lot more complex. It's a lot it's much more modern. It has 99 queries with like really really complex joins as you can see here as well. U
[09:04] basically fails but we outperform across all all sizes. Uh we actually have a lot more performance to gain on the bigger uh scale factors that we're going to actually be improving over the next uh over the next couple of quarters. But
[09:17] but here we are next. Now moving to uh concurrency. So all these benchmarks that I was just sharing uh previously uh were mostly single user but like all of us here they want to basically onboard a lot of users on board dashboards and
[09:31] basically onboard folks on the platform to actually use it and in this particular case uh latency under concurrency becomes like really really uh important. So what we've what we what we have here is that on the x-axis we
[09:43] have uh throughput or queries per second. On the y-axis you have your latency. So ideally what you want to have is that as you onboard more and more users and you end up running more queries per second, your latency stays
[09:55] bound and all of a sudden you don't end up missing your SLAs's and as we show here on the chart is that uh real time lakehouse is able to deliver like 12,000 QPS while maintaining a P90 latency of
[10:07] of subsecond. Uh the Gen 2 offering basically peaks at 200 QPS. Uh the interactive one peaks at 300 QPS. uh the uh the yellow one picks around like 1500 QPS and it starts having some errors and
[10:20] the latency spikes from there. Now I'm going to give it back to Jeremy. >> You want to cover inverse? >> All right.
[10:32] couple customer uh points and then I'm going to dive into uh the end toend level 200 uh conversation. So first um we're going to talk about Inverse. If you missed this in the keynote, just want to cover this one more time. Uh,
[10:45] Indris um, uh, was they came to us couple months ago with uh, some queries that they wanted to test that they wanted to uh, improve the latency on. Um, they had queries that range from a
[10:58] couple hundred or sorry around 20 milliseconds all the way up to around 10 seconds. And as they um, onboarded into Lakehouse real time, uh, the new results looked something like that. Pretty flat.
[11:11] um pretty low latency. Uh we were able to improve uh this first query by about 100x and they um actually are um uh one of our design partners now. They've come on board and they are excite we're going
[11:24] to be working closely together on uh making sure that we roll lakehouse out RT out uh successfully. Uh this is their quote from Paul. Uh Paul was very excited to see the 100x improvement that
[11:36] he got where he had tens of milliseconds results. Um, and this is not just one customer. We saw this with point-click care where they had a 10x performance improvement. Bali's had a 7x performance improvement.
[11:49] Um, a 5x per performance improvement from a threat lookup security operations company. And then how actually had also another 4x performance improvement. So now let's get into the 200. So how does
[12:01] data bicks keep your data real time and your your latency real time end to end? And there are three tools that keep you real time. First one is zero bus. This is what I mentioned earlier. This handles ingest. This was a product that
[12:13] I believe was launched last year. Gade last year I believe. Um and second is predict optimization and liquid clustering. And the third one is lakehouse real time. Now the key
[12:25] takeaway here is that these systems aren't just silos. They all talk to each other and they all work together. This is kind of the the the future stack of how data bricks is working together to make performance great. uh these systems
[12:38] are working very closely intertwined and we're going to cover kind of how they feed into each other. Um so first let's dive into zerobus. So um first at a high level this is what zerobus is. It's a serverless engine. Um
[12:52] it's not a message bus that you run. Um any format can go in. You can have proto JSON arrow open telemetry and this system autoscales and load balances
[13:04] itself. You don't have to manage that. You don't have to stand up an extra server, nothing. It does it all automatically. And what it does is it has two kind of key modes. One is that zero bus node has a right- ahead log.
[13:16] And this is where we uh accept the data. We make sure that it uh that it's durable. The second is that we have a delta writer. The delta writer then uh passes this data into a managed delta.
[13:30] Um and uh this lands directly into your lakehouse within a matter of seconds. Now a lot of customers will ask like okay when do I know of when to uh ditch the bus or when not to and so what we
[13:44] tell them is it's kind of the default answer is is that zero bus is the best performant thing on uh data bricks for ingestion. Um if the lakehouse is your sole destination if you need high
[13:56] throughput concurrent rights to your lakehouse this is the right tool to use. If you need near realtime second level freshness again zero bus is the thing to use. Keep a traditional message bus in
[14:09] this situation where you need fan out. So if you have one source of one source of truth that needs to go to many consumers then uh your traditional bus cafka is probably the right solution.
[14:21] Now I'm going to go through best practices of of zerobus. The first one is is the protocol. Use gRPC in production if you can. we have a rest uh um system as well. Um second is is that
[14:36] encoding so you can you should match the encoding to the work to the workload. So first of all protobuff uh is is built for record by record but arrow is best for fast batch while JSON is offered as
[14:50] well but it's it's more convenient but it's just slower. Third is we recommend batching as much as possible. Fourth is uh scaling your uh producers
[15:02] not just one thread. So every connection can do about a 100 megabytes a second. So if you need to get the gigabyte uh per second, multiple gigabytes per second, you just uh share the connection out across all of those. I think you'll
[15:15] see if you look on our blog, I think there was a blog released earlier this or last week where they did like 12 gigabytes per second and they just used multiple connections and they scaled it out. Next is uh don't wait for the um don't
[15:30] block per record uh on the acknowledgement. The delta write a the write ahead log will take that you don't have to wait on it. Um when the pipeline appends you can you don't have to block on it. You can just keep moving forward.
[15:42] And then lastly is same region uh colllocate as much as possible and try to keep a stable schema. These are the best practices that you're going to find for zerobus. How to get the absolute best performance possible. I'll wait a
[15:55] second because I know some people are taking photos. All right, good. All right, next is um now as I mentioned, zerobus uh pat lands the data, but then all that data is now
[16:08] um it's not just landing in your lake and just waiting. What's happening is zerobus is sending signals out to predictive optimization and to liquid clustering of hey, there's new data. Make sure that this data stays fast.
[16:20] And there's
[16:33] optimize, if you don't recluster your data, if you don't vacuum out the data that's stale, you end up in a situation where kind of like this uh you see the line that's kind of going up and down here, your read latency will go up and
[16:45] down. And the reason is because you have things shift, right? Like you have new workload types to come in. uh you have your new data uh new data that lands into your database. And so um if you can
[16:59] keep if predictive optimization can read the data, can read the queries and it can make the adjustments for you, your latency can stay flat. And these systems all need to work together to kind of make this happen. I'm going to kind of
[17:11] go over real quickly what PO does. So first every query and every write that happens um you end up getting um it predicted optimization will predict the table benefits and it does these three things. It runs optimize, vacuum and
[17:24] analyze. Optimize is does your compaction on small files and it and it triggers reclustering by query keys. Vacuum will get rid of those dead files and it will reclaim storage so it'll make sure your storage stays cheap. And
[17:36] third is it'll run analyze. This is one of the key pieces for lighthouse real time to be fast is that you want to prune really well. And so when analyze runs it actually will go through um and it will uh make sure that we have the
[17:49] latest stats the uh on the files and that data will pass straight into uh lakehouse real time for those models to understand exactly what to do. All right.
[18:02] Now um on the previous slide you saw where we had um optimize as the first block. Optimize triggers liquid clustering. So if you've enabled liquid clustering with auto um you're going to
[18:14] get the best of both worlds here. So liquid clustering uh is clustering without any um uh uh work done on your side. You enable it one time and then
[18:26] liquid clustering will work with lakehouse real time to monitor the incoming queries and it will make adjustments to uh the data layout as needed. So as as your query patterns change liquid clustering will continue
[18:38] to adopt that and it does this on your your workloads and it does it in real time across all your tables. Um this next slide will cover kind of real quickly uh what automatic liquid
[18:50] clustering does. This is kind of how the internals work. Basically this is how you enable it. Up at the top left uh you run creator alter table uh cluster by auto and it does three things. It does clustering key selection clustering on
[19:03] right and background clustering. And as the data comes in we take that telemetry uh we do a model evaluation and then we we trigger the optimization from there. You enable it once and then leave it
[19:16] alone. This is going to be your best practices and I'm going to hand it over to you non. Uh so uh these are your best practices for PO and liquid. So the first one is is that you turn it on at the catalog level if possible. It will cascade to all the schema and the tables
[19:29] available. Second is is that try to keep your tables as you see managed tables. This is the fastest for metadata. Third is let it pick the clustering keys as your workloads change. You can't be
[19:41] in there monitoring every table across your whole whole organization all the time. If you'll just set it to auto, it will monitor the data for you. It will make the adjustments for you. You don't can you can stay handsree. Fourth is
[19:54] don't run hand optimize. Uh if you run optimize, it see it will it can mess with the uh maintenance the predictor and so you don't want to do that. Just let it run it itself. And then fifth is you can monitor cost and monitor the
[20:07] impact through your uh predict optimization system tables. Now I want to hand it over to Nong, the star of the uh lakehouse real time show. >> Thank you. I don't know about that. Hello everyone. Uh my name is Nong. I'm
[20:20] I created and lead the rate and development effort um at data bricks. Um so I just want to kind of extend some of the topics that Reynold covered about a little bit, how we approach building it, kind of what it is and and some of the lessons that that we've learned along
[20:32] the way. As Mustafa said, it is a complete rewrite of huge portions of the engine. It's not an indexing solution. And it's not a caching solution. It is a full-blown database, not an immemory database. Um that took a lot of the
[20:45] design principles we've had uh for quite some time and um put it put it together. So I think as we look across the industry, you normally see that um sort of different systems excel at certain
[20:57] types of workloads, right? You have some systems that are really good at time series or low latency or batch or um complex data warehousing, but we've rarely seen a system that sort of has the best aspects of both and and really
[21:10] hard to put it together. Right? This is the the same paper that Reynold is talking about kind of the stone breaker paper. this kind of one-sizefits-all that one database for analytics that can do all these things has been something that eluded us and that's exactly the
[21:22] goal we set out to do raiden right some of Mustafa's performance where we can simultaneously hit you know 12,000 QPS and single-digit latencies all the way up to you know terabyte multi- terabyte
[21:34] data sets with complex joins and have great performance sort of best-in-class performance across the board so that's the mission that uh we've been on um and I think one of the reasons reason we
[21:46] feel um that sort of a system has never really been able to do this is that uh there are a lot of innovation in data analytics technologies, right? A lot of different ways to to push the state-of-the-art, but they all come with
[22:00] some amount of trade-offs, right? you you sort of design your system to be particularly good at one workload and then it's not ideal at the other ones and it takes you quite a long time to get your novel idea sort of into
[22:12] production testing everything and by then your system has kind of oified a little bit right you can sort of stretch your arms and work on workloads that are adjacent to you but you're sort of missing out on what is the best you can do um across the the entire um spectrum
[22:26] of workloads right these things just kind of come with with these trade-offs and that I think is somewhat fundamental per idea. So Reynold showed this slide. This is sort of a sampling of um seminal papers from kind of the RDBMS research
[22:41] land. But I want to make it a little bit more concrete and sort of dive into two of them. So this is one of the papers from I want to say maybe 10 12 years ago sort of this idea of um say morsels driven execution. So in any distributed
[22:54] database, you're going to take the the whole thing you need to all the work you need to do. You're scanning a big table, divide it up into chunks. That's how you got parallelism. That's how we've always gotten parallelism. Morso is an idea that takes this step further um to
[23:07] divide it even more finely, right? Very like morsels as opposed to say tasks or or something like that. And this is a great idea. Um uh it's a certainly an idea we leverage in different parts of the system, but it comes with some
[23:19] trade-offs. It has complexity in head of cube blocking, right? when you run in high concurrency workloads, how do you make that work? Uh it has sort of metadata management complexity. When you break the task into so many little pieces, how do you keep track of them?
[23:32] And that might hurt you in concurrency. So this is an example of what we mean that these research ideas and these things that have been put into practice have a lot of upside, but then in some workloads they don't work so well. And I think Reynold talked about a little bit in his keynote. So that's just kind of
[23:45] kind of one example and some of the challenges with with systems in the past. Um I'll just kind of sample a couple of these. So I'm not going to go into sort of the the history of database research. Um so sees store right the
[23:57] original sort of um columnar database research paper sort of the core technology behind Vertica and now in sort of modern day Apache parquet and and a whole bunch of things like that. So obviously we love column stores for analytics right tremendous speeds up uh
[24:11] insane benefit for a lot of queries. But if you have say a workload today that has extremely wide columns you want to put images in your parquet table or um the table is extremely wide right thousands tens of thousands of columns
[24:24] some of these techniques start breaking down same idea right very helpful for some workloads but trade-offs in others and again we feel that's kind of um kind of the pattern we we've seen and again I'm not throwing shade obviously at any
[24:36] of these this work all these authors know know these things uh but putting them into together into practice has kind of been been a challenge So how did we approach this problem? Um so we have the fortune of having operated uh this platform for quite some
[24:50] time right operated um data bricks uh DBS SQL right just access to all the data that people have scanned. So we got a really really diverse workload uh a lot of uh production traces so
[25:03] understanding what the customers are doing with their permission and just trillions of queries that have been executed on the platform um over time and we've done is basically so you know kind of the examples I talked about where you have some trade-offs is
[25:15] because when you're designing the system you're very focused on a particular workload you're trying to optimize you sort of miss the other parts of the problem space and because we have such a diverse variety uh from the history of the company, we're able to integrate all
[25:28] of them into into the system from the beginning. Meaning initially we are able to design the system more holistically uh to capture this variety that we have from from operating it for for so long. Um so that's kind of one one way and
[25:42] we've integrated this um quite a bit throughout. Now you can think of this as we're trying to get the blended techniques of you know those research ideas I I briefly showed and people have operated the system on the other um in other places. And then the other thing
[25:55] that we've um we sort of managed to assemble in quite some time short time is is sort of having people who have been doing this for for quite some time right people who have you know worked at these companies I'm sure many companies have slides that look like that but it's
[26:08] really the people who have led significant portions um of the engine development in in their prior life right um so I think that's that's also kind of uh been extremely helpful for us to get to to where we've gotten to um and sort
[26:22] of putting all that together. Um, we do believe you can get to a world where where sort of one sizefits-all, right? We can get to that extremely low latency, extremely high throughput and you um and being able to scale up and
[26:35] we'll continue um to do that. And again, it's sort of um you know, I think the lessons that database designers learned over time is that the workload is always much more diverse than you think, right? the benchmarks you run any analysis you
[26:49] do um in production it's always um always changing and I think that property is is also somewhat fundamental right essentially the customer all of you will push the system harder to do more uh so we have built sort of this
[27:01] system to continuously uh figure out how to do this essentially have it learn from the traces tell us what areas need improvement help us guide the choices we're making so that the trade-offs right not just that it's really fast in
[27:13] one place uh we didn't have performance problems in other places we're able to sort sort of get the right balance um and and the right solution. Um and you know we leverage this sort of principle really throughout the entire development
[27:26] of the of the system. So you know we use those traces to learn and how to assemble the different sort of state-of-the-art algorithms. But it's also um sort of the way we we build the entire system meaning um we're able to
[27:38] simulate synthetic traces as part of our just core development. um sort of just get that huge variety of workload testing so we can uh we can do all all these things that that we've shared. And
[27:50] again, I think that's that's kind of the trick to the the one sizefits-all, right? You can't be too far in one direction. Um and and this production access um is what lets us balance those two. Uh and then lastly, you know, I
[28:03] think we've been talking about um SQL, but really the goal is to have kind of the the complete uh lakehouse. So I think other talks at this session will talk about things like uh fulltech search right having that integrated into
[28:16] Raiden and the real-time lakehouse uh things like geospatial types and then we're working towards the full support of sort of the spark connect data frame that entire set of ecosystem APIs that's what you would expect from from the
[28:28] lakehouse right so not not just SQL only uh being able to do all those and that's um that's coming soon um and then maybe kind of a peak forward I think there was maybe a dedic dedicated uh talk about this which is
[28:43] you know for all the results we've shown so far this is Raiden against iceberg and delta meaning against your single source of truth the stuff on your cloud storage again no sort of intermediate indexing or caching uh none of that
[28:56] stuff but really sort of Raiden is a platform for us um to innovate in other ways uh so in this particular example we've called neoren uh kind of a maybe a precursor to um LTAP or like a slight
[29:12] variation of it uh which is the ability for Raiden to directly query um the Postgress data pages right so instead of post instead of querying delta we can query the data pages as they're stored in Postgress and it's a similar approach
[29:25] to the the LTAP um solution and you can see of course you know RAM being built uh for analytics can be significantly faster than the Postgress and analytic style workloads right help you remove
[29:37] some of that data movement data silo and and so on and so forth. And of course, all of this is coherent with the entire data bricks platform. So credentials in UC auditing, lineage, all that kind of stuff, all put together um in one
[29:50] system. All right, Jeremy's going to come back. >> Thanks. Great job. All right. So, um, next we just want to
[30:02] kind of go over the takeaways, uh, of the conversation, then we can do some Q&A, especially, uh, with the team we have here. Uh, so the first thing is, um, kind of the the the highle takeaway
[30:15] here is it's one system, one copy. That's what uh, we're trying to build. Uh, we can serve real-time data directly on your governed lake uh, that you can ingest into. um it's governed end to end
[30:27] all the model all the lineage all the audit capabilities are still there and the third piece is is that lakehouse RT now offers this to be millisecond fast massive concurrency um and it stays fast
[30:39] the data bricks ecosystem will work to keep your data fast um from all the things we talked about today from the zerobust ingesting it to the predictive optimization and liquid clustering uh
[30:51] working to make sure that the files are pruned uh or the files are clustered correctly ordered correctly. We have all the stats to make sure that the engine can be as fast as possible. And then uh lastly, these are just kind
[31:04] of the high level best practices. The first one is uh use zero bus. You can push data straight into your lake. There's no broker. There's no streaming SEC that you have to manage. Batch the records for throughput. The second is is
[31:16] use predict optimization and liquid clustering. All you have to do is set it to auto, then walk away. You don't have to touch it anymore. And the third one is use lakehouse RT for these millisecond uh massive concurrency type of workloads. You can set the size to
[31:29] auto and uh let it uh run itself.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.