Optimizing AI/BI Dashboard Performance: Best Practices for Speed and Scale

Summary

This video explains the four performance pillars for AI/BI dashboards on the Databricks Data and AI platform: caching and materialization, query optimization, dashboard design for progressive loading, and warehouse compute configuration.
Automatic materialization and server-side cache scheduling can dramatically reduce initial load times and enable instant filter interactivity without relying on live query execution for every user request.
The video introduces Lakehouse RT (code-named Raiden), a new warehouse type designed for sub-second BI query performance, and demonstrates the Optimize button for one-click automatic SQL query improvement.

Optimizing AI/BI Dashboard Performance: Best Practices for Speed and Scale

Slow queries, long load times, and inconsistent responsiveness don't just frustrate users — they limit adoption and erode trust in your data. In this video, we'll teach you how to apply the right design patterns and leverage the latest AI/BI Dashboard capabilities — including automatic materialization to deliver consistently fast, interactive performance at scale. You'll walk away knowing how to: - Design datasets and queries for fast, interactive dashboard performance - Leverage caching, query optimization, and warehouse configuration effectively - Use automatic materialization for the right queries to maximize performance gains - Identify and fix common performance pitfalls
Talk By: Daniel Streit, Staff Software Engineer, Databricks ; Eason Gao, Senior Product Manager, Databricks ;

Chapters

FAQs

What causes slow AI/BI dashboards on Databricks?

Slow dashboards typically result from missing or stale caches, unoptimized queries, poor dashboard design choices such as loading all data upfront, or undersized warehouse configurations. This video walks through how to diagnose and address each of the four performance pillars to achieve fast initial loads and instant filter interactivity.

How does materialization improve AI/BI dashboard performance?

Materialized tables pre-compute expensive joins and aggregations so dashboards query pre-built results rather than re-running complex SQL on every page load. This video explains when to use materialized tables versus metric views, and how to configure auto-refresh schedules to keep materialized data current.

What does the Optimize button do in Databricks AI/BI dashboards?

The Optimize button automatically rewrites dashboard SQL queries to apply performance best practices such as aggregation pushdown and improved filter selectivity. It offers a one-click path to faster queries without requiring manual SQL analysis or rewriting.

What is Lakehouse RT (Raiden) in Databricks?

Lakehouse RT, code-named Raiden, is a new warehouse type on the Databricks Data and AI platform engineered to deliver sub-second response times for BI queries. This video introduces it as a compute option for teams that require the fastest possible dashboard interactivity and cannot tolerate even brief load delays.

Full transcript

[00:08] Uh great to meet you all. Uh my name's Ethan. I'm a product manager here at Databricks. I am responsible primarily for our dashboards product. Um and we're excited to talk to you about performance today. Uh Daniel? >> Yeah, my name is Dan. I also work with
[00:20] Ethan on the AI BI dashboards. I've been on the product for about a year now, mostly focused on performance and trying to make them faster. >> Cool. All right. So, we have half an hour. We want to leave time at the end, about 10 minutes after that for
[00:32] questions. Um out of curiosity, how many people are here because they have a dashboard they're trying to make faster? Love it. Okay, okay. We'll we'll we'll get practical. That's great. And can I
[00:44] get a show of hands if it's like a few dashboards are slow? Okay, and how about like dashboards are generally slow and I'm looking for Okay, so there's a mix, but it sounds
[00:56] like some of us are seeing spiky dashboards. Some of us are seeing kind of every dashboard needs a little bit of help. I think today is actually good because we have some pointed advice and then we have some kind of more specific
[01:08] things you can target individual dashboards. So, we'll we'll get into both. Um just to take a step back, just to ground all of us, I think when we all saw demos of AI BI, um performance is always one of the promises. At least I
[01:20] hope that was conveyed when you guys had tried the tool, right? Like you saw demos where filters were changing and like responsiveness was instantaneous, initial loads were in the orders of seconds, and there was a chance that maybe, you know, when you went home and tried something, you didn't get the same
[01:33] speed that you saw. Um and so today we want to kind of demystify like what made those demos so fast, like what were we doing under the hood? Um and then those are all techniques that you can all use as well, and they're something they're best practice that we encourage. So, so again, the goals are initial
[01:46] loads are super quick, right? Instant interactivity should be a requirement. Um and then it shouldn't be super dependent on if you have a cache or not, right? So, even if you couldn't hit a cache at this at one point, if that
[01:58] result wasn't preloaded, how can we still make those fast? We're going to talk through kind of each of those building blocks today. So, there's kind of four key layers that we like to think about when we consider performance as a topic.
[02:10] Uh the first one is is caching and materialization, which essentially means doing the heavy lifting of running the query and the joins and aggregations before the end user actually asks the question and executes the query.
[02:22] There's also what we call query optimization, which is actually writing more efficient SQL. There's designing the dashboard, and there's actually techniques with layout and kind of how you display your filters and where you put your tables that can affect inter- interactive performance.
[02:35] And then we're going to talk about the new warehouse that we're developing, and that's something you might have heard in the keynote this morning, and we'll talk about about that a little bit more. And I kind of the key takeaway, and we'll keep harping on this, is that each
[02:48] one of these things are contribution contribute to faster performance, but they really compound when you use them together. So, keep that in mind. If you're doing some of these things but not the other things, when you try them, you might be surprised at how good of a result you can get.
[03:00] Okay. So, the first thing we want to get through today is materialization. And to do that, I'll hand it to Daniel to talk about that.
[03:16] >> Um yeah, before we talk about optimizing, I want to talk a little bit about how the system works and in particular about how caching works, cuz caching is one of the key levers to make anything fast. And with AI BI dashboards, we have multiple layers of
[03:29] caching in order to make things fast at different levels. So, I'm going to go ahead and start at the top layer, which is the >> The clicker's struggling. >>
[03:41] >> But the button works. Okay. I want to start at the top layer, which is the browser cache. And in the browser cache, we have two two types of caching. One is just ubiquitous for every request that we make for your data, there is a cache
[03:54] that uses an LRU um and stores all the data so that if you revisit the same widget or the same page, you should have a near instant uh response. The other type of caching we do on the client is uh what we call
[04:06] in-browser compute. And here we'll actually try to download your entire data set and apply your filters locally in the browser to make execution or sorry, interactivity of those local
[04:19] queries um near instant as well. And so the main constraint here is the amount of data that we can bring into the browser and right now those limits are 100,000 rows and 100 megabytes. And when you meet those rows or sorry, those
[04:32] limits, you'll find that we can make those dashboards uh extremely fast and extremely interactive. >> Yeah, and and I think it's worth calling out that's a per data set. >> Correct, per data set. Mhm. So, if you have multiple data sets, that is okay.
[04:49] Um the next layer of caching is on the server. And so anytime that anybody executes a query on your dashboard, it is going to be cached and we have a and persisted for 24 hours. Um so, if you visited the cache visited
[05:03] the same dashboard uh this morning and then you try to visit it again in the afternoon, it can uh load near instant because it won't be executing any of the queries, it'll just be loading that data um from the server cache. The benefit of
[05:15] this is also you can start to add schedules so that instead of running all your queries first thing when you get there in the morning, you can have a schedule run your queries and then when you get there, the cache will
[05:28] already be warm and your dashboard will load near instantly. >> Yeah, and I think it's worth saying if for initial loads, this is the single most helpful thing you can do. >> Mhm. >> Is setting up a schedule using the parameters that you expect your viewers
[05:41] to look at initially. Um and then sub in those default values, run them every 24 hours so that the first initial loads will always hit the server cache for your viewers. >> Yeah, and I want to reiterate your point there about uh choosing the parameters
[05:53] that you're most likely to hit because the schedules do allow you to um pick what the most common parameters are going to be. And so, of course, that's not going to work for every uh permutation, but you can hit a lot of the common initial loads that you were
[06:06] going to be looking for. And then the last thing I want to emphasize about the shared cache is that it's by credential. So, if you are using shared credentials on your dashboard, then everybody who views or executes a
[06:18] query is going to share the same cache with everybody else. And so, if I visit it in the morning and then because Ethan's always 10 minutes later than me, he comes in and views the same dashboard, well, he's going to actually just see the same data that I did, and
[06:30] he won't have to um go ahead and load that data again. So, that's essentially how the the caching works. And there is a key limitation here. As we kind of have been alluding
[06:42] to, only previously run queries are getting cached. Anytime you do something new or change a filter or a parameter that hasn't been run yet, you run back to the same issue of going back to your raw
[06:54] data to execute the query. So, what we want to talk about today is adding a new third layer of caching uh called materialization. And this is going to sit between your server cache, which is
[07:08] executing the widgets queries that are going to be uh powering your dashboard, and your raw data. So, it's a materialization of your base tables that are going to make up the foundation the foundation of your uh dashboard.
[07:25] Um so, the big question becomes when to materialize. Materialization is most effective when your base tables reduce the size or or complexity of the
[07:38] raw data. So, if you have multiple or costly joins or you're doing expensive computations within your queries or you're reducing the data so you're only interested in a small subset of your raw tables.
[07:50] All of these ways will reduce the raw amount of table raw amount of data that your base tables have versus your raw tables and then those become much very powerful opportunities to materialize
[08:02] and the subsequent widget queries will only run on those materialized data saving of the scan of your full data sets. Um and this can make interactivity much faster because instead of of course
[08:15] doing those full scans, you are only scanning the materialized parts of those data sets. So materialization basically just means do
[08:28] the expensive work once ahead of time and save the results. So every viewer reads something small and ready instead of recomputing it. Um it's durable and you get to choose how fresh it is. So we're going to do a
[08:40] little bit of a demo in a couple of minutes about setting up a materialization on a metric view. And you get to actually define many aspects of what that materialization looks like in order to um
[08:52] in order to tune it to exactly what you want, the freshness that you want, the granularity that you want. Here I'm going to go ahead and pass it on to uh Eason to talk a little bit through automatic materialization.
[09:04] >> Yeah, thank you. So I know materialization itself like Daniel mentioned is not a new concept in Databricks and DB SQL as a platform. Um but when we recommend a
[09:16] materialization for dashboard authors, you know, in years past um there were sort of two key complaints uh with that process. The The first one was I actually physically create a new table when I materialize. And that means I
[09:29] have to re-point all of my existing dashboards, rewrite those queries and point towards this materialized view. And yeah, like that that's like takes work and it's a little bit annoying, but but ultimately I can't do this at scale because that's a lot of management to
[09:41] maintain. The other thing is because you've created a new table, now the author of the dashboard is now responsible for the governance of that new materialized view. Um and now you kind of have spread apart like who is actually owning the
[09:53] table, who owns the metric view, who owns the materialized version that people actually use as fast. Right? Um and then the one thing that people maybe didn't ask about but kind of implicitly knew is that when you materialized in
[10:06] that world, you could only do the base layer of granularity. Which means if your table was say like a sales data by kind of a global company, um you might have had transactions that were at the granularity of individual cities or individual neighborhoods. And
[10:19] if you wanted to slice by like a region or a country, you had to re-aggregate at query time, which took away a lot of the benefits of materialization if you had aggregates that you always wanted to do. So, what we did this year with materialization uh in metric views is
[10:32] that we wanted to solve those three problems um and give you a one-click experience that keeps everything kind of in the package that you already had. Um so so the key thing here is now with automatic materialization, uh you can
[10:44] just kind of right-click in a dashboard and say, "Hey, materialize this for me." Um and under the hood, we're going to generate a materialized view that will be used when a query hits the base table or the base materialized view. Uh the
[10:56] metric view. So, what that means is you no longer have to rewrite your queries to point at this new thing. We're always going to have We're going to maintain and do that routing for you. Um the other thing is it's permanent its permissions are inherited. So, the materialized view is going to subscribe
[11:09] to the same permissions that is used uh to define that metric view in the first place. So, that takes a lot of the maintenance out of it as well. Now, I I kind of mentioned there's kind of two ways to do this. I want to spend a couple minutes on this slide to kind
[11:21] of describe like in detail when you guys get home like what the difference will be. There's two types There's two places you can materialize with automation today. Uh one is a metric view that exists in Unity Catalog. So, this is the type of
[11:34] metric view that you can use in Genie, use it in Genie 1, you can use it across multiple dashboards. A materialization for that asset is in public preview. Um, and you can also materialize now what we call a dashboard local metric
[11:46] view, and that is in private preview. And so, kind of the path you're imagining here, and maybe this is how a lot of you are already building metric views is you start in the dashboard, you know, you're opening up a new kind of
[11:58] layer of reporting for your company, maybe you're doing it on a new advertising campaign. As part of that, you're building a metric view locally in that dashboard with like, "Hey, these are my key measures and KPIs for this campaign, right?" And then, you can materialize that specific metric view
[12:11] just within that dashboard, so that, you know, that initial view run of reviews are quick and people can look at them, take advantage of materialization, look at that one dashboard. But, as soon as you maybe that marketing campaign starts to get larger and people start wanting
[12:23] multiple different reports, you want to start feeding it into Genie, you can promote that local metric view into Unity Catalog for reuse. And when you do that, you can also set up materialization there as well, and the mechanisms underneath are the same.
[12:36] So, what that practically means is in both of these worlds, whether it's shared in Unity Catalog or local to a dashboard, neither case requires you to rewrite any queries. Um, like I mentioned, queries will completely just hit the materialized layer when that's
[12:49] available. Um, and also there's a little bit of a difference in aggregation support right now. So, we'll talk about this a little bit more in the demo, but today in Unity Catalog, you can already support aggregates, and if you get on board into the private preview, you'll notice that
[13:01] you can't say materialize the like sales by city and then sales by region separately. That that is in development. Um, reusability I just talked about. There's a bit of a difference in how schedules are configured as well. Um, so
[13:14] when you materialize at a Unity Catalog level, obviously there's no kind of inherent scheduling mechanism in a metric view so far. And so, you can actually this for each individual materialization and per aggregate, you
[13:26] can choose how often that is refreshed. Um and then for the dashboard version, which is, you know, when you get on that private preview, um it subscribes to the dashboard schedule. So, if you have a schedule running once every Monday and then once every Wednesday, every time the
[13:38] dashboard schedule is run, your materializations will automatically be updated as well. Kind of the key takeaway I kind of want folks to walk away from this slide is, you know, where your materialization would benefit, like what Daniel just
[13:51] talked about, right? Like in all those examples where you're not just doing plain table scans, um the turning on materialization now is really simple um and is really simple for us to just test. Turn it on, run a few queries, check out your load times
[14:04] and your inactivity times, and you know, if it's performing to the way you'd expect, you can keep it. If you're not seeing a lot of benefit, you can turn it off, right? So, the the kind of cost of testing out what this looks like is now way lower than it used to be.
[14:16] Okay, so that was a bunch of theory. I think it's more fun to show you guys what the UI actually looks like. So, for that, I'll pass it back over to Daniel. >> All right, I'm going to do a little bit of a um a sync over to mirror modes here, so
[14:28] pardon me and cross your fingers for me, please. >> We're having some display challenges today.
[14:46] >> Uh yeah. >> I'm just going to go to my All right. So, this is just a basic materialized view that we created. It's using um Sorry, yes, it's a metric view. The the MVs have been coming a a lot
[14:59] lately. Um yes. So, this is basically just um food inspection data in Chicago, and what I'm going to do is
[15:12] Okay, I have to go to your I know because you're in your computer. Okay. Okay. Got it. Okay, sorry. I was not the owner of that or I was on my computer, which is bricked over there, but I'm Easton was not the owner of that materialized
[15:24] view, so we'll switch over to another one here. Okay. So, when you're in the metric view editor here, what you'll see is this materializations tab here, and I can go ahead and add one here. And what it's going to start by is giving me a
[15:36] name here, and what kind of data are we looking at right now? Well, so there's two different styles of aggregation. The first is aggregated, and then there's unaggregated. I'm going to actually start with unaggregated, in which case
[15:48] there is actually going to be no further options, and then I would say I want a schedule on this, and I can determine how frequency how frequently I'd like that um materialization. And so, I like it a
[16:00] little bit I like my data fresh, so I'm going to go every 3 hours. Go ahead and save that, and save that, and now I've got a materialization.
[16:14] There we go. Nice. Okay, and what does that look like in the actual metric view configuration? So, if I go down to the bottom here, you can see that that is basically just persisting configuration into our uh metric view, and it's
[16:26] relatively human readable, and uh relatively easy to make sense of. >> Can I Can I add a comment there? >> Course. >> So, what what Daniel's showing is the unaggregated materialization. So, in a real world, if you imagine you
[16:39] have a query with an expensive join, and you know, hey, like it's taking a minute to load, but 30 seconds is actually joining these three tables. Um when you materialize unaggregated, it means that that kind of the work to
[16:51] join, instead of like not necessarily aggregating or grouping, but the join itself is now automated and pushed into a layer of that is beyond the dashboard. So, anytime a viewer now looks at that dashboard, at least that part of your query can be skipped.
[17:03] >> Absolutely, and including any filters or other parts of your query that might um be reduced upstream. Okay. There is also option of materializing aggregates, in which case
[17:16] you can determine which dimensions you want materialized as well as the um measures that you want included in that materialized materialization. So, here I've got this. I'm going to call this one
[17:28] customer segment. And then I'm going to go ahead save that. And then again, I'll just pop over to the uh YAML and show that
[17:41] we do now have a new materialization. Cool. All right. Let us go back to
[18:04] >> Yeah, and the like I mentioned, the the neat thing about automatic materialization is that you set up all of that as part of the metric view, but your queries stay exactly the same, your data governance stays exactly the same. You just re-point queries that used to
[18:16] go to the base tables, now add those materialized views. >> Yeah. Cool. Awesome. All right. I want to go into some >> Yeah. >> execution optimizations? >> Definitely. So, materialization is, like I mentioned,
[18:29] one technique, right, to make that uh query execution faster by offloading the expensive work. Um but if you go one level deeper, it's like, "Hey, I can make I can run SQL that is expensive on a schedule, but what if I just wrote SQL
[18:42] that was cheaper or I can have my colleagues write SQL that is cheaper, right?" Um traditionally, to write better SQL, you kind of needed to know SQL pretty well. Um but you know, now we're trying to with AI BI really opening up, kind of creating BI assets
[18:55] for even less technical users, um one thing we're really excited to to announce, um is that AI BI dashboards, as of last week, uh now support this new optimizer genie code button.
[19:07] Um so, the thing to know here is that we're really the goal of it is to automate the SQL improvements for you. Whether you you you know don't know how to make it faster, don't have time to make it faster, or you're trying to
[19:19] enable the rest of your organization organization to do this. So now this is inbuilt in Genie code. If you're still writing data sets directly or SQL, you know, for example, you needed to use parameters and you're not sure how to make it faster.
[19:31] This button in the bottom right corner of your dashboard query editor will light up whenever Genie code finds opportunities to make that faster for you. Um and I >> Yeah. >> Oh yeah, I would just note that it's not
[19:44] limited just to your SQL, but it'll also look at your tables and make sure your partitions are reasonably set up and your clustering is set up and you're using managed tables if possible and whatnot. So it's going to do a holistic look at your entire execution pipeline,
[19:57] not just your SQL. >> Yeah, that's a that's a great call out. So I mean even even faster than maybe converting everything to metric views and then setting a materialization. Maybe the first thing you guys can do when you get back if you have those low
[20:09] dashboards is hey, if it's SQL like see if this materialize or optimize button comes up. If it is, hit the button and Genie code will walk you through hey, like these are suggestions I'm changing, you know, these are implications you should be
[20:21] aware of and as soon as you approve them, we'll rewrite them in and show you what the execution path looks like. Um this is something we intend to continue to maintain as we find new techniques, new patterns and heuristics where you know, Genie code like kind of
[20:34] learns and how to make SQL better, you'll notice that like this button will start to get more and more effective over time. This is an area that we're actually investing in platform wide. So you'll not only be able to optimize here, you'll be able to optimize a SQL
[20:46] editor and notebook and other services where you can execute SQL. Um one thing we love for you guys to to partner with you guys on is if you know of SQL kind of improvement techniques that you know are good, but you're not seeing the optimize button work for.
[20:59] Talk to your account team. If that's something you you guys want to see incorporated into the tool for the rest of your org, you want to see it added as a base instruction. Please work with us. We're super happy to kind of aggregate everybody's expertise to make this tool better for everyone.
[21:12] Um, and just to give you kind of a little bit more under the hood on what this optimize button actually does. Um, we've essentially powered Genie with a bunch of heuristics uh on kind of here are all of the ways that we know of that
[21:24] can make SQL faster and here's generally what they look like and what you should do. Um, when you write a query now, uh Genie will go through and evaluate the areas of opportunity and estimate how much time it's going to save you for that query.
[21:36] Um, and when you do that, uh anything kind of over 10% is actually what gets bubbled up. So, when you hit the optimize it's not going to be one change, it's actually going to be a couple of suggestions that you can all accept at the same time. So, you know, some examples of what it
[21:49] can do today is, you know, if it's seeing that, you know, you're not adding a where clause on a partition key, right? So, the where that you can make, you know, significantly faster scans, it'll it kind of explain an opportunity to do that. Um, if you're seeing
[22:01] exploding joins, it's going to help you resolve those. You know, if you're filtering after you join, it's going to help you filter before that join. So, again, scanning less rows and things like that. Um, if you're familiar with SQL, this is some pretty basic and obviously there are more
[22:14] um kind of advanced heuristics. Uh before a lot of authors who maybe don't have the time or the energy necessarily to like review and scrutinize their SQL themselves, hopefully this will be a big help. Okay. And now we want to get into actual
[22:27] dashboard design. So, for that I'll hand it back to Daniel. >> Yeah. So, there's a few different things that as you're building out your dashboard you can do to make it easier to make them run faster. And I think especially with metric views, one of those
[22:40] opportunities is to use a single data set as much as possible. Um, I know for myself that I've often built multiple data sets on the same underlying tables in order to do different views of that
[22:52] data, whether they're aggregations or or whatnot. Um, with metric views and the upcoming semantic models, this becomes much less necessary. And if you add in the
[23:04] aggregations of the I'm sorry, the materialized aggregations of the metric views, um having a single definition of your data becomes much more viable and also allows for all
[23:17] kinds of performance optimizations. So, if you're have, you know, at worst case scenario, one data set backing one widget, um I think we can start to move a lot more towards
[23:31] reducing the data set sprawl and having a single metric view that describes your entire uh realm of data and using the relationships uh to um enrich it. >> And And you get nice things like cross
[23:44] filtering and all that stuff without having to Hey, like I'm going to add a data range picker and I'm going to add it to 17 different data sets
[23:56] >> Exactly. >> Um yeah. Um another thing that I want to talk about, and this is something that I've seen commonly both with my own dashboards, but with also with a lot of customers, is including many widgets on the first
[24:09] page. And so, oftentimes like you'll start with like a high-level look, and then as you go down the page, you'll go into deeper and deeper views. And some of those lower-level deeper views are actually quite data-intensive. Um
[24:22] One of these strategies for making a dashboard faster is to move those more intensive visualizations off of that first page. And so, for example, here we're showing like Okay, you have maybe you have a sales summary um first page
[24:36] that you want to land on, but if you ever want to go into the details, which maybe has like the actual row-level data that you can then inspect at a more granular level, you can do that through a drill-through. And so, instead of
[24:48] having that on the same page, you shift it to the next page, and that allows the dashboard to load incrementally. So, instead of going through all the detail of loading the transaction details in addition to the sales summary, the sales
[25:00] summary can load in, you know, multiple couple of seconds and then as needed the transaction details load on demand. >> Yeah, and and when we say load, we mean we we literally don't execute the queries needed to power that table until
[25:13] the user goes to that second page. Um so yeah, if you have expensive queries that are specifically to power individual visualizations, um separating them make a big big difference. >> Yeah, and it's not even just about the time that the uh queries execute. It's
[25:26] also just about the load on the warehouse and not executing everything at the same time. So if you know something is going to be a secondary affordance, you can basically signal to the system that it doesn't need to load with the first ones by shifting it to a
[25:38] separate page and then loading it dynamically. Um a similar thing, especially on the idea of a lot of queries hitting the warehouse simultaneously, is to batch um filter
[25:52] updates. This is particularly relevant for large data sets. I think that with the smaller data sets it comes less uh important. So on the smaller data sets, as you're clicking around your filters and you're watching the data update live, even
[26:05] though you may have multiple filters that you know you want to apply, it's kind of okay for that dashboard to update live as you go. In fact, it's somewhat of a satisfying experience. If though on the other hand you're looking at really large and complex
[26:17] data, in particular filters with like a high um domain size, you know, like 20, 30,000 uh IDs or or usernames or those types of things, then those type of interactive filters
[26:30] can actually become a bottleneck. And a simple way to address that is to enable the apply button. This is just an option in the settings for the dashboard, and now you'll be able to click multiple
[26:43] filters before triggering the actual queries. And then once you click the apply button, it will um update the entire dashboard all at once. And so, you're saving a lot of strain on the warehouse in these cases when
[26:56] otherwise it would have just been um every click triggering a new query that you're not particularly interested in. All right. How about optimizing the warehouse?
[27:08] >> Yeah. So, so this is the last piece. Um you imagine, "Hey, I've done my materialization. I've written my SQL as well as I could. I've optimized my dashboard. That really expensive scatter plot is now on page five instead of page
[27:21] one." I still want it to be faster. Um well, the last mechanism is just straight-up faster compute, right? Um and so, something we're really excited to be working on right now is integrating this lakehouse RT real-time
[27:35] lakehouse real-time warehouse uh with AI BI. Uh so, this is currently in private preview. Um and just for the folks in the room, the reason this is in private preview is that, you know, as we develop this lake this kind of new warehouse,
[27:49] the types of SQL queries and the types of tables that I can query on is kind of a journey to build out full support for. So, as more and more support comes online, then more types of queries start to become, you know, incorporated into the optimization, we want to open up
[28:02] more and more access. But, right now in private preview, you know, we're really keen to test with customers like yourselves, "Hey, like for your canonical dashboards, you know, are the accelerations that we're building into this warehouse for your workloads actually matching what you're doing? Or
[28:15] are the queries that we think are we're making faster not like the queries that you guys actually run?" Um so, that's that's what we're testing in preview. Um kind of the key takeaways for for this particular project is that one, you know, there's no data movement. You
[28:28] know, your governance still stays central. You know, uh in terms of user experience, it's as easy as just picking a different warehouse in that drop-down that you guys already use. Um but, because this new warehouse is
[28:40] kind of reactive to the types of queries that it can run, um we're doing a lot of optimization once we understand what query you're doing to apply different techniques under the hood to accelerate different types of cases. Um and traditionally, you know, a lot of the
[28:53] the warehousing and DB sequel was made for kind of really large scale enterprise data warehouse scale uh computations, and that's still something that, you know, this is intended to support. Uh but we really wanted to narrow in on kind of the the sub-second
[29:06] BI level queries as well, and that's what this is for. Um and so if this is something you guys are interested in, um I hope you are, uh talk to your account team, uh make sure you get yourself signed up for the preview. Um we'd love to work with
[29:18] everybody in this room, especially because we're all keen on this performance topic, uh to kind of dig deeper and see kind of where this works, where this doesn't work, and make it better together. Um I know the session happened before uh the one that we're in now, but I just
[29:31] wanted to call it out cuz recordings go out. Um if you want to learn more specifically about that engine, um there's a 40-minute talk that happened earlier this afternoon with that team. Um we'd recommend you guys check that out after the event.
[29:44] Okay, so so real quick on key takeaways, um I just want to just go through those four pillars again, you know, these are all the opportunities you guys have, right? So, set up materialization for expensive queries that you guys can't make faster, optimize your sequel with the sequel optimization button, leverage
[29:58] the things Daniel just talked about like the apply button, moving heavy things off to the second page, right? Um and then get yourself signed up for a faster compute engine. Hopefully with these four things, and this is all the things that kind of we do and the tricks that we do to make sure our demos are always
[30:11] fast, um everybody's dashboards can look like the ones that we all come come come to expect. >> Yeah. >> So. >> Yeah, and I would just add that those things compound, you know, just throwing a faster warehouse at it's not going to fix it, but when you add materialization
[30:23] and optimize queries and reducing the load by uh layout of your dashboard, all these things come together to make really fast dashboards. Uh Yeah. >> So, hopefully that was helpful. Thank you guys so much. Uh we have 10 minutes
[30:36] of questions, so if you have questions, go free to stay and we'd love to take them.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.