Modern Data Modeling at Scale: Advanced Patterns

Summary

Databricks lakehouse natively supports traditional relational and dimensional data modeling patterns—including star and snowflake schemas, primary and foreign key constraints, identity columns for surrogate keys, and column-level data quality rules.
Metric views provide a unified semantic layer enabling the same business metric definitions to be reused consistently across Genie, SQL queries, machine learning pipelines, and Lakebase, eliminating divergent definitions across teams.
This video walks through ten specific data modeling capabilities in the lakehouse, from physical file layout optimizations with Delta to Lakehouse RT for sub-second BI queries, demonstrating that sound data modeling matters even more in the AI era.

Modern Data Modeling at Scale: Advanced Patterns

Unlock the truth behind data modeling in Databricks. this video will tackle the top 10 myths surrounding relational and dimensional data modeling. Attendees will gain a clear understanding of what Databricks Lakehouse truly supports today, including how to leverage primary and foreign keys, identity columns for surrogate keys, column-level data quality constraints and much more. this video will talk through the lens of medallion architecture, explaining how to implement data models across Bronze, Silver and Gold tables. Whether you’re migrating from a legacy warehouse or building new analytics solutions, you’ll leave equipped to fully leverage Databricks’ capabilities, and design scalable, high-performance data models for enterprise analytics.
Talk By: Kyle Hale, DBSQL Product Specialist, Databricks ; Shannon Barrow, Principal SA and Practice Lead, Databricks ;

Chapters

FAQs

Do traditional star and snowflake schemas work in Databricks lakehouse?

Yes, star and snowflake schemas work natively in Delta Lake on the Databricks lakehouse. Organizations migrating from legacy data warehouses can implement the same dimensional modeling patterns they already use, with primary and foreign key RELY constraints that the query optimizer can leverage for performance improvements.

What are metric views in Databricks and how do they unify analytics?

Metric views are a semantic layer in Databricks that allows organizations to define business metrics and measures once and reuse them consistently across the entire platform—including Genie natural-language queries, SQL notebooks, machine learning features, and Lakebase. This prevents divergent metric definitions from causing inconsistencies across business reporting and AI applications.

Why does data modeling matter more in the AI era?

In the AI era, well-structured data models provide the foundation for reliable AI outputs, since AI systems depend on consistent, governed data to function accurately. The Databricks lakehouse now encompasses not just ETL and BI but also data sharing, Lakebase, Lakehouse RT, and AI applications—all of which benefit from sound underlying data modeling.

What is Lakehouse RT and how does it enable sub-second BI queries?

Lakehouse RT is a Databricks capability for achieving sub-second BI query performance on Delta tables without duplicating data into a separate analytical store. It enables real-time decision-making directly on the lakehouse, which this video covers as part of its discussion of physical data modeling and file layout optimization strategies.

Full transcript

[00:07] All right, welcome to modern data modeling at scale advanced patterns. Uh I am Kyle Hill and I'm joined here by Shannon Barrow and we're going to talk today about high performance panel uh patterns and capabilities for enterprise analytics here at data bricks. Cool. Um
[00:21] you guys seen a lot of these today a lot of asterisks. I don't think we're like sh well maybe a couple little things with like some asterisks like could believe etc. So, you know, like just take it with like if we have forward-looking statements may or may not come true,
[00:33] put your surveys. Good. All right. Got it. Okay. Uh your speakers today. So, I'll I'll go first. I'm on the right. It's me Kyle. I'm a lead product specialist uh here at Data Bricks for Gene and ABI. I've been here
[00:45] just over four years. Uh like I said, I lead product for it. Uh I'm a six-time data speaker, so I do this a lot. Uh and you can trust me because I've deleted production. So, uh I'll take you over to Shannon. >> Okay. Uh my name is Shannon Barrow. I am
[00:58] a principal at Data Bricks. I've been here for seven years. We we've been here so long we they're asking us to just do the same talk over again. So we did this one last year. Hopefully we'll bring a little bit of new fresh insight this year. >> Little tweaks. Cool. All right. So what
[01:12] is our agenda today? Why are we here? So the evolving data modeling landscape. We're just going to talk about that at a high level. We're going to talk about these 10 capabilities. We're going to literally just kind of go through them like a top 10 list. I'll do the first five. Sharon do the back. Sharon do back
[01:24] five. We'll do Q&A. Oh, we got a couple of demos maybe or I don't know. We'll see. Okay, demos TBD. All right, fair enough. Okay, so a quick refresher on the lakehouse. The lakehouse is no longer just ETL and BI, you know, like
[01:37] this whole thing we've done is AI. So, really cool stuff. We've got data sharing. We have lakebased now. We have lakehouse RT, uh, brand new kind of offerings here. Um, and so we want to talk a little bit about data modeling
[01:49] and the core principles that haven't really changed with data modeling and why they matter even more than now that we have these additional systems and ways to express those data modeling. And as you should think about them, the importance of data modeling in your overall lakehouse. Um, I think the main
[02:03] thing I want to take before I kind of jump right into the capabilities is that again, I just gonna say it again like data modeling is something that's like tried, true, and tested as a quality improver separate from hardware, separate from new engines, separate from, you know, kind of new uh AI
[02:16] technologies to go speed up experiences and stuff in the back end. Data modeling is still going to sit at the core of these. It's not going away. If anything, it's more important than ever. We're coming, you know, we always talk the world is like every company's a data
[02:28] company disguise. Data is the new oil, etc., like it's just more and more true literally every year. So, it's probably one of the reasons they called us back to do this presentation. I really think like this is like core critical like understanding. This is like the the
[02:40] principles that underpin how to actually deliver these types of systems in in production operations. Okay. So, today we're going to talk about kind of 10 beliefs, myths, what you call them, 10 things that are true in the lakehouse,
[02:52] even if they weren't when you last looked. So, there's some new stuff, but also some things that are true to like some people are coming from data warehousing backgrounds. that come from traditional data modeling backgrounds. You may not know about the lakehouse or the capabilities of data bricks and want
[03:05] to help you understand some reality there. Cool. All right. So, capability number one, sorry, the belief is tiny. Maybe flicks that a little bit, but okay. Belief is that data doesn't really support relational or dimensional model.
[03:17] So, it's a lake, right? And like lakes do things differently in the lakehouse. The reality is that dimensional modeling is still the most effective design for analytics and it runs natively in the lakehouse. So today we give you delta and iceberg formats giving you uh acid
[03:31] properties schemas full DDL the relational foundation is there we do support star and snowflake schemas and if you ever read any of my like blogs things that I post if you ever have me in a room um you'll tell me I'll tell
[03:43] you you must use a star schema unless if you give me an answer I'm going to make you write it 50 times on a blackboard anyway and just punish you for giving you a bad answer. Um so they translate cleanly to manage delta tables. We
[03:55] support slowly changing dimensions, not just type two, one through six. Conformed dimensions and surrogate keys. Like all these things that you do in traditional data warehousing and BI applications, they absolutely apply in the lakehouse. You can't skip them.
[04:08] There's no shortcuts to them. There's no like, oh, we don't have to do that. We have a lakehouse. Not true. In fact, lakeouses are pretty good. We built really good performance engines. They perform better on these engines because of those underlying capabilities of Delta Iceberg. They're columner stores
[04:20] paret. They have metadata that describes those columns. So we can do data skipping. Those things really take advantage of guess what back tables that are long skinny b of numbers then short fat wide dimension tables that we can
[04:32] skip rows in very aggressively inside that back table. Second thing is like materialized views, metric views, feature tables. If you've sat through a session today that kind of talked about those as capabilities, great. They're awesome. They're all
[04:45] depend on relational shape. And so we invested in these for a reason. We actually want you and we're encouraging you to do more dimensional modeling and relational modeling through these features. These features are actually there to kind of like teach you since
[04:57] some of you are stubborn and don't want to do it the way out of the box. This is the right way to do things. Materialize aggregations from star schemas. Build metric views on star and snowflake schemas and build your feature tables in ways that they're very column and
[05:10] structured and they're very dimensional by nature. You don't always need a dimension model. Sure. But in 2026, you should consider it first. Okay, the next one's pretty straightforward. You can't really use primary foreign keys in data bricks. This used to kind of be true.
[05:22] It's not true anymore. We have PKFK constraints. They are GA and our query optimizer uses them to make joins faster. So, first off, you'll notice the very first word in bold and the bullet points isformational. So, our constraints, they're not enforced. We
[05:36] don't do things like three and F style. You can absolutely insert a value that in a column that is not in your actual upstream system. So, you should rely on data expectations, data quality. We'll have a little question later about DQ. You should be relying on those. You
[05:49] should be doing that anyway in in traditional warehouse systems when you're doing big batch loads. What you do, the first thing you do is you turn off the constraints that require foreign keys. You bunch of you load a bunch of data in and then you go turn those constraints back on after the fact. Um,
[06:01] and so that's like what you do in our systems as well. So informationational constraints, you declare them. We have a special keyword rely. If you've never used it, you should check it out because when you use rely, it turns left joints into inner joins and it turns certain
[06:13] inner joins into no joints at all. Obviously, reducing joints or having more discriminating joins helps with performance. So, these are really good for for those. So, you should rely when you have put in the work to make your PKFK strains enforced. So, when you've
[06:27] done the work and you know that every value in your PK is representing your FK and there's no strays on either side, turn on rely, you get a performance boost. Cool. All right. We do do support concurrent identity columns. You can generate surrogate keys at scale across
[06:39] parallel rights. Pretty easy technique. There's your classic identity column. Yep, we've got it. And we can do it across distributed compute. It's actually like a kind of an interesting engineering problem that we solved for you. Uh entity relation type relationship diagrams and catalog
[06:51] explorer auto render from your constraints. So when you define PKFKs, you get that nice pretty star schema you can go look at, print out, show your boss, your friends. Um and then enforce constraints remain on the road map. But today rely is how your pipeline decides
[07:04] integrity uh you know in actually uses it inside the optimiz. Okay. Capability number three AI assisted modeling. So AI helps you query data but you still handcode the model. I would tell you that like AI is an
[07:16] excellent way to help you design and evolve your model especially when you have dimensional models. Um and so really like one you can use authoring so we can generate DDL and slowly changing dimension 2 templates and CDC SQL from
[07:28] natural language and there's a con context aware from your unity cog schemas you have your bronze tables you claims them to silver you know what your dimensions are you can kind of pointed at it today I have a demo and fortunately we're on Shan's machine I
[07:40] might show it if you're you want to see it after class I can show you exactly how to do this but you can really tell it hey these dimensions are in there I'm telling you they're in there so that's you you the AI don't have to guess that are in there. I just need you to go build me to sedd2 to describe these
[07:53] pipelines and then I can record them. It does an awesome job with this. Genie relies on semantic learning. So day-to-day usage actually trains a semantic model. Your catalog improves itself when people ask questions. We saw this with Geneontology. So we're now actually taking in all the questions
[08:06] that people put into these systems to actually go make improvements to the ontology itself and also make recommendations for gaps in your semantic model. So it doesn't directly like kind of just go fix your semantic model, but it will say, "Hey, people
[08:18] keep missing this metric or getting this question wrong or giving negative feedback on it because you're missing some attribute. You should consider adding it." So it will make those recommendations and kind of really make it really easy for you as people ask questions and give that feedback to kind
[08:30] of incorporate that back in your semantic models. And then finally, with LakeFlow Connect, we have declarative shapes. Again, we kind of like I'm just repeating the question again, but with materialized views and streaming tables and manage connectors, you declare intent. So you actually go in there and you don't really like handcode ETL. you
[08:43] just kind of tell it here's the shape that I want this data to be in how much materialization I want willing to pay for us with frames um and the amount of streaming that I want and it takes care of all of the actual behaviors underneath in the physical model. So
[08:56] moving to declarative pipelines and kind of declarative overall jobs but even just declarative shape of your data and then letting the tool figure out the best way to actually go physically manifest that. So AI is just like a new part of this and makes it really easy. The more you can make your own systems
[09:09] declarative, the better AI tools can respond to it because they speak language and declarative systems, they speak language. All right, last capability like for for this kind of area with the semantic layer is that you know that you need a proprietary BI tool to have this layer.
[09:22] This is not true. We have Unity catalog metric views, the RGA. We've open sourced them into Spark uh and they're shared across every consumer. We have all of the key features that you would want today in a BI tool. So we have dimensions, we have measures, we have
[09:34] windows, we have joins, we do have multiffac join. Now we have many to many relationships and you can even do nested metric view. So metric view can be a metric view of metric views. Um and so those things are really good because you can have one definition. You can use it
[09:47] today inside data brick SQL. So it's a full SQL dialect. You can use it obviously in our AIBI dashboards just like you would with the BI tool. You can use it in AIBI genie. So any metric view that you build, you can just store that in your genie space, one object with all
[10:00] of your dimensions and measures and query against that. And then external BI tools like Tableau, Omni, Sigma, we have a lot of great partners today who are working with us on building metric views uh into their platform. So you can use them there as well. One of the really
[10:12] nice things about metric views is you don't have to start with like this kind of like grand vision in mind. You can actually start with an AIBI dashboard, start with a data set, build up your met your measures and dimensions in that dashboard as a local metric view. So
[10:25] just go up to that dashboard and then with one click you can promote it to a full unity calmetric. So we have this great capability for you to like build sandbox experiment get this thing right and then have a promotion process. Uh genie's natural language answers agree
[10:37] with your dashboards because they query the same definition. Right? So we have a metric view here. We're not having one data set over here and another kind of raw set inside genispace. We're all using this high quality semantic layer in metric views with great performance
[10:50] from all the things I listed above. And and why now? So the main thing is that in agent AI, you know, agents are going to consume more and more and more of the overall use of these tools and they extremely rely on semantic layers to
[11:04] offset the generation of queries. So rather than having to generate SQL against your systems and so on, what they're going to do is they're going to look at these tools and treat them as orchestrators. And I said this in my last chat, anytime you have an opportunity to move an AI system from a
[11:18] probabilistic system to a deterministic system, you should always do that. So if it can route the query, it goes, "Oh, you got a query. I have an answer for that and it's over here in this this metric view. I don't have to calculate that query. I just hand it the columns
[11:30] and it gives me the results back and then I replay them back to you." That's awesome. That's a huge win. So using metric views, using these in your tools, AI tools consume them as well. Genie consumes them. Absolutely. Loves metric
[11:42] views because then it just becomes a deterministic routing system from that query prompt. Just get the columns right. It's easiest thing to do. put it into that system instead of having to go generate a query that may or may not be right and even if it's right it'll be right tomorrow. So this guarantees that
[11:55] consistency that overall kind of like moving to a deterministic system for for AI. So again also helps humans by the way. Yeah it's good. Okay. Cool. All right. Um and so agents have data but
[12:07] not the business context right. And so this is kind of like an example. So on the lefth hand side sorry for the tiny text but what's our revenue per active customer in Q3? And so you know you have all these business concepts here. What is our definition of revenue? Is it net?
[12:19] Is it gross? Is it recognized? Which definition applies? U customer, is that logged in this month? Paid this month. How do we define that in code? Uh source of truth. Which of these 47 revenue tables is the trusted one, right? So
[12:33] like if you just leave it up to Genie, it'll take its best guess. That's what AI does. It's really eager and overconfident. It'll tell you everything you want to hear. Um but it doesn't going to find that truth in there, right? It doesn't have that access, that context. So what the agent actually sees
[12:46] is just these things on the right. It sees some columns and it has to make guesses based off just that. It's not going to do. So it's really really important that without business context the agent will guess or hallucinate. So again just really really reiterating that agents have the data. They do not
[12:59] have the context. They cannot manufacture the context. They will just hallucinate the context. But your answer and your job is if you're building these data modeling, invest in these tools and supply it to them so that they don't have the one that they don't have to worry about the context because you've
[13:12] already supplied it in the metric view and they're just routed to it. Yeah. So the way to frame it AI doesn't have an intelligence problem. It has a context problem. Um and so existing solutions are kind of incomplete for this. So what we see is that BI semantic layers by themselves just kind of focus
[13:25] solely on single tool. That's probably the biggest part. So the BI tool has all the answers. That's great. Go to this BI tool. What about my agents themselves? What about applications? What about just SQL and code? I don't necessarily want
[13:37] to always have to be use the BI tool as the only interface to that. So this is probably the big challenge there. So with knowledge modeling, you get a lot of business entity reasoning and you get ontologies and glosseries and domain models, but these require lots of upfront modeling. These are like usually
[13:50] like things that are like formal exercises that take years and years and go nowhere inside your organization. They're usually just some guy in the corner like building ontology. Um and they get scale really quickly. They're siloed in different stacks. really hard to bring into everyday analytics. And
[14:03] then finally, like LLM providers. I think we kind of know this like already. I've said it like a hundred times. They're really good at user specific and good skills. You can try your damnedest to make these tools kind of just like stay in their lane and give me the right answers, but they're not reusable cost
[14:15] users. Have many versions of the truth. They have no governance. So enterprise context requires a holistic approach. That is the geneontology that's kind of been offering today. The reason I bring this up in a data modeling session, I
[14:27] want to be really clear is that the thing at the bottom of this whole thing, metrics, glossery, and domains. So metadata modeling also just as important as data modeling. And when you do these
[14:39] things, well, the thing at the top automatically gets better, too. So this is exactly we were talking about that virtuous cycle loop of people using geneontology. That's the AI's domain. That's what it uses to come up with those answers, process these questions,
[14:51] and so on. But it's relying on those underlying metadata models and data models. So it is really imperative that you again build really good systems at the foundation for these tools to leverage in the experiences that they're providing to your users. You don't do
[15:04] this and you just go a genie's got it got no problem. I mean you heard it from I think Ali's mouth greatest thing since sliced bread. You're going to have a bad time. So again just making sure that you know when you're thinking about like how does it work? It extracts knowledge
[15:16] snippets. It calculates the re relevance here with graph rank. It applies permissions and resolves conflicts and it has a feedback loop. And really what this means though when we say extract knowledge snippets that's data model.
[15:28] That's metadata model. That's the thing that we're here to do. This is where those things literally. So now we're actually able to when you define this metric view and you're like hey I built this nice atomic unit. It's actually able to dissect it and bring that all
[15:40] apart and put that into its knowledge snippets and then reconstruct it at the time of finding the right answer. So it knows all the measures and dimensions and their interplay with each other. And then when the user asks a question that can be answered by that, it finds it using those knowledge snippets and then
[15:53] it uses a metric view to give them a deterministic answer. Really, really important that data modeling is going to be the fuel for consistent and high quality AI experiences and there's really no substitution for it. You know, otherwise you just get hallucination.
[16:07] Yeah. And so fundamentally that is what gene is, right? It's an automatic context graph to deliver grounded answers. So it's a context graph for your business. It's automatically maintained based off of data sources across your company and it's fully
[16:20] secure which is great. But that thing in the middle automatically maintained. Yeah. But it's maintained but it's maintained on the top of the data modeling that you did before. That's what it's using as that fuel for maintenance. So your data modeling quality directly reflected in the genie
[16:32] quality you're going to see uh in genie ontology and therefore in all the genie experiences you have in data. All right. Last but not least, switch over to Shan.
[16:45] Look at the time. Good. Okay. All right. So, uh, you know, row and column level security on data bricks means writing custom views. So, we do have today row filters, column mask, and Aback. So, attribute-based access control are all
[16:57] GA and they propagate wherever your data does. So, these are UDF- based. They're applied at the table level and they're evaluated every time a query goes through those tables. Those same policies are enforced whether you're
[17:09] quering it from data bricks SQL from genie from ML feature serving lakebase one definition everywhere. So this is a key difference between most of these tools where you have to go redefine these policies or rules at every single system and and as soon as you leave the
[17:22] system you have to go and you lose that that governance. So um we made sure that Aback itself is GA. So instead of governing by individual grants so Shannon you get access to this and Kyle you get access to that. Instead, we can have a policy that the team in the field
[17:34] gets access to this or people in a certain business unit or people in a certain corporate function, they get that access. And so once we apply, so this works both ways. So one, when somebody moves into an organization, they now acquire that tag and they
[17:47] automatically get access to those systems. When we get a new asset and we tag it with that thing, it's automatically available for those users. And then obviously in reverse when we remove a tag or remove a user or they switch functions it's really easy for us to just by switching those roles
[17:59] switching those tags to change the access that they have on different. So it really makes it really easy for you to kind of like move people between systems and it's really what it's about. It's about portability of those permissions as people move through roles
[18:11] leave join companies and data itself moves through different life cycles. You can tag columns like PII once and the policy auto applies to every table that tag including future tables which I love. Uh and then finally your model
[18:23] security travels with it. So whether you're going through directly quering in SQL to BI to AI to operational apps again this unity catalog governance is following it. The rowle security policies are following it aback's following it wherever that is and that's
[18:35] really like the promise of unity catalog. That's why like some ways why a lot of you are here like because we built something right here that delivers that data governance on your good data modeling and makes it available wherever it goes. So your data modeling can be
[18:47] reused everywhere with the confidence that security follows. All right. Um, that is it for me is up for Mr. Barrow. All right. Thank you. >> You skipped one. >> Oh, I did. What? I got an extra slide.
[19:00] Oh, I forgot the slide. Yeah. Yeah. You want to tell the slide or you want to skip it? >> I'm good. >> Sorry, I forgot about that slide. I forgot the second one there. >> So, we I was running late because of this slide, by the way. >> This slide almost caused uh Claude to
[19:14] eat my entire presentation. >> Oh, no. >> Yes. I this was originally two slides and I said combine this and the next thing you know is cats and dogs all the you know it's it's raining acid you know everything's destroyed anyway uh it's
[19:27] also the reason why these uh these little placeholders didn't get removed because I had to restore back from a previous version so all these placeholders that actually never had a video were not supposed to be there all right so you know we can now treat a transaction as a real unit of work
[19:40] across tables and even multiple formats you can begin a transaction update a delta table insert into another table leap from an iceberg table and then committed all as one atomic publish coordinated by the catalog. So since
[19:54] this uh last year that's expanded into a few important ways uh you know multi-statement transactions are now GA. I think we were actually announcing it last year. We actually had most people ask about it after but yes it is it was in preview at the time. you know rowle
[20:06] conflict detection is GA so concurrent writers do not have to lock the whole tables and external engines like spark flink and ductb can now participate through catalog commits so the bigger message is that this is an open standard
[20:19] contributed to delta lake this is not a data bricks only trick it's the foundation for interoperable coordinated rights across the lakehouse
[20:33] so we unified the open by API story with why we're unifying the data formats so you on the left everything that's new since data and AI summit 2025 is you know kind of listed here so we know what's changed since last year um but for external access to manage delta external rights are in beta credential
[20:46] vending GA volume credential vendings and public preview and aback for external reads is coming uh on the right you know the why so content metadata doesn't need to differ between Delta and iceberg so we're designing one unified
[20:58] metadata layer that improves both co-designed across both communities and improved together rather than being constrained by either format's legacy. So customers should not have to choose
[21:10] between delta and iceberg anymore. Both formats already share the same underlying data with parquet and deletion vectors. And now the industry is converging on a unified metadata layer as well. So that means delta and
[21:22] iceberg clients can read and write against the same foundation without transition I'm sorry translation overhead or uniform sitting in the middle. For Delta customers specifically, that means a better Delta,
[21:34] faster reads through bounded log replay and manifest level pushdown, plus more efficient writes through incremental tree updates and parallel bulk operation. So the real takeaway is that it's not about picking a winner between
[21:46] formats. It's about giving customers a better architecture underneath the hood. You know, a common uh assumption is that data quality is and data bras stops at table constraints, but the story is also
[21:59] much broader. the foundation. You've got check, you've got not null, you've got unique, and type enforcement on delta and iceberg tables. On top of that, LakeFlow pipelines let you declare expectations inline and decide whether
[22:12] to warn, drop, or quarantine bad data. And now the newer layer is data quality monitoring, which is schema level, one-click monitoring that learns normal patterns, watches for freshness and completeness across important tables,
[22:24] and prioritizes issues using Unity catalog lineage. So the takeaway is the data quality on the lakehouse is not just enforced. It's declarable, observable, and increasingly intelligent.
[22:40] You know, last year I spent a lot of time on this one, but we're going to summarize it this time because we're not going to have time for demos and such, but you know, capability 9, this is the physical model. So like I a lot of times I tell Kyle that he focuses a lot more on the logical model and I'm much more, you know, uh focused on how do we get
[22:53] this data stored so people actually want to query it. So uh the physical model deep dive you know so the reality is that auto liquid clustering is now in G you know last year I demoed that uh we we saw every layer of performance increase as we went from you know no
[23:06] clustering or partitions to what happens if we finally have stats to what happens if we have it clustered and we even tried autoclustering and autoclustering has gotten extremely powerful now in fact there's new ways now where you can actually enforce that op that clustering
[23:18] to happen even after only a couple of queries without even waiting for the background job to come and do it. So if you're in, you know, a testing phase where you're comparing one platform or a product to another or you're comparing different columns to another, you can enforce that to happen immediately. Um,
[23:31] you know, predictive optimization also runs optimize, vacuum, and analyze. You know, a lot of our customers, some of the biggest advantages they get from from this is just because they they rarely vacuum their tables. And so you'll see cases where they drop their storage costs by 80% or more just
[23:44] because they've actually run they're actually running these vacuum commands. So this actually does this on every uh this maintenance on every single one of the UC managed tables. It keeps those autostats current. It keeps the
[23:56] optimized data current. All the workload aware layout reads your query history. And the only I'm sorry, data layout decision worth making by hand in 2026 is actually just to turn this on. >> Yeah. And just real quick, this is the only technique in this whole slide that
[24:08] saves you money. All the rest of you, we want you to spend more and do more, but this is one that will actually go save you money. So take this one home and do it. So what's funny is like I I did a lot of the benchmarking for the product team for this session this week that
[24:20] they're getting announced. But um this was the first year I've tested some data where the partitioning choice was actually worse in terms of performance in TCO than clustering. So it took a while but then clustering is actually now better than even just choosing
[24:32] partitioning at storage level. So last one. So you know the the same unity catalog model drives three paradigms now. So I think that everybody saw in the keynote now we've obviously we had lakebased get introduced last
[24:44] year and then now we have lakehouse RT which ironically they did still announce as Raiden in this in the meeting this morning. um you know what what does this mean now actually if I've got three different paradigms of data I've got
[24:56] data still stored in lakebase I've got data stored in my lakehouse you know and we have you know a new engine with lakehouse RT did did anyone catch an LTAP did that did that get announced >> got an out there >> so is it who knows what LTA is does
[25:10] anyone remember >> exactly right >> it's HTAP with better marketing it's it's it's HTAP with an L, right? Um but
[25:22] but what they have actually done with it is actually much more powerful because you know you could try and do something that's extremely difficult with the hybrid, you know, access approaches. But in our case, they actually did make this
[25:34] very performance. So that what does that mean when it comes to laying out this data? If I'm going to now use, you know, the lakehouse RT, this new real-time lakehouse um optimized performance engine uh and the data brick SQL side, how's that going to actually read this
[25:47] in a performant way from from the data stored in lakebased that Postgress data, right? So, right now we have three different ways to now access actually four if you consider traditional DBSQL, but okay, four different ways to access
[25:59] it. So, you know, here's how the lakehouse RT works under the hood. You know, agents, apps, and BI read in real time. Lakebase handles transactions and lakehouse RT handles real-time analytics. So with performance isolation between them both over a single copy of
[26:12] data on the lake in open delta and iceberg format. But you know stepping back to the why right voters are putting AI agents into production operational and analytical systems are converging and pipelines remain a drag you know so how do we make
[26:24] sure that we're actually optimizing for how we're working against this data in real time and you you continue to use the same DBSQL warehouse that you would before except now you're actually going to be using this new lakehouse RT warehouse type
[26:37] >> so it did eat it look this is one it actually combined another one so anyway so what's the catch so no catch just one copy of data no migration same price as DBSQL
[26:51] and this is where we are today so these are the three different formats okay these are the three different engines if you will of how you can get to this data all right so Delta and iceberg are both scalable open data formats they co-evolved as parallel standards share
[27:04] an interoperable data layer of parquet and deletion vectors the content metadata differs translation layers like uniform bridge that split translation um between them actually work pretty well u with the new versions of v4 V5 coming
[27:16] out. But now you've got three different ways to access it from the lakebase side. Put DBS SQL or lakehouse RT. So again, Lakebase, I always describe this as like if you know in your house where your keys are. You walk into your house
[27:28] and go pick up your keys and you can leave. If you know exactly what you're looking for, you use lakebase, right? If you do not know what you were looking for, that's where you're going to use DBS SQL. That's when you're using Lakehouse RT. Okay? That's whenever I
[27:40] need to actually analyze the data to try to find my answer. If I don't know the answer yet, you use Lake House RT, right? So, here's the top 10 at a glance. All right, summarizing everything that we've
[27:52] seen. So, it's quick recap here. Um, >> so you know, take this back to your team. >> Let him take a picture. You took too fast. >> I went too fast. >> Go back. Sorry. I didn't take a picture. Just take a picture.
[28:05] >> Oh, you made up the slide. >> Yeah. Not you. Not you. Good. >> It ate my notes, too. Did you see that?
[28:17] >> Yeah, it did. I saw that. >> I started reading the previous slides. It almost destroyed it. >> I guess >> we could have just come and done stand up the whole time. >> AI is the best. >> I don't have my slides, guys. We're just going to do jokes.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.