Skip to main content

Data Modeling 101 for Data Lakehouse Demystified

Summary

  • Lakehouse data modeling paradigm: The data lakehouse combines data lake flexibility with data warehouse structure, requiring new modeling techniques to unlock full potential.
  • Dimensional modeling for the lakehouse: Traditional dimensional modeling adapts to lakehouse architecture, enabling BI dashboards, ML pipelines, and self-service analytics.
  • Demystifying architecture patterns: Practical guidance helps organizations adopt the right modeling techniques for their analytics and AI workloads.

Data Modeling 101 for Data Lakehouse Demystified

Watch: Data Modeling 101 for Data Lakehouse Demystified
Talk by Luan Moreno Medeiros Maciel, Lead Data Engineer, Pythian
In today’s data-driven world, the Data Lakehouse has emerged as a powerful architectural paradigm that unifies the flexibility of data lakes with the reliability and structure of traditional data warehouses. However, organizations must adopt the right data modeling techniques to unlock its full potential to ensure scalability, maintainability and efficiency. this video is designed for beginners looking to demystify the complexities of data modeling for the lakehouse and make informed design decisions. We’ll break down Medallion Architecture, explore key data modeling techniques and walk through the maturity stages of a successful data platform — transitioning from raw, unstructured data to well-organized, query-efficient models.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

How does a data lakehouse differ from traditional data warehouses?

The lakehouse combines low-cost storage like data lakes with ACID guarantees and structured schema like data warehouses, enabling diverse workloads (BI, ML, real-time analytics) on unified data.

What modeling patterns work best for lakehouses?

Dimensional modeling, medallion architecture (bronze/silver/gold layers), and event-driven modeling all work well—the choice depends on your analytics and AI workload requirements.

How does medallion architecture organize data quality?

Bronze layer ingests raw data as-is, silver layer applies transformations and cleaning, and gold layer curates business-ready datasets—each layer supports specific workload patterns.

Can lakehouse models serve both BI and ML use cases?

Yes, well-designed lakehouse models with proper governance and quality gates can efficiently support both analytics dashboards and machine learning feature engineering from unified data sources.

Full transcript

0:00 Introduction

All right. All right guys, can you hear me? Well over here. Over there. Okay. So, first and foremost, thank you so much for coming to the session. There's a lot of great sessions happening right now. And amongst all of those, you just choose to be here. Thank you so much for that. Well, today my talk is going to be about data modeling 101. And I would like to dismify a little bit about lake housing and data modeling.
And there's a reason why the audience is crowded because even though data modeling is just from the past decade, we still have a lot of issues and a lot of unknowns regarding how we're going to put modeling principles and strategies inside on top of data lakehouse. And this is what I would like to discuss today. But before I get started, I would like to introduce myself a little bit. Promise I'm not going to take too much of your time, but that's my name, Luan.
I'm from Brazil. I I've been boots on the ground for the past 17 years. Uh for the 17 years, I would say the 78 they're totally focused on data engineering stuff. So I saw the rise of the data leak housing. Uh I was in fact in the event where Matis Haria just released the Delta Lake uh which was a a huge leap on the data analytics ecosystem. I'm just sparking data bricks expert uh I'm just Microsoft MVP for the past 10 years in a row and also I became the very first Brazilian to hold the data bricks MVP.
Data bricks MVP is a program that talks about how people share content across the globe and we are here exactly to do it. We have a booth on the expo as well. So if you'd like to come and visit us, it would be a pleasure to talk a little bit more. And also I have the pleasure to be with the data bricks product advisory board. So I'm just deeply focused and in tune with the data bricks folks to talk a little bit about the experience the data engineering things like such as lakeflow zig and stuff like that.
Well uh before we get started I would like to get a sense a little bit about the audience. So just a quick show of hands for people that currently implementing or have like the Medadium architecture on your organization is just nice a solid 80%. That's that's perfect. Now just for the honest ones uh bear with me. Uh you have gold tables in your lake house that you're not sure if anyone actually uses. Okay. I think 30%.

2:05 Audience

Okay. And lastly, you ever debated whether something should go in silver or gold based on whether it's operational or analytical? Yeah, you know, just don't be shy because I have this problem a lot. I just quite get this problem a lot. And this is this is what I'm trying to unfold here. I'm going to try to bring some experiences from the trenches alongside for all my ears of implementing solutions and data league solutions for median and um large customers as well.
Um, and of course they would like to start off the foundation, right? So when we talk about lakehouse, we're talking about I I like to put in this way. There's a lot of ways to show this, but I would like to see as a three layers. The first layer is where you sit and how you sit the data into the data lake. So just pushing the data from different sources into the data lake which is a file format, right? And here we would like to make sure that we're using optimized file formats such as AVO or RC and park.

3:11 Data Lakehouse

The second stage is where your data lakehouse lies. So you have your different uh styles of lake housing which is Delta Lake, iceberg and hoodie which is a table format. So here you truly get the best of your data lake plus the dare housing capabilities meaning that you have asset you have concurrency you have uh scheming forcement you have the whole shebang all the things that would love to have on the data warehousing we have by this uh by the data lake house and lastly we have the medallion architecture which is truly a data design on top of data lake house and that's interesting because if you look for a Second, when we think a little bit about bronze, silver and gold, we are truly think about the maturity level and the process and stages that takes to your data through the release to the final user.
And final user could be another pipeline, could be a user truly consuming it, can be a level report, can be a dashboard, could be anything you want. And this is where I think um it gets a little bit messy when you start to think a little bit a bit about each one of them and responsibilities about each one of these layers. And this talk is going to be exactly about that. How do we unfold the bronze, silver and gold in such a way that we can deliver the maximum capacity and how do we use modeling techniques inside of these layers to take the most of the data lakehouse.
Okay. So yeah, let's talk about this right now. The bronze layer, um, I would say that fairly bronze layer is just pretty straightforward. Uh, is the easiest part. Basically, you just have to think onetoone mapping, one to one replication. Meaning that if you have different data source and if you'd like to bring the data from the different data sources into your medallion architecture, what you're going to do is you're going to bring them um as onetoone mapping which means that you have a direct replication from the source data.

5:17 Bronze Layer

So if you have a Kafka topics with a 100 tables, 100 topics, you're going to end up with a 100 bronze tables on your bronze layer, right? um we don't have any business logic we don't do anything about the data except for adding metadata and I strongly recommend you to think a little bit about metadata metadata drives the whole good process around data processing at scale so always considering when you're moving data from source to the bronze adding data metadata such as ingestion time stamp uh source um source uh form um who did that detail process.
So as more metadata you bring to the table, the better you're going to see how the data fits into the bronze architecture. Um so the bottom line here is just that start simple by capturing everything and then transforming nothing. Um and there's a lot of uh questions around this regarding oh should I just bring all data to here? Yeah well that truly depends of your business. uh but in the end of the day what I realized across a different lakehouse platform that I was trying to implement is that eventually people would like to consume pretty much all the data from your data sources.
So eventually what we discovered is would be much easier to have one system that ingest the data and just push to there instead of having different ETLs doing this. So we just want to make sure they're not copying different same data here multiple times. We just have to reduce this and eventually what's going to happen here is you have a layer that sits in between the data sources and the bronze layer that brings you the capability to inject ingest data uh incrementally or in CDC form.
All right, that's just pretty basic stuff. Second, we're going to talk about the silver layer. And I don't know about you guys, but that's where things get a little bit nerdier here. Um, that's a lot of articles out there. There's a lot of great books out there like the data league definitive guide, the medium architecture implementations. Those are really great books that pretty much talks about the nuts and bolts in how you're going to build up your bronze, silver, and gold layer.

7:38 Silver Layer

But be careful because this part is the trickiest one. It's really hard to make that right. And in the way that we managed to do this, it's going to truly unfolds how we're going to leverage that data into the gold layer. So we're going to be talking a little bit about the silver layer because this is in fact is the most important one. We have two basic techniques here. We can truly use third normal form to pretty much bring the data from bronze.
You just push into silver. You just apply minimal transformations on it. Uh the focus here is to be right optimize it because you would like to make sure that ingesting data as fast as possible. Um or in the end of the day what you would like to do is since data lake housing is about data integration you could use like some immunos capability to receive data from different sources then just collapse them into the ODS and then you deliver this data to the gold layer.
But if you read about the articles out there, you're going to see that whenever people are talking about silver layer, they're always talking about just enough transformation. And what does it mean just enough transformations, right? Uh there's a little bit, it looks simple, but if you just dive deep into that, it gets a little bit hard to understand what does it mean? Because in the end of the day, we have to have enterprisewide view inside of the silver.
The silver is going to be the place where we're going to group, organize, cleanse the data to deliver faster gold layers, gold tables to the end to the final users. So that part is pretty important. And lastly, the gold layer. The gold layer is where truly the users are going to read data from. They're going to consume your data. And in terms of data modeling techniques on in the ways that you can leverage this is you have two options truly here.

9:44 Gold Layer

Uh actually you have other ones but these are the most commonly used ones which is the story schema. Everybody is just aware of this time schema. Just raise your hands just a little bit. Okay just pretty much everybody. Good. That's style and we can opt to go to full-blown one big table. It's funny when start deal with data teams and we're discussing which one is the best to deliver there's always this talk about uh Kimbo or one big tables are best and truly depends of the use case that you're trying to achieve there's no right and wrong answer here in fact you just need to understand about the business requirement in order to choose between them but sometimes you're going to see that even inside of domains or departments you're going to see like this delivering processing happening not in only only on one way you're going to be two ways you're going to have some story schema and you have some OBTs there's a lot of fighting against this because people are trying to always put the you know the feet on the ground and say hey you must use s schema or you must use OBT but actually what we're going to see here at this presentation is that you don't have to actually fight you don't have to actually choose to choose one or another instead you can have both if you have a foundational silver layer well established and that's the provoking question that I would like to raise right now can we use both because there's a lot of discussions about that we're going to discover this um in a minute so the common thing that we that I get from out of the consulting engagements that I've been part of is that we built by the box we have all the layers they are working bronze silver and gold but isn't working properly.
It's not working the way that used to work. And the answer is not actually about the pattern because pattern is about you following something. It doesn't mean that it's going to be 100% fitable to your use case. You have to adjust as per your needs. But it's really about the maturity stage they're in right now. And by discovering this stage that you're right now, that going to gives you like a little bit of space to think about what comes next.
And that's where I started to talk a little bit about how I'm going to approach the session here. Because being out of the woods just in this consulting engagements and see how customers operate inside of Medallion architecture, I figured out that there's no right and wrong answer, but instead maybe could have a methodology on top of Medallion architecture to help you to achieve what you truly want. Because in the end of the day, we just need to get speed.
You have to bring data to your end user. You have to guarantee that you're going to be able to deliver as fast as possible. You know, a lot of things are changing in a minute. You have to be adaptable. So, this is the idea of medallion architecture. maybe not stick uh with all the needs that you truly want right now, but also thinking a little bit through the process going to help you to unravel what you can take the maximum of each one of this.

13:12 Copy Paste Architecture

So we have this first stage what we usually call copy and paste architecture and what does it mean is basically what we're doing is we have different data sources we bring this data to bronze and I'm just push some SQL statement here to make easier for us to understand and there's some people laughing here because it truly happens and that's normal uh you're just adding some metadata for example current time stamp and just make this as a low time stamp which is fine we are adding metadata on top of bronze that's extremely correct This is the best process, best way to do it.
Secondly, you're just bringing the data with silver. However, you're not applying anything. You don't have any domain logic applied. You don't have any other merging uh strategy applied here. You're just simply removing taking out the customer ID which is not new because it may not be acceptable for your end user. And look what happens in the end. You have renaming columns without any optimizations. That doesn't ring any bells for you guys.
I have seen this happening a lot. And if you ask me is this wrong approach? No, it's not because sometimes uh in the way they approach this, you're just thinking a little bit about the straightforward ETL process kind of workflow that you used to do in the past with datous if you think. So you have your ETL, you have your staging area on bronze, you do your transformations on silver, and then you build up the story schema on the gold layer and that's fine, but there's like some problems here with this implementation.
So first you're just copying data in different places and trust me I I was in a customer and we're like doing like some assessments and we found out that for the data duplication he had 10 terabytes of data because of this because they're basically moving the same data across all the layers and delivering the data in gold. And guess what? A lot of customers and a lot of data platform engines, they say, well, you cannot let people get into the silver, which means that if you want to access interact with me, you have to be on gold.
So that's what happens. People are just always getting data from gold. And if you have a data data scientist for example, they would like to consume the raw data, guess what happens because of this? He would need to consume from gold. And then you have to duplicate the data from silver to gold to make that happen. So that's happening right now. People are doing that and there's no problem unless you start to feel like the hurdles of it, right?
Um you have data duplication. You're not reusing the logic. You have zero transformation. You have a poor performance because you're not optimizing for the end user. You're not adding groupings. You're not adding like capabilities like summing. you're not building uh the logics underneath the reporting engine. Um okay so that's the first phase. The second stage is about the awakening of the domain um idea. So, and that's the tricky part and this is where I start to see people truly understanding the whole meaning of the modern data architectures right now which is eventually re you realize that well I have my different data sources that comes from different services ultimately I would ended up with multiple users table for example like as you can see in an example you have Postgress user you have my SQL user you have Salesforce user user.

16:10 Awakening the Domain

And if you think about the copy and paste architecture, what's going to happen is you're copying the data all over until the gold to be consumed with minimal transformations. But here you're being a little bit better just being smart because what you're doing is you realize that users may be a domain inside of your your organization for example and then what you're doing is you're collapsing all the data of users data inside of one big table call it consolidated user and that's what we call domain.
So you can have a users domain, you can have a transactions domain, you can have a uh product domain, so on and so forth. And that happens a lot because now once you start to share the data from gold your data analytics team or your analyt your data analytics or your analytics engineer they would look up for the domain tables and he's going to be able to construct the data pretty easily because all the money all the transformations happen prior that part.
So you're helping your teams to be faster. You are helping them to unlock the best of the Medadano architecture and also you're bringing the responsibility to the right place. You don't want your business users to interact with your mudging state with your swamping state I would say. You just want to make sure that people will be consumed by a clean area. And you can go even further. You have two ways to do this. The first one is sometimes you see customers that they have like heavy transformations between bronze and silver.
So the option would be option A which is adding a technical cleaning path before building your domain table. And the second you can say oh no I'm good. I know what I'm doing. We know that our team is pretty mature to take this data. So what we're going to do is we're going to remove the cleaning part because we don't have to replicate the data from bronze and silver and instead I'm going to go full-blown to the domain area and then we can just go straight up to the domains and that not only happens with the data replication but it happens with a lot of other things.
We start to cover way more. You start to reduce the amount of details. Can you imagine how many tails we have by you know having this all disparate sources bronze silver tables you you have a job you start a job that runs in two hours and eventually you realize they're running 23 because you're having so many now to fill up the gaps each one of these tables that eventually you ended up with a lot of ETLs running to fill up the data so here you're streamlining the process by adding domain stable does it make sense Okay, so in the end of the day, silver is no longer a clean state table.
It's way more than that. It's just like becomes a semantic core of the business. And if we stop and think for a second and take take a step back that pretty much lines up with the data warehousing, the mo the enterprise data warehousing stack. What you want to make sure is in some place here inside of the Medano architecture, you're going to have your facts. you're going to have the clear vision about all the data that is going to be participating of your delivery.
Well, now we solved the silver area. We pretty much building the the domain tables, but then people starts to create a lot of golds and that's where I see like the gold cut here. um we start to be pretty much smart on the silver side and we relax a little bit on the gold area and that's a little bit dangerous because what you're doing right now is we got it smart we just push data from bronze to silver we consolidated this data into domain tables and now I'm building golds out of this but the problem lies here is you have eventually other people is nick in and also instead of going to your silver zone and building their goal tables.

20:10 The Gold Cut

No, why they should do that, right? I'm going to just plug inside of this KPI or your ops team instead and I'm going to get the data from there because it's pretty much what I want. I would have to do small transformations to get there. And that's a bummer because you know what's going to happen if and it's going to happen eventually the KPI authors should be reside in different department and they're going to have like different business needs they're going to change the structure of this data and guess what's going to happen with all the downstream users consuming from this data possibly it's going to break so that's why we avoid to consume the data from other goals instead we have to plug into the silver layer.
So that's the reason why we establishing a pattern we establish best practice inside of silver to guarantee that we can deliver the data in the best way as possible. Something that people sometimes u underestimate is about the data contracts between the silver and gold. I have seen a lot of people talking about data contracts between the gold and the end user but they're not talking too much about silver and gold and guess what happens silver layer is our fabric right so that's where the point where data engineers are working they're just doing transformations they're just doing all the munching the all the stuff it happens here so once you release the data you just want to make sure that's going to go through a contract this way you're not breaking any downstreams pipelines connecting to that.
So we have the gold spraw silver domains are enforced by the contract layers unfortunately and you start to create a lot of goats and goats can become really expensive in terms of processing. So yeah, you should start to see long waiting times about spark jobs running by consuming the data, doing other transformations on top of that and then just sprawled everywhere. But the point is there has to be a better way. Uh well we have seen the chaos, we felt the drift, we thought the gold layers bra.
So how do we truly design the data lake house in a purpose and in a way that could unlock its full capability? So that's funny because in one of these consulting engagements uh the customer had a challenge to transform and operate 10 terabytes of data and we are thinking about the strategies in how to do that. Um customers is struggling a little bit with the domains awakening plus the gold cut. A lot of golds being just linked up with other goals.

23:26 Customer Challenge

It was super messy. So we started fresh and we thought about a methodology and how we're going to push into the mandum architecture in order to be able to accomplish this whole end to end process. So let me show you something. Well lake house meets discipline. So data lake is about scalable data integrations. I never seen data lake house being used for one data source. I have seen data lakehouse being used for many different data sources just CFKA object storage databases NoSQL databases APIs FTPs yada yada but there's another component pretty interesting called a data vault which is about a structured and governed integration that sits on top of the data warehousing.
So together they can become like the M in heaven. And the reason why is both they were born and raised with the intent to be data integration driven. So let's talk a little bit about data vault. Well, data vault was first introduced in 19s by. That guy pretty much picked up the word from Bimmans and he just worked through it. uh we have had a lot of updates across this last 30 years. He has been updating um this methodology since then and he he just worked with really large customers.

24:55 Data Vault

I don't know if you guys heard about Locket Martin. So that's the company behind NASA, behind all these big companies, the DOS. Um, so he just got up for the challenge to be able to ingest pabytes of data. And then ideally was a little bit tricky to do that, right? We wouldn't have the same technologies and the same characteristics that we have today. So the idea was well now um and mainly because of AI as well you're generating a lot of data and eventually data engineers data scientist MLOps engineers whatever you name it we're going to have to be able to data transformations so ideally you have a lot of data you just bring this data inputs to a place you just store them there but then you have to be able to build a methodology that sits on top on your data warehousing or your data lake house that allows you to build something scalable, right?
Robust, auditable and resilient. So here what Den says is the logical modeling works more than the physical modeling which means that if you design a data vault you can either apply inside of your modern data warehousing such as snowflake data bricks SQL whatever or you can push it to the data to the medallion architecture and that's where we're going to see the perfect match between these two systems they're going to work together in order to offer you the best of breed.
So the data vault modeling is just about three major entities. Uh we have hubs, links and satellites. Unfortunately, we don't have time to talk, you know, in details about each one of them. I'm going to be available all this week here at summit if you like to hear more about data vault. Also, yesterday we had a talk about data modeling strategies. What is a workshop that ran on Monday? They're going to run this workshop on Thursday as well.
They talk about the different modeling techniques. pretty interesting. Um, if you have a chance, I and if you're interested by the topic, I strongly recommend you to do so. So, basically what what it's doing is here's is what we call the raw data vault and we're going to see in a picture all the layers um how they establish and how they organize inside of the medal architecture. But the hubs is the B business object.
It hosts all the metadata and keys which is the business key. So imagine for example that you have a user and that user is not recognized by the user ID because user ID spuns across different sources and user ID one in Salesforce it doesn't mean that it's user ID one in MySQL. So how do we make this link? So we have to have a business key. So the key of this modeling technique is to have a business key to group to link this data.
So let's imagine for example that SSN would be like the user key. So we're going to host all the keys in a hub. We're going to host all the distributed attributes of a user in the satellites. We're going to push all the attributes to there except for the key. And then we're going to have the links which basically is the joint between the different hubs. So what happens is I no longer have sedd anymore because the key is decoupled from the scripted attributes.
We don't have to handle this anymore. We are handling with a pen only technique which is ingesting for you to get the latest updates. So here you no longer have sedd. You can get the latest update from your table look about the descripted information and it's pretty fast because we don't have to check whenever you're ingesting data into this model. We don't have to check for a business if the business key already exist in a hub.
So let's say for instance that we have a user that got like the birth uh column updated. The social security number is the same. We don't have to touch in the hub. The link between that user and the transaction is already established. I don't have to do that as well. The only change that I have to propagate is on the satellite which is a new entry of the scripted attribute which means that this model is made for pretty intensive write operations and that helps a lot in a world where we have to ingest fast.
We have to deliver even faster and we have to make sure that everything is working concisely end to end. So how do we plot the data vault inside of medallion architecture? That's a little bit challenge but let me show you how we are usually doing this. And first you have your data that comes from gold. This is what we call our persistent staging area which is our bronze area which is our staging uh area is the same. So then as per the characteristics of bringing data from bronze to silver ideally what you wanted to do is you have to maintain the same granularity of the data that you're bringing.

30:07 Plotting Data Vault

So we add here hard business rules. So what we're doing here guys is that we're bringing a methodology inside of the medallion architecture in order to be able to deliver the best of its capability. So we adding hard business rules that are going to guarantee data quality rules that are hard business rules that is going to guarantee that we're not going to change the granularity of the data. Now here's the interesting part the silver zone.
So there's a lot of people fighting against which is the best model to use in medallion architecture and I don't think this way. I think that there's no one-sizefits-all. There's no model that comes with all the bells and whistles but if we bring all of them in a structured way in a methodology we may have like the best model out there and this is how I think and that's how I usually approach systems and it's been working really great.
We have this first entry which is the raw data vault about the objects that I showed you right now like previously the hubs links and satellites. So we store this data inside of the silver layer. We push all the silver data to there. We maintain the historical information so it becomes auditable. We just make sure that's a pen only. So extremely fast they are persistent tables. Secondly, we have an extra layer that sits inside of business uh inside of the data vault which is the business data business vault.
And that's interesting because now what we're doing is we are building the same vision of the raw data vault. We are not changing the grain of the data. But look at this. We are allowing people to consume this data and we don't have to release it inside of gold anymore because we can create views that sits on top of the raw data vault and we can allow people to get into this zone that is still organized because we have different other type of tables that we can deliver like snapshotting of the data latest latest registry historical information of that and we can apply all the transformations without conflicting to anyone.
So now data engineers instead of having to instead of your data analytics team sharing the data for the gold, what you can allow them is that they can tap and play, they can plug and play into the business vault and can still consume the data that they want to consume without me replicating the data. And that's the best part. Now since we went through the bronze to silver, we just did the transformations, the cleansing, we did the verification, we did the matching, the uniformization of the data.
Now we can add soft business rules which is truly changing the meaning of the data somehow. So we adding total spend, we adding customer segment for example and we're adding some metadata layers on top of that to make sure that you can track this from back to source. And now you can choose which model you want to deliver. Isn't that good? We don't have to fight anymore saying, "Hey, let's use data vault. Let's use s schema.
Let's use OBTs." Instead, you're going to take a different approach. You're going to access your customer. You're going to understand the business requirements. You're going to understand all the needs that the customer wants. And based on that you're going to deliver the best gold experience that they can usually have. You don't have to pick between OBTs and his story schemas because you have a well structured foundational layer sits on that sitting on silver layer.
Now you can build anything you want. You can build views. You can build a gigantic story schema. You can even I mean we we have customers that we don't house anymore the sedd changes on the story schema or they doing that once a week we reprocessing the story schema again just picking the latest data from the silver because now we can do this and what they're doing is that instead of handling CD which is pretty expensive you don't have to do this anymore.
You're just taking the data from silver because that became your gold standard place where you can just plug into it and deliver the data in the way you want. We don't have to fight about what's the best model anymore. We are now thinking the best way and how we going to make changes through a methodology that truly works. Well, just looking a little bit how we going to build this on data bricks and data vault. Yeah, we what's going to realize is that once you get off this session, you're going to see a lot of articles of data bricks talking a bunch of data vault and there's like amazing ways to do this.
You can ingest the data through CDC capabilities. You can just add DLT to run your journaling tables, your bronze tables. You're going to create a data vault through Unity catalog just hubs, links and sets. Eventually you're going to have your business data sitting there as well through like uh materialized views if you want and then you're going to delivering the way you wants OBTs virtualizations applications whatever you want and the best part is you have the freedom now to offer to the analytics engineer to use DBT SQL mesh whatever you want to plug either on the information delivery layer or into the business layer allowing allowing them to make any changes they want without sacrificing the data engineering changes. because we are working two different sil silos.
This is just a little bit of picture in how we would build that here. This is like uh a use case where we ingest trillions of data through the vault using DT and then you should see something around that. And the next question is are you up for the challenge? Right. So I brought up some the 19-day challenge. This is something that is to do with the customers talking a little bit about the steps that you're going to take in order to bring that methodology this process into your requirements into your business right so first you access and you have the awakening part the second part is where you design and align your goals just building the PC and city captives on board and lastly you deliver and celebrate this because eventually you want to make sure that you're following a procedure This is what we just built as like a skeleton to deliver to customers and we have been following this across the board and it's just it's just working pretty well so far and I mean just the last message is that trust isn't built overnight but starts in 19 days embrace the vaulted lake house and architecture data future now I want to say thank you so much for being the session and I hope that was fruitful don't forget to complete your survey and thank you so

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.