Skip to main content

Vibe Data Modeling: AI Data Models That Match Your Business

Summary

  • Industry versus business models: Generic industry data models don't capture an organization's specific language and business rules like custom business data models do.
  • Vibe agent automation: The Vibe Data Modeling agent applies 250 rules to validate data relationships, naming consistency, and depth automatically over 10 hours.
  • AI as assistant: The Vibe agent acts as a 10x assistant for data modelers, enabling faster pipeline updates while preserving human expertise and judgment.

Vibe Data Modeling: AI Data Models That Match Your Business

Watch: Vibe Data Modeling: AI Data Models That Match Your Business
Vibe Data Modeling is a new open-source initiative for architecting lakehouses around agile, AI-driven business models instead of bloated, off-the-shelf industry models. Youssef Mrini (Developer Advocate) sits down with Amir Ali (Lead Solution Architect) and Franco Patano (Principal Architect) to unpack Databricks Industry Data Models and the Vibe Data Modeling agent.
Industry models represent a generic average of a sector. Business data models capture the DNA, jargon, and rules of a specific organization.
In this video:
• Why generic industry models fall short, and how business data models encode your organization's language and business rules
• How the Vibe Data Modeling agent applies about 250 data modeling rules, running around 10 hours as it validates depth, relationships, and naming consistency
• Why AI does not replace data modelers: a 10x assistant to build, validate, and update pipelines at the speed of the business
• A shift from bottom-up Bronze to Silver to Gold, starting backward from Gold-layer business semantics even before Bronze exists
• How to use 40 out-of-the-box industry data models from Databricks Industry Solutions on GitHub: browse, download as Excel to collect business feedback ("vibes"), and iterate custom versions with the agent
Browse the models and agent on GitHub: https://github.com/databricks-industry-solutions/lakehouse-industry-data-models
Learn more about Vibe Data Modeling: https://www.databricks.com/blog/reimagining-data-modeling-lakehouse-introducing-vibe-data-modeling
Chapters

Chapters

FAQs

What's the difference between industry and business data models?

Industry data models are generic averages of a sector, while business data models capture your organization's specific language, jargon, and unique business rules for better alignment.

How does the Vibe agent work on data models?

The Vibe agent applies approximately 250 data modeling rules to validate depth, relationships, and naming consistency, completing validation in around 10 hours of automated processing.

Does AI replace traditional data modelers?

No, the Vibe agent functions as a 10x assistant to data modelers, helping them build, validate, and update pipelines at business speed while keeping humans in control of decisions.

How do you customize industry data models with Vibe?

You can browse and download the 40 out-of-the-box industry data models from Databricks Industry Solutions on GitHub as Excel files, gather business feedback, and iterate custom versions with the agent.

Full transcript

0:00 Introduction to Vibe Data Modeling & Industry Data Models

For three decades, data heavy sectors been sold a highly compelling shortcut. Just buy off-the-shelf industry data model and you will save a year of engineering work. But anyone who has actually tried this knows the painful reality. You end up inheriting hundreds of bloated tables. Naming convention that doesn't match your business and relationships your workload don't even need. The supposed shortcut ends up taking months of timing and rewiring to get the baseline.
But what if you could deploy a structurally sound, highly specific business data model for your industry, complete with built-in governance, primary keys, and metric views in a matter of hours. Welcome to today's podcast. We're sitting down with the mind behind a massive new open source initiative, Vibe data modeling and data bricks industry data mods. We're joined by and Franco Patano to discuss how they used a rulesdriven AI Asian to build out of the box silver layer models for 40 of the world's biggest industries and how this project is about to fundamentally change how they you architect your lakehouse.
Hi Amar. Hi Ali. >> Hi. >> Maybe you can start by introducing yourself. >> Sure. Um I'm a lead solution architect with data bricks for the past like 5 years and uh yeah I'm excited about the session today introducing vibe data modeling and showing everyone the power and how it serve their needs. >> Hey everybody uh. I'm a principal architect at data bricks. Uh I've been around for seven and a half years now and uh super interested in showing you all the importance of uh vibe data modeling and these industry data models today.

1:12 Speaker Introductions: Amr Ali and Franco Patano

Amazing. So let me start with the first question. What's the difference between industry data models and business data models? And why industry data models not as attractive as before? >> So an industry data model is kind of the persistence of what we consider like this is what it a model for the industry is. It's kind of generic and we produce it with the vibe data modeling process. We kind of produced all of these industry models. uh you know Ammer and I have built an a team of bricksters that actually validated these models through all of our industries and that's kind of like a result set of what customers can take off the shelf and customize it into a business data model and they would do that with our Vive data modeling agent.

1:48 Industry Data Models vs. Business Data Models

They would take an industry model and then they would customize it to their business data model. So those are the differences in how these pieces fit together. >> And can you remind me what do you mean by vibe data modeling? The main reason we created Vibe data modeling is because like said industry data models if you think about it people used to create them because it's the average of all industries like if you have a tele data model it could serve the need for all Telos but it doesn't have your DNA it doesn't have your signature as a business if you use one specific rule in the business it doesn't have to be represented in that model because it has to fit all other telos so business data models are the one that are having your DNA and a specific when we build the vibe data modeling the mean the word vibe here represents the context that you will be adding to this agent.
So it capture all this DNA of your organization. You can give it all your subsystems all your uh you know um you know like data models all your jargon uh business rules and the Asian will mix it up and generate a model that represents your business. is not an industry model that is generic to everyone but it represents your own business. >> Okay, understand but what makes it different from any other data model generation that can use a frontier model? >> Yeah, that's a very good question.

3:41 What Makes the Vibe Agent Different From Frontier Models?

And I think uh if you go to charge or clothes and ask it give me a data model for an airline you know it will give you like a bunch of tables that you know if you look at them yeah it looks like they are airlines or teleico tables but if you go deeper they are inconsistent they don't have you know uh proper relationships there is a missing attributes and they're missing fundamental concepts and entities. So the difference is the vibe modeling agent is equipped with about 250 data modeling rules that ensure there is a good depth and coverage.
Uh relationships are respected naming conventions uh you know uh foreign key primary keys what is a many one to many relationship what is a many many to many relationships it goes into the depth. So the depth is what matters and the difference between this vibe data modeling and any other uh modeling tool or generic uh frontier model generating data models is the rules. The rules is really the essence of it. These rules ensure at every step of the generation of the uh business data model these rules are enforced at all times.
That's why if you run the gen the uh vibe data modeling it will take a few hours. It's not going to be it's not like a a 30 minutes or a 20 minute job. is taking hours about 10 hours to get a proper detail for your industry. Why? Because it ensures at every step these rules are validated ensures that they are consistent and applied everywhere. >> Okay. And I'm pretty sure you get you get this question a lot, but if natural language and AI agents can adapt and evolve schemas on the fly, uh does the traditional role of the dedicated data modelers will still be there in the next few five years maybe? >> Listen, AI is amazing. it every single day we find that it can do more and more and it can do better on these exams that we give it.

5:19 Will AI Replace Traditional Data Modelers?

But the one thing it's never going to replace is the humans in the loop. The business side needs to be consulted ultimately like they make a lot of these decisions. A data modeler is the interface between you can call it a meat proxy if you will uh between the business and actually producing it. I think this does not replace a data modeler. This is a true 10x opportunity for data modelers out there. They they can use these tools to build, validate, and push to production much faster than they have ever had to in the past.
Data modeling projects that I have been on have taken months to a year and most of it is parlay with the business. It's not actually building. So I don't think this does away with the data modeler. I think that it this is like a true 10x opportunity for the data modelers out there and we're hearing it on LinkedIn, Reddit, everywhere that they absolutely love it. Anyone who actually realizes what we've built here, they're like, "You just made my life so much easier."
So, I don't think it goes away. I think we make these people more effective. >> Yeah. If I had more examples here, I think uh part of it the people should think about the vibe data modeling as an assistant to the data modeler. Uh so it help them do a better job. The data modeler if you think if you have if you think like a business that have like customers and a debate in the business about whether should we allow customer to have multiple accounts or one account, right?
So some business will will apply or will allow multiple accounts because the business rules allows this others won't. But what this will be take debates inside the could take hours maybe days and the data modeler will be there the business analyst the business leaders maybe the company CEO himself will make the decision right but whatever the final outcome once it is sealed and documented this is the role this is the business business like decision uh uh role has ended this is the start of the data model your data model should reflect whatever the decision it's one account or multiple account now taking this and doing it.
This is the assistant. The vibe data modeling can assist and make sure this consistency of one account or multiple account is consistent everywhere in the data model. You shouldn't see places where it's one account and other places is multiple account. And this is the power of the VI model. It ensure the rules ensure consistency everywhere. So it helps the data modeler do a better job and become more efficient. Well, now you know.
It means you will be become more efficient and maybe achieve way more than you expected. For decades, offthe-shelf industry models turn in turned into bloated and maintainable nightmares for you. What was the breaking point that made you realize that the old approach enterprise data modeling was completely broken? Personally, I had uh a really a few bad experiences with data modeling where we did take offtheshelf data models and try implementing them and most of the time it cost more in development time to actually customize the model and put it into production than it did for the original model.

8:19 The Breaking Point of Off-the-Shelf Enterprise Data Models

And then it had us asking ourselves like, well, what if we would have just built it from scratch? Like would would it have taken us less time? Um, I think the moment I saw that what Ammer was building and he he kind of I don't I don't remember if it was in a slack you pitched the idea or somewhere like I caught a whiff of it and I thought about it for a second and I was like well yeah data modeling is just a bunch of rules and interfacing with the business and then implementing that into translating that to code to schema in databases.
All of this except the meat proxy side. You can automate with AI. You just got to like tell it. Uh at that day I was like this is going to be revolutionary. Like no we're we're not going to uh buy have to buy these things off the shelf and spend years customizing them only for them to be out of date. The other thing I just want to drop here that's super important that Ammer talks about is not only can we create them super fast, efficiently, and durably but then we keep them up tod date at that same speed.
We all know that the business is now changing so rapidly that you know from day to day your data models can change and the Vive data modeling tool helps your data modelers keep these things up to date. And so no longer is it like well our data model has drifted and we no longer use it because it takes five weeks to update it. Those days don't exist anymore. For customers that are adopting this paradigm their data model is keeping up with the speed of their business which pays dividends. >> Yeah.
And I add to that I had the same problem Franco I think it's a problem everywhere building canonical data models or you know industry data models used to call it. We need to understand the reason why people created this industrial model in the first place. The reason was it's a very you know heavy operation to be done. You're looking at modeling thousands of tables, thousands of attributes, hundreds of relationships.
It's a time consuming. If you want to do it to your business, you have to dedicate resources, maybe a team of four or five people that go through every business domain, sit with the business, document all the rules and this. So nobody can afford this, right? So what they do is they created this entities that takes care of the data model. But the problem with this entities is it's not going to create your own model. It's going to create everybody's model, right?
It's the average of the whole industry. So when you take it back to your uh I think this is what Frankini here is you take it back to your organization to customize it, it takes time again. But the time you you take you customize it, you're already turning it into from an industry model to a business model. The already another version came about, right? So you're already stalled even before you go live. So this cycle used to take a lot of time.
What we offering here is we published the industrial model as an example of this Asian can produce this and we believe it or not all the model that you see the 40 industrial models were all generated without any vibes without any prompts. It just giving it the industry and it understand the context and genate all that. What we asking people to do is this industrial data models that we publish, they are the baseline. They are not the final product.
You should take them and use the vibe modeling agent to get version two and version three and version four until at one point in time you have a version that is fully suited to your organization because you have been vibing and adding all the rules and all the you know the business uh decisions into it and the model will react and evolve with time. So this is the promise that we uh we offer you here. >> We do live in a very exciting era.

12:34 The Importance of Documented Data Models in the Era of AI

You did mention vibe coding so many times. How important those are going though to be those data models, documented data models for the AI era. >> This going to be super important. Uh one of my good friends, longtime Bickster at uh at data bricks, wrote a blog uh a while ago. Uh a data model is worth a thousand tokens I believe what it's called. And he researched and found that uh just a regular agent on on a catalog with no data model versus an agent on on a on a good data model uh was able to use less tokens, significantly less tokens, and gave better responses.
That's what everybody is looking for, right? They're looking for cheaper and faster and smarter, right? Better better responses. So I think it's crucial to the era of AI and what we're finding when we're talking to customers is it's essentially the foundation to the ontology when you're thinking about building an ontology. A lot of you know there's many sources for it. There's structured unstructured the structured ontology comes from your data model and a data model that is well doumented. it's got it's relationships that are proper and it's been well thought out and structured.
These agents uh perform significantly better there. So I believe it's the absolute foundation for a proper ontology and then that gives you much better outcomes uh with these with the AI tools. I'll just add a little bit here the it to understand how data model will influence the way you work. Uh data model is the foundation. There's no doubt about that. But for years the way we approach it the data engineering is you got the bronze layer everything coming from the source system.

14:10 A New Mindset: Flipping the Data Engineering Pipeline From Gold to Bronze

You build your server layer on top of your bronze layer which is the cleaned you know augmented and have all these rules and then you build your business semantics and gold layer on top of your server layer. Now with the vibe data modeling this this cycle will actually change. It's not going to be what you call it like move left from or like the layer from bronze to silver, silver to gold. You actually could start from your gold layer, your business data.
You can vibe your entire business data model and get a readym made business data layer, semantic layer. Uh even before you have your bronze layer, you see this is it just it changes your perspective. You start from the customer or the consumer. You lock in your business semantics and now you can get your bronze layer and everything in between just blumbing or by blinds right that connect the whole thing together but you started from a different place from where you would have started if you already using the traditional approach and that's why people looking at vibe data modeling as a tool that is going to change the their perspective about how even they build their pipelines.
You don't you no longer have to start like a top or a waterfall approach. You could start from gold and go backward to bronze which is a different mindset than people use usually do their own data engineering. >> Yeah. And I think uh everything you've mentioned would be very important for geneontology. Uh as you may have mentioned we have business semantics that are carried by UC metrics. uh we have the data model. I will explain how to join the tables together to get the most of them.
Of course, having a good documentation, primary key, foreign key to be able to find this uh this uh entity relationship diagram to understand how things flows. Uh pages where you can add your own documentation and the entire purpose is to make sure that you burn less tokens, your response are faster and more accurate. And regarding uh what you have uh released so far, if I'm not wrong, it's about 40 data models. What if uh these data models are not true uh representation of my business? >> It isn't and we agree with that.

16:28 How to Iterate and Customize From the 40 Industry Base Models

I mean we like we discussed the industry data model is not going to be every uh your own business. This is the average of the whole industry. So yeah, if you go browse like retail model, you'll see stuff that applies to your retail business but stuff that aren't. And we we published the 40 industry data model as uh you know an example that the agent the vibe modeling agent can do this amazing 40 industry models and go to the depth.
So your job is like to look at the model browse see the depth see the domains coverage the attributes the relationships you know the what you see there is the perfect model in a normal scenario right but we know the businesses are not that perfect. They usually have different rules. They have different stuff. They have different you know relationship. In some ways they are weird relationship but it it does represent the business.
We agree with that. So you take this base model and you use the vibe modeling agent and start an iterative to make sure that you get to a a model that fits your business. For every process you or every iteration you give it a vibes. And the vibes is all about my rules. This is my rules. In my business, we use multiple accounts per customer, right? Someone else could say, "No, my business is strictly one account per customer." two different uh you know retailers, one could have multiple, one could have one to one and they are descendant from the same model but the vibe modeling agent allows you to customize iterate and at the final outcome it could be version five version six you will have a model that's very close to your uh organization business data model and that's the aim of the whole thing.
I want to further clarify just one point that gets a lot of confusion when out on social when we talk about this because we use the term vibe. First I want to say if any data modeler or any person who's done this exercise before has ever gotten complete clear and concise requirements on what to build on your data model upfront from the business. You let us know until that day comes. The best you get from the business are I think this, I think that.

18:21 Demystifying the Term "Vibe": Requirements vs. Slop

You're really getting their vibes. Like they don't understand. And you have to understand like from their point of view, they're operating today. Like you're asking them to envision tomorrow. The best they're going to give you is vibes. What we're producing with this is not slop. Like a lot of people they they hear vibe coding, vibe this, vibe that, and they think, "Oh, you're going to produce slop." No. that there this is a a completely well-defined uh many customers have used this and developed their media models quickly and push them to production and are using it actively today.
What we mean by that is that the best you get from the business for the most part are vibes. You get vibes from them. You don't get clear, concise, complete requirements that we would love that actually give give us the requirements upfront. we'll just go build it. We wouldn't have to have a data nominator waste 6 months trying to figure out like how to build these things. So that's what we mean by vibe. Vibe is not not synonymous with slop here.
It's like that's the best we can get from the business and we're actually trying to produce the best data model given all of the rules on what this means to your business. And then the industry models are kind of like a starting point. No one likes starting from scratch. Starting from scratch is scary but if you have a good starting point like the industry models then you go build your business model uh that's where the value is some people actually suggested we should call it aantic data models we could have but I mean the why people are fixated on the name the if you look at even the platform in data bricks things changes all the time data model is the and the changes in the data model is a killer because it's the foundation is the at the bottom and there is features in the platform that compensate for this change like evol schema evolution like you know because it is very crucial someone added a new data model added a column or deleted a column or like you changed an attribute in the source system and your entire blind breaks because someone changed like a like a name or something.
So we don't want to be in that situation. We you wanted to have something that evolves very quickly. Changing the data model is a very heavy operation. You have to synchronize from the source system to the buy lines to the consumer system in the dashboards and god knows what else downstream. You have to synchronize the whole thing. With this you can be more agile. You can do this changes rebuild your pipeline using the vibes which is like think of it as your new requirements that comes along.
So the agent will respond to that. >> And I guess this is my last question but also the most important one. Where can I find those 40 models? >> Yeah they are published in our repo. If you go to Google and search data bricks industry data model, you will see there is a reboot in data bricks industry solutions. And in this rebu you will see the entire 40 data models and also the AI agent into the same rebu. And there is also a nice uh to see like a viewer where you can browse your the entire models.

21:29 Where to Access the 40 Models and How to Get Started With Genie

You pick the models that you want. You can see it on screen. Go through the domains. Go through the entities. Look read the documentation even before you adapt it. You can even download it as an Excel file and share it with your business and they look at it and see what tables and columns and you know entities are represented it and you can collect vibes by sharing this. So they will tell you we looked at the like the customer entity and it's a missing uh I don't know uh customer secondary address.
So customer have two addresses one primary and secondary for example. So you can vibe this and go to the agent say every customer must have two addresses primary and secondary and the agent will rebuild the model and give you a customer entity that have two primary and secondary and so on. You can change domain names, you can add domains, remove domains, you know these things takes months. You need to see it for yourself when you try the Asians and see how quick and agile the Asian respond to the vipes or the requirements.
If you don't know where to start and if you're like, "Oh my gosh, you guys are talking about a lot of stuff like I am. Where's step one?" If anyone follows me on social, you know, I I really don't like skipping steps. Why don't you pull up Genie inside of data bricks and tell it I want to look at the datab bricks industry data models and vi data modeling. Uh give it the link. The internet search is rolling out. By the time you're watching this, you should have web search in Genie.
Uh Genie code definitely has it. Ask it. It has access to your entire environ like I want to be able to implement this this where do I start? And if you don't know where to start, Genie will help you. >> Yeah, I'll give people a teaser. Also, uh we we build the industrial data models, right? Which is kind of like your silver layer. We have your bronze layer, which is the source system. What comes in between, right? We we're gonna publish a blog very soon around how can you make the link.
So you have your entire end to end from bronze, silver and gold all connected. So stay tuned. It's going to be very exciting and I think people will find it and it's the good thing about it is it's going to be published as Genie skills. So your Genie will do the job for you. You just add the industry model, give it the skills that we're also going to publish very soon and Genie will do the whole thing. We tried this with like some customer the feedback was phenomenal.
People absolutely loved it. So we happy to share it but stay tuned. It's coming very soon. >> Thank you so much uh guys. It was a real pleasure to host you and you will find the repo in the description of this video. Thank you so much and see you

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.