Trust You Can Measure: Data Quality Standards in The Lakehouse
Summary
- Data Governance Score: Databricks built a Data Governance Score to systematically measure and surface trust signals across hundreds of thousands of tables on the Databricks Data and AI platform.
- Quality standards at scale: Learn how to leverage Unity Catalog and governed tags to establish data quality benchmarks and drive consistent governance decisions across your entire lakehouse.
- Trust in practice: Discover actionable strategies for data engineers, platform owners and business leaders to raise data quality standards and build confidence in data assets used for critical decisions.
Trust You Can Measure: Data Quality Standards in The Lakehouse

Talk by Amit Pahwa, Staff Software Engineer , Databricks ; Sergiy Kanyshchev, Staff Software Engineer, Databricks
Do you trust your data? If you’ve ever struggled to figure out which datasets are reliable, well-governed, or safe to use, you’re not alone. At Databricks, our own internal lakehouse faced the same challenge—hundreds of thousands of tables, but no easy way to tell which data met quality standards. In this talk, the Databricks Data Platform team shares how we tackled this problem by building the Data Governance Score—a way to systematically measure and surface trust signals across the entire lakehouse. You’ll learn how we leverage Unity Catalog, governed tags, and enforcement to drive better data decisions at scale. Whether you're a data engineer, platform owner, or business leader, you’ll leave with practical ideas on how to raise the bar for data quality and trust in your own data ecosystem.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00The importance of trust3:37Challenges of scale9:20Initial certification efforts13:07Leveraging Unity Catalog17:38Moving beyond certification24:02Establishing quality signals26:05Implementing governance scores33:30Operationalizing the score38:41Future goals and conclusion
FAQs
What is the Data Governance Score?
It's a systematic way to measure trust across your entire lakehouse by combining quality signals—ownership, documentation, freshness, testing—into a score that surfaces reliable data assets.
How does this help data users?
Instead of manually determining which datasets are reliable, users can see governance scores and trust signals at a glance, accelerating their ability to make informed data decisions.
How can organizations define and enforce data quality standards across the lakehouse?
Data quality standards combine schema validation, statistical checks, and domain-specific rules that detect anomalies, inconsistencies, and missing values before data is used for analysis or AI. The Databricks Data and AI platform enables automated quality checks in data pipelines that monitor key metrics, alert when quality degrades, and prevent distribution of bad data. By treating data quality as a measurable property like other system metrics, organizations build confidence that their data foundation supports reliable insights and model predictions.
What data quality dimensions are essential for accurate analytics and AI?
Essential data quality dimensions include completeness to ensure datasets have all required records, accuracy to verify values match their real-world counterparts, consistency to ensure the same entity has the same values across tables, and timeliness to confirm data reflects current state. Different analytical and AI use cases have varying tolerances for imperfection in these dimensions, requiring frameworks that allow teams to define appropriate thresholds. Databricks Delta Live Tables integrates quality monitoring throughout the pipeline, enabling teams to enforce standards appropriate for their specific requirements and use cases.
How does data quality monitoring build organizational trust in analytics and AI?
Comprehensive data quality monitoring and lineage tracking enable organizations to trace insights back to source data and understand how transformation and aggregation affected quality at each stage. When teams can demonstrate that data meets defined quality standards and understand how each value was calculated, stakeholders develop confidence in analytical conclusions and AI predictions. Delta Lake provides this traceability through audit logs, schema versioning, and time-travel capabilities that enable investigation when anomalies or discrepancies arise.
Full transcript
0:00 The importance of trust
Welcome everyone. Going to start with a little bit of legal ease. The legal team thinks it's really really important that you know not to believe anything I say up here. So there's a lot of paragraphs there. Um also a quick reminder, there's surveys that you can complete on the app. We really really appreciate that feedback. So please do complete those surveys and then to get started. Does anyone here have kids? That's more hands than I expected.
That's good. Uh I'm actually also a parent. I have two daughters. Um clicker. This is my daughter. A little giblified. uh she's starting kindergarten this year and as part of that my wife and I went and toured a ton of different kindergartens. Uh we did that because we want to make informed an informed decision. It's she's our oldest. We don't know what we're doing. So we went and gathered data. Uh as part of that interview process, we looked at the schools.
We talked to the teachers. We even went and talked to a bunch of parents of other kids at these schools. Um, none of you work with me except for a couple of you in the back. Um, I'm I'm a little OCD about those things. I I really need information in order to make a choice. Uh, so a lot of this was self-inflicted, but it was a bit overwhelming. Uh, I I really really wanted to be confident in that decision. It took some time, but we did make a choice.
She's scheduled to go to this kindergarten in fall. And once we made that choice, there was still a lot more to decide. Did any of you ride a bus to school? See, I actually never did. Neither did my wife. So, we had no idea if we should let our oldest daughter take a bus to school. Do we trust her? Do we trust the school system? Do we trust the driver? Do we trust that the bus is going to be okay? the other kids on the bus.
We weren't really sure. Uh, and the kindergarten did something really, really great at the open house. They actually let us go on a bus ride, a short little bus ride around the neighborhood. Uh, the driver was really pleasant. Uh, they even gave the kids little erasers shaped like school buses. It was really cute and it did a lot to build trust. At the end of that, I felt very confident. I I felt like I could trust the school.
I could trust the driver. I could trust the school system. It did a lot. But what if this was the bus, right? Like, would you trust that bus? Would you trust that driver? Do you want Pennywise driving your kid to school? That doesn't seem like a good choice, right? and what signal is a school system is the school giving me if that's the bus that takes me on a tour, right? Should I should I trust that school? Should I trust that school system?
Right? There's broader implications there. And that's actually I'm I'm not here to talk to you about school buses. I'm not here to talk to you about uh kindergarten. I'm here to talk to you about trust. Sergey and I are here to talk to you about trust, specifically trust in data because that's what our internal data platform team at data bicks really cares about. We really care that the company is able to trust data and that's because we own our internal data.
3:37 Challenges of scale
We own our internal data bricks workspaces like you. We are customers of data bricks using the same tools that hopefully you know and love, right? And we want the company to trust these workspaces, the data in these workspaces. We want the company to trust us. And why, right? That's actually kind of an obvious thing, right? But these workspaces feed data into every single decision we make at the company. We need everyone to trust that data and have confidence in those decisions.
We need people to trust that data because trust is the difference between like going by facts or going by your gut. It's the difference between completely automating a solution or having to constantly do manual review. It's the difference between having the faith to hand your CEO a dashboard and letting himself serve or having to rely on a team of analysts to pre-generate a bunch of reports that on a weekly basis and email it to them.
Right? That's really slow. That's really painful. It's the difference between success and failure in some cases between IPO and bankruptcy. So this is what we were living with a hundred. Think about how big that number is. 100,000. And I haven't given you the context on it yet, but that's still a big number, right? When I joined data bricks a few years ago, back in 2021, we lived in a pre-Unity catalog world with over 100,000 tables in our primary workspace.
It's a lot. It's not 5,000 or 10,000 and some of you are dealing with. Yeah, exactly right. A hundred,000. It's a lot of tables. We only had a few thousand employees. That's double-digit number of peuh tables per employee. How many revenue tables do you think that is? Right? How do you know which one to use? What criteria do you use to make a decision? Do you go with the first one you see at the top of the list? Do you use the one with the best name?
That sounds fairly reliable, right? Or how about the one that gives you the number you like the most? The biggest, most impressive number. Hey. Right now, what we found is the reality at a lot of companies. You everyone has that trusted person that they go to, right? What what table should I be using? Oh, that one, right? But how trustworthy is that person? How much do they actually know? Do they know all the 100,000 tables?
No. Is it really just a proxy for the ORC chart? Are they just choosing the one that's closest to them? And if you ask them why this table, do they have a good answer? Not Samantha created that table. You can trust her. She's really good. Not, oh, that's the one we use for our dashboard. Sometimes you should just use that one, right? It's can they actually tell you the difference between these tables? Do they have the information to tell you which table to actually use?
I'm going to step back a little bit. So at data bicks, we believe data and AI can and should be infused across every function of your business. And so in order to do that, as I said before, we strive to do that ourselves. To support this, as you can guess, we with 100,000 tables, we support the single largest and most complex data bricks implementation in the world. nearly a hundred workspaces, over 10,000 users spanning all supported clouds and using nearly every single feature that data bricks provides, including some that haven't been released yet that you have no idea exist.
We do this so we can make the most out of the data that we have. So every single decision can be informed, so more actions can be automated so we can continue to make the product better. We want to infuse data into everything that we do. But that can be really difficult. Scaling out use of your data and getting the most out of it can be really, really difficult because in order to scale out use, people have to trust the data.
If they don't trust it, they're not really going to use it. A lot of the times they're just going to go based off gut because again trust is trust in your data is the difference between facts and gut right it's a difference between automation and review is a difference between success and failure IPO bankruptcy it is the role of our internal data team to ensure we have that trust so what's so difficult about trust As I mentioned in 2021 pre-Unity catalog world, we had over 100,000 tables, various quality, no clear ownership, no comments, no documentation.
9:20 Initial certification efforts
All you had was a schema name, table name to go off of. Very difficult to find the data you're looking for. So if you wanted to look up revenue, we had 43 different revenue tables to look at. every or and product team was creating their own. So if you wanted Salesforce data, we actually had six different bronze schemas owned by four different teams. They were duplicating things within teams. So how do you find data you trust?
You ask around and that's actually how certification was born inside the company. So the certification was the idea that you can instead of having five different tables recommended by different people based off some proxy for the org chart, you can get all those knowledgeable people in a room and have them tell you, hey, no, everyone should really just be using this table. We'll create some core set of tables vetted by the experts of unimpeachable quality upholding the highest operational standards in the company.
It would be the basis for all other tables in the company for all other pipelines. It's a very ambitious goal. I don't know how many of you may have started your certification journey already, but it's it's pretty difficult. The first thing we did was we aligned on some standards for certification and it's a short list, but I'll list it there. Naming conventions for tables and comments, modeling standards. We we really like star schema. uh recom uh requirements for table and column comments, SLAs's completeness, change management, incident management.
That's really difficult to standardize across all the different orgs in your company. Almost none of our tables met these requirements. So, we got to work and it turned out to be a really non-trivial amount of work. Upgrading all those pipelines is really difficult and changing table names and column names, that's a breaking change. So, how do you get consumption to move? New use cases, sure, but but existing dashboards.
How about that dashboard the intern built last summer? It's used by most of the sales team. Is anyone maintaining that dashboard? Is anyone going to sign up to upgrade and migrate that dashboard? No, they're not. Spoiler. And on top of that, how do you make certification work? Right? How do you specify if a table's certified? If Alice just joined the company, how does she know to even look for certified tables? How do you get this these certified tables in front of everyone when they're searching for data?
With 100,000 other tables, it's still going to be a needle in a haststack. And what about dashboards? Most consumption isn't even aware of the tables. If you are consuming some data as a leader, you're looking at dashboards, not tables. You can't expect Mr. Stark to go double check the source tables whenever he's looking at a chart, right? So, how do you know if a dashboard certified? We didn't have good answers to these questions.
We didn't have the tools. If you can't tell, this is one of my favorite movies, Iron Man. Um, if if I knew how to play and make the deck play hype music, this is the one I would probably do it for. I'm not trying to sell you Unity Catalog, but I really really appreciate what Unity Catalog has done for us internally. For those of you who read the data bricks tech blog, you know that we performed an enormous UC migration.
13:07 Leveraging Unity Catalog
We did it very, very early on. It was pretty painful at the time. And we learned a lot from that to make it a lot easier to migrate to UC. And with Unity catalog in the picture, we now had access to a bunch of tools to make this all easier. Table tags meant we no longer had to rely on a schema where we would certify all the tables. We could now certify tables in place. There would just be a tag associated with a table. you see lineage and metadata provided a ton of extra signals for users and we even had the dedicated certified tag if you've heard of the certified tag in UC which influenced search ranking so if you were searching for revenue now that appears at the top of search results most importantly it gave us controls over visibility access controls a lot of those 100,000 tables were not very good.
We're were never meant to be consumed by people. So, it gave us the ability to lock those down, to hide them. Now, we quickly made use of these features and the data practitioner experience did change. It was much better. As a new employee searching for revenue, you were no longer ambushed by 43 tables. There was instead one table clearly marked as certified with documentation directly in UC within the catalog. But we still had an issue.
Certifying data was really difficult and timeconuming. Only the most critical tables had the time and resources to pass certification. And that bandwidth comes and goes. It's not permanent. you have time this quarter, maybe not next, right? So as as the product and pipelines evolve, sometimes they drift out of compliance, right? To make matters worse, many making changes to certified tables was so high friction that requests from teams started to pile up.
So they began to unblock themselves. Yeah, more tables. More and more tables. You get a table. You get a table. by the way. We fixed an issue over here, but we created another one over here. Or so we thought. But when we actually looked beneath the surface, more tables were not actually the problem. Tables are a natural result of leveraging your data. It's fine as long as they're high quality, as long as you know where to go.
We had been so focused on that tip of the iceberg, we actually weren't measuring any anything underneath. We knew that 1% of tables was good, but we weren't actually really looking at the rest. We were just saying don't use them, right? It turns out certification was having a ripple effect. By creating those core certified tables, we created a foundation for all the other teams to easily build on. The reliability and thorough documentation car carried through to many of the other pipelines.
And most importantly, by simply clarifying the expectations through certification, producers knew what was expected of them and consumers knew what signals to look for. the signals that indicated care and quality, the signals for trust inside the company. So while these tables weren't certified, they didn't satisfy all the criteria. They satisfied some and that meant that things were better. We just didn't have a way of measuring it.
We had been treating it like a binary issue, but in reality there's a gradient of quality. and how do you measure that? So, I' I'd like to hand off to Sergey now who's going to tell you a bit more about how we evolved from binary certification to comprehensive measurement of our lighthouse. Thanks and hi and welcome to data and AI summit. Again I work with Amiton data platform team which means that most of the time we don't develop features for datab bricks but we use datab bricks same way as many of you do datab bricks at datab bricks.
17:38 Moving beyond certification
So let's recap where we were with certification. Our data practitioners had high trust in a handful of high quality certified data sets. The tip of the iceberg but it did not scale. Our lakehouse had 100,000 tables and the company keeps growing. Certifying each one of them isn't just unrealistic, it's unsustainable. Why? First is unsustainable both for producers and the governing body of subject matter experts, most of whom are high up in company ranks.
Second, what happens if a data set degrades? There's no paved path to uncertify or certify data set. And third, our original process heavily relied on naming conventions for tables and columns. That put even more load on producers since they would have to migrate all downstream usage for what? Just to earn a certification page. And despite all of that, pre-production tables would still get used. Don't get me wrong, certification is a great outcome for the top 1%.
But what about the rest 99? Most of my talk will be about this bottom bottom of the iceberg. So we needed something broader than certification, a way to bring visibility and accountability to the entire lakehouse. And what behavior do we want from people? We want data producers to follow best practices especially for anything that meant to be shared widely like gold data sets. Ideally, such behavior gets rewarded. Somehow those ready for production tables get promoted.
We want consumers to be picky about the data they use. We want them not just to grab what's available, but to make an informed choice. We want consumers to vote with their trust and create irresistible market for producers. Eventually, consumers get rewarded with clear contracts. Another behavior we want to nourish is thinking about how data gets shared and exposed. Amit likes school buses. I like music. So, I came up with this analogy.
Please bear with me. This is about exposure and visibility versus quality. So, what's an example of high quality, high exposure, performing at Grammys? That's the picture of my daughter, by the way. She d she made it. High quality, low exposure. performing at her house with her dog and her brother. Low quality, high exposure. It's me covering and sharing it publicly on YouTube. Now, back to data sets. Thinking about my data sets, how how would I put them on this chart.
On the left, I have all my locked down data sets, both high and low quality. I did good job and shared them right with the with the right people. I followed least privilege principle. On the top, I had my best data sets. On the bottom, I have my toy development testing data sets. The one I did not ever mean to share publicly. So, I may not care much about quality of those. Okay, follow me. Top right is widely shared highquality data sets.
My golden data. It's properly cleaned, processed, and anonymized and gets shared with large number of people and maybe with a whole company. This is the best of the best. This may even get me promoted. Top right is where you want to be. Your data is clean. Your pipelines sparkle. You may even get a monument in the middle of your company's headquarters. And finally, bottom right, low quality, high exposure. This is the danger zone.
This is how legends are born and lawsuits. My broken join made it to CEO's board deck and now we think there are three million customers in Antarctica. Nicely done. I'm famous now. Probably fired. The behavior we want to incentivize here is to not have any data sets in this bottom right quadrant. And we think there are two paths forward. If the data should not have been widely shared in the first place, we propose data producers to lock it down and move it to the left.
If the data was meant to be shared, then the owners need to improve quality and move it to the top. Which path would producers choose? It depends. Sometimes they prefer to lock it down. Sometimes they prefer to improve quality. But often they prefer to do nothing. In which case we as administrators or data platform team can take an action on their behalf and force a lockdown. Notifications, not deleting the data. We don't have enough evidence it's necessary but locking the data down.
And what makes such enforcement easy? That's right. Unity catalog and all the governance features specifically attribute-based access control. Now after this detour into exposure and access, let's get back to expanding certification to something broader. We need a system that brings accountability and trust to the entire lakehouse to all data sets not just top 1%. We needed to set the right incentives. As a data platform team, we want to make right things easy and wrong things hard.
24:02 Establishing quality signals
It has to be fully automated and cover broad spectrum of data quality, reliability, documentation, governance without adding manual work for producers or committees. It can be a tiering system, think bronze, silver, gold or a continuous score as long as it can scale and evolve. Let me pause on evolve here. Business changes. The quality of our lakehouse keeps improving. The issues we focused on yesterday may already be fixed.
For example, if we had a huge campaign and improved data freshness across the board, we may not need to prioritize freshness anymore. When measuring quality, we can focus on the next big thing. It's crucial for our measure of quality to evolve. Let me use another analogy. Last one for the day I promise. It's about grocery store. So certification is equivalent to certified organic labels that you see on a fraction of products but small fraction.
What we want is something like nutritional labels and we want them to be attached to every single product or data set. In our case, we want to democratize quality signals. And what can help with that? Yes, I thought you learned by now. Unity catalog. When you go and establish trust in your lakehouse, you will have a set of criteria that matters to you. We are sharing what matters to ours, what we try to use when we try to build the best data platform.
And now we'll show you how we used catalog's feature to make all of it work. Introducing data governance score. It's a number between zero and 100 that measures how production ready and trustworthy a data set is. It measures how well data producers follow key rules and best practices. It's like a credit score for your data. Since it's a composite score, we preserve individual subcores for each subcomponent. We develop this score inhouse within our data platform.
26:05 Implementing governance scores
It is not part of datab bricks product but it showcases how easy it is to elevate data quality when you're within datab bricks ecosystem. Let's dive into major score components. Again these are our opinionated standards. It's the value proposition of a data platform team to have those opinions. If your company has a data platform team or is building one, you will have your own set of opinions. So we are sharing ours.
No surprises. Three pillars there. Documentation. Is a data set described and well annotated? Reliability. Is it fresh, stable, and properly owned? Governance. Does it meet access, privacy, and handling standards? Let's zoom into each of those pillars and let's start with documentation and see how to score high at that dimension. Here is an example of a table overview in Unity catalog. What does it tell me? It tells me nothing.
There is no documentation, no comments. The owner is service principle, which is great. But do I ask service principle a question? Who do I go to? The first piece of data governance score is documentation. We have specific things that we want to be annotated annotated on tables owner, Slack and email. That way consumer can go directly to the person who understands the semantics of the data set, not the person who is responsible for operating table.
Ownership built trust. You can't trust what no one owns. And to make these data sets fully self-sufficient, we require descriptions and column comments. We score them. Having column comments is similar to well doumented code. No need for satellite docs explaining the data. Everything you need is right there with your data in Unity catalog. People can use it. People can find it. People. Speaking of people, you know who else can benefit from good documentation? if you have standards around documentation, it's not just humans who utilize this.
In the new world, um, LLMs need as much context as possible. So even if your today's workflows don't use table metadata, they likely will tomorrow. And by the way in datab bricks genius spaces already use stable metadata. So if you have good documentation you can get more out of datab bricks even today. Let's move on. Let's say as a consumer I picked a table. It's the same table from our previous example and let's say I built a dashboard on top of it.
Is that dashboard something I can share broadly? Is it going to be reliable? What kind of contract do I provide my consumers with? Does the data land on time? Does it have data checks? Will it be around tomorrow? I don't know. I have no context on that. So, that's the second piece of data governance score, reliability, no personal accounts. How do I get proper support if the table is owned by from second floor? What do I do with is on vacation?
All tables meant for business needs to be owned by a service principle or service account annotated and documented by a human but owned by service principle. Age did this table pop up yesterday. Will it be around next week? I don't know. So we track the age of the table. I want to be a table around for at least few weeks before I use it for my production dashboard and send it to some exec with my name on it. All tables with SLA must define those SLAs's.
Then we can track regularly if they meet them or not. If there are data quality checks, producers should define those, declare those. They should be visible to consumers. Consumers should know what things are checked on and what are not. That builds some confidence in the data and that starts to build contracts between producers and consumers. Big one governance. It's in the name data governance score. It's so important we even have a special waiting scheme for it in the score.
Does a table meet our company data handling standards? Does it have PII tags? Does it have retention? Does it handle sensitive data properly? Is it tied to our GDPR tooling? Are we following principle of list privilege? It is about measuring it and scoring it. We look at access patterns. Is it overshared? Who's using it? Is it relied upon? If no one's using it, am I missing something? Cost. As a data platform team, we want to know, do those pipelines producing all this beautiful data cost us millions?
ETL jobs should be tacked with budget and it also gets scored. We covered three pillars. documentation, reliability, governance. Let's talk about the weights. Again, the score and weights keep changing as business evolves, but those are the weights we use today internally. If there was one slide to take a picture of, that probably is the slide. It's a glimpse into our secret sauce. And these are the weights we use today for our score for each score component.
Your mileage may vary. To start off, we care deeply about documentation. It was incredibly important, the lowest bar you can set. So, we score documentation heavily. As we move on and good documentation becomes more of a standard practice, we don't need to notch people as much as before remind them. So, we can start weighing it less and shift our focus to the next piece of the score. You may have noticed a crown next to uh data handling standards.
It's because headlining standards are so important that failing those checks brings your score down to zero immediately. You cannot get high score and poor data handling. Now when you know what data governance score is about, let me show you how we implemented it fully within datab bricks. It all starts with a snapshot of your current world. There is a lakeflow job that collects the metadata about state of the world from Unity catalog.
It uses SDK and system tables to augment this data with access patterns. All of it gets used for score computation and the scores get stored into our persistent storage. We use delta table. What do we do with the data? Well, two things. We provide visibility and make it actionable. How do we provide visibility? First, we have a mission control dashboard where you can explore scores for each individual tables and each component.
33:30 Operationalizing the score
Second, we tag each table with its score. Each table in Unity catalog gets tagged. Now, wouldn't it be silly if an owner of the table could just go and change the score to 100 100? It would. So luckily we have a new feature of Unity catalog tag policies which makes tags governed. So it's only service principles appointed by administrators or our team data platform can set those tags. And how do we make it actionable?
By enforcing the exposure. That's the job that does the access lockdown that I talked about earlier. And now now let me quickly go through through some examples. This is a signature of a function we provide to our pi spark data engineers to create documentation and declare ownership est and so on. This is literally a snippet from our codebase. When they use this function table comments in unity catalog look like this.
The comments are structured. Its markdown is supported and we can parse those comments back in a structured way. This is same table as I showed before but with documentation it looks a bit nicer. It has owner slack columns documented score computation is contained within a stateless function executed by pi spark job. Results of scoring are stored in a source of truth table and from there two things happen. First, as I said before, each table in UC gets annotated with a score and through the tech policies.
Only service principles that have privilege to add this score can do it. Second thing that happens is the data gets displayed in our mission control dashboard. This is a Unity catalog regular AIBI dashboard. We didn't have to build any UI do any front-end work. It just came handy out of databicks package. On the left here you see a high quality table. All the boxes are checked and green. The score is high. And in the bottom you can see an access pattern which is more than 6,000 people have access to the table and it's used by more than 700.
That's a good usage. That's fine. And on the right you can see a table with a lower score. It's missing comment. DQ checks SLA. It's been around only for one week. And although it's also open to 6,000 people, it's accessed by two. So the way with the way we run campaign, our producers of this right table would be faced with a choice either to lock it down or improve quality. Speaking of campaigns, sometimes you need to know the bird eye view of the quality of your lakehouse.
Let's say I got asked to improve quality of the my whole org by 25%. Where do I even start? So this is an example of how we aggregate scores per organization. And again the tables in the red which is top colors don't meet our quality standard. And the producers would be faced with a choice either to improve quality and move them to the green or lock down and move them to the gray. It is all about incentives. One incentivizing mechanism that we can employ is to use Unity catalog cleanage to propagate the score as a tech downstream to tables and dashboards.
Imagine your gold table is downstream of a low quality one. Can we trust it? Garbage in, garbage out. Specifically for AI use cases. And furthermore, someone built a userfacing dashboard on top of this data. It makes sense to propagate quality signals all the way to the consumer. Imagine looking at dashboard and seeing large tags saying warning this dashboard was built on the low score data. It's nice. Get some signal here.
Lineage and system tables and tech policies from Unity catalog come very handy. It is all about incentives. The idea is to create a strong market where consumers are incentivized to use highest quality data and raise the bar for the producers who in turn are incentivized to continuously improve data quality. And that's how we scale data platform and unlock trust with Unity catalog. And now let me hand it back to wrap things up.
38:41 Future goals and conclusion
Thanks. We're back to my favorite slide. Um, we've we've talked a bit about our journey, where we were. We had a mess of a 100,000 tables, no signals for quality, no standards, and where we are now, right? We enabled Unity catalog and got access to all these tools and now we have certified data sets providing core centralized highquality data and we have data governance score providing observability and standards for everything else.
Going back to the bus, the company's on the bus. We're on the bus at least, right? We use data to inform our decisions, not our gut. We're using data to automate solutions without constant review. We're handing our CEO dashboards and not cringing every time we do it. We're seeing success, not failure. We foreshadow a little bit about, well, where do we go from here? I'm just kidding. That's uh we raise the bar. Sergey was talking about we need to evolve the score.
So, we're going to do that. We're going to add new criteria. We're going to tweak the weights, set incentives. We're going to drive improvements for quality, reliability, and health in our lakehouse. And we're going to continue to raise the bar, continue to unlock more value. We want we also want to extend this score to cover more and more assets. Right now we're covering tables and dashboards, but what about features?
What about metrics? What about genie spaces or apps? We now have a blueprint for how to measure and drive that incremental growth. And now so do you. So if you enjoyed this talk and would like to learn more about the internal data platform team here at data bicks, we have a bunch of talks scheduled. Uh you can see the remaining ones here on the slide. Uh and if you have any questions or would like to chat, we'll be sticking around for a few minutes.
You can either find us here or outside. Uh thank you everyone.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.