Unlocking Data Intelligence: A Beginner’s Guide to Unity Catalog

Summary

• Unified Governance Foundation: Unity Catalog provides a single place for access control, lineage, discovery, auditing, and business semantics across all data and AI assets.
• Data Intelligence Through Governance: Combining data and AI governance enables deep data understanding and delivers contextual, domain-specific insights that boost productivity.
• Open Interoperability Standards: Unity Catalog enables open collaboration and avoids vendor lock-in by supporting multiple formats, tools, and platforms.

Unlocking Data Intelligence: A Beginner’s Guide to Unity Catalog

Talk by Chris Grabiel, Lead Product Architect, Databricks ; Sachin Thakur, Principal Product Marketing Manager, Databricks
Getting started with data and AI governance in the modern data stack? Unity Catalog is your gateway to secure, discoverable and well-governed data and AI assets. In this video, we’ll break down what Unity Catalog is, why it matters and how it simplifies access control, lineage, discovery, auditing, business semantics and secure, open collaboration — all from a single place. We’ll explore how it enables open interoperability across formats, tools and platforms, helping you avoid lock-in and build on open standards. Most importantly, you’ll learn how Unity Catalog lays the foundation for data intelligence — by unifying governance across data and AI, enabling AI tuned to your business. It helps build a deep understanding of your data and delivers contextual, domain-specific insights that boost productivity for both technical and business users across any workload.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and welcome
• 1:07 Data governance challenges
• 4:02 The fragmented landscape
• 6:19 Unity Catalog overview
• 8:27 Governance capabilities
• 12:13 Interoperability and sharing
• 14:47 AI-powered intelligence
• 20:01 Live product demonstration
• 32:06 Conclusion and Q&A

FAQs

What is Unity Catalog and why should beginners care about data governance?

Unity Catalog is a unified governance solution that simplifies data and AI governance in a modern data stack. For beginners, it means having a single place to manage who accesses data, understand data lineage, discover trusted datasets, and ensure compliance—removing complexity from data governance.

How does Unity Catalog enable data intelligence?

Unity Catalog unifies governance across data and AI workloads, creating a deep understanding of your data assets. This foundation enables domain-specific insights and contextual intelligence, allowing both technical and business users to make better decisions faster.

Can Unity Catalog work with tools and formats outside Databricks?

Yes, Unity Catalog supports open interoperability across different formats and tools, enabling secure collaboration without lock-in. This openness means organizations can build on open standards while maintaining robust governance and control.

What governance capabilities does Unity Catalog provide?

Unity Catalog offers access control, data lineage tracking, discoverability, comprehensive auditing, and business semantics management. Together, these capabilities ensure data is secure, traceable, well-documented, and compliant with organizational policies.

Full transcript

0:00 Introduction and welcome

Hey everyone, welcome to today's session on beginner's guide to unity catalog. I'm a principal product marketing manager here at Databix. I'm also today joined by who's a lead product architect here at Databix. And Chris has helped hundreds if not thousands of customers get value unity catalog. So we're excited to have you guys here. uh but I must say this is one of the most important sessions at summit because everything that we're building at databics uh intelligence platform is built on top of unity catalog.
So from tomorrow onwards every keynote session every breakout session that you'll go to you'll hear one way or another in the importance of unity catalog and what value it brings to the table. So first of all round of applause to everyone who registered for the session and super excited to you know answer any question you guys have. But before we dig in, just want to uh put the disclaimer up front. U we'll be talking about some of the forward-looking statements around product road map, some of the stuff.
So please bear in mind those timelines might change. So just an FYI. So let's get started. So I know all of you guys have questions. So we thought like you know we also have some questions. So when and I were coming uh to deliver this presentation, we thought hey you know we'll ask this question to the audience. Why did you decide to attend this session? What does governance mean to you in your organization? We thought instead of directly answer asking this question, we'll put some scenarios and see like you know whether these scenarios relate to you guys or not.

1:07 Data governance challenges

So let's start with the first one. How please raise your hands if you think your catalog strategy looks like you are fixing IKEA from boxes from different parts glued together multiple architecture and please be candid about like you know your your catalog strategy. Raise your hands. It's after lunch. Everybody's got to wake up. You're close to beer. You're close to dinner. You're close to everything. So, just stay with us for another hour or so.
All right. All right. The next one. You have built an awesome beautiful data wiki to explain what this table is all about. And someone and you get a ping on a slack or teams. Hey, can you explain what this column is all about? What this table is all about from like any business analyst or a engineer and you were like, hey, why don't you go to section this or this of the doc? I've explained in detail. Please raise your hands if you come across this scenario in your team.
There we go. Awesome. There you go. How about this one that you have trained a model and you're ready to deploy it into production and the first thing comes to your mind hey like you know on which tier are this model trained on you asked this question and no one has answered not 100% sure whether they you are using sort of proprietary information or not. Have you come across this scenario? Please raise your hand. Feel free to be as candid as possible.
Awesome. So how about this one? You are like data steward. You have set up policies across teams, across different projects, across regions. Now your catalog looks like a graveyard of policies and you wonder, hey man, what did I do? Please raise your hands. Awesome. And this one is my favorite. Um, you get a question from a exec what's the revenue per quarter? Um, what's the generation looks like? Uh, what's the sales per store?
And you get this dashboard as a source of truth. Someone sends spreadsheet as a source of truth and someone has this chatbot created as a source of truth but none of them has accurate answers. Raise your hand if you come across this issue where business caps don't align among teams. Awesome. So this is the world of governance we're talking about here guys. It hits everyone in an organization from a data architect who's figuring out the first problem.
Hey, how to glue different data sources into one unified view or somebody who's an AI engineer figuring out hey should I deploy this AI model in the production or not because I don't know the lineage of this AI model or somebody who's figuring out the business KPIs answering to a business executive and figuring out hey like what's the source of truth of business KPIs as you can see the governance hits every everyone in organization across different personas.
But the next question would be how did we come up to this stage like what are the biggest issues that we that are reasoned behind these challenges. So fundamental issue is today data escape is fragmented fragmented across different asset types. You can have files tables models you can have data spread across different sources data leaks data warehouses catalogs you might have data stored in different formats. Now you have different personas like data engineers data scientists and of course AI agent is also in persona now in in today's world.

4:02 The fragmented landscape

So data landscape today is super complex and it's very difficult to define governance such a complex landscape and what are the challenges what are the uh negative business outcome you get out of these challenges of course like the increased compliance and security risk vendor lock in especially if you're dealing with a platform that only works with a specific format or specific asset type and you cannot port your data outside that platform that's also a challenge and of course operation efficiencies like one of the example that I shared was like hey you will get Slack and team message all the time can you give me access to data and you'll say I have given it let me know it works or not right and then someone again pings you back hey I still don't have only have read access I write access as well right so so there's a lot of operation efficiencies that happen if you don't have unified governance in your in your platform so our point of view is you need an open and unified solution but the reality is there aren't many in the market if you look at the vendor landscape uh at least the top ones you get a very fragmented sort governance architecture across different uh platforms.
Let's start with snowflake. You might have seen Horizon Polaris two different cataloges working together. One works on iceberg, one works on the proprietary format. Try sort of syncing the policies across these cataloges is a is a challenge or the likes of uh AWS you have glue, you have S3 tables, you have lake formation. So you literally have to stitch together your sort of governance framework across these multiple services.
Uh or likes of fabric um I mean they don't have operation catalog yet. So you have perview governance acts as a engine level per engine level. So you define policy per engine then if you switch engine your policies won't not work you have to define it again governance is at the file level and for GCP the same scenario you have data plex you have other data stores in there you have to plug them together to create this view so we have a different approach um four years ago Mati Zaharia our CTO and co-founder decided hey should we build on some some existing architecture like hive which has been for 17 years or should we like redo something from ground up and that's what we did uh with unity catalog So what exactly is Unity catalog?

6:19 Unity Catalog overview

It's a unified governance solution for all assets. Doesn't matter whether structured data, unstructured data, files, tables, AI models, dashboard, you get one governance model. But is not just a simple catalog. We also build capabilities on top of that around like uh access controls, lineage, monitoring, um data sharing, all those capabilities bundled together under this one sort of solution which we call it as a heart of like data intelligence platform.
This is what powers data intelligence in your organization and if you have seen traditional catalog only deal primarily like on access control to some extent on discovery side but only on like table format as well. So and as I mentioned like it is the core of data intelligence platform and tomorrow on I said like in keynotes and session you'll hear it again and again how unity catalog is powering all the workloads on top of the market that you've seen.
So what are the business value that you get out of having a unified governance framework one you have enhanced security and compliance across your data and AI because you have your own unified solution you break your lock ecosystem so it doesn't matter which format you're using uh which tool you're using as long uh as long as interoperable you get this freedom to choose the best tools for your team and you gain operation efficiencies once you have this built-in intelligence around discovery accessibility so that any user on the platform can access data uh irrespective of your technical proficiency and we have seen sort of like benefits across um different organizations smaller larger Pepsico hit like 30% sort of reduce onboarding time because of the unified access uh policies DJ did 50% uh jump sort of in the self-discovery sort of management British petroleum dropped like uh reduce cost by $1 million over an year so there are a lot of benefits when you think about unified governance is not just I said it's not just about access controls and discovery there is a a lot of things tied to governance.
So let's look at some of these capabilities one by one and how unity catalog helps you deliver around those values. The first of all unity catalog brings all the data that you have across fragmented ecosystem under one secure view. So and it doesn't matter whether the data is managed by database or it or if it lives outside database we have technology called lakeos federation which makes it easy for you to register your data outside database into unity catalog. you get a singular view.

8:27 Governance capabilities

So which means um you can define access policies in a singular view. You can get lineage monitoring everything in one place. And the best part is you don't have to load your data. No ETL required. just you have to register give a path of the data source be it like you know SQL server snowflake whatever you'd get those tables registered in uni catalog and it works across all formats iceberg delta park whatever you pick and as I mentioned earlier is not just on for table data it works for files tables ML models as well so you get like one sort of singular view of all your data and a assets once you have all your data in the second part of the governance strategy essentially is to build access policies so you with unity catalog you get one interface very famili people people familiar a lot of people are familiar with that interface SQL based simple like grant select grant modify statements you can apply it for ML models you can grant for tables you can get for files any structure unstructured data you get one sort of interface to define policies um you get fine grain access controls with aback uh with the row and column level filtering so you can get sort of detailed fine grain access controls and we also build this automation that helps users is to request sort of access to a particular asset right from the catalog.
So you know it becomes super easy instead of like pinging somebody on Slack hey can you give me access and someone has to go look up hey where which group does this user belong to should I give it access or not you can access request for access right from the catalog so a lot of awesome capabilities on self-service access workflows um unicata also provides a lot of benefits around on observability side of things um compliance security cost budgeting so first of all you can autocl classify all your sensitive data um this capability we built in unity catalog right now in in beer.
Um you can also sort of like uh uh do automated quality monitoring. So all the assets in unity catalog uh in an overview page you'll see these things flashing now which is like quality which gives you an indication about correctness uh the completeness of the data freshness of the data. So any user who goes into UC is not just looking at the table but gets some additional context. Hey is this data even stale? Should I use it or not?
So all this intelligence capabilities we're building right in the uh explorer and also unity catalog comes with lot of rich system tables from the cost from lineage to billing everything is built in so that you can build rich sort of dashboard governant dashboards on the observability side of things and this one is my favorite like with Unity catalog also provides automated lineage so it helps you understand how's data flowing across your organization uh right from a source which could be the Kafka stream for example to a table how it's transformed how it's consumed downstream to a dashboard or ML model so you get end toend lineage and it's real-time lineage the moment you write a query it gets logged that's the benefit of having a sort of builtin governance solution on a platform um so it helps when a lot of use cases uh like impact assessment if table if I drop this table who's going to get impacted who's using it or root cause analysis uh you know there is error in this table let me go back and check where this table is coming from what's a you know what's a source and so like things like that so it helps you do the root cause analysis as well and then on the second pillar which is more interoperability which is I believe is one of the key concerns a lot of folks have whenever they deal with catalogs hey is it a lock in or not like you know kind of story so unity catalog took a very pioneering approach last year when we open source unity catalog but not only open source catalog we also open source unity cat APIs so what does it mean it means any engine can implement those APIs and You can read and write to unity catalog from any be it iceberg client or be is like a delta client any so there are a lot of integrations that happened over the last one year you can see confluent uh fabric star rocks bunch leino flink bunch of other things um platforms have integrated uh and if there is an engine or uh if you're using any platform that has not done that integration it's super easy basically just tell the partner basically to implement those open APIs so it helps you build a data architecture based on open foundations.

12:13 Interoperability and sharing

So that's one of the core pillars of Unity catalog making sure you use the tools that you are that you love without getting logged in. Um Unity catalog also power open uh open delta sharing protocol. This is the foundational protocol we open source four years ago. Essentially what it enables is you can share data and AI sets across platforms without need to duplicate them. Um and not only the data or any AI asset that's managed by data bricks you can federate it to unity catalog as I mentioned earlier with lakehouse federation and share that federated assets outside as well and the best part is you're not copying data it stays where it is be my SQL be postgress you just point it to UC and share it outside so you might have seen some other sharing solution where you actually have to load the data in many a times then you can do some cross region crosscloud sharing but thing is you have to load the data into the platform and also data u is the foundational technology that's powering clean rooms UC plus delta sharing so uh clean room solution database clean room solution is a multi-party collaboration solution so you can bring your private data in and both the parties or multi-arties can sort of collaborate on the data run some mutually approved jobs without having access to the raw data essentially so you get a secure environment to do collaboration across crowds across regions And finally uh from marketplace perspective this is also you see powering the database marketplace you have like tons of resources tons of um we have around 1200 listings from 250 providers it's super easy as a data consumer you go to marketplace requested data it shows them in UC as it is as it as if it is your native asset um and then for the next section I'll invite talk about some of the intelligence capabilities we're bringing in and you left me 25 minutes so if I run out of content I'll have to break into song and dance or something else entertaining here.

14:47 AI-powered intelligence

So, um, right. So, uh, late in the day, I'm going to try to close this out strong here, talk about some of the built-in intelligence that we're packing into UC, and then I think mostly because I must just hate myself, we will do a live demo in an unscripted environment. Uh, uh, it's going to go fine is what I've been telling myself. Uh, and so that'll be sort of bookended and we'll take a break from the slides. I'm sure you guys have only seen a couple thousand slides at this point.
So, um, we'll do that. So, um, why does all this matter? Uh, when I talk to a lot of customers, especially those who've been with us for a long time and they're still on, uh, Hive, there's they've got lots of local HMS. They've got lots of workspaces all over the place. You see, and sort of the migration, a lot of that stuff. We get a lot of questions about why why do I care? My jobs are working. My my reports are working.
Like, we have a business catalog and people pretend to use it. Like, everything's fine. Um, well, the reason that this matters is we built UC from the ground up because this context matters to AI. And even before we all rode this roller coaster hype cycle to chat bots and everything else last year, we sort of the writing was on the wall for this, right? Good AI is only smart about your business because it has context about your business.
And so, uh, if you'll kind of look the eagle item amongst you here in this top right, this is something that we're actually making more and more transparent in the platform. So when you interact with Genie, your assistant, you get a right or wrong answer. You give it an up vote, you give it a down vote, and now we have the ability to say you weren't just wrong, but why were you wrong? And there's even the ability to fix it yourself if you happen to know what code is correct for that SQL.
Um, and then you can see here on the very bottom, and this actually pops up, uh, or will start in many of your environments popping up in a much more obvious way. Genie's like, cool, I learned from that. I'll I'll consider that next time. A lot of that was happening already on the back end, but it wasn't very transparent to our users. And so we talk about it a lot and there's an assumption that it's happening, but this is actually showing what's happening behind the scenes.
Um, and so all that context matters. Uh, anytime we talk about metadata and description on columns and all this sort of stuff, that's really why, uh, it helps with search. Yes, your users can, you know, look up acronyms and whatever else. That's all fine. Uh, but it's really to power all of your AI services. Uh and then um all that metadata gives you a way to curate and organize all of your data. And when we think about curation, um this is something that we will see or should see or might see in a few minutes when I click into the environment.
Um I'm not going to spoil uh the Kool-Aid too much. There's some sessions later in the week that talk about discover. Um but we thought it was high time to give the catalog kind of a fresh coat of paint for our business users, for our engineers. And so we've been working really hard to build this kind of best-in-class kind of familiar experience. Um, search is great. Making global search better is always important. Um, but actually giving uh users and data stewards in the catalog the ability to certify different reports and pieces of data and the ability to rise these things to the top and actually group them into domains and subdomains and other concepts for your business.
And so this is something um, if you catch talk uh, I think it's either Wednesday or Thursday, I'm not sure. it's hard to keep track. Uh, but there's a whole session that's specific to just this that I really encourage you to catch. Um, so the other thing that you've probably heard a little bit about and we'll definitely hear more about this week in the keynote and everything else is u metrics or metric views or UC metrics or there's lots of names.
Naming things is hard. Uh, I think I've heard it four times today. In other settings, that's the hardest part of computer science is just naming things. Uh second only to renaming things once they've been named. Um and so a lot of what we've done with metrics is what we know and what a lot of us that have been in this business for a long time in different roles, different companies, so much of this logic is pushed into your BI layer.
You have custom SQL and Tableau. You have PowerBI data sets that have metric definitions and acronyms defined and all this stuff. And that's great if you only have one viz tool. if you're only ever going to have one viz tool and if you only use your data for BI and reporting it's less great if you want to use it for AI and you want to use it for a bunch of different tools that aren't in this single surface area. So metric views is kind of us tiptoeing into the space great way to mark up your data and write a whole lot more uh rich uh information about it supports full markdown and lots of other things and again this will be this will get a whole session dedicated to it as well.
So I encourage you to check that out. Um, and then the other thing, and I think we've touched on this before, but system tables, you get observability across all of those rich interoperable systems that was talking about. The other thing that's great, and we do this at data bricks quite a bit, but use data bricks on data bricks. So, when you start thinking about using Genie on system tables or you start thinking about building agents and apps and things on data that are already in the platform, that's when you're really kind of starting to unlock the power of what's in there.
And so, um, without further ado, uh, I think we're going to do this. I think that's enough disclaimer. We'll see. For those of you that have been here before, you've seen the cookie demo and lots of other things. So, things can go wrong. It does happen. And, of course, right away, I got to log back in. So, that's always the best part of the demo. And I have no internet. So, we're we're off to a great start here. There we go.

20:01 Live product demonstration

Perfect. Minor knit. We'll survive that one. That was just a flesh wound. All right. So, um, couple things. And actually, can I swap back? I realized I had one more. It's very important to the storytelling. It's very important because I got to wake everybody back up. Hold that thought. All right. How many of you are in an HOA or have to deal with someone who manages your building or somebody? And then I'd like to put I'd like to see a show of hands uh for how many of you just love interacting with those with those bodies.
I expect to see lots of hands in here. Really? Nobody. Building permits like throwing a shut up. Nothing. You don't love that. Okay. So, um what I thought we'd do for the demo is we're going to do a little role play. So, I am a power- hungry, newly knited admin for my HOA, and I have been dispatched to dig into the production data that comes off of our system, this lawn and order system. It's a pun in case you didn't catch that.
I'll point it out for you. Um, this system is going to give all sorts of observations, let's call them, from one neighbor to another about the quality of their lawn and the val, you know, the the cleanliness of their pool and what their pets are up to. Really important stuff. So, um I think we all spend enough time in demos looking at marketing data and retail data and e-commer, you know, like let's do something different.
So, all right, back to my four button now. Cool. We did that. All right. So, uh I'm going to spend some time in this newly kned role. Uh I've been charged with sort of poking around a little bit, starting to develop reporting packages. We'll eventually develop agents and everything, right? The neighborhood will just run itself. Um, but I'm gonna poke around, see what kind of access I've been given, see what kind of data I can find as a user to build reports.
And so I'm gonna start from the very beginning. And in this case, uh, I'm actually going to start on this discover page, which is something that I kind of just highlighted a little bit ago. Um, and thankfully even in the, uh, demo, uh, environment, it actually loaded, which is a plus. So what you're going to see here is, uh, curated in two ways. either manually by a user who has certified it and kind of set it to be featured for that area or we've got algorithmic surfacing as well.
So if there's like really high view counts and it's been certified and it meets some of these conditions, those things are going to surface to the top. And so you kind of pair the best of both. Hey, just because people a lot of people are looking at it, it doesn't necessarily mean it's good or that they should be. Um and so we give you some controls to kind of do this this dual check, right? So I'm going to come in here.
You know, again, I don't we must be sharing an environment with a bunch of other people here. I I'm not seeing a whole lot of HOA stuff. You know, I've got triedand-true taxi cab data. We've all played with that enough. I'm not seeing it's got workspace usage. Let's sort of poke back. Let's see what we got in here. Attributions, contacts, accounts. Okay, so in this case, I'm not finding what I need, right? Uh which was planned, which is a sidebar.
Uh I don't want to spoil too much of the discover session. I encourage you to go and do that. But as a user now, I'm I'm kind of stumped, right? It's not floated up here for me. I I don't have this one-click experience. So, what am I going to do now? I'm going to come in here and I'm going to do a stone cold lawn and order search. And what am I going to get? Well, I'm going to get some notebooks. Looks like maybe some of this is ETL related.
I'm going to get some tables. All right. Some scratch SQL. You know, I wonder if there's any dashboards. Oh, there is. Command center. Well, that sounds what I'm looking for. I'm sort of power hungry. That sounds very official. I'm going to click into that. Okay. So, now we're getting somewhere. So, so what am I looking at here? I've got the ability to look at reports by my neighbors, who I mostly hate, but sometimes like, uh, by year, street.
I've got a raw tech search. I've got incidents by type. Looks like we're very interested in mailbox and lawn care related incidents. These are a real big problem in our neighborhood. Uh, I've got pet information, which is questionable to store in the system, but nonetheless, I seem to have that. Um, what else do I Oh, there's an ask Genie button down here. Interesting. So, somebody's already been in here, and I think built some of these reports, but I don't know that I trust yet.
So, we'll we'll get to that in a second. But, it looks like we've started to talk to Jeie. We we've asked what the top pet names are, which doesn't seem relevant to me in my role. Although Bilbo Baggins, uh, Admiral Feathers, Barkley Cooper, these are all great names. Uh, I don't know that helps me actually enforce the neighborhood rules, though. So, maybe let's let's come back to that. Um, I want to know what's under this dashboard.
Uh, what I know as a user for the HOA is that we've got a lot more data. I I'm going to come back to this dashboard. I think I'm going to work with it later, but let's actually look at what's under the hood. Okay, so the dashboard's certified. In this case, it's a single table coming from the silver layer thing. Let's click into that and see what we're working with. All right. So now I'm seeing again. So I've got quality monitoring on this table.
It looks like healthy as as if you can see here on the right side. I've got good uh uh healthy state on my table and the freshness looks good, which means the jobs that have been monitoring, they seem to be running appropriately. We've got a whole session to talk about. Again, uh quality monitoring. I feel like I'm just pitching you on other sessions, but that's sort of the nature of this one. Um lots to talk about there.
Uh I'm seeing again certified. It's Delta. It's pretty popular. It's managed. Um I see that it's got keys defined. So I wonder if I've got keys defined. Can I actually Okay, so I can I can look at the relationship here and not just lineage of like notebooks seating other tables, but the actual key relationships between tables. So that silver tables being seated backward from bronze coming from this house and this resident table.
Okay, that's interesting. All right, sample data here. Uh, one thing you'll notice for those of you that may uh have used UC before is we're starting to push intelligence and genie into more spaces of the product. So you're seeing here as soon as I click into sample data, I'm getting an AI prompt right here, right on the table, right in the sample data. And you know what? That seems interesting. What is the yard score?
The average yard score. Clearly, we're rating our our neighbors on how well they stripe their lawns. What's our average yard score for people that happen to have a pool? Um, okay, that looks good. I think this is just going to give me a score, right? It's not going to tell me. So, three. I don't really know what that means, though. So, I'm going to come up here. I say that's that's okay. But, uh, what is the average yard score for houses with a swimming pool versus those without?
Typing in front of a crowd. Always the hardest problem. All right. So now I'm actually going to get the flag. That looks better to me. So I'll have some reference for what I'm talking about. Okay. So it turns out that people that have a pool take better care of their lawns. This is something my HOA is going to govern around now. Maybe we'll have some rules in place. Maybe we'll force you to put a pool in just because we're feeling extra mean.
Uh so this is good. This is good context sample data. I'll come back to this. What else can I look at? H it's a bunch of config and stuff. I I don't care so much about that. Uh permissions. Okay. Okay, I can see what permissions are applied lineage. Okay, so there's a lot of upstream and downstream stuff on this table. In this case, you know, I can see from silver, upstream from gold. I'm driving some other summary tables.
Okay, so this is giving me kind of a general idea of where some of the stuff comes from. Uh insights again, as the longer you have some of these tables and resources, the more information you'll have um to this totally not contrived demo. You can see there's a limited amount of history uh from a user standpoint, but uh if you had long-standing stuff in here, you would see ongoing queries and usage by other users. Um lots of information about the notebooks and dashboards and queries that are using that resource.
Um and then frequently join tables as Genie or as UC becomes more and more aware of what your users are doing with it. You'll see that surfaced here as well. Um quality uh is something I'm not going to spoil too much today. Uh but again, if you if you have quality monitoring on in these first two rules, it's going to be looking uh for a healthy state based on freshness and completeness. Freshness is kind of thinking about, hey, if this is a daily job, I've fired seven out of seven days or I've fired six out of seven days, so I'm freshish.
Uh and then completeness is going to look at, okay, what's your average sort of net rate of change? So, how many rows tend to be created, upserted, deleted? What what is that rate of change? Um we've got more rules coming that are out of the box. We're kind of taking more of an opinion on this stuff. Uh, and lots more to come there. Uh, so that is all super interesting. Um, what I'm thinking about now is let's see if we can go into this.
I've I've been to Silver. I've kind of seen what I'm working with. I have a feeling somebody has started some some SQL and things already. So, I'm going to start kind of all the way back from the beginning. And I'm going to say, take me back where I was. because I think I saw something that something start somebody started for me. Oh yeah. So there's a scratch SQL file. So somebody started to play with some of this data.
Okay. Interesting. That's my gold. This is still pretty aggregated. Let's go down look at some of these other things. Okay. Again, I think I'm kind of aggregated by household. Oh, that's interesting. It looks like I'm not allowed to see phone numbers in here. So this this must be a policy that's stopping me from seeing that. I wonder if there are other policies on data. If I go all the way down to kind of the raw. Okay, so there's lots of stuff.
So there must be a tag policy here. Um or fine grain access controls that I've set on the catalog. So the legacy way of dealing with this would have been you build all these dynamic views and roles and all these things. Um I'm actually really curious why this is happening to me because I feel like the job they gave me uh I should kind of have the power to see these things. I'm gonna have to take that up with somebody.
I'm really curious. Let's see. Okay, so there are some masking rules here. So, I've got a restricted rule. I've got a private. Um, let's look at this hide one. Okay, so this is applying to me. I am an HOA user inside of the context of this beautiful lawn and order application that I'm using. Uh, it looks like I'm not part of the admins group. I'll have I've got a bone to pick about that if I got to do my job right. I'm going to have to have admin privileges clearly.
Uh, and okay, it applies a whole catalog and it applies to anything that has a tag of SSN. Looks like this is a compound for a driver's license. Okay. And then if it finds that, then it's going to run the any data through this function. So now I know that anytime that we add passport, driver's license, phone number, whatever it is, as a regular user, it's probably going to be masked away from me. All right. Well, that's kind of the end of my live unscripted demo to sort of take you through uh uh the from a user journey standpoint what you'll see in UC.
There's a whole other set of demo cases that we could do from an admin point of view on system tables and like digging deeper into quality monitoring and a lot of that, but I think the more mature you see gets, the more it's important to understand what the business user experience is, what that what that developer experience is. There's plenty of time for us to get into the weeds on the admin stuff, but uh I just kind of wanted to give you a light uh walkthrough of what the user experience is shaping up to be.
Uh and now I think I'm back to here. We did it. All right. So, um like we talked about and we'll just kind of close this and then I think we're going to have about fiveish minutes for questions or dancing. Uh whatever your preference is. Uh I warned you. uh Unity catalog uh is really way ahead of a lot of what we're seeing in the marketplace. We've been talking about having an open unified catalog for a long time. I wish I had another slide I really like to use, but suddenly it's very invogue to have an open unified catalog.

32:06 Conclusion and Q&A

So much so that everybody's using the same words that we've been using for some years now. Um Sachin talked about uh what we do with Lakehouse Federation to kind of get things into UC's visibility uh which gives you centralized auditing and all the cool stuff that I just talked about. Um, I didn't call it out here, but you may have noticed there was a ton of cataloges on the side there that were like Postgress and Snowflake and all this stuff.
Those are still living exactly where they were, and yet I could have interacted with them in a notebook if I felt like doing that in the demo. On the other side, we've got uh open APIs. So again, anytime we have a conversation now about, hey, does X work with Unity catalog? My favorite thing to say is, I don't know, you should ask them because our open APIs is public code. if they want to integrate that, they have all the tools they need to do that.
Uh, and we've seen massive uptick in terms of that happening with the meta managers and the business cataloges and a lot of the popular clients. Um, and again, we're kind of the only shop that gives you a unified experience for iceberg and delta and parquet and everything format agnostic. Uh, I would say it's by accident, but I don't think you buy billion-dollar companies by accident. we became the best iceberg catalog and I think there's going to be a lot um to think about there as well.
So, right at five minutes, um I'll flash this slide up since I think everybody can remember the word questions with a question mark as a prompt. Um uh we've got several brain dates and AMAs and things set up through the week. So, if you know you don't want to be handed a hot mic or or ask now, there should be opportunity as those fill up. At least the ones that I'm doing, I know I'll continue to schedule more. So you have opportunity to get time with us that way.
Um, but any questions that we can answer in the room?

Learn more about the Databricks Data and AI platform.