What’s New in Unity Catalog With Live Demos

Summary

Access control innovations: Unity Catalog's latest enhancements provide fine-grained permission management and attribute-based access control for complex governance requirements.
Discovery and lineage improvements: Enhanced data discovery with AI-powered recommendations and automatic lineage tracking help users navigate and trust data assets.
Monitoring and quality features: New monitoring capabilities detect data quality anomalies and lineage health, enabling proactive governance and issue prevention.

What’s New in Unity Catalog With Live Demos

Talk by Murt Neemuchwala, Product Manager, Databricks ; Paul Roome, Senior Staff Product Manager, Databricks
Join the Unity Catalog product team for an exclusive deep dive into the latest innovations and upcoming features of Unity Catalog! Explore cutting-edge advancements in access control, discovery, lineage and monitoring — plus get a sneak peek at what’s coming next. Packed with live demos, expert insights and best practices from thousands of customers running Unity Catalog in production, this video is also your chance to engage directly with product experts and get answers to your most pressing questions. Don’t miss this opportunity to stay ahead of the curve and elevate your data governance strategy!
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What access control improvements are new in Unity Catalog?

Latest features include attribute-based access control (ABAC) for dynamic permissions, fine-grained resource-level access, and integrated audit logs for compliance and security.

How does AI-powered discovery help data users?

AI recommendations surface relevant datasets based on user roles and previous queries, natural language search improves findability, and AI-generated metadata enriches asset descriptions.

What does automatic lineage tracking show users?

Automatic lineage displays complete data pipelines—showing how raw data transforms through gold layers—helping users understand data provenance, dependencies, and impact of changes.

How do monitoring features prevent data quality issues?

Continuous monitoring detects schema changes, data anomalies, stale datasets, and lineage breaks—enabling teams to proactively address problems before they impact downstream consumers.

Full transcript

`0:00` Welcome and introduction

Hello, Bricsters. Welcome back for another exciting day here in San Francisco. We're so excited to have you all back. I know this is the last day. We're all a little bit sad, but we still got quite a few items left on the list. I hope you've been taking advantage of all of the incredible learnings that we've had. We're going to have some breakout sessions here coming up next. But real quickly before we do that, let's take a look at what is on the on the schedule for today.
So we heard our keynote speak this morning. Really incredible. Some additional announcements. We heard about declarative pipelines going live, the GA of Lake Flow, and now I mean I I I kind of have to giggle. DBSQL is 25% faster for the same price. I mean we just we just keep getting better. And we're so excited for you all to be joining us with all of these amazing announcements. If you are here on campus with us in FF SF, we would love for you to come downstairs to the expo, visit all of our sponsor booths, check out the more demos, take in our lightning talks, and talk to the experts, connect with your peers.
That's what we're here to do. Coming up now, we're going to have a couple of breakout sessions. I hope you've been enjoying those. And as well as the other days this week, we've had our certifications. It's been really cool to see so many people walking around with all of these jackets that are embroidered with their certification logos. You don't just get to put those logos on your LinkedIn accounts. You actually walk away with some really cool swag.
So, we're really excited for you all to be here and hear some additional breakouts. Right now, we're going to turn it over to two gentlemen from the Unity Catalog team and they're going to talk through a couple additional items from what we heard earlier this morning. What's new in Unity Catalog? are going to give us some live demos and they're also going to talk about access controls, lineage, and some additional items that are coming soon to a Unity catalog near you.
So, with that, I'm going to throw it over to the stage. go ahead. Take it away. Thank you. So, thank you for joining us to talk about what's new in Unity Catalog. So I wanted to begin with what to expect in this talk. So you've got down there and you've got up here. We're from the product team. We look after Unity catalog and we wanted to give you an insidider glance into what we're really thinking about uh what's coming in the road map and what's hot in the world of cataloging.

`2:11` Evolution of data catalogs

Secondly, we're going to roll the dice. We're going to do live demos. So fingers crossed for that. And thirdly, we go really fast in this talk. Okay? So we're going to cover a lot of topics. If you feel like we just scratched the surface of a particular topic, then check out your uh session schedule and recordings because they're probably covered in a deeper dive as well. So with that, I wanted to begin with where we were as an industry 10 years ago.
Okay, so this screenshot is actually from Sparks Summit 2015, which is almost 10 years ago to the day. And um you see a youngl looking on screen there demonstrating the latest offerings in Spark's IO packages. And I want to point out the three patterns I see on the screen. So the first one we see Spark reading from a file. So this case we are reading from this file called people. json. Okay. Um the second one we're actually accessing a database.
So we're accessing a database by hard coding this JDBC URL in there. And then the third time we are um reading a table in Spark from Hive. So we're actually using a different systems catalog. So the way that I would summarize this is in the beginning Spark had really great connectivity but it had no unified catalog. Okay, three different sources. I had to think about how to access them all. So from a end user perspective there was kind of you know there was some knowledge you had to know in terms of how to access these different sources.
And if you think about how do you discover data, um it's basically a non-existing surface. You have to go to the individual source systems. You have to hunt around files and folders in S3 and ADLS to find your data. Okay, so that wasn't good. And then I think of the poor governance team, right? So the governance team needs to think about how do I secure access to all of these different types of data? How do I audit them?
How do I do data quality? And that problem's way harder because I've got to rationalize all of those different sources. The good news is there's this famous quote. It says that all problems in computer science can be solved by another level of indirection. Okay? And for me, the catalog is that level of indirection. So cataloges abstract data access. So the user and the governance teams don't need to think about it as much.
Okay. So in the example I have on the screen, the catalog is sitting between the user and the data and it's providing this higher level abstraction where you can just r you can think about tables instead of files. Um so from the end user perspective they now have a unified way of accessing this this data. They just interact with these different table objects and they might happen to be pulling data from these source different source systems.
And then in terms of the discovery surface, the user has a list and can search and filter through all of these different higher level objects versus hunting through the source systems as well. And then the governance team, they're way happier. They can think about who has access to each object or table and who did access them instead of worrying about the end different sources. So you might be thinking where does Unity Catalog come in now?
Well, you probably saw in the keynotes, we're all in on this abstraction of the catalog, right? So, we really think about kind of three different layers of the catalog. The grinning face you see, that's what I think of as the UX and discovery surface. Then you have the governance capabilities, things like access control, auditing, lineage, and so on. And then you have the different types of data and AI AI asset that you ultimately access, right?
So these are the three layers of Unity catalog and um our goal with Unity is really to reach all users. Okay, provide best-in-class governance across all engines. You saw that in the keynote today. You don't have to use data bricks to access these assets and then um allow you to access all different types of data in AI assets uh in this pattern. So what I wanted to do in this talk, we're actually going to have a look at uh we're going to go through chapters and look at these individual kind of building blocks and talk about how we're advancing towards that goal that I shared in the previous slide.
So in chapter one, we're going to talk about the kind of governance capabilities box and how we're evolving uh recently and in the next year as well. So access control, this is a really big topic in in governance. Uh and we're going to talk about a few topics here. We're going to start with attribute-based access control. So what do you need to know about this? There's three things. Okay. So the first thing attributes attribute based access control.

`6:57` Advanced access control

It starts with attributes. The clues in the name. So what are they? They provide context about assets, users or environments. So in the example you see on the screen, I have this small table. The column called SSN is tagged. That's a type of attribute with this this tag called PII, right? That's giving me context about that column. Next you need to know about policies. So policies are a new object type that we have in Unity catalog.
And basically they they are rules that dictate users access by looking at the attributes. Okay. And then thirdly what do you do with this policy? Well you actually go and stick it on one of your nodes in the Unity catalog hierarchy. So maybe you go put it on your catalog uh on the top level at the catalog level and then it will inherit down. So if you put the policy on the catalog it will protect you know the thousands or tens of thousands of tables that are in that catalog.
Okay. So how does this work? So uh you should all go try this out by the way because it's now in beta and you can launch that uh in your workspaces. But let's say we had this use case of protecting all data tagged PII in this catalog. How would we do that? Okay, so firstly we would run this feature called data classification. This would automatically find all of the instances of PII in our lakehouse. Okay, it uses both the metadata and also it looks at the at the rows of data and runs a classifier over that to try and find uh that PII data.
Secondly, you would author this policy only admins can see assets tagged PII and you would stick it on the catalog node there. And then finally, this is super a super safe uh paradigm because the individual table owners if they get annoyed with you, they can't override this. This is specified by this highle admin. And so uh those individual table owners are governed by the rules that come down from the governance on the on the catalog level.
Okay. So this is Aback and I'm going to pass over to Mr. actually who's going to do a live demo of this in action. Awesome. Thanks. So, I'll be doing a quick demo just to show youall just how easy it is to use attribute-based access control. So, you see here I'm in a production catalog and I have a bunch of different kinds of data that's been added to this. And I'm worried as an admin that there might be some potentially sensitive information that's been ingested.
So, I see there's some customer data that may have PII such as names or email addresses. I have some log data here that I'm worried might have things like IP addresses. And uh as an admin, I want to be able to first find all the sensitive data and then set up a policy to go ahead and mask it. So, I'm going to start by clicking on the catalog and I'm going to go into the details tab. And you can see there's a feature here called data classification that mentioned.
So what this does is it'll go through all the tables and all the schemas in this catalog and it'll automatically tag the specific columns with the type of PII that it detects. And so I've gone ahead and enabled this and I can click the see results button which will open up a out of the box dashboard which gives me insights into the types of PI that were detected um errors with detection classifications over time. So if new data enters this catalog, it'll automatically be tagged and then if I have a policy set up, continue to be masked.
So I'm going to switch back over here and now when I go into my tables, I will find that the sensitive information has been tagged with a type of PII. So you see there's some names here and email addresses that have been tagged. Um, and in my log data, there's IP addresses that have been classified by our data classification feature. But these are still kind of uh they've been tagged, but they're still in plain text.
So I want to go ahead now and set up a policy to mask them based on that attribute. So the first step in doing this is I'm going to go ahead and create a very simple masking function. So I'm going to switch over here and I'm going to run this create a replace statement that's going to uh just very simple function that says it takes in a string and it returns a masked value. So now that I've created this, I'm going to set up a attribute-based access control policy based on this masking function.
So I'm going to click new policy. I'm going to call it mask all PII. This policy mask catalog, I'm going to apply this policy to all my account users, but I'm going to make a special exception for my uh super admins uh group. So essentially it's saying that all the account users if they're regular users will see these values as mast but the super admins can still access sensitive information. What's really cool about this is that previously you would have to put this kind of logic in your uh function itself with native uh column masking but with attribute based access control you can abstract it into the policy which makes it far more performant performant on read events.
So I'm going to choose it's a column masking policy type. it targets this production catalog. I'm going to go ahead and select the masking function that I just created. So masking function and you can see it's very simple. It just returns a masked value for that. And I want this to be applied whenever uh there's a PII tag on the column. So I'm just going to click meets condition and I'm going to paste in just a simple hashtag either any of these phone number or kind of all the tags you saw with sensitive information.
And I'm going to create this policy. And just like that, it should now go ahead and mask all the sensitive information. So, I'm going to go ahead and verify this. So, I'm going to head over into my uh customers table. I'm going to hit sample data, and I'm going to refresh the tab. And you can see now all the sensitive information has been masked. And I'm going to check with my log data as well. And refresh. And hopefully all the IP addresses are now masked.
Awesome. So to recap, all I did was enable data classification on the catalog. Um set up a masking function and put a catalog level policy to mask all PII. I'll hand it back. Excellent. Exciting stuff. So let's continue how the chapter on access control here. So the next thing we're going to talk about is exclusive data access via arbback. Now the first thing to say everyone has something called arbback. Okay, forget everything you know about arbback because uh everyone uses it in different ways.

`13:41` Quality and governance features

So what do we mean? So we're trying to solve this use case where she is a very privileged user and we we allow her to access data in these three different trials but we don't want her to do it uh we don't want her to be able to join between these trials. she should have to access them one by one. Okay, so this is our use case. How would this happen today? Well, without arbback, uh the way that access level is determined is by taking a union of her permissions and the permissions of any group she belongs to.
And so today she could join across these trials. So how will this work with the new feature? So with Arbback, we're going to introduce this new concept called an exclusive group. So you're going to be able to go to your groups in data bricks or in from your IDP and you put this tag on it that says it's exclusive. What does that mean? It means that to get the permissions of that group, you first have to go ahead and assume that group.
Okay. So you see in the little diagram here, is no longer acting as herself. She's now acting as group one. And that's the only way to get the permissions of group one because it's an exclusive group. So now she can access trial one's data but she can't access trial two or three. So we've solved our use case. Okay. So this is coming to your data bricks instance uh very soon. Okay. Just a quick fire one here on val volume path permissions.
So what are volumes? You're probably using them. They're a way to store and access and govern unstructured data and files. Okay. The challenge today access is at the volume level. So if I want to grant access to this volume, I have to choose well, she has to be able to see all of the files in that volume. Okay. And customers are asking for some a little bit more. So maybe I have a use case when my volume has two folders in it and I want to partition access by those two folders.
So uh what we're exploring is the ability to actually be able to say, well, you're allowed to see data in this volume, but only folder A and can see folder B. uh without this you would have had to delete the volume create two which might cause downtime and uh and disruption. So uh lots of customers are really excited about this. Okay. So governance is so much more than just securing access to data. Okay. Governance is also about things like data quality and I want to spend the next couple of minutes talking about this.
So we spent the last year speaking to hundreds of customers about data quality and we learned three things. So the first one of them, the first challenge that customers were having is that the way that they that it's been approached uh historically is by writing a number of custom rules that you can only do so on a small number of tables. Okay? So it's custom, it's manual, and you can only service a small number of tables.
Secondly, because those rules are custom, they're actually very hard to tune, right? what works last year doesn't work today. And it's, you know, for things different seasons, you know, you have a a busy Thanksgiving period and you're in retail, you might suddenly get a lot of drift in how how well those rules work. So, you get a really noisy environment. And then challenge three, we find that data quality is a very asymmetric property in the enterprise. uh and only typically only a few people know about the quality of the data and certainly the end consumers don't always know about data quality.
Okay. So what are we going to introduce? So the first question that we're going to help you try and answer is the age-old question. Is the data up to date? Okay. Um so how are we going to do that? You're going to go into catalog explorer. You're just going to go and enable data quality monitoring for a whole schema. So that might be applied to tens of thousands of objects. And then what we'll do, we'll automatically build up this per table time series analysis of the commit history of the tables.
So in the screenshot you see there, all the greens is when we've seen a commit happen, a write happen to this table. Okay? And we can obviously then use an AI model to predict when the next commit should happen. And if it doesn't, then we we think this the table might have gone stale. Okay? And that's what's happening in this tan element down there. You see that perhaps this table is actually gone stale? And remember, we're doing this automatically across all of the tables you've enabled this on.
So all of the tables in the schema. Secondly, we're going to help you answer the question, is the data complete? You know, does it have enough records in it? We're going to use a very similar methodology here where once you've enabled it on the schema, we build up this time series analysis for every table that looks how looks at how many records are getting ingested into the table every every day. Okay? And then flag whether we think that's within the you know within the norm.
In this case, we think the completeness is as expected. But this is how we turn those manual rules into automatic AI data quality insights. The second thing that we saw, the noisy environments. How are we going to solve that? Well, here we have two alerts. Maybe they're on two different tables. The good news is we have the context of what's going on in the platform to help us prioritize between these alerts. So table one perhaps there's only one happy user querying this table, but table table two has thousands of users and critical reports that are actually uh you know using this table.
And we can determine that by looking at the lineage that we have in Unity catalog. So in this case, you probably want to look at alert number two before number one because it impacts uh more critical things. So we are able to prioritize incidents based on downstream impact. The third challenge of asymmetric knowledge of data quality, we're going to solve with the power of a a small dot. Okay, UX is very powerful. So you in the screenshot here, you see like an orange blob or a green blob, a little dot.
Wherever you get your data from, we're actually going to start servicing these little data quality icons so that at a quick snapshot, you can see whether the data, you know, is complete, whether it's uh, you know, it's missing records or or whether it's out of date. Excellent. So that's how we help more users understand about data quality. Okay, so that concludes chapter one. We're going to move on to chapter two now.
And I'm going to do one more section here before we we get another live demo. And you saw this in Mate's keynote, but I I thought it deserved a a little bit of a deeper treatment. And this is to do with metric views. Okay, so the question that we're trying to answer here is the following. Let's say my boss walks into me. I'm a SQL analyst. And she says, tell me, you know, I've got a board meeting. What's the average revenue per customer by segment for fulfilled orders?

`20:34` Metric views and definitions

I would do what every self-respecting SQL analyst does and go and ask an AI to write me a SQL query, right? And the AI would probably go find a table called orders. It would find a column called revenue and it would write a very relevant and correct looking SQL query like this, right? And I would go give my boss the answer. However, Mr. who's a more discerning user. He knows that I'm not supposed to use the column called revenue.
It's supposed to be revis dbx and I need to do something slightly nuance when I check the order status. So the problem here is that we had the same question but we have two different answers depending on our underlying knowledge of the data. Okay, that's a governance professional's nightmare. Okay, so how are we going to solve this? Well, metric views is designed to capture that logic up front. So, a central admin or governance persona would define these different things so the users don't have to guess afterwards.
So, particularly they'll define dimensions, things you want to be able to slice and dice the data by and measures, things you want to be able to aggregate and sum by. And then you get all the goodness of Unity catalog. You can control who can edit these definitions and who can use them as well. Um, and so when and come along, we simply reference those predetermined measures and dimensions in our SQL query, and so does the AI, and we all get the same answer.
Okay, so I'm going to hand over to do a demo of this in in real life. Thanks. All right, so I'll switch back over to my data bricks instance and I'll do a demo on metric views in Unity Catalog. So, I'm going to pretend that I'm a a SQL analyst and I've kind of been tasked with figuring out uh what was the revenue uh in the last fiscal quarter. So, typically I would find some table with some sales data. I see there's a column here with revenue in USD and um I like using our text to SQL functionality.
So I would maybe ask what was the revenue in the most recent fiscal quarter. And this will probably generate a query that looks at that uh revenue amount in that column. And when I switch over to my SQL editor here and run it, it'll give me a number. And it looks like we did about 3. 2 billion in revenue the last fiscal quarter. So we're doing quite well. Um, but this number actually is not the right one based on our uh internal business logic and I'll explain why.
Um, so I'm going to switch back over to the catalog explorer and I'm going to create a new metric view. So this is this new type of object in Unity catalog in my gold schema. So a metric view is defined through a YAML definition. Um, you can see there's an example here, but I'm going to uh cut this out and paste from my notebook a predetermined uh definition. And I'll go ahead and explain what exactly this is doing. So, as mentioned, there's a couple different aspects of a metric view definition.
So, there's the source table. So, I'm still referencing those same tables in my silver sales schema. Um, and I've defined this measure. So, this is like the canonical metric that can include your business logic. So, I'm basically just saying that um the way we calculate revenue, we have discounts in the form of rebates to customers. Um and there's different kind of reasons why we give rebates. So, uh I'm looking up in our uh contacts table to see if they were referred.
We're going to give them a 10% discount in the form of a rebate. So, we want our revenue to account for this. Um so, I've added some logic in my uh measure there. And then I've also defined some dimensions which I expect my consumers to be able to slice and dice this revenue number by things like customer name, product category, fiscal year, quarter. So once I've defined these dimensions and this metric based on my business semantics, I can go ahead and give it a name.
So I'm just going to call this my certified revenue and click create. And this successfully created a metric view in Unity catalog. So um now whenever consumers come to see uh this data they can kind of go ahead and query this similar to a table and it'll have this metric. So um I'm going to go ahead and query this and I'm going to use our genie functionality which is inbuilt here. And I can see Genie has prompted me um kind of what I'm looking for.
Show me monthly sales revenue by fiscal year quarter. And this is that same kind of calculation that I was running against that base table which now I'll run against the metric view which has that additional logic about rebates. Um and it looks like our revenue has been growing. And the most recent fiscal quarter it's actually 2. 7 billion. So uh kind of a difference from the 3. 2 billion number I got before. Awesome.
So to summarize, I created a metric view with some uh logic encoded defined the measures and dimensions and now my consumers can access this just like they would a table in Unity catalog. I'll hand it back. Excellent. So let's continue our chapter on the different data and AI assets that you can consume in Unity and govern. We talked a lot last year about Lakehouse Federation. Now this is the technology that allows you to take one of your data warehouses or databases register it in unity and it will crawl those systems so you can access it in place and govern it in place.

`26:19` Federation and managed tables

Really powerful technology we added terod data and Oracle this year. Customers are really enjoying that. However, I wanted to wanted to speak about a new pattern of lakehouse federation that we have and that's called catalog federation. So you can see in the diagram here, we now support the ability to take an external catalog system such as Apache Hive, AWS Glue or anything that speaks iceberg and you can basically register that in Unity catalog and we will crawl that system and uh surface all of those assets in data bricks so you can query and govern them.
The really cool thing about this is that you get access to all of these different types of file format. So, you know, you probably have a lot of text data or JSON or parquet or CSV um as well as, you know, the more modern formats like lake uh delta and iceberg. And this is a way to bring them all into Unity in one fell swoop swoop. So, we've actually seen this emerge as a really great way to actually uh migrate to Unity, just bring in all of that data as an external uh catalog and federate it in and then you can start using the benefits of Unity catalog.
And I wanted to just pause for a second here. So we talked at the start about this abstraction that cataloges provide and lakehouse federation in my opinion is the prime example of this. Right? So I have 10 different sources you see on the right hand side here. And they're all federated or registered in unity and cat the catalog is providing this abstraction. The user doesn't need to know that my data is in glue or bigquery.
They just simply query a table. Okay, so this is really the power of the abstraction that I was getting to at the start. Okay, one more topic in in this chapter. So we're going to talk about manage tables. Okay, and um the open secret is 10 years ago in the screenshot I showed at the start, no one liked managed tables in Spark and Hive, right? Like everyone wanted to use external tables. The opposite is true of Unity catalog manage tables.
Okay. And I want to spend a couple of minutes explaining why why that why that's changed. So what is a manage table? Firstly, it's the simplest way to create a table. You simply say create table and give it a name. This will be a manage table. So it's super simple. But the defining characteristic of this table is that the source of truth for the metadata is stored in Unity catalog. And critically when you want to speak to that table, you have to speak to Unity first. you cannot go to the file system.
Okay. And you might be thinking, that's kind of a red flag. I like the file system. So why why should you switch? So the first thing, the very fact that you have to speak to Unity to speak to the table is the superpower. It means that we can see all of the queries that are hitting against this table and use them to optimize the layout of that table to make it run faster. Okay. And we can also do automatic maintenance operations.
So we can delete files, we can compact files to make it cheaper for you to run. So long story short, manage tables are both faster and lower cost for your environment. But part two, you might still be thinking, but, I I like the file system still. I have external systems. I I don't just use data bricks. I use open source spark and trino, etc. I need to be access to the tables. Surely that means I need to use uh external tables or or go to the file system. you don't have to choose anymore.
Okay, so you can actually access your Unity catalog manage tables from all of these different clients you see on the right hand side here. And you saw the exciting demo uh that was done in the keynote. You can actually both read and write to your manage tables using uh these open APIs. And we're the only vendor that allows you to do that to operate on these manage tables from any engine. Okay. Um, one slight variant of this again we spent a while on in the keynote.
You now get these two special words here. If your company is standardized on iceberg and you want to benefit from that, you can just use this create table using iceberg syntax. And the good news is you don't need to know anything more. Okay, you get the same benefits we just talked about. We optimize the files and the layout, do the optimization, and they're super open and interoperable as well. So either choice is a really great choice.
Okay. So we're just going to spend uh a few minutes talking about the grinning face at the top of the screen here. This is what I was using to denote the kind of UX and discovery experience of Unity. Now we've been talking about the power of this abstraction, but in reality this isn't a full picture. There's not one user, there's thousands of users and the catalog can see everything they're doing. Okay. And what this does, it actually allows us to build a really rich discovery experiences for the users because we can do things like analyze query uh patterns, see which tables are commonly joined with other ones, derive lineage, do AI suggested comments.

`31:19` Discovery and data usage

Um, and this really powers the discovery experience of all of the users. So, we kind of have a superpower there. So, we've we've used that to build a lot of different stuff on the discovery side of the catalog this year. Some of it you see on the screen right here, but this is best demoed. I'm actually going to pass over to actually do this justice by giving you a live demo of some of these features. Awesome. All right.
So, I'll switch over back to my data bricks instance to show you a demo on our discovery experience. So typically um you would have pointed your uh analysts and business consumers to the catalog explorer where they could uh find and discover data sets. Um we want to make this journey even easier for nontechnical users which is why we've introduced a brand new discovery surface called the discover hub. So I'm going to navigate to the discover hub by clicking discover on my lefthand nav.
And you'll see that there's a bunch of objects that have already been featured here. So I'm going to click into the featured tables and there's actually two ways that u objects populate in this featured section. So the first is through admin certification. So your data producers can go ahead and certify certain assets for certain users and these will kind of be promoted in this page and they can go ahead and look at them.
And then the second way is through our AI suggestions. So, as mentioned, Unity catalog has a lot of metadata around query patterns, who's accessing these tables, the lineage for them, and based on that, we can kind of give you custom suggestions on here's the uh data sets that you are most likely to want to use. Another really cool thing about discover page is we're adding this concept of data domains in Unity Catalog, which uh allow for custom curation of things based on the business areas and use cases.
So u for a quick analogy like in the catalog explorer you may have a lot of data sources organized by um kind of the source system or the medallion architecture which your business users might not be familiar with. Like if you think about uh something like Netflix whenever people are browsing it they don't care about kind of as much on which uh movie studio produced a movie more than kind of what genre it's part of. And so similarly here you can organize things by domains based on kind of your intent of what you want to do with that data.
So, as a consumer, I'm going to go in and click on, for example, I want to understand the performance of my marketing campaign. And I can see that there's pre-curated dashboards and tables. Um, and so I'm going to click into one of these. So, I see there's attributions table with a lot of queries against it. And I actually don't even have data access to this table, but I have a full discovery experience where I can kind of understand what this uh data is.
I can see that there's a a view relationships button which lets me see an erd to see what are the related tables here and the keys to join them on. Um I can also kind of get insights around um who are the top users of this table. Um so if I have questions I can reach out to these folks. I can see what they're frequently joined with and related assets that are downstream of this. And this kind of gives me a lot of trust signals um into this is the right kind of data that I'm looking for.
Um, what's really cool about this is I can also access the full lineage for these tables. So, um, I can see here's some downstream assets based on these tables. And what's really cool is Unity has always had really great autocaptured lineage. And we recently launched a new feature called bring your own lineage, which allows you to uh add metadata for lineage living outside of data bricks. So, you can see that here there's an annotation that this actually comes upstream from a Salesforce object and I can see this in the lineage graph. um which gives me kind of more insight on where this data came from.
And then similarly u I can see that there's a downstream uh dashboard outside of data bricks in Tableau that's dependent on this uh table downstream. So I can get a full endto-end view of my data flows within Unity as well. All right. So I've seen a lot of trust signals. I see this has been certified by my admin team and a lot of people are querying this. Um kind of the next thing I would want to do is get data access to this um schema.
So we have a new feature request for access where I can directly request the permissions that I want and my admin can can configure who this would alert and they can kind of go ahead and provision me access. So I'm going to do this live. Um I'm going to request my admin to uh give me access to this accounts table. I'm going to request read access. I need access to run some marketing analytics. And in one click, this will go ahead and send over um to the admin team.
They'll receive uh an email of this kind where um there's a new access request for their object. Uh and actually happens to be the admin for this uh schema. So uh he's going to he's just gone ahead and granted me uh data access to these data sets. And just like that, now I can go ahead and uh query this data in natural language after just finding it just now and discovering it and getting all the trust signals through our data discovery experience.
Awesome. I'll hand it back. Excellent. Okay, so to wrap up, we looked at this very powerful abstraction called the catalog. We talked about these three different layers and how we're advancing towards the goal of servicing all users with best-in-class governance across all engines for all data and AI assets. Um really, you know, we want to support all of these different types of uh of ways to interact with the data. Um I'm going to leave you by asking for your feedback.

`37:19` Closing and next steps

Definitely go into the application and give us your your feedback directly in the app. It helps us to improve every year. and uh lots of really good materials uh if you scan this QR code to continue your learning journey. But thank you for attending the talk.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.