Skip to main content

Unity Catalog Deep Dive: Practitioner's Guide to Best Practices and Patterns

Summary

  • Metastore architecture and setup: Pamela and Jinlin guide practitioners through configuring metastores, connecting to diverse data sources and organizing assets for scale—foundational governance concepts.
  • Fine-grained access control: Unity Catalog's row and column security enable teams to share data securely, granting different users access to different data subsets without creating duplicate copies.
  • Lineage and discovery: Automated lineage tracking and data discovery tools help teams understand where data flows from and how assets relate to each other—critical for compliance and impact analysis.

Unity Catalog Deep Dive: Practitioner's Guide to Best Practices and Patterns

Watch: Unity Catalog Deep Dive: Practitioner's Guide to Best Practices and Patterns
Talk by JINLIN HE, Sr. Solutions Architect, Databricks ; Pamela Pettit, Sr. Solutions Architect, Databricks
Join this deep dive session for practitioners on Unity Catalog, Databricks’ unified data governance solution, to explore its capabilities for managing data and AI assets across workflows. Unity Catalog provides fine-grained access control, automated lineage tracking, quality monitoring and policy enforcement and observability at scale. Whether your focus is data pipelines, analytics or machine learning and generative AI workflows, this video offers actionable insights on leveraging Unity Catalog’s open interoperability across tools and platforms to boost productivity and drive innovation. Learn governance best practices, including catalog configurations, access strategies for collaboration and controls for securing sensitive data. Additionally, discover how to design effective multi-cloud and multi-region deployments to ensure global compliance.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is a metastore and why does it matter?

A metastore is the governance layer managing all data and AI assets—catalogs, tables, volumes, models. It tracks ownership, lineage, and permissions, enabling organizations to govern at scale.

How does row and column security work in Unity Catalog?

Security policies restrict access at the row level (hiding specific records) or column level (hiding sensitive fields). Different users see different data without duplicating tables—reducing sprawl and compliance risk.

How does lineage tracking support compliance?

Automated lineage shows the path data travels through transformations, from source to insights. This is critical for audits, understanding data quality issues, and tracing the origin of downstream insights.

Can Unity Catalog span multiple clouds and regions?

Yes. Unity Catalog supports multi-cloud and multi-region deployments, enabling global organizations to govern all data assets with one system while meeting regional compliance requirements.

Full transcript

0:00 Introduction and overview

Okay, we're gonna get started now. Thanks so much for uh attending today's session. Um this is the Unity catalog deep dive, a practitioners guide to best practices and patterns. Uh I'm joined by my co-host. We're both senior solution architects with data bricks and we'll be taking you through today's presentation. um standard forward-looking statement. You may see features here that are preview um some mention of things coming in the road map.
So just keep that in mind as you're reviewing the presentation today. And just to give you an overview of what we're going over. Uh we're going to start with setting up your meta store, get into the data foundations, how to connect to all of your sources so you can start bringing in your data. Um a bit of understanding around discovery for adoption. So how you can assist your consumers to find the right data to use within their projects and then once they find that data how they can scale both your uh BI business analysts as well as your AI data scientists and Gen AI engineers.
Um, and then we'll close out with a little bit on best practices for organizing your metas store as well as observability for auditing purposes and then we'll wrap up with a little bit of time at the end for Q&A. All right, I'm going to pass it off to now who's going to give us an overview of getting started. Okay. Hi everyone. Maybe before we start, let's show a hand of uh who are in the audience who have heard of Unity catalog.

1:26 Unity catalog fundamentals

Raise your hand. Oh, okay. This is great. Okay, keep your hands up um if you are actually managing Unity Catalog for your organization. Okay, great. We're great. And then also last one, keep your hands up if you are excited to hear about how do you use Unity catalog to not only govern your traditional data assets but also all these emerging AI assets. Okay, this is great. Okay, you're in the right spot. Okay, let's get started.
But um just in case because I also see some of the people in the audience have not heard of unit catalog before. Uh FYI, Unix catalog is the governance layer for data bricks and when we think about data governance right to its simplest uh form or term is how your user interact with your data. So on a data platform like data bricks that means it's kind of three components right first on user management. So how do you you integrate with your identity providers to create your users and then there is a category of data assets.
How do you manage your data assets on top of your your cloud storage? And throughout the session, we're also going to talk about how Unity catalog help you kind of abstract all these data assets in a logical way. So you are not always interacting with your cloud storage all the time. And the third category is your data bricks workspace assets. Yeah. And then putting this all together then a lot of emerging data governance questions becomes like how does a user discover your data assets?
How do you make sure your uh users are having the right access to your data and how are your dollar DBUs spent across the platform and how do you audit users uh actions performed on your platform. So we're going to guide through guide you through all these um questions and best practices throughout this session. Okay, first on a really high level, right? Like how does Unity catalog fit in my cloud deployment and then how should I think about assigning admin roles.
So to start, this is how because data bricks is multiloud, right? So this is how um unity catalog metal store align with your cloud account. So if we start with AWS for each data bricks account it can be associated with multiple AWS accounts and then for one AWS account that can span across multiple data bricks workspaces as well and very similarly for Azure that's one database account can be associated with multiple Azure subscriptions and for GCP it's um can be associated with multiple GCP projects and how you should think about aligning workspaces to different cloud cloud account is really is majority is driven by billing boundaries for your cloud account.

3:36 Metastore and admin setup

Um just a few notes here is that it is possible for um for one database account to have multiple metas stores. It's just that you have the limitation of having one meta store per region. Yeah, just a note there. Okay, then okay now how do we assign the admin rows across all these boundaries? Right. So at the top you have your account admins. Um this is where probably you are at the very start of your uh database journey right you are creating all this initial assets around database account.
So you're creating your uh meta stores uh you're creating different workspaces. So it makes sense for a central governance team or platform ops team to assume this account admin role. And then moving on we have workspace admin. So the way to think about this is the workspace admin deal with the day-to-day operation within the workspace. So they are managing all the user access within the workspace. They're thinking about preview feature enrollment.
So it makes sense for um your IT team, platform team uh and devops team to assume this workspace admin role. And then next is the unity catalog meta store. So this the metas store admin holds the highest privilege over all your data asset because again right metas store is the abstraction on top of your data assets and since we already mentioned before it's possible for one unity catalog metas store to span across multiple datab bricks workspaces it again makes sense for a central governance team uh platform ops team to assume this metas store admin role and for them to then delegate privileges into different data owners in different BUS and the last one is marketplace admin.
This one the definition is more defined. So uh yeah it's the alignment there is more distinct. Okay we're going to move on to connecting different sources to PML. Okay. Uh so we're going to go over setting your data foundation. Uh so we'll take a look at um thinking about how you can connect to all of your upstream sources whether that's cloud storage or like upstream warehouses um and how to configure your cataloges and the assets that you're engaging with.

6:31 Data foundation connections

Uh so starting at the very beginning at the lowest level you have your storage credentials. Um this is the authentication mechanism that Unity catalog is going to leverage to interact with your cloud storage. Um you configure these in the UI. You can also configure these programmatically using REST API endpoints or Terraform. Um in order to create external locations, uh you need the storage uh credential created first.
Um so these come in a pair. Once you have that credential, you can get into uh external locations which are the uh connectors to your cloud storage. Um this allows you to um configure your storage across your cataloges. You can use one storage credential for many external locations um within your account and you can have many external locations associated with different cataloges. Uh so when we think about storage credentials and external locations just a couple notes on how this get configured create across each cloud.
Um typically when you're deploying within AWS Azure GCP the native storage is what you want to go with. Um but you do have some additional options. So for Azure and GCP um as a storage credential you can also configure um an IM role to connect readonly access to S3 buckets. Um as you might manage imagine that might create some you know egress costs. So another option that you have if you need to do crosscloud connects is configuring cloudflare uh credentials so that you can connect to R2 storage uh which avoids those data egress fees.
And then finally, um, you do have the ability to configure a connection to your DBF, uh, DBFS root location. Um, this is a legacy configuration. You're going to need a real good reason to be doing this. One of which might be if you're in the midst of a migration from Legacy Hive Metaur to Unity Catalog. U, but make sure to really plan through that option. All right. Okay. So in addition to storage credentials, you also have service credentials where storage credentials allow you to connect to cloud object storage.
Uh service credentials allow you to connect to your um cloud service uh cloud services. Uh so this might be like AWS glue or an AWS secrets manager, that kind of thing. Um and then lastly, you have connections. So we looked at cloud object storage, cloud services. Um, connections allow you to configure a connection to some upstream source system like warehouses like an Oracle SQL server. Um, typically things that are set up for federation.

9:25 Advanced data connections

Okay, let's take a quick look at kind of visually how this is all connected so far. You have your Unity catalog metas store um, which you're able to create storage credentials in to connect to your cloud object storage. You can use one storage credential for multiple external locations like we said, but you can also configure additional storage credentials that might connect to other cloud storage um services. You can also configure a connection.
That connection is tied directly to whatever your upstream source system is. And then you also have your service credentials which are uh the connection to your cloud services. All right. So, we have our connection set up. Now, we want to set up our catalog. Um, typically those um storage credentials are at the lowest level. Maybe those are a one-time configuration. And you can use your um external location for many different cataloges.

10:24 Catalog and asset organization

So, when you're configuring a catalog, this is the top layer of your threelevel namespace, catalog, schema, assets. Um, and this is where all of your data and AI assets are going to be contained. Um, so you can do u manage storage here if you like. Um, connected to your external location. Within your catalog, you have schemas. Uh, and then within your schemas, you have all of your data and AI assets. And we'll get started taking a look at volumes and tables. volumes are just going to be that abstraction layer on top of those external locations that allow you to give more fine grained access to subdirectories within your uh cloud storage account.
So we talked a lot about storage locations. Uh let's just take a look at how you might think about inheritance of where your OB uh data is going to be written out to. So at the very top in your meta store you have the option to configure a uh default uh external location for where your data should be written to. Um if you configure your uh option here then your cataloges and schemas if there's no other external location defined are going to inherit that storage location.

11:24 Volumes and tables deep dive

So they'll be written out all your data will be written out to whatever external location is for your meta store. uh if you define you have the option to override this and you and if you don't define your external location at the metas store level then you will be required to define one for each catalog that you create. Um and then if you uh don't overwrite this any schemas within that catalog the data will be written out to the storage location of the catalog but you do also have the option to override this one more time at the schema level.
Um so within your catalog if you want to have different areas for each schema you can um define your storage location with your schema. As a best practice um it's a recommendation to not configure the storage location at the meta store level and instead configure it for each catalog. This typically aligns with like the physical isolation requirements and the um access differences that you have as you're setting up your cataloges.
Uh so we took that uh storage route all the way through to schemas, but you do have one more layer where you can continue to override where your data will be written out to. Um and this is at the volume and table level. And so for this you can create either managed that means it's inheriting the uh location from this schema. Um otherwise unmanaged or external volumes or external tables that's when you're defining the storage location as you're creating either the volume or the table.
Um for managed tables there's a key difference here. If you drop a table it's going to also clean up all of the underlying data for you. And with manage tables, you also have the option to take advantage of our optimization, which handles all of the uh analyze, vacuum, and optimize operations on top of your tables to keep them performant. So, as a best practice, we would say lead with manage tables for simplicity and performance.
And so everything that we've looked at so far is kind of standard cataloges and assets, but remember we set up that connection before. And so that's what gives us our foreign cataloges. So you create a foreign catalog instead of using um you're creating it with your uh connection that you set up and this will provide federated access to your upstream systems. So when we put all this together, we're extending out from those external locations.

14:01 Foreign catalog federation

Up there you can see the catalog configured with our um external location. And schema one is inheriting all of the uh data writeouts from our catalog. But you can see schema 2, we're overriding the location with external location two there. And then you can see our connection is connected to our foreign catalog. All right, let's get into a quick demo and take a look at what all this looks like in the platform. So, I'm currently in a data bricks workspace and you can see I'm at my home base which is my catalog here.

14:57 Connecting to data demo

Um, so within the catalog uh in this workspace I'm a meta store admin so I have access to everything and we're going to start with external data at the top here. Within external data we can see we can configure our credentials. So when we create a credential here um because this is an AWS workspace I have access to create the AWS and Cloudflare credential types um I can give my credential a name. This is just the internal reference name for my uh for me to see within the list here.
I provide my IM ro and then a comment about what this is used for. I already have a credential created. And one thing that I just want to call out here is this credential um follows the same permissioning structure as every other single asset that you have within Unity Catalog. So every object under management within Unity Catalog, you'll see this permissions tab where you'll get the grant options where you can explicitly assign permissions for users to be able to perform actions with that object.

15:50 Catalog and volume demo

So if your u metas store admin creates a a storage credential that it wants multiple users maybe across business units to be able to use to define their own ex locations um they're going to have to grant that user the ability to create external location with that storage credential. So once we grant that permission um now we can go instead of credentials to our external locations and here I can create an external location directly within the UI.
Um but I can also create this programmatically. Uh so I can use a SQL statement to create an external location give it a name reference my uh storage credential that I defined previously. Um, and then this will create that external location directly within uh my catalog. Once I have my external location, then I'm off to the races. I can create my catalog using the u managed location that I have for that external location.
I can create a schema by leveraging that inherited location. So because I'm not defining a location for this uh schema here, it's going to inherit the location from the catalog. And then finally, I can even overwrite that once more. So I can define a volume location. Maybe I already have some data that has been landed from upstream systems within my uh cloud object storage that I want to be able to read to kick off my data engineering pipelines.
In that case, I can create an external volume here, define my location, and just give it a comment that this is my volume for all of my landed data for my files for ingestion. And just to prove to you that all of this works, when I go to my uh catalog here, we can see the catalog I created, my sales schema here, and here's my landing volume um with my data that I can browse through. So just like you can browse through your data in the actual cloud object storage, you can browse through those files here.

18:03 Table management demo

The difference is now you can assign permissions on top of that data for reading these files and all of these uh interactions with those files areable auditable through Unity catalog system tables. So, if I have my volume location and this data sitting there, the next step that I want to do is to be able to create tables. You have a lot of different options here. Um, you will need to grant users the ability to read your volume in order to read that data and process it into tables.
Um, but this is just one example. I have a a lakeflow pipeline here that's reading from my volume locations, processing that data, and merging it in to create dimension tables. So, if we take a look at my tables here, um I have that raw table that we just created as well as my dimension table. All right. Um one last piece here is we talked about the connections and federation. So, within my um catalog here, I actually have a legacy table, a hive meta store table.
It's this line item table here that I want to be able to join against my data sets. But I also want to be able to interact with this table as if it were part of Unity catalog. I want to be able to get all the lineage, all the additional assets. So I could go ahead and upgrade this table, which you should absolutely do. We'll get into migration in a moment. Um but we you can also do is um create a connection to your internal hive metas store and do hive metas store federation.
Uh so we'll take a look at how we can do that here. So if I go back to my catalog explorer the same place we created the credentials and the external locations we have connections. So when I create a connection, I can give it a name and then I can choose my connection type here. And one of the options in addition to all those other warehouses and services that you can connect to is hive meta store. Um you can connect to AWS glue or external hive meta stores, but you also have this option for your internal hive meta store.

20:14 Lineage and federation demo

Um which allows you to set up this connection. I've already created a connection. So once you have that um you can come back to your uh catalog here and instead of going to external data we'll go to add data and create a catalog and instead of standard we'll just click foreign and then we can search for um the connection that we've set up for our internal hive meta store. Once we have that um connection set up we can go ahead and access that uh table here.
So this is my um federated Hive meta store. I can search for that same schema that I saw um within the uh legacy Hive meta store. Um and when I access this table, I'll be able to see all the same rich information as any other Unity catalog table. And then users can continue to interact with this. I can go into my uh SQL editor. I can select from this hive metas store using the three-level namespace just as I do from my regular tables here and I can create a table and now when I go to that table here if I view the uh lineage I'll be able to see the complete lineage including my uh federated hive meta store um which goes all the way down to our column level lineage here.
Oh, all right. So, we've created a couple assets here. Yeah, created a couple assets. I don't do this. Okay. Yeah. All right. So, we're back. So, discovery for adoption. We want to take a look at how you can classify and organize different types of data. Uh what permissions are needed for data discovery and how you can able enable your users to discover your data. So we already took a look at the lineage that you have.

22:56 Data discovery and lineage

Um this is going to give you upstream views across all your cataloges and schemas. Um but you also want to uh configure access controls for your users. Uh so centralized access governance is managed across every single object here um where you'll be able to grant privileges to different types of uh principles whether those are users, groups or service principles um on all of your different objects. One of those permissions is the browse permission which can be applied at the catalog clean room and external location level.
Um, the browse permission is great because when you apply this at the catalog level, it will allow your consumers to see all of the um, metadata about assets within your catalog, but they won't be able to select or query against the underlying data. So, this is a great way for your users to be able to discover the assets that are already created while still governing the access request flow. And one way that we're supporting uh governance of the access request flow is with requests for access.
Um you can configure destinations of where those requests should go and then um you can approve or deny requests from your users. So this allows your users to selfservice how they're interacting with the how they're discovering data assets and how they're gaining access to them. once they um for discovery we have contextaware search and discovery so users can use natural language um like where can I find sales data um it will use all of the metadata information like the description about the tables tags thing to surface up the assets that are relevant for those users to help you with this process and make discovery easier for your end users we do have AI generated documentation uh so for your assets both at the table and the column level.

25:06 Governed tags and comments

You can generate um descriptions based off of the schema of your table and then you can edit them from there. As a best practice, curate your tags and comments as early as possible. Um this information is used not only for discovery but it also supports the data bicks AI powered features uh which include things like your AIBI genie spaces. Um one of our features now that we have in beta is governed tags. So as tags become more critical for um you know auditing, resource management, access control, we want to make sure that we have clear uh framework for who is able to create tags, who is able to modify the values of those tags and who is able to apply those tags to different types of assets within your meta store.
Um so tags like I said they'll be used for discovery. They can also be used for attribution. So for your um jobs, if you add tags to your jobs, you'll be able to view that uh usage and query against it within your system table. Um but they're also going to be used for governance. So if you create like a PII tag with different categories of PII data, this will allow you to apply governance rules to the data within Unity catalog.
And one key way that we do this is with functions. Uh so functions are another object under management within Unity catalog. Uh they're created in SQL native syntax but that can be a router wrapper around Python as well as different types of AI functions. Um but the key function that you have is these rowle filtering and column level masking. So all of the logic that you'll have for keeping your uh fine grain security uh applied to your tables.

26:42 Row and column security

So the syntax is creating a function and then the logic you want to apply. So in this example um we see two functions here. One for uh only showing specific rows and another for masking our sensitive column data. So the first thing that we do in this function is we're testing for group membership. If you're not a member of the admin group then you shouldn't be able to see the data. And then um if you're not a member then we set up the uh governance.
So in the left hand side um we're doing row filtering and on the right hand side we're defining logic for our column masking. And then we can use the alter table statement to apply these functions directly to our tables. Um but this can get quite cumbersome. You're probably not going to have one table with PII data. you'll probably have many tables with PII data that you want to apply the same function across all of them and that's where those govern tags come in um with attribute-based access control.
So, attribute-based access control allows you to define a governance policy that can be applied at the catalog and schema level um for you to leverage those govern tags and apply your functions, those security functions to your assets. Um, this feature is currently in preview. Um, but we'll take a look at how this works. Anyone with manage permission um can apply these policies to your Unity catalog assets. So let's take a look at the demo here.
Oh yes. Okay. So I'm currently in my uh catalog explorer here. And one of the first things that we want to do is we want to create our govern tags. So up here we have um tag policies. And if I go into tag policies, I can create a tag key here uh like PII. I can give it a description uh like governance for PII data and then I can define my allowed values like address um you know ID, maybe social security number. So whatever you want to apply here.

28:44 Security implementation demo

Once we have this configured um just like with every other object we have that permissions tab and we can grant our permission sets across our end users. Uh so for a particular type of user we can allow them manage where they can manage and govern the uh values for that tag. Um or we can allow them to assign that tag to different types of objects that we have. But in order to use this with a policy, we want to create a function.
Uh so here's a notebook. Just like we were creating tables, we can create a function within that same schema that we were working in. And this is a function that's going to mask an address. So if we're not a member of the HR admin group, uh we want to go ahead and replace the actual address with the masked address. And if you look at the groups that I'm a member of here, you'll see I'm not a member of the HR admin group.
Um, and before right now, we've created the function, but we haven't applied it anywhere. Um, so here, if I look at this customer dim table, um, I have these addresses here, and I want to be able to mask them. So, here's my function directly within my catalog that I registered here. And so I'm going to go to my uh catalog at the top and within my catalog I can create policies. Um so here we can create a new policy. We can give it a name.
We can apply this policy to all account users. So every user who tries to interact with this asset, they have to go through the policy first. And then we can set up the type of policy that we want, whether that's a row filter or a column mask. And here we're applying our policy to our catalog that we have. And now we can select our function. So that same function that we just created and registered, we can select. And then we can choose um when the column has a tag value of that PII uh data that we set up.
I think that was yeah here with a value of address applied to it. Um then we want to apply that security function. So I've already created a policy here. Uh it does the same thing leverages that masked function applies it where the PII um data exists. So if we look at my uh sales schema here and if we go to my customer dim table which has the customer address here you can see that I've added the UCPI address tag and when I go to my sample data now when we take a look we can see customer address has been fully masked.
Um, so this will no longer show that data and now I didn't have to use the alter statement. Anywhere that tag is applied, the policy automatically goes into effect. Oops. Yeah. All right. So, I think that's about it for this demo. I will pass it over to now. Okay. Cool. Thank you. And okay so has walked us through how do we set up all the foundational data assets right so downstream unavoidably probably we're going to go on to different BI and AI workloads and as platform admins it's good for you to know that what are some of the new variety of uh securable objects from Unity catalog that's going to emerge for this BI and AI workloads.

32:51 BI and AI asset overview

So the type of question you're going to ask probably are like okay what are some of the um considerations around all all my BI assets and now I'm have a new persona like data scientist group I have to deal with and then what are some of the security measures around geni applications. So we're going to iterate through all the different BI and AI asset one by one. So first is metric views. So this is a BI feature that's in preview.
What it does is it helps you uh define really complex um complicated business logic into reusable metrics. So imagine that now you can standardize the definition of revenue at your organization across teams and you define it once you can use it in multiple downstream AIBI workloads. Things like uh write it in a query refer it to in a query in a dashboard in a genace. Genispace by the way is our um out of the box text to SQL agent and I will also say that because it helps you manage the data quality help your teams reduce all the duplicated queries they have to write because now it's one single definition of metric right so I will also say it's a governance feature and something I want really want to add is uh I think uh from this morning's keynote from anthropy also alluded to this is that in the ji world data governance is actually really really relevant just because you uh it should be on top of your mind to prime your data state in a good in a clean streamlined state so that in the future you're prepping for a near future where it's not only human users who are interacting or consuming your data assets but also agentic applications and the more streamline clean your data state is um it's even though we have all the optimization on the agent application side to improve the performance of the agent itself.
But if you have a really clean data state, it's going to help the final performance of the agentic application as well. So data governance is only going to be more relevant in a geni world. Uh and also if you want to learn more about metric views, we have a session tomorrow uh on that. I'll pause two seconds so you can take a photo. Okay. So that's the BI side of uh feature. And if now we're talking about traditional ML workloads, so your data science team, they also probably want to create all these features.

35:48 ML feature tables

And the good thing that Unity catalog make it easy for platform admins to deal with uh feature assets is that our approach for feature table is really um it's just a delta table with a primary key. So in terms of governance perspective as an admin you don't need to do anything extra to manage a feature table. So later on we'll show that the permission around a feature table is identical as if you are managing a delta table and also uh I will ask all the platform admins to evangelize to your data science team that it is actually a good governance practice to use feature table because we have built in a feature store with mlflow.
So MLflow if everybody is familiar MLflow is kind of like the model catalog or the uni unity catalog for your machine learning models right so today um Unity catalog actually integrates with MLflow and I will talk more in that as well so when you use feature table it has really tight integration with MLflow so that you are actually bundling your features and your models together during training and inference time and then you will be able to see all the lineage and we'll show later as well.
So feature table deal with your structured data inputs right and now as we know that with all the innovation around genai we it it's become more effective that we can actually deal with unstructured text because today your user they can have uh say say again use the announcement we have in the morning uh agent bricks to say ingest say uh a directory of pdf files of unstructured uh text. Once you have a delta table of already parsed text, you can directly create a vector search index on top of that and this is using our vector search product.

37:15 Vector search indexes

One thing to note that is uh this index is hosted on our compute. That compute is the vector search endpoint. So it's just kind of association of assets there. Okay. Finally, we reach the model object of Unity catalog. And something to know, I already talked about how Unity catalog now acts as your MLflow model registry. And that's why when you register a model, previously all the information around MLflow on all the experiment level run level uh metrics information that today you can access that from unit catalog UI.

38:07 Model registry and MLflow

Another thing to note is that when other teams come to you to say okay I have a model today in data bricks we have expanded the definition of model. So traditionally you probably think model is say like a a secular model and later on we move into deep learning. So we have transformer models and today your team may be creating agentic applications and malflow support this uh agentic flavor as well. So that when you see it's a model in unity catalog it could be a agentic model as well. uh but we have because all the permissioning governance around whether it's a agent application whether it's psycharn uh model it should be the same right so we have abstracted everything into just model so to make all the governance flow easier okay so functions again so this is not a duplicate slides I know already talked about this in the previous session where in function you can define arbitrary uh block of code log logic in SQL or Python.

39:21 Tool calling functions

But in the context of building agents, uh probably two calling is a term that has come up is when agents know that hey we define all this task now we can trigger this tool to call uh to perform some additional action and usually you probably want to all this tool calling to interact with all the different external systems and this is also make it really easy where if you interact with an external system you need some sort of uh authorization right oh sorry authentication Right.
So um what walked through before in the demo for the UC connection that's actually where you can actually govern your credentials. Say in this case we have example of sending a message to Slack. So all the tokens and credential for Slack that can be uh registered through UC connection and then for when you're defining uh pinging a rest API to Slack that's where you can wrap that logic in a function that refers to a pre-established UC connection.
Also to note that if you are building agents that's calling out to internal data bricks uh products like vector search model serving and even genie today you can do that. So use genie as a sub aent that type of situation we already built in some uh APIs. So it's directly uh a function use as a function. Okay. When we talk about ML ops we can't run away to think about okay not it's not just about input of the model but also output of the model right.

40:58 Inference tables and MLOps

So today um there are multiple ways to consume a model. So you can do it in a batch inference manner. So you are probably already defining some target uh payload table uh sorry prediction table downstream and for inference table this is a case where you are hosting your model to uh serverless model serving data bricks model serving and you have the option to enable inference table what we're doing behind the scene is we are automatically persisting this payloads into a inference table and again it's um just another delta table so for admins the permission model it shouldn't be anything different and especially it's automatically persisted as like underscore payload table so if you scroll through all your UC assets you see all the payload table those are generated from a model serving endpoint okay finally closing the loop on just all the data ops ML ops we also have lakehouse monitoring feature this is a feature that um is not just relevant for ML so because we not only care about the output of the model but also inputs right so today on monitors on any table you can create a monitor there are different modes I can walk you through in the in a in the demo but we have expanded out so you can create on any input table of any pipelines even nonML related you can create a monitor on top of model outputs say if you want to monitor how my model performance uh change over time and today if you are creating an agent application we can also create monitors on top of your agent outputs to see okay what's the uh reliability what's the quality that come out of agent application and the type of assets that's created when you create monitor are metric and profile tables and then on top of that we create a dashboards and how those dashboards are going to propagate is whoever are the owners of the table and the monitors they are going going to create all this and then downstream they're going to grant access to more users on the metric table, profile table and the dashboard.

42:02 Lakehouse monitoring

And we also have a whole session on lakehouse monitoring around data quality. Uh it's also happening tomorrow. Uh take a photo if you want to join that. Okay. So now looking at what we have configured so far in the AI BI area. So let's zoom into all the maroon colored boxes. So under this schema forecasting we have metric views. Again this is a way for your BI users to centralize their definition for a business metrics and then your data scientists may be creating feature tables and again feature table is just another delta table with a primary key and it can be bundle up with a model.

43:38 AI and BI demo

So all the lineage will be traced and then on the unstructured side you have your documents table. These are tables uh you you'll see in a demo very soon so I won't go over it. And then on top of that you can create vector index which were hosted on a vector search endpoint. That's where the your end user is going to consume from uh semantic search. And then you can have models and this model again can be traditional ML model, Asian model, uh transformer models and you can also define functions.
These are tool calls uh that can interact with external systems when your agent needs to be. And then the last one is inference table. This is when your model hosted on a real time model serving endpoint. Those are the automatically persisted payload tables. Okay, with that I'm going to go to the demo screen. Okay, cool. So I created this uh schema for all the AIBI assets and actually if we collapse it you you'll able to see that it's just really convenient for Unity catalog to have all the pretty uh heterogeneous type of assets under one so you can see them all at one place.

45:05 Metric and feature demo

So if we were to start uh I have a table here with all the customer service data uh like transaction level and likely my BI analyst want to create metric views on top and how you can do that well one option is to go from the source table and just directly create metric view on top. So if we already pre-created this metric view and we can just zoom in to see um how is the definition like so there are two concepts for sure the session on metric view will zoom into this just so very high level intro here is dimensions are all the columns are very likely frequently used when you do like a select column or filter by column.
So these are the column name and they map to the concept of userdefined dimensions and then for all the actual aggregate logic that's um stored in different measures. So um the metadata that's exposed is all this actual name of the measures but the actual calculation logic is stored there and as you can see that uh you can actually from here use this metric view in all the uh different sets of workflows downstream. Okay so that's more on the BI side.
We already talked about that um feature store table. So you already see that there is a primary key there. So by definition it's already a feature table and I already pre-trained uh using this data set to train a model. So you can actually see some lineage here where this is the feature table we are we're looking at and downstream of course there are some feature engineering logic and in real life from this table to the feature store is likely to be like a a batch ETL job that compute all this feature engineering logic and then downstream since I already log to ML flow training this model with this feature table so that's why UC understands that there is a lineage here.
And uh yeah, that's it for the feature model lineage. And just to note that um other than catalog, we recently also have this tab for feature discovery. So this actually hosts all the feature tables that UC is aware of and you can again do all the search. So this is where multiple data science team can come here and search for feature sets that's relevant to them. Okay, going back to the catalog, uh that we talk about the structured data sets input, right?

47:38 Model and tool demo

And what if you have some data sets like this, these are all the products. If we were to expand, it's all this um unstructured text in your in your table. And traditionally, people might do a lot of reax and try to extract uh meaningful informations. But today with vector search you can directly create a vector search index on top of this table to specify this is a column I want to embed into vector and to be able to vector search later and I already precreated this index.
Um as I mentioned before this index actually have a lot of associated assets. First this is a source table that's created on and then this is the endpoint the compute that this index is hosted on and uh if you were to say query index right that's the endpoint we're quering for uh semantic search and then these pipelines are just uh behind the scene automatically generated pipeline because we are you are able to do multiple different sync mode for the delta table and the vector uh search index Okay, so that's all the appstream data side.
So finally, let's let's take a look at the actual model. So uh for users who have interacted with MLflow before, this interface should look pretty similar because for each registered model, you'll have multiple versions of runs or experiments. And we can zoom into one to see um this is the source run it was generated. uh we can go there first and this takes you to now MLflow. So all the metrics that traditional was tracked by MLflow you can find them here and also that's also why I was saying MLflow was acting like a UC for models.

49:26 Agent and monitoring demo

Uh so this is all the artifact is tracking and that's why MLflow is able to package that and serve uh models to an endpoint very easily. Okay. So if we were to quickly go back, I also want to show you the lineage for the from the model's perspective. So because model is kind of different type of asset, right? So you'll be able to see that hey, this is a notebook that this model was locked and upstream this is a feature store, a feature table that it was trended on.
And then downstream we served this model on the serving endpoint and because I turned on the inference table feature on the serving endpoint the downstream payload are persisted. Remember the suffix of payload tables. Uh and because all these upstream and downstream assets they are ultimately just delta tables. Again um I can go in here to illustrate. You will see that the permission for a feature table similar for a payload table.
This is identical with if you are to uh grant permission on an actual table. Okay. So that's traditional ML models, right? And we talk about for agentic models, they may want to do uh something more complicated. So for example here I've defined a function to slack uh post messages to slack and here I'm defining in SQL and of course uh just like you see function before you can do uh basically wrap it around like a python logic as well and here I'm referring to this connection which is pre-registered with unity catalog as a UC connection and so this makes it really easy for AI developers that all the credentials are predefined you can uh navigated through Unity catalog and I have also created a very simple agent that use this function and again you see that even though this is a agent model the interface look exactly like the same as if it were like a traditional ML model.

51:58 Agent tracing and monitoring

So if I were to go inside into versioning, um something that is just a little different for agent models and probably this more interesting for AI developers is that we also persist all these traces. So these traces are really important when it comes to develop agent uh applications because here I'm just triggering one function but uh in a complicated agent application you're triggering a lot of functions. what is going in and what's coming out of each function is really valuable for debugging.
So those information are persisted here and we could also see from the lineage standpoint that uh it knows that okay here is just one function but when you have a lot of tool calls that's where uh UC is aware that the model uh is encompassing all these different functions. Okay, we just have one more asset left which is a monitors and I'll show you how do you normally create that. So since I mentioned you can actually create monitors on any table.
So I'm just going to choose any table and this under the quality tab this is where you can start creating a monitor on that table. And as I mentioned there are different profile types. Basically you will choose as a user um say if it's just a normal upstream data table I'll probably use a snapshot just to see let's calculate all this statistical uh distribution of my columns over time. So that's a snapshot mode. And if you're creating a monitor on like a model output table you probably want to measure okay how does this model performance change over time.
So that's the inference mode and then if you have time series specific data that's the time series profile you want to create and just quickly show you how does a monitor actually looks like. This is a pre-created monitor and the asset you have is all these profile metrics and drift metrics and on top of that this dashboard is automatically created. Okay, it takes some time to load. And just want to show you like right now because I don't have many time span of the data but this is where um you'll very likely uh encounter into is that you are probably the owner of this dashboard but you want to share it with a a wider audience.
So today we support embedded credential mode. That means even to share with a user that's not within this workspace, you can embed the credentials so that they can assume the privilege as the dashboard publisher. So uh be able to access this dashboard. Okay. So that's all the AIB asset that we have introduced uh to show you in action if we were to go back to okay is that this is actually not done yet, right? because the last layer consumption you probably want to expose that to say a dashboard a genie space which again is our text to SQL agent or an app.

55:18 AI app architecture

So very quickly iterating through each is that for dashboards uh we already talked about if you want to share it to a wider audience that are not the audience within this workspace uh we support embedded credentials and for genie spaces uh it's really a text to SQL agent that means even though the interface is that you are asking question natural language to uh ask about your structured data but behind the scene is just executing the queries on a SQL warehouse.
So everything you have predefined on the Unity catalog side on tables though Genie is going to respect that. So that's also very neat when uh if you are to create your text to SQL agent from scratch from another external system uh it won't embed the permission model as tightly. And then lastly is data bricks apps. Uh we I think we also announced a lot of uh feature updates on apps where now we support probably almost every Python app framework out there like streamllet uh flask and for app it can be a little more complicated because there are more things you can do or more components inter you can interact within the app.
So I'm actually going to walk you through a scenario. Say we're building a AIB app like a agent app. So in this case I have all the product metadata. So very likely your AIBI users are say business analysts and product managers and they want to be able to ask one interface or one app all these questions or perform all these tasks. So they can ask questions like which products are returned the most. And as we all know this really translate into a task that's like typical product analytics right so you're likely want to perform some SQL queries on your structured data and then the source of truth are already in your unit catalog right it already exists in a metric view in a feature table so that's how the breakown breakdown is and then user can also ask question like what's the return policy on this product category and the task that's like a general knowledge uh QA task.
So that the source the uh the source of truth likely exists in a vector search table index and then finally they probably want to perform some actions say send post all these insights to slack channel and the source of truth of all the slack connections those also pre-exist in UC already. So that's a a UC function and behind the scene this is the architecture of that right you probably have a agent app sit directly in front the user and then behind the scene you have your genie spaces um then genie spaces runs on a SQL warehouse it knows where to execute those generated SQL queries and then when you uh want to perform like a tool calling action on Slack That's where you're predefined the function two calling function also executed on your SQL warehouse and then finally if you're asking questions that can only have answer exist in unstructured data source that goes to a vector search endpoint and then all the model logic orchestration logic lies in a model and again it's an expanded definition of model and uh as admins now you probably are a little worried because okay now I have this really complex permission dependency graph how am I going to manage it the good news is a lot of these permissioning are already predefined in UC right and then what we have done is through uh database agent framework and of course there are other session covering this in details as well but what it does is this agent framework creates a system service principle and it's going to check so if I'm creating this app it's going to check if jilling has all the necessary permissions to all these dependent resources.

59:09 Governance and observability

So these resources are um the UC resources on the left but also all the workspace assets in the middle. So it checks if if the creator has all this uh access to this dependency resources, it's going to create this service principle that has the minimum level of permission to all the resources and then that's why when you create this through agent framework, it's going to automatically grant you the right access to all the dependent resources.
So I've so you can uh have select on this table you can have execute on this function and model. Um something that's also in preview feature is we're working on because there are scenarios you probably want to differentiate the permissions through to users right because they probably have different permission around different structured tables. So we're also going to release on behalf of user support. So that's where by default we're going to have system service principle have minimum permission to everything every every dependent resources but then you can specify on behalf of user to down scope for a certain sensitive resource you can uh let the service principle assume the credential of the user.
Um okay so now that finished all the section on AIBI uh I'll let go through um organization visibility. All right this is our like last section so we're we're coming to the end and then we'll get into some questions. Um okay so some key questions for uh observability and uh organization. Um, how do you monitor not just usage and costs to make sure you're staying on budget, but how do you audit the actions that users are performing on all of your Unity catalog objects?
Um, you might also want to consider from an organization perspective, uh, if you have, I think I saw there are quite a few metas store admins in here, you're probably supporting many different business units that all have their own initiatives, maybe have their own budgets, maybe have their own timelines that want to manage their own objects. So, how do we organize all of the cataloges within our meta store to support some of that like decentralized ownership?

1:01:52 System tables and budgets

Yeah, thank you. Uh okay so for auditing the first thing that we have is lakehouse observability with system tables. So system tables are going to be created automatically for you. Um and these track all of the information uh in and all of the actions performed on your Unity catalog objects. Um so this allows you to understand not just the lineage of how those objects are created but the um audit for who's interacting with them from an access perspective as well as the billing information from usage.
And one more uh session alert um today a bit later we have the data bricks observability session on uh using system tables to monitor and manage your data bricks instance. Um so for these system tables we've basically curated the data model for you for you to be able to um query against those tables using SQL to do all the auditing but we've also given you an outofthe-box AIBI cost dashboard. Um so this will sit directly on top of your system tables and allow you to very quickly um start understanding how your assets are being used.
In addition to that, we also allow you to create and monitor budgets. Uh so budgets can be applied to a subset of workspaces. They can also be applied using tags. Um and this allows you to track usage based off of those workspace uh workspaces and or tags um across your business units or projects to see if they're staying on budget. Okay. So we'll do a quick demo just to see how this all connects within the workspace.

1:03:38 Observability demo

So to create the actual um usage dashboard, you have to be a meta store admin. So I'm currently in my account console. I'm no longer in my workspace. And in the account console, you have this usage tab where you'll be able to click this setup dash dashboard option and you can select a dashboard or select a workspace where you want to import that out of the box usage dashboard. Once you import it, it's going to look something like this.
Uh so you'll have a selection across your workspaces. You'll be able to filter across um time frames of what your usage is across all these different types of uh like origin products. You also get a view by tag uh to be able to see what the usage cost is um across those tags here. Uh so for our budget and environment tags, how is our what does our spend look like? And then one of the most useful ones um from my perspective is this last group here which is the top spending.
Uh so it takes a lot of time to do optimization on top of your workloads and you want to make sure that the time that you're investing actually pays off in terms of the reduced costs that you have. So this section here shows you across all of the different types of resources that you have what are the top end most expensive resources. Um so right now it's also showing me kind of like all the rest. Uh but here we're able to see by job ID.
You could do this by job name, by endpoint ID. Um how the spend of these jobs are changing over time. Uh so here we can see this huge spike from this job here. That's probably a place where you want to invest time to do some optimization. Maybe somebody left something running, maybe somebody misconfigured something. So that would be a great starting point. So that's the usage dashboard. Um again in the account console under usage um we have the budgets preview and here's where you'll be able to define a budget.
You can give it a name what the total amount is and then you can define your associated workspaces where we're going to acrue and aggregate all the usage um or key value pairs and then you can define an email address to get alerts about consumption against this budget. All right. So we'll wrap up with uh just a few more slides here around isolation features. Okay. So in Unity catalog we talked a bit about the admin roles and delegation.

1:06:07 Organization best practices

You have your metas store admin and then at the top there you can apply different owners for different cataloges. Um for each catalog you also have the option for workspace to catalog binding. So your meta store spans workspaces, but you might only want users who have uh like read and write privilege to be able to read your production catalog from your dev workspace. You don't want them to be able to write there. Uh so workspace catalog binding allows you to limit the workspaces that your catalog is accessible to um and how they can interact with that catalog.
And then storage isolation is if you have like physical isolation requirements um for your cataloges, you can define different locations across your cataloges. Uh so you might use a cat a particular storage location maybe for an entire catalog or maybe just for a schema within a catalog. Um and this gives you that physical isolation as well as logical isolation. Um, now let's take a look at you have all these different cataloges.
Let's think about the dimensions of data that you have. So if you're curating data and AI assets, they'll probably go through the typical development life cycle. So you have your environment, dev, QA, production. Um, when you're talking about data, you probably also have validation. So like your medallion architecture, bronze, silver, gold. Doesn't have to be bronze, silver, gold. It could be layer 1, 2, 3, N, whatever it is. and then your business units.
Uh so maybe your central IP team is managing a different collection of assets from your HR team who's again managing a different collection of assets from your research and development team. So how do we think about organizing these cataloges? You could take a one-dimension approach. We create a single catalog across each environment uh like dev, QA, and prod where your central IT team is maybe managing this At that point, you might create uh schemas at the project level and maybe you delegate ownership of those schemas to particular business groups.

1:08:23 Data sharing and marketplace

Um in that case, what you'll probably end up with is within that project schema, that business group is going to have endtoend objects for that project within that schema. Right? So you still kind of have those three dimensions. Um this is just if we're using one dimension at the top level. You could also take an approach where you're applying the data validation layer at the schema layer and then you wouldn't uh you'd be able to separate that across the schemas.
You could also take a two-level approach at the catalog level. So in this example, we have both environment and validation. Uh so dev bronze, dev silver, dev gold. And then that would mean that you have the same three cataloges across your higher environments QA and production. So that same uh like dev gold catalog we'd mirror with uh prodold in production. But what this allows you to do is remember that browse permission I talked about.
Um here we can apply the browse permission on the prod gold catalog which means our consumers can discover all of these assets that are validated assets for consumption and we can lock down those um upstream cataloges the silver and bronze. Maybe they shouldn't be accessed by users. We certainly don't want them to access bronze. That's that's how we get bad metrics, right? um maybe the silver catalog should only be accessible by your data scientists.
Um but you know this gives you a way to kind of really simply allow them to discover the assets that matter to them rather than having an end user discover assets that may be across the entire data validation structure. And in the last example we can take a look at a hybrid approach. Uh so maybe your central IT team is doing the heavy lifting for data ingestion from your upstream source systems which makes sense. A lot of times in larger organizations they're the ones with the access to all the upstream systems.
So they're configuring the connections and you might have a metadata framework to ingest all that lower level data. Um so they might have a setup like this where their cataloges are all two-dimension. Um but then for your downstream consumers and business units uh you might set up a separate catalog for them where they have browse permission on that prod gold catalog to consume the validated assets and then the catalog that they manage they're able to set up their own project schemas however they like and they might have some smaller sources that they're ingesting so they might still have their own like end to end uh view within their project schemas.
But this gives them a little bit of autonomy without sacrificing uh the discovery of validated assets that they should be consuming. These are just a few examples though. Um as a best practice, generally separate cataloges by responsibility. It'll probably make your lives easier from a governance perspective. Um, use workspace catalog binding where required, but do what's right for you. Use the minimum amount of cataloges that you need to support your business and physical isolation requirements.
So once you have all these different cataloges, we can take a look at kind of like a practical permissioning example. Uh, so here we have cataloges and schemas that are created by our central team. and the central team is the owner. Uh but you see there in that production catalog, they're actually giving use access and maybe browse to the other business units. Those other business units could have their own cataloges where they're the owner.

1:12:06 Conclusion and resources

Um and they're giving use or browse privilege back to central IT or to other uh business units they're sharing to. And this can kind of go on for however many business units you have within your organization. Um, especially in larger organizations, you might have your data bricks deployments across clouds, maybe an AWS, GCP, and Azure. Um, or you might have them across regions like a AMIA team and a North America team.
In that case, um, delta sharing is going to be a tool in your toolkit to be able to share your data globally. Uh, so delta sharing is powered by Unity catalog. It allows you to share assets without replication across clouds across regions. Um, but it also allows you to share with non-databicks users. So you of course can do data bricks to data brick sharing. Um, but the data consumer can be any compatible client. And so here's just a look at that kind of multicloud deployment that we were talking about.
Um you're able to see that delta sharing can be used across cloud regions for your data bricks workspaces. It can also be used across cloud providers. Uh but it could also even be used across companies. Um as a best practice though keep your like if you have dev QA and prod cataloges keep them all in the same region meta store. Um and you know those business unit cataloges if they're all if they have their own devqa prod as well um and use del uh data bricks to data bricks delta sharing between your cloud providers and regions.
Uh two more things that are powered by delta sharing is data bricks marketplace. Um this is where data providers can put forward different assets that's not just like data tables but also models uh notebooks accelerators for data consumers to consume. So you might be able to find some augmented data that you want to incorporate into your BI and AI projects that you can just add as a delta shared catalog directly in Unity catalog from the marketplace.
And then finally we have clean rooms. Clean rooms are also powered by uh delta sharing. Um it's a privacy ccentric collaboration tool. Uh so two companies if you want to share data uh you can do that using delta sharing into the clean room. Um and then you can mutually approve the code that will be executed um for your data science or BI logic. All right. So, a couple places to go from here. Um, you can try out all this code.
We've got hands-on tutorials that you can pip install into your workspaces um on governance specifically as well as the end toend flow for creating all of these assets. And then if you are currently um in a legacy datab bricks workspace with a hive meta store, we have the UCX tool which allows you to do an audit of all of the assets that you have in hive metas store and provide recommendations on how to upgrade those assets as well as your groups um into the account console and your uh uh Unity catalog meta store.
And then finally, Unity catalog is open source. Uh so you can learn more at the website. Um we have LinkedIn and Slack community. Um but if you've built out all your governance in data bricks, you can keep your governance even if you're not using data bricks. Um so you can use open-source CC to consume all of your data bricks assets that you've governed in Unity catalog. All right, thank you for sticking with us. Um, we'll open it up for about 13 minutes of questions if you uh have any.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.