Unity Catalog 101: Unified Governance for data, AI and applications

Summary

Unity Catalog is the unified governance solution on the Databricks Data and AI platform that enables organizations to discover trusted data, apply consistent access policies, and govern AI assets — including models, MCP tools, and agents — from a single control plane.
Governance capabilities span automated PII detection, fine-grained attribute-based access control, AI-powered catalog curation, and data quality monitoring, covering structured tables, unstructured data, notebooks, and AI assets.
Data lineage tracking works across all asset types, and AI-powered spend governance helps organizations manage the growing cost impact of proliferating agents and agentic workloads across the data estate.

Unity Catalog 101: Unified Governance for data, AI and applications

As organizations build AI-powered applications and autonomous agents, governance becomes more complex. Data, models and tools must be discovered, understood and securely accessed across teams, platforms and engines, yet governance is often fragmented across catalogs, security systems and AI tooling. Unity Catalog is the unified governance solution for all AI agents, applications and data on the Databricks Platform. It enables organizations to discover trusted data with shared business context, apply consistent governance policies across data and AI assets and automate governance workflows using built-in intelligence, all while governing open data formats and assets across clouds and platforms. In this introductory session, we’ll walk through how Unity Catalog works and how it enables three foundational capabilities: discovery with business context, unified AI governance and AI-powered automation. You’ll learn how organizations use Unity Catalog to simplify governance, reduce data and AI sprawl and build trusted AI systems on an open and governed data foundation.
Talk By: Liv Sun, Associate Product Manager, Databricks ; Murt Neemuchwala, Product Manager, Databricks ;

Chapters

FAQs

What is Unity Catalog?

Unity Catalog is Databricks' unified governance solution for all data, AI models, agents, and applications on the Databricks Data and AI platform. It provides a consistent way to discover, access, audit, and govern assets — from Delta tables and unstructured files to AI models, MCP servers, and agentic tools — across clouds and engines.

What types of assets can Unity Catalog govern?

Unity Catalog governs structured tables, unstructured data like PDFs and images, notebooks, AI models, skills, MCP tools, and agents within a single catalog. This breadth of coverage ensures consistent access control and lineage tracking regardless of the type of data consumer or the format of the data being governed.

How does Unity Catalog support AI governance?

Unity Catalog governs AI assets — including models, MCP servers, and agent tools — as securable objects with the same access control and audit capabilities used for data assets. It also provides automated PII detection, fine-grained attribute-based policies, and AI-powered spend governance to manage the growing complexity of agentic workloads.

What is the business cost of fragmented governance without Unity Catalog?

Fragmented governance — where different catalogs and security systems are used for different teams and data types — creates collaboration barriers, compliance gaps, and unmanageable maintenance overhead as data estates grow. Unity Catalog addresses this by providing a single governed layer that scales consistently across users, agents, and data modalities without multiplying per-persona catalog configurations.

Full transcript

[00:07] Thank you for spending your late afternoon with us. If you don't know anything about Unity Catalog, then you're in the right place. Woo! Yeah. Um, I'm Liv. I'm a product manager for Unity Catalog and I'm joined by Mert
[00:19] here, also a product manager. Um, I'm thrilled to be sharing the stage with him because when I first joined Data Bricks, Mert gave me the 101 on the product. So, it's an honor. what's on the docket today. So, we'll start with
[00:32] Unity Catalog's origin story and then we'll move towards some Unity Catalog features that you can leverage to simplify your governance and then we'll make sure to sprinkle in plenty of demos throughout.
[00:44] Okay, you've seen this a million times already. To summarize, we might be talking about features that haven't quite hit your laptops yet. So, just a heads up. And if you really enjoyed this talk, make sure to uh let us know in the
[00:57] surveys. Okay, so your enterprise probably manages hundreds and thousands of data assets and on top of that you likely have
[01:11] thousands of users, right? These could be your data engineers, your ML scientists as well as business users, right? So you can think of um your analysts or your clients. Finally, as of this past year, you're
[01:24] now also managing hundreds of agents, and those are notoriously difficult to manage as well. Now, your assets aren't just tables. Uh, you have unstructured data, right? So,
[01:36] that could be a PDF. It could be maybe your branding assets, just to name a few. And you probably have a bunch of AI assets as well, right? Models, MCPs,
[01:48] skills, tools. You have all of these data types, not just tables, and you want to govern them all. Um, anyone know this TV show? It's one of my favorites. Yeah, it's Severance.
[02:02] And I'm showing you this because you don't want your data estate to look like this. You likely maybe in this case have several cataloges, right? You have um data engineers with one catalog
[02:15] solution, analysts with another catalog solution goes on and each catalog has its own separate governance layer on top of that. And when data lives in different cataloges, the problem is that
[02:28] data teams can't work collaboratively together. Uh they're severed. So don't get mad at me for this pseudo math here. Um, but let's just imagine if you take hundreds and thousands of data
[02:41] assets of various types, you take your thousands of users, uh, both business and technical, you take your hundreds of agents and your catalog per persona, then what are we left with? You're left
[02:53] with a massive governance headache. Yeah. So, if we take it back to five years ago, we've been talking to customers for quite a long time. Um, and this was a
[03:06] huge pain point for customers. Here's our co-founder Mate. He was presenting this exact problem five years ago at the data and AI summit. Essentially, governance is really complex when you
[03:18] have so much data. Okay? And governance problems don't stay governance problems. They quietly transform into your business. So, what does that mean exactly? I'll give you a
[03:31] couple examples here. So, we talked about fragmented governance and some examples of what that could turn into. One example is you might not pass an audit because a policy isn't truly
[03:44] global across all your cataloges. Or maybe you have crossunctional teams working on a crossf functional project, but they're all working on different versions of the same table.
[03:56] Second, say you don't have flexible access policy uh access control or maybe you have incomplete metadata. So pay say maybe you don't have tags that are properly describing your assets. So the
[04:09] business problem there is that employees are chasing down the right person in order to grant them access to the right asset, wasting a lot of time. or maybe you're a new of a new hireer and they don't know whether to use table A, B, or
[04:22] C because they don't know which one's trusted, right? And then the third problem here is when you don't have intelligent governance tooling built into your catalog, then you're going to struggle with this exponentially growing
[04:35] estate. And why would you want to be so focused on governance when you could be focused on innovation? Okay, so Unity catalog, we pioneered this uh back in 2021 and this is the
[04:49] solution for the modern enterprises for solving the modern enterprises governance problems. It governs every type of agent or sorry every type of asset as shown on the left and for every type of persona shown on the right that
[05:03] includes the AI agent as both a consumer and an asset to govern. Now with agents in the mix, governance is of the utmost importance, right? So
[05:15] those agents we talked about, they're probably one of the top consumers of your data. And the challenge with them is this, right? They don't have a sense of what's right versus wrong. Left to their own devices, they're a threat to your security. But with good governance,
[05:29] you can give them the right context and instructions and safely use them to your advantage. Um, and so with all that context, I'll let Mer walk us through how you can precisely use Unity Catalog to govern
[05:41] your data in AI. >> Cool. Thanks, Lib. Um, so in today's talk, we'll cover a few different chapters for what makes up governance in Unity Catalog. So we'll start out by
[05:55] talking about um unified data and AI assets. So how you can bring these all into one place uh and unify them under a single permission model. Then we'll talk about the second chapter context rich discovery um which is
[06:07] around how unity catalog allows you to leverage uh defined business semantics so both your users and agents can find the data that they need and have the context to be able to use it in the way it's intended. And then third we'll be
[06:21] talking about AI powered governance controls and how intelligent tooling is built into UC to do things like data classification quality monitoring and general observability over your entire data estate. Cool.
[06:33] Let's dive right into chapter one. So the first bit of unifying your governance in one place is making all of your data available in one place. And so that means having a single catalog for all your structured tabular assets in
[06:47] the data format of your choice on the open data lake whether that's iceberg or delta or part and being able to have that alongside all your unstructured data. So that can include PDFs and uh
[06:59] images and uh JSON files uh and then all your AI tooling uh that's becoming more and more prevalent as of late. And so this will allow you to create a single source of truth of all your data no matter where it lives. But rather than
[07:12] tell you about it, let me just give you a quick walkthrough of what this looks like in practice. So, I'll switch over to the Unity catalog, catalog explorer, and I'm going to zoom into a production catalog I have here. And I have a few
[07:25] different schemas. And so, I'm going to go into my default schema. And you'll see I have various tabular assets here in Delta and iceberg format. Uh, I have my views that I've built on top of these tables, my business metrics, which we
[07:38] call metric views. Uh, more on that a little bit later. I also have my unstructured data uh stored in volumes in UC. So these are things like your PDFs, images, even uh video files can be
[07:51] uploaded and governed here in one place. Um and then uh other ML assets like functions, models um and all of these are under a single permission model uh
[08:04] that you can enforce at the schema or even at the catalog level. So you can delegate your data science team can access these alongside your data analysts and your other personas in the organization. What you'll also see is that I have a
[08:17] bunch of what we call federated assets. So from different sources like I have my BigQuery data um I have data in my AWS Glue catalog. I can bring that in here through connectors. Um as well as my
[08:29] operational data from all the databases. Um and so I'll switch back over to the slides. And so what you saw there is powered by what we call lakehouse federation. And so this is our uh
[08:41] offering within Unity catalog that allows you to build a unified view of your data state no matter where that data lives. And so whether it's in a SAS application or whether it's an external database system, you don't have to copy
[08:53] this data in order to govern it in UC and make it discoverable for all your different personas. And there's two flavors of lakehouse federation. The first is query federation. So this is for an external
[09:05] database system that has its own compute and um if you register these uh with unity catalog whenever your consumers discover and query them we'll push down the query to the source uh in order to efficiently run it in that compute. And
[09:19] the other flavor is catalog federation. So this is if you're maintaining either a legacy hive um in your own kind of implementation or you may be using one of the popular managed hives like AWS glue um you can directly federate that
[09:32] entire metas store into unity catalog um and this is really useful both of these kinds of federations for interoperability um because you know your data estate may be fragmented you probably have a lot of different um not
[09:44] only um database systems but maybe multiple different cataloging tools as well and And so speaking of interoperability, openness is a big theme that we really pride ourselves on with Unity Catalog.
[09:57] And what does openness mean to us? So the first is you should be able to access all of your data um no matter which uh system or tool it's in using any engine of your choice. So you shouldn't just be locked into data
[10:09] bricks uh compute. Second is you should be able to define your access policies once in Unity catalog and these should be able to be enforced for all your data consumers no matter where that data lives. And then three, you shouldn't have to
[10:21] copy your data and create redundant copies just to give your ML team access versus giving your BI team access. So you can store it in one copy in the open data format of your choice and it's all governed within Unity Catalog.
[10:35] So the components of an open lakehouse. Um the first is compute. Uh Unity catalog allows you to use the query engine of your choice. You can use duct db, trino, flink, even open source spark. Um, and still have all your data
[10:48] uh governed with unity catalog access policies. Um, and these are all stored in the open table format of your choice. Um, as part of the announcement this week, so you may have heard uh we announced uh the newest spec of iceberg
[11:00] as well as um external access to the managed iceberg tables within Unity catalog with any engine of your choice.
[11:12] Another critical bit of openness is open data sharing. So many of you probably work for enterprises that have either partnerships or relationships with other companies either as suppliers or vendors and you may want to share data for various use cases where you can combine
[11:24] this data in order to do some kind of analytics or ML. But crossorganization data sharing is typically very complex and expensive. Um most vendors will typically lock you in like if you choose
[11:36] their sharing. Um it's it's typically not an open protocol. Um it leads to replication or having to if you're trying to share across vendors having to replicate data. It's also hard to balance privacy and flexibility uh to
[11:50] keep your data in sync when you're sharing it with uh another organization as you make changes in the source and having it reflected in the destination. And thirdly, data sharing is typically not optimized for AI. So you might have
[12:04] your tabular data assets, but you also probably have notebooks, um, other AI uh, models or, uh, tools that leverage this data, and you want to be able to share those alongside your data, uh, when you're doing this kind of
[12:16] crossorganization data sharing. And so, Unity catalog solution to this is what we call the open sharing protocol. Uh this was formerly known as delta sharing but we've since expanded this uh to give first class support to iceberg entities
[12:29] as well. Um and so the great thing about open sharing is it allows you to share not only your structured data sets but also your um uh notebooks and AI models that use this data as well as your unstructured data. And you can share
[12:41] these across clouds and ac across regions without having to replicate these and creating additional copies. And you can share this data directly from where it lives, which is really powerful because you can federate data
[12:53] into data bricks and then share it this external data that lives elsewhere using Unity catalogs uh governance controls and the open sharing protocol. Um and so this allows you to share openly to your
[13:05] partner of your choosing. And so these other organizations don't even need to be on data bricks as long as they're one of the compatible clients for the open sharing protocol which includes a lot of leading firms.
[13:17] And the open sharing protocol also powers uh the data bricks marketplace uh which is really exciting because the data bricks marketplace has now become the largest and fastest growing data marketplace in the world. Uh we have over 280 providers with new partners
[13:30] being added every day. And so this is really useful if you may not have established relationships with other organizations but you may want to get access to common data sets like for you know climate or financial data that many
[13:42] of these vendors make available and monetize. Um and similarly you can monetize your own organizations data by making it available in the marketplace for other folks to discover and find.
[13:56] Cool. So we talked uh a bit about unifying these data and AI assets. So we'll jump into the second chapter uh around contextrich discovery. So now that you've made all your data available in one place, another
[14:08] challenge that arises as your data state grows is finding what's relevant based on your use case and your persona becomes harder and harder. And this is increasingly not just a people problem. It's also an agent problem. um these
[14:22] agents with access to all these federated data sources in one place might not know what to use and how to use it. And so one of the tools we have to make this really easy is uh with the
[14:35] introduction of discover page and domains in Unity catalog. So you can browse your data um the similar way to how your business operates. So rather than technical cataloges and schemas, you can make these data available to
[14:47] your consumers by business domain, subdomains, um your consumers can find what's relevant to them based on personalized recommendations. And these are all powered by um things like c a c certification which is done by data
[15:00] stewards uh as well as curation and then also the system being able to see what are the popularly queried assets by users of um that are in a similar uh persona as you based on things like group membership um and similar kinds of
[15:14] asset query patterns. And so let me just jump in to show you what this looks like in practice. So I'm back in a data bricks workspace and so typically um you'll see like
[15:27] things in the catalog schema hierarchy here in the catalog explorer. The discover page is an entirely new uh kind of internal marketplace experience. And so you'll see here that all the data is organized by business domains. So I have
[15:40] my finance and revenue data um as well as my fraud and risk data, marketing and growth. Um and this has been curated by my data stewards. So, I'm going to click into a domain, and you'll see that there's additional subdomains here
[15:53] around revenue management and sales performance. Um, and there's kind of a a glossery here with kind of terms around what our uh core functions of this domain are. And so, when I click into one of these subdomains, you'll see that
[16:06] there's various different asset types like dashboards and tables. Uh, I can see my admins have certified um this product revenue forecast dashboard, so I know I can trust it. I can also see the tables that are available um and their
[16:18] owners and it also gives me an indication of like this has been queried by other folks in the last few days. So it tells me this is the right uh revenue table to look at when I'm trying to uh derive some metrics
[16:31] and so this discover page experience is going to be going um going to be publicly available in this coming week. Cool. So um once you've made all this data discoverable the kind of second
[16:44] order problem that arises is being able to get the right insights from your data and so for example you may have a question like what is my revenue per active customer and there's a lot of implicit questions that you have
[16:56] whenever you kind of are trying to get that insight which is how is revenue defined in your organization um what does active customers mean which tables or views should be used in order to get like the right definition of this business metric so that's really a big
[17:09] challenge that we with a lot of our customers and getting the right number is hard. Um so like we're trying to figure out what your um you might be looking at certain fact tables. There's a bunch of data sources
[17:22] and depending on how you query the data, you might get different numbers. So analyst one might get one number for ARR another on a different day querying it uh might get a different one and the AI
[17:34] agent that's querying it might get a separate one because it looks at uh different data or does a different kind of aggregation on your business logic and so our solution to that in Unity catalog is metric views which are now
[17:47] generally available and metric views allow you to make your business metrics a first class asset in Unity catalog so you can define your metrics in one place and then govern them and no matter which interface through our AIBI
[18:00] dashboards, our Genie or any other SQL client or notebooks access these they'll have a centralized definition of this. And so getting started with metric views is really easy. So you just pick your
[18:13] sources. So these are the tables or views that your uh contain your raw data. You can pick measures which are basically your aggregations or KPIs to encode your business logic. So thing things like how you define sales or
[18:25] churn within your organization and then defining dimensions which are basically filters or groupings so you can slice and dice this data. So you can see like revenue per product line or per region. Um and metric views define this in like
[18:37] a YAML format that's super simple and you can help uh use Genie to author these. Um and the great thing about metric views is that it allows you to define a metric once and answer many different kinds of business questions.
[18:49] So you don't have to kind of define a new metric view for each different region or for each different product line. You can mix and match these measurements dimensions um which allows slicing and filtering without having to
[19:01] replicate this business logic in multiple places. And you can also certify these metrics. So um your data stewards can kind of add that added level of trust for all your consumers to trust them. Um and like with anything in data bricks we want optimal performance.
[19:13] So we offer materialization which we recently released with these so that if you have frequent queries these can be precomputed and so you have really low latency for getting um uh your latest business metrics
[19:33] and like with everything else in Unity catalog we believe in openness. So um our metric views are open with a whole host of partners um in our uh semantic foundation. So this partner ecosystem allows you to define the metric views in
[19:45] Unity and tools like Tableau, um, Monte Carlo are all able to access that business logic. So you just have to define and govern once and make it's available to all these different tools that you may use at your organization.
[19:58] Cool. So we talked a lot about business semantics and um, being able to offer data discovery via domains. Another key bit of metadata context that we believe is really crucial to Unity Catalog is
[20:10] lineage metadata. So, a lot of you might be familiar with data lineage in terms of, you know, powering uh compliance, being able to prove that sensitive data isn't spreading in your organization, isn't being used by people that
[20:23] shouldn't have access to it. Uh it's also really useful for workflows like impact analysis. So, when you're making schema changes, uh deprecating or masking columns, you can see what all downstream consumers and assets would get impacted. But we think lineage is
[20:36] also really really powerful for agents as a context for discovering data, understanding how it was produced and how it's being consumed so that they know how to use it in the right way.
[20:48] And so um in Unity catalog lineage is completely automated for all the workloads that you run in data bricks. So we capture this down to the column level at runtime on every query no matter the language um and no matter the
[21:01] asset type. So with your structured data or unstructured data, we'll capture the full lineage of where uh how it was produced as well as how it's being consumed in surfaces like Genie and dashboards as well as queries and
[21:13] notebooks.
[21:27] my sales tables. And you'll see there's a lineage tab here, which I'll click into. And what you'll find here is a list of assets. So, I can see my ETL pipeline that helped produce this table uh and a
[21:40] notebook that that pipeline runs. I can also see downstream assets like a Genie agent querying this um a few days ago, as well as a dashboard that relies on this data set. I can also pull up the
[21:52] lineage graph. So I can visualize these asset dependencies and understand how this data was produced um as well as how it was transformed via these notebooks. Um and then any derived tables from this
[22:06] that other data consumers might have created uh as well as business metrics that rely on this. This is really powerful to understand how my data flows and the dependencies and um it goes down to the column level.
[22:18] So I can click into a particular column and see how that field is transformed um or consumed downstream.
[22:32] But a lot in a lot of your organizations um likely your data flow doesn't start or end in data bricks. So you might have data originating from various source systems like SAS applications or operational databases. And it's also likely not just being consumed in our BI
[22:45] surfaces like Genie or dashboards. you probably have external tools uh that are leveraging the data within from Unity Catalog via JDBC connections or whatnot. And so uh this week we're really excited
[22:57] to announce the general availability of external lineage in Unity catalog. And this is really powerful because it allows you to extend your lineage graph beyond just data bricks assets. And so uh the key bits of this are you can
[23:09] register your external data assets as a first class object in Unity catalog and then you can define the relationships down to the column level along with the transformations. Um and so this is you can do this both manually. We have a API
[23:22] and SDK where you can define these external lineage relationships as well as automated connectors. So um if you use any of our LakeFlow managed ingestion connectors, we automatically will bring in these external metadata objects representing the source systems
[23:35] that these um that the data was ingested from and we directly annotate this so that when you open up your lineage graph, you can see the first mile Salesforce object or MySQL table that uh
[23:47] was ingested from in order to produce your bronze table and then your downstream assets. Cool. So we talked about uh the context for discovery in Unity catalog including
[23:59] lineage discover and domains as well as um lineage. Uh and now I'll hand it back over to Liv talk about our third chapter.
[24:13] Thanks M. So to recap, we've unified the assets, we've made them discoverable, and now it's my favorite part, which is the builtin AI intelligence to make governance a lot simpler. So this is chapter three.
[24:27] So you probably already do these jobs today. Um you have a way that you're securing your sensitive data and you're trying to monitor for things like access um uh performance cost and more. So I'm
[24:41] going to show you how you can leverage AI tooling inside Unity Catalog to simplify both of these jobs. And so securing can take a lot of forms, but for the purpose of this 101, we'll focus on detecting columns that contain
[24:54] PII data and then applying a column mask on top of that. So let's take a step back here. Um, it's a real pain to secure PII data today. So first, you shouldn't have to be using
[25:07] reax on table by table and maintaining complicated logic inside your job queries and scripts. And second, that shouldn't have to be on your table owners to be enforcing that logic and
[25:21] writing that logic. That's simply not scalable. And it they also aren't the compliance expert after all. Data bicks solution to this starts with govern tags. What are govern tags?
[25:33] Govern tags are your account level vocabulary for describing your assets. And they can be used for a variety of use cases ranging from costs to certification. But in our example today, I'll show how they can be used for
[25:45] sensitivity classification. Tags are a lot more than just a label, right? They are a key value pair. They are permission uh permission controlled and they are applied via inheritance
[25:57] down our three-level namespace in Unity catalog. And by the way, uh govern tags are just one of the three attribute types that we support today. We also support identity attributes and context
[26:09] attributes which are newly in private preview. So with govern tags they are exactly the primitive that what what makes data classification now possible. Data
[26:22] classification is an AI intelligent PII detection system. This makes sure that you don't have to be using Reax to scour for those social security numbers, those phone numbers, um, or any other types of
[26:35] PII that might be hiding in plain sight in your data estate. So, I have a list of really cool capabilities of data classification. We'll dig in a little bit later uh, once I show you in the live demo.
[26:49] And then after we detected the PII, now we can use those tags to then apply an attribute-based access control policy or Aback policy for short. Aback is
[27:03] responsible for enforcing access based on the presence of attributes including govern tags. I love Aback policies. There's just a lot that you can do with them. Um, for example, it's set and forget. So Mer has
[27:16] mentioned this theme again of you know setting something once and then letting us take care of the rest. For example, it has this inheritance. So you can protect thousands of tables at once. And
[27:28] in addition, I also love that you have a bunch of levers for complete control over how you define your policy. So for example, you can define what scope the policy is applied to. You can define
[27:40] what tags that is particularly matches. Um, and I'm actually just gonna stop explaining and show you live.
[27:55] Okay, back in the catalog explorer, let's say that I'm a retailer and I want to start a marketing campaign. And for this marketing campaign, we need this customer data table that's already in my Unity catalog. But uh before I share it
[28:08] with my marketing team, the addresses and the phone numbers um are not in compliance. I don't want to be exposing that PII. So what I'll go ahead and first do is apply uh enable data
[28:21] classification. I can go into my catalog and right here on the right side is the single toggle for enabling data classification. Now for the purpose of this demo, I've
[28:34] already enabled it. Um it does take a short time to run, but we can immediately see the results here. After data classification has run, um it seems like we've scanned the three tables inside my catalog. Imagine
[28:47] there's probably thousands in a more realistic environment and we've located um we've located the two columns that contain PII and applied the tag location
[28:59] and phone number. So, some pretty cool capabilities here that I'll show off. um you can exclude certain tags if it doesn't follow your organization's best practice right here and it'll learn from that behavior through our smart scanning
[29:13] technology. And the smart scanning technology also uh make sure to only tag the new assets and new tables and new columns. So that way you're not rerunning and wasting compute on the
[29:25] same tables. We also newly support custom classifiers. So you can bring your own govern tags uh in addition to the 40 out of the box classifiers that we already support.
[29:37] Let's now double check that uh it has been successfully applied. So we'll go back to that customer data table. And here we can see the tags are now on our columns. Okay. So the next step as we
[29:52] talked about is enforcing via a policy. So I could just click a new policy here, but the best practice is to do so at the highest possible level. So we'll go back into our catalog and we can add a policy here. New policy. We give you this
[30:06] really handy form where you can just fill in information um about uh including the name and description. So I'll just give it this name and description. For the purpose here, we
[30:19] want to hide the contents inside columns. So I'll apply a column mask. But I do want to feature this grant access. This is a new type of AVAP policy that we newly support um which
[30:31] will enable you to use course grade access control via via tags. I'm going to apply this to all account users excluding just their super admins who have maybe special privileges if
[30:44] they need to see that you know address and phone number information. We're not going to apply any additional conditions on the tables. I just want this to be applied to all tables inside my catalog. But I will make sure that we will apply
[30:56] this policy when it matches the location tag as well as the phone number tag. Cool. And our final step here is to
[31:09] apply a masking function. This is where you get a lot of flexibility here um to determine what you want to replace the original contents with. I want a simple mask. Thankfully, Genie here helps me
[31:21] not have to write SQL. And I c all I need to do is check the syntax to make sure everything looks good to go. I can also verify the SQL in the right-h hand panel and create the policy.
[31:49] just click into our customer data to make sure that the columns are now successfully masked.
[32:01] And there we go. Now my table is ready to be uh more broadly shared.
[32:13] So on to the next job that's quite painful to do by hand, monitoring. We get a lot of questions. Um we've heard a lot of customers want to know have these questions about their data, right? Is my data fresh? Where is most of my AI spend
[32:25] coming from what are my most queried assets? And you can get answers to these questions with the right monitoring intelligence.
[32:37] So to answer those data questions though, we first need to talk about having visibility over our huge data estate. We need high level visibility because otherwise these governance gaps are hiding. So for example, how would you know um if there's a broken pipeline
[32:50] or how would you know when everyone is using the wrong numbers from maybe an old fiscal quarter, right? The key here is observability. So that brings us to like the first monitoring feature that I'll show off.
[33:03] This AI feature is data quality monitoring. It allows you to track a whole suite of data quality indicators. Um that includes freshness, completeness, percent null as well as an
[33:15] overall quality score that anyone can understand. And then you can zoom out as you can see here and you get a holistic view. So we leverage Genie as well as lineage as Mer presented earlier. So you
[33:27] know exactly what the root cause of what went wrong. On top of that, you can prioritize which tables to respond to based on downstream impact. And third, you can configure notifications so that the owners get
[33:41] properly notified if there's a quality problem. Now, um what if you want something even more proactive for the most urgent of problems? You might have heard about
[33:53] Genie Zero Ops agent. This was presented in our keynote speaker. Brand new feature. What does this really look like? In the simplest of terms, essentially what it does is it surfaces
[34:05] a quality problem to you in a notification. For example, in Slack where you already are doing a lot of your day-to-day work. Then we will give you an impact analysis summarized in, you know, natural language.
[34:18] And then we'll suggest a fix. And this is a smart fix because we'll already make sure that it doesn't break any downstream assets and dependencies. We already check for that. And so overall
[34:32] the zero ops agent really automates the maintenance of your data and ML pipelines. This is all built inside UC. These insights already are governed by you know your permissions only. Um and
[34:44] so we're super excited about this. Like I think this is really going to change the game when it comes to data operations. Okay. So we've talked about monitoring data quality. What about all the other
[34:57] things we care about, right? What about monitoring AI, monitoring cost, monitoring performance? Well, we have the governance hub, a single pane of glass across all your workspaces, metas stores, clouds, and you can get insights
[35:11] about data, AI, cost, and performance in one central hub. So, this is really important, right? We talked a lot about scale. With scale, you need a bird's eye view.
[35:24] I'll just show you what this looks like in practice.
[35:40] Okay, we are in the governance hub. In the top left, we can see the four things uh the four types of insights we can get from the account level. Starting with the data tab, just right off the bat, we can see that we govern a lot of assets.
[35:53] Uh 60 million assets across our account. Uh just short of a thousand tags. Now, something I might be curious about is what are my most popular assets like which which tables are being queried a
[36:05] lot and are they properly being governed? So, I'll go ahead and review data usage. Um here you can see these are my top uh tables. Um, I could even
[36:17] filter down into a specific metas store if I wanted. And I also have a time series that looks at usage over time. Now, we're not doing so well on governance. Uh, we don't have data quality monitoring enabled. We have no
[36:29] tags on any of these top assets. Thankfully, this is a demo environment, so we're doing a lot better in in in reality. But don't make this be your enterprise. So, the next thing I'm curious about is
[36:41] AI and cost, right? like how much are we actually spending on AI? So, you've probably heard a little bit about AI gateway. Assuming that all of those assets have been registered in Unity Catalog and AI gateway, you can actually
[36:54] now see your AI spend as well as your AI usage across all your users um and even get breakdowns by model. So, one question I'm particularly curious about is what are my top users? Like what's
[37:07] the average spend per user? and like do I need to add some type of cap, some type of limit because certain people are spending too much. Well, let's take a look. I'll jump into AI usage. Um, right now it's grouped by models. I can uh
[37:21] look at users instead. And immediately I can see in this list, looks like Young Bin and Jean are both spending a lot of money. Um, $5,000 in
[37:35] the past month. So they're really token maxing here. Maybe it's a good thing for enterprise. Maybe you need to add a cap, but that's an additional conversation that maybe would be really hard to glean without this view um from the governance
[37:48] hub. So the last thing I'll show off with governance hub is how to turn these insights into action. So I'm back in the data tab and
[38:00] right, sometimes it's hard to see where those blind spots live. Thankfully, we have data recommendations here on the right s right hand panel. We will scan your data estate and surface
[38:12] recommendations for what to take action on and to to close those governance gaps. Um, say for example this third one, unused assets suggested for deprecation. I've talked to a lot of
[38:24] customers and customers are always telling me, hey, I want to save on storage costs like I want to deprecate assets so I can save some money there. Other customers are telling me, hey,
[38:36] like uh we have duplicates of the same table. Can I ddup and you know deprecate the rep uh the copies? So instead of sorting through all of those tables in your uh in your estate, you can just go
[38:49] ahead and review what we've suggested and take action from there. Okay. So that's governance hub. this bird's eye view um central place to
[39:01] monitor data AI uh and cost. I'm gonna bring it back to Meritt to close us out. >> Cool. Thanks, Liv. Um so to summarize
[39:14] today's talk, um we talked about how with agents in the mix, governance has never been more important to enforce consistent access control. Uh and we covered three different chapters of what makes up governance and Unity catalog.
[39:27] First being unified data and AI assets and bringing them under one governance model regardless of your table format. Number two, putting the right context which includes uh business semantics
[39:39] being able to define these via metric views and then defining your domains and subdomains so that your users can find the data they need and then having the lineage context so that uh agents and users quering this can see how that data
[39:51] was produced and how it's being consumed. And then lastly, we talked about the AI powered governance tooling built directly into the catalog to do things like data classification, access control, uh, and monitoring for cost and
[40:03] for, uh, AI spend. Uh, if you're interested in learning more, we covered a lot of different things here in this L100 session. We have a bunch of deep dive sessions
[40:15] tomorrow um around external data access and how you can connect other compute to Unity catalog around metric views, how exactly they work, how to model these YAML files, a Unity AI gateway uh to
[40:27] govern all your multiAI access across your company, uh open sharing, uh cost performance monitoring, open table formats. So uh feel free to take a photo of this and join us tomorrow for some of
[40:39] these more detailed talks on this topic. Cool. Uh, that's the end of our talk. If you have any questions, uh, we'll stick around here. Feel free to come chat with us. Uh, and thanks so much for taking
[40:51] the time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.