Unity Catalog: Advanced Field-Proven Patterns from the Experts

Summary

• This video covers field-proven Unity Catalog governance patterns for scaling from centralized to federated, domain-driven operating models without creating bottlenecks.
• Presenters demonstrate row- and column-level security, AI-powered metadata curation, and automated table descriptions to accelerate data discovery at scale.
• The video introduces Unity AI Gateway for governing AI agents, MCP servers, and model access as part of an end-to-end AI-ready governance framework.

Unity Catalog: Advanced Field-Proven Patterns from the Experts

Join us for a real-world exploration of Databricks Unity Catalog designed for data leaders scaling their platforms. We will tackle the challenge of governance at scale, demonstrating how to balance centralized management with decentralized, domain-driven operating models. Learn field-tested strategies for catalog organization that establish structured guardrails. These patterns will allow your teams to innovate rapidly without traditional bottlenecks. We will also cover how to accelerate metadata curation to empower true data discovery and a frictionless self-service experience. Leave with flexible best practices for cross-boundary collaboration, secure data sharing, and AI readiness that adapt to your organization's unique architectural needs.
Talk By: Jyotsna Bharadwaj, Senior Solutions Architect, Databricks ; Pamela Pettit, Sr. Solutions Architect, Databricks ;

Chapters

• 00:00 Introduction and session agenda
• 02:02 Catalog organization: centralized vs federated governance models
• 05:11 Multi-dimensional catalog design by zone and business unit
• 13:42 Access control: user vs group grants and grant placement
• 22:14 Row and column security with dynamic view functions
• 37:43 AI-powered metadata flywheel: automated curation at scale
• 46:45 Data discovery demo: AI-generated table descriptions
• 54:04 AI agents as data consumers: new governance requirements
• 61:00 Unity AI Gateway: governing model and MCP server access
• 68:03 Agentic access control: service principals and session-user chaining
• 75:15 Governing MCP servers with Unity AI Gateway
• 83:50 Summary: maturity roadmap and best practices

FAQs

What is Unity Catalog and how does it support governance at scale?

Unity Catalog is Databricks' open governance platform that provides centralized management of data and AI assets. This video shows how to balance centralized control with federated, domain-driven models so teams can innovate without filing tickets or creating bottlenecks.

How do you organize Unity Catalog for large enterprises?

The presenters recommend a multi-dimensional catalog design organized by zones and business units, providing structured guardrails that let domain teams ship governed data products independently. This federated approach reduces the central platform team's workload while maintaining governance standards.

How does Unity AI Gateway govern AI agents and MCP servers?

Unity AI Gateway extends Unity Catalog governance to cover model access, MCP server calls, and agentic workloads. It supports service-principal-based agentic access control and session-user chaining so enterprises can apply the same security framework to AI agents as to data assets.

What are best practices for row- and column-level security in Unity Catalog?

This video covers dynamic view functions for row- and column-level security, allowing fine-grained access policies without duplicating data. Presenters also demonstrate how to place grants at the right level—user versus group—to maintain scalable, auditable access control.

Full transcript

[00:07] Thanks for joining our session today. We're going to go ahead and get started because we've got a lot of content to cover and only 90 minutes to do it. Uh so this session is Unity catalog advanced fieldproven patterns from the
[00:20] experts. I guess that sets the bar pretty high but we hope you guys will find some useful patterns here. Uh there is an expectation coming into this session that you have some knowledge of data bricks unity catalog some of the
[00:32] foundational elements within unity catalog uh because we're really going to be focusing on how to scale all of this and what it looks like to put things into production and what patterns you can think about when you're scaling these things out. Uh I'm Pamela. I'm a
[00:46] senior solution architect with data bricks one of our Unity catalog specialists and I'm joined by my colleague Jets here. >> Yeah. Hi everyone, I'm Jotra Bhadwaj. I'm a senior solutions architect part of
[00:58] the manufacturing vertical and uh welcome to our session. All right, we will go ahead and jump in now. Uh usual forward-looking statement.
[01:10] We're going to be showing some things that are preview uh features, beta features. So, you know, no guarantees. Uh and so this is just a highle view of what we're going to be trying to help you guys uh take away from today's
[01:23] session. How do we kind of organize our people and our objects uh without creating bottlenecks? How do we secure everything without slowing people down? Um how can people discover data assets
[01:37] and AI assets without having to file a ticket? um how do we extend all of the data security we're doing out into AI without rebuilding a whole new security framework and how do we sustain this
[01:50] without having to be on call? How do we make sure everything keeps running? Um and then finally we'll give you some tips at the end to kind of start with exactly what you have. So figure out where you are uh kind of in the spectrum
[02:02] here and then figure out how you can take the next best action. All right. So organize without the bottlenecks. Uh the big question here is can a new domain team ship a governed
[02:16] data product without filing a ticket. Uh can they get everything done without waiting on it here? So failure might be a strong word
[02:28] see or friction points that prevent scale. One is the centralized bottleneck. Right? So if the platform team owns everything, then everything has to go through them and there's
[02:40] eventually going to be a cue of what takes priority. Uh so every change is a ticket and your innovation speed is essentially your approval speed or the execution speed of any of your tickets which is going to be a hamper on your
[02:52] business. The other side is kind of decentralized chaos. um no standards, no consistency, everybody's kind of starting from the same place and tends to become a compliance nightmare. Uh
[03:06] these are extreme ends of the spectrum. Most folks kind of sit somewhere in between um but it's still not quite working. And I think one of the things that allows us to start to answer this
[03:20] question is to start with the foundation before we think of technical implementation. How are we actually organizing our data? What are the dimensions of data that we're talking about here? Because that will inform who
[03:32] needs to participate in the creation, curation, and consumption of that data. So these are a couple examples here. Environment pretty much everybody has, right? Pre-pro, prod, dev, QA,
[03:44] production, validation is another really big one. You might have zones in your medallion architecture or specific phases you're moving your data through as you're kind of curating it for consumption. And then your business
[03:56] units, right? Central it might take in all your source aligned data sets, but you've got business teams like HR and finance that might be closer to the consumption layer that still want to be able to contribute.
[04:08] So if we understand kind of the dimensions at play with our data, then we can think about how to organize everything. So this is an example within Unity catalog of using one dimensional
[04:21] asset at the catalog level. Um so if we have our cataloges separated as dev, QA and prod that means we're putting the emphasis on our schema level or at the
[04:33] table level to address those other dimensions, right? Um this is fine but it can also start to limit the uh capabilities that you can adopt from Unity catalog. One of those is the
[04:47] browse capability. Right? Browse you apply at the catalog level. It allows users to see all of the available assets but not query any of them. So they can know they exist but you can still have the guardrails of whether or not they
[04:59] can uh interact with them. If we use this method, I have no way for users not to discover that bronze schema there or those bronze tables. And so now you start to create some confusion. Should
[05:11] they really be looking at bronze? Is that the right step here? So another example is let's do two dimensions of data maybe environment and val validation at the catalog level. Right?
[05:23] that allows me to isolate my bronze layer, give um all my consumers access to my production gold layer to discover everything with that bronze permission and then even segment out that small group of data scientists that always
[05:36] want the granular raw data and maybe they can get into silver here, right? Um so that that gives us a little bit more organization and then you can also have a hybrid approach, right? is that that two-dimensional approach still kind of
[05:50] gets at lots of people, lots of business units writing out to prod gold which might start to become overloaded. So we can separate out um a hybrid here where
[06:02] we have finance dev and maybe we're creating a separate catalog and because finance likely isn't creating ingestion pipelines from their source systems they're generally probably pretty project aligned and so maybe it's okay
[06:15] for them to be able to create their own version of you know bronze silver gold as they're building out their metrics so they can push the business folder just forward just building on top of still those same assets from prod Right? It's
[06:28] kind of different views. Are we source aligned? Are we derive products? Are we consume products across the layers here? So, you have a lot of different options here on how to implement. But those dimensions certainly play a role and
[06:41] they're going to inform uh what capabilities you can adopt or not adopt and how you're going to scale. The other thing that plays a big role especially when we looked at that last group, right, that finance dev is
[06:54] maturity. So you have some teams that'll fall all the way in that autonomous spectrum, right? So they might have a lot of capabilities. They might be teams that have done a lot of Power SQL usage
[07:07] before. They might be teams that are just maybe a little bit more agile to understanding new technologies or more self-starters. And then you've got teams on the other side that are more assisted. And you know, they just wanted
[07:21] the answer. They don't want to have to figure out the problem, but they still might need to contribute just with a different set of skills. And so we want to be able to for both of these teams um
[07:33] give them a path to collaborate within the guard rails that we've set up. And so in the autonomous approach, maybe that approach we took with finance dev works. We give them manage permissions
[07:46] on their catalog. They're still gated within that catalog. They still can create all of their assets and move at the speed of their business, but all of that is visible to us in our Unity catalog uh system tables. And then on
[07:59] the assisted side, maybe we create one centrally managed catalog and that becomes a sandbox for them to work in. Or maybe we create only cataloges in dev for these users. They can't ever promote
[08:11] anything to prod and they're partnering with the central team to actually do any of that promotion. So it goes through a certification gate, but we're filtering out those autonomous users who can be their own certification gate because
[08:23] we've already verified that they know how to set up CI/CD pipelines. They're using the correct automation users. They understand basic principles of generating good reliable data that they can support in production.
[08:36] So the big thing here is making sure that we have a path for collaboration. Maturity plays a role here. Again, these are probably the two extremes within any
[08:48] organization. You're going to have gray area in between there of what teams are able to support or not support. And so, understanding and having a path forward for that maturity will allow you to better support those teams and have
[09:00] everybody moving quickly.
[09:14] actually like the owner and manager of those cataloges. They're able to verify things going into production. We're giving them a lot of trust to do this, but we're also verifying that what they're putting into production actually
[09:28] is within our our goals. Um, definitely be careful here, too. A lot of teams want may want this permission and not be ready for it. And so making sure that you have clear guard
[09:40] rails and are able to check off like will you actually run and maintain this? Can we verify that what you've put into production is not going to blow out our budget because it's a really weird query that takes forever to run or um are you
[09:53] actually going to maintain the metadata here? Are these just going to be uh schemas that get created and abandoned? like those are kind of people discussions that inform the success of a model like this.
[10:06] The other example is the assisted centralized sandbox. And I'm using sandbox loosely here, right? It doesn't have to be one sandbox catalog. Um, but this gives you a way for users to still
[10:18] have some autonomy. So maybe that might be at the schema level with a first or last name schema within that single catalog. Maybe it's team based or project based. Um and then that gives them a space that contains schema space
[10:32] to create their assets without like uh conflicting or colliding with anybody else. But they're not responsible for any of the promotions. So central IT or your core platform team, core data
[10:44] engineering team, those are the ones that are owning kind of all the cataloges here. And your users or your groups don't have access to anything higher than dev, right? All of that
[10:56] those assets allow them to rapidly prototype, get all the business knowledge on paper for us to experiment with and then fasttrack the handoff to the central IT team so they can actually
[11:08] push that into production with all the right guardrails in place. Um, so generally what we're thinking here is what looks right at 10 tables,
[11:21] uh, might break at 10 10k tables, 10,000 tables. Uh, so if you just have one mega catalog and you have all of this like fan out of business users that want to contribute and collaborate, you're going
[11:34] to end up with hundreds of schemas within that catalog with all different types of dimensionality to them, right? bronze, silver, gold, valid, not valid, which comes a lot of permissions to
[11:46] manage. Um, per team cataloges for immature teams that can lead to that kind of sprawl of the cataloges that you have. So, we have to be judicious of where we create these. Um, naming by
[11:59] technology. Uh, so this really becomes meaningless to consumers. Maybe there's an edge case here where we might do some of this for like very close source align data like right when it comes out of the
[12:11] source. But generally this you want your catalog to reflect how your people work with your data and that will generally allow you to align all of the governance
[12:23] around it to the way your teams work together as well. So it'll give you a little bit of flexibility. Um, and yeah, definitely no giving full catalog ownership to teams that aren't ready for it. That will just lead to no governance
[12:38] or everybody has access to everything and somebody's going to have to come in and eventually fix all of that compliance. So, as an example here, um, all of this
[12:50] information we can track within our system tables. So, one of the big questions that we would have here are what are what are the assets being created? What are stale assets that were
[13:03] created but weren't updated in like the last 90 or 120 days? Because those might be candidates for automatic cleanup, especially in that assisted environment. If you have a lot of stale assets for a
[13:16] autonomous team that you just gave rights to, maybe they did a one-off project and then completely abandoned it. That's also something we probably want to be aware of and clean up. So this dashboard I created with Genie Code
[13:30] right on top of our system tables. It just gives us a view across all of our critical assets we have to review for each catalog as well as our permissions for all. Uh so what our grants are, what are direct
[13:42] user grants versus uh group based grants, um where these grants are being applied, if they're applied at the catalog, schema or table level. Um and
[13:54] then an example of the most active users in the last 120 days. So we can kind of see where is the adoption coming from, what artifacts are they interacting with
[14:06] and how frequently are they interacting with those uh assets. Those are things that it will allow you to actually measure your operating model and see is your platform kind of growing
[14:19] deliberately. Are we creating assets that we continue to maintain that people are interacting with that have permissions that reflect the compliance that we want or um are we creating
[14:31] something that kind of creates a graveyard of abandoned assets that no one will interact with. So you can't scale what you can't see and if you measure all of this you can kind of move forward with confidence here um on what
[14:44] your uh onboarding and adoption and scale plan will be going forward. So lesson number one when we talk about organize centralize the rules. If everybody knows
[14:57] the rules then you can decentralize the execution. Okay. Section two uh secure. Can you add security to 100 new tables in an hour?
[15:09] Uh so can we secure our data without slowing down? And that's not even slowing down the centralized team. But can we secure our data that 10 teams are
[15:21] creating in parallel for other data assets when we have one global policy for what PII means? Uh these are some of the questions that we want to answer here. And to get started, I think the first big thing to understand is tag
[15:35] types because tags are fundamental to the security that you're going to be able to apply at scale across your objects. The top two tag types are the key ones here, but I just wanted to give
[15:48] you a lens into kind of the tag types that exist across data bricks. So you don't you have query tags, right? Those are on your SQL warehouse. So you can attribute different queries on the warehouse. You have usage tags. Those
[16:00] are on your jobs, right? Or the compute that you're running to understand how your pipelines execute. But on the top here, we have free form tags. These are the unregulated tag assets. These are
[16:13] the tags that existed before Unity catalog came about. Um, you don't need any permissions for these. Anybody can create a free form tag and assign it to an object. On the other side, you have
[16:26] Unity catalog govern tags. Govern tags are an uh governable object within Unity catalog. You need permissions to create govern tags. You need permissions to
[16:38] assign govern tags to objects. Um and if a tag is defined as a govern tag, you cannot define it as a free form tag. So there is a guard rail there. Um, but
[16:52] govern tags are the backbone of all of the security that we're going to do going forward. So, we want to make sure everybody's moving towards govern tags and we're slowly decommissioning these free form tags as quickly as we can.
[17:06] But we also want to make sure if we're moving towards govern tags now, we're moving towards kind of a team sport because govern tags are at the Unity catalog meta store level, which means they're at the account level, which
[17:18] means they'll go across all of your cataloges. And if we go back to our dimensions of data, we might have many different teams working on cataloges, creating new data assets that may have
[17:30] to create uh governance policies on top of them. We don't want everybody creating like finance PII, HR PII, supply chain PII, commercial PII. We want one PII because generally at the
[17:43] organizational level, that's likely what the lawyers are going to tell us. Everybody has the same definition of PII. But there are some cases where your business unit team probably has their own definitions. In your marketing
[17:57] segment, they might have a tag for the different types of uh maybe subscriptions that they have with customers to classify things out that maybe influences the data that they share with partners. That would be
[18:10] something that's very specific to your commercial or your marketing team that's separate from like the organizational level. So we need some shared operating model here that allows us to establish those crossbu kind of global tags allows
[18:24] us to align on those domain specific tags and then gives us a way to verify what tags are out there who's creating them and where they're being applied across the organization. So in a perfect
[18:38] world, you know, you might have your meta store admins, which is probably sitting with your central IT team being the owner of allowing people to create tag privilege for the governed tags.
[18:50] Those people that you grant that privilege to might be your satellite champions across your business units. The ones that have some of that stewardship knowledge or understand how the business works, is willing to collaborate with the business to make
[19:03] the definitions. uh they also are the ones that are going to have to adhere to any of the rules of how these govern tags are applied across the assets within their business unit. >> Can I ask a question? I'm sorry.
[19:17] >> I was trying to get to the first section. I'm going to take you back slides to just ask me a question right so you talked about the failure right centralized you know
[19:31] >> your organization obviously I'm sure there are requests coming from various business I want this data I want this to be done how do you prioritize >> yeah so the question was how do you
[19:44] prioritize data in a fan out mode if you've got one team that's responsible for bringing on all this data And you've got lots of teams that want data. And I think we are all aware that with the uh
[19:57] fastpaced kind of leaprog advancements within AI, everybody wants more data. People who were just wanting to submit a question before, they want data now too. And they have business knowledge that
[20:10] can help contribute to the curation of data. And so if that's where that failure mode comes in, if you only have one team doing things, the way you prioritize it is whatever falls in your
[20:22] ticket queue a lot of times or whoever makes the most noise. The way you scale it is by enabling those teams to be able to contribute. Whether that's in a fast prototype or as autonomous teams that
[20:34] can develop and produce their own data assets on top of our source line data assets going forward. Um we will we did block out time for questions at the end. So we will take more questions at the
[20:46] end of the session. Um but we do have a lot more demos to go through. So I want to make sure that we can keep pace.
[20:58] apply the permissions for the tags? What leads are going to actually create and manage those tags? And then how are we going to sprawl out permissions for users to assign those tags and make them
[21:10] aware of it. So this this becomes kind of that trio there for how you scale. So if we got our tags down, tags, like I said before, is the backbone of security. So the other big thing that we
[21:22] have here are row filters and column masks. Um tags will allow you to do your row filtering at scale or your column masking at scale. uh your row filter is going to return a
[21:36] boolean response to see whether or not you can see that that row. Um same thing with the column mask. It's going to give back the transformed row uh if users don't have permission to see that data.
[21:48] Um these are the SQL UDFs that get applied to the table or the column. Um this evaluation here can be per query against the session user. So the
[22:02] identity making the query against whatever object you're looking to get the data out of. Uh there are key built-ins here that we can reference not just the session user but also is account group member as a part of our
[22:14] security function logic. So we can hook into the identities there. Um, and the identity functions here resolve once per query, which means we can scale this out even if you have multiple account group
[22:27] cases that you have to handle for some of the security applications for row filtering or column masking. When we have these security functions, then we can create our attribute-based
[22:40] access control policies. So the policy is the definition that we can create at the catalog, the schema or the table level. And it leverages the security functions, the logic to apply our
[22:54] security to specific identities or exclude our security application from specific identities. And the logic that it applies is the security function that
[23:06] we've registered. So, we want to make sure that if we're going to be creating a policy that applies our row filtering at scale across all of our assets, that these functions are performant
[23:19] that they're going to work across all the data that we're interacting with. Um, so some things we want to make sure we're doing is we want to make sure we have non-correlated um like a subquery that doesn't reference an outer table if
[23:33] we can help it. Um, if we do have to reference an an outer table, let's try and keep it as small as possible. If we can keep it as small as possible, we can broadcast that table, which will keep all of the execution much more
[23:45] performant. Uh, so generally that means kind of like a materialized view rather than a a regular view on top of a streaming table. Um, and as much as we
[23:57] can, we'd love to keep these deterministic. If we can guarantee the same output results from our input, that's going to make our our functions here more performant. For column masking, um, you can actually
[24:10] kind of flip the logic here a little bit. And so some column masking will be matrix based, but some column masking is kind of an artifact of row filtering. Like I
[24:23] can see all of these records because I have permission to this subset of data. And so in those cases, we don't have to do any masking there. That's kind of the else case here. Um, but we can say where
[24:37] we do need the reductions applied. So broad users that have general read access across many rows rather than already getting the specialized filter down roles, they get column masking by default.
[24:51] So some tips and tricks here for scaling your Aback. um leverage those in uh include and accept clauses in your attribute-based access policy as much as
[25:04] possible. Um anything we can move out of our security functions and embed within the Aback policy is going to help us in the long run. If we do have to reference like a mapping table for some of those
[25:16] matrices that don't apply cleanly to your identity groups, make sure that um those tables are small enough to be broadcast. uh when we're querying them against our other tables.
[25:29] Um compose your tags and use those tags effectively with many different policies rather than kind of creating one monster policy that applies to everything with a massive security function.
[25:42] Um test at scale if you can before production as well would be the last one. And then I talked about the to accept principal targeting. Um, so make sure that we're using that as much as
[25:56] possible. So we're gonna get into a little bit of a demo now. I'm gonna jump into the platform and just show you briefly uh what this kind of looks like, what some of the code blocks are and what kind of
[26:08] the order of operations might look like for generating uh this data here. All right. So like I said, our first step is govern tags. we have to know
[26:21] what we want to govern. Uh so we can create govern tags programmatically. Um I'm just doing it in SQL here. You could do this using the Python SDK as well. Um if that's your preference.
[26:33] So here I'm creating three different tags. I'm creating a PII tag for uh name, email, phone, and date of birth. I'm creating a row access tag that will
[26:45] work with my uh country column here for uh country filtering. And then I'm also creating a PII tag for uh sensitivity. So I can see all my govern tags here,
[26:59] verify that they were created and that they have the values that I want as well as when they were created. So again, verification of what's available across the platform and what tags we've created here.
[27:12] Um the next step is actually applying these tags. I just went ahead and manually applied these tags in the code here. You can do this in an automated fashion with data classification. Um,
[27:24] particularly with the new beta feature for custom classifiers in data classification that allows you to step outside of the system govern tags like class address and move into the custom
[27:37] tags that you curate. So one other thing that I'm setting up here is just a really simple access control lookup table. Like I said, for some use cases, you might need that
[27:50] lookup table. We just want to keep it small and effective so that we can see the impact on performance within our query. Uh so I have a simple uh access table here that just shows that I have access to three country codes, US,
[28:05] Canada, and Great Britain maybe.
[28:18] organization. Uh, so the next thing that I want to do is create my row filter. So this is the function that I talked about, right? Um, create a replace function. This registers this function in Unity catalog
[28:31] so that it can be reused wherever. Um, one thing that you could do is you could have like a common assets in enterprise IT that maybe you create a governance schema in and register all your functions there so that you don't have
[28:45] sprawl of your governance functions across every catalog. If you want your autonomous teams to own those, you can do that as well. Um, this is returning a boolean here.
[28:58] I'm telling it that it's going to read SQL data. So, it's making a query out to my mapping table here. And just for the sake of the demonstration, I'm labeling this as deterministic that I'll always get the same output. This is not
[29:10] deterministic because I'm querying a table that could change. I could come back up here and give myself access to France and then the next time I run this query, it's not going to give me the same
[29:22] security results. Right? So technically this would be a little bit more non-deterministic but I want to show you the output of the effect of what this would look like.
[29:34] The next one we have is our masking functions. Uh so I have a function here to mask names. I have a function here to mask emails um as well as phone numbers.
[29:46] And then I also have a mask of a date of birth that's actually just going to remove the month and day and zero in on the year. So not a full mask. Uh so yeah, so these are the the
[29:58] functions that I'm creating here. These are the functions I'm going to reference in my attribute based access policy. Um so in these policies here, the first one I'm creating is this row filter uh table
[30:12] here. You can see I'm applying it to all users within this group. This C3 rose scoped users. It's a horrible name. This is probably not the group that you would have within your Entra identity, right?
[30:25] But for clarity's sake, this this would be the group that we apply it to. And then I've got my accept group. So I've got my C3 admins. They're exempt from any of these filtering rules. They can
[30:38] see all data. Again, in the real world, would our admins be able to see all data? Probably not. They're subject to rules, too. But there's probably somebody that needs to see all data. So, whoever that is, you can accept them.
[30:50] Uh, right here. The big thing that allows us to apply this is we want to apply this for these users or not for these users across all our tables that have this tag, key, and
[31:04] value pair, right? our case, our sensitivity tag and the PII value. Um, so it matches anywhere it has the column with the tag value of our row
[31:18] access and our uh country here. And then using our country column here, that's what we're going to pass into our functions. Uh so that we know the values of the rows of the table we're querying
[31:31] and how to map them against our our uh validation of what the user can actually access. So that's generally the format, right? Create our policy where we want to apply it. And here I'm applying
[31:44] everything at the table level, but you could apply this to a catalog as a whole or a schema as a whole. define the users that the policy should apply to or the users that should be exempt from this policy and then how you identify the
[31:57] table assets where this policy policy should be applied because it's not going to be blanket to everybody, right? It's only where those govern tags ex exist. Okay, so I've got my verification. Now
[32:10] it's just going through and making sure that everything exists. We have our effective policies on our customers table. So these are the things that we set up our column masks and our row
[32:22] filter. Um we can actually see and interact with the definition of these policies describe what's going on just like we would describe a table.
[32:34] Um we can verify the govern tags that are on top of that particular table. So if you go into system information schema, the table tags will allow you to see what tag uh key value pairs are here
[32:48] on this table. And we can do the same thing for our column tags, right? So across all columns within this uh table, we can query against the information schema,
[33:00] see what tags are there. And then we can also um do our sample query here just to see that our tags are working. So I am in the row filter group
[33:12] that we talked about. That means that I'm only going to see GBUSDA here which is happening. And my date of birth here is going to be masked uh because this is going to apply to me here. But I do get
[33:25] to see the email and phone number because the email and phone number only applied to my broad access users. So in my policy, I'm filtering out those different identity groups, who it should
[33:37] apply to, who it shouldn't. I'm standardizing and centralizing my security function once, and making sure that it's uh effective at scale. And the way we can see it's effective at scale is with this explain statement here. So
[33:51] if we've done our job correctly with the security functions that we're using, even in the case of mapping tables, when we explain this query here, we'll see this line in the explanation. This
[34:04] photon secure uh secure view here. This lets us know that it's doing all of the kind of pre-calculation work of what users should see or not see so that we're not actually querying any data
[34:18] that they shouldn't have access to. Um the other piece here is the shows us our um statistics across the tables that are relevant for this query which is not
[34:30] only the customers table but also that mapping table that we have to execute as a part of our security function to find out which regions we have access to. This is important right now. This is
[34:42] missing what's missing the statistics on these tables. Those statistics are super important for you to have performant queries. You can do this, you can get these statistics manually with uh
[34:56] running an analyze command, but you can also get them out of the box if you turn on predictive optimization. That's going to run those analyze u vacuum automatic liquid clustering kind of all of those
[35:08] optimization features for you. Um so this is another important one. You have multiple ways to address this, but those two things influence the performance of your security at scale across your
[35:20] objects. So, one last piece here is once we've got our our all of our setup done, we still have to measure it, right? We have
[35:32] to make sure that users are in compliance with the security we're setting up here. And so, I've created another table here or another dashboard. I didn't create it. Genie code created
[35:44] it. Uh but it gets the job done. It's showing me my tag compliance. So across my tables, where do I have free form tags versus govern tags? Um it shows them at scale here. And it shows those
[35:58] tag operations. So who is actually creating tag assets? Uh who's updating tags to be applied? what are our most frequently applied tags across our
[36:10] organization. And so this gives us a really comprehensive way to see who's applying tags to what objects, what popular tags there are, what the trend is over time, and where we're falling
[36:24] outside of compliance with those uh free form tags instead of our govern tags. So for every step we want to take here, we want to make sure that we're doing that governance check that we're not just setting the rules, we're measuring that
[36:37] people are adhering to them. So the second big lesson here for secure is one policy today protects every table you build tomorrow. And now I will pass it
[36:49] off to Jatna to see how we go beyond data here. Yeah, thank you Pamela. All right. So the next thing that we are going to talk about which is a critical portion of scaling governance in your
[37:01] organization is being able to discover data. So so far we have spoken about uh being able to define permissions, access control, tagging them, classifying them and so on and so forth. But none of this
[37:15] actually guarantees that a new hire in your organization will be able to find data without filing ticket. So discovery and governance these are two parts these
[37:29] are two spectrums of governance. So let's see how we can do that. So this is what I call as the metadata flywheel. So let's walk through the loop
[37:43] of the metadata flywheel starting with curation. So automate the curation process. It should be part of your ingest pipeline and it should not be like a separate activity which is part
[37:57] of your cleanup or part of your project backlog. Make it integral part of your injection pipeline. The second uh turn in this loop is about quality. You
[38:10] cannot undermine the importance of high quality descriptions or metadata because that is what brings the meaningful context to the raw data which helps
[38:22] everyone in your organization find it and also that is what helps your AI to work on your data. The third part of this is discovery. So once you have high
[38:35] quality descriptions around your data, metadata is then there and it accelerate discovery of accurate assets as they are needed.
[38:48] And once the discovery is made easy, what you can do is that everyone in the organization can trust that data and clarity in the data that builds trust in
[39:01] data which lets broader adoption and it truly democratizes the usage of your data foundation and this is what leads to self-service as well. And once
[39:15] self-service is there when people in the organization and even AI they are able to find the data the right data the accurate data the governed data what the the next step of this is that u this is
[39:30] effective self service and there are fewer uh tickets which are then logged freeing up time of your engineering team and they can then focus on more business important things. So your metadata it
[39:44] does more than describe your data. It actually compounds the benefit of your data foundation.
[39:56] So the next let's try and figure out how do you populate the metadata that matters and do that at scale. So there are certain features which are built into the platform as well as there are solution accelerators and custom
[40:09] solutions which you can build. So starting with AI generated comments maybe few of you are already familiar with that these this is a built-in feature using that what you can do is
[40:21] that at the table level as well as at the column level uh you can generate those uh generate the comments the AI under the hood helps you do that but this is one object at a time when it
[40:36] comes to data classification and being able to identify sensitive data. Uh there are classifiers which are available. There are custom classifiers
[40:48] which are available as well as Familia spoke about that. So this is again something which is native to the platform and you can use that. It scans the column values against built-in or custom rules and then it applies those
[41:02] governed tags based on the rules that you configure. This can either be continuous or schedule. And then uh there is a solution accelerator by data data bricks uh
[41:15] engineering team. It is open-source solution accelerator. So you can leverage that. It can run in scheduled manner. Uh the benefit of that is that you can run it uh across uh your
[41:28] catalogs and schemas and scale it more efficiently in comparison to let's say doing uh AI generated comments. And then if you want a completely custom solution, you can do that as well. U as
[41:42] you know inside data bricks we already have foundation model endpoints which are available to you. You can call those model endpoints programmatically and uh generate uh the table context to
[41:55] generate the descriptions and again this can be scaled this can be scheduled. This is a completely custom solution which you can work through. Um so the goal here is that we are not completely
[42:08] eliminating the humans out of the loop. The humans can still review those comments but what is important is that you definitely need that metadata and you do not need to write from scratch.
[42:21] Take advantage of any of these features which are available to you.
[42:33] So once you have uh metadata in uh in place the payoff of that is uh easy discovery and all your raw data scattered data uh which was non-navigable that can become like a
[42:45] structured map which your business teams can easily navigate to and find information and there are three main building blocks of discovery with domains. So what are domains? Uh these
[42:58] are account level groupings and these you can actually map to the way your enterprise or your business works. So uh few examples of uh domains could be finance team, supply chain team, your
[43:12] procurement team, your marketing team and so on and so forth. So these are like groupings and then you can uh put in different assets into uh domains and
[43:24] tag them for easy discovery. The second part of that is certified trust signals. So um inside Unity catalog you can actually tag u different assets and mark
[43:37] them as certified and once they are tagged as certified what happens is that as part of the your search results uh these assets come as on the top of your
[43:49] search results. So that is the second building block. The third building block over here is the whole discover experience. So this is a curated browsing interface across your tables,
[44:02] your dashboards, Genie spaces, apps and so on and so forth which you can actually um experience through Genie 1. So Ali today talked about talked a lot about uh Genie 1. So all your business
[44:16] users can actually use this discover experience. Uh it hides all the complexities for the business users and helps them easily discover these assets. So the bottom line over here is that you
[44:30] govern once. So you define all your principles, governance principle once but then you signal them everywhere and sandbox prototypes they stay visible to the builders your data engineers. uh but
[44:44] then the certified assets once you mark them you tag them as certified they can then surface to business users and that is what builds trust in your data asset estate.
[45:02] So um this is interesting over here. So, Genie as you know this is the natural language agent. Uh using this you can interact in natural language uh with your data and this is where your metadata accuracy matters even more. So
[45:18] let's consider on the left let's say you have unlabeled data you have a column name with the name uh CLV. I myself don't understand what CLV means. So imagine genie trying to understand that
[45:31] without any description over here. So the outcome of unlabelled data over here is uh it generates the wrong uh answer it generates the wrong join maybe it gener it it does a uh wrong aggregation
[45:46] and once the business user or any user for that matter sees that what what do they do? They file a support ticket and then it goes into that entire cycle of people trying to figure out what needs
[45:58] to be done to fix over there. On the right, what we have over here is governed metadata. So again the same column CLV but with better description. CLV stands for customer lifetime value
[46:11] in USD. We are giving some additional description saying that this is sum of all completed orders since account creation. outcome of that genie is able to write accurate query and what you get
[46:26] is uh correct results and the reward of this is that users can selfservice a trusted correct answer instantly no need of filing any ticket over here.
[46:45] So again uh taking the same example of global retail corp that we started with. So global retail crop corp as the name suggests it is a retail corporation which has these different uh uh departments let's say customer and
[46:57] finance supply chain so on and so forth which can easily map to what I introduce you to as domains and what we are going to see over here is that how you can generate uh descriptions on tables
[47:10] columns etc. We'll also see that how you can build a AIBI dashboard on the same data foundation which would respect all the governance principles that we defined on these tables as well as see
[47:23] how Genie space uh can be used with this data. So with that introduction uh let's go and into uh my workspace. What you see
[47:36] over here is a catalog. I have these different schemas like customer 360 uh supply chain and uh finance for example and I have these different tables over
[47:50] here. >> Zoom it slightly more. >> Yeah, sure. >> Is it better now? >> Okay. >> Uh I know it's a little small. We run
[48:02] out of real estate a little bit. We'll describe this and then uh you know some of the big things will be easier to see here. So this is more just centered on where things exist generally.
[48:14] >> Yeah. >> All right. So let's take the example of this table loyalty members. As you can see there is no description on this particular table over here. What I can do is I can use this built-in feature of
[48:27] AI generate. Uh click on that and that generates description for me out of the box. I can read it. I can of course edit it as well. But if it looks good then I can accept it. I can do similarly I can
[48:42] add or generate comments for these different columns as well. So here for example I have points balance without any description. I can again click on generate comment and then uh this inbuilt utility generates comments uh
[48:56] for me even at the column level. So it's as easy as that. if you want to use the in-built uh features over here. The next thing that I introduced was around uh
[49:08] domains. So as part of the discover experience uh let me again try to zoom make it bigger. Okay. So this is the discover experience. What you can do is
[49:22] based on your the way your enterprise works, you can create different domains mapping to your uh different departments. In fact, you can this is a really interesting one which I want to
[49:35] pull up. So this is a domain created for system tables for example in which you can see uh different tables and different assets which are mapped onto
[49:47] it. So this helps you organize and discover your uh data easily. So this is domains. Now let's say on that same data um I
[50:00] want to create a dashboard but I want it to respect the same governance principles which I had set up as part of my let's say column level masking or rowle uh filtering. So because for this
[50:15] particular data set I have a filter set up which says that I can view data only for these two regions. So even my this dashboard respects that governance. Um
[50:29] and also because I have masking set up on my sensitive data as you can see over here it has already masked uh email and I cannot whe
[50:44] whether it is uh by running SQL or whether I'm seeing it in my dashboard I see the same governance principle applied over here and let's say if I want to ask a genie a
[51:00] question over here. So this is the genie which is embedded inside my dashboard and if I ask the question what is the distribution of customers by region. So it should respect the same rowle
[51:13] filtering which was set up as part of my table definition and my policies. Right? So this should give you a perspective that we are defining
[51:25] governance once and it is being used by these different surfaces but the governance is still being respected. So if you see over here it again brings in
[51:37] uh these two regions to which I have access to and it doesn't talk give me the details of other regions like uh uh APAC etc.
[51:49] So the other thing which I wanted to talk about is this interesting dashboard. So as Pamela was talking about defining governance is one thing but being able to monitor it is another
[52:01] thing. So this is a really useful dashboard which you can use as part of your enterprises as well. uh but what it gives you is a complete snapshot of how
[52:14] ready your data is as far as metadata definitions are concerned. So for example it tells me that 30% of my tables have proper descriptions 16 just
[52:28] 16% of uh columns have description how are my PII tag how is my PII tag coverage certification coverage etc right so you can see all of that you can
[52:41] also see that how these different cataloges and schemas are doing in terms of being tagged and covered It also gives me uh trends, trend details and uh
[52:54] table readiness by catalog, column description readiness by catalog, documentation stailness. It also gives me uh freshness and uh how stale or fresh my data is. So this is a really
[53:08] useful uh dashboard using which you can actually monitor and see how well your metadata is defined as far as your data estate is concerned.
[53:31] Okay. Yeah. So this was the dashboard that we just uh saw. So the lesson the third lesson that we just learned over here is that metadata is the difference between self service
[53:44] and a ticket. So you cannot undermine the importance of uh uh classifying and uh writing right descriptions around your data.
[54:04] Next up, so as we all know today, the consumers of your data are no longer just humans or people and uh very soon or probably they already have all the autonomous agents,
[54:17] the AI agents will perhaps outnumber uh humans in terms of data access. So the question that I want to pose over here is that can you trust your AI agents are
[54:32] respecting your excess controls and can you extend that same governance that we so far spoke about for your data can you extend that and make it applicable to
[54:46] your for your AI without rebuilding it so that's the question which we are going to discuss over here but But before I go into that, let's try and
[54:58] understand the anatomy of uh how an AI agent uh works. So let's say a user sends a message to an agent. So the agent uses the LLM any of your
[55:12] foundation models whether it is cloud or Gemini or GPT or any of your custom models. So the agent uses the LLM as a brain. So whenever uh whenever a user
[55:24] request comes in, it uses LLM as a bra as the brain. But the LLM doesn't understand your enterprise data. It has zero context about how your enterprise works. All the LLM understands is
[55:38] because it has been trained on all the data from internet. That is all what it understands. So in order to answer the question being asked by the user what it needs is maybe it needs different MCP
[55:50] tools it pro it could uh need uh your data which is in uh DBSQL data brick SQL because you might have some custom business logic it might need some UC
[56:02] functions either in Python or maybe in uh SQL um it might try to use genie space for natural language interaction even data
[56:14] bricks app could be one of the tools and if you if we are talking about unstructured data that is where vector search comes into play so it could be one of the tools and for OLTP based
[56:26] interaction lakebased could be a tool as well so as you can see over here uh the spectrum becomes really wide and the boundary of an agent is not just confined to one
[56:40] single table the way it cost for simple data queries. In fact, if you see over here, there could be third party services as well which a agent might need to um uh to answer a user's
[56:53] question. So, this is where the real governance gap uh comes in because the way your data queries interact with your data versus how a agent interacts with your
[57:06] data is completely different. So starting with execution a single SQL query it hits your warehouse and it returns the answer. So it it is that simple. But as I was just
[57:18] explaining in the previous slide agentic interactions they are multi-step there are multiple hops in that there are tool calls there is LLM reasoning and there are chained actions. So it is complicated when it comes to identity.
[57:32] So managing the identity is extremely important in case of simple data queries. What happens is that it is user let's say jotna I get authenticated to
[57:44] uh the SQL warehouse and then based on the permissions defined in unity catalog the the query the results of the query are returned but then in case of agents
[57:58] um there are multiple hops and there are so many different stakeholders which are involved in agentic interactions that the identity needs to be forwarded from
[58:10] one system to another and it has to be faithful forwarding. It cannot happen that someone else is impersonating Jotna and trying to get answers uh uh as as
[58:22] needed. So that is about identity and identity propagation is probably one of the biggest and hardest problem to solve. The third thing um the where the
[58:35] governance gap comes into play is boundaries data queries one table maybe multiple tables depending upon your query but then agentic interactions they travels multiple tables external
[58:48] services so on and so forth. So the blast radius is extremely big over here and you cannot confine it within the lakehouse
[59:03] grants you can define policies and they can handle everything but then when it comes to agentic interactions uh the needs are completely different you need more controls so you need to
[59:17] rate limit so there's so much talk happening about token maxing and so on and so forth. So you definitely need rate limiting being able to put those budget controls in place. Then you need guard rails.
[59:30] These agents are interacting with sensitive data and without having proper guardrails both for request and response when it comes to traffic of your AI uh you cannot trust your AI and then you
[59:43] need credential forwarding. And lastly uh in this comparison when it comes to audit for your SQL queries query history suffices you can look at
[59:55] the query history you can see what was asked what was the plan how the data was returned and you can monitor uh your query but when it comes to agentic interaction
[60:07] uh every tool every LLM in invocation needs to be monitored and that is how you can see you can see and you can trace how your agent is performing and
[60:21] learn from that evaluation loop as well. So as you can see there is a vast difference between the way agents work and how the data queries operate. The
[60:33] good news is that these agentic interactions they are built over your data foundation. So leverage that leverage that substrate leverage all the governance that you have defined on top
[60:46] of your data. So data queries inherit inherit that governance automatically and as I said agents do too at that data layer. everything above that layer all these
[61:00] different tools etc and monitoring the traffic that is where you need something additional and this is where unity AI gateway comes into play. I think Ali
[61:12] today briefly introduced that I will give you little bit more insight into how Unity AI gateway works. So there are two parts of um AI
[61:25] governance. uh one is AI asset governance. So all your models, your functions, your connections, your MCP servers, so on and so forth. These are
[61:37] what are the assets and Unity catalog already manages these assets just like you have your tables, your functions, uh your unstructured data. Similarly, you
[61:49] can have all these models, functions, connections also part of your catalog and schemas. So these are securable objects. They are governed with standard
[62:01] UC privileges. For example, grant revoke. Uh you can see complete lineage. You can also see the you can audit all of these assets as well just like any of your data assets. So this is one part of
[62:15] your AI governance. The second part of this is that AI traffic needs to be governed as well. And this is where Unity AI gateway extends the governance which was part of
[62:30] of your um data governance and in as part of the traffic governance you can uh put controls on every request every response which is being uh passed
[62:43] through this particular gateway for LLMs your foundation models even external models you can you can control access you can enforce rate limits. You can track usage and cost across providers.
[62:59] So what this Unity AI gateway does is that it gives you control. It helps you monitor the cost. It helps you with your compliance and it gives you choice. We
[63:12] give you choice. We are model agnostic. So you have choice of using any of the models based on your uh use case whether it is proprietary models or whether custom models or even open source model.
[63:24] So remember those fours and that is where four C's as I call them and this is where uh the AI traffic uh governance is handled by Unity AI gateway and then
[63:36] don't forget MCPS because MCPS for tool interaction this is a standard which was created by Enthropic but then this is a key ingredient of how you set up your uh agents and these are all managed uh
[63:50] through Unity AI gateway. So all in all, Unity AI gateway it gives you that central control plane uh in which you can monitor all your AI assets
[64:04] as well as keep a keep complete inventory of your uh different agents and get observability around that. So, Unity AI gateway uh in conclusion.
[64:18] So, Unity AI gateway extends Unity catalogs data governance to AI resources. The same access control lineage audit that protects your data is now applied to your AI assets and AI
[64:31] traffic. So, this is how the Unity AI gateway fits in. So your uh clients, let's say users, apps or even different agents when they have to interact with your AI
[64:44] backbone, they go through this Unity AI gateway. So every request and response passes through the Unity AI gateway. It helps you permission and rate limit. You
[64:56] have complete payload, payload logging, usage tracking. You can build uh uh AI guardrails for PII reduction. Even custom uh functions you can write for
[65:08] that. You can also define fall back. So let's say uh you have high traffic and Gemini is not able to handle that load. You can maybe fall back on some other model as well. And then you can do
[65:20] traffic splitting. And at the bottom as you can see this is this choice part. So you have choice of using different models u based on your needs. So this is
[65:32] the multiAI governance um which is part of Unity AI gateway. Any model, any agent, any MCP server, all of that gets handled through Unity AI gateway.
[65:44] Okay. So again a few more details around uh what Unity AI gateway offers. It is extending your security to your AI surface. It is one control plane for all
[65:58] your AI traffic. Um you have standard APIs across all providers. So you can swap models uh by conf uh by config using open AI compatible or native
[66:11] provider APIs. So this is again that choice part. It is governance layer not just serving. So don't treat it just like a ser serving
[66:23] layer. You have all these uh inbuilt governance principles in it. You get enterprise observability and cost control in it as well. So there is a built-in crossworkspace dashboard with
[66:36] cost attribution per endpoint, per user, per team, per model based on your needs. You can configure that and we'll see that in a bit. So this is the cost part of it and most importantly it is
[66:50] consistent governance for agents and tools. So one thing which I forgot to highlight is that every enterprise today is using these coding agents. Wipe coding has become a norm and uh you want
[67:05] to be able to control and get complete visibility into your coding agents as well. So that can be handled by Unity AI gateway as well.
[67:22] Yeah. So now that we have understood what Unity AI gateway can do, uh this is how a production ready agent architecture should look like. Uh you have a user asking the question but uh
[67:34] unlike the previous slide which I showed you, every request and response is now going through the AI gateway and that is how you get complete uh governance and observability for for your uh AI as
[67:48] well. Okay, the other important thing which I spoke um in my previous slide was around the identity chain. So most security teams
[68:03] today the question that they have in their mind is that let's say if an autonomous agent queries my enterprise data who is it acting as uh is it using
[68:16] some all powerful uh identity and uh trying to access my data using some backdoor uh bypassing my security
[68:28] policies. So that is the fear which most u security teams have in mind. But with data bricks and the governance foundation that we are talking about you
[68:40] can be confident that that doesn't happen because inside data bricks the regardless of how and where you're using your data the identity propagates. So
[68:54] let's see these three different surfaces which are listed over here and let's see how the identity propagates. Um so if it is a SQL query which you are trying to fire using SQL warehouse. So if it is
[69:08] Alice Alice gets authenticated to a SQL warehouse all the governance and access control um defined as part of Unity catalog gets respected. So the sessions
[69:21] user stays as Alice. So this is how the identity works for uh SQL warehouse based interaction. Genie space which is again an agent but then it is focused
[69:34] more towards SQL queries. Similar to SQL warehouse in this case as well. Alice authenticates to Genie. Uh same identity propagates to SQL warehouse. The UC
[69:47] permissions are respected and the session user again remains as Alice. Now comes our agents and this is where it gets complicated. So agents they have
[70:00] all these different tools but then we are working with AI gateway. So what happens is that Alice authenticates to AI gateway and then through the AI
[70:12] gateway the agent is accessed and then the MCP server. All the permissions and access control defined as part of Unity catalog they are respected and the session user again remains as Alice. So
[70:25] the bottom line is that no matter how many steps are involved, every request resolves back to the real user, the identity is uh propagated and your
[70:38] existing access rules just works. So this is really important especially in context of AI.
[70:51] Um when it comes to which identity to use I'm sure all of you are probably aware of service principle so you can use u so in in case the security is personal or individual then you can use
[71:03] enduser pass through or o based uh and then that would be for example jotsna so the role level filters and all your uh permissions and access control that gets
[71:15] filtered based on jotsna And you should use it when you are let's say building interactive agents or chat bots and so on and so forth. But then uh service principle based um identity management
[71:29] is important as well. And in this case it will filter based on your service principle. You could use it for bad job system to system or machine to machine interaction. But you have both these
[71:42] options. Use them based on your use case. All right. So with that we will again go back into a brief demo of few of these
[71:55] uh features. What we are going to see is we'll see how genie space is exposed as tool. We'll see an external data source, an external MCP and how you can uh define service policies again using uh
[72:09] AI gateway and we will see another interaction ser uh surface using supervisor agent. So with that let me jump into the demo.
[72:25] All right. So as I said all these different assets which are needed for building your agents they can be part of unity catalog. So as you can see over here I have these different unity UC functions
[72:38] which I can use uh when defining my agents they can all be permission controlled. So all these tools my data all of this can exist at as part of my Unity catalog. And next, if in case I
[72:52] want to create an agent, what I can do is um I can just go over here and I can add all of these different tools to my supervisor agent and based on my
[73:06] permission, it will um come up with the answers. So let's say if I ask uh so here if you see I have this particular MCP connection to which I do not have
[73:19] any access. So if I ask it, what is today's weather in San Francisco?
[73:33] There's this table over here. So then because I do not have access over here, it says that it is it it cannot uh it cannot fulfill the request. The other example which I want to show
[73:45] you over here is of another agent which I have and in this case I am using another MCP connection to which I have access as well as I have genie space over here and I have also provided
[73:58] different functions for it to um work through. And in this case, if I ask it, who is my who is the customer with highest
[74:14] loyalty points? Then it should be able to work through my data, use the same uh governance which we had defined as part of our uh data stack and come up with
[74:28] answers. Yeah. So I think it is working through it. But I guess you get the point.
[74:48] Yep. And then if I ask it get me the email of customer ID 4 96. And let's see if it can get that. But because I had defined rules so
[75:02] hopefully it will still mask it. So as you can see over here because there were masking rules which were defined on this particular column regardless of the fact that I'm invoking
[75:15] this uh as an AI agent it is able to respect all those rules. Okay. The other thing which I wanted to show you is with the AI uh Unity AI gateway u
[75:30] what comes is the this is this is a new thing which has been introduced which we call as service policies. So just like at the cot at the catalog level you can
[75:42] define uh aback rules. Similarly for your MCB connections you can define these different service uh policies. So for example here I have a MCP
[75:54] connection. This is my MCP connection. Permissions of course I can define but then in addition to that I can define service policies. And with these service policies what happens is that I can
[76:06] actually control which tools as part of my MCP definition I want to include in this particular service policy. So I can basically uh enable disable them over
[76:18] here. In addition to that, I can define uh policy. So let me just quickly uh click on that. This is where the traffic uh the AI traffic control which I spoke
[76:32] about comes into play. So you can define input guardrails, you can define output guardrails and these are the different types of guardrails you can define. You can also define custom uh functions
[76:45] based on your needs. So here is an example of that in which I have used a custom functions to deny any usage of the tool let's say U research. So this
[76:58] is something which has been recently added to Unity AI gateway and is uh really useful uh in controlling the access to MCP servers. So lastly what I wanted to show you was
[77:12] um uh the Unity AI gateway dashboard. So one thing which I want to highlight is that Unity AI gateway is not workspace specific just so just like your metas
[77:25] store it's it is at a level above. So using this particular dashboard you can track uh you can get a overview of your different workspaces, how many daily
[77:37] requests, daily tokens, how many unique users based on the model uh and the destination type. You can get all of these uh different statistics over here.
[77:49] You can also get stat statistics on your usage. um what was the medium time to first uh bite, median latency, error count, so on and so forth. The these are
[78:03] the usage details. And as I said, even for coding agents uh which every enterprise is today using, you can get insights into that as well and put in your um uh controls.
[78:17] And here for external MCP servers. So for example, this was the one uh which I had used. You can see how many daily requests were there for MC for every MCP servers. Uh the status code of how
[78:30] successful or failed requests were there as well as daily unique MCP servers etc. So at a high level all of this information is part of system tables. Uh
[78:42] this is a dashboard which is which comes out of the box but then of course you can customize it based on your needs. All right.
[78:58] Okay. So, this was the rash word which I just showed you. Uh, so lesson four. So, what did we learn? Security follows the data not the
[79:10] application. And inside our platform for AI, it is not a separate tool. You do not have to bring in a separate uh product. uh security and governance of
[79:22] AI is built in and Unity catalog still powers that. So use that to scale your AI. I'll hand it back to Pamela. Okay, I'll wrap us up real quick here with the
[79:35] last lesson because it kind of compounds everything we just looked at, right? How do we sustain this? Um can our governance still work when our best engineer goes on vacation? Uh we want people to not have to be on call here.
[79:47] So the first big thing is principles of a stand a shared platform. Make sure we standardize once. Uh we're decoupling delivery from all of kind of the shells that we're creating across workspaces and the spaces of data and AI that
[80:00] people are working and that we have documented joint design patterns so people know when a handshake needs to happen. The other big thing is we talked about that maturity curve of autonomous
[80:13] versus assisted users. Um, it's not just data we care about, right? And it's not even just data and AI we care about. In order for assets to truly be meaningful for the business, you also need the code
[80:25] that generates them and any resources layered on top of them. Dashboards, applications, genie chats, agents, all of those kinds of things. So, we need to make sure that we have a common uh
[80:37] framework for how to promote all of these things to production and that we're thinking about that across all of the maturity curve within our organization. We also want to make sure that we're standardizing kind of a single
[80:49] deployment contract that people can fit within. And so, who is actually able to publish new assets to Unity Catalog and who's responsible for owning them going forward and how do people collaborate on
[81:01] them? um those questions should all fit within these same steps here, right? But those automation surfaces also become really important here. So if
[81:13] we're separating out, if we're decoupling kind of the shells like the workspace deployment, maybe lower uh order Unity catalog objects like storage credentials, external locations, maybe the catalog level from all of the things
[81:27] we're creating within that catalog. We want to have um where these people operate. So your enterprise IT team, they might be the ones managing your Terraform code bases. One framework deploys all your workspaces across all
[81:39] the business units that are working autonomously. for those autonomous teams and even for the assisted teams, you might have this bridge of a BU liaison that manages some uh auditing of the downstream teams as
[81:52] well as uh owns any BU specific Terraform which might be uh Unity catalog permissions applied on top of assets under management within that team. And then your product and delivery team shouldn't have to think about any
[82:04] of that. They're just turnurning out assets that fit within the constructs that we've set up for them. So a pipeline sequence might look like your central IT team doing the uh like bootstraps for identity secrets service
[82:18] principles um integrations. Uh they might manage the account state as well. So the deployment of your actual meta store and all of the configurations there and they might own that workspace shell that gets deployed. We should
[82:31] probably define that once. There's lots of network connectivity there that we don't want random business users having to repeat sign off on. But then within that, your business unit has a handshake and can pick up from there to manage all of the workspace configurations, the
[82:44] Unity catalog configurations within their setup and then manage their continuous pipeline deployments here. So it allows for parallel work streams without collisions. So our delivery scenarios look like
[82:58] usually the routine is standalone execution for your autonomous teams, right? It's not every day you have to deploy a new meta store or even a new workspace, but joint execution becomes a handshake for on-prem routing, front
[83:12] door rules, new BI tools, new integrations outside your estate that require new networking patterns that have to be approved. So an example of this might even be data sharing, right? I have one business unit that wants to
[83:25] share data with another business unit. In most cases, it probably won't trigger a global ORC policy that hasn't already been established. And so, they can manage that on their own and just do
[83:37] workspace catalog binding or grant the permissions for whoever needs that asset. In some cases, this is where Handshake might come in with your enterprise IT team to say, "Ah, that's actually data that we don't have a global policy for yet, uh, that applies
[83:50] to all of our cataloges. Let's get that defined, get it documented, and then you can start to share that freely across teams and we're sure that that data will be secure. So the big lesson here is if it depends
[84:02] on somebody remembering, it's not going to scale, right? So all of those things we looked at across lessons one through four really give us a lot of confidence here. And I'm just going to close out
[84:15] our last two slides here to finish up and see if we can get two questions. So the big thing here is start with what you have. Um if you have if your team had one week to improve governance,
[84:27] where do they start? This is kind of some of those requirements, right? We ask questions at the beginning of each slide. These are what some of those questions are. Can your new hire find your data? Uh can something ungoverned make it into
[84:40] production? And this becomes our scorecard. So across these dimensions, start out with definitions, right? put all the technology aside and make sure we know who the people are and what the
[84:53] process is for it. Once that's defined, great, you're at the manual stage. Go ahead and prototype. Make sure it works. Figure out how things connect. Once it's connected, now you can scale it out. Um, this is where you can start doing the
[85:06] Aback policies. You're testing your identity chains with your agentic solutions. You're doing guardrails. And then from there, this is where you're scaling. You're automating your daily tests. you have all of your alerts on
[85:18] top of those dashboards that we saw so that you're not going to find errors. Those errors are coming to you and you're able to solve them going forward. So start with what you have, score it, and then improve from there. And I'm
[85:30] sure next year there's going to be a whole new suite of things that we're all going to have to account for that we didn't have to account for two years ago. And then we'll have a new matrix here. So wherever you're at, it's all good. But just keep moving forward.

Learn more about the Databricks Data and AI platform.