What’s New in Unity Catalog With Live Demos!

Summary

Unity Catalog now governs AI primitives — including skills, MCP servers, and model endpoints — as securable objects alongside data, with a Unity AI Gateway that controls access to both internal and external AI models.
Automated sensitive data detection with PII masking in AI Gateway request and response flows extends data protection to agentic workloads without requiring manual classification of every new asset.
Genie Ontology brings domains, KPIs, and certified data products to Unity Catalog, providing the business context that enables agents to produce trustworthy answers from governed data.

What’s New in Unity Catalog With Live Demos!

Unity Catalog is evolving fast! Join the product team for a demo-packed dive into the latest capabilities shipping now and what’s on the horizon for data governance. We’ll unveil powerful new features across access control, data lineage, AI governance, observability, discovery and many more! Walk away with a complete understanding of what Unity Catalog can do for your organization today — and what advancements are coming next!
Talk By: Lianne Zelsman, Sr. Product Manager, Databricks ; Paul Roome, Senior Staff Product Manager, Databricks ;

Chapters

FAQs

What AI primitives can be governed in Unity Catalog?

Unity Catalog now treats skills, MCP servers, and model endpoints as first-class securable objects, meaning they can be discovered, permissioned, and audited alongside data tables and volumes. The catalog UI supports creating and managing MCP services directly, making AI tool governance consistent with the existing data governance experience.

What is the Unity AI Gateway?

The Unity AI Gateway is a control plane within Unity Catalog that governs access to external AI models and services, applying the same consistent access policies used for data assets. It also enables PII masking in AI request and response flows, preventing sensitive data from leaving the governed environment during agentic interactions.

Why is governance becoming more critical as agent query volumes grow?

Internal Databricks environments show agent-driven query volumes growing exponentially and recently surpassing human-driven queries in scale, creating a new class of governance challenges around data access, cost, and trustworthiness. Unity Catalog's AI primitives and Genie Ontology are designed to address these challenges as agents become the primary data consumers on the platform.

What is Genie Ontology and how does it improve AI answer quality?

Genie Ontology stores domains, KPIs, and certified data product definitions in Unity Catalog, providing agents with the business context needed to generate accurate and trustworthy answers. By grounding agent reasoning in governed semantic definitions rather than raw schema metadata, Genie Ontology reduces hallucinations on company-specific data questions.

Full transcript

[00:07] All right, let's go ahead and get started. So, thank you for joining us for this session to talk about what's new in Unity Catalog. You got myself, Paul, and you have Leanne as well. Uh, we're from the product team. We look
[00:19] after Unity catalog, and we wanted to give you a sneak peek into our road map and talk about what's hot in the world of cataloging. Uh before we get into that, just a quick standard disclaimer. Some of the features we look at today,
[00:31] they're on our road map. They may change in form before they reach your laptops. Okay. So I wanted to begin with the problem that we saw 5 years ago. We saw a fundamental problem with the way that
[00:44] governance was run in organizations. We saw many different data personas. You can see them on the left hand side here. Trying to access many different types of data and the governance team struggling to keep up. Okay. So let's have a look
[00:56] at an example. Perhaps the data analyst wanted to access customer data. Well, you would go to the warehouse perhaps and configure some ACL to do that. The data engineer going wanting to access images. Well, you know, maybe that was
[01:09] via the AWS identity and access management system. And the data scientist wanting to access log data. Well, you know, maybe the EC2 credentials is the way that we provision access there. Okay. And so on. And so we
[01:22] started with a simple mission. We wanted to bring unified governance to the lakehouse. So what does that mean? We wanted a consistent way of doing access control, discovery, lineage, auditing,
[01:36] and all of these other capabilities regardless of the type of data consumer we had or the type of data. Okay, so that was our mission. Since then, some stuff has changed,
[01:48] right? Like I think we all feel a little bit like this guy in the foreground here. Okay, there's been some trends happening in the industry and we're actually seeing real changes
[02:00] in the data platform. Okay, so this is just one graphic. This looks at an internal environment we have at data bricks and it looks at the query volume um over the last couple of years for different slices of queries. And I
[02:12] didn't tell you what those slices are yet, but you can see something magical happens sort of midway through last year, right? The yellow component starts to go exponential. This is the number of queries driven by agents. Okay, so query uh agents are
[02:26] just driving the platform now, right? We have a new player in our game called the data agent. Okay, so let's meet the data agent. This is our data agent and he you
[02:38] know he he might look a little bit like you or I okay but there's a few things you need to know about the data agent. So firstly, the data agent, he's called the data agent, but he actually connects to all of our systems, right? Not just
[02:52] data bricks. He he wants to connect to Slack, he wants to connect to drive, he wants to connect to all of the systems. He runs queries on our behalf, right? Which is awesome but terrifying, right?
[03:05] and he discovers data that we didn't even know existed which again has so much potential but you know when you're giving the numbers to your CEO you want to make sure it's on trusted validated
[03:17] data okay so what we wanted to do in this talk is have a look at how we need to extend the catalog so that we can have this new player in our game right
[03:29] how can we extend the catalog for the data agent so we're going to talk about extending the catalog for AI primitives is controlling access in the aentic era and adding context so that agents can do smart things. Okay, so the first
[03:42] chapter. So this is how I see the typical enterprise AI adoption journey. Okay, you start with one simple model. The support team, they're sick of triaging
[03:56] uh support calls. They want to build an agent to do it. So they pull from one simple uh MCP, use a simple model. The legal team, they hear about this thing. They they do something very similar. The
[04:09] marketing team, they get FOMO, right? They want to hook into the into the support agent. They want to add their own data and build their own uh agent and so on. Okay? And you're left
[04:21] thinking, how much is this costing me? Okay. What data am I sending to these external model providers? And who has access to all of these different agents? Okay, we call this agents sprawl and
[04:34] this is what we're seeing happening in the enterprise right now. The good news is there's a tried and tested way in enterprise technology to deal with this kind of sprawl. Okay, and
[04:49] we talked about it about five minutes ago, right? It's the catalog. It's really good at untangling sprawl. So the first set of capabilities we want to talk about is how we're extending the catalog to add these new AI primitives
[05:03] required to untangle that mess. Okay, so you can see them on the right hand side here in green. We're adding skills agents themselves, the model service, meaning the actual invocable model
[05:15] endpoint and MCPs are securable objects in Unity catalog. What does this mean? This means you get all of the same governance capabilities that you did for tables and files for these new types of
[05:28] AI AI assets. Okay, so specifically you can now create these uh you know skills and MCPS and so on and reuse them across users and workspaces. Okay, that's one of the benefits. you can access control
[05:41] them using the familiar SQL grant style access controls we have in Unity catalog and use things like arbback and aback as well and then you get consistent lineage and auditing just like you do for all of
[05:53] the other Unity catalog securable types. This is a a quick look at, you know, what it looks like. In the mo in the middle here, I have my customer support endpoint and Unity catalog has
[06:06] automatically tracked exactly which models are going into this. We're using a Quen model and a Claude model and who's consuming it, right? And this is automatically built and maintained by Unity catalog. Okay,
[06:20] excellent. Okay, so that was the first set of capabilities we're adding, but that's not it. We're adding somewhere else. We call it Unity AI gateway. You can think of this as a new enforcement mechanism. So what does this do? Think
[06:33] of it like a control valve, right? You've got your users on the left hand side, including AI agents that are using all of this stuff and you've got your securable assets on the right hand side. And Unity AI gateway is the control
[06:45] valve that helps you control things like cost uh where you're sending your data and so on. Okay. Um, so let's have a look at some specific examples. With Unity AI Gateway, you can control and
[06:57] track spend across all of the users, models, and agents. You can set budgets so that people don't go crazy with it. You can enforce policies and guard rails that say things like, "Make sure we're not sending any PII to external model
[07:10] providers," or, "Guard me against prompt injection for my users." And then every single request to the model is actually logged in a delta table. So you get full audit history of exactly the request and
[07:23] the response from the model which you can use for model quality and also for kind of access auditing use cases as well. Okay. Um so this is just one screenshot. This shows you how you know
[07:36] you can track cost down to the user level exactly how many tokens they're spending on what model uh and how much that cost you uh using using Unity AI gateway.
[07:49] Okay. And I'm gonna welcome Leanne to stage to tour to give you a quick demo of these. There we go. >> Thank you very much.
[08:01] >> All right. So Paul just told you about all of the exciting stuff that we are adding into the catalog. So I want to show you what that looks like. So in case you're not familiar, this is Unity catalog. Uh I'm currently in the catalog explorer view right now. And if I go
[08:13] into this retail prod catalog that I have, I can see all of the assets that are underneath the catalog. So all of my schemas and then all of the individual object types that are underneath those schemas. So you see the tables, the
[08:26] volumes, the functions, the models, everything that was there before. But now you're also seeing those new services that Paul was talking about, the AI related services. So let's see what it looks like to go create one of
[08:38] those. So if I go to my gold schema and I go to create and I go to service now I can create an MCP service from here or I could create a model service from here. So let's test creating a
[08:51] model service. I'm just going to call it test demo model service because I'm not very creative with naming today. You can choose which specific model that this
[09:03] model service is going to be for. Let's do Opus 4.7. and I click create. And now I have a new model service in Unity Catalog that benefits from all of the goodies that Unity Catalog gives
[09:17] you. So you've got all of this governance setup over here where you can see your usage tracking. You can set up some policies which we'll get into more detail about later. You can set up an inference table. So maybe let's try
[09:30] doing that. So the inference table is going to be where any of the request and response payload gets logged. So, let's say where that table should be. We're going to put it in that same catalog.
[09:42] Not supported for schemas and default storage. All right, let's go back.
[09:55] I didn't pray to the demo gods today, so everything has to go slightly arai. Let's go to permissions first. So since it's in Unity catalog, I can grant and revoke permissions for anyone that I
[10:08] want access to this new model service. So let's say someone more than just me and Paul should get access to it. How about my co-orker Jod? We're going to give her execute permissions on this model service so that she can run it and
[10:21] do all of the capabilities that come with this model service. And because it's in UC, you also get the lineage capabilities. So since this is brand new and I just created it, you're really just seeing the upstream model, but any
[10:34] downstream payloads, downstream sources that are now using this model service, any interactions that happen in the future are going to show up in this graph
[10:51] and let's dive into access control. So, as Paul alluded to, data catalog expectations are changing with this AI world. And that means that the access control needs for the data catalog are also evolving. So, we have this meme of this dog. This
[11:06] is kind of what it feels like right now for governance teams where you have so much new data coming in. There's new tables. There's new users. There's potentially thousands of agents that are acting on behalf of users, accessing data that has never been found before.
[11:20] and had they're doing things with the data that hasn't been done before and it's kind of just chaos. Thankfully with our Unity catalog access control framework, we have a solution for this and it comes in three main parts. So our
[11:33] arbback role-based access control for isolation boundaries and role switching, Aback policies which is the attribute-based access control for powerful scalable management and then observability via governance hub. you
[11:46] can actually understand what is happening with all of that access. Let's start with our back. So today in Unity Catalog, all of our permissions follow union semantics. So
[11:58] if you are Alice and you are a member of groups 1, two, and three, Alice's permissions are now a union of groups one, two, and three. Which makes sense
[12:10] until it starts to cause some problems because what this means is as Alice continues to work, gets more access to more projects, suddenly she has access to everything all the time simultaneously and there's no isolation
[12:24] boundary. So she can query, she can join, she can share data across all of these different data sets. And that might not always be something that you want. So this is where role-based access control comes in arbback. So now we have
[12:39] this new type of group that is assumable. So instead of all group permissions unioning unioning together, you can say that certain groups are can only be assumed exclusively. Think of
[12:51] this like switching hats. So if Alice is working on project 3, she can assume the role for project 3 and only get access to project 3 data exclusively. She
[13:03] cannot access anything from project one or project two. She would have to switch into those roles to get access to that data. And the good part here is that this is an instance switch. You don't have to log out. You don't have to
[13:15] switch workspaces. It's just a workspace toggle that you can quickly switch back and forth between. So, we've talked about isolation boundaries and role switching, but how do you actually control what data can be
[13:29] accessed and what can you do with that data by whatever role you are letting someone have? That's where Aback policies come in. So, we've got this three-part system, three capabilities, one unified system, no manual steps in
[13:44] between. It's all done for you. So starting with data classification, this is the system that detects any sensitive data that comes in. So things like social security numbers, credit card information, etc. and classifies it as
[13:57] sensitive and creates a tag. Then you have your govern tags that are the taxonomy that make sure that that classification is consistent across your environment. And then you have aback
[14:09] policies that actually enforce access rules based on those tags. So let's dive into each of those. Starting with data classification. Like I mentioned, this is your continuous AI powered scanning. So automatically
[14:22] detects any sensitive data patterns that come in. So SSN's, emails, etc. We have a bunch of built-in classifiers that are available that cover things like PII, PCI, some common regional tags. But we
[14:36] also now have custom classifiers. So these can learn from already tagged columns to detect business specific patterns without you having to manually manage those rules.
[14:48] So data classification is detecting that sensitive data and it is creating these tags automatically for you. So let's talk about how those tags are governed. Think about governed tags like your
[15:00] standardized texonomy at the account level. These are the key value pairs that you can control centrally. And the important component here is that they are governed tags, which means that
[15:12] they're permission controlled. Only the authorized users that you decide are authorized can create and tag objects. And they also follow inheritance. So if you tag a catalog, all schemas and all
[15:24] objects within that schema will inherit that same tag. Now to the last component, Aback policies. So this is what actually enforces access control based on those
[15:36] tags. Aback policies have four main components. You have the scope. So this is the object where the policy is attached. You have the principles. So this is who the policy applies to or does not apply to. You have actions. So
[15:51] this is what the policy is doing, whether it's masking data, filtering data, etc. And then you have conditions. This is when the policy is triggering. So the attribute-based expressions that you define. So you can end up with a
[16:05] policy that like the one on this screen that says mask all marketing data if it's tagged PII for everyone except for the marketing team. The important thing here to reiterate is
[16:18] it's there's no manual steps in between these three capabilities. It's an automatic continuous flow and a loop. though you can be mostly hands-off and not have to continually intercede and try to manually manage everything at a
[16:32] table by table or grant bygrant basis. So let's take a look at what that looks like. Switching back to the catalog I was in before.
[16:44] If I go to retail prod, I've already enabled data classification for this. I did this yesterday so that it could run its first scan. Um, I've only enabled it specifically for this gold schema, but
[16:57] you can choose to enable it for all schemas or exclude specific ones. And if I want to see details on how the scanning is working, I can go and I can look and see everywhere where it has detected sensitive data. Now, this is
[17:10] just the very standard outofbox PII classification. I have not set up any custom rules or any fancy magic which you can do. This is very much just the single button click of like find
[17:22] sensitive data and this is what it found for me and I can see everywhere where sensitive data is detected where and what uh tables that that is in. So let's go see what I can do with those tags.
[17:37] So since data classification is running for me, if I now go to let's just check any table in this catalog, I can now see that any tables that have
[17:50] some of that sensitive information already have these tags. I didn't have to add these. The data classification system added these for me. Now comes the fun part. So, I've already pre-created
[18:02] an Aback policy on the catalog where I'm saying that for all account users except for this specific accepted group called PII readers, I want to mask
[18:16] any columns that contain any of that sensitive data that the data classification system found. So now if I go and look at one of those tables. So going back to that same table
[18:29] that I was in because I am one of all account users. I'm seeing mask data. So any of those columns that were tagged I now can't see
[18:41] that information because it was sensitive. But let's look at what happens if I switch into that that group that I had exempted in the policy. So I'd exempted that PII readers group. If I go to the arbback roll switcher, I can
[18:54] switch to assume PII readers. And now I can see all of the sensitive data because I'm in this PII readers role.
[19:08] Awesome. Now the fun part is we are extending our Aback capabilities. We've talked about what we have currently, which is row filters and column masks, but those are all about controlling the visibility of
[19:20] data in tables, but what about all of the stuff that Paul talked about, all those new objects coming into Unity Catalog? This is where service policies come in. So now you can control and filter interactions over MCPs, model
[19:35] services, agents, all of that stuff. Uh so you could mask PII in request or responses if say the service call is being made by an agent. And then we're also extending to access
[19:48] permissions. So you can create grant or deny policies that say grant or deny certain permissions from whatever users or groups across entire cataloges or
[20:00] schemas or many cataloges or many schemas based on attributes that either the data has or that the users have. So speaking of attributes, we're also
[20:12] extending the types of attributes that we're going to support in these policies. So we've already talked about govern tags. That is the information about the data itself. But now we're introducing attributes about both the
[20:25] principle and the request. So for the principle we're going to have identity attributes. So this is things like what geography is the user in? What uh what employment type do they have? What
[20:39] organization do they belong to in your company? And then on the context attribute side, this is looking at and controlling how the data can be accessed. So you could say um is it
[20:51] being accessed by a user or is it being accessed by an agent on behalf of a user and set different rules based on that context. And this can all be combined into a single policy. So you can have
[21:03] richer conditions and way fewer policies to have all of this power. So let's take a look at what that looks like. Coming back to this same catalog that I was already in.
[21:16] Now, previously I had shown granting access to my colleague Jodie to give her execute permissions on a single model. But what if I need to control access across all the models in my catalog? How
[21:28] about if we're looking at say the system AI catalog? I need to switch back to my personal role so I can see everything
[21:40] in the system.ai schema. That's where we have all of our outofbox available foundational models. What if I want to control who can access these? So I could go and create a aback
[21:54] grant policy by clicking new policy. Let's call this model access policy. It's going to be a
[22:06] grant type policy and we're going to apply it to let's say just members of our ML team.
[22:19] And then I can set a rule to say I only want to grant this access when models have a verified tag. So I know that these models are safe for the ML team to use based on our verification system. So hashtag
[22:32] value
[22:47] and then I'll create that policy. So what this now does is any time a new model comes in or a model gets tagged as verified anyone in that ML team group will automatically be granted that execute permission. I don't have to go
[22:59] do that manually every single time anymore. And then what about the system policies that we were talking about? If I go back to that model ser sorry service policies. If I go back to that model
[23:11] service that I had created called test model service and I create a policy. We have a bunch of outofbox guardrail types that you can choose from. Um, you
[23:25] can also create your own guardrail type if you want to get super fancy. Um, you could choose to have a custom function. You could have Genie help you create that custom function. Like right now
[23:37] it's suggesting maybe I want to block all insecure URLs. Um, but maybe for this I just want to block and redact any PII. So let's go back to the PII reduction policy type.
[23:50] I've chosen my model service evaluator. Click create policy. And now in just a couple of seconds, I now am automatically making sure that any
[24:02] request or response that is using these models in this model service, the any PII data is automatically redacted. I didn't have to do anything else to configure this.
[24:16] All right. And last but not least, observability. So none of the stuff I talked about is helpful if you don't actually know if it's working or not. So with governance hub which is our centralized access observability for unity catalog you can
[24:30] finally understand information about what all someone has access to what they can do with that access and how they got access. So whether it's from a manual grant through an Aback grant policy like
[24:43] we just showed or maybe from uh groups that they are members of and then based on that information you can now prioritize taking action on things like missing metadata classification policy coverage gaps anything that's important
[24:56] to you. So this is now supplementing what was already existing in governance hub around cost and data insights. Now we have access insights. So remember this meme with all of the
[25:09] chaos. Now that we have this three-part Unity catalog access control framework, hopefully your governance teams can sleep better at night because all of this chaos is controlled.
[25:21] And I will pass back to Paul to talk about context. Awesome. Thank you. So last chapter now we're going to talk about adding context
[25:33] into the catalog. Let's remind ourselves about why we want to do that. So this is our data agent if you remember our new player in the game. And the data agent has the potential to be our best friend,
[25:46] our best co-worker or you know you can probably have any relationship you want in a professional setting with a data agent. No judgment. Uh but we don't want the data agents turn into this guy. Okay? We don't want to be watching this
[25:59] movie in two years time. We don't want the data agent that is hilariously confident but terrifyingly wrong. Right? And we don't want to live in the AI era where it's not what you know, it's what you think you know. And we definitely
[26:12] don't want the masterclass in being wrong. Okay? And to avoid this, we need to get one thing right. Okay? And that's context. So let's have a look at an example of that. So let's say our data agent is asking what's our revenue per
[26:26] active customer in Q3. Okay, there's a few things that we probably want the data agent to know to answer this correctly. The data agent should know what revenue is and what active customers are and probably what Q3 is,
[26:39] right? And then the data agent should be able to find a good source of data for those. What the agent actually sees is a list of technical column and table names and maybe some comments if you've done
[26:51] your uh done your homework. But without the stuff in red, we're going to lack that business context and we're going to have some assumptions. We're probably going to get some hallucinations. Okay, so this is what we
[27:04] need to get right. So you might have heard Ali say this this morning. We really think the bottleneck now is not about intelligence. AI doesn't have an intelligence problem. It has a context problem to be useful in the enterprise
[27:16] now. Okay. So how are we going to provide this context? Okay. So I want to tell you about the stack that we're building up in Unity catalog adding to the catalog so that we can make agents smart
[27:28] with this context. So it starts with great connectivity to all of your data, right? Your da your data in in Delta and iceberg, your federated sources in your other cataloges. Connect it all into
[27:40] data bricks, but also your business applications. So much context is in Slack, in Drive and Teams and SharePoint. Okay, connect that in. And then also your workspace assets, your data bricks, notebooks and and
[27:53] dashboards. So much context in that. Okay, so this is the building block. But what do we do with it? The first thing we we do is we build what we call unity catalog semantics over the top. This the goal of this is
[28:07] to augment the data with the terms the concept and the KPIs which allows the business to understand it. Okay. So we have domains that allow you to organize
[28:19] your data around business area. Okay. We have glosseries which help agents understand different concepts in your your organization. like what is an active customer and we have metrics
[28:31] which provide deterministic uh executable KPIs right they're basically tables the thing to remember about this layer this is the user controlled layer right the user along with AI tooling
[28:46] curates this stuff it's awesome because it's highly precise it's really authoritative but the sad truth is it will always be incomplete right because humans can only model so much you can
[28:58] never write down everything the business knows and so that's why we think this is necessary but not sufficient in the context stack and the missing piece is the geneontology
[29:10] okay so here's the difference with the geneontology the geneontology is able to learn from how users act not just what they define okay so my favorite example of this is in my product team we have
[29:24] this definition of a highly engaged customer in Unity catalog, right? Uh hopefully you're all highly engaged customers. Um we've never written this definition down anywhere, right? We've
[29:36] never written it down formally. It's in a dashboard. We check it once a week. Geneontology is able to mine that dashboard, find this business term, highly engaged customer, and then figure
[29:48] out how to calculate it based on the logic in the in the dashboard, and then serve it to the user to answer their questions. when you ask the ask genie, you know, how are my highly engaged customers? Okay, so the combination of
[30:03] the ontology provides you the scale knowledge and then the unity catalog semantics is that highly authoritative modeled layer. Um so the two really complement each
[30:16] other and then you can consume these from you know genie via the GMCP to any external agent via SQL interfaces to the SQL objects here as well. So this is what we think is the next generation of
[30:28] context stack in the catalog. Um we oh let's have a look at a quick example of this. So let's let's take another example just to bring all of this together. Okay so let's say our data agent asks why did activation drop
[30:42] last month even though signups went up. What are the re reasoning steps it's going to go through? Well, firstly, the data a data agent is going to contact the glossery and it's going to say, well, do you know anything about
[30:54] activation? And in this case, yes, there's a glossery item for it. Awesome. So, it now knows what glosseries are and it has a source of data. It can leverage a metric view for the actual calculation. But let's say signups, you
[31:06] know, that's never been defined before by a human. Well, the geneontology is able to infer this perhaps from a notebook. This activation funnel not notebook is in my diagram here. But it's
[31:18] automatically inferred what signups mean and given us a data source. Okay. So just putting this together, the glossery page allows us to reason over the concept, right? What is activation? The
[31:30] metric view is leveraged for the actual calculation itself to get the data. Uh geneontology is that learned layer. This gives that scale knowledge and augments the fact that we haven't actually
[31:43] formally defined what signups are. Okay, so they all three get used together. And I'm going to give a quick demo of some of these components in action. Okay, so
[31:56] let me come over here. And the first thing I just want to orient you on, I'm in this new surface in data bricks. It's called discover. It's in beta right now, but it's actually going public preview this week. And Leanne was showing you
[32:08] kind of the catalog explorer. The catalog explorer is awesome because I can see all the data that I've ever had access to there. But it's also terrifying because it has all the data I've had access to. And so what you find
[32:22] in discover you get the data that has been highly curated and certified for high volume usage. Right? And the second thing it's oriented around business area
[32:34] not around catalog. Okay. So, I've created eight different business domains here. I've created retail, orders, customer success, and so on. And when I click into one of these, you see that
[32:47] you load in your different asset types into here. So, this domain of customer success has dashboards in it. It has metric views, it has genie spaces, it has tables, but these are the stuff that's really highly relevant to uh this
[33:00] business area of customer success. Okay, the one-stop shop, but I want to focus on this glossery section here. Okay, this is the new thing I talked about in the the semantic
[33:12] stack. So, let me click into one of my glossery terms here. So, here I've defined a term called LTV. Okay, it's got synonyms of CLV or lifetime value. And you can see the structure of the
[33:25] glossery is that of kind of a unstructured kind of wiki page confluence format, right? We're leaning into the, you know, the lingua franker of the agent, which is markdown, basically. Um, I can come in here. I can
[33:38] edit this. I can say, you know, how to get help. Uh, I can make that a heading. I can tag people in here. I can say, you know, Hongi is my myme here. Um, and
[33:51] then I can start to build up relationships between these different pages as well. So I can say you know this LTV concept actually relates to the attribution u page that I've defined elsewhere. I could add extra details
[34:03] about exactly how these things relate. Okay. Um so that's the general structure of it. We also link down to your actual assets in your lakehouse. Right? That's the what the related assets are here. So that you can think of these kind of like
[34:16] the the agent next steps. If the agent is trying to reason about LTV, well, it probably wants to know find a metric view that contains this this particular concept. Okay. Um, the last thing I'll say on this particular page, we're also
[34:29] really leaning into the crowdsource kind of model here. So, any user can come in and say, you know, actually this is factually incorrect. This should be, you know, 24 months, you know, and this bit's wrong. And they can make a
[34:42] suggested edit. Okay. But Mudet, who's the domain admin, has to come in and actually suggest these before they go live. Okay, so this is a glossery page, but you're
[34:54] probably thinking, you've been talking a lot about AI. Where where's AI coming here? Do I have to write this all myself? Well, that's the really exciting thing about this. We actually expect most of these pages to be written by our
[35:07] agents. And this is where there's a really nice symbi symbiosis between the geneontology and this glossery layer. So let's say that I wanted to author a page on incremental revenue. I could come
[35:19] into Genie code. I can say tell me about incremental revenue and we're going to ping it's going to ping genie uh sorry the geneontology here pull down everything it knows about incremental revenue and give me a page bootstrapped
[35:33] for me. Okay. Not only that, it's going to also start to detect drift over time where the geneontology knows something that my glossery page doesn't. So, it's going to really help keep these things up to date so that we have really rich
[35:45] and accurate context. Okay. Um, so we just looked at couple of different elements of the stack, domains and glossery. Here I'm going to give the world's quickest demo on geneontology
[35:58] and then uh we'll we'll move on. So let's let's have a look at the genieontology layer here. And before we came into the session, I asked Genie
[36:10] one. So that's where I am in the data bricks right here. I'm in Genie one. I asked Genie1, give me a summary of the customer profiles visiting the summit this year. Okay. I told it not to actually return data so I don't like,
[36:23] you know, give any of our secrets away. But um it gave it gave me a nice response, but I don't want to focus on the response. I want to focus on how it got there. So you can see down at the bottom we have our sources are cited
[36:36] here and one of those is the geneontology. Okay. So it says to to provide you this answer I actually use the ontology. So let's have a look at one of these things. Uh let's have a look at this
[36:48] one. So the geneontology has mined this piece of knowledge basically that the this particular funnyl looking table here is the primary table that you need to know about if you are thinking about
[37:02] summit registration and attendance right and then it actually goes on to give me the exact filter clauses that I need to use to or that the agent needs to use to actually uh prosecute this table. How
[37:14] did it do this? Well, it actually found this particular dashboard here which has this high authority score. Lots of folks look at it. It's been certified. The author is very trusted and so it knows
[37:26] that this is reliable knowledge that it mind. And down at the bottom here, I can see a nice little lineage graph of all this. So, I can see that it was created in the marketing domain. Here's my snippet and my dashboard and my author
[37:39] there. Okay. So, just to go back to my diagram really quickly, we looked at the geneontology again. given us the scale knowledge. Okay. Complements the uniccast with semantics. Excellent. Okay. And I'm going to pass over to Leanne.
[37:52] >> All right. Let's bring this all home so I can get you out of here into all your happy hours. So you have heard us talk about these three things today. All of the new
[38:04] AI primitives that are coming to Unity catalog, how you can manage access for those new items at scale. and then having that intelligent layer so that
[38:16] everything that is interacting with the data all of these agents they actually can do useful things. So now this data agent that we know has leveled up. It's now unlocked safe
[38:28] access to all the systems uh and models. There's access control at scale and you have full enterprise context. Congratulations. Uh, and just finally, everything we
[38:42] talked about today is great, but we could only touch on some things that are coming and there is so much more that is going to be talked about this week. So, we have listed some sessions here that we think would be super useful if you
[38:55] heard anything today that you want to deep dive on. So, there's going to be a dedicated access control session tomorrow. There's a bunch of AI governance specific sessions, but we've listed two here that would be really
[39:07] great to go to. Um, there's also some open lake house sessions which we did not have any time to talk about today at all. Um, and then some other sessions that you may have missed today that we think are particularly interesting. If you didn't make it to them, check out
[39:20] the recordings online once they are available. Um, there's so much coming to Unity Catalog and we could only cover some of it today. So, please go enjoy the rest of the conference. Uh, and if
[39:32] you have any questions, we'll hang around a bit in the hallway after. I think we're going to get kicked out of this room, but you can come talk to us. Um, and otherwise, thank you so much for joining. Hope you have a great week.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.