Unity Catalog Ontology: Enterprise Context for Trustworthy AI

Summary

Databricks Unity Catalog now includes a business glossary, domains, and Genie ontology to give AI agents verified enterprise context and prevent hallucination when answering business questions.
The onto-rank algorithm re-ranks knowledge graph results using verification status, time decay, authority signals, and domain expertise to ensure agents surface the most trustworthy answers.
Genie ontology ingests unstructured business knowledge from Slack, Confluence, and SharePoint alongside governed lakehouse assets, updating automatically as data and teams evolve.

Unity Catalog Ontology: Enterprise Context for Trustworthy AI

Ontology is the missing layer between your data platform and trustworthy AI. Databricks Unity Catalog now includes business glossary, domains, and automated Genie ontology to ensure AI agents ground their answers in verified enterprise context. Learn how Genie builds a live, permission-aware knowledge graph that understands relationships between data assets, business terms, and trusted individuals across your organization.
In this talk, Raj Gossain walks through how Genie ontology prevents AI hallucination by giving agents visibility into your lakehouse assets, federated data sources, unstructured business knowledge from Slack, Confluence, and SharePoint, plus certified tables and metrics. Discover how the onto-rank algorithm re-ranks results based on verification status, time decay, authority signals, and domain expertise, and how this knowledge graph updates automatically as your data and team evolve.
🤝

Chapters

FAQs

What is Genie ontology and why does it matter for AI accuracy?

Genie ontology is a live, permission-aware knowledge graph within Databricks Unity Catalog that gives AI agents visibility into your data assets, business terms, and trusted subject-matter experts across the organization. Without this business context, LLMs are prone to hallucination; ontology grounds their answers in verified enterprise knowledge so results can be trusted.

How does the onto-rank algorithm improve knowledge graph results?

Onto-rank re-ranks results based on four signals: verification status, time decay, the authority of who certified the information, and domain expertise. This ensures agents surface the most current and credible answers rather than relying on stale or unverified data buried in the knowledge graph.

What unstructured data sources can Genie ontology connect to?

Genie ontology can ingest unstructured business knowledge from Slack, Confluence, and SharePoint in addition to governed lakehouse assets and federated data sources in Databricks. This broader sweep captures the institutional knowledge that typically lives outside formal databases in most organizations.

How does Unity Catalog's business glossary help organize enterprise data?

The business glossary capability within Unity Catalog lets customers define key business terms and organize data assets across business units, functions, and geographies using domains. This structured semantic layer feeds into Genie ontology so agents understand what business concepts mean in the context of a specific organization rather than relying on generic definitions.

Full transcript

[00:20] Hello and welcome back. We are joined by Raj. Raj, would you like to give us a little intro? I'd be happy to. Raj Gosain, I run one of the product teams at Unity Catalog, on Unity Catalog I should say. I've been at Databricks for actually 3 years and this is my fourth summit now. Oh, okay. Amazing. Great. Actually, it's my fourth summit
[00:35] too. It is, okay. Yeah, I my 3-year anniversary is in 3 days, so yeah. Excellent. Congrats. Exactly. Exactly. Cool. So, today we're discussing about ontology. Okay. Yeah. So, why is it important and how does Databricks fit into the whole ontology game?
[00:51] And and maybe for those newer to Unity Catalog, give them a bit of a background on that. Absolutely. Okay. Everyone remembers the moment last year when Matei open-sourced that uh uh All right. So, okay, let me let me step back a little bit. So, Unity Catalog is our unified governance layer for data
[01:06] and AI at Databricks. Honestly, it's a big reason why we've been so successful in getting customers to migrate to the lakehouse architecture. Provides security, lineage, governance capabilities, auditing, and we've added semantics capabilities to it. And so, I think, you know, why does ontology
[01:22] matter? You know, the biggest challenge that people have with LLMs is if an LLM doesn't have context around your business, it's going to hallucinate, it's going to make up answers, and it's going to give you stuff that you can't trust. And so, we thought it was essential to build a knowledge or
[01:39] context layer into Databricks, and it spans both the Genie ontology as well as semantics within Unity Catalog. Yeah. Very nice. Okay. Cool. And then, in terms of you know, how it is relevant to AI and agents, can you give us an idea of
[01:55] Absolutely. we parse it? How do we fetch No, no, it's it's a great question. So, let me start with the Unity Catalog layer. So, what we've done with UC is we have visibility into all the assets that are in the lakehouse as well as those that are federated into Databricks.
[02:12] So, we calculate lineage, we can infer popularity. What we've added in the UC layer, which feeds the Genie ontology, is what we call Unity Catalog semantics. So, we've added domains that help customers organize their assets across, you know, business units, functions,
[02:28] geographies, what have you. But, in addition, we're just now adding, and you'll you'll hear more about this in the next couple days, a new business glossary capability within Unity Catalog. And that allows customers to really define key business terms. And so, this is Think of this is like the
[02:44] ground truth, the source of truth for what does a customer actually mean? What does a churn customer mean? What What is a an aircraft engine mean if you're in the you know, the airline or aircraft manufacturing business? And those business terms previously they haven't
[03:01] actually existed in Unity Catalog. And so, Genie was trying to figure out like, okay, what does that definition actually mean? By having that ground truth within Unity Catalog, we're giving the Genie ontology insights into how data's organized,
[03:17] business terminology that customers use in their organization. And then that Genie ontology gets automatically generated based on everything that's in Unity Catalog, all the sources that Databricks has visibility into. And it essentially creates a live, always
[03:34] up-to-date knowledge graph. And that knowledge graph is used to inform Genie when anybody asks a question in Genie 1 or in a Genie space, that ontology is going to guide Genie to maximize the likelihood that Genie's
[03:49] going to provide answers that are kind of verifiable, understandable, probably correct, and it's going to do it in a very token efficient fashion, which is one of the big concerns that customers have these days. And then the knowledge graph, I mean maybe not
[04:06] like graph in the traditional sense, but the way those connections feed to each other, is there a way we should think about that or like try to even like mentally visualize that process? Yeah, no, it it it's a great question. The way I would think about it is that knowledge graph is stitching together
[04:22] and identifying relationships between data assets, code assets like dashboards or Genie spaces or you know, take a take a notebook as an example. It's also associating that with individuals. So,
[04:39] are there sources of data that are trustworthy, but are there also individuals that are trustworthy? So, if I've written a you know, a dashboard on a particular topic and Genie gets a question about that topic, the ontology will help the LLM
[04:56] understand, oh, Raj is actually an authority figure on these five topics, you know, it could be you know, STP pipelines, it could be generative AI, it could be machine learning workloads or what have you. And so, that guidance, that relationship
[05:12] between assets, individuals, and their authority figures gives Genie essentially insights into what data signals it can trust so that it can provide the best possible answers to answer the question that that user has at that moment. So, all of this then knowledge, who is
[05:29] supposed to be to build it? I'm just curious, are we expecting like the business users to come and It's a great question. So, I think one of the historical challenges with ontologies, like this is technology that's been defined by library scientists and knowledge management experts for 25 plus years.
[05:47] The problem with legacy ontologies is they tend to go out of date. They get stale. There's drift. Um there's only so much knowledge that you can curate. Now, the UC Glossary layer is going to be designed so that it's as easy as possible for that glossary source of truth to be authoritative and stay up to
[06:04] date. But for that long tail of knowledge that exists within an organization, it's impossible for humans, data stewards to keep that knowledge up to date. So, I think the big innovation for the Genie ontology is it's always monitoring the activity in the
[06:19] lakehouse, all of the consumption patterns, the queries that are submitted, the ML models that are being run and trained, um and individuals that are accessing and creating new artifacts. And so, that uh ontology layer is always up to date. It's always
[06:34] permission aware. That's the other thing that's absolutely critical is you may have access to certain data that I don't have access to. Right. Your subset of the graph in the Genie ontology should look different than mine because should be aware of and have access to things that I I shouldn't have access
[06:50] to. And so, keeping that that knowledge graph up to date, have it be a call aware and personalized to each individual in the organization is one of the key innovations that uh you know, makes Genie ontology I think state of the art and really built for the agentic
[07:07] era, which is really what we're the problem that we're trying to solve. Right. So, you have basically the combination of what Bilal was talking about, which is these uh agents have context for Databricks. So, they're the best for building agents within Databricks. They have context on
[07:22] the business, whatever the company provides, and maybe that takes a little uh work in the beginning for it to learn that context, and has the context of who's actually using it and trying to access data. Yeah. That's exactly Yeah, no, all of that comes together. And I'm glad you mentioned connections to the business.
[07:40] Like, historically Databricks and Unity Catalog have visibility to assets that you ingest into the lakehouse, assets that you create, or federated data sources. I can federate to an external Iceberg catalog, or say a SaaS platform. The other thing that we've been building
[07:56] with the Genie team is connectors to unstructured business knowledge, so that business context can come from all of that knowledge that exists in your organization that's not structured and doesn't live in Databricks. So, Slack channels, Teams channels, stuff in
[08:11] SharePoint, stuff in uh you know, Confluence, or or um any of those unstructured data platforms that have enterprise knowledge, Genie ontology has visibility into that as well. And so, it can infer things about
[08:27] what's happening at this very moment. It can compute answers based on what it knows and what's current, as opposed to simply being a smart search engine, which is what a lot of LLMs end up resulting in is they're great retrieval engines, but
[08:42] they're only looking for is there a document that can answer this question, not hey, if I want a an update for sales in the last month for this product line. There's data there's events that have taken place over the last 7 days that
[08:57] are going to inform that answer. You want an agent that knows not only what that product line means and and what it's about, but also what's the source of truth for that data, and that's where the ontology comes in. What you're describing is clear that now when we are talking about ontologies,
[09:13] not linked with Genie space as it was before, right? That was one of the biggest questions we had Uh. with Genie spaces when we were really talking about it with our EMEA like European customers, right? It was like, okay, whatever I'm doing is on this specific space, how can I expose it to everything else? Now what you're
[09:28] describing actually solves for that. It absolutely does. And I think the best part I I had a bunch of customer conversations earlier today, in fact. Customers that have spent time curating and defining these Genie spaces, that's actually super valuable work because those Genie spaces are authoritative
[09:44] sources of knowledge for the Genie ontology. So, if you've got a curated Genie space that answers finance questions about a particular product area or marketing questions for a set of campaigns, that's now authoritative knowledge that the Genie ontology has access to and that Genie 1 can take a
[10:01] look at and and generate answers based on that information. Same thing with UC metric views. Again, authoritative sources of truth in Unity catalog for business KPIs, that curation, that modeling work makes Genie ontology that much smarter. And so, I really think
[10:17] it's the combination of sort of this curated, human stewarded, authoritative knowledge coupled with all of the things that Genie ontology has visibility to, that's that virtuous cycle that makes Genie 1 so powerful. So, that maybe you don't have to curate as many Genie
[10:33] spaces going forward, but the ones that you have are going to be super valuable. Exactly. That's exactly right. And then you you also talked about certification of assets, right? Yes. Can you explain a little bit what does this represent and how it is fetched into the ontology to re-rank, maybe?
[10:50] Yeah, no, that's exactly right. So, um when we talk about the ranking algorithm within the Genie ontology, it's called onto rank, so ontology rank. And the idea is it's going to take all of these signals, again, authority and trust signals from individuals, how popular,
[11:06] you know, broadly consumed are particular dashboards. It's going to take authority signals from things like these glossary pages and metrics. The additional signal that stewards or even agents can apply is a certification tag. And so, that certification is almost it's almost like a little Good
[11:22] Housekeeping Seal of Approval. It's like, you can trust this. This is the certified dashboard for revenue analytics for this business unit. Or, this is the certified table for insights into customer performance, or a metric
[11:37] view, or what have you. And so, again, it's those small additional signals that humans can apply to data and workspace assets in Databricks that can, again, improve the quality of the answers that Genie 1 ends up providing, and again, be
[11:53] as token efficient as possible because the LLM's not searching across everything. It can kind of narrow its search to trusted assets to the extent that those assets have been certified. So, this all works together into a pretty seamless fashion. So, you're almost creating,
[12:09] like when you get the results, like a re-ranking system that um you know, the it's all learning in that time period. time. And so, how exactly does that work, like taking in the new information? It So, Genie and Genie code, which powers a lot
[12:27] of the underlying capabilities, that's building this ontology, because it's connected to our data platform as opposed to a bolt-on ontology or knowledge graph, or what have you, it's got visibility into all activity within the lakehouse. New data that's been
[12:42] ingested, you know, schema evolution for critical pipelines, new artifacts that have been created. And so, as a result of that, if things change, the Genie ontology is aware of it. It's going to re-rank stuff on an ongoing
[12:58] basis. And so, as trust signals change, say somebody leaves the company, or maybe, you know, product A replaced product B, Yeah. that signal is going to exist in your data platform, in your pipelines. And as a result, the Genie ontology is going to become aware of that, it's going to
[13:14] self-update. And again, you're going to maximize the likelihood that you're going to get an answer that is trustworthy, and you know, believable, as opposed to the The doing what they want to do. Like if you're on ChatGPT, they're going to find an answer whether it's true or not, they're going to give you something and
[13:31] you don't want that in your enterprise company. Right. And Raj, that was a lot of awesome information that was diving in pretty deep. If you can give the 60-some thousand people that are registered right now and watching one sentence of like, "Hey, how like what
[13:48] was the takeaway? What's the walkaway?" that you want them to have? I think the takeaway is Genie Anthology is going to do the work for you. Get your customers on automated identity management so you can make Databricks available to all of your employees. It's
[14:04] super simple. It works with Entra, it works with Okta, and then turn on Genie 1 and give your users access. If you add context with metrics and Unity Catalog glossary, it's going to be even better, but you don't even have to do that. You can just get started and
[14:21] point Genie 1 at your lakehouse and unleash the power of Genie 1, the best co-work agent you could possibly have access to. It doesn't hurt to try, I think. Yes, exactly. You just got to get started. All right, Raj. Well, thank you so much for joining us. Good to see you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.