Sponsored by: Collibra | From Pilot to Production: Governing AI Agents on Databricks with Collibra

Summary

This video explains how Collibra acts as an enterprise context layer across a Databricks ecosystem, bridging structured and unstructured data to provide the metadata and governance context that AI agents need to move from pilot to production.
Collibra integrates with Unity Catalog through bidirectional metadata sync and exposes an MCP Server that functions as a governance agent alongside Genie, allowing AI systems to answer questions about where data came from, what policies apply, and whether it belongs to a data product.
Three types of context are covered—connected metadata via Unity Catalog sync, actionable context via the Collibra MCP governance agent, and embedded context for AI-ready data products—with a live adverse event analysis demo showing Genie and Collibra MCP working together.

Chapters

FAQs

Why do organizations need both Unity Catalog and Collibra for AI governance?

Unity Catalog governs technical metadata—tables, schemas, access permissions—while Collibra provides business context such as data glossary definitions, data product standards, lineage documentation, and governance policies. When an AI agent needs to answer questions about what data means or whether it can be trusted, Collibra's MCP Server fills the gap that Unity Catalog alone does not cover.

What is the Collibra MCP Server and how does it work with Genie?

The Collibra MCP Server acts as a governance agent that can answer questions about the data itself—its origin, associated policies, and data product membership—running alongside Genie which answers questions from the data. Together they form a two-agent system where Genie handles analytical queries and Collibra MCP handles governance and lineage questions.

What is bidirectional sync between Collibra and Unity Catalog?

Bidirectional sync allows tags and governed metadata to flow both from Unity Catalog into Collibra and from Collibra back into Unity Catalog, keeping both systems aligned without manual reconciliation. This video demonstrates how governed tags managed in Collibra can be pushed into Unity Catalog so that downstream Databricks assets inherit the correct business classifications.

What are AI-ready data products in the context of Collibra and Databricks?

AI-ready data products are a roadmap capability described in this video where Collibra embeds governance context—including lineage, quality scores, and policy tags—directly into data assets so that AI agents can evaluate trustworthiness before using the data. This embedded context layer is designed to make the gap between governed data and AI-consumable data as small as possible.

Full transcript

[00:08] Welcome everybody. My name is Jeff Coass. I'm a staff customer engineer with Collibra. The only thing between me and you guys going to the airport is me. It's the last presentation. So, my mom told me they save the best for last.
[00:21] I think that was she was talking about desserts, but anyway, I'll go with it. Appreciate everybody coming out. Um today I'm going to be talking about how we can go from pilot to production uh specifically around AI agents with
[00:33] Databricks and uh also Collibra. So, my background, I spent I've been with Collibra about a year and a half. Uh before that, I spent a little over a year at Databricks. And before that, I spent many, many, many
[00:46] years at Informatica. So, I've done a lot with data integration, with data quality, um master data management, GenAI projects, and of course, governance.
[01:00] If there's any questions, I don't know if I'll have enough time for questions during the the presentation, but after, please come talk to me if you guys have any any questions. If you want to get more information about anything. So, going to pilot to production,
[01:13] or first thing everybody's talking about is context, right? If I'm going to go from pilot to production, I need context in my uh chatbots. Now, for us, we typically think of it is kind of
[01:25] three different types of context. And I'm going to do a demo. I only have a few slides. I'm going to try to spend as much time showing you guys live demos. So, the first thing is, you know, I have
[01:37] Unity Catalog and I have tags over here, or governed tags, and I also have Collibra. How can I move one to the other and vice versa? How can I push from Collibra into
[01:49] um Unity Catalog? So, I'm going to show that uh in a demo. The next thing is we call it actionable context. And to me, this is I've been doing this demo all week and it really seems to resonate. And it
[02:03] helps to answer the question of, why do I need two catalogs? I have Unity, why do I need Collibra, right? So, what I can do is I can have um Genie space to help answer questions
[02:15] of the data. But then I can have our MCP server answer questions about the data, where it came from, is it part of a data product, what standards and policies are associated with it, right? So, think of it now I have a governance agent. How
[02:29] can I trust this data if I have questions, right? Where do I go to? And the final one is embedded context and this really this is um this is on our road map, so full disclosure it's not released, but um I wanted to present at
[02:42] least a slide so you can see here's where we're going with with our integration with Databricks. So, as I mentioned the first thing we're going to talk about and I'm going to
[02:54] show you in a demo in a second is the connected uh metadata. So, you can see over here I have Unity catalog, I have my descriptions, tags, policies, right? Documentation. Now, we can pull that into Collibra, but
[03:08] also you if you have more of a data mesh framework and your governance framework, we can also push that push um metadata from uh Collibra into Unity.
[03:20] So, let me get out of here. Let me get out of here for a second. And then So, notice over here, so I have my own this is my own Databricks instance that we use at Collibra, my demo instance
[03:33] over here. And you can see I have some tags right over here. Now, here's my Collibra instance, here's my my platform, right?
[03:46] And I've already ingested, so if you've if you're not familiar with Collibra, we have what's called Edge and Edge does the heavy lifting. It does the ingestion from um from different metadata sources, things
[03:59] like obviously Unity Catalog. We're agnostic. We also support Snowflake, BigQuery, literally everybody, right? So, we want to have a holistic approach, holistic diagram, so you can see this
[04:11] knowledge graph across your entire data estate. And that also includes dashboarding um uh things like Tableau, things like Power BI. So, not just dashboarding tools, but also reports as well.
[04:34] what I've done is I've I've leveraged Edge, I've created a connection typically through JDBC, so I've set it up. And then over here, you can see I have a couple of different tabs. I have inbound. What do I want to bring from Unity
[04:46] over here, and I'm only bringing main path um main Jeff, right? Cuz I only want to bring my my schemas. I don't want to bring anybody else's schemas into my my instance,
[04:58] right?
[05:12] create all sorts of different um assets and curate them within Collibra, and then push them into um directly into uh Unity Catalog. Now,
[05:24] I'm going to open up one of these right here. So, if I start to add different tags, I can see here's all the different asset types. Now, what what what I'm doing is I'm setting up something to say, if I
[05:37] make a change to a description or something at the database level, it will automatically propagate into Unity as a tag. And then I define whatever the tag is. I
[05:49] can define whatever name. So if I choose something like database over here, I can see I have all sorts of um attributes that are filtered just on database. If I choose something over here, like table,
[06:02] my attributes, notice they change. Right? So it's very, very simple for us or for you guys to to um create whatever tags that you want to move whatever metadata you want into
[06:16] Unity Catalog. All right, let me go back into my slides. So um one thing, you know, when you're moving from from pilot to production with agents,
[06:30] a lot of times it's really difficult to figure out what you have, what the status is, um you know, are there are there agents that are at risk? Are there agents that need reviewed? Um and again, we take a holistic
[06:43] approach. You know, we have a really tight relationship with Databricks, but we also know, when I worked at Databricks, some of my customers had um uh they were deploying models in Sagemaker and Bedrock. And the weird
[06:56] thing is, the the um AI team didn't know what what different lines of business were doing. Right? So it's kind of shadow IT going on. And so what we do is we can pull in things from like Foundry, things from
[07:09] Bedrock or different agents. And so I put together a little a a quick minute video just to show you guys what
[07:21] our registry looks like over here. So you can see I have, you know, a whole bunch of different assets, use cases. I can have a trust score. You can control the weighting of the trust as well
[07:33] uh of the trust score. I can filter on whatever it is I'm looking for. Now, what's also interesting is we're pulling in through Open Telemetry um some of the metrics from Databricks
[07:45] from their, um, for the different agents. So, now I can see, you know, from a quality perspective in a single pane of glass what's going on with, um, with some of my agents. We also keep track of token consumptions
[07:59] which is also very nice. And so, over here we can see the different KPIs. I can see, um, you know, my average trust score, uh,
[08:11] which AI monitors, you know, I have a pass rate for that. I can start to drill in. It's very interactive. So, over here I can see the trust score over time as well. I can see things
[08:24] like, you know, if I care about, you know, safety monitor, right? How is that in my current cycle versus my previous cycle?
[08:38] And then I can also see the tokens consumption as well. All right, let me keep going. Now, one of my favorite parts of this presentation is Genie with MCP server,
[08:50] right? So, as I mentioned at the beginning for, um, uh, for more actionable type of context, what I really want to do is I want to ask questions. I want my consumers, my
[09:02] data consumers to ask questions of the data, but if they have questions about where the data came from, it shouldn't matter. They should be able to do it in a single, um, in a through a single supervisor agent.
[09:15] So, let me jump out of here and get back into another another demo. Now, full transparency, I already ran this. Sometimes it could take a little bit. I only have 20 minutes and I've already killed more than more than half of it.
[09:27] So, I'll just walk you through what I ran earlier, um, about an hour ago. So, what What at is I have a supervisor agent. I'm over here under my agents.
[09:40] And then under the supervisor agent, I have a Genie space that I have one table for pharmaceutical company, one of our customers that was joint with Databricks. The Databricks solutions architect helped build this with us. So,
[09:54] she did the the Delta tables and then I did the Genie space on top of it as well with their MCP server. So, over here you can see I asked the question for the drug test prior, what
[10:07] are the top emerging safety signals of the last 6 months broken down by age group? Show me the most frequently reported adverse events by age. And Genie is amazing at taking English,
[10:19] figuring out the sequel, and bringing back really, really nice, useful information. That's why there's so much interest in Genie, right? Now, I can ask it maybe I'm maybe I'm new to
[10:32] the company or maybe I'm new to this data set, but see I have event type over here. What if I need more information about that? So, I can go and and I asked the question over here. I said um
[10:45] what does event type mean? And you could see it it just it still used Genie. So, it said, "Okay, well based on on the data, I'm going to answer the question." But I have a data steward that curated
[10:57] this. So, there's a business term in my glossary with very specific information and details about about this asset, right? And so over here I said use Collibra to answer my
[11:10] question, please. Now, notice over here it went out and it brought back from Collibra right here. Perfect. According to Collibra's pharmaceutical glossary, here's the official definition. Right? I
[11:23] don't want some generic information. I want official curated information. Maybe I have something that I want everybody all my employees to know, right? That's why you want to have data governance. So,
[11:35] now I keep going. What if I saw something like con commit medication? What does that mean? So, I go through here and again, it's going to start answering questions. And again, this is all the information.
[11:48] It even has context, right? It has context because I curated this asset and we're going to see this in a second when I show you guys our knowledge graph, how everything is related. To me, it's still table stakes, right?
[12:01] Ask a question of business glossary. Okay, that's that's good. What if I want to use this specific asset in you know, as a data product for creating dashboards, right? So, I can
[12:13] ask another question. Is it contained in a data product right here? And so, the supervising agent's going to be yes, according to Collibra um metadata, it's contained in a data product. And now it's going through cuz
[12:26] we exposed our various tools to find out more information about this adverse events data product. Notice I have key attributes, the business value, the business case, the description. All the information that we curated is here
[12:40] now. So, now I can ask questions of the data and questions around my govern through my governance agent, right? I can see it's in development. Um the target delivery date, it's part of a domain. Um it has related business
[12:53] terms, the architecture, who's responsible. I was the requester. Um we have a compliance officer. Right? So, what if I want to use this now? I keep asking questions, right? Are there are there any standards or
[13:05] policies related directly or indirectly to adverse events data product? And now this is a long one, so I'm going to just scroll all the way down cuz it took a while for it to answer.
[13:21] This is also nice because you can see it it it can use multiple tools if it needs to, right? AI is smart enough to say, okay, well, I got to get this. Maybe I need to get some more things. Maybe there's indirect relationships." And then eventually I get Oops. Over
[13:35] here. I have standards and policies, right? There's MedDRA standard over here for medical coding. I can see the last review date. I can see the purpose of it. Key requirements. The reason we're
[13:47] seeing this is because we have governed assets in Collibra. Now, I also have other policies as well, coding and management policies. I have
[14:00] ConMed usage policy. I have a data contract. It even figured out there's a data contract and it's showing me here's the SLAs right here according to the data contract.
[14:14] And then I even see the business terms governed. And it's really nice because it In this case, it gave me a summary of it. And my final question is, "How can I ensure that my adverse events data product adheres to industry standards and policies?" And what it basically did for me is it
[14:27] gave me an implementation checklist. It literally built a governance framework, like a whole series of steps. And it did that because it said, "Okay, you have this type of data according to
[14:40] Genie, this these policies and standards that are already in Collibra, and now industry standards, I'm going to go through and make this recommendation." Which is pretty cool if you think about
[14:52] it. Now, I only have a couple minutes left. Um let me go into I'm going to show you how easy it is to create this agent really quick, and then I'm going to shift gears. So, to create the agent over here,
[15:04] I literally create a supervisor agent. And then I add over here my Genie space that I have. And then it's going to take about 5 or 10 seconds to start initializing it and
[15:17] building it. And then I go back in here and I say let me grab my connection. My UC connection is my my Collibra MCP server.
[15:29] And then I say, "Okay." Notice I have drug safety adverse events, Collibra right here. And then I can say, "Tell me about your capabilities."
[15:46] And here's all the capabilities it can do. So, again, if I just have Genie, I have just data analysis and query. I have all this other information. We have a tool for lineage, right? Where did the
[15:58] data come from? Uh you know, maybe if I make a change to this data, what what um Tableau uh dashboards and reports are available? What Power BI reports are available? Are they part of a data product, right? So, I have all this
[16:12] wonderful information that's available to me. Now, I'm going to keep going cuz I look like I have I'm running out of time. I do apologize. Let me go back over here.
[16:30] So, as I mentioned, now one of the things we're working on is instead of me doing what I just did, manually going into Databricks, now what I want to do is I could be a data product owner and I could through our
[16:42] the um the Collibra AI and the Databricks uh AI or API, I should say, automatically spin up a Genie space, give it all the context, the semantic
[16:54] context it needs, so I have all this information, all done through a a a data product owner. So, how nice is that? I don't even have to go in and and touch um
[17:06] I don't have to be like an admin and and and touch uh Databricks. It's all going to be spun up for me. And again, that's, you know, Safe Harbor Act. I don't know when it's exactly coming out, So, um
[17:21] All right. Last thing I want to talk about. So, we talked about, you know, if you're going to go from from pilot to production, it's typically crawl, walk, run. I always tell my customers, don't try to boil the ocean, right? Start with structured data, bring in your models,
[17:34] your AI agents, start to bring those together, and then my customers have the aha moment. Well, I have all these rag applications. I'm building a chatbot. What about unstructured data? What about files,
[17:47] right? When I worked at um at Databricks, I had one large insurance company, and we had to wait 2 months for a chatbot. I had my RSA, I had everybody on board, I'm a Zoom session,
[17:59] and my customer was like, "Wait, we don't know what files we don't to use." I'm like, "You guys know, you know, how to do like forest random classifiers from Hugging Face, but you
[18:12] can't find files?" They're like, "The way they explain explain it to me is a dumping ground, right?" So, what we see, so we acquired a company called Deezy Labs, and we loved them so much, we kept the brand. So, what Deezy Labs does is it
[18:25] allows you to point to things like SharePoint, these dumping grounds where you have tons and tons of files, and it will make sense of it. It will leverage AI to figure out what tags should you have,
[18:37] and then from there, we can generate the metadata, and then, you know, if I'm building a chatbot for like legal or HR, I only want legal documents. I don't want sensitive documents. I want them
[18:49] fresh within a certain date range, right? So, what we'll do is we have um we have pre-built connectors to over here to different sources. I you typically show uh SharePoint, and then the curation stage, so actually, let me
[19:02] back up. We do the OCR and the chunking first. If you guys already do OCR and chunking, we connect to the vector database, so that's fine with us. And then over here in curation stage, that's where the magic really happens.
[19:15] That's where we can leverage AI. If I don't know what I don't know, it can auto suggest all these different tags for me. Now, the other thing is I have one retail customer that wants to do 150 petabytes this year alone, then twice as
[19:28] much next year. I can't even imagine what the cost for tokens for that would be to ship everything to a model. So, we can do um pattern-based, but not just pattern-based, pattern-based with context. And why that's important,
[19:40] you'll get much better accuracy. So, if I'm looking for a you know, nine digits, it could be a social security number, could be a phone number, whatever. We can figure out automatically suggest, here's some of the keywords for you.
[19:53] And then the last thing we do, um we make it look like a structured format over here, almost like a table with a file name and all the the um all the metadata with it. We create
[20:05] what's called a data slice, think of that AI-ready data product. And then from there, we can you scan it once, I could push it to Collibra to be um governed within Collibra, I could be part of a data product or AI use case. I
[20:19] can enrich the source, I can hydrate back to SharePoint as well. Or what I could do is we have an open SDK uh that we we provide, and I could have be in my Databricks notebook,
[20:31] and I could call out, connect to the my data slice to get the 10 files that I need. Now, the other thing we can do with the data slice, we put on a set schedule. So, as new files come in a SharePoint, we'll figure out what's a
[20:43] new file or something gets deleted, will automatically refresh the data slice, so you don't have to worry about your your chatbot getting stale. Now, it looks like I'm out of time, so I do appreciate everybody coming.
[20:55] Um we'll be around for another few minutes if you have any more questions. Um again, thanks so much. Appreciate you guys.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.