Skip to main content

Open Source Unity Catalog: Getting Started, Best Practices and Governance at Scale

Summary

  • Unity Catalog open source: One year after open sourcing, this video covers UC OSS capabilities, latest features and the growing ecosystem for data and AI governance.
  • AI functions integration: Discover how UC OSS integrates with GenAI and enables SQL UDFs, AI functions, and ML workflows on the Databricks Data and AI platform.
  • Governance at scale: Learn best practices for using Unity Catalog to govern hundreds of thousands of tables, models and AI assets in modern lakehouse environments.

Open Source Unity Catalog: Getting Started, Best Practices and Governance at Scale

Watch: Open Source Unity Catalog: Getting Started, Best Practices and Governance at Scale
Talk by Ben Wilson, Sr Software Engineer, Databricks ; Tathagata Das, Sr. Staff Software Engineer, Databricks
How to use UC OSS, what features are available, and intro to the ecosystem. We'll dive into the latest release and get hands-on with demos for working with your UC data and AI assets — including tables, volumes, models and AI functions.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is Unity Catalog open source?

UC OSS is the open source version of Databricks' governance solution, providing catalog, access control and governance features. This video shows how it has evolved over the past year with new AI-focused capabilities.

What are AI functions in Unity Catalog?

AI functions enable you to define and register GenAI capabilities as reusable SQL functions, integrating large language models directly into data workflows on the Databricks Data and AI platform.

Can I use UC OSS for free?

Yes, Unity Catalog is open source and free to use. This video demonstrates how to get started and shows the latest features available in the OSS version.

How does UC help with governance at scale?

Unity Catalog provides centralized control of data assets, lineage tracking and enforced access policies, enabling organizations to govern thousands of tables, volumes and models consistently.

Full transcript

0:00 Introduction and agenda

Good morning everyone. Has everyone had their coffee? Still having their coffee? Still waking up. Someone needs a punch in the face. Okay, at least that hopefully that woke it up. Uh, okay. Everyone welcome to the I guess the first or second day of the day of the data plus AI summit. Everyone here is clearly brave enough to start the attend the first session and not sleep. So thank you very much extra thank you for coming in in this first session especially to those who are not morning people and that includes me.
Uh so here today we are going to talk about the unity catalog opensource project. Exactly a year ago this time at the summit in the keynote mate Zarya open sourced the unity catalog project and here we here to talk about what happened in the last one year what the fun stuff we have been building and uh everything under the sun related to unity catalog open source project. So to introduce myself I'm an engineer at data bricks.
I have been working on the spark ecosystem for a long time. Uh from RDDs, Spark streaming, structure streaming, delta lake have been involved del involved with all of them since their beginning. Now also involved with unity catalog. Uh and Uh I'm one of the core ML flow maintainers. It's usually nothing to do with Unity catalog, but we also do open source support for ML packages at data bricks. And one of the packages that we support is Unity Catalog AI, which I'll talk about a little bit throughout this presentation.
Awesome. Okay. So, we're going to talk about from the basics what makes a great catalog and why is UC to motivate why is UC the right catalog for you for the next generation of workloads? What can you do with UC today and what have we been doing in the UCOSs project till over the last one year? So let's start with the very basic first question. What makes a great lakehouse catalog? To understand this, let's first start by asking the question, what do modern data plus AI workloads look like?

2:10 Modern data and AI workloads

Because they're evolving, right? Today is a new world genai and everything which means that all the technologies that are needed to support the new kind of workloads that are coming in needs to evolve with the to adapt to the new demands of such workloads as well. So what does a modern data workload look like? To understand that let's start a company together. Let's sell some shoes. So imagine there's a website I'm I don't want to take any names but uh myshoe. com let's say not Nike not Adidas uh myshoe. com you have a website where customers can buy shoes so you have transaction data coming in and you want to build this backend pipeline where everybody in the company as well as end customers can ultimately deal with the data that is coming in.
So how would you traditionally start building it? You have transactional data coming in. You will have a data engineer which preps the data, collects the data from various different sources, build up a ETL pipeline for bronze, silver, gold tables, takes the data, cleans it up, structures it, puts it in structure tables like orders, customers, etc. So that other folks like data scientists can generate more cleaned up versions, reports, etc.
The golden layer of the medallion architecture on these tables. And then you have different sets of people in the company u uh who want to read these reports understand what's going on with our with the our revenue and u are shoes selling well how much profit we have how much uh costs we have etc etc so financial analysts want to look at those reports etc but now their job is to forecast that how is our business growing so they want more advanced stuff and that's where classic ML models come in like this we have been doing for a while that I want to demand project uh the financial analyst once here's the projections of demand and financial so it goes back to the data scientist or the ML engineer and like okay we need ML model the ML engineer takes the greatest and best tools in the ML ecosystem and builds here's a model um finanes happy okay I can make projections and stuff but that also means that there's now a growing set of reports like hundreds of tables people are dealing with etc now the CEO comes in talks with an analyst this is too complicated I want it to be dumbed down I want to ask simple questions and then there are also customers who are asking okay they need customer service where's my order why is my order late uh what's the ETA etc so all of these customers going to ask questions.
Warehouse managers who's maintaining all the shipping and stuff of the shoes, they're going to want to know, okay, what are uh how much capacity do we need? How much do we need to reorder from the factories and stuff? How many orders are in transit, etc. And that's where the new geni comes in. We need uh today we want to build AI agents that can answer these questions without them having to deal with SQL or Python or all the lower level tools.
Just ask questions and stuff. And so this is what a new age pipeline looks like from end to end. Let's see how we're going to build this. What are the technologies we will need? On the ETL pipeline side there is the traditional technologies you have to deal with tables. So there's hive meta store as a catalog which may have different you may have table in different formats you have getting data some from some things in JSON some things in park some things in a more modern format like delta lake or catalog or or iceberg so a catalog has to handle all these different kinds of data tables and then these tables will be stored on cloud storage S3 Azure data lake Google GCS HDFS you name And then they will be have you know plethora of different processing engines to process that data multiple tools whatever you you want to use whatever tools is best for the job you need to do on the data then on the ML side actually you should talk about on the ML side he's the ML AI guy not me yes we were talking about how what CD was talking about with regards to the forecasting models so your data science team is going to build some let's say they're building profit models relatively easy to get started maybe they do some stats model stuff later on or some deep learning approaches.
You when that team is building those models and getting ready to deploy them so that you can use them for analytics it's very important to know what is in staging what has been validated and what is about to ship to production. So, MLoflow builds a native integration with Unity catalog both managed on the data brick side as well as open source Unity catalog directly so that you can manage access and promotion to different staging environments to eventually production and when you're going to replace those models you can then retire them and have a lineage tracking of that process.
So, it's super critical to have that as part of the that's the Moflow Unity catalog model registry integration. But also with AI agents, it's very important to make sure that we have the same sorts of rigorous controls about what is that robot effectively actually accessing. Yeah, in fact, you actually jumped ahead that what how Unity catalog solves it. Let's understand in the world without Unity catalog you have these previous Unity catalog you had these different sets of uh systems like MLflow weights and balances that used to track models and registries that is completely different from hive catalog right so there are like two different worlds altogether so there's the table world and there's the model AI world and the two can never meet well not never that's the point of this So and then you have to deal with multiple different table formats.
So hive catalog has been this def facto standard for almost two decades now but technologies underneath the formats have evolved. uh classic table formats JSON, CSV, park these are still relevant but they do not give all the things that a modern table workload needs like you don't get asset guarantees you don't the uh you don't get whether the data on the file system is actually in sync with the metadata in the catalog all of these are problems that every data engineer has to deal with and so that's why modern table formats like delta and iceberg needs more from the catalog that often the catalog is unable meta store and this traditional stuff is unable to give.
You need to evolve the catalog the table catalog itself for these new age formats new to add new capabilities to it. And then if you zoom out again, every persona in this there are like lots of people involved here and you have data that is everyone needs to access but you don't want critical data to be accessible to everyone. So every person here needs different accesses and and the system needs to ensure that they don't have access to data that they shouldn't have access to.
For example, a data engineer is creating all the tables, managing all tables. They need to have access to the the bronze and the silver tables that they are working on. So they definitely need access to the data but they don't need access to anything else like like uh more uh serious data like revenue projection etc. They shouldn't have access to that. Similarly, data scientist actually needs access to some of the some of the silver layer tables to generate the reports and stuff but the data scientist should not have access to for example things like customer data like that is PII data data scientist may not you may not want as a infrastructure provider you may not want them to access to have that financial analyst customers they all need different access for example customers they should have access to their own data so if They want they should to be able to access and query their own orders but not others orders.
They need to be able to update their own customer billing information or addresses or whatever but not anybody else's and they need to have access to the agent as well. Warehouse managers again a different model. So everybody has different thing but on the other hand AI agent needs to have access to all the data because it has to answer all the questions right. So the AIG needs to be smart where depending on who is asking the question, it needs to assume that caller's permission and only access the data that caller is allowed to access.
So this is a new world of uh access control that is needed to really secure the data that it goes beyond tables and these traditional models. And the other problem I want to highlight is even within the domain of tables or anything that is on the storage today the standard is you need to configure your engine spark flink whatever with permanent cloud credentials permanent S3 access key secret key pair things like that uh permanently in your engine and so it is makes it much harder to change the access you have to go update all your applications update configuration files in multiple different places.
If you want to change somebody's access, you want to revoke someone's access because the company the person left the company and or move to a different role, you want to revoke access that those are definitely much harder because of permanently configured credentials. So to summarize, data and a assets are siloed into different parts of the company and it's very hard to share things between them or have a overall view of what are what who has access to what and so governance is fragmented which increases administrative cost and then modern table formats are not supported by the traditional cataloges.
So you don't get sufficient transaction guarantees until you do a little bit more or you don't get or you can get for instance delta lake you can get transaction guarantees but there's a risk that your catalog and the delta table on disk may be out of sync. So our goal while building this Unity catalog was to give users the flexibility to use data and AI assets without sacrificing security to have a single uh pane where you can look at the overall and see what are all the assets that are present and how am I able to govern them.

13:07 Unity Catalog's core mission

So Unity catalog became started as a multiformat from day one multiformat multimodel and a unified platform. Let's walk through the problems once again. So we are taking from these disjoint silos and having Unity catalog be the single source of truth that tracks all assets from tables to models as was talking about a little bit earlier. And then all accesses from every person, every engine needs to go through UC so that UC can dictate whether you should have access or not have access. that can do the access control in one single place and so therefore you can control it in one from one single place and then for UC to have this level of control you need to be able to trust UC.
UC is a trusted entity where you are putting all your security policy in UC which also means that UC should be the right person to dictate whether a particular user or a particular engine that needs to access the table should have access or not. So the UC is the one who will get which will do vend credentials temporary credentials to engines s such as spark to access the tables based on the user who has queried. For example, if the warehouse manager wants to query about order data only he is only allowed to query orders.
So only queries that are about orders and nothing else not customer data and stuff only those queries UC will say you're authorized to your the spark query is authorized on behalf of the warehouse manager to query the order table. So unicettle will give it temporary credentials and the while the spark just for the duration of the spark query. I'll go into details later. And then the third problem is uh better support for modern formats.
So we're building in Unity catalog new APIs that will allow Delta Lake iceberg uh iceberg already has rest catalog. support iceberg rest catalog within Unity catalog rest catalog AP spec within unity catalog but we are building all this additional stuff that is needed for modern formats to have more support for transactional updates while keeping the catalog in sync of what is what all is going on or all the changes going on in the table.
So if you're changing the schema of your table, your catalog schema will also reflect what is the current schema of the table without losing support for existing classic formats. So you don't have to choose that I will pick only this format and nothing else and stuff. So give you the maximum flexibility. So we have unified data in assets. We have unified governance and we are giving modern table formats the support from the catalog that they need.
And so there in this talk we're going to cover not everything but this is the high level overview. The goal is to provide governance discovery eventually lineage uh monitoring overall across all these different assets tables views volumes I didn't go into volumes but um functions models some of these we're going to go in so what can you do with ucoss to understand UC to understand the object model how does how are these different data AI assets stored in UC we have the catalog that is uh there's a three-level name space.

16:47 Object model and configuration

So you can have any number of catalog objects my catalog one my catalog 2 inside it catalog there can be any number of schemas databases and inside it schema schema and database is is equivalent term here and inside the schema you can have tables volumes models functions other things and then independent of this three-level name space there are other objects that are also present there are things like credentials and external location I'll go into detail I'll walk through them but they are important for that goal of the UC being able to have access to all the data or all the storage so that it can vend selectively vend access to engines on a temporary basis we'll walk through that so for tables let's let's start with the basics what does a what does a table entity look like so at the very basic level it has the properties it has is it's it's it needs to know what is it's part of a catalog it's part of with schema.
It has a unique identifier. It has all the column information like what is the table's schema? What are the columns inside it? Properties. What format is it? CSV, JSON, parket, delta. What type of table is it? Is it external or managed? Like there are definitions here whether is it fully managed by the platform that UC is part of or is it something that is external to it and UC does not have full control over it or views there are different definitions I will not go into detail but ultimately but the most one of the most important bit is the location of where the table is and so you can have the there's a hierarchy in the catalog that ultimately points to the hierarchy in the cloud path where the where the table reside and then there are APIs that UC provides rest APIs where there's also SDKs and stuff that allow you to operate on these tables on the meta at the metadata level not at a data level because you engines are supposed to operate at the data level.
You the engines will ask the catalog to get the metadata of the table before operating on the data. So to create the standard crowd operations create get list tables and stuff but the mo the interesting bit we have added here is you can gen ask UC to generate temporary table credentials and as I said for that to work UC needs to have full access to the cloud store. it it needs to know whether how any user can access the cloud and UC must have full control and so for UC to know that these are the two additional entities that you can populate in UC for example if you are a uh you are let's say administrator and you want to configure that all the tables in this particular catalog let's say my catalog uh let's say I was using my catalog my schema So say I'm an administrator I want to set up that every all tables in my catalog is going to have uh be located in this particular S3 bucket.
So as an administrator I will configure this as an external location uh and a credential. So external location is it points to the path of that S3 bucket where all the tables will reside and the credential is things like IM role etc that the UC can assume to generate credentials for reading writing within that bucket. So the UC has the full knowledge of this. Now using these two information. So so the steps to configure you see for working with tables as that all of this works is first you configure the credential you configure the external location and link it with the credential that using this particular credential you can access the external location and then you can start defining tables in that particular catalog which is using that external location.
So let's look at the steps. When you ask UC to create a temporary credential, let's say uh I want a temporary credential for my delta table. What UC would do in the background is that it'll first of all it will authorize if the user you're as a user you're providing spark here is my personal access token or oath token. Spark will use that forward that to the catalog and say that I want to get credentials. I am this user.
I will get credentials for accessing this table. First of UC will check is this user valid? Does this user have access to this table? So authentication authorization I won't go into that detail here but then the UC lookup here is the location of the table. It needs to match the location of the table with any of the external location. Is my table present within an external location or not? Once it finds the match then it knows that okay UC has the capability the corresponding cred my credential and external location it has the capability to actually generate a new credential that gives that user or spark masquerading as the user access to only the delta table on a temporary basis.
So just to be clear that UC has access to the entire bucket but the user should have access only to the delta tables location. So UC generates a scoped down credential which is limited time limited by let's say 1 hour and gives it to the spark engine or whichever engine you are working with and that engine uses that scoped on credential to access only that table. it has should have access to and nothing else and that's how temporary scoped on credentials work and that's how you have governance and so if you are a you if you are an administrator and you want to revoke access to a user you update UC that uh this user should not have access to this table anymore after the temporary credential expires you don't you they will not get any new access that's how UC has full control uh In the interest of time, I'm not going to go through a demo for this, but I think there's a lot more exciting stuff going on on the models and AI side.

23:36 AI functions and GenAI integration

So, yeah. So, just as TD was talking about with table access controls as well as having a place to store securable objects that you really do care about, we extended the same thing once we years ago when we looked at what Unity catalog built the MLFlow team. We said well this actually solves a lot of our problems. Why would we go and build our own thing when we can just use this? So that's what we did. We actually used that entire structure to store what you see on the right hand side here which is basically a typical model architecture struct structure of the serialized state of a model in MLflow.
Anybody in here ever used emlow? Raise your hands. Couple people. Yeah, more people. So when you look into the MLflow UI, you actually see this. So you can actually take that entire data structure, send it to Unity catalog and have the same functionality in MLflow with stuff like promoting from different you know locations. I have a couple of things in dev. I want to push it to staging. You can use UC to control that and actually provide access controls for doing promotions across uh places.
So it's super important. Extending from that, we also looked at what Unity catalog built with some other cool stuff. So functions started off with tabular access. It's a way for you to define a userdefined function in a couple of different code you know languages of your choice. Uh we got you know most people use it for SQL uh historically and it's great. It's got you know the ability like required things like you have to define your catalog your schema the actual function name and there's a bunch of optional things metadata that are in there that became important for us later on uh for when we wanted to do the Genai stuff.
Uh but the standard APIs that are there that are part of Unity catalog, they make a ton of sense for what most people were using it historically for. I need to apply custom logic to a table in order to get an answer to a question that I have and I need that to be secure. But when we moved over to the Genai era, you know, go back in time about a year and a half, we start looking at what capabilities are there for interfacing with an LLM.
LLM, you talk to it, they tend to make stuff up quite a bit, right? So over the last couple of years, people have been trying to extend the functionality and capabilities of LLMs by giving them access to deterministic tool executions. So we wanted to build the same thing, right? Like all of our users wanted that on MLflow. So instead of us going and reinventing the wheel about, oh, how do we build a serialization platform for this, we looked at UC because it had most of what we needed.
Now for the typical basic use of these these UC functions, we just kind of define say a Python function like this like you see on screen. We have a name of that function that's going to get registered to a catalog in a schema and registered this function name and applying this to a data table. Spark is going to intelligently know or whatever your execution engine is intellig intelligently going to know how to execute this Python code. it doesn't need to really understand anything external to it because you've already defined what the type interface is.
So if it's operating on, you know, in this case we're operating on a string doesn't really matter what we call this function. It's irrelevant like as a user we're the one calling it. We should know by looking at the code what it can do. However, when you hook something like this up to a Genai agent and tell it, hey, I have this function definition and it's called sanitize input, it's going to infer what the functionality of that tool is and decide whether or not to use it based solely on the name.
So, there's no metadata here that really tells what's going on. So what I'm simulating here at the bottom on this slide is a user asking a question, hey on our shoe sales company, who like who's our most valuable, you know, user because we want to send some marketing information to them. And if this is the actual tool that it has, the Genai agent, it's going to reason before it actually does a tool execution, that's what it's going to basically talk to itself.
It's like, hey, I need to clean up this data because I shouldn't be sending like PII data. And then it'll execute this function. And it the actual agent doesn't look at the source code. It doesn't have access to it. It just has access to a name and hopefully some metadata about what that function's for. So, it's going to return after executing the agent is going to assume that it's already cleaned out all of the important stuff that this user shouldn't have access to.
But as we can see, it gives all the information you would never want to display about a customer. Now on in the JAI world, we actually need that additional metadata. We need to know what the function's name is. Yes, like and when we're talking about UC function storage, we need to know the catalog, the schema, the function name, but we also need to tell it what is this function used for? What are its capabilities? What are the argument inputs that I that need to be used?
Those are also required for standard UC functions. But in the AI world or the genai world, we need a little bit more information. So we need comments about every single argument. We also need types. And we need types because we're operating typically in Python, which is an untyped language, but we need to be able to convert between Python types and the backend JVM types. Now once we have this registered and the actual agent understands what this function's for, it's going to know what it like when to actually call this and what the expected output is.
So in the event of this this scenario, it would actually realize that it didn't actually sanitize anything but some strings. So it wouldn't even call this thing. Now, when we moved into building this out for real, because of the the limitations that were in place because UC functions were never designed to do what we were trying to make them do, we realized that we needed to extend Unity catalog. So, we created Unity Catalog AI, which is an extension on the Unity Catalog protocols that provides an SDK interface that basically sets it up so that when we define a tool, aka a Python function in order for Genai to use, it has all of the data it needs to know when to use it, how to use it, what the expected use of that tool is going to be, and then some we do some validations as well.
But it allows us to do a much more safe application. We know what it's going to be having access to. We know when a user interfaces with that Genai agent, whether it has access to certain functions as well because just as TD said, we can set those controls as well. So you have a very sensitive function that's there that you only want a subset of your company to have access to. You can set that. So most users can still talk to the agent, but for that super special, highly secretive one, maybe there's only four or five people that can talk to an agent and have that capability in it.
So with us building this SDK, it allows us to have the kind of the best of all worlds with respect to function execution in AI, data science, and the Genai era where we can have a software engineer write a fairly complex function that can be reused pretty much everywhere. So a data scientist who's trying to create feature engineering sets from maybe traditional ML, they have this function that they need to execute and apply to their data. before you see AI, that's code copy in multiple repos or in multiple notebooks and it's a mess like trying to manage that.
I know I used to do that. It's it's terrible. It's a headache. But with this, we can actually have other software engineers use that function. We can have the analysts, we can have like Tableau connected to these, you know, Unity catalog and you can have visualizations directly just using this function that's been vetted and as you see in the top right here, unit tested. We can have this set up so that as it's in Unity catalog, we can have daily checks that are like, hey, run this against, you know, this data that I have.
Make sure this function is still doing what I expect it to do. And we can connect it to an agent. Three minutes. What's that? Three minutes. Yeah. Yeah. Yeah. So, the main thing about what this SDK that we built is it's a translation layer. So the function we allow it to translate into a way that we can write a Python callable into Unity catalog. Then from there we can decode that back into something that's human readable and an agent can actually use.
And this is kind of a high level of overview of how agents actually use these tools. So we'll have this agent definition. It has an LLM. It has some tool definitions. But what we're actually doing is we have a client in the Unity catalog AI SDK that allows us to define these functions, write them to UC and then when the geni agent needs to actually access like what functions do I have available and how do I access like actually run these that's what we actually built here.
So it's the interface between magical genai stuff as well and you know deterministic function execution. So we're going to go through a quick live demo. of seeing what this actually looks like. And what we're going to be doing is talking to Anthropic. Anybody here like a Claude fan? I know I am. Huge fan. Cloud 4 is awesome. So, we're going to actually be doing that. And to set up everything, we, you know, basically have to be running Unity Catalog open source.

32:59 Live demo: GenAI and UC AI

That was the I term that you guys just self check and making sure that the catalog is up and running. And this is kind of what you're going to see when you actually do that. You're going to go to the root of where you have UC docker compose up and it'll build it. We can go over and verify we have the UI running and we have, you know, the default setup there. And here all I'm doing is I'm basically contacting you know Anthropic to say what models do you have available today and show me the latest and greatest.
We have cloud opus 4. We have cloud sonnet. These things are things that just released relatively soon. And for this use case in this demo, what we're going to be doing is asking Claude if it can estimate based on some loan characteristics, if I make a monthly payment on my mortgage, when at a particular rate and some some data associated with it. If I pay this like overpay my mortgage, when when am I going to be done paying my mortgage?
So fun questions like that. And it's something that like it doesn't seem super fun and exciting for a demo, but when you think about it, it's actually a useful tool if you're like, "Hey, I I want to know if I pay an extra $250 a month, am I going to save like six months on my total mortgage payment?" And it seems like it's a trivial thing to answer, right? That hey, a human can do that, no problem. The function for it isn't really that hard.
Compounded interest calculation. However, genai even though it can write its own code, it's incapable of actually answering this question. The reason being if anthropic will answer me, it'll give us an answer. However, Anthrop and it says right there, I I can help you calculate when your mortgage will be paid off. It understands it parsed all that language that we passed into it. It knew even though I I just wrote some arbitrary human readable question there.
It comes up with some math here. It seems legit. And by the way, every time you run this, you're going to get a different answer because that's how Genai works. It'll tell you in different order of stuff, but it's going to come up with something. It's going to say, "Hey, my loan's going to be paid off in 117 months." And it says, "Hey, if you start it today," does anybody notice what's wrong with that? what it says today, December 2024.
That's kind of a little far off. That was when it last was trained. So the latest date that this cloud model had was December 2024. It trained. They went through a bunch of fine-tuning. They've been did a whole bunch of stuff and then finally they released it in the end of May of 2025. So it has no idea what day it is and it shouldn't really, right? You shouldn't have this thing uh connected to live services like that.
Uh, but it's going to say, "Hey, around September 1st, 2034 is when you know your mortgage is going to be paid off, provided all that stuff. But like I said, they make stuff up. I don't trust them." So, what I did is I went and wrote this function, and you can kind of see here here on screen. We won't go over it, but it's it's basic math. But if we actually run this, we get May 1st. So, we're off by many months. Even if we do the math of like adjusting, okay, there was like seven months ago and then, you know, add it again, it's off.
Now, if I if we had more time, I'd run this 10 times and we would see that every single iteration, it bounces around on what month it thinks. Does anybody know why? It's a fun little question. No. Okay, I'll tell you. Floating calculations. So, what Claude is doing, it's executing code, but it's writing basic JavaScript code. So it's using just floats in order to do this. This calculation is immortizing every month in order to do the calculation.
So it's a recursive you know calculation that those floatingoint errors propagate throughout that. So when you get a long enough time period horizon it's way off and it's going to be it's going to adjust how it writes the code slightly differently every time because it's just generating stuff. So it starts getting really really you know unstable. So so a whole different definition of variable mod rates. Exactly. Yeah.
So what we can do is we can actually take that function that we just defined and we can just write this to Unity catalog. So I wrote that in Python. Now UC functions has the ability to store metadata about a Python callable so that I can execute it wherever I want. But with the Unity catalog AI SDK, I can just pass in the Python callable with defining a Google style doc string type hints and making sure that I have comments there so that we can extract all of that and store it in UC in a way that a Genai agent like Claude 4 in this instance will understand what this thing is for.
Now even though we have that stored want to make sure that hey yeah I wrote this but I want to make sure that I actually wrote it correctly. So I can actually execute this directly in UC and that's what we see here this UC client execute function that's part of the UD catalog AI SDK and it gives us the correct answer. We need to wrap up. Let's just see what happens when we actually talk to Claude with the tool. It's going to say instead of making a bunch of stuff up, it's going to say, "I'll help you with this."
But it's also going to say, "I need to call a tool." And it knows which tool to call in Unity catalog because we have that translation layer. And when we call it with that, it gives us the correct answer. So this is the power of Gen AI with tools and we can do this in a very simple way where we can write a function in the way that we know how to write it. It stores it in a way that's not really human readable but the Genai agent can actually interface with it.
Okay, we are totally out of time so I'm going to wrap up the rest of it very very quickly. Uh let's switch back to the slides. But so there's a lot of things we didn't get get to. Ask us questions right after this. We are at our first our growth of the UC project has been much bigger than anything I've ever seen. Much faster than Delta Lake itself. We have lots of release going on both in the main core line as well as the UC AI release line.

40:01 Conclusion and project growth

We have over 500,000 downloads a month for things that we are releasing as well as other companies are actually integrating with it as we speak on every month. This list is growing and lots of future directions. Come talk to us in the slack on GitHub uh etc etc. So I wrap it up in 10 seconds. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.