Demystifying Upgrading to Unity Catalog — Challenges, Design and Execution

Summary

• Unity Catalog upgrade challenges: This video discusses common difficulties enterprises face upgrading from non-UC setups, including governance fragmentation, multiple data formats, and integration complexity across diverse data platforms.
• Repeatable migration patterns: Learn how Databricks and partner Celebal Technologies overcame upgrade difficulties to create reusable patterns and assets that enable successful migrations across the largest Databricks customers.
• Unified governance benefits: Discover how UC provides a single source of truth for organization data and AI, enabling open connectivity, lineage, monitoring, and collaborative sharing across the enterprise.

Demystifying Upgrading to Unity Catalog — Challenges, Design and Execution

Talk by Anirudh Kala, CEO & Co-Founder, Celebal Technologies ; Dipankar Kushari, Lead Specialist Solutions Architect, Databricks
Databricks Unity Catalog (UC) is the industry’s only unified and open governance solution for data and AI, built into the Databricks Data and AI platform. UC provides a single source of truth for organization’s data and AI, providing open connectivity to any data source, any format, lineage, monitoring and support for open sharing and collaboration. In this video we will discuss the challenges in upgrading to UC from your existing databricks Non-UC set up. We will discuss a few customer use cases and how we overcame difficulties and created a repeatable pattern and reusable assets to replicate the success of upgrading to UC across some of the largest databricks customers. It is co-presented with our partner Celebal Technologies.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What are the main challenges when upgrading to Unity Catalog?

This video discusses challenges including governance fragmentation across multiple data sources, managing diverse data formats like Delta Lake and Iceberg, integrating numerous data platforms, and maintaining data quality during migration.

How did Celebal Technologies and Databricks develop repeatable upgrade patterns?

Through experiences upgrading some of the largest Databricks customers, the teams identified common challenges and created reusable migration patterns and assets that can be replicated across other organizations.

Which AI orchestration platforms integrate multiple data sources?

This video shows how agent orchestration frameworks integrate with the Databricks Data and AI platform to access multiple data sources securely. Unified governance enables agents to query diverse systems while respecting access policies. The platform simplifies multi-source agent deployment.

How do I securely connect AI agents to multiple data sources?

This video demonstrates secure multi-source agent connectivity on the Databricks Data and AI platform through centralized governance and authentication. Unity Catalog enforces access policies consistently across sources. The platform enables safe agent operation without exposing credentials.

How do I govern Delta Lake and Iceberg together?

This video shows that Unity Catalog on the Databricks Data and AI platform provides unified governance across Delta Lake and Iceberg tables. Organizations can use both formats without duplicating governance policies. The centralized approach simplifies compliance and security.

Full transcript

Hello everyone. Raise your hand if you're currently going through a Unity catalog upgrade. Nice. Nice. So today we are going to talk about Unity catalog. uh why we need unity catalog and then um we'll also look at some of the use cases and also help you understand how you can efficiently upgrade to unity catalog. So this is Deepankar Kushari. I'm a specialist solutions architect with datab bricks. Been with datab bricks for more than 3 years now and I specializes in data governance.
I have with me uh Aneru Aner. Hi everyone. I am the co-founder and CEO of Celible Tech. We've been one of the partners who've done almost 100 plus migrations and upgrades to Unity catalog last year. uh more than happy to discuss some of the success stories. Thank you. So let's get started. Now first thing I want to want us to understand that governance is hard and this is a very typical situation for most of the enterprises where they have multiple data sources.
They're using diverse data assets and AI assets and also they have numerous data formats. So for examples you are definitely dealing with data lake as well as you still have data warehouses and you are also dealing with multiple cataloges and databases at the same time you have data assets not only just tables views you are also dealing with files in your data lake and you have notebooks dashboard and as well as of course in today's world AI models and then nobody uses just one data format right delta Lake, iceberg, CSBs and parket.
These are some of the well-known formats that we are dealing with. So, governing all of these data assets and data sources and all these data formats and data um stores across all of these solutions, it's hard. It's fragmented. Second thing is these solutions or data platforms, they don't really work well with each other. they they don't really always integrate well. There is always some kind of additional integration or some customuilt solution that is needed.
You can also think of some ETL pipeline that is always required uh to integrate all of these solutions together to make sense of your data. And also we have to remember in today's world it's very important to harness the power of AI and ML and lot of these um data stores they don't have really out ofthe-box uh solutions or uh built-in intelligence and what are the impact of these challenges what is what are we losing out of it so with fragmented governance what we see is there is compliance and security issues quality risk and as well as operational overhead and efficiencies.
I talked about little bit about how to how how it is difficult and timeconuming to build all these integrations and custom solutions between these systems. But also remember if you have to govern your uh data sources or your systems, you are probably duplicating or duplicating your policies across all these systems. And not only that, a simple example could be uh administrator who is trying to see who has done or who has accessed what data in all these disparate data systems.
Uh then he has to build another solution on top of all of these and these does these systems doesn't really integrate well with each other. So there is no real open connectivity. So these are uh creating vendor lockins as well as again higher operational cost. And finally, lack of built-in intelligence is um slowing down the speed of innovation as well as it's a missed opportunity for uh competitive edge. So how does a good solution look like?
What does a platform should have that can provide you or uh kind of get rid of all of these challenges? Uh first thing we are talking about is unified governance. So irrespective of where your data is because we talked about your data is in all of these different data stores. Irrespective of where your data is, you should be able to access that data in a unified way from a platform if at all there is a platform like that.
Secondly, open connectivities irrespective of what compute engine you are using, you should be able to access your data. You should not be restricted by a specific compute engine to be able to access only a certain segment of data. And the third thing is in today's world you should be able to harness uh the power of a IML and um you should have features available out of the box within your system which can help you to do repetitive task or to able able to uh perform certain things where you don't need to spend time rather you can build something which brings more value to your business.
So we could think of autocl classification of your data because we are talking about governance. So let's focus on those areas. There are so many use cases for a IML. I'm not going to talk about all of those. But auto tagging or maybe optimizing your data as they are getting ingested. So with keeping all of these challenges in mind and also how a platform should look like with unified governance, open connectivity, built-in intelligence, we have built unity catalog the first unified catalog for both data and AI.
Unity catalog is multimodel. That means it supports not only just data that means tables or views but also it supports your IML models. Unity catalog supports multiple data formats not only just data lake but iceberg and parket and CSVs and many more and uh unity catalog also supports accessing the data. So your data when governed by unity catalog can be accessed by any engine not only just datab clusters or SQL wrowers.
Um now this session is about um upgrading to Unity catalog. So we also need to understand what was there before. So there these workspaces were kind of this monolithic structure where uh primarily we have to focus on this user management the meta store and the access control which were part or local to the workspace. So that means each of these workspaces were shipped with a hive meta store and your users were managed locally within the workspace and finally access controls were also maintained within the workspace.
With introduction of unity catalog what we did is we brought all of these up one level at the account which we call at the account level. So now your meta store is at the account level. Your users are managed at the account level as well as your access control is also managed at the meta store at the account level. And then you have your workspaces. Those are think of it more like a area where you do the compute or perform your compute and you attach those workspaces to the meta store to access your metadata and your access control your data.
And since we had this logical isolation at these workspaces um now that all of those are kind of centralized at the meta store level at the account level. Um so we had to bring um another level of um isolation which we called catalog. So that's the third level name space that we introduced with unity catalog. So you have your meta store at the top and uh there can be many cataloges created within the meta store and then for each of these catalog you can create multiple schemas also known as databases and then within the database bases you have these data assets and AI assets.
So you have AI models as well as you have tables, views, volumes, functions and so on and so forth. We also didn't get rid of hive meta store yet right? So we have still have hive meta store as one of your catalog. You can still access them but eventually as you will migrate to unity catalog and you are at some point you would want to decommission your hive meta store. So will talk more about those things. The other thing is very important because you might ask that okay I had this workspace my developers only have access to the workspace and they can only see the meta store uh data or metadata that is accessible by that workspace.
So how are you isolating? Now you you brought everything at the centralized at the at the meta store level which is centralized and accessible from all these different workspaces. But how do you really isolate that? So we are also we built a way or mechanism to isolate these cataloges and not only cataloges external locations and other uh unity catalog assets which can be isolated from different workspaces. So as an example when uh a developer who is part of a development group is using development workspace for doing all the activities building ETL pipelines and so on and so forth um they are only supposed to have access to the development catalog.
They don't really have any business to have access to the prod catalog. So you can determine that okay these dev catalog can only be attached to this workspace which is aligned with the developers and then if the tester needs access to the staging catalog they can be attached to the staging workspace and now let's say the tester also wants to have access to the plot but only in read mode read only mode then uh you can do that too with the catalog workspace binding.
Um so in a nutshell what we had as a logical division between these workspaces is also achievable that isolation is also achievable even if your catalog or your data is maintained centrally at the meta store level at the account. So unity catalog is really unifying the lakehouse. Uh we talked about unified governance. So with catalog federation you can achieve that even if your data is maintained across all these different systems across all these different cataloges let's say hype meta store uh glue horizon snowflake horizon or any IRC uh iceberg rest catalog you can still federate those cataloges to unity catalog and then access via UC open API or unity catalog open API or IRC which is iceberg rest API um uh to access your data.
So we started this talk about talking about a platform ideal platform which can give you that unified governance. So Unity catalog is that platform where you can get that unification via um leas federation federation and using the open APIs. Also, if you want to collaborate, you can use data sharing in a secured manner to share your data with other clients that can read that share. And finally, Unity catalog is part of the databris data intelligence platform.
So, you get all the inbuilt intelligence and harness the power of a IML in UC. So with all of these, Unity catalog is the most open, powerful and interoperable catalog for both data and AI. And finally, let's look at this picture which shows the feature set that gets unlocked when you use Unity catalog compared to hive meta store. So all of these feature set needs metadata or special processing that is just not possible to be available in hive meta store.
So when you move to Unity catalog or when you use Unity catalog you get all these rich feature set. And with that I want to hand it over to who will talk about how Cell Tech has helped customer to upgrade to Unity catalog across different um verticals and unlock the full potential of the lakehouse. Sure. Thank you so much. Um let me ask you a I mean let me start with a question rather. Uh how many of you have heard the term UCX?
All right. Uh so for those who have not UCX basically is a utility by data bricks. It's obviously open free to use to upgrade to unity catalog. And maybe then let's have a follow-up question. uh now that you've seen Unity catalog has got the federation for Oracle for Snowflake and whatnot. So let's say you've got uh some tables in Oracle that you can now federate inside Unity catalog then do we really need to upgrade to the Unity catalog or push these tables to Unity catalog or are we good to stay with that?
Anyone I mean if you want to answer more than happy to take it right now. We also have a Q&A at the end. All right. See when you federate basically that lets you run the warehouses with all the potential of unity catalog effectively it's a techn you eventually want one policy system you want one governance system you should want one lineage system and that's it right so with that I'll come to the introduction of celtech we've been a datab bricks partner for last 8 years now uh we've been the APJ partner of the year for last 5 years this year in Last year we were also the unity part unity catalog partner of the year as well as we were the uh this year we've been the energy uh partner of the year work very strongly across different industries and unity catalog is right at the epicenter of all the innovations that we do.
Um so what we did was basically we use UCX the utility I just talked about and on top of UCX we built something called as the Unity launcher. It's one of its kind migration accelerator which basically enables you to migrate not just within the workspace but also interworkspace. So we'll see what that is. So from an exploration perspective from hive meta store the most common meta store that we come across in our customer base Apache ranger and AWS glue.
Glue is of course the ETL tool but it also can push on certain policies that you would want to integrate with uh with Unity catalog. One big challenge with Hive Metas store is of course it doesn't allow you all the things that Debbanka pointed out the experience integration with not just the data catalog but also integration with processes with persona based integrations with so with security I'm sorry ourbacks and the feature attributes.
So the feature attribute basically is a concept where you could go and give security control to an attribute. The attribute could be a column, it could be a row and so on and so forth. And then what we also did was we did integration with Cibbra, Microsoft Perview and IMUA because you may land up having multiple systems of governance and right now what the system does is it allows a real time and a backward integration.
That means a policy built in Calibbra could then be in real time ingested into Unity catalog and a policy in you know created in Unity catalog could go back into Calibra so that if you got a a full-on security construct which is basically right from ingestion to your storage to your processes to your ERPs to your CRM and so on and so forth. Unity catalog can then become the integral and again the epicenter of all your security constructs that you're trying to build.
So and with Apache Ranger specifically so I don't know if if if anyone is from uh communication and media we found Apache Ranger was one of the most commonly used tool even for downstream security constructs. So with Apache Ranger we of course did not sunset sunset it but the downward integration the backward integration the front integration and the up integration all for integrations were actually possible using launcher and that too within the second week of implementation.
So what we offered to all our customers was that we could migrate all your existing tech debts in security or in access controls or in policies to uh Unity catalog using Unity launcher. So as you can see the first three are more migrations the other three are integrations. Hence basically it was able to provide you best of both the worlds at the same place. And what were we migrating? First the user groups, the data assets, job and cluster upgrades.
So your DB uh runtime upgrades, your notebook upgrades, your three-step or threelevel namespace upgrades and so on and so forth. Then ACL migrations, your policies and in fact end to end testing. So basically all integrated in one solution itself. As I said, we did around 118 migrations last year. um it was double the number of migration with the second in the race. So basically we could do all of that because this solution was uh we built a solution to migrate this the entire customer landscape and not just you know let's say a part of it but in fact the entire uh set of assets and after that we actually made it open source.
Now you can actually download it and start using it for your own upgrades. In case you are, as I saw so many hands going up for upgrades to uh Unity catalog, you can actually use it and expedite your overall journey. And the next few slides I'm going to talk about some of the challenges we face so that in case you are facing those challenges, I'm more than happy to uh discuss more and in fact help you uh with some of the major roadblocks that we were able to fix.
So the first challenge was validating jobs in the lower environments. Um I hope I I I don't want to go but I am sure that uh some of you probably would have similar challenges upgradations of data right in hive meta store to UC workspace now basically in the lower environments to the prod. And then I would say last one which is one of the most common one in Azure which is ADF not triggering a UC enabled job cluster. This is something we found at virtually every single customer that we worked for.
And then finally, what we did was we maintained a two-hour downtime for all assets migrated using CI/CD and the product environment and this three-level name space. So you saw that three-le name space, right? That was the catalog dot, you know, the schema. t name, right? So all those upgrades, we were able to make those changes in the notebooks as well as upgrading the notebooks uh for UC. So we achieved a 84% UC adoption within the first 13 weeks.
Now this customer had almost 100 plus workspaces uh and about about 100 plus terabytes of data that we had to migrate. That means uh make them available inside Unity catalog registered inside Unity catalog. In some cases we make migrate manage table. In certain cases we actually converted them from managed in hive to external in uni catalog. So one of the largest oil and gas companies here in the US. We did uh migration of the assets.
The assets were primary the SQL udfs and the dataix notebooks. Um and of course something that hive metas store lacks which is data security and privacy governance data lineage again something that was not available out of the box in high metas store. Uh migrated the entire SSRS reports on governance. So in fact they had built the arbback um you know logging on SSRS. We're able to migrate the logging in um in uni catalog to the other BI tools.
Um now again the same thing came up. We had those ADF triggers. We had uh the similar challenges of down of the lower environments not having validated table names and so on and so forth. Uh so of course this this is again like a large large scale migration. So we started with a mount you know we started with a small MVP and then we of course upgraded them to a full-blown unity catalog implementation u the mount points and the SBS path right big challenge that we've seen uh with unity catalog with the uh the earth file governance systems so now with uni catalog we're able to do that across the system uh with multiple storage points and of course we delivered this entire thing in 6 weeks the reason why I wanted to show you these two case studies.
I've got one more and I wanted to understand that migrating to Unity catalog is not just a function of certain changes in the system that you wish to make or certain you know uristics that you could follow. It has its own customization. Every customer that you work for they could not use unity launcher like as is because customers have their ACL lying in metas store there. Someone had it in even in u in the Azure um you know central governance system there.
Some had in perview they wanted to keep it in perview as well as have it in UD catalog. So these permutation and combinations were very eclectic. I mean this was a mix of probably everything possible but now what is available out of the box because of UCX and the and what we've open source the Unity launcher you could actually fit it to your own purpose fit it to your own use and run these kind of solutions pretty pretty much the same way now this is one of the largest cyber security customers across the globe in fact you'll see their uh boards uh with their their uh sponsor name outside one of the biggest challenge they faced was the increased risk of data breaches, security, vulnerabilities, privacy and violations.
Why? Because they've had their governance platform split into 25 small small uh solutions all across their data platforms and not just data platform even applications. So what we offered to them was an upgrade to unity catalog. Not just an upgrade in fact we were able to kind of migrate them migrate them to data bricks but most importantly today their entire security construct which includes all the applications that are SAS based consumerf facing as well as the data platform is all running on uni catalog one single instance with multiple it's it's a multi-tenant data warehouse that they offer to the customers I mean similar to any you know splank or uh you know green plum that you must have seen but most important ly the entire data universe is now managed with unique catalog and using AIBI.
So now the users can go and ask for their ask for some of the outcomes on the security data without actually without the system having to first create a silo for them because uh that security construct probably won't work the kind of security framework they had. So now with indic catalog that person has identified in the real time and they use AIB genie to actually run their work or run their reports in the real time.
So this was one of the I would say one of the most u uh compelling outcomes we were able to drive and this this two in 10 weeks they took about 12 months before they came to us and we delivered this using launcher. So almost 100 plus delivered engagements 80 plus customers 118 to be precise. um 98% on-time deliveries when I say on time basically means before time and what the way we start is I'll probably talk about that uh from a whole holistic purpose uh where we start is we actually have an analyzer that can go deep uh in your governance platform figure out the policies ACL uh personage arbacks uh even if you put in processes for example someone uses SAP now in SAP the biggest difference is that the the the security is at the object level.
So I may be able to access my supply chain data but not my production even though supply chain is very closely related to production. So when I use the same uh let's say construct in my AI genie or an agent or a lakeflow or a lakehouse u you know delta table the same construct is automatically transferred. That means if I was able to only see my supply chain data, I see exactly that integrated with SAP, integrated with Oracle, integrated with Salesforce, Service Now and of course about a couple of dozen more popular solutions and of course deployed across seven industries.
So we've already got the template for uh different industries with respect to the processes. So let's say you got uh if I talk about uh discrete manufacturing. So from production manage production plant manager to supply chain manager to procurement manager uh right to drop drop schedule manager and so on and so forth. So all those personas were pre-shipped in UC. Last but not the least our approach for unity catalog migration.
We start with an analyzer. It's a pre-eployable utility that we can push and get the entire inventory. So basically I could make uh the overall analysis of your workspace within the first day of this deployment which takes about 2 minutes. Then the entire premigration assessment basically how do we migrate what do we migrate and we have a pre-built template as per the industry as I talked about and then from there to project planning asset migration and finally post migration.
All of this we've delivered using a well- definfined template exactly the same way for each of our customers and we able to do this in as I said in few weeks. So what is the real UC challenge? One big UC challenge that we've seen is the inter workspace mig upgrades. So I'm not just upgrading the existing workspace but actually upgrading it to the next one. So to the new one. So from no isolation of production assets to a use enabled workspace and external applications external application the word is application and not just a data asset.
So migrating that which was not supported in UC to now uh not now start supporting UC mountpoint incompatibility I just talked about that and streaming conflicts one major issue we found uh you that we found in DT we're able to you know resolve that using unity catalog I think all of this is one but the most important point the most important outcome for has been that we've been able to then take it to the next level in our AI security so our AI persona the AI governance methodology that we are now putting up.
Of course, there is hallucination, there's prompt injections and so on and so forth. But our ability to use the right persona while prompting is something that has really helped and that's kind of built on the uh unity catalog end to end descriptive cataloging right that we've done for each column uh from a source. So if I go to a as I said go to a manufacturing customer I know what personas are allocated to what kind of data sets.
So now if I have a prompt that is outside the realm of that persona, I'm able to stop that using Unity Catalog. And I think that's been a big change for all of our customers. Well, with that u we can get into any Q&A. So if you have any questions, more than happy to answer.

Learn more about the Databricks Data and AI platform.