Unity Catalog Upgrades Made Easy. Step-by-Step Guide for Databricks Labs UCX

Summary

• Simplified migration process: The Databricks Labs UCX project automates key components of Unity Catalog upgrades including group, table, and code migration from Hive Metastore.
• Risk assessment dashboard: The UCX Assessment Dashboard provides comprehensive insights to reduce upgrade risks and prepare effectively before beginning the UC migration process.
• Lakehouse federation support: UCX works alongside Lakehouse Federation to provide a streamlined and efficient upgrade path for organizations moving to Unity Catalog governance.

Unity Catalog Upgrades Made Easy. Step-by-Step Guide for Databricks Labs UCX

Talk by Liran Bareket, Lead Product Specialist, Databricks ; Vuong ‎, Lead Solutions Architect, Databricks
The Databricks labs project UCX aims to optimize the Unity Catalog (UC) upgrade process, ensuring a seamless transition for businesses. this video will delve into various aspects of the UCX project including the installation and configuration of UCX, the use of the UCX Assessment Dashboard to reduce upgrade risks and prepare effectively for a UC upgrade, and the automation of key components such as group, table and code migration. Attendees will gain comprehensive insights into leveraging UCX and Lakehouse Federation for a streamlined and efficient upgrade process. this video is aimed at customers new to UCX as well as veterans.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What does UCX automate in the Unity Catalog upgrade process?

UCX automates installation, configuration, and key migration components including group migration, table migration, and code migration. The automation reduces manual work and potential errors during UC upgrades.

What is the UCX Assessment Dashboard?

The UCX Assessment Dashboard provides comprehensive insights into your current environment to reduce upgrade risks and help you prepare for UC migration. It identifies potential issues before you start the upgrade process.

What is Lakehouse Federation and how does it relate to UCX?

Lakehouse Federation is a feature that works alongside UCX to streamline and improve the UC upgrade process. This video explains how federation capabilities enable more efficient migration paths.

Is UCX suitable for customers new to Unity Catalog?

Yes, this video is designed for both customers new to UCX and experienced veterans. UCX provides comprehensive guidance and automation regardless of your prior experience with Unity Catalog upgrades.

Full transcript

So today we have a host so I'll do the intro but yeah we're talking about Unity catalog upgrades uh in particular we're talking about the UCX the labs project. So Luran Luran nice to meet you. So quick introduction about us. Um so I'm I'm a lead solutions architect data bricks uh working in the Asia Pacific region have done quite a lot of work in the on the platform but also have contribute to a lot of open source projects so including projects like the terform provider and I think particularly today we're talking about is UCX sure lan I'm a I've been a unity catalog product specialist pretty much from the onset and uh started with vong UC and others the UCX project to automate the M migration process.
So yeah, UCX when we talk about UC migration and I'm sure a lot of people would have heard about that has been ongoing you know for a long time right moving from my hive meta store to unity catalog um we want to talk about four things today right mostly motivation you know why did we actually build the toolkit why do we build UCX to support migration how do you then actually start preparation right how do you actually execute the migration with the capability ilities in UCX and then how do you make sure that it actually works.
So a little bit about the motivation um obviously why we want to migrate to UC obviously best of unit bricks is built on unity catalog I'm sure in the keynote tomorrow there'll be a lot of topics you know we having a lot of capabilities a lot of features that get unlocked I think the latest one that literally got the release note like today or yesterday was the attribute based access control so we're actually now having that available which I think basically means if you want to take the advantage Unity catalog right you need to oh of data bricks you need to move to unity catalog everything you know around system tables be able to do data sharing in place sharing data you know having AI governance capabilities having that lineage track you know and all of that requires unity catalog so obviously we need to do that right and normally you see you got have meta store right where your datas are reside it's a two-level namespace it's under the schema and is workspace specific and then you move into metas store.
So part of that is the first step we need to do is migration of the groups because groups originally are workspace local objects. When I talk about unity catalog we thinking in terms of accountwide because these have to be cross workspaces. So first we need to make sure these workspace local groups become account level and then we need to then move our do the migration of the data as required. Right? Sometimes especially if data sit on like DBFS route you need to move that as well and then once that's done we basically need to make sure our data has the right metadata that gets migrated so metadatas that sits in have meta store that are legacy that are workspace specific needs to move to UC and then finally once we move all the metadata we need to make sure our code are refactored right instead of referring to two-level name space it should be now referring three-level name space.
These are theories, right? But we realize that's a lot of work. So when you think about it, right, you have potentially tens of or maybe hundreds of workspaces. There are thousands of tables or views. You have lots of jobs, lots of notebooks, and there are coachings that need to happen. You need to make sure these are done in an iterative manner. So you need to be able to plan all of this and actually doing this manually almost impossible. and try to do this in a more programmaticical manner.
Yeah. Cool. But obviously folks come up with their own things. So which is why we're talking about UCX or a reintroduction because this has been around for a year or so a bit longer. So when we talk about UCX, why is it UCX? It's is the name is called Unity Catalog upgrade toolkit. Um but UCX because it's cool, right? Everything that ends with an X is a cool product. In a nutshell, it's a dat labs project. It's developed in-house by a field engineers.
So, including folks like Luran, myself, and there's quite a few other contributors to it as well. Been keeping this project up to date, fixing a lot of bugs, adding new features, and the key capabilities that we have added, right? is the assessment capabilities which is inventoriize the entire existing hive medicine and gives you an indication of how to migrate doing what we talk about right group migration making sure reg all the foundation objects in unity catalog are registered making sure we can do the table migration and then the code migration and the additional utilities as well and it's been quite a successful project you can see the adoptions uh across the customers right we're talking you know a few hundreds of customers are using this to support their UC migration and we're also getting quite a lot of stars on GitHubs etc everyone likes stars and forks and I think giving one of the success story right we're talking a customer migrating roughly 600 workspaces which is a lot right lots of business units and they were able to complete the migration project in just about over six months.
Um that is quite a impressive pace without automation without having toolkits like that I don't think it will be possible to do so. So let's talk about preparation right what we need to do firstly is installation so obviously how do we install UCX uh it's available via the datick CLI. So actually in under when you install databick CLI it actually has a labs group command and it will actually be a databick labs install UCX.
So we can we can get the video to play u but otherwise I can narrate what it does but effectively is a interactive prompt right you install datab bricks cla I would say if you ever work with data bricks you need to do anything programmatically data brick cli is pretty much the way to go and you run data bricks lat install ucx it will ask you a series of questions and you can iterate through that so effectively it would ask you you know what are the workspace what are the profile for authentication purposes.
It will then deploy the UCX library and all right this is working right. So basically it would because UCX is a Python library. So you will start the prompting right giving some naming conventions for example what the schema do you want to store the artifacts from UCX any logging we want to do and there's a lot of defaults already set right we want to do different threadings do we how how do we upgrade the groups doing the m you know the upgrades do we need to scope our migration to only specific databases Do we want to skip the migration um of some objects?
And if you think about it, right, it's the assessment is probably where most folks have been using data bricks um you see elaps UCX for because it gives you essentially inventory of all the hive objects, right? The tables and the views, the UDFs, including also the access control list that histo like that was the legacy version, right? We call it the legacy table alle and we also inventoriize all the workspace objects as well right and we use the API to programmatically go through this.
So all the workflows, all the clusters, all the compute resources, the query is a dashboard because again think of the need to rewrite rename some of these two-level namespace objects to three-le namespace and once we start running the assessment you can UCX will actually deploy a assessment job here and the assessment job here actually has dependencies of a lot of objects right first we crawl like the tables and we want to crawl the grants we want to crawl the workflows and then we will then summarize all of that in a dashboard.
So this dashboard easy to look at the entirety right how many tables have we captured how many credentials right how many jobs have we inventoriize and all of this give you a quick overview of how much effort it requires. So you can see here right and out of the objects that we inventoriize it also give us an overview in terms of what patterns can be used to migrate right if you have a table that is system DBFS roots that's probably more effort to migrate than if you already have an external table that sits in S3 location and it also go through the code identify potentially any compatibility issues because as you're aware moving to Unity catalog also means moving to the new access mode in data bricks, right?
It's not a no isolation free-for-all anymore. And you can see here lots of details that allows you to step through your migration, right? Going through the one, two, three, four steps as we do. And there's one more thing that I want to highlight, right? As we actually look at this and try to estimate a t-shirt sizing in terms of effort, as I mentioned, right? Delta table that sits already on S3 slightly easier to manage than let's say tables that are on DBFS in a you know packet format or something.
One more thing to think about is when we moving we're now moving from tables that sit in high meta store that are scope to single individual workspace. We're now moving to tables that sit in unity catalog. So these will sit across all the BUS actually how do we map that to the structure. So one of the thing that we ask C um users to do is how do you map the source to the destination right because you think again look at that picture right we have two level name space in high metaol we're now moving to three-level namespace in UC so where does that sit right are we moving table as is to a new schema that says under the catalog or do we want to do some reorganization like are we cleaning up our houses and this is quite useful And what we've seen is a lot of customers have done the assessment and they've got a good view of the effort and they start going through the UC migration and that's where we see the execution it can be opinionated because it depends on the scenario depends on user level.
There are different ways people approach the problem. I think our opinionated approach is that UCX has a lot of capabilities. It has a lot of tools on how to sequence the actual execution. So I'll hand over to talk about this opinionated way. Thanks V. One thing I want to mention with when you install UCX, it asks you a lot of questions and typically when it asks you the questions, you're not at a point to provide all the answers, specifically if you do it for the first time.
Don't worry about it. you can guess like a lot of all all of that can be then fixed in the configuration file. So let's talk about how you actually execute the migration. So the first thing you do is the group migration. Group migration there's no really other way to do other than using UCX. H the idea that we're moving from workspace specific group to account groups that part is simple. It's part specifically if you use something if you import the groups from external external uh repository like a a pretty simple to do to remap it to the account.
Where it gets more complicated is that all the internal objects in the workspace are mapped to these groups, the permissions. And it's very it's very hard to the point that of impossible to actually explore all the places where you reference these groups to remap them to the account groups. And that's what we do with group migration. We list the we list the existing permissions. We that's all done automatically using this the same workflow.
We create the account groups for you. That's optional whether you want want us to create it or you want to create them yourself and we can make the same groups with the same name or we can create mapping. We have cases where you want different groups. You move from a from a condition where you have in dev workspace QA and prod the same name group but used differently to a place where all the groups are in the same repository.
So you may want to prefix them or suffix them and we're allowed to do all that. We validate the account group. So we run validation and we can run validation on the nested groups. So we run validation, make sure the membership on the account group is the same as the workspace group. We rename the workspace groups as a backup. We remap all the permissions and then we get rid of the workspace groups. That's all done automatically by group migration.
This is by far the most tried and true elements of UCX. Let's talk about table and views migration. So we have now two options and that's new that hasn't been the case last year. We have now two options for migration to to have in mind. One is the traditional the table migration. We call it bulk. It's not a great name. H you can do it iteratively but essentially this is where we migrate the tables from HMS to UC. We will show you exactly how we do it because it's different from manage group, manage tables, external tables, delta or non-delta.
We have all this determination. The next option that is available and we call it incremental is HMS federation. So now UCX allows you to turn on HMS federation and migrate the HMS catalog into Unity catalog as one one to one. We don't offer mapping there. So all the names remain the same. The schemas and tables names remain the same. But we it it is an implicit upgrade. Now when do we recommend using HMS federation over traditional UC migration?
If you have a case where you want to get on UC fast, you don't want to touch the existing code or you have upstream processes that are too complex to migrate and you want to leverage a lot of the UC capabilities downstream downstream visav SQL Genie all the new features you can use HMS federation and what is new is that we allow you to migrate it incrementally. So first migrate all the tables all the pro downstream processes AI BI dashboards all that to UC and only later upgrade your ETL to UC we have an option a button on each individual table upgrade in HMS federation so that's new for you right now obviously we migrate tables we migrate views we repoint the views to the new the new location of the table as well.
Let's skip. Okay. So, this is kind of the mapping of what we do in in table migration. Uh we we have a lot of strategies as you said. We we figure out if it's an external to manage. If we need to copy the data or just copy the metadata, uh whether we do sync, whether we do deep clone, we have a lot of options. All these options are managed by the workflow. If you have a case that you think we don't cover, reach out to us because it may be covered and you haven't surfaced it or we may even customize it for your need.
We try to cover all the migration cases for tables but we may have missed some. All right. So this is the workflow that migrate the tables. As you can see it when when the table so when uh yeah you don't see great like I should have I we're running workflow right now right? Yeah like I'm running the migratable workflow h and as you can see not great C but like each one of them is a migration is a task that migrates a certain type of tables.
So you see it's it's very elaborate and at the end of it we update the migration status. I'll touch on that soon. Next slide. Yeah, let's um Okay, that's fine. So the next thing is that's how we migrate to federated. It's a single command migrate to federated. We allow you to migrate from workspace groups sorry from workspace HMS or from external HMS. We support glue my SQL SQL server posgress and in this case I migrate from glue.
I specify the glue. For glue, you have to provide a service credential and now I create the location for the new catalog. That's that's a nuance in a HMS federation. And you will see that once it's created, I can go and is that next slide? Yeah, let's connect I think yeah you can see here that the new catalog was created and the catalog has a one to one depiction of what you had in HMS. I didn't have to specify mapping to tables or anything like that and I can access the HMS tables directly.
Yes, I think I think this is a game changer, right? Because if you thinking of if you still have legacy hive meta store, this is a new and better options because it allows you to effectively switch all your downstream to start using benefit from Unity catalog features things like you know AIB genie even some of the more ML focused workloads and then allows you to then incrementally upgrade your ETL. I think that's probably the biggest change.
Yeah, that's the biggest sense from last year from when we presented it last. All right, next slide. Okay, code linting. Once you migrated all your tables to HMS where we are linting the code or we have to migrate the code we are build. So the way we l code we have two main options. We can link existing workflow or we can link static code. static code as in code in your git repository, SQL queries, libraries. So we can look for code in jobs and dashboards, in notebooks, in files, in wells, in eggs, cluster configuration and pi, pi.
We create a graph, it's internal process. We create the relationship between all that. So if you provide us a notebook, we will not stop at linting the notebook. We will go into the library. It or other notebook it references and we'll surface them and finally we will link and we will show you all the problem in the code okay next so what are the what are the examples right oh yeah so yeah uh yeah like I list few example the JVM if you access the JVM directly if you have a direct file references we will highlight everything that might not work with you see this is an example in pispark right Now this is the state of it and again I'm not sure it's going to change soon.
We haven't decided yet but we will highlight all the references that you have to that are two name two levels name space and we will tell you what table you have to change it on SQL we will change it for you on SQL query or dashboards we will change it for you we will back it up and you don't have to touch it. Next uh this is some more I would say involved examples where you reference a file a table name that from a prompt or something like that.
So we will highlight that and you will have to change it on your own. Um, next and the last one that we're working on and it's should come later this year is a update external reference external file reference to the table that host this uh location. And this is especially important now that we encourage everybody to move to manage tables. This will be used for when you migrate from external to manage table regardless of HMS to UC.
So you will you we will use the same function. I think yeah the um because as we're switching now from paths to table references because we are decoupling right you want to be path independent effectively your tables should be what you refer to that's how you grant your permissions. The underlying location is a function of the platform that will then handles the management and the optimization of that because essentially the layer that would con would dictate how the table maps to a path is unity catalog right unity catalog should be the source of truth you shouldn't try to go to the path in directly go to the tables so we we are fixing think SQL that is embedded in pispark and SQL that is independent of pispark we do both of them like and uh right now again we're offering linking in future version we will offer we will offer fixing fixing is a little complicated because the table name is not always listed outright in the code sometimes it's in a variable all that kind of stuff like so that makes it more difficult all right finally and this is something that is relatively new but is very powerful is we have the concept of validation.
We have right now it's an experimental workflow that essentially is doing reconciliation. The concept of reconciliation is to compare the source table to the target table. And there are few cases where where it is desirable. In in many cases where you have an external table, we just reference the re we just copy the metadata from HMS to see. Reconciliation doesn't make a lot of sense because it's the same table. If it's delta, the metadata is in the table and not in the in the catalog.
So it doesn't really matter but cases where it is not delta or if it's manage table on DBFS routt and we copy the table to UC and you forgot to turn off the upstream process or intentionally you keep both system alive. The reconciliation will alert to you when the source and the target become estranged. We and these are the tests we are running. We are checking for the metadata and we we provide you reconciliation of the data itself like and you can set the threshold.
You can look for sampling or you can look at every single record. We're doing it by check sum. It's pretty efficient. We run this process on a nightly basis and we give you a report on how much your migration progressed and how many tables are the same or not the same. So the con the idea is that you will not migrate all at once rather you will have a rolling migration and every night you will get a progress report to see how you're doing.
Finally, these sanity checks have nothing to do with UCX. It is on you to check that UCX and your processes are all fine before turning off HMS. Do all the workflow run successfully? Are the dashboards there and available to your customers? Right? People have access. This is something that is very important because it's a big break from the way HMS and HMS security works and who owns the newly created object is kind of important.
This is something it's a concept that's not available prior to UC but who owns is very important. It's important that the ownership will be assigned mostly to groups and not the individual because there are certain that has changed lately. We added the manage uh permission but before we added the manage permission ownership was very fundamental. So it's important to understand you do all these sanity check before you disconnect your HMS which is quite relevant because we now have ability to completely turn off HMS as well in in a workspace that was a long time in waiting as well right before you can you complete your migration you need to make sure that no one keeps writing data to meta store as well so you want to turn that off so that is now also available So it's a it's a little things that basically make the migration now a seamless end to end journey.
All right. Few special consideration few things that the secret thing that we wanted to share with you is no migration happens in isolation. So when you when you migrate you have to think about migrating the tables migrating and skip over migrating the related views. You m you migrate multiple notebooks you migrate the codes and you migrate the job that are related to this code and finally you repeat that into in multiple um multiple workspaces.
So click over and that's what I described before. The migration the progress dashboards is what gives you a view of how your migration is doing across all your workspaces. Uh it's an experimental and we hope you can participate with us in uh getting it uh getting it mature. So it's an experimental workflow that right now is disabled when you install your CX. All you have to do is enable and play with it and you will get this beautiful dashboard.
All right, let's talk about road map and what and what's coming and uh we're we have small enough group so I'll run quickly through that and then I want to little bit hear from you what do you want and what worked for you what doesn't work for you and where do you want us to head next because this is something that we contemplate the migration progress report that's already experimental is the first thing the direct path lookup that I described is something that is coming later this here the converting mounts to volumes controversial feature we're working on it and finally AI and ML assets right so far we barely touch on AI and ML assets upgrade we're going to tackle that next and finally okay one more thing this is secret between us and you that joined us here That's all right.
There's a little tidbit, a little secret in Unity catalog in unit UCX that come ships with UCX. We call it the debug notebook. The debug notebook uh the debug notebook is there when you install UCX. It allows you to run UCX from a notebook and have access to all the UCX context and all the features of UCX. It's really, really cool. I I have I have an example here. I can share the example with you. It gives you access to all the facilities we have with UCX from the command line and you don't have to worry about logging in or running it from your laptop and it is you it was used there originally for de debugging or troubleshooting but it can be used for much more.
I'm showing you here a use a a customer I think you might be even in the room that request us to update manage table after they were upgraded for a little while. So we create we created this code using the UCX but using the debug in a notebook and the code that would have been I don't know few few pages ended up being half of it is importing so really like eight lines and I scan all the all the tables I see what is the source what is the target and I keep them in sync very very easy to do you can do a lot of thing you have access to All the migrated table, migrated assets, you have access, you have access to a lot of facilities.
Everything that you see in assessment, you have access as a an object in the context. It's very simple. We can provide you example. You can reach out to us for more idea. The most important thing is to copy the notebook out of the folder. It's in the application folder under UCX. The reason is if you will change it there next time you install UCX it will be overridden. So copy the debug notebook before you use it and handle it with care.
With great power comes great responsibility. So make sure you do that. Finally this is how you get UCX. Again this is a a link to the new documentation site. This is your this is how you start how you engage with UCX. From there there's a link to the GitHub page if you want to look into the code or if you have aspiration to use this code for other things and that's what we had. Uh any questions? All right. Thank you very much.
Thank you.

Learn more about the Databricks Data and AI platform.