Skip to main content

Deploying Unity Catalog OSS on Kubernetes: Simplifying Infrastructure Management

Summary

  • Kubernetes deployment: This video demonstrates how to deploy Unity Catalog OSS on Kubernetes, leveraging cloud-agnostic infrastructure for efficient governance at scale.
  • Helm automation: Helm simplifies installation, configuration, and credential management, making Unity Catalog deployment straightforward even for complex environments.
  • Enterprise readiness: The approach provides high availability, security, scalability, and disaster recovery—essential for enterprise data governance infrastructure.

Deploying Unity Catalog OSS on Kubernetes: Simplifying Infrastructure Management

Watch: Deploying Unity Catalog OSS on Kubernetes: Simplifying Infrastructure Management
Talk by Vasilii Bulatov, Senior Software Engineer, Nebius
In modern data infrastructure, efficient and scalable data governance is essential for ensuring security, compliance, and accessibility. this video explores how to deploy Unity Catalog OSS on Kubernetes, leveraging its cloud-agnostic nature and efficient resource management. Helm makes Unity Catalog deployment simple and easy by providing a simplified installation process, easy configuration and credentials management.
The session will cover why Kubernetes is the ideal platform, provide a technical breakdown of Unity Catalog on Kubernetes, and include a live showcase of its seamless deployment process. By the end, participants will confidently configure and deploy Unity Catalog OSS in their preferred Kubernetes environment and integrate it into their existing infrastructure.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why is Kubernetes ideal for Unity Catalog deployment?

Kubernetes provides cloud-agnostic infrastructure, efficient resource management, high availability, scalability, and automated disaster recovery—all critical for enterprise data governance systems.

What role does Helm play in Unity Catalog deployment?

Helm automates the installation process, simplifies configuration management, handles credential management securely, and enables repeatable deployments across different Kubernetes environments.

How do I set up Unity Catalog OSS with Spark?

This video demonstrates deploying Unity Catalog open-source on Kubernetes, showing how to set up the governance layer with Spark. The process uses Helm for simplified infrastructure management and enables organizations to run open-source Unity Catalog locally while maintaining enterprise governance.

Do I still need to manage infrastructure with serverless?

This video shows that while serverless Databricks eliminates certain infrastructure concerns, open-source deployments on Kubernetes still require management. However, the benefits include complete control, customization, and the ability to run governance on your infrastructure alongside Spark workloads.

What are the main components of Unity Catalog OSS?

Unity Catalog consists of a UI for user interaction and a server handling governance logic. It also requires a data backend for storage, networking, and security infrastructure for enterprise deployment.

Full transcript

Um so today I'll be presenting a new way to deploy Unity catalog oss. Um spoiler alert it's Kubernetes. Um so quick question um how much how many of you are familiar with Kubernetes? Please please raise hand raise hands. Okay awesome. Um then you will be like familiar with everything. So let's move on. Um yeah, so. Um I work at Nibbius. Uh we are a GPU cloud company and I specifically work at managed services uh team.
We provide a lot of managed services and I do a lot of stuff with Kubernetes um like every day. Um so uh let's dive in. Um so what is Unity Catalog OSS? Some of you may be not familiar with it. It's relatively new. It was presented by data bricks last year at last year's uh data NA summit. Um it's basically an open source implementation of Unity catalog. Um it's completely open source and free to use. Um yeah, feel free to scan a QR code to learn more.
There's a lot of stuff. Um uh yeah. So what could be difficult about deploying Unity catalog, right? Um it basically consists of two main components of a UI and a server. Um seems simple enough but if we dive deeper um a bit deeper we find find out that there's a like data back end where your data is obviously stored. Um yeah seems simple enough still um but I mean we need to somehow provide a users of that unity catalog access to that catalog.
So somehow we need to manage networking um provide access um like open to specific like corporate network or like some uh secure way. Um yeah uh we need to protect everything uh if you're an enterprise. So we want uh uni catalog overall like database all the components be secure and to not leak data anywhere. And of course uh we want to be ready for some emergencies, right? So we don't want to lose connection to all of our data the most busy day of um like so we want it to be highly available.
We want to have multiple replicas of everything and we want it to be like stored preferably in multiple data u data centers and all shebang and yeah so we want Unity catalog I mean for enterprise for um business use to be highly available we want it to be scalable we want to manage it uh pretty easily like to not uh spend hours trying to do a small update right uh we want it to be disaster ready and secure Um and I mean basically that's um we want Unity catalog to have following technical stuff from that.
So we want Unity catalog to be able to support multi- instance support um authentication authorization. Uh we want our database to be highly available. We want it to be um have backups be ready for any disasters. Uh we want uh it to be secure. We want to manage routing. We want to be like load balancing and all that stuff. Uh and of course for in infrastructure guys we want it to be um manageable like to not increase their already quite high load u most of the times right.
Um yeah so what are these uh what of these stuff is already covered right? So first two are covered by uni catalog itself. Um uni catalog already supports multi- instance support. It uh supports um authentication using oath. It uh it supports a role-based access control and all that stuff is already available um for a long time. Um next two are basically handled by your database provider. I mean most of the most of you um probably use a cloud like Amazon um like Azure or anything else um that already provides you like with a simple highly available setups and backups and for last part like um there's no apparent solution yet um until now.
So um I would like to introduce Kubernetes as a solution for like um the rest of the stuff we like it will allow us to deploy it easily. It will will allow us to configure networking and not really have a big issue with that. Right? So why Kubernetes? Um Kubernetes is an industry standard. Um most of the companies use it. Um most of the uh cloud providers um have a offering of their Kubernetes products. Uh it's been around for years.
So it's reliable and stable and it has a huge ecosystem of various different extensions that allow you to do magic on uh Kubernetes cluster. Um and um since the beginning of this year, Unity catalog is already has Docker support. So it's a perfect match. Um yeah, so we have um like some kind of like this architecture like um how can we move it into Kubernetes? Um so yeah here's um like graphical uh representation right so um we are basically providing a helm chart uh which is basically a recipe um to deploy unity catalog on kubernetes cluster and allow you to manage it configure it and don't really um like suffer from it um like uh it will sort out networking it will u sort out uh deployments and like if your for example one of the instances is gone because of a hardware issue on the like underlying cloud.
Um it will just restart it and rescheduled on a different node. So that won't be a problem. It will be highly available. Um you can observe it, monitor it using like regular Kubernetes stuff. As I said before, it has a lot of um ecosystem and there's a lot of different uh options that you can go for. Um and yeah, database provider basically covers most of the stuff. Um yeah so actually from the out of the box Unity catalog provides a file database probably if you have already opened Unity catalog OSS probably try to run it.
Uh it uses um H2 database but under the hood it uses hibernate. So basically you can use any database um but out of the box it works perfectly with uh posgress. So um yeah, Helmchart allows you to either configure a one instance with a uh files database or a multi- instance with posgress. So um what does Helmchart allow us to do? Of course we have Unity catalog on Kubernetes. That's itself like quite a big step, right?
Um it provides us all in one solution. We have we get networking, we can configure security as we want. Uh we have scaling, we have high availability. Um and one of the really important stuff it is cloud agnostic so it will work on any cloud provider or even on premise um manually spun up uh kubernetes cluster so that is not an issue and um we've tried to make a helmchart really configurable so you can configure um like how many uh UI or server instances you want you may want to configure database uh even more stuff I mean you can spun up sidecar containers Um and a lot of different stuff um like resources all the Kubernetes stuff you can configure and extra you can configure security options you can configure authentication right right here um you can configure um like unique catalog settings uh various really deep security stuff um and yeah so it is actually already merged into main branch it's not yet available on some public registries yet it will be fixed in a in fixed soon.
I I hope. Um yeah, so you can uh try it. Um yeah, here's a QR code where you can dive in. Um and I would appreciate if you would contribute to that and make it better. Um yeah, so what's um can we improve actually? Yeah, let's uh touch a bit of that. Um yeah, so actually yeah, so we want to bring a chart to public registries to allow even more people to just install it. Um, we want improved security. We uh want uh Unity catalog to be secure out of the box.
So basically you install it, you get them like good enough secure uh settings. Right now there's um it's it's a first version. So there's of course hacky workarounds. Um but that's u will be fixed soon. Um yeah and user management on authentication. Um that's um unique catalog issue probably um I mean that's what I I had issues with and I would be happy to contribute and improve that. Um yeah and of course uh we discussed with the team like there could be a more exciting stuff actually with integrating with more cloudnative ecosystem like uh more deep integration with various Kubernetes native stuff.
Um I won't say like uh specifically what they are but there's a lot and I mean it's exciting. So basically you can dive in and join us take a look. Um yeah so this basically concludes the part of presentation and I would like to do a live demo. Yeah. Uh let's um take a look. Um yeah probably too small. Let's make it bigger. Um yeah so uh I have um like cloned the repository and we are basically ready to go. I have already spun up a Kubernetes cluster and uh just want to like try and deploy um Unity catalog um Unity catalog uh I want to do it nameace you see and um I want to create a namespace if it doesn't exist so hopefully it will work um yeah so it's uh started deploying Um yeah, let's take a look uh here.
So yeah, we see um in instances spinning up. Um so this will be a single instance Unity catalog um that will have one server and one UI. Um there's a limitation of uh file uh based uh file based uh data storage that you cannot basically share it between multiple instances and even if you can it won't be stable and good enough. Um yeah so it's um going a bit slow um I hope it will spin up pulling image so hang in there uh yeah so I will tell a bit more about the helm chart um it supports um ingress and um gateway API integration out of the box so basically if you're spinning up a like more like uh more or less a enterprise like uh installation, you can basically use that to instantly get access to your Unity catalog like integrate with external DNS for example um to instantly get a DNS record that will already like pop up the screen of like pop up the UI um and uh go with it.
Um yeah, so we see that server has started. We can take a look at the logs. Yeah, so it's started. Uh we were waiting a bit for UI. Yeah, could it should take half a minute probably. Um we have some time. Yeah, so it's uh running. Um we can do port forwarding and we should uh dive right in um to the UI of Unity catalog. Uh okay um starting up. So yeah um that's um uh Unity catalog UI. Um this is some catalog that I created before.
Um I mean I can create some um some cat schemas. Um I can create basically anything and it's uh good to go. Um yeah and actually I have one more demo here um of the cluster that I've actually like was crafting the whole night. Um yeah so it is basically a um u multi-instance uh works on posgress and has a um like let me quickly uh take a look here because I forgot the name of the DNS records. Um, yeah. Yeah. So, here it goes.
Um, yep. Um, it has um all off with Google. Um, it has a like it has TLS, so it's like secure connection. You can work in it. Uh, you can connect like using your uh Unity catalog CLI. Um and it uses our Helm chart uh with some integration with cloud. Um and yeah, uh it was not that difficult to do. Um and you just need to like set up all you need to set up uh your S3 credentials. You just need to hook it up in one single uh values YAML.
Um which we can take a look here. Um one second. Oh no, that's wrong VS code. Sorry. Um, yeah, that should be it. Uh, yeah. So, val YAML uh we have like quite a big uh values file but don't worry it's quite documented. So if you open up in GitHub you will see um like all the like the fields how to configure it uh some examples and like you can configure it um like like I I did here like so I set up a like connection to um Google's cloud storage.
I've made a like credential secrets I and I reference it here. I've made a Google Oth as well and also referencing it here. I mean uh I'm setting up u users like initial users here. Um I mean that's a workar around unfortunately. Um yeah so I'm configuring u posgress credentials and it all is done using uh kubernetes secrets. So it is quite secure. you can configure I mean here I'm dropping all the like security stuff dropping capabil Linux capabilities running it as a user like doing um resource management um I mean all that stuff is already available um yeah so I think that's basically it we have five more four more minutes I mean if you have questions um yeah uh let's let's

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.