How Nubank improves Governance, Security and User Experience with Unity Catalog

Summary

• Large-scale governance: Nubank migrated 1.1 million tables and 4,000+ notebooks to Unity Catalog for governance.
• PII protection: Fine-grained access controls and column-level security protect sensitive customer data across 3k users.
• Simplified discovery: Centralized data management improves user experience by simplifying data discovery and access.

How Nubank improves Governance, Security and User Experience with Unity Catalog

Talk by juliano.fonseca juliano.fonseca, Engineering Manager, Nubank
At Nubank, we successfully migrated to Unity Catalog, addressing the needs of our large-scale data environment with 3k active users, over 4k notebooks and jobs and 1.1 million tables, including sensitive PII data. Our primary objectives were to enhance data governance, security and user experience. Key points: Comprehensive data access monitoring and control implementation Enhanced security measures for handling PII and sensitive data Efficient migration of 4,000+ notebooks and jobs to the new system Improved cataloging and governance for 1.1 million tables Implementation of robust access controls and permissions model Optimized user experience and productivity through centralized data management This migration significantly improved our data governance capabilities, enhanced security measures and provided a more user-friendly experience for our large user base, ultimately leading to better control and utilization of our vast data resources.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why did Nubank migrate to Unity Catalog?

Nubank's previous system created data silos and made governance difficult at scale. Unity Catalog provided centralized governance, fine-grained access control, and data discovery capabilities needed for 1.1 million tables and 3,000+ active users.

How does Unity Catalog protect sensitive data?

Unity Catalog provides column-level security, row-level filtering, and audit logging. This allows Nubank to safely share PII and sensitive financial data while controlling who sees which columns and rows based on identity and roles.

What was the migration scope?

Nubank migrated 1.1 million tables, over 4,000 notebooks, and equivalent jobs to Unity Catalog. This represents one of the largest Unity Catalog migrations, requiring careful planning and validation.

How has this improved user experience?

Data discovery is now simplified through centralized catalogs. Users spend less time searching for data and more time analyzing. Permissions work transparently, reducing friction in accessing governed data.

Full transcript

Hello everyone. I hope you are doing very well today. I'm super excited to be here and to be presenting about how Nubank improves data governance, security, and user experience using Unity Catalog. Well. I am an engineering manager at Nubank and I have been working with data for the past 9 years of my career. Mainly in platform teams, but I have been always been very close to the business teams as well. And more specifically, I have been working with Databricks since 2017.
Back then, there were only notebooks, jobs, and all-purpose clusters. So, it's a lot of time. I have a lot of stories. Uh but before jumping in to talk about how we have been uh dealing with governance, security, and user experience at Nubank, let me give you some background about Nubank itself. Uh well, first, also we have the forward-looking statement, so you must have been used it to this today. This information This presentation has informational purpose only.
All right, so about Nubank. Nubank was founded in uh 2013 in Brazil and is currently one of the largest financial service platforms in the entire world. Right now in 2025, only 12 years after the company's foundation, we are one of the largest financial institutions in Latin America in number of customers. Uh we're operating Brazil, Mexico, and Colombia. And we have more than 180 million customers across all of those countries.
At Nubank, we take things very seriously. So, we fight complexity and bureaucracy by providing transparent, straightforward products and experiences to our customers, making sure that they are having the best experience possible when managing their financial lives. And you must be wondering, in a company that is that size, with that many customers, how do we provide those great experiences? How do we fight bureaucracy?
Well, I would say that a great part of providing good experiences, fighting bureaucracy, is being passionate by data. Is using data in our favor to remove all those blockers. So, at the core of our company, besides being fanatical for providing the best products and experiences to our customers, uh is using data to take the most efficient and smart decisions as always as possible. So, in around that process of making the best decisions possible is a series of process that are composed by data development, ad hoc analysis, reporting, and that's that entire process, what we call data exploration process, is made on Databricks.
So, data Databricks is the main tool used at Nubank for the data exploration process and it has been a part of our stack since since 20 2016. Uh and currently executing all of those activities, those data exploration, data development, we have a very diverse range of users. So, we have both business analysts, product managers, uh we have software engineers, but we also have our even our operations team using Databricks to make more informed decisions.
And just for you to have an idea of how important data is at Nubank and how big of a deal the data exploration process is, I'd like to share with you some numbers about how Databricks is being used today. So, currently, we have more than 3,900 uh monthly active users in the platform. We have more than 25,000 notebooks and 120,000 tables being created on a monthly basis in the platform. We have more than 2,300 uh all-purpose clusters being used by all of those users.
More than 2,500 scheduled jobs with 50,000 executions monthly. And all of that is pretty impressive, right? So, those are all big numbers. And you must imagine that to have that scale, it is very important, it is necessary that we have an environment that is well governed, that is secure, that have a lot of controls uh to make sure that we are operating in the most appropriate level. Uh and that's where Unity Catalog comes into the picture.
So, we have decided to use Unity Catalog on Databricks to do governance, to manage our assets, to enhance our security, and improve user experience. But before Unity Catalog, we were managing the whole environment uh mostly using Hive Metastore. As you know, Hive Metastore uh is a veteran of the industry. It's a great tool that for us, it has served its purpose, but as the years went by, it started to impose a few challenges for us.
Uh especially in such a massive environment as the one as we have. And among the main limitations that we find with Hive Metastore is that we had a very little control over personal information through Hive. Uh mostly because Hive doesn't provide any access control mechanism. So, it's a great for the metadata management, for storing our tables, but not necessary for access control. So, that required a series of different solutions to work work around all of those access management challenges.
For us, given that Hive didn't provide that, uh we had to implement a different set of capabilities in the platform and mostly push the access control to the storage level, which was kind of sub-optimal given that we already have a catalog, we have a meta meta data management solution, but even though we still have to manage the access through the through the storage layer. And there was also There was also an an additional challenge that was uh we had some dependencies on the IM pass-through solution from Databricks, which is a solution that is already deprecated and it's not receiving any any updates.
And last but not least, for almost every custom use case that we had, uh the solution was single-user clusters. If we had any custom permission, any set of permissions that we that we should address, that we couldn't with Hive or uh storage storage level permissions, we should use single-user clusters. And at the end, this is very expensive, right? Both from an infrastructure perspective, where in an environment that is such massive such massive with that amount of users, you have a big amount of single-user clusters, uh but also from an operational perspective, having to manage more than thousands and thousands of clusters is pretty demanding from a very from a very small team.
So, this is the kind of scenario that we were at before, where we had all-purpose clusters, which we we used to call the general-purpose clusters, clusters where everyone uh would use and and access Hive Metastore. We had the single-user clusters, uh and then we had the system users and the job clusters. All of those types of access types of uh users, they used Hive Metastore and a few of them even bypassed Hive. They used directly the storage layer uh access control that we had developed.
So, basically, three different types of access using different solutions. Sometimes using Hive Metastore, sometimes using uh directly the storage level uh access control that we had developed. At scale, this was very demanding for us and it required a lot of support uh from our team and it often caused us some problems. And that's when Unity Catalog became so interesting for us because uh we noticed that Unity Catalog could not only address many of those those uh access management uh and governance issues that we had, but also had a bunch of features that were very interesting for our context.
So, for example, uh it for a long time, we we wanted to use uh Databricks SQL and Databricks SQL is only accessible through Unity Catalog, specifically the serverless version. Uh second, it would allow us to break down all of our metadata management into multiple catalogs. So, for example, we could have a staging catalog where users could play around, have their data, uh have more freedom without any big concern uh with the data they were already sharing because it was staging.
We could have the production catalog where we would have more restricted access control. So, users would not be able to write any data set. They would only have view permissions on certain tables depending on the level of access that they had. We could have even a user catalog, that is the catalog where users can put their own tables and manage their own access. So, uh I think this was a great advantage for us instead of, "Okay, we can provide different levels of permissions with different types of catalogs, so we can organize all of our metadata management uh in a in a better way.
Uh And besides that, we were also interested in having more visibility of the platform. And with Unity Unity Catalog, that was possible. Uh mostly through system tables. So, we could have more visibility around the costs that we're having with both Databricks SQL, with all-purpose clusters, with jobs. Uh but we also were able to build a lot of monitoring and observability on top using audit logs, using uh query history through tables, using system tables.
Uh so, we could leverage all of those capabilities to increase the visibility that we have around the platform. And last but not least, Unity Catalog also provide a very good set of APIs that allowed us to manage access and to manage the platform in a more programmatic way instead of depending on infra as a code or even manual operations through the UI. So, this was very interesting for us and uh we could leverage all of those capabilities for our use case.
So, now instead of having to manage those different types of access uh and handle each use case both on Hive Metastore or in the storage layer permissioning, uh we could have only Unity Catalog. So, we could have could have the group clusters, we could have the single user clusters, and any other system that system that wanted to access our tables and data sets uh doing so through Unity Catalog and leveraging the same the same permissions.
Uh And all of that, it looks pretty pretty beautiful and easy in the slides, right? So, when you look at okay, just a bunch of great features, let's move it. But now it's where it comes the tricky part. So, uh actually doing the transition and moving to Unity Catalog uh it is where we face most of the challenge. So, remember in the first slides where I mentioned the size of our platform, right? The number of users, the number of assets.
Well, now that is translated into a real big challenge for us. Uh creating your whole metadata management and catalog structure from scratch, uh I think it's pretty straightforward. It's pretty easy. So, you have a an a new environment and you can create it as you want. But once you already an entire operation running on top of Databricks and having uh such a huge amount of users and assets, this becomes a a a real challenge.
Uh and also at the same time, we didn't wanted to spend years and years in a never-ending migration. Uh because that's very costly, that's not effective, we do not generate the value to the business as we want. And so, we took some actions to avoid that scenario and that's what I'm going to tell you how did we get there. So, I think one of the most important aspects that I should warn you that I should warn you from the beginning is that you should prepare very well for the project.
I know that this might seem obvious. Of course, who wouldn't prepare, right? But depending on the amount of assets that you have, on how deep you are in the platform, how big is your usage, uh it is very complex to migrate. So, I I and I think there are some very important aspects that you should know about before migrating. So, uh a lot will change for both you as probably platform admins, but also for your users. And some changes can be more subtle.
They can be more transparent for the users, but there are a few of them that the users will need to adapt, that the users will have a part of in in into that process. So, if your users, they are used to using Hive Metastore, their experience will change a lot when they move to Unity Catalog. Uh mostly because now they can have a role in managing permissions as well. So, they can create assets, they can create tables, they can create volumes, uh and they can be owners of those assets.
So, when we talk about governance, you should know that they will have a role in that too in how they share those assets. You can control that, you can decide to put everything under a box and do not allow users to do everything, but you can also uh bring them to be part of the process. And I think this is very important, something that you should be aware of. Clusters and notebooks, they do not behave the same with Unity Catalog, especially if you have been using Hive Metastore or any other solution for a long time.
So, not all the same operations that you had working uh in the past, they will work with Unity Catalog. Uh so, your notebooks might break, your jobs might break, and you should be aware of that you'll need to make some changes and even users will need to make those changes when they are coding, when they are developing and using Databricks. Uh and also, instead of accessing directly the storage layer, instead of accessing directly your buckets, now you will have volumes, which is this intermediary layer, this logical layer for you to access data.
So, this can be really tricky for the users as well because they won't even be able to reference the buckets anymore. So, you need to be aware that you need to reference the volumes and that will change. So, users will need to have at least some context about how the uh the storage layer works at Unity Catalog. And in terms of technical changes, it's very important for you to be aware that uh Spark Connect becomes the main protocol for communication between the client and the nodes and the resources infrastructure resources.
Uh that's a very significant change because Spark Connect, uh different from the standalone version, it has some different capabilities where it has a layer of governance where you cannot execute all of the operations that you were used to to execute in the standalone version in the past. So, that's where the code change part, the migrating some operations, some kinds of uh code patterns uh will have to adapt. And at Nubank, this is this isn't our first migration.
Uh we have done a few migrations in the past. It's not that we are proud of it and that we like to do migrations. It's just that they they happen for various reasons. And by doing those migrations, we have learned a few things that are very important, especially in complex and user-demanding uh and especially user experience disrupter migrations. So, I think the most important thing is that inventory is the key when doing a migration of this size.
So, if you do not have visibility of what do you need to migrate, of what is important for your business, or of what are the main assets that are critical for your operation, uh it will be very hard to prioritize and you can end up end up in one of those cases of a never-ending migration. So, the inventory was crucial for us to coordinate the effort with our users uh and make sure that all the impacted teams were aware of what was happening.
And at the end of the project, we have executed a massive migration. So, it was more than just enabling Unity Catalog in a workspace and having just a few tables over there. We had more than 2,000 uh clusters created. We had 1. 1 million tables migrated from Hive Metastore to Unity Catalog. Uh and as I as I mentioned, we had a lot of data, so we have to move 50 petabytes of data around between buckets. And at the end of the project, we realized that more than that what started actually as a governance project, uh as something that we wanted to improve in terms of security and access control, uh had most of the impact in the users.
So, it it was kind of a a a user-facing project, not only a governance one. It impacted the user experiences and how the users experienced experienced the platform. More than 7,000 users were impacted uh across across the board having more than 4,000 notebooks being migrated at the end of the day. And I always like to say that where there are data assets, there is probably an opportunity for a cleanup. Cleanups are very important, they are part of the process, and you should do them regularly.
Okay, so once the inventory was complete and we have mapped everything that needed to be migrated, we have full visibility of what was the scope of the migration, all the users impacted were uh aligned and aware of everything that was going to happen. Uh it was time to actually operationalize the change. So, put in practice all the plan that we have defined to execute this this migration. Uh because at the end we impacted all the business units of the company.
We have users all over the place, it the usage of Databricks is spread throughout the entire company. So, it was a big impact. And at the end of the day for us, it was more than a collective more collective effort than anything else. So, we've tried everything. We had our team a platform team, centralized team doing part of the migration. Uh we had the users executing part of the migration. We had We have hired external partners.
We've tried AI. We've done it all. So, for what was considered the most important assets that were critical for the business operation, such as jobs, clusters, and very important process, we took the ownership and we've migrated ourselves, the data the platform team, to guarantee that we would have the minimal disruption because those were very important and very critical process for the business. So, uh on the other hand, uh at the at the end of the process, there was a very interesting lesson for us.
We learned a lot. So, uh clear and straightforward communication can be an asset in a migration this size. So, being really clear to your users of what is expected of them, of what is going to happen, of what are the impacts uh they're going to have on a day on their day-to-day, it's super important. And we were able to engage our users to migrate their uh medium-to-low complexity assets uh and engage them into being part of this process as well because as we have too many things, we could not centralize into into a one single team to execute everything.
So, creating playbooks, creating documentation, uh and engaging them to be part of this process was super important and it was a uh an important lesson that we have learned. And during the migration process, it was not all flowers for us. So, of course, given the size of our environment, given the amount of users and assets that we have, this was a challenge alone. But, we also had some other issues that really impacted the migration uh and like it was sometimes a bit hard for harmful for the project, but we were able to overcome it.
And given how significant were those challenges, how how hard it was for us to solve it, I would like to share with you a few of them so you have visibility and you'll be prepared in case you are doing this the same kind of migration. So, by far, the biggest technical challenge that we had was the code migration. That might be That might not be the case for everyone, uh but it's really important for you to be aware and do not underestimate the code incompatibility that you might have.
In the bank specific code base in the in the bank specific case, uh most of our Spark code base is entirely on Scala. And with Unity Catalog, as I mentioned in the past, we have the enforcement of Spark Connect, which which is this different protocol for communication between the client and the nodes and the executors. Uh so, with Spark Connect in shared clusters, which are the the best way possible to use a Unity Catalog, you have multiple users with different set of permissionings permissions accessing different data into the same infrastructure resources because Spark Connect layer allows that.
Uh but with this new protocol, we have some capabilities and so some APIs that do not work the same way as they used to. And we've done a a big migration in terms of code. We have adapted most of our code base, but at the end, there were some APIs and some functionalities that we were not able to adapt it. Uh and that was really hard. We've worked with the Spark team. We've tried to launch a new DBR version fixing the problems that we had, but there were some problems that we were just not able to solve it.
Uh in Python and SQL, most of these changes, they are kind of transparent for the users because Python kind of already have a an abstraction layer when we talk about Spark. Uh but that's not the case for Scala. So, those limitations were a blocker for us and it made us change our strategy in the last minute. Like we have done everything and then we noticed that we wouldn't be able to use shared clusters. Uh so, we had to shift our strategy and start to using group class group clusters instead.
Uh so, group clusters, if you're not familiar with it, but they they can use Unity Catalog. They can leverage the same functionalities and permissions that we have with Unity Catalog as a whole, but instead of having multiple users with different types of access inside the same infrastructure resources, you have one cluster that have one group and all the users that are part of the group of that group have the same permissions.
So, it's less granular than having the user level permissions, but at the same time, we could leverage Unity Catalog and it was a matter of adapting our strategy to use those those shared clusters instead. Still around code migration, one curious thing that we noticed during this process of moving to Unity Catalog was that notebooks have the tendency to be very ephemeral. So, users create dozens, hundreds, thousands of notebooks per month.
Uh and most of them, they never look back. Like it's a one-time thing. So, this is really important when you are considering transition to Unity Catalog and having to making having to apply a bunch of changes because sometimes you can find a blocker because a given set of notebooks is not compatible, but when you look at it, like those notebooks are not relevant. So, this can really help you avoid this kind of dependency that can even block and hurt the project.
The second challenge is around the storage migration. So, in Unity Catalog, there is no DBFS. So, the storage layer was was placed was replaced by volumes and that mean that all of your buckets, every reference that you have to a bucket, it now needs to reference a volume. And volumes are very interesting because they provide a very low-level access control. So, you can manage permissions at an user level in a volume.
Doesn't necessarily need to be at the physical data. So, it's it's this intermediary layer that provides a lot of control for you in terms of governance of the platform. But at the same time, if you have hundreds or thousands of buckets, you need to replace every single reference to those buckets by the volumes. You need to mount volumes for all of those buckets. And this can be can this can be a a a bit challenge. Uh in our case, it was a a huge challenge because users were used it to create their own buckets.
They had a lot of autonomy in that process. So, they created a bucket, they stored some data there, they queried and they used it through Databricks. Uh and now with volumes, we had to create all of this intermediary layer of access with volumes. And it was very disruptive for the for the users. They didn't really understood why do they had that layer of access because they were used to buckets. Uh at the same time, at the end, uh they they really engaged into that and they saw the benefits of having more control and don't even having to worry about buckets about the physical data itself.
This is not exactly a challenge, uh but rather a learning that I would like to share is that when you have when you already have an entire operation running on top of Databricks and you are planning to implement Unity Catalog, uh it will basically turn everything upside down. I must be honest, that was the case for us. When we first first started, everything was different. Uh users will need to code in a different way.
They will have permissions now, something that they were probably not used it before. The clusters will not be the same. Uh the way they access the data will not be the same. So, in that context, uh what I can say is avoid surprises at all costs because uh there might be surprises that can be really costly for you in terms of timeline, in terms of effort. So, talk to people. Uh understand how do they operate on a daily basis.
Uh and understand what is the type of work that each kind of each type of users of yours does in the platform. So, if you have time and resources to do that, implement observability and monitoring before migrating. That can really help you throughout the process of uh moving to from a legacy environment to Unity Catalog because you can know how things are progressing. You can know if you're having an incident, if you're reaching any limit of the platform.
So, it really pays off. Uh and of course, it can help you avoid going into the wrong wrong direction in in the project. Uh and be aware that depending on the size of your environment, you might need to deal with uh more costs. You'll probably need to duplicate assets from tables to jobs to clusters. You'll need to validate that. So, when planning uh a movement of this way, consider to put some budget reserve some budget for this kind of project.
You don't want your finance team reaching out because you're over budget and you have to run around to save costs in the middle of migration. All right. So, where we are at today after all of that effort, all of those challenges, and all of their those learnings. Well, I can say that for sure at the end of the project we are in a much better place than we were when we first started. Uh mostly in those through three very important pillars that I mentioned.
Governance, security, and user experience. We have an access control layer defined at a group level. We have all users, all Databricks users, being part of a group and having their defined set of permissions, the tables that they have access to, the volumes that they have access to. Uh we have proactive monitoring and observability of the platform using system tables. We are able to get at a very granular level and have really control of everything that is happening, having a lot of visibility uh of how the platform is behaving, what users are doing, uh how much each component component is costing, uh how each feature is being adopted, and including life cycle management of assets.
As I mentioned, there are a lot of assets that are ephemeral and it's important to that you put in place life cycle management policies to make sure that you have a organized environment that and that you can can keep scaling even for a very large user base. Uh and with Unity Catalog we have a a granularity uh of access that allow us to address uh all the requirements from regulatory institutions uh and being compliant to the industry standards around personal data sharing, around strategical data sharing from the company.
Uh in terms of secure security, we are in a much better place where we were able to really apply all of those policies and remove the burden of having to manage a bunch of different components that would manage that same kind of access before. Uh and I can say that one 100% 100% sure that the experience overall is way better. Mostly because now the experience for users is more cohesive. They have the same access in notebooks, they have the same access uh in Databricks SQL, uh using DLT, using basically every feature of the platform.
So, all those access are shared. Users don't need to worry, oh do I have access to this data in Databricks SQL? Do I have access to this volume in notebooks? No, like it's the same experience across the entire tool. Uh they don't need to basically deal with different behaviors and this was really a pain point before. Uh and what started as a as a pain point became more of a collaborative journey uh of users being responsible with a high level of autonomy for their own assets.
So, now they manage permissions, they are responsible for what they create, uh they know who they share their data with, which is something that we didn't have visibility before, even the users didn't have. So, they do not depend on a centralized team anymore. And this was very important to us. Uh and at the end uh I think this this project was very challenging. It was more than 9 months working in this migration. Like it was really really demanding from the team.
And I think kind of migrations are all about that, right? It's really demanding. It's really expensive. You need to put a lot of effort. Uh but when you are doing the right thing, it always pays off. So, I think that's it. Thank you so much for your time.

Learn more about the Databricks Data and AI platform.