Nubank's Fine-Grained Access Control at Scale: From Permission Groups to Unity Catalog

Summary

Nubank, the largest fintech bank in Latin America with over 130 million customers across Brazil, Mexico, and Colombia, built a permission groups policy engine that translates access policies into Databricks Unity Catalog grants across hundreds of thousands of tables and thousands of users.
A three-line defense model ensures consistent governance: governance-aware data owners classify their data, platform-provided default rules enforce a private-by-design baseline, and compliance-driven auditing demonstrates adherence to data protection regulations across multiple jurisdictions.
Read-time column masking protects sensitive data without modifying underlying stored values, and a permission group taxonomy covering domain-level, subdomain-level, and column-level access is synchronized with Unity Catalog through SCIM APIs.

Nubank's Fine-Grained Access Control at Scale: From Permission Groups to Unity Catalog

Enterprise data access control at scale across multiple platforms presents a fundamental challenge. When managing hundreds of thousands of tables and thousands of users across fragmented systems, ensuring consistent permissions becomes nearly impossible. Nubank solved this by designing an internal permission groups policy engine, then translating these policies into Databricks Unity Catalog grants.
Learn how Nubank implements fine-grained access control using a three-line defense model: governance-aware data owners, platform-provided default rules, and compliance-driven auditing. Discover the permission group taxonomy covering domain-level, subdomain-level, and column-level access. Explore how policy translation works through SCIM APIs, how read-time masking protects sensitive columns without modifying underlying data, and why this architectural pattern applies to any organization building enterprise data platforms on Databricks.
🤝

Chapters

FAQs

How does Nubank manage access control at scale on Databricks?

Nubank built an internal permission groups policy engine that translates access policies into Databricks Unity Catalog grants using SCIM APIs, rather than integrating Unity Catalog directly into each of their many subsystems. This approach covers domain-level, subdomain-level, and column-level access in a single consistent framework that spans batch ETLs, streaming Flink applications, ML models, and human analysts.

What is Nubank's three-line defense model for data governance?

Nubank's three-line defense consists of governance-aware data owners who classify and manage their own data, platform-provided default rules that enforce a private-by-design baseline, and compliance-driven auditing that demonstrates regulatory adherence across Brazil, Mexico, Colombia, and Europe. This model ensures that access to personal and sensitive data is always scoped to a declared purpose, satisfying data protection laws in each jurisdiction.

How does column-level masking work in Nubank's Unity Catalog setup?

Nubank applies read-time masking functions at the column level within Unity Catalog so that sensitive data is obfuscated at query execution without modifying the underlying stored data. The masking functions check the querying user's group membership and return either the real column value or a masked substitute based on whichever policy applies, enabling compliance without data duplication.

Why did Nubank choose a policy translation approach instead of integrating Unity Catalog directly into every system?

Nubank operates across multiple subsystems built over more than a decade, each with its own design philosophy and technology stack, making direct Unity Catalog integration into every system impractical. By building a policy translation layer that projects a single source of truth into Unity Catalog grants, the team enforces consistent access control without requiring changes to every consuming system or data pipeline.

Full transcript

[00:09] Hello everyone. Good afternoon. Uh I guess this is the last talk. I'm not sure. Uh but I hope everybody had a good lunch and I'll try not to bore you all uh with a lot of details and keep it simple. So let's get started. Um
[00:24] standard forward-l lookinging statement here. Please complete your surveys and hello again. I am Nikita. I am the staff engineer at New Bank uh in the data platform team. For those who don't know about New Bank or haven't heard of it uh
[00:41] or hearing it about first time, it's one of the leading financial services platform uh at the moment and it's the largest fintech bank of Latin America. So we have approximately 130 million plus customers serving across Brazil,
[00:59] Mexico and Colombia. And today we are going to talk about how we implement access controls. So you have been hearing probably a lot about access control, open lakehouse architectures, Aback and how to scale access and apply
[01:16] it consistently. And I'm going to talk about how we do that at New Bank. So a quick regulatory check. I will spare the legal details here, but as I mentioned, we operate across Latin
[01:33] America and soon US and we have around 10,000 employees across uh the Brazil, Mexico and Colombia as well as Europe and US. So we need to abide by the data protection regimes in in the regions uh
[01:50] mentioned here. But all these laws converge on five basic principles that is not uh a lot of people should have access to personal data or sensitive information.
[02:05] We need to know our data that is how sensitive it is where it is. The laws also go ahead and mention about technical measures like masking and pseudonmization and not just to have policy documents.
[02:23] Also actions should be scoped to a declared purpose. That means if we are collecting data only for credit card or for loans, it should not be used by the marketing team. So it should need to be scoped uh for that purpose. And last but
[02:39] not least, uh we need to demonstrate compliance and not just documented. So this is a very simplified view of our data landscape. uh it's a composition of multiple subsystems
[02:56] and our storage layer mostly is based on AWS in S3 and if not all most of the data is loaded onto data bricks for further analysis. So a table can be accessed by batch ETLs
[03:14] uh streaming flink applications ML models uh and then there are human users on data bricks uh that also need access to data. Now each of these subsystems that we have it's built over a a decade they
[03:31] have their own design philosophy their own text stack and their own access controls and with this fragmented access control audit becomes really hard and there's no single source of truth of who has access to what and or who should
[03:46] have access to what and moreover it's not a good user experience uh as well because for example if a user wants to access credit card data coming from a streaming application uh and join it with credit card data coming from ML
[04:02] workflows. The user might or might not have access to one of the two tables they want to join. So with this we want to have consistent access across uh all our subsystems and the different platforms that we use.
[04:19] Moreover, a table uh we have a lot of tables that are really big and can have 50 to 100 columns even and these columns will have different sensitivity levels. So a user does not need to have access
[04:35] to all of these columns, right? So implementing data minimization with the fragmented access controls is also quite hard. And all these issues become tenfold um when the tables are growing from a few
[04:50] thousands to a few hundreds of thousands of tables that we have today and being accessed by approximately 10,000 users on data bricks that we have and for that reason uh we designed access control model inhouse. Now you
[05:07] might have heard a lot about the catalog federation. catalog federation. We could use the multi-engine platforms like Unity and Glue, but the federation does not really federate permissions, right? And this is exactly why we build the
[05:24] access control model inhouse to have a single source of truth uh for who has access to what and then use this platforms to translate our permissions uh on the platforms that we don't own. Let's see how to do that. So our model
[05:43] is based primarily on data domains and data subdomains which is a direct consequence of us uh adopting data mesh as our architectural paradigm and data domains are the the bigger
[05:59] business boundary and subdomains are the granular functions within it. Ultimately uh the data domains and subdomains translate onto unity catalog as permission uh groups and we'll see that later how but
[06:15] the data domains and subdomains is a strict two-level hierarchy. So every subdomain will belong to a domain and every table uh belongs to a subdomain and at both the levels like data domain and the subdomain we have people that
[06:31] are accountable for governance at that layer. So for subdomain we have data owners which are responsible for approving or denying access requests uh to the data for that subdomain.
[06:50] Now the key principle that we operate on is that keeping tables private by default. That is when data is ingested in our data platform coming from the transactional environment like data sources we have datomic uh Dynamob and these this ingested data is more
[07:07] prone to have more sensitive data. So these tables uh are kept private to their subdomains and the batch transform table which are aggregations and which is mostly meant for data consumers to use those tables are kept private to the
[07:23] domain. So these are the platform rules that we apply by default when a table is created and the data owners can uh go and override these default settings uh anytime they want. So they can make a table more broader and make it public or
[07:40] they can restrict access by making the table private from to domain to private to subdomain. And most human users should fall or will fall under the least friction access. That means most of the the user should
[07:56] fall under the domain category. That is they should have access to their own tables without being able to request access. And this model is backed by three lines of three lines of defense. One, the data owners who are responsible for conf
[08:13] configuring access. So they know their data and they should be able to configure if it should be public or overwrite based on uh their knowledge. Then second is the platform team. That's us providing the default rules providing the system to be able to control access.
[08:29] And third is the the audit and the compliance teams that we have that translate the regulations to requirements for implementing controls. Now the more sensitive the data the more
[08:45] friction to access it. So the common cases as I mentioned uh should require least uh should not require any access requests or tickets uh to access the data. So if a table is public and if it's p private to the domain and if
[09:02] you're part of the domain they can just go and access the tables on data bricks and they don't need to request um access. Now if the user is making cross-domain queries or if there are tables that are kept more private within
[09:17] the user's domain for example HR so there might be multiple people in HR but for example employee salaries need to be more confidential so the the owners of the subdomain that handles the salaries can keep it more private in that case
[09:35] the user needs to request access uh to that subdomain and in both these cases access like the zero friction path as well as when you need to request access the personal or the sensitive information is still not accessible and if you need access to the personal data
[09:51] the user uh needs to make another request uh to get access to the personal data which is time bound. So you cannot have indefinite access to the sensitive information. uh the we'll see uh later as well the uh
[10:07] personal data access is associated uh to the subdomain. So you will get access to the personal data only at that subdomain level and not uh all of the data.
[10:24] Uh the the main mechanism that ties this model together uh is what we call as the permission groups. These can be considered as our policy engine. Um and just a note uh that the the domain
[10:40] subdomain organization uh might look very similar to uh uh company structure right the people structure that might be in the Oracle peopleoft for example but it's still deliberately kept different because if
[10:56] reors can happen in a company more than we can imagine right so but the data organization does not need to change um and the permission groups are tied to the data organization. So we have all domains permission groups.
[11:11] So we have four types of permission groups. All domains which cover the public use cases like the public tables when you join the company you're part of that group you can go to data bricks and access all your all the public tables for the domain permission groups. We
[11:27] again map it. So there is a fine link between the data organization and the uh company structure but they they're still separate. So for the domain permission groups we do still add users automatically in the domain permission
[11:43] groups based on their division or business unit uh where they are working in and they will get automatic access to the tables that are private to their domain. And if the user changes teams or changes their uh divisions or business
[11:59] units uh the access is revoked and they're automatically added to the uh new business division or the new domain permission group. Then are the granular access that is the subdomain permission group where we by default only add the
[12:15] data owners uh of that subdomain and the approved members. So the members that are approved by the data owners. So this is request based and only the data owners get default access uh to their own data. And the last one
[12:31] is the column level permission group. So every column level permission group is associated with a subdomain permission group. And as mentioned earlier it is time bound. There is no default access. Even the data owners uh are not added to this permission group by default. So
[12:49] they need to request access. So another data owner or data accountable responsible can grant access to the personal data or the sensitive data. Uh the key here is so every table is associated with a permission group and
[13:04] once uh the table is associated with the permission group and you're part of the permission group the the user gets access to the table. Um so the permission groups like the whole
[13:20] membership management and the permission groups it only gives us who has access uh who has access but the enforcement happens on the the platforms like AWS and and data bricks. So the the abstraction that permission groups gives
[13:37] us it does not really enforce the the policies. It's a internal system uh that manages the policies. So it's deliberately platform agnostic uh so that we can use uh this policy engine to be able to have the consistent
[13:53] permissions across different internal subsystems as well as the different platforms uh that we use. So we looked at a couple of alternatives to have our policies through permission groups on these different platforms like direct
[14:10] integration gateways authorization bridges but with the engineering costs and the latency for reading the data was quite high. So we went ahead with policy translation and what that means is that
[14:25] for example if we have a new platform later we don't need to recreate the whole policy engine we just need a new translation mechanism and for data bricks since datab bricks is investing a lot in unity catalog aback uh and everything related to access on the data
[14:41] lake layer we leverage what datab bricks gives us and translate our uh internal permissions to the data bricks permissions permions and let's see how the translation works. So we have a permission group service uh
[14:58] in our environment which manages the creation removal of the permission groups as well as the membership management based on the the structure as well as when the domains and subdomains are created and the permission groups uh
[15:16] service then creates the account level groups on data bricks to so we have a one-on-one mapping between the permission groups that we have internally and the account level groups that we have on data bricks. Now why we
[15:33] have account level groups is so that we can have consistent access on all the workspaces uh in data bricks and also unity catalog does uh table level grants on account level uh groups uh and we use the skim APIs for
[15:52] translating or creating the permission groups and also the membership management. Now the policy enforcement. So we saw who gets access. Now let's see how tables are made accessible to those
[16:08] users. So there are three layers. One is the table level access. Then is the column level access and at query runtime unity catalog actually makes sure uh to enforce those permissions. So the query uh layer is the third one.
[16:25] So everything we discussed until now the permission groups the skim translation the membership management they all converge at one moment that is when a table gets created or registered in unity catalog. Uh our internal
[16:40] subsystems they are required to have the permission groups as a metadata on on the tables uh when they when they create the data assets. So our table control plane then validates on this metadata along with other data
[16:57] lake requirements. Once it all looks fine, it triggers the permissions through our translator on unity catalog. So when a table gets created and if it has permission groups it's basically a unity catalog API call that applies the
[17:14] grant on the table that is which usually looks like grant select on the table to this account level permission group which map maps back to our internal permission group. So and unity catalog's default behavior
[17:30] works in our favor. So if the table has no permission groups configured that means the table is private by default and it's not visible to anybody. It's it's completely private and there are no table grants on it. So nobody can access that table.
[17:46] Now column level strategy. So moving on to the path with most friction. Internally we have defined uh a sensitive data taxonomy which is a bit of a broader categorization of our
[18:02] detailed classification that we run on every tables which is personal data like tax ideas, names, addresses and so on. Financial data, balances, transactions. We have sensitive data uh like the origin, health, biometrics and could be
[18:20] IPS, geoloccation uh and so on and we have confidential uh data that is could be internal business metrics, strategic data or MNPI that users want to keep confidential and the strategy again here is to use
[18:36] what Unity cataloges gives us that is the readtime masking and there are a bunch of benefits that we get from this is for we use hashing as our main um uh technical measure to mask
[18:53] the data and like to protect the sensitive data based on the categories. So with the right time hashing we if we change we we basically change the the raw values right. So and we have to maintain the hashed data and so on. So quite expensive. So read time
[19:09] masking actually helps us not to tamper with the data. The data is as is and unity catalog does the heavy lifting of applying the column masking at query time. Also when we rotate solves to keep changing the the hashes, we don't have
[19:26] to rewrite the whole data if we would have done right time masking. It's merely about uh editing our functions that do the column masking. Um and Unity catalog will again make sure to apply the changed uh hash
[19:42] and classification uh is reflected instantly. So we basically just change the masking function on the column or or remove it if the classification change which is quite instant with the the APIs uh that
[19:57] we have. So how the column level enforcement happens is whenever a column permission group is created when we create the permission groups we also create a userdefined function in uh in data bricks with which
[20:16] has the hashing technology or the hashing algorithm that we use the salt reference and uh the check that actually checks apply the mask or unmask only if the user is is in the column permission group. So the column masking functions
[20:34] have these three important information and when whenever a table is provisioned so our table control plane makes sure to check the classification inventory if there's sensitive data it will then trigger column masking through APIs uh
[20:52] and apply the the mask on the columns of the table in question. So that's how column masking takes place. Now looking at the full chain, we do
[21:07] apply column masking first. So whenever a table is being provisioned, we check the uh provisioned column masking function and alter the table and set the mask on unity catalog. If the masking fails, we do not apply table
[21:23] level grants and we will keep on checking until the column masking is applied. Uh then the table grants are applied. And this is very important because we don't want to have the sensitive data uh open even if there's column masking failed. So we uh we avoid
[21:40] giving table level grants and looking at the whole chain. Um that's how the entire end to-end flow looks like where unity catalog sits at the heart of the the enforcement of the
[21:56] permissions. So whenever data domains and data subdomains are created by data owners the permission group service provisions the column masking functions as well as the permission groups through the schem
[22:12] APIs on data bricks. The data producers owners will create tables. There will be a classification that runs on it and the table provisioning mechanism, the table control plane will make sure that it
[22:27] adds the column masking as well as the table level grants uh on that table in Unity catalog. And when the data consumers try to access this information, uh, Unity catalog makes sure that okay, does this user have permission to access
[22:45] the table? If so, sure. Um, the user can access. If not, the user will get permission denied. And when if the user is accessing the sensitive data, the columns will get
[23:01] masked by Unity catalog. So if you're member of the if the user is a member of the particular column permission group they will see the raw data and if not they will see the the masked data.
[23:18] Now all of this yes there's no Aback in here. Uh and we've seen a lot of talks in the summit regarding Aback but when we started the project uh I think it was a bad timing uh Aback did not fit
[23:36] uh what we wanted. So it was a timing issue and hence the whole model uh and the translation that we do today is not based on Aback. Uh also Aback requires a higher version of data bricks runtime and we deliberately run on lower
[23:53] versions because datab bricks is our biggest query surface and we don't want to break things for users by continuously upgrading without testing. So we take our time to to test the the new versions and then go on to the the
[24:09] higher version. Also the limits that ABACK had at the time uh did not fit our footprint. So we are looking at hundreds of thousands of tables uh to apply the policies and the current limitations on
[24:25] the uh the number of policies on schemas and cataloges did not fit the solution at the time. But that does not mean that we are Aback is wrong for us or nothing. We are continuously watching Aback. And
[24:40] again with everything announced this week, I think we have much better options now to put Aback on top of the translation that we already have especially with uh user identities can be tagged um and things like that. So we're continuously monitoring that. So
[24:58] yeah uh we will look at AVAC and try to incorporate it in our solution. Key takeaways so far. So to be able to have consistent access across different multi-engine platforms
[25:14] as well as our internal subsystem. It's it's useful to build the policy engine inhouse and then leverage what the native platforms are giving and translate it on that. We make the default safe. So to have data
[25:30] minimization and least privilege access. All our tables are private by default and then we give the option to the data owners uh and whoever is accountable to actually review that access and change it. Moreover, uh friction increases with
[25:47] sensitivity and scale. So the the more sensitive the data, the more hard to get access to it. And we design for detection and not prevention. So if we have the policies in house that means we can always use what data bricks gives us
[26:04] in terms of who has access that data and compare it with our internal policies to have detection loops to uh manage or know about unauthorized access uh because we don't want to make it too hard for the users and of course
[26:20] we cannot uh look at every place and we don't want our users to go and create shadow ETLs in elsewhere. which is a bit which is more uh dangerous for visibility and audit. So we try to keep it uh simple and most of the users will
[26:38] fall under unrestricted path and from there we detect unauthorized access and that's it. Thank you everyone for listening.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.