Dealing With Sensitive Data on Databricks at Natura

Summary

• Data governance strategy: This video presents a comprehensive approach to protecting sensitive data in Databricks environments, even from high-privileged roles like Account Console Admins and Workspace Admins.
• Security mechanisms: This video covers encryption, secret scopes, column masking, fine-grained access controls on tables, and auditing capabilities as layered defense strategies.
• Real-world implementation: Natura, a multinational cosmetics company, demonstrates how federated governance and strict access controls protect employee data while maintaining data democratization.

Dealing With Sensitive Data on Databricks at Natura

Talk by Daniel Shimura, Data Architect, Natura
Ensuring the protection of sensitive data within a Databricks environment requires robust mechanisms to prevent unauthorized access, even by high-privileged roles such as Databricks Administrators: Account Console Admins, Workspace Admins, and Unity Catalog Admins. To address this, a comprehensive data governance and access control strategy can be implemented, leveraging encryption, secret scope, column mask, fine-grained access on tables and auditing capabilities.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How can sensitive data be protected even from Databricks administrators?

By implementing layered security including encryption, column masking, fine-grained table access controls, and audit logging. This ensures data protection regardless of administrative privileges in the Databricks environment.

What is the federated model for data governance?

The federated model distributes governance responsibilities across business units acting as data spokes, enabling agility while leveraging central governance standards and reducing administrative burden.

How can we prevent unauthorized access across our data and AI environment?

This video features Natura's approach to handling sensitive data on the Databricks Data and AI platform. Natura implements comprehensive access controls through Unity Catalog with role-based access, row-level and column-level security policies, encryption at rest and in transit, and comprehensive audit logging. These layered security controls prevent unauthorized access while enabling authorized teams to work with sensitive data productively.

How does Natura manage sensitive HR data?

Natura uses Databricks to integrate HR data from systems like Workday and ADP while protecting sensitive employee information through column masking and fine-grained access controls for business intelligence purposes.

What are the three types of Databricks administrators?

Account Console Admins manage users and workspace creation across the account; Workspace Admins manage resources within a workspace; Unity Catalog Admins manage data governance and access policies within the catalog.

Full transcript

Good afternoon everyone. Thank you so much for being here. Uh that's a honor to be here and have you all guys picking my presentation to talk about uh sensitive data at Natura. Um. My folks called me Gold. Um, I'm data architect at Natura. Uh, Natura is a multinational BC Corp cosmetic company. Being well. Well-being is our guiding principle to deliver positive impact on the planet and everyone surround us. In order to democratize data, my leadership, my senior manager, my director and my the teams of data architecture team is structure our working model with the federated happy spoke federated model where we can leverage governance and agility to the business units acting as a I spoke um the HR department was looking for uh easy and automatically way how to integrate their data from their systems such as workday, ADP and CRIX into their business intelligence tools uh so that they could have faster and more actionable business insights uh and also building up the foundations for adoption of AI.
One of the biggest concern uh the HR department had was how to deal with the company's natural sensitive data and they came to the data hub department to address this that issue and that request also includes uh that uh the datab bricks administrators could leverage the governance of the environment. They could organize everything inside datab bricks but they don't own the data they own the environment. So talking about uh datab bricks administrators what are the types of databicks administrators and who are them?
Uh there are three types of databicks administrator. The first one is account console admin and then unit catalog admin and then workspace admin. Um account console is a web- based interface where the administrators manage all the aspects of the account datab bricks account. So I'm highlighting here uh the most important uh features that are related to security. Uh so um the they the account mean manage users groups and services principles.
In our case, we have integrated uh the single sign on solution with AD uh Asia AD uh as identity provider and also we have done the synchronization between the groups from AD to the groups on data bricks. But the account console administrator via API are able to add users on a group even though the user doesn't belong to the uh the group in AD. Uh the administrator also can create workspaces and for each workspace it can flag who is going to be the administrators for that workspace and also uh they can enable features like uh enable met stores unit catalog met store and pick one administrator. the is is either a single user or a single group.
Moving on for um um the unit catalog. Unit catalog is a datab bricks management solution uh uh that uh centralizes um access control um data lineage data discoverability and audit as well. On this picture we have the structure of all the objects that the administrator the unit catalog UC administrator can grant or revoke permissions. Um the workspace is a web- based uh interface where the users work in a collaborative way with data building their pipelines uh with their notebooks and jobs and so on using uh SQL warehouses to export data and I'm super highlighting here the secret score which is the vault for the datab bricks I'm highlighting it because the secret scope where we going to store the keys to an encrypted data at runtime during the query execution.
So the workspace administrator can grant access control list for a group. In this case, we're granting permissions for the people analytics group to have permissions to retrieve the key from the secret scope. So the datab bricks uh intelligent data platform provides out of the box uh three set of things that help us to address the security and governance uh which are the hierarchy permission on the met store dynamic views and column masking.
In this example, I've took a snapshot of the UI that all the permissions that we can grant or revoke for a principle. The principle can be uh either user uh service principle or group. And this is the permissions that we we can do on the table which is the bottom level of the the hierarchic that we have seen before. This a a snapshot from the UI for the catal the schema and all the permissions that we can grant or revoke for schema.
Note that some permissions are exclusively to the schema like use schema. Other permissions are related to the bottom level of object of the hierarchy like see select and then the last one on the hierarchic is the catalog. If you grant permission to select on the catalog all the use all the tables that the principal will have permission will be able to query that. Um and then moving on um uh the dynamic views is a way to control logic to display the data.
You can hide columns or create columns or uh create a logic to show the data for uh for a certain group of users. leverage the built-in functions that are written in Iscala that is super fast doesn't do serialization the serialization issues with Python UDFs and then we can use is member of a a group or is account group member of auditor on this example if the user the current user is a member of the auditor group it will read the emails on the column emails otherwise it's going to see the text redacted.
Um so the issue of this approach is that we you're going to have to manage two sets of objects permission the table itself and the dynamic view. So datab bricks enhance this idea and use this ideas to use udf functions to apply for output of of the query. Uh so in here it's creating a uh SQL udf which is a SSN mask that receives a string as a input and then checks whether is uh the current user is a human resource department belongs to the group user resource department.
If the case is true, it's going to retrieve the social security number SSN. Otherwise, it's going to be star star. There are uh basically you change the data definition for the table. There are two ways to do that. Either upon the creation of the table or with the example here that we just create a table and says uh SSN is string mask and then the function that we have just created. And the other way we can uh alter the table and set the masking function.
So building all this the solution and rep and get everything that we have the sensitive data is stored at rest on SC3 uh is going to be encrypted field by field. Uh we're going to leverage access control list for the secret scope where we store the keys to encrypt the data at runtime. only authorized people will have will be able to query the tables that has has sensitive data and as a extra layer of protection as I mentioned before uh u unal account console administrator are able to add a person or a user in a group in a diverted way.
So we are creating a white list table that will have the authorized users and then we query this authorized users and check with the UDF function uh current user if the current user belongs on the table. We create a UDF that if it creates it turns true otherwise it turns false. So it has to double check the logic uh that belongs to a group and also belongs to the uh white list table and then we leverage column masking functions uh that datab bricks has provided out of the box for us.
And then just to have a better picture uh this is the white list uh table that I just mentioned where we going to insert all the users and we're going to grant permissions for only few people that can insert delete or update users on the white list table and also uh each each table each sensitive data table that has sensitive data will have the column masking functions. Uh the the only authorized people will have branding permissions to select on this table.
Uh we know that this mechanism can be broken but thanks for the audit system that datab bricks provides we can uh check if this the mechanism that we have built to protect the sensitive data has been broken. uh either creating alerts on on several actions either if a UDF function has been changed or if someone has granted permission in a diverted way or uh if someone tries to retrieve the key uh from secret scope that shouldn't retrieve and basically there are several things that we can do as alert to make sure the mechanism is saved and finally this is our uh Natura's overview architecture.
I'm illustrating how work day uh out of the box saves the data export the data to S3 and the data will be encrypted with PGP algorithm which is a asymmetric algorithm so that we have the public key in our day and the private key to encrypt the data will be stored on secret manager and as soon as the file lands on S3 It triggers a lambda and that lambida at runtime restore the key from the secret manager and encrypted the data and encrypt it field by field retrieving other key from another secret manage manager key and encrypted build by field at rest on another bucket.
And then we have our regular way to do the our medallion architecture bronze, silver and gold to build the HR products and that's it. I'm super thankful for you guys to be here. That's my very presentation. I'm super happy to share our natur strategy how we are dealing with security security for sensitive data and if you like you can talk any further questions of this stage. Thank you so much.

Learn more about the Databricks Data and AI platform.