Securing the Databricks Lakehouse: Multi-Layered Defense With Zero Trust and Governance

Summary

Natura implemented a zero-trust security architecture for its Databricks lakehouse that defends against both external threats—unauthorized access, credential theft, and DDoS attacks—and internal threats such as accidental data leaks, malicious exfiltration, and over-privileged access to PII.
Identity governance uses Azure Active Directory single sign-on with SAML 2.0 and MFA, synchronized to Databricks via SCIM, so that revoking a user from Azure AD immediately removes their access across all Databricks workspaces.
Data layer governance through Unity Catalog uses hierarchical permissions, dynamic views, column and row masking, attribute-based access control with policy tags, AI-driven PII classification, and Delta Sharing with automatic masking for governed partner data access.

Securing the Databricks Lakehouse: Multi-Layered Defense With Zero Trust and Governance

Organizations transitioning mission-critical workloads to Databricks cannot rely on the traditional castle-and-moat security model. Instead, they need comprehensive defense-in-depth that protects against both external adversaries and internal risks, from credential theft and DDoS attacks to accidental data leaks and malicious exfiltration.
Daniel Goldszmit from Natura walks through a zero-trust architecture spanning identity, network, and data layers. Learn how to implement corporate SSO with SAML 2.0 and MFA, synchronize identity groups across Azure AD using SCIM, enforce network boundaries with IP allow lists and VPC endpoints, separate workspace topologies for different firewall rules, and govern data access through Unity Catalog's hierarchical permissions, dynamic views, attribute-based access control (ABAC), and AI-driven PII classification. Includes Delta Sharing strategies for governed partner data with automatic masking and row filtering.
🤝

Chapters

FAQs

What is a zero-trust security model and why does a Databricks lakehouse require it?

A zero-trust model replaces the traditional castle-and-moat assumption that everything inside the network perimeter is trusted, instead requiring continuous verification of identity and authorization for every access request. This video explains that Databricks deployments face both external threats—such as credential theft and DDoS attacks on the control plane—and internal threats like accidental data leaks and over-privileged access to PII, making zero-trust the necessary security posture.

How does Natura implement identity management for Databricks access?

Natura uses Azure Active Directory as its corporate identity provider, implementing single sign-on with SAML 2.0 and multi-factor authentication so users log into Databricks with their corporate credentials. SCIM is configured to synchronize Azure AD groups into Databricks automatically, meaning that removing a user from Azure AD immediately revokes their access to all Databricks workspaces.

How does Unity Catalog enforce fine-grained data access control at Natura?

Unity Catalog's hierarchical permission model allows Natura to grant access at the catalog, schema, or table level, with dynamic views, column masking, and row filtering providing additional granularity for sensitive data. This video also describes attribute-based access control using policy tags, where access rules are evaluated at query time based on both the data's classification tags and the requesting user's attributes.

What is AI-driven data classification and how does Natura use it in Databricks?

AI-driven data classification automatically scans data assets and applies classification tags—such as PII labels—based on the content of the data rather than requiring manual tagging by data owners. This video explains that Natura uses this capability within Databricks to keep sensitive data classifications current as new tables and columns are added, feeding those tags into the attribute-based access control policy engine that governs who can access sensitive information.

Full transcript

[00:08] My name is Daniel Goldsmith. My folks call me Gold. I'm data architect at Natura. Natura is a a B Corp multinational cosmetic company. Being well, well-being, is our guiding principle to deliver positive impact to
[00:25] the planet and everyone around us. Today, I'm going to share with you guys uh our journey securing Databricks Lakehouse a multi-layer defense against outside and inside threats.
[00:41] If you think about a Databricks platform, you have to think beyond the mentality of castle and moat to protect the Databricks. Of course, you have to protect how people get in your Databricks.
[00:56] Uh so, from the outside perspective threats, we have unauthorized access, credential theft, DDoS attacks against the control plane. And also, the other side that we have to
[01:13] to think about as well is the inside threats, accidental data leaks, malicious data exfiltration, or even users with over privileged upon sensitive data like PII.
[01:28] So, if you think about the castle again, the first thing that you have to protect is how people log in. And the best way to do that is to leverage the corporate identity provider. In our case, we have uh Azure AD.
[01:45] Or actually is Azure Active Directory, formerly known as Azure AD. Uh we have implemented a single sign-on using the same user to log in on AD, Azure AD, and the same passwords. Uh Uh and we have implemented the SAML
[02:02] 2.0 protocol. Databricks also provides uh OpenID uh connectors for other identity providers. And we also leverage the the benefit of multi-factor
[02:19] authentication against uh when people log in on Databricks. And one thing that Databricks has uh innovated is you set up the unified logins, making sure that all the workspaces across the account console
[02:35] will have the same way to to do the login. And the benefits of using the centralized way to log in is that if if you remove the user from the uh Azure AD, uh the user is no longer able to connect on
[02:51] Databricks. And and oops. And then, one one other one other thing is that uh we we use um uh
[03:06] groups to to govern how we grant permissions inside the Databricks. And then, we leverage the the uh the the the corporate
[03:23] uh authentication for and and implemented the the SCIM system cross-identity management using the same identity provider, the Azure AD, that does the synchronization with the groups.
[03:38] And once the group uh is synchronized, it will be appear on the account console. And on this picture, we can visualize uh a better
[03:54] a better way that we have the identity provider there and the identity provider does the provisioning for the groups, which means uh we assign a group for Azure enterprise application and the
[04:09] Azure enterprise application runs every 40 minutes and create the group if it doesn't exist in the account console or it makes sure that all the members that belongs to the account to the Azure AD group will be will have
[04:26] on the account console as well. And once the group it is on the account console, we can add those groups in any other workspace that we have. Now that we have already talked about the identity management, let's see what
[04:43] we can do for the harder thing, the network layer. Imagine that you can guarantee that people are allowed to Databricks only if they are in the company infrastructure, either a headquarter or any other branch or they
[05:01] are in the inside a VPN, virtual private network. So it means if they are connecting from any other resources like a cyber cafe, the connection will be blocked. And how
[05:16] it works, the company has Oh, sorry. Sorry. Sorry. Yeah, the company has a contract internet service provider and that internet service provider is going to provide the company a public IP and
[05:34] those public IPs we we collect all the list of the public IPs from the company and via a REST API, we we create or uh update uh IP allow list. So, it means no one can connect on Databricks if they
[05:52] are not uh previously allowed and making sure that they are in the company infrastructure. And then, in this way, if some For example, a personal access token has been leaked uh and they are not connecting on the
[06:09] uh the company infrastructure, that personal access token is not able to to be used because the connection is will be blocked. And how we don't or have done that in in the in Natura, we created a regular
[06:25] table inside the Unity Catalog, and we register all the CIDRs uh IP IP list there, and we query the table and update uh with one single call the API. And then,
[06:40] we update and then sometimes if you were creating that, we create a new IP allow list as well. Um Um and moving forward, how we can protect and make sure that authentic user
[06:56] can connect on Databricks, but uh they can access the data, but they can't send the data in an adverted way outside the company. Uh and this is called data exfiltration. And in order to avoid data exfiltration,
[07:12] you we can apply egress filtering. And we also can configure on the uh the cloud storage protection uh uh the S3 bucket. And in this picture, better illustrated what I'm talking about. Um
[07:28] S3 uh we created a VPC endpoint, which is a private link, and that the private link, we make sure that all the route that uh for the Unity Catalog bucket that you use it uh
[07:43] has to go via the the AWS backbone. If there are any connections that is not from the VPC endpoint, the connection is blocked based on the a policy that is attached on the bucket.
[08:00] And now uh as I was talking about the firewall, there are two types of firewall. Uh data we can spin up a serverless compute, and if you have a serverless compute Databricks provides us out of the box a
[08:17] uh egress uh secure gateway where we can configure any URL that's are allowed or blocked. And if you are running on uh all purpose cluster or or uh job cluster, which is a classic compute, uh
[08:34] that connections goes via uh our our firewall. We have a Fortinet there, and then we can either block or allow. And this is the figures uh six and five that we can see there.
[08:49] And then we have the this is topology of our workspaces. We have separated workspaces for ingestion and workspaces for transformation. And then this way we can apply different rules on the firewall.
[09:05] Uh it means some workspaces are allowed to do the ingestion uh to to reach some URL, but other workspaces are not allowed. And then let's talk about Unity Catalog.
[09:20] Uh what is Unity Catalog? It's uh uh a a govern platform for data and asset uh for for permi- for for
[09:35] that provides secure access control uh for tables and and any other resources. And it we are focused here on the the features of Unity Catalog. I have to mention here that it provides lineage
[09:52] and audit the out of the box, but in here we are more focused on the hierarchical model, how we grant permissions. And if someone has permission to query a table, it doesn't mean that it has permission to see sensitive data or to
[10:10] see all the table is there. And then Databricks came in with some solutions to do column masking and row filtering. So, in this picture we have a better visualization uh how is the hierarchical object
[10:25] models, where catalog is is own set of permissions and is hierarchical. We can see that select, if you grant the select on the catalog, it's going to expand for the permissions for the select all the tables beneath that catalog. And each
[10:42] object has its own sets of granting permissions or revoke permissions. And then uh dynamic view is the first solution that the Databricks has provided to us uh to do the column masking and row
[10:58] filtering. And this is a view, and in this view we have this the whole solution there. And if you are a CC admin, you're going to be able to see everything, but if you are US uh regional analyst, you're going to be able to see only US data and that data
[11:16] will be masked. The downside of this strategy is have is that you have to granted permissions uh or manage permissions for the tables and the views as well. And then Databricks came with the the idea to what if we change the definition
[11:33] of the table and apply a function either for a row or apply a function for the rows of results of a query on the table. And then the changing the definition we can create a function and do the alter table.
[11:49] And the same is if you are HR manager on this case, you're going to be able to see the whole data unmasked. Otherwise, any other person that doesn't belong to that group leverage the the the built-in functions is member.
[12:06] We can do this logic. And then uh the same strategy we can do the row filtering for applying changing the definition of the table. And the downside of this strategy is that we
[12:22] have to change a lot of tables. And then Databricks thought how we can work with attribute builds, which basically is tags. And then we create a back solution, attribute-based access
[12:38] control, where we we created a policy. That policy can be applied on the catalog, schema, and tables. And basically, what the policy has is the groups that is the target for the dynamic
[12:53] evaluate the function. The groups that there are the exceptions. And the tags that can be attached on the on a table or a schema or even a column if you are doing a column masking.
[13:08] And the function itself. So, in this example here, we have a syntax for creating a function and the the policy itself. Databricks also provides out of the box the UI way to
[13:24] create those functions and you don't we don't have to memorize the syntax there. In this example, uh the table has to have the tag HR and any column that have the tag PII is going to take a place.
[13:40] The uh the the the masking the column masking. And the same strategy, we can do the the row filtering with where we create a a function and a policy and we attach uh on uh the meta store objects.
[13:57] And the the the challenge of this, as you can see, we going to have a govern tags uh the govern tags belongs to a the account console, but we can have several thousands of say less several thousands
[14:13] of tags, especially if you are trying to do uh uh PII protecting data from personal identity identifications. And then Databricks came with the solution that what if we can do the data
[14:28] classification using leveraging AI with a super smart LLMs there. And that feature just came GA a couple of weeks ago. And uh we can either accept or reject those
[14:46] solution those classifications. And once we accepted those classifications, we can auto apply auto tagging and then it's going to do the the the protection in a massively way.
[15:05] And then at Natura, we use quite a lot uh sharing data with our partners. And the best way to do the in a govern way uh with auditing is the uh the protocol Delta Sharing, where we create a Delta Share, we decide which assets we going to share either tables, views, and and
[15:23] nowadays we can share notebooks and also models. And decide the target, the recipients that is going to receive the those data. That can be either Databricks or no Databricks. And that solution works really neat with
[15:39] ABAC sharing. The with ABAC sharing, which means ABAC solution, we can decide whether we are field applying row filtering or column masking. In our case, we have several column masking stuff. Uh and quite a few row filtering there.
[15:57] And also, we can decide uh the person that has permission to see the data unmasked and do the sharing is going to propagate it for the Delta Sharing. Uh and it it is working really well. We try
[16:12] to use a column DDL definition. Change the definition of table, but that doesn't fit well with with the protocol Delta Sharing. And then, just to summarize our solution,
[16:29] uh let's say that our key takeaways for identity governance leverage SSO, a scheme and MFA. And for the network layer, uh IP allow lists, firewall for cloud
[16:44] computes, and egress filtering for serverless computes. Uh and Unity Catalog uh to govern the permissions, all the security objects, and also leverage ABAC solutions. Uh
[17:01] Thank you so much for being here. Uh I'll be happy to answer any further questions off the stage. Thank you, guys.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.