Securing Data and AI for the Enterprise: What’s New in Databricks Platform Security

Summary

Automatic Identity Management (AIM) automates identity provisioning and enforces least-privilege at scale, with native integration for Azure AD and Okta and just-in-time group provisioning for both human users and AI agent service principals.
Private Network Gateway enables secure, managed connectivity to data sources, while context-based network policies replace static IP ACLs with adaptive controls for Genie, apps, AI/BI, and agentic workloads.
New compliance milestones include an Azure GovCloud environment, expanded AWS GovCloud capabilities, and certifications for ISMAP, HITRUST, and FedRAMP High, making enterprise-grade protection the default.

Securing Data and AI for the Enterprise: What’s New in Databricks Platform Security

Databricks is shipping a wave of platform security, networking and compliance capabilities that make enterprise-grade protection the default — not an afterthought. In this video, we cover the highlights including Private Network Gateway for managed, secure connectivity to your data sources, context-based ingress policies that bring adaptive network controls to Genie, apps, AI/BI, and agentic workloads and cloudless compute with a cross-cloud backbone that simplifies networking across environments. We’ll also showcase automatic identity management (AIM), which automates identity provisioning and enforces least-privilege at scale. On the compliance side, we’ll announce a new Azure GovCloud environment, expanded AWS GovCloud capabilities and new certifications (ISMAP, HITRUST, FedRAMP High). Plus, secure cross-workspace access, serverless egress logs, account frontend private link, CMK for UC Catalogs and more. Come see how Databricks is making security a platform advantage.
Talk By: Jason Wu, Senior Director, Databricks ; Samrat Ray, Director of Product Management, Databricks ;

Chapters

FAQs

What is Automatic Identity Management (AIM) in Databricks?

AIM automates the provisioning of user identities and enforces least-privilege access at scale, integrating natively with identity providers like Azure AD and Okta. It provides just-in-time group provisioning and governance controls for AI agent service principals, addressing the identity sprawl that comes with agentic AI adoption.

What is Private Network Gateway in Databricks?

Private Network Gateway is a Databricks feature that provides managed, secure connectivity between workspaces and data sources without requiring teams to configure individual network rules for each connection. It simplifies networking for agentic workloads, Genie, and AI/BI applications across complex enterprise environments.

How does Databricks handle network security for agentic workloads?

Context-based ingress policies bring adaptive network controls to Genie, apps, AI/BI dashboards, and agentic workloads, replacing static IP ACLs with rules that adapt based on user, device, and workload context. This reduces the attack surface without blocking legitimate access from business users who need to reach data products.

What compliance certifications has Databricks recently achieved?

Databricks has announced a new Azure GovCloud environment, expanded AWS GovCloud capabilities, and new certifications including ISMAP, HITRUST, and FedRAMP High. These additions extend enterprise-grade compliance coverage for regulated industries such as government, healthcare, and financial services.

Full transcript

[00:08] Thank you. Uh hope folks at the back can hear me clearly. Uh So today uh we'll be talking about securing data and AI uh for the enterprise. And uh So uh
[00:21] it's a perfunctory forward-looking statement that whatever we're talking about today like the forward-looking statements are not a commitment to deliver something.
[00:33] Uh So I'm Samrat Ray um and with me is Jason Boo and uh together we'll be covering about talking to you about uh
[00:45] what's changing with platform security. And before I jump into the actual uh topic, uh sometimes we use the word platform, so we'll just define what this actually means. The word platform here means uh platform includes everything from
[00:58] uh your networking to encryption, so your data protection uh to um authentication, uh identity uh including um agentic identity and stuff. Uh it
[01:10] also includes things like logging etc., right? And compliance. Right, so it covers a uh vast set of surface area that if you are a security conscious customer or if you're a platform admin, this is
[01:22] definitely something that uh is relevant to you. With that uh so let's kick off with something that's kind of obvious and I'm sure many of you uh been experiencing this in your organization,
[01:36] which is that um agentic AI is driving data democratization. And what do we mean? It sounds like a fancy buzzword, uh data democratization. So what do we mean by that actually? Uh what it means is that uh if you look at it, if you
[01:50] look at uh a platform like Databricks and who was using it directly, it was a select set of users that were uh technically proficient, uh data engineers, and maybe after that SQL analysts who are kind of using that
[02:02] platform. So, it was a limited set of users that were fairly technical that were gaining access to it from very secured environments. Typically, if you're a security-conscious customers, you had locked it down completely, and then it was just treated
[02:14] like a back office sort of a system that only you can gain access from very secure networks. That's changing. Right? This has already been the case with the in Databricks with things like apps and with AI and BI dashboards. Uh
[02:28] where uh we had already seen customers building data products that were directly consumable by the business user. But, that has been supercharged with um the advent of agents and use of agents
[02:40] within the context of data processing. So, um this is not a Databricks-specific phenomenon. This is a secular movement we see across the board for the uh data platform. And uh what we are seeing now
[02:52] is that in order for you to deliver the true value that you can bring and to have your company be competitive and truly be a data-and-insight-driven company, you need to enable data access and in
[03:04] interactive data access in many cases, not just like a offline doc with a, you know, with a screenshot of a dashboard, uh but actually interactive access to pretty much close to everyone in the company.
[03:16] Right? Um uh because everyone can derive some value. Right? Uh and because it's everyone, you can no longer say that they have to be in some select network all the time. They might be on in the field using a mobile
[03:30] device. They might be on the corporate network versus an IT network, right? If you have those kind of uh segmentation. And further, the other thing that's happening with agents is earlier you had this uh view where all the data was, you know, uh in
[03:44] the lakehouse, and you're mostly accessing data in the lakehouse, and that's what you were building it off. That continues to be the core corpus of your data, but because agents can deal with unstructured systems very well and
[03:56] semi-structured, we are seeing a lot of scenarios where uh taking the data that's in the lakehouse and then joining it with data that's in Slack, that's in your uh SharePoint, that's in Google Drive, and that combination is very
[04:09] powerful and is uh driving a whole new set of use cases on our platform. So, that's great. Now, we're giving you all these cool technologies which you have heard in the keynotes that's enabling these workflows.
[04:22] But, one of the challenges that comes with it is what happens in terms of security? All these things that you've configured before, maybe you have configured a front-end private link, you have configured uh various kinds of controls. Like, how
[04:35] does that actually work with this new world, right? So, clearly, platform and security has to keep up with this new requirement. We need to move from a world where all or nothing security to a
[04:47] world that is actually more granular and context-based. You had heard the word context used in uh our keynotes. It can mean two things. Uh context can mean like the context given to AI models
[05:00] so that uh or so that they can perform well, but in the security world, when we say context, it means the context of the request. Who is accessing from where? And that becomes a key requirement, right? Further, you need to scale. You
[05:12] need to scale across several dimensions. You need to scale across the user dimensions because earlier, as we said, you if you're just serving a select set of users, you now need to be able to govern access across your entire enterprise. It's much more like a SaaS
[05:25] enablement in your enterprise than it is like a data platform enablement, right? Second, uh you need to be able to securely connect to a very large number of back-end endpoints than which you had to do before. Every agent and workflow
[05:37] developer will come and say, "I want to connect to X, Y, and Z." Again, you need to have a scalable way to securely connect. Just because you want to scale doesn't mean security goes out of the door.
[05:49] So, with that, uh let's talk about uh in today's session, we'll actually cover a few things. We'll cover uh how do you scale and streamline your operations while you're while you're scaling to meet all these
[06:01] multi-dimensional uh scaling requirements. Second, how do you mitigate key risks with uh agents running on your platform? Because this is a new thing, right? And how do you mitigate There was a lot of discussions about uh
[06:14] control as um Ali called it, but what does it actually translate to if you are thinking from a platform perspective? What does that mean? Uh and lastly, how do you continue to meet your um availability and compliance
[06:26] requirements, which actually become even more critical um as you enable more business-critical functions to be directly reliant on the platform?
[06:39] So, let's talk about the first problem, which is how do you scale to uh efficiently scale to 100K+ users and agents? So, one of the key features that we've launched recently is a feature called automatic identity management.
[06:53] So, automatic identity management, the goal of it is to allow you to scale to a large number of users while streamlining your operations at the same time. So, how do we achieve that? First of
[07:05] all, instead of your old approach, which was like to have manual scripting or some way of doing SCIM pushes, working with the your IDP admins, uh we now instead of that, we now provide a
[07:18] native integration to at least two IDPs right now. We may do more in the future, which is to Azure AD and to Okta. Right? With these native integrations, you do not need any form of further
[07:30] scripting and script-based push mechanisms. The entire directory is now natively integrated into the platform through graph APIs and is available. So, what
[07:42] does that mean actually? The first thing that it means is when a user signs in, if they're in the directory, right? They can sign on. Doesn't mean that when they sign on, they have access to any data. It doesn't change your access control
[07:56] parts. They're still your the UC policies, workspace entitlements, all of that still apply, right? But it means that you don't have to go just because you now have created a product for your business users, you don't have to go and make sure that
[08:09] your identity admin has provisioned all the business users correctly and stuff like that. This is done uh seamlessly for you. Right? And the other thing is we don't aggressively just replicate your entire directory
[08:21] because that could be a pain and may not be the best security practice. So, we do just-in-time. So, when somebody signs in for the first time, you share an asset with the business user and they and you have given them actual access through
[08:33] either UC or workspace entitlements, then when they sign in for the first time, we automatically get all their group memberships. We automatically get all their all the associated including nested groups and are able to that therefore
[08:47] determine what permissions they have on the platform real time. The second thing is that we now also beyond nested groups which I just talked about, we also now have included support for service principles.
[09:00] So, a lot of times now we are hearing more and more with the agentic world that you want to also see service principles be rooted in your directory and not just be local. So, you might have that requirement and we are now going we
[09:13] now will support that seamlessly as well. And lastly, as we said, this is engineered for scale and so it is it can scale a lot to a lot higher numbers of identities than the older SCIM mechanism.
[09:26] So, that's AIM and but there are few security considerations, right? Like whenever we create something like this, there's always folks are thinking, "Okay, what about the guardrails that I had in place? What about the sort of my compliance requirements I had to meet?
[09:39] So, I just want to um provide a few of the security considerations that we have heard from our customers and shared like how we are thinking about it. So, the first is there are certain maybe some users or
[09:51] groups that you don't want to show up at all on the platform. So, for that we provide a hard guardrail called account access deny list. So, if you add a group or users to it, they cannot gain access to the platform even if they're
[10:04] part of another group that actually is mapped to the platform. So, this is a pretty hard guardrail, right? Um the second thing which I mentioned briefly before is just because somebody is be able to sign on with AIM
[10:17] doesn't mean they have access to any asset unless it's been um appropriately shared or given granted access to by the right authority on the platform. The third thing uh is we now have a new
[10:29] role in the workspace called consumer access um it's uh entitlement. Um so, it's a new kind of a role in the workspace. If you give that access, the the user it's typically meant for the consumers of the
[10:41] business users. So, they can consume products that your team the data team creates, but they do not have the permission to create new notebooks or new data products themselves, right? Uh so, that's uh fairly important for you
[10:54] to keep the sort of risk down is to make sure that most of your business users are in the consumer access entitlement and not our full-fledged workspace users.
[11:09] So, to summarize the key benefits of AIM as we like to call it in short is one it streamlines your identity management workflows. No need to deal with manual scripts, reconciliation, etc. for nested groups, service principles, all of that. It minimizes permission drift. So, let's say you move
[11:22] a user out of a group. The next time they log in, it immediately reconciles and the permissions and if they the previous group had some access to some assets and the new group doesn't, they lose access right away. So, unlike SCIM where you had to make sure that you had
[11:34] a SCIM push before that could take effect, you don't have to worry about that. It happens automatically the next time they log in. Right? Uh so, the other thing is it's optimized as I said for business user collaboration.
[11:46] Uh you can now share a dashboard with somebody who has never signed into Databricks before, doesn't even know what Databricks is. They just know that you have a data product, you're sharing it with them and they want to access it. They can just sign in with the standard
[11:58] single sign-on and be able to access that. So, we talked about identity and scaling identity. Now, we talk a bit about access. Access has two parts. We think about
[12:10] access to data, which is Unity Catalog policies or um you know, like uh the IAM side of it. But, uh it's not We have to think of it more from a zero-trust perspective. Now, it's
[12:22] no longer sufficient just to think of networking and IAM as two separate layers. That thing just doesn't work in this new world, right? Because you now have different personas. Like, you have a data engineer who might be having highly privileged access into your
[12:34] workspaces and into your data, and they log in they might have to log in only from a highly secured uh production network, and that is the only place from where they can log in. But, you might also have a ton of business users, some of whom might be accessing over the
[12:47] internet, maybe over a mobile gateway using um a mobile device. Others maybe just using an agentic interface that you have created on the corporate network, and then that agent is then talking to Databricks. Right? Uh
[12:59] And lastly, we're seeing a lot of partner integrations happening as well. A lot of you are creating the what was being shared through Delta Sharing. We're now seeing a new world where people are sharing dashboards with partners instead of Delta Sharing. So, it's more of a more finished product
[13:12] that is being shared across your value chain. Uh in which case again, you need to give them access not just to the data, but you're actually giving them access to some asset on in the workspace. So, how do you manage this environment?
[13:26] To ma- you manage this, we've come up with a new policy. This is part of your object called network policy. So, those who have configured serverless egress control would be familiar with this. This is the policy you use to configure egress control.
[13:38] We now have added an ingress section in there. And that is uh it is replacing effectively your IP ACLs or your just using front end private link or just using an IP vanilla IP ACL. It allows
[13:51] you to now define more granular rules. So, you can define rules like based upon not just a network source, oh this is the where they must come from, but this is the identity. So, for example, for partner onboarding, you might say this
[14:03] is the service principal my partner is going to use and this is their IP range. Here's a second partner, that's a second service principal, and that's a different IP range. So, now you're able to actually bind these identities to specific IP ranges, right? When you're
[14:15] giving access outside. Secondly, you can reduce the scope that they actually have access to. So, for example, if you only wanted to share dashboards, you could decide I only want to give access to embedded dashboards. I'll point out the scope reduction is
[14:28] primarily on the API surface area, not on the UI. So, the UI is still If you have access to the console, you have access to the console, but a lot of access is happening programmatically, so that's where you can reduce the access.
[14:40] So, I'll give you a couple of quick examples on this. Uh the first one is uh um like you want to give access from a SAS tool. This is a very common thing. Many of you use Power BI or you're using Tableau. Uh this was actually, even in
[14:52] the current IP ACLs, was a really hard thing to do. You had to pick huge CIDRs, put it into all your workspaces, and then uh every time there was a new IP added by the provider, which happens from time to time, you had to go and keep that in mind and keep refreshing
[15:05] it. We have really simplified that for you over here. One, you don't have to remember the CIDR. You can just say I want to give access from Microsoft Power BI or from uh you know, Tableau. And you can say, "Oh, I want to only give access
[15:17] to the SQL gateway." Secondly, there's no good reason why service principals or my long-lived path should ever be used from that network. Maybe that is a sign of breach. I only want to give access to human users. So, you've suddenly reduced risk,
[15:29] simplified your policy. Also, you define it once and you can attach it to multiple workspaces now at the account level. Second, I only want to open up access to apps. This is a very common pattern we're
[15:41] seeing with this agent Akira, right? Where people are building agents, but instead of directly giving access to agents, they want to do some pre-filtering, they want to do some their own interface a little bit instead of just the default interface that
[15:53] Databricks offers, right? So, then you create an app that is actually the front end, and then that calls Genie, and you can some Genie space, but you can still expose the the chat interface through your own app console, which is just a
[16:05] pass-through to the Genie API. And you can have your AI gateway sit between your app and Genie, for example. Right? But in this case, what might happen is you might want to expose that app to all
[16:17] your field. So, it might be accessible over the internet, right? But that doesn't mean you want to expose your entire workspace to the internet, and that is now possible. Right? Because you can just scope it to
[16:29] the app over here, what you're gaining access to. Another example, the reverse example, is you've got some highly privileged service principles which have a lot of admin permissions, etc. You want to make sure they're never are used from the internet even as you're opening
[16:41] up the workspace. So, you can put a deny, we support a concept of a deny, which is like a hard guardrail again, that prevents the that service principle from being used from anything but the network. So, the not in clause says this is the network
[16:54] from where they can access it. Anywhere else, they are not allowed. Right? Um So, we talked about context-based ingress, but I just wanted to call out that that was just one part of a broader
[17:07] solution. You have heard the term context use uh in multiple places, and it is part of a broader solution of contextual access control that we're offering on the platform. So, you can start all the way from the client if you are using
[17:20] Omnigen, for example, that has contextual policy that is on the agent itself. So, it's on the client to the workspace access. That's what we just covered, context-based ingress. Then, going to the next step, you can do data. Actually, this is based on context now.
[17:32] So, that is a new like we just had a uh session on Unity Catalog, ABAC policies. We are extending that to have context attributes. And lastly, uh you can now then take the same UC ABAC policies, but use it in the context of your AI
[17:45] gateway, the Unity AI gateway, to then govern access to tools and uh models themselves, right? So, that is end-to-end context-based access control on the platform. And here's an example of
[17:57] uh where uh we're taking the context and actually applying it to data. Right? Uh here's a new UC ABAC policy where you are saying the the key thing that we're saying is if it's happening on behalf of those who are familiar with
[18:09] on behalf of authentication. So, when you use an agent, you want the agent to use your identity to gain access, not their own identity. But, you also want to know that it's actually the agent and not you that's directly accessing.
[18:21] That's called on behalf of. So, that's the authentication mechanism where we understand there's an app ID which is the agent, and there is a user ID that's you that's accessing. And then, you can say, "How do I want to downscope access?" So, you can downscope access to the platform and to data. In this
[18:34] particular example, we're showing that if we are accessing via an agent, we want to reduce PII exposure, so you mask the PII. But, you could also deny access to certain operations, uh to certain data sets, etc. from uh when accessed
[18:47] via an agent. With that, I'll hand it over to uh Jason to talk about uh the other uh dimensions, including uh how do we connect to back and such scale. Thank you.
[19:00] >> All right. Thank you, Samer. Turn it off. >> So, we have a few more exciting products and uh uh capabilities

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.