Skip to main content

Unlocking Access: Simplifying Identity Management at Scale With Databricks

Summary

  • Identity complexity: As enterprises scale across clouds and teams, IAM becomes fragmented—Databricks simplifies identity provisioning via IDP integration.
  • Automatic management: Automatic Identity Management (AIM) eliminates manual SCIM provisioning for Azure Entra ID, reducing administrative overhead.
  • Unified governance: IDP integration, externally managed groups, and PIM support enable secure, scalable identity management without IT bottlenecks.

Unlocking Access: Simplifying Identity Management at Scale With Databricks

Watch: Unlocking Access: Simplifying Identity Management at Scale With Databricks
Talk by Hari Selvarajan, Specialist Solutions Architect, Databricks ; Keegan Dubbs, Staff Product Manager, Databricks
Effective Identity and Access Management (IAM) is essential for securing enterprise environments while enabling innovation and collaboration. As companies scale, ensuring users have the right access without adding administrative overhead is critical. In this video, we’ll explore how Databricks is simplifying identity management by integrating with customers’ Identity Providers (IDPs). Learn about Automatic Identity Management in Azure Databricks, which eliminates SCIM for Entra ID users and ensures scalable identity provisioning for other IDPs. We'll also cover externally managed groups, PIM integration and upcoming enhancements like a bring-your-own-IDP model for Google Cloud. Through a customer success story and live demo, see how Databricks is making IAM more scalable, secure and user-friendly.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What is Automatic Identity Management (AIM)?

AIM eliminates manual SCIM configuration by automatically synchronizing users from Azure Entra ID to Databricks. This reduces setup time and maintenance overhead while ensuring secure identity provisioning.

How does Databricks integrate with Identity Providers?

Databricks connects directly to customer IDPs (Azure Entra, Okta, etc.), enabling automatic user provisioning, group management, and compliance with centralized identity governance.

What is Privileged Identity Management (PIM) and why is it important?

PIM controls and audits access to sensitive resources by requiring just-in-time elevation with approval workflows. This reduces security risks from standing admin access.

Who are the speakers on identity management?

Keegan Dubbs (Databricks Staff Product Manager) and Hari Selvarajan (Specialist Solutions Architect) discuss scalable identity governance for enterprises.

Full transcript

0:00 Introduction and challenges

First of all, thank you everyone for coming. We know this is is the last session of the day and that we are standing between you and the welcome party or happy hour if you so choose. Um so let's get into it. All right, you've probably have seen this numerous times today. This is just the forward-looking statement that our lawyers make us show. We will be talking about some things that are roadmap items here today. So, please don't hold our feet to the fire on any exact deadlines that we communicate.
And I say that especially speaking as a product manager. So, let's quickly introduce ourselves. I'm part of the product team here at Data Bricks. I own the identity life cycle parts of the product. So that effectively has to do with how do we get identities into the system, make sure they can authenticate securely and then that they have the right level of access when they're in data bricks. And with me today, yeah, hi everyone.
I work as a specialist solutions architect uh mainly focusing on platform and security. I'm based out of UK looking after customers in the UK and Ireland market. Awesome. All right. So just to kick things off and kind of lay the scene, we wanted to um first start with the challenges that we know we all identify with in identity management. So first of all, the identity landscape is complex and um it's growing and it's distributed and it's diverse and we know for you all as administrators, it's often the case that as the organization grows, you're not just responsible potentially for more users, but you may have more tools, additional use cases.
Hey, you may be even juggling data bricks ac across multiple different clouds. And all of this can be incredibly challenging because as the business scales, you want innovation to keep pace. But if administration administrative piece can't keep pace, then that can cause friction and of course is uh is a problem for securing access. And the other thing that we know is oftentimes there's not just one centralized IT team or one team that's maybe managing the IDP is the exact same as the team that's administering or managing data bricks.
And so that can cause a lot of challenges as you're trying to get the identities into the system to ultimately do the governance within data bricks. So what's at stake here? This is a very real statistic which on average for most of the customers that are using data bricks or most of our customers there's less than 5% of their organization that's actually enabled and provisioned to be able to use data bricks and this isn't necessarily because uh you as the administrators don't want those users to access data bricks and it's certainly not because the users don't want to access data bricks.
It actually has a lot more to do with just the friction around the identity administration of getting the right users and groups into data bricks to make sure that those are the trusted ones that you want to have access. And we know this is true because um you know one of my favorite parts of my job as a product manager is I get to work with and listen to all of you as customers. And on the identity part of the product team at data bicks, there's actually only two of us product managers and we're both in the room right now.
And I can say that between the two of us, we have more than 200 conversations annually with customers like yourselves. And a lot of the key themes that we hear are really consistent. One of them tends to be this kind of friction point between the team that controls uh let's say like security and identity maybe isn't the same team that's handling the data governance pieces within within data bricks and so there can be a little bit of a disconnect or a friction there.
We've also heard time and time again that it's really frustrating that we don't have the amount of admin granularity that you want within data bricks. And then the last piece is that we know it can be painful and manual to provision tens of thousands or hundreds of thousands of users at a time. So what are we doing about it? Well, we have made it our mission and it actually is our mission statement as a team as an identity team at data bicks to make data bricks more simple.
And the way we want to do that is through three key initiatives. One, we want to reduce the admin effort and overhead. And the way that we're doing that is by investing in features that make it easier for you to autoprovision identities into your account. We also are investing heavily in the platform so that we can scale securely with you as the business grows. And then finally, we know that uh for the vast majority of the rest of the data bricks team, they're often thinking about things across clouds, but in the identity world, it's actually matrixed because we're not just talking about clouds, we're also talking about identity providers.
And so, we need solutions that work across all clouds and all IDPS. So, that gets us to what we'll be covering today. We have three key things that we want to dig in on. The first has been my metaphorical baby which is automatic identity management and this is our native integration against entra ID. Today it's available in public preview on Azure uh but we will be bringing that across all three clouds. Then Hari's going to run us through privileged identity management.
Um we'll d go deep specifically into the entra ID solution but this can be leveraged for any of the other IDPS that have similar type PIM integrations. And then finally we'll conclude with what's available today for you all as the uh feature set for administration to get you more of that like simplified identity admin experience in data bricks. Um so before we get into it though I am curious how many of you are Azure customers?
Okay, great. You are gonna love this session. Okay. Um, also I will tell you I'm doing a live demo and I may have been logged out of my my Azure portal. So we'll just bear with me as we go through this. All right, let's double click into automatic identity management or as we often call now AIM. All right, a little bit of background around why did we build this feature. So uh when I started at data bicks, there were three really common pain points we were hearing from our Azure customers.

6:11 Automatic identity management

The first is that uh it was very difficult and manual and you needed to have custom scripting or you know cross your fingers and pray to be able to sync in nested groups or groups containing SPS uh which both of those are actually things more unique to Entra as an identity provider. The second was that as we started to expand in use cases, customers would be creating dashboards and things and they'd want to share them out to other users or groups and unless those users or groups were provisioned, they couldn't share to their team members.
And that was really frustrating. And then the third was that a lot of times you have other Microsoft or Azure tools that you're using and you're used to them operating in a certain firstparty way. And data bicks is actually firstparty in the Azure portal. So we got questions around why can't you leverage that? Why don't you operate just the same way that PowerBI does or the way that Synapse does? Why do I have to deal with SKIM?
So that is exactly why we introduced automatic identity management which is the feature that we've been working on now for the last several several months. And there's really two key use cases here that we're trying to introduce. The kind of topline narrative is that we want to remove the need to pre-provision users or groups and pre-provision them all using account skim. Instead, we want much more of that seamless or automatic experience the same way that you would get with other first-party tools.
So, the first use case is really about the administrative experience. So for workspace admins, being able to have the more distributed model that you can bring in identities as needed into your workspace without involving the account admin. And then the second is more of that frictionless sharing experience. You create a new dashboard or you create a genie room, you want to share it out with the rest of your organization.
Great. I can do that without needing to involve the account admin or deal with potentially writing a request to have a new group created and then that be pushed through skim. And maybe it's at best case a 40-minute process. At worst case, I've heard from customers it can be a couple days. So what does this actually look like in practice? This is my very rudimentary architecture diagram. So the engineers in the room, please don't come for me.
Uh but what you're looking at here on the screen is that the upper section where it says skim sync, this is the old world. So in the old world where you would need to skim provision identities, you effectively would be sending copies of the identity from Entra ID to datab bricks and we'd be storing them in our local identity store or identity directory. Then let's say if you want to run a query in data bricks, you're in DBSQL, we are actually checking against that local identity directory to ensure that you have the right level of access.
Um, with the major shift with moving to automatic is that we're no longer only using the local identity store as the identity directory source of truth. We're now natively using Entra ID also as the identity directory. And so the way we're doing this under the hood is through a series of graph API calls out to enter ID to be able to read those identities and then on demand as users are coming in or SPS uh and accessing data bricks we're autoprovisioning them and we're also making checks to see hey is a member still of AM team group in Entra ID.
So let's walk through an example because that should make it a little bit easier to understand. So the uh let's go through what it looks like for workspace provisioning. So imagine I am a workspace admin and I want to add group FU to my workspace. So what that will look like is in the UI I can now go and search for group FU and if that group exists as an account group in datab bricks or if any any group with the name fu exists in entra ID then I'm actually able to see those identities in data bricks and I can select the foo group from entra ID then when I actually provision or add group foo to my workspace two things are happening under the hood first we're creating and provisioning ing the shell of group fu to the account and we're also granting and giving it access to the workspace.
I want to be really clear that the big difference here between aim and skim is we are not provisioning all of the all of the members of the group at that point in time. So it's really just the shell of the group and as users sign in the group would fill up. Uh and that's exactly what step number three is. So let's say user one is a member of group fu in entra ID when they sign in and authenticate via SSO they are just in time provisioned or autoprovisioned into the account.
We also check what groups is user one a member of in entra ID and if they are a member of group fu we place them in group fu in data bricks and when user one signs in they automatically get access to workspace one. Then while the user is in an active session, we will are making regular checks back to enter ID to see if the me group membership or anything else changed. So let's say it was the insane use case that you were fired while being in the middle of an active session.
Uh that would be reflected in real time and you would lose access to data bricks. But hopefully that doesn't happen to any of you. All right. So hopefully you can see these. Okay. But I wanted to show you actually what it looks like in the UI uh for us to be able to return the identities from enter ID. So on the lefth hand side I have the picture of adding a group for the workspace admin use case. And then the second place we've updated this in the UI is for the sharing of dashboards.
Uh, which one other thing to call out here, dashboards, genie rooms, and datab bricks apps are unique data objects in datab bricks because they're the only ones that can be shared outside of the workspace boundary to account users. So when you share a a dashboard to an Entra ID identity, if that person has never been provisioned and never accessed data bricks, they will sign in and become an account user. they are not automatically granted access to the workspace that dashboard is coming from.
I want to make that super clear because we we we want to respect the uh sanctity of the workspace. Okay. Now, we're going to get into some of the FAQ section sections because I've been in several different customer calls with folks who are in the public preview and these are the common themes that come up. So the first and probably most important is customers want to know about how do we migrate from skim to AIM. Um and there's a couple things I want to call out or like best practices to recommend.

13:01 AIM migration and support

So the first is really the only pre-work and the best pre-work for you is to make sure that all the identities that you're currently syncing via skim have external ID populated. Users, groups, and SPs all have an external ID that comes from Entra ID. Um, as long as the external ID is populated, then we're able to recognize on the datab brick side that's been provisioned over SKIM in data bricks is the same that exists in Entra ID and once AIM is on, we effectively tie those identities together and we're able to have um automatic takeover management of the identities.
The other thing that I want to bring up is uh when you enable automatic, you are enabling this at the account level and it's turned on for all workspaces where they have UC enabled. Um or you may have heard the term identity federation before and we we don't actually recommend that you disconnect your SKIM connector immediately. AIM and SKIM are interoperable and so please run them side by side until you feel comfortable to remove the skim connector and have ensured that the identity syncing is taking place.
And then the last piece here which is we're not actually shutting off account SKIM. So you can still make calls to account SKIM while you have AIM enabled. And for the time being this is actually best practice and recommended as our only API for you to be able to provision identities. You'll see in my road map slide um we are delivering something there uh before GA but in the near term for public preview this is your best API to still provision once but then have automatic manage the identity after that.
All right. Okay. Next piece is around nested group support. So I wanted to run through an example and I did color code it to try to hopefully make it easier. Um, but what is what does AIM support for nested groups look like? All right, so imagine that you have parent group P with child group C and it gets added into workspace one. Uh, AIM will respect the access control of both of the parent and child group which means that any member that is a member of P or C gets access and can sign in to access uh, workspace 1.
The thing I do want to make clear here though is uh only P gets provisioned to the account. Child group C is not autoprovisioned. And this was an intentional design decision for a couple different reasons. But one being that let's say that you have a nested group that has 30 layers. We don't want to unnecessarily provision groups to the account unless you as the administrators have the intention of granting additional permissions to those groups in data bricks.
So that's the context here. Um, and where it comes out as transparently the kind of funky behavior today in public preview is that child group C since it's not provisioned, it won't show it won't be um uh it's not automatically provisioned to the workspace and it won't show up in UIs that are only returning provisioned identities. So those are the notebooks. Um, and I think this will be I see some confused faces, but I am hoping this becomes more clear in the demo.
And if it doesn't, we can loop back on it at the end. Okay. SPs are a little bit more clear. SPS are respected within groups. So if you add a group containing SPS to data bricks, those SPs will be able to authenticate and they uh they get autoprovisioned similarly to the way that users do. It's just the first time authentication is in the command line rather than through the UI. Uh and the other time that SPs can be autoprovisioned is if an admin assigns a role or grants a permission that at that point in time also autoprovisions the identity.
Okay, now to the demo which hopefully my I didn't get logged out of anything. So the first thing I wanted to show is just very quickly like where do I enable this in my account? This is an again it's like an account setting. So I'm in the account console right now and on the previews tab it is the third one down you can see it's just this automatic identity management which is in public preview today. Then I wanted to go to show you this in the in the account user management UI.

17:28 Demonstration of AIM

So right now I I searched for identities with the name seamless and this is the whole concept of the like provisioned versus non-provisioned identities. So what with the change of automatic we're making it that all identities in entra ID can be read by data bricks. We are not and this is intentional we are not provisioning every single one of the identities that you have in entra ID to the account. I don't think you probably want us to do that either.
Um, we are however we have that visibility. So that's why it's reflected back to the admin in the account admin UI. So you can see that here with the status where this this identity seamless 10 is active because it's been provisioned to the account. Um, whereas seamless user 12 is it says actually like inactive, no usage. This identity has not been provisioned. Um, but if I were to add them to the workspace, they would be provisioned.
The other thing I wanted to call out was the source type. And I'm uh going to now jump into the workspace admin UI to show you two other things and then turn it. Okay. So now we're in the workspace admin UI. We are on the groups page. I already provisioned this group before our session. I have automatic on in this account. I added this group over AIM and you can see I have all of these users here today. Um if I wanted to look at, is has not been provisioned to the account yet because he hasn't signed signed in.
I can on demand provision him here which will uh add will provision him to the account. The other thing I wanted to show you is just how quickly AIM operates with respecting the uh respecting enter ID as the identity directory and source of truth. So we are now in the Azure portal and this is this is that same group just in Entra ID. So I'm going to go here and I'm actually going to add someone to the group. Let's search for someone with the name K.
Okay, let's add. Oh no, she's might be a guest user. Let's add uh Kimman. Okay, so I have now see Azure didn't even show it as quickly. Um, I've now added Kimon to the DICE workspace group. And I'm going to go back and in my workspace, while that's loading, I can see that Kyman is there as an identity. So, if Kimman were to sign in, thank you. I think it deserves a round of applause as well. I'm sure my engineering team would feel the same way.
Um, great. So, one other thing I did want to show you guys, which is the actual flow to let's say add a nested group to the workspace. So, I'm going to go here and I'm going to search for dice sales. And you can tell that these are identities that are from Entra ID and not in the account because they're appended with the in Microsoft Entra ID. So, I'm going to add this group. And then when I go search for this, I actually see well, I see the parent group and the children group.
But if I go into dice sales, I see both of the children group or child child groups have been added to the workspace. One last thing to show you, which is what does this look like for a dashboard. I've got this beautiful dashboard here. And if I want to share, same thing. So I own this dashboard. I am not only going to share to um the DICE dashboard group, but I'm also going to share to, both of whom have not been provisioned to the account.
And this should go through. This is now under the hood adding the dice dashboard group and wing to the account. So that is what I wanted to cover on the demo piece. Um very quickly roadmap we do have other good things coming before we get to G if this ever loads. One I already made mention of which is the API support. um we we do want to be we want to be able to give you full abil full capabilities similar to what you have in skim today but in an AIM world.
The second is you may have caught this. Uh one of the big pieces of customer feedback that we've heard is this is awesome. I love that you guys can do this but I also don't want you to read the entirety of my entra ID out to all of my users. Can you give me some way to limit the tenant boundary? So, we're working on a an allow list uh that can be defined by the account admin to be able to limit that scope. And then the last piece is that kind of oddity that I spoke about with the nested group behavior in the workspace and like the notebook UI.
We'll be fixing that so that if a nonpro provisioned identity has access to the workspace, you actually see that identity to be able to share with. So, with that, I will pass it off to. Thanks, Keegan. Right. So the next section is about privileged identity management. I want to start this with an interesting uh statistics. So this is a report from Microsoft in their 2024 state of multicloud security report where they found that they manage over 200 million identities in across all customers uh in Entra and they have like observed over 55,000 permissions and of that only 2% of the permissions assigned were actually being used and over 50% are at high risk.

23:37 Privileged identity management

So that tells us something that we are overpromissioning users. We're giving more permissions than what we really need. That actually affects uh exposes customers or enterprises to like security breach uh compliance issues or even like uh doesn't affect their operational efficiency. So which is why uh Azure introduced privileged identity management. Now why do you need it? uh basically it can help you do things like providing just in time privilege access to resources.
So that means you don't need to give them access permanently. You only give them access only when they need it when they need to do any uh uh any configuration which requires elevator priv privileges. Uh you can set timebound access which means that you can set duration from like an hour to a few days or a couple of weeks. So you are assured that at the end of the duration the access is automatically revoked. So an admin doesn't have to follow up again on the permissions and it the final thing is mainly use scenarios where you are doing like a production configuration or a break scenarios where you need elevated access to do uh any exception workflows.
So where you need a temporary access to a a privilege role and do that operation and after that the access is no longer required. So with that it helps you with like minimizing your risk of any security breach. It improves your operational efficiency and there helps inating with your compliance and reporting. So that's how pim works. Now how can we use it in databicks? Now within databicks also we have many different roles which are privileged in nature like you have like the account admin which gives you like the full power to do literally anything you want in databas admin which like the root uh unity catalog admin.
Then you have individual workspace admin. So if you need to do any of those production workspace configuration or account level configuration, BIM is a good candidate for that. The other could use cases could be like if you're trying to onboard an external user, you want to bring them as a standard user in a workspace for a 6 months period, you can set a time bound access and you're assured that at the end of their contract, the access is automatically rewarded.
And then if you have scenarios where you have like a data mesh architecture, you have you published a unity catalog which is used by a lot of downstream consumers and you often have a data steward who is like responsible for the catalog in terms of uh configuring the metadata or permissioning. You want to be able to ensure that only restricted users have access and only they get access only when they need it. So that's a third scenario.
And finally, this is something you could do in Unity catalog native features as well. If we can use in pim also it's possible like if you have PII data where you're using features like role level security column masking to restrict by default you want no one to be able to see the PI data but only when they need it. So a user could access a temporary elevated permission to be a member of a group and that lets him access some of the PI data and then after that it's revoked.
So these are some of the examples how you can use PIM in in Azure data bricks. the how how PIM actually works. Uh so PIM in data in Azure works on like three different options. You can do like PIM at a at a role level that's like Azure custom native role or custom role. You can do PIM at groups and you can do PIM at an Azure resource. But the way database is designed we we encourage customers to manage permissions by means of group like either an account admin or a meta store admin or or a workspace admin.
So we recommend that you use the pim with groups feature to be able to manage database access permissions. So how do we go about setting it? So it's mainly requires three things to do it. Starting with setting up the pim itself. Now by default PIM is not a feature which is available in the default entra license. You need to be having either entra p2 license or entra governance license which gives you that additional pim feature.
So you need to identify what are the groups which we need to be able to control using pim. So good example would be okay I have a group which is an account admin. So I want to be able to control privilege access using pim. So you identify those groups and activate pim only for those groups and then you set okay which are the users who can access temporary permission to be that groups. You don't want to give that to all the users.
So you set eligible users who can it could be someone who is a production team or it could be a devops team who deploy the actual production changes. They are the one who needs to be eligible to be able to u uh request access to a pim. And finally within pim you have options like you can set things like the maximum duration you want to give access. You can set things like it needs to have a approval workflow. There needs to be a justification given and whole lot of features which you can set.
So that's how you set up the pim in the completely on the entra side and then on the database side as I mentioned before you manage permissions using groups. So you don't give users direct access you manage access using groups and add users to the groups. So like in example you have a group for account admin one for metas store similarly for catalog admins or workspace admin and so on and then finally you need to onboard those groups into into databicks.
So in this case we have automatic entity management where you can add those eligible groups into the account. So like as as showed like in the workspace URL you can do it from the UI like in the account console we don't have the UI option right now but then I have the command which says we can use the database CLI to add a entra group into uh into the database account console. So you notice that I'm passing the external ID which is which maps to the object ID in entra.
So that is how automatic guarantee management recognizes that this group in database is same as the equivalent group on the intra side and that's it once that is done. So you're sorted. So that means like you have the pin setup you have the permission setup and you have onboarded those groups into into database. So if you want to walk through a scenario, it could be like if a user wants to create something like an account level configuration, let's say NCC or a private endpoint, he goes to the pin group, he requests access for the membership, it could trigger an approval workflow if it is configured and someone who is in approval or owner and approves it.
And then the next time he logs in uh using the power of automatic identity management it instantly checks whether the user has a permission member of that particular group and he has and then he can perform the change and at the end of the pin duration which you set the access is revoked. So the next time the user logs in he doesn't have access to that. So this is one way how you could do a pim. So I do have a video recording uh to show how this actually works.
So this is on Azure data bricks right. So this is a console where I I'm as an admin user. I I go to my tenant. I have a group and I call it DB account admin. And you would notice that it currently doesn't have any members. So it's an empty group. Uh I have enabled PIM on this uh DB group and then I'm the owner. I go to the eligible assignment. I have added a test user who is one of the user who can request elevator access if he needs.
And then I have some settings done. If you go to the settings, you can see that I have set the maximum allowed duration is 1 hour and things like uh there is an approval. That means the if someone requests access, it should come to me. I should approve it. And then you have options like you need to give a justification for activations and all those details. And there are lot of other features available. Now I switch to uh incognito mode I log in as the actual test user.
So right now he's not a member of the group. So when he tries to log to the account account console uh he's not able to login because he doesn't have permission. So what he does let's say he wants to do a production change process. He goes to the Microsoft Entra. He goes to the PIM and selects the group. He has access to only one of the group and he see an option where he can activate it. He clicks on activate. He gives a justification.
So let's say he could say that I need temporary account admin because I need to create a private endpoint um on say UK south region or so on or you could make sure that they they mention a change request number from their service now because they're doing as part of a production request. He gives the justification and he says activate which internally creates the approval workflow. This is all pimp features so not databicks side.
So I switch back to my role. So I go to the pim again. This time if if the request is there, it should come as a approval request for me. I click on approve request and I switch to the groups and I see that there is a request waiting for my approval. That's the one which is initiated by the test user. I click on it. I click say approve. I can verify that what is the reason the justification he has given and I say approve and that's confirmed.
So that's how pim works. So what happens behind the scenes? stream ensure that the user is added as a member of the group. So you can you can verify that if you go to the I switch back to my test user I verify whether I have access if my user has been added there. I select the DB account admin but this time it has one member um and that is the test user. So that's added. So this is all done in pimp side. Now the user goes to account console. he logs off and tries to tries to log in this time.
Hopefully, it should let him access. So, it it could be like few seconds uh delay like you say. Even though it's a recording, there was still a glitch. So, I had to do a Yeah, I'll be passing this on to the engineering team. So, I had to do a refresh and this time the user see the option called as manage account. And that's it. He clicks on it. Now, he is the account admin. He can see all the options there. he is able to do whatever change operation and this is done instantly because because of the power of automatic identity management no need to wait for the schem sync to happen you can verify that in the DB account admin you can go to the membership and you will see the test user is added in databics also that's all done he can go to the cloud resources he can do all his changes he can create his private endpoints and whatever he wants to do once that is done the end of the maximum one hour duration the next time the user tries to login the membership is revoked so he won't be able to log into the account console like how he experienced in the beginning or if he was in an active session at the end of the hour with automatic it would be yeah end up being um booted out right so so that's how a quick demo of how it works uh the final thing is about how do you monitor the access logs so there are couple of ways you can do it you can directly go into the pay audit history which gives you each and every user what requests they have accessed or what privilege they were assigned for which duration and you can cross reference that with the unity catalog system audit table to make sure that user has in done changes which is intended for.
So that's are two ways you can monitor and for any audit purpose. Uh so we have seen how pim in combination with automatic guarantee management works for Azure data bricks but let's say if you are on AWS and most of the customers on AWS they have um Azure entra as their IDP yes you can't use AM for now but then you can still use the pim concept just that you'll have all the facilities where you can do the instant privilege access but the only catch is yes because you need to be dependent on a skim sync to happen so it won't be instant it may take few minutes to like up to 20 minutes on some cases 30 minutes uh for the sync to happen but it gives you the same functionality which you want to do and if you're using other IDPS like octa has similar thing called as octa privileged access Google has something similar called as google privilege access manager they all work on the same concept where to give a temporary elevated access to users based on time bond with justifications and so on so that's in short about how pim works the final section is about the future of identity at datab bricks I'll hand it.

36:22 Future of identity and scale

Okay. And we're going to go through this kind of quickly but just to bring it all full circle. So we had talked about in the beginning kind of the vision of where we're trying to go with all of this which is to make administration make make the uh admin overhead piece less um less time consuming in data bricks. So we wanted to talk through like what do we actually have available for you depending on the cloud that you're in.
So we spent a lot of time going very deep on what we have today in Azure with automatic identity management. But we also today have for AWS and coming very soon for GCP just in time provisioning which gives more of the autoprovisioning of users use case in both of those clouds. So what does this actually look like? This is called just in time um or often called JIT provisioning. And effectively what this does is if you con if you turn it on uh and you have SSO configured anyone who's an identity in your IDP is able to um on demand signed into data bricks and be autoprovisioned and the real benefit of this really comes from the sharing use case where let's say again um I spoke about that in the beginning that apps genie rooms and dashboards can be shared to account users.
So let's say that you created a data bicks app and you want almost everyone in your organization to be able to access it because I don't know maybe it's an HR app or something. You can then set it to say share to all account users and if you turn on JIT in the account then anyone with the link who's in the IDP can sign in and access that data bricks app. They'll be provisioned as an account user. They will not be added to any workspaces and so it is secure in that sense of the respecting the workspace boundary.
Um I do also just want to call out best practice is to still have skim configured for deprovisioning. Uh and this is unique to SSO. The second thing I want to call out as a best practice is if you're trying to get this sharing account sharing goodness uh on AWS, you do need to have unified login configured. And this is actually the new default setting for how SSO works in AWS. And effectively what this means is you configure SSO or a single sign on once at the account and then that configuration is pushed down to all of the different workspaces.
This is much easier from an administrative perspective because you only have to configure it once and for users they only see one chicklet instead of 10 or potentially hundreds. Um the last thing I want to call out is you may have seen my asterisks that JIT is coming soon to GCP and that is because we are also changing the identity model on GCP. Today if you spin up a new GCP account uh SSO is autoconfigured against Google identity.
We will be shift we heard feedback from customers. So we'll be shifting to account for that feedback so that when you create a GCP account you can configure the SSO and the skin provisioning against whichever identity provider uh you so choose. Now finally to leave you with the last and potentially best part which is we are saying that we want you to bring more users into your account. Well what are we doing to support you on that scale-wise?
So one thing to call out is we have had this for some time and we are continuing to invest which is we have on demand limits where we can scale to hundreds of thousands of users and groups at this point. So if you need additional scale within your account, please work with your account team. They'll write in and we can work to address that. Second, we're actually also working behind the scenes in the next few quarters to increase the default limits by moving customers to a more scalable version of Skim.
For the vast majority of customers, this will not require um any admin intervention. Uh for the ones that there is, we will be in contact to work on uh work on the details there and move you to the scalable version of SKIM. And I'm over time. So, as a reminder, what we covered AIM, PIM, and then the future of identity at data bricks. Thank you everyone. And uh we'll stick around for questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.