Unity Catalog Lakeguard: Secure and Efficient Compute for Your Enterprise

Summary

• Fine-grained governance: This video demonstrates how Unity Catalog's Lakeguard enforces role-based access control across shared compute, even when multiple users concurrently access data.
• Flexible compute options: Learn how to choose the right compute model—serverless, clusters, workflows, or SQL warehouses—while maintaining consistent security and governance policies.
• External data integration: Discover how to confidently integrate data from lakes, databases, Salesforce, and cloud functions into your Databricks workloads with unified governance.

Unity Catalog Lakeguard: Secure and Efficient Compute for Your Enterprise

Talk by Jakob Mund, Staff Product Manager, Databricks ; Scott Van Woudenberg, Sr. Staff Product Manager, Databricks
Modern data workloads span multiple sources — data lakes, databases, apps like Salesforce and services like cloud functions. But as teams scale, secure data access and governance across shared compute becomes critical. In this video, learn how to confidently integrate external data and services into your workloads using Spark and Unity Catalog on Databricks. We'll explore compute options like serverless, clusters, workflows and SQL warehouses, and show how Unity Catalog’s Lakeguard enforces fine-grained governance — even when concurrently sharing compute by multiple users. Walk away ready to choose the right compute model for your team’s needs — without sacrificing security or efficiency.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Lakeguard and how does it provide fine-grained access control?

Lakeguard is a capability of Unity Catalog that enforces fine-grained governance policies at the table and column level. Even when multiple users share the same compute cluster, Lakeguard ensures each user can only access data their permissions allow, enforcing access control based on individual user and group membership.

What are the two compute access modes available in Databricks?

This video explains two modes: Standard access mode, which enforces fine-grained access control for multiple concurrent users, and Dedicated access mode, which dedicates compute to a single user or group in exchange for unrestricted Spark API access, including RDD APIs.

How does serverless compute handle security and governance?

With serverless notebooks and jobs, security is configuration-less. The platform automatically uses the standard access mode for multi-user governance, applying fine-grained permissions without additional setup. This simplifies the developer experience while maintaining enterprise security.

Can you mix different compute types with the same governance policies?

Yes, this video demonstrates that Unity Catalog governance applies consistently across all compute types—serverless, clusters (in auto or standard mode), workflows, and SQL warehouses. This unified governance simplifies security management across your Databricks Data and AI platform.

Full transcript

All right, even better. That individual users permission will be taken into account. So, uh if I execute code on that cluster um and I have like user level permission that only I have and no other user, I will still be able to get that data. If I'm um a user who is there in multiple groups, I will get the permission of each of my groups. It's completely additive. All right. Uh and this uh architecture is also used for serverless notebooks and serless jobs.
And it's also used if you have a UC enabled SQL warehouse. This is also the architecture that we have. We have a second exit mode, the dedicated one. will later tell you all about this, but just a quick overview. This one is an access mode where you dedicate compute to a single user or a single group. And the key message is if you do this, you get privilege machine access, meaning you can run the full unrestricted Spark API, including things like RDD APIs.
Um, but the fine grain access control is um boiled down to the single user or the single group that you assigned or dedicated your cluster to. will again tell you more about this. Uh but that's the general overview of what you have. Now how is this relevant for you if you use serverless? Not at all. It's configuration less for you. We use the standard access mode for this one. You don't have to do anything. If you're using normal clusters, you can uh the UI experience looks like four.
If you create a compute, there is an advanced setting that you can set and there by default it is in what we call the auto mode. uh by auto mode is will determine what is the best access mode for you based on the other configuration settings that you did for your cluster. Um so if you're picking for instance an older version of datab runtime we sometimes might recommend that you start with the dedicated access mode but when you move to a newer one uh we mostly um will will guide you in the auto to the standard access mode because uh it has more features.
All right. Um you can do the same in the API. Um there is a field called data security mode. Um we recommend that you set that to the auto mode as well. This is equivalent to the UI. Um and if you want to do that at scale, we recommend that you use cluster policies to set this for all your clusters. All right. Um so that's the primer on UC compute. Uh let's jump into standard clusters and serverless. Let's recall that we have two goals here for secure and efficient compute.
We want to make sure that users are separated from each other and from the spark engine to prevent the overfetching problem. We we achieve this um using what we call lake guard which is a bunch of technologies uh that enforce the governance at the runtime layer. uh and on a spark driver what we do is that we essentially isolate the spark driver and each user session on that machine. So each notebook and job and client code that you're running is completely separated from other users and from the driver using a combination of a sandber technique for the workloads itself and spark connect so that your notebook can still use spark code.
All right. uh and we did something similar on the executives because on executives you are still able to run user provided code and we have to ensure that the same applies for the userdefined code that cannot escape to the spark driver and not ex not like see anything that another user runs or see something that's going on in the spark executed JVM. So we use the sandbox isolation technique for this as well. All right.
Um so what does it get us in terms of new features and workloads? So there are three things that I want to talk about today. Uh the first one is machine learning. Um with data bricks runtime 16. 4 you can already run a couple of machine learning workloads on a standard cluster. So what you can do is and it's included out of the box with database runtime is you can use scikitlearn uh and you can use mlflow for like for your mops uh and your tracking experiments um and on top of this what is provided in that works runtime you can install pytorch x to boost tensorflow on top of it all of this will run in single node mode meaning that it will run on the driver only it will not be distributed to the um to the executives and Now new with DBR17 will also allow you to run a spark ML uh on a datab bricks UC standard cluster it will be both um single node or distributed so it will use make use of the executors and you don't have to install anything it is bundled in data bricks runtime not even the ML version of it just the plain one um if you want to use GPUs our recommendation is that you try out the new serverless GPU compute that we have in serverless um it essentially allows you to run your GBU workloads on serless using the same um architecture.
All right. Uh the second important thing that I want to talk about is structured streaming. So up until recently structured streaming was something that was not fully supported in standard clusters. That is because it requires the same user isolation that we need for UDFs. But now for the driver with a couple of more technical challenges to overcome. Um but since the recent uh the most recent DBR version 164 uh we have the full streaming API support.
So you're including userdefined code or user provided code in terms of for each for each batch flat group with state transform with state and pendas. Uh so this is also the code example here as you see um there is a function that we define here which is our um for each batch function and we provide this here and when you do this even though this is user code that is actually executed by spark and that also does some spark calls within it it's all isolated using a combination of the UDF sandboxing on the executors and uh the spark connect protocol.
All right, finally last thing I want to talk about today for standard clusters is how you can connect to external cloud services. Um you might be familiar with the concept that we had before where you could attach instance profiles or service principles to your cluster and then every user gets essentially a permissions uh that is part of that instance profile. Uh this is something that wouldn't work with the UC governance model uh in UC because each user might have different permissions.
So we added a replacement for this one which we believe is better uh which we call UC service credentials and UC service credentials allow you to authenticate with cloud services uh using the same mechanism for shortlived credentials as we have with instance profiles. Um plus you get access to uh you you can govern the access to those service credentials using unity catalog. So you can essentially if you look at the code this is just normal B to3 um AWS cloud SDK code and you provide a name for service credentials this is let's call it my secrets manager credential h and if I as a user execute this code unity catalog will check whether I actually have permission to use this credential and if I do so under the hood unity catalog will get shortlive shortlift credential and use this one to authenticate um with the B3 session to access secrets manager.
All right. Um this is a replacement for instance profiles or service principles um or Google service accounts. It's available on all UC clusters. It's available for delta life table pipelines. It's available also for us userdefined functions um on UC enabled SQL warehouses. All right. Um, that's that's what's that's how it works on a Senate cluster and that what's new for the Senate cluster and I'll hand it over to tell you how all of this works on a dedicated one hopefully.
Okay. Yeah, my mag's hot. Thank you. Thanks. Um, so yeah, I'm just going to talk a little bit about uh dedicated clusters or dedicated access mode assigned to a single entity. Um, essentially shorthand we, you know, call them dedicated clusters although that's not the formal product name. Um, so before I dive into it though, uh, how they, you know, kind of how to create them, how they work, uh, some of the some of the use cases, I want to start by talking about like some of the reasons you might use them.
So hopefully we can kind of go through and convince like most of most of the most of these use cases uh, will hopefully work on serverless or standard, but in in some certain ones still you still need to use dedicated. So we'll go into it here. So first and foremost, the kind of last remaining language that doesn't work on serverless or standard is R. So the R programming language. So if you're using R, you're going to need to use dedicated access mode.
Um if you are using distributed ML aside from Spark ML, which just talked about, like so distributed PyTorch or 10 distributed TensorFlow or distributed XG Boost, these are all still only supported on dedicated access mode. So you'll need to use that if you're using uh the Docker container service. So basically you're providing uh providing an a customized image that we then combine with our DB runtime and you're using that to deploy your clusters.
Um that's still only supported for with a dedicated access mode. Um there are a few like a handful of Spark APIs that are not currently not yet supported on standard and serverless. And then there's also a few kind of corner cases for streaming data sources and streaming data source configurations that are not yet supported on um serverless or standard. So definitely recommend um taking a look at the that we have a documentation page that's like cluster access modes limitations that kind of goes through all of these kind of little corner cases.
In general um this list at the top used to be a lot longer like even when I started here like nine months ago this list was quite a bit longer. Um, so some of the ones that are, you know, if you if you've been with data bricks for a while, you might not necessarily be keeping up with all the news, but like some of the things that are no longer reasons to use dedicated access mode, GPUs. Uh, for one, we just recently announced the serverless GPU compute that mentioned earlier.
It's in beta. You can try it out. This is you can get access to GPU Nvidia GPUs on using serverless compute. Um, Scola is also now in GA uh supported for standard clusters and we're working to bring it to serverless cluster or serverless as well. So serverless compute as well. So that's going to be coming soon also. So pretty soon Scola will work on standard and serverless. Um and then spark submit jobs that we now added a capability where you can just use a jar task instead.
So if you're if that's a a use case that you have um you can use a jar task instead on a standard cluster and be fine. Um, but again, there are some kind of nuances and corner cases and stuff that still remain. We're looking to close those as as quickly as we can. Um, if you have any other reasons or issues that prevent you from using standard or serless, we'd definitely like to talk to you. Um, help us prioritize the ones that are important to everybody to get all of this stuff working on on the on serverless and standard.
Cool. Um, so now a little bit about how dedicated works like what are the key characteristics? Um, like I think mentioned this earlier, it's kind of the full unrestricted Spark API access. So this is the this is the you know uh close to the original kind of Spark concept single user single entity single assign principle uh entity that talked about earlier. Um it and the way we the way it works today is you either that authorizing authorizing uh principle is either a user or a group.
So you can assign a dedicated cluster to a single user or to a single group and all of the permissions uh for any activity that is executed or run on that dedicated cluster is the permissions are scoped to that assigned uh authorizing principle. So no other permissions are honored. So if that authorizing principle is a single user for example um that single user permissions will be used. If it's a group and I'm a group member, only that group's permissions are used, not my individual users permissions.
So, that's just something to keep in mind. I'll get into some of the kind of nuances that that brings about in in a couple of slides. So, um, in terms of how to create dedicated access mode clusters in the UX, uh, already showed part of this, but it's essentially instead of the auto mode, you pick manual and also it's under the advanced settings kind of toward the bottom of the create compute UI. You click manual, select dedicated from the dropdown, and then you put in your authorizing principle essentially.
So it's either a single user or group. Um the bottom slide down here just kind of shows you or the bottom picture just kind of shows you for if it's a group cluster, if you assign that dedicated cluster to a group, the all of the group members automatically inherit the group automatically inherits the can attach to. So all group members inherit that permission. So they can attach to and use the group cluster that we do that automatically.
So you don't have to mess with that at all yourselves. Um yeah. So then uh as far as like the behaviors I was talking about um like kind of about the per the specific permissions that are used this is an example where um for any of the Unity catalog securables. So we're talking about tables, schemas, uh you know uh endpoints, uh models, connections, the the various UC Unity catalog securables. um the permission of the authorizing principle is the only permission that's used.
So if your if it's a group cluster for example and you're that group does not have uh permission to explicitly been granted permission to read a table, a select star on that table is going to fail. Um but as long as the group has permissions to it, it will work fine. So down below you see the example of like all right well is an a member of a of a group but they maybe try try to run a query on one of the tables like a temp table they've created in their in you know in their that in their user uh user schema their personal scratch schema or whatever um they're going to get they try to run that select star from a group cluster that they're that they're a member of that they're going to get the insufficient privileges perh permissions error some most of the time the permission the error that you can get is something like insufficient permissions.
Sometimes the error is not found simply for kind of better security practices. So just if you see not found oftentimes it's a permissions issue. Um just keep that in mind. Um let's see. Uh oh also want to mention um this also applies if you if your users belong to multiple groups only the groups permissions uh of the cluster that they're using are are used. So even if they belong to other groups if they if they're attached to and running a notebook against group you know a group clust a dedicated cluster assigned to group A only group A's permissions are used even if they also belong to like groups B C and D.
So again this is all this there's only a single authorizing principle that is used for dedicated clusters whether that's a user or a group. Um so then also this also applies to any unity catalog securables that are created from the group cluster or from the dedicated cluster. So essentially any of those objects are going to be owned by the authorizing principle. So in this example again I use group clusters where I create a schema um that if it you know if it doesn't exist and assuming I'm running this on a group cluster uh the marketing DS group is the is the own the name of the group that I assign that group clusters to then the owner of this object of this new schema that I just created is going to be the marketing DS group not my my individual user.
So um workspace objects in general work fairly similar. So if I, you know, I just have an example here where I'd use DBFS to copy a file from like, you know, an example data set into into my own a group workspace folder that I created. Um, again, when you then pull that up and look in the in the in the UI, you'll see that the owner is the group, not the not the user of that file. So again, just workspace objects are mostly consistent with that.
There are a few exceptions such as if you use ML flow or sorry if you use like autoML for like um regression or something like that. Some of the artifacts are created that belong to the user and they belong to the user. They're kind of the workspace artifacts but the like the stuff that actually gets executed by the runtime is all belongs to the group. So but we're working on kind of fixing all of those little like the edge cases.
Um so the audit logs for dedicated access mode clusters um the because like uh like the authorizing principle um is the entity that is used for all authorization uh access and permissions um we actually have to record we actually record both the authorizing principle and the user that actually was authenticated and actually executed the command. So it's like so we have two different um um uh tags in the identity metadata column of the system access audit log.
The run by is the authenticating principle. So single user or group. Uh and then the run as is whoever the user was logged in as when they when they executed that command on the dedicated cluster. So um both of those pieces of information are available in the audit logs so that there's always a record of you know hey it was group cluster this command to drop a table came from a group cluster but it was actually run by so and so you can go yell at them for dropping a table that they shouldn't have.
Um, so, uh, let's see. Uh, okay. Yeah. And then kind of touched on this a little bit already, but, um, just for dedicated access mode clusters, we still handle the securing the, uh, the, uh, the two kind of issues that mentioned, we still make sure that like those those security controls and governance of Unity catalog are not, uh, are not bypassed by that. So essentially for the fine grain access control which is the primary use case here that's relevant to group clusters again we do that fine grain access control filtering off of the cluster and what we call the filtering service so that the unoverfetched data is never never reaches the the dedicated cluster whether it's for a group or a user.
So this this solves the issue um the both of the issues that talked about as that for securing your that lake guard help us help us enforce on spark. Um right now there's a there's additional cost for this serverless running on the filtering service. So keep that in mind. If you're using fg fine grain access control from dedicated access mode clusters, there'll be an additional server serverless compute filtering cost.
Um but then fine grain access control also still supports both reads and writes. Uh the rights are supported via merge into and append. Um so yeah, that's pretty much that. So um pretty much done uh you know as far as uh the content we have. Um we definitely want to open it up for questions but hopefully this helped you all understand a little bit more about how we've taken Spark single use you know kind of the way Spark is currently architected provided additional isolation and pro and protection and government via Unity catalog via Unity catalog lake guard and yeah uh if we have any questions from folks we want to open it up feel free to ask we can come around with the microphone

Learn more about the Databricks Data and AI platform.