Supporting Fine-Grained Access Control with Server-Side Scan Planning

Summary

Unity Catalog's move to server-side scan planning, built on Iceberg REST Catalog Scan APIs, enables enforcement of fine-grained access controls — including row filtering and column masking — when external engines like Starburst query UC-governed Iceberg tables, without requiring any bespoke policy logic in the external engine.
The server-side scan planning flow works by having Unity Catalog intercept the metadata scan phase of an Iceberg read, apply the relevant row and column policies server-side, and return only the authorized filtered data file list to Starburst — which then reads only those files without ever seeing filtered-out rows.
Databricks and Starburst collaborated to build this capability, with a hospitality company customer story illustrating a real multi-vendor lakehouse deployment where consistent fine-grained access controls now apply uniformly across engines without policy drift or duplication.

Supporting Fine-Grained Access Control with Server-Side Scan Planning

Open table formats such as Iceberg allow organizations to run multiple engines on shared data, but this openness makes governance difficult. Unity Catalog already supports universal table-level access control for external engines like Starburst via credential vending, but that alone cannot support fine-grained policies like column masking or row filtering. UC’s move to server-side scan planning, built on Iceberg REST Catalog Scan APIs, allows enforcement of fine-grained access control in Starburst: policies are defined and applied server-side, and Starburst uses the filtered scan plan from UC to respect them, with no bespoke policy logic. Starburst and Databricks share how they collaborated to bring fine-grained access control to Starburst on UC-managed Iceberg tables.
Talk By: Alex Jiang, Product Manager, Databricks ; Jack Fitzpatrick, Senior Software Engineer, Starburst Data ;

Chapters

FAQs

What is server-side scan planning in Unity Catalog?

Server-side scan planning is a capability in Unity Catalog that intercepts the Iceberg metadata scan phase — before any data files are read — and applies row filter and column masking policies server-side, returning to the querying engine only a filtered list of data files that respect the access policies. This means external engines like Starburst receive a sanitized scan plan from Unity Catalog and never encounter the filtered-out rows or masked columns, enforcing fine-grained access controls without any policy logic needing to exist in the external engine itself. This video explains the architecture using the Iceberg REST Catalog Scan Planning API as the standard mechanism for this server-side enforcement.

Why was credential vending alone not enough for fine-grained access control?

Unity Catalog's credential vending approach already supported table-level access control for external engines by issuing temporary credentials scoped to the authorized tables. However, credential vending alone cannot enforce sub-table policies like column masking or row filtering because it grants access to the underlying files without the ability to apply those constraints at the storage layer. Server-side scan planning fills this gap by inserting Unity Catalog as the policy enforcement point in the Iceberg read path before any files are accessed.

How does Starburst integrate with Unity Catalog for fine-grained access?

In the architecture described in this video, Starburst submits an Iceberg table scan request to Unity Catalog through the Iceberg REST Catalog Scan Planning API rather than directly accessing the metadata files on storage. Unity Catalog evaluates the requesting user's row filter and column masking policies, generates temporary sanitized data file pointers covering only the authorized rows, and returns this filtered scan plan to Starburst. Starburst then reads only the files in the filtered scan plan, so the fine-grained access policy is enforced entirely server-side with no duplication of policy logic in Starburst.

What is the real-world impact of fine-grained access control across multiple query engines?

Without a unified fine-grained access control mechanism, organizations running multiple query engines on shared data must redefine row-level and column-level policies in every engine separately, which leads to policy drift, inconsistencies, and the risk of data being effectively ungoverned in one platform even when it is protected in another. This video presents a hospitality company customer story where a multi-vendor lakehouse now enforces consistent fine-grained access controls across engines through Unity Catalog server-side scan planning. The approach also simplifies auditing because policies are defined and logged centrally in Unity Catalog regardless of which engine executes the query.

Full transcript

[00:08] Hi everyone, my name is Alex. I'm joined by Jack and we're here to talk about how Unity Catalog enforces fine grain access controls when Starburst is quering data governed by Unity Catalog.
[00:21] So just as a disclaimer, uh this content is forformational purposes only. There's no promises for what we do in the future. And as a reminder, we strongly encourage you to complete your surveys.
[00:33] This will help us make next year's data and AI summit much better and also give feedback to us to make our sessions better in the future as well. >> Can you can you all hear me by the way? Okay. >> Okay. >> Awesome. >> Um,
[00:45] >> speak louder. You said >> Oh. Oh, too. Okay. Sorry. I saw everyone take their headphones off, so I was like, I don't know if you can't hear me or if I'm too loud. Um, as an introduction, my name is Alex. I'm a product manager at data bicks and I work
[00:58] on unity catalog interoperability as well as open table formats. If you have questions about any of these topics, feel free to reach out to me or if you want to know where to take your team to dinner tonight, I'm also the guy to talk
[01:10] to. Uh my contact information is here if you want to save it for future reference. And then >> we heard that there's volume controls as well on the headphones, so if it's too loud, you can mess with that a little bit. Um my name is Jack Fitzpatrick. I'm
[01:22] based in Boston, Massachusetts. um talk to me about iceberg rest cataloges uh starburst data we do data federation and lakehouses and backpacking as well I'm really passionate about that so >> awesome with that in mind let's get
[01:35] started so first thing I want to talk about is our vision for unifying governance from unity catalog and just to set the stage here interoperability and your data estate is not easy you see
[01:48] this problem where data policies and engines are often stuck into different silos so on the screen Here we see three different data platforms each of which are kind of isolated from one another and that creates several problems. The
[02:01] first of which is that data is locked into platforms and in order to ensure that this data is accessed across multiple platforms you often have to duplicate them multiple times which adds to your storage costs. Moreover,
[02:15] policies are also fragmented. you have to redefine them across every single platform you use which can cause drift and inconsistencies and often lead to data being completely ungoverned. Finally, auditing is also decentralized.
[02:29] Each platform has a different audit system which makes it really hard for your admins to know what data was accessed and by whom. So in order to address some of these challenges, we saw the industry converge
[02:42] on cataloges and open table formats to break data silos. And this allowed engines to read and write to just one copy of data. So you have data in iceberg or the delta lake format. You
[02:54] have hive metas store as your metas store. And this allows one copy of data to be read or written to from any engine. But the issue with this is that policies are still siloed into data
[03:06] platforms. There's no good way to solve the problem of admins having to redefine policies across multiple platforms even though the data copy problem was solved for.
[03:18] So that's where we introduced Unity Catalog as a way to keep governance centralized. That way you could keep one copy of data and one copy of policies and ensure that any engine could access
[03:30] your data with your access controls enforced. And the way that Unity catalog does this is whenever a user is using an engine, it'll check the access controls on the user running the query and if
[03:42] that user has access in Unity catalog, it'll vend credentials to query the data. If the user doesn't have access, it will deny credentials to access the data. So it works pretty seamlessly.
[03:56] But the key challenge is how do we do this with fine grain access controls? And when I say fine grain access controls, I mean policies that filter out specific rows or mask specific columns. This is a big challenge and there's two broad challenges here.
[04:09] The first of which is that credential vetting which is the mechanism by which Unity catalog can enforce policies is coarse grained. You either get access to a table or you don't. There's no notion of getting access to a subset of rows or a subset of columns.
[04:23] And moreover, not all engines can be trusted to enforce policies defined in Unity catalog. A lot of engines are in the open source community. They don't really have any understanding of policies. And even those that do have
[04:35] some notion of policies, they might not understand UC or Unity cataloges policies specifically. So we believe that this problem should be solved for by the catalog enforcing
[04:47] all policies whether it's coarse grained or fine grained. So our vision is centralized governance that applies not only for coarse grain access controls but also fine grain
[04:59] access controls. So if you're a user using an engine and you want to access data with row filters and column masks and force unity catalog can actually sanitize that data for you and ensure
[05:11] that the external engine only gets data that the end user has authorized or has access to. So the vision is essentially enforcing fine grain governance without any custom logic or using any third
[05:23] party tools. So yeah this is the problem. How exactly can Unity catalog make this fine grain governance vision a reality? Um it sounds good in practice but in implementation it's not that easy. Um
[05:37] but one thing to note is that Unity catalog is an iceberg rest catalog. So we offer open iceberg rest catalog APIs that allow external iceberg clients to read, create, and write to iceberg
[05:49] tables within Unity catalog. Any iceberg engine can integrate with these APIs and use the APIs to access data in iceberg storage. And the good thing about this is that within the iceberg rest
[06:01] specification, there's something called the scan planning API which can solve the problem of fine grain governance. And with that said, I'm going to pass it off to Jack to discuss this solution. Yep. So, how can we, you know, enforce
[06:14] these fine grained governance with Starburst? Well, it's going to be using the iceberg rest catalog APIs and serverside scan planning. So, if you're not familiar with scan planning, what that is is uh simply just when you're
[06:26] reading an iceberg table, you read the metadata files first to get a sense of the table, uh the schema, and what data files you need to read to satisfy the query. And so, planning out what files you need to read is just scan planning.
[06:39] And usually that's done on the client side, but with iceberg 1.7, the iceberg restspec was updated to include this endpoint uh plan. And what you can do is submit what columns you want to read,
[06:51] certain filters that you want to apply to the data. And in response from the catalog server, you'll receive a scan plan. And so this scan plan has a list of file scan tasks. And that just tells you what data files to read and what
[07:04] ranges within that data file to read from, as well as what deletes to apply and things like that. and it will give you a list of storage credentials to use to be able to read those data files. And so this comes with several benefits. The first one being that it offloads
[07:16] planning from the client to the server. Uh the server has a much more comprehensive view of these tables and so it's you know might be easier and more performant for them to be able to give us the scan plan directly. Secondly, um the catalog is able to
[07:29] control visibility. So the catalog itself can redact certain data and this is getting to the fine grain access controls that we were talking about earlier. Thirdly, um all of this is still secure access. These storage credentials allow us to vent access to
[07:42] specifically the files that the client needs to see. Uh nothing else. And then finally, um this is execution ready. So these file scan tasks are actually a concept in the iceberg library. And uh most execution engine should be able to
[07:55] run these straight up with no additional modifications. So if we want to talk a bit about the general flow of serverside scan planning, we have a SQL query coming into the Starburst query engine. It does its stage creation. It does its split
[08:07] creation. And splits here are really just kind of chunks of data that the query engine is going to be reading from. So in this case, it's just going to be an iceberg table. So we hop over to the iceberg client where we start planning the files. So with typical
[08:19] client side scan planning, what we would do is read directly from the metadata files. We would create our own scan tasks. This is on the starburst coordinator uh because starburst query engine is distributed. So this would be
[08:31] on the coordinator. And then we would launch scan tasks to the workers to read directly from the base data files. So with client side planning, we are only ever reading from the iceberg files themselves. Um but serverside scan
[08:43] planning gives Unity catalog a hook into that process. So now we can go through the scan API. Uh Unity catalog will apply its fine grain access controls. It will look at the metadata files. It will look at the policies that apply to a
[08:56] certain user and it would generate these temporary data files that are sanitized already. And by returning the scan tasks back to us that reference those temporary data files, uh, Starburst is able to only ever read that sanitized
[09:10] data without having to do any of the policy application ourselves. Um, all of that's already done by generating those temporary data files. So from another perspective, we have a user here that would want to query Starburst. Starburst
[09:23] would use the scan API to Unity Catalog. Unity catalog would then apply whatever policies it needs to and as a result it would create these sanitized data files. Then Unity Catalog would create a scan
[09:36] plan for us, send it back to Starburst and Starburst uses those scan tasks as well as the vended credentials to access those sanitized data files and then the data that it returns back to the user is
[09:48] going to be sanitized. All of the policies that are set up on the Unity catalog side will be respected by the Starburst query. So let's talk a bit about support with Starburst. Um server
[10:00] side scan planning support will be shipped with Starburst enterprise platform 481 for any iceberg rest catalog. Uh in this case for an example we have a query that you might see with Starburst. Uh it's joining two different connectors. So we have a bunch of
[10:13] different connectors across many technologies. The top table here is going to be an iceberg connector using the Unity REST catalog. And any policies that you create on the Unity side are
[10:25] going to be pulled into the query already sanitized. And so when you join it with any one of our number other connectors that we have in our catalog, uh you're going to get that data federation along with the policies from
[10:37] Unity catalog enforced. Cool. Uh a quick note on identity just to talk about this a bit. Um this is unchanged from how it currently works with the Unity catalog, but it's worthwhile noting because it's an aspect
[10:50] of it. Um first the user would sign into Starburst. They use whatever ID provider they prefer to get a token which is then forwarded along to Unity catalog using those REST endpoints. Uh some cloud providers do require a token exchange.
[11:03] So we might need to mint a new data bricks token to use specifically uh for this. And then finally, Unity catalog would identify the principle using the token. It would evaluate any of the policies it needs to using that
[11:16] principle and then it would be able to return sanitized data back to us. So all of this is remains seamless. Cool. And so now we can talk a bit about a case study. Uh this is a real Starburst customer in the travel and
[11:28] hospitality space and um they are a multi- vendor lakehouse. Uh they're undergoing a modernization. So they're migrating to a new architecture. Uh and their current tech stack kind of looks a
[11:41] little like this. They're using Starburst, of course. They're using Octa for an identity provider. Um they are using multiple cataloges for iceberg. In this case, they're using both Unity catalog as well as glue. And um they are also using Apache Ranger. And so this is
[11:55] to apply an additional layer of governance on top of Starburst because the policies from Unity Catalog are not respected by Starburst. So they need something else to be able to make that work. And this is a problem for them. They would really like to have those
[12:08] policies defined in Unity catalog to be applied to Starburst without needing that extra layer of governance in between. Um so without serverside scan planning in the current world there are a couple of bad solutions to this problem. They could only use data bricks
[12:21] which is good but Starburst brings a data federation that they would really like. They have multiple cataloges. They want to be able to query across other connectors as well. And so they lose that interoperability to be able to use external engines that are really well
[12:34] suited for the problems that they have. Uh secondly, they could just duplicate all of these policies. You know, any policy defined in Unity, they could also define in Apache Ranger. But this is also really not a good solution because
[12:46] uh they could be different policy languages. They could they're different technologies. The policies might be defined in different ways. They would need to keep them all in sync. They would need to make sure that there's no inconsistencies or drifts across it. And it's hard to apply, you know, auditing
[12:59] when data can change in different ways depending on the query engine that you're using to query it. Uh, finally, they could just accept the gaps in governance. Either they could use lowest common denominator access controls. So
[13:11] just saying like what we were saying before where they cut off access to the table completely if you're certain user uh or they just accept the compliance risk. But that's also not great. You don't want to be uh you know have a
[13:23] compliance risk. So what we want to do and what we did do is allow uh unity catalog policies to be applied to starburst. This governance must extend to starburst as well.
[13:35] And so to talk a bit about the flow like we did earlier um a user will use starburst to query data. Starburst will hit that scan API endpoint. Unity catalog will consult its own policies
[13:48] and see what that user is allowed to see. They will generate the sanitized data. So in this case, we're just masking the email column. Um they will return that scan plan back to Starburst referencing these new temporary data
[14:01] files and then Starburst will just read those data files and return the results to the user. And so this is great because Unity catalog remains the single source of governance for all of that data. Uh there is a unified audit trail
[14:14] so we know exactly why data changes in a certain way. And finally um we don't need those redundant policies anymore. um we don't need that extra governance layer to be able to duplicate whatever policies already existed in Unity
[14:27] catalog. Cool. So we can move on to a brief demo. Um apologies if this is a bit small. I'll explain it. Um so here we just have a test table with some PII. You know we
[14:39] have first name, last name, email, phone number, address, SSN. We would probably want to mask SSN. And there's 10 rows here. So if we were to query this with Starburst, we would just get all that data directly. And this is using client
[14:52] side scan planning. So there's no server side scan planning happening yet. And uh the important part is if we were to continue using client side scan planning after we apply the policies we're about to apply, this is the data we would get
[15:05] back. There would be no change. Um so back in data bricks we can add some policies. We can say we want to mask SSN. So now it's just a string of X's. And we can also filter addresses. So we
[15:18] filtered out anyone from Washington or California. Uh so there are seven rows now and all of the SSNs are masked as we would expect them to be in data bricks but now that is also reflected on the starburst side. So here you can see the
[15:30] SSNs are masked. You can see the rows are also filtered which is great. Um we are getting that serverside scan planning and we're able to read the sanitized data. And I want to stress that the data is sanitized by the time
[15:42] it hits Starburst. There's no additional masking or filtering that's happening in Starburst. All of this is already filtered by the time it reads it. We can also apply some more complex um filters
[15:54] or masks. So you can use govern tags that data bricks offers uh to apply to specific columns or to specific users. Uh here we mask phone number as well as a string of Y's. And over on starburst
[16:07] that is successfully masked as well. So mission accomplished. We are respecting those policies with uh no like bespoke logic on the starburst side. There was no additional masking or filtering that had to happen in the Starburst query
[16:20] engine. We were respecting what Unity uh was putting forward. So if you're interested to learn more, Starburst does have a booth 6:14 starting tomorrow at the expo. Uh I will be there from 10:30
[16:32] to 2 on Tuesday and Wednesday to do live demos of serverside scan planning. Uh Starburst also has a theater session. Our co-founder Matt Fuller is going to be talking about unlocking enterprise AI. And then finally, we have some resources for you if you are curious.
[16:45] Um, a couple blog posts. The first one from Starburst, the second one from Data Bricks, and the third one is documentation for uh Starburst Enterprise Platform 481. Um, still a
[16:57] little bit of a work in progress, but you can see an early version of it there.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.