Unlocking Cross-Engine Governance With Scan APIs

Summary

This video explains how Scan APIs address a critical gap in open lakehouse governance, where external query engines previously bypassed Unity Catalog's access controls by reading Delta files directly from cloud storage without server-side policy enforcement.
The Scan API architecture makes Unity Catalog the authoritative arbiter of file access: when an external engine queries a table, Unity Catalog evaluates RBAC and ABAC policies server-side and returns only the file paths the engine is authorized to read.
A single governance layer through Scan APIs eliminates the need to duplicate or synchronize access policies across multiple systems, and centralizes audit logging so organizations can track data access consistently across all engines.

Unlocking Cross-Engine Governance With Scan APIs

While companies have embraced the data lakehouse architecture and benefits it provides, maintaining data governance has become a much more complicated story. Many companies today implement cumbersome workarounds, such as processing and maintaining multiple copies of datasets, duplicating access policies in different systems, or worse bypassing access controls altogether. In this talk, we'll explore some of the current problems with data governance and the data lakehouse architecture and how Scan APIs can address some of those challenges.
Talk By: Alex Reid, Staff Software Engineer, Databricks, Inc. ;

Chapters

FAQs

What governance problem do Scan APIs solve in the open lakehouse?

In an open lakehouse environment, multiple query engines — such as Databricks Runtime, EMR, Starburst, and Snowflake — can read Delta and Iceberg files directly from cloud storage, which can bypass Unity Catalog's governance controls. Scan APIs solve this by making Unity Catalog the decision point for which files an engine is allowed to read, so governance is enforced server-side rather than relying on each engine to respect access policies.

How does the Unity Catalog Scan API work?

When a query engine requests data from a table, it calls the Unity Catalog Scan API instead of scanning cloud storage directly. Unity Catalog evaluates the engine's credentials against RBAC and ABAC policies server-side, then returns only the filtered set of file paths the engine is authorized to access — so the engine never sees the files it is not permitted to read.

Why is a single governance layer through Scan APIs beneficial?

Before Scan APIs, organizations had to maintain and synchronize separate access policies in each query engine, leading to inconsistencies and fragmented audit logs. With Scan APIs, all policy enforcement happens in Unity Catalog, so there is no need to duplicate policies or reconcile audit trails — every engine gets consistent governance from one authoritative source.

What is demonstrated in the Scan API live demo?

The video includes a production demo showing the Scan API in action within the Databricks live environment, illustrating how an external engine makes a scan request and receives a filtered catalog API response containing only the permitted file paths. The demo shows ABAC and RBAC policies being applied server-side before results are returned to the engine.

Full transcript

[00:07] First of all, thank everyone for joining my session today. Uh I'm going to be talking about uh unlocking cross engine governance with scan APIs or basically trying to take another step um towards kind of completing that data governance
[00:22] um you know centralized data governance story. Um we're looking statements. complete your surveys. Uh, a little bit about myself. I'm Alex Reid. I'm a staff software
[00:34] engineer here at Data Bricks. I work on the, uh, open lakehouse platform and UC manage tables. Um, so, um, a little bit about me. You can talk to me about iceberg and iceberg rest catalog APIs,
[00:47] uh, UC manage tables, uh, sports, basketball, football, whatever. All right. Uh before we get into kind of the problem and the solution, I kind of
[00:59] just want to give a little bit of background of kind of like how we got here. Um so before this whole open lakehouse uh kind of infrastructure, uh we were basically um had a lot of
[01:13] issues um with how the data was kind of stuck in these kind of data silos. Um basically meaning that like data was locked into these platforms. Um, and if you wanted to kind of use different
[01:26] tools, you had to replicate the data into many different systems, which was a lot of work, very costly. Um, also if you wanted governance uh on on top of these data sets u in different
[01:39] systems, you kind of had to have policies in in two different places, multiple different places. Uh, it got fragmented. There's inconsistencies between them. Uh and then also just auditing, you
[01:52] know, around like how people are kind of using your data, who's accessing this data was also just very decentralized. You had audit logs over in this system, audit logs over in this system. Um and it just wasn't a very cohesive uh
[02:06] efficient story. Um and then open table formats came about um and this kind of concept of the open lakehouse and kind of broke down those data silos
[02:18] could start mixing and matching tools. So different o like engines like data bricks runtime your oss engines uh EMR starburst uh snowflake all talking to
[02:31] one centralized catalog um that was basically doing the governance for you and then basically vending a credential back to these engines in order to kind of read those data files or read those tables um which made this a lot a better
[02:44] story um basically kind of solve some of those data silo problems Um and so how that looks like kind of end to end in this flow is basically a
[02:56] user uses like a query engine and makes a query towards some table uh in unity catalog and this engine basically uses UC or um iceberg rest APIs to kind of
[03:09] interact with the catalog and kind of say like yeah give me this table load me this table metadata. Uh and Unity takes care of checking basically the user uh privileges, the access uh any access controls like is
[03:22] this user allowed to even like read this uh table and if so it basically returns the table metadata to back to the client back to the user engine and is like also gives
[03:35] it a storage credential. Um and this storage credential the query engine uses to basically read the files out of basically cloud storage. Pretty simple. Um it's all based around vending cloud credentials
[03:49] and then obviously the query engine returns the results to the user. But what about tables with fine grain access controls? So this is stuff like uh I have a column mask uh on a certain
[04:01] column of my table. So I have PII data uh sensitive data or maybe I have row filters that only certain users can uh I want them to see. Okay. How how do we
[04:13] how do we kind of govern that? Right? If we go back to this previous example of how we kind of, you know, did stuff with vended credentials,
[04:25] we see that we can't safely allow access to these tables that have FGAC applied using that same method, right? Uh because basically we're giving a storage credential back to the user, right? uh
[04:40] but that storage credential has access to the underlying table files right it's sitting there in cloud storage and all that is all the raw data with all the PII data in it right and so we can't trust that this query engine is going to
[04:53] be applying the same policies uh and filtering the data so that you know users can't see it even if they were trying to we we're basically giving them a key to access PII data um that you
[05:05] know obviously can't trust right um So basically that story isn't complete yet, right? So we have any FGAC applied tables. I have my regular tables. I can't query my FGA tables. I can only
[05:18] query my you know non-FGAT tables, nonPI tables, you know, in an external engine, right? So what options do we have for these offg tables? Typically what I've seen in
[05:32] in the industry is a couple of different options. The first one being probably the worst uh which is like okay well we'll relax some of these policies or we'll just open up access to everyone
[05:44] right or uh very crude here like this user this like whole group of users can access this and this whole group can't uh which is not you know these are just
[05:56] like security escalations waiting to happen because you're you're you're not really applying any security at this point it's kind of defeats the purpose of using FKA and most of the time this is basically a non-starter for any sort of security compliance. Um the other
[06:09] option is well let's try and duplicate or sync policies from like one catalog system into the other. Um but that's a very complicated and burdensome thing to
[06:22] do. Uh you know sometimes your syncs fail, syncs are you know uh not always up to date. Policies are fragmented. Things could drift. what if somebody updates, you know, a policy in this system that you're like syncing to and then things get out of whack. Um, the
[06:36] auditing is still decentralized because you're kind of still using two different systems here. Um, the next one is like duplicating the data. So, okay, let's I have, you know, a PII FGAC table. I'm
[06:49] going to basically do some ETL out of it. Uh, do all the filtering and then store it into another table that's basically filtered and clean. Um, but that's complex ETL, right? Uh, there's
[07:02] higher storage costs because I have to duplicate the data. Um, there's issues, right? What if if something happens in the ETL pipeline, oh no, now I have to kind of restate data. Uh, that just
[07:14] becomes just basically an operational nightmare. Um, but like all those old infome infomercials used to say, there's got to be a better way.
[07:30] Uh and the solution um that we've come up with here is basically adding scan API support to Unity's catalog APIs. Now you're asking what what is a scan API? So
[07:43] backing up a little bit, um basically the iceberg rest um the icebergs has a rest spec uh API um that allows basically the engines to
[07:55] kind of talk with the catalog and most of the time this is like to do you know load the table metadata to update a table to create a table. Um there's this nice open standard spec um where you
[08:09] know there's that's what controls the dialogue and the communication between the external engine and unity catalog. A while back in the iceberg respect they added some more APIs called the scan
[08:22] plan APIs. Um and what these are used for is basically to take away or decouple from the external engine um the responsibility of
[08:37] basically planning out the query plan um from the external engine. So if we think about spark if I can go back a little bit here uh
[08:52] I'll just use this um in the query engine for number two, it basically says, "Okay, load me this table. Give me back the table credential." Uh and Spark is like, "Okay, I'm going to read the table metadata. I'm going to go figure out
[09:05] from the query and from the filtering um what files do I need to read basically um and then kind of distribute that down into the workers. Now with the scan APIs basically
[09:20] this is now saying telling Spark just to like hey catalog you tell me what files I need to read to kind of surface to be to be able to uh satisfy this query or execute this query. Um so basically the
[09:35] engine passes along you know the projections what columns I want to select uh what filters I want to apply uh what limits I want to apply basically all the different kind of things the description about what the query is and
[09:48] the catalog's responsibility is to now then go and read that metadata and kind of figure out what files I need to return back to the engine uh to satisfy that query. Now we can use this
[10:03] ah here. Yeah, see it offloads the planning. Uh basically we're catalog is controlling the visibility into like what the the the client's going to see basically what files it needs to read uh and can basically then return a scope
[10:17] credential to only read certain types of files. Okay. So with that same diagram, we'll kind of
[10:29] switch it up here a little bit. So when the client basically executes a query or like issues a query to the external engine, the external engine then says, "Oh, uh, I see that this is an FGAC
[10:41] table. Okay, I'm going to use the scan API and call into the Unity catalog scan API." UC checks the FGAAC policies u and also then goes down and says okay
[10:55] uh based upon this user's policies and fgak policies I'm going to create a temporary view or a temporary files that are filtered to basically what the user
[11:08] is asking for. So basically, Unity catalog is doing a lot of the work here on kind of applying the column mass, applying the row filters to create a sanitiz data set or data files for the
[11:20] for um the query. And in turn, it returns the list of those data files back to the engine and says here engine, here's the list of files you need to read. Here's a scope credential to only be able to read those files. So
[11:33] basically we've um segregated this out here so that like the query engine no longer has access to the raw table files that has all the PII data but instead has access only to this
[11:46] filtered sanitized data files. This is a a little bit of a breakdown of kind of how it works um more kind of in the query engine level. Um in the plan
[12:01] files um and then you can see here when you get down into the catalog API, right? It's like the scan API, it applies the FGAC and makes temporary files. It creates these scan tasks which is a list of files uh and gives it back
[12:14] to the query engine. Um and you can see the temporary files. So a little bit of different view kind of flowchart uh of what the last one um kind of showed as well.
[12:31] Now let's talk about this in kind of a practical sense. Uh give a kind of a simplified example. Um let's say I have a data set uh a table called hotels that has hotel name,
[12:44] city, rate uh and guest email. And guest email is the PII sensitive data. I don't want uh users to be able to see the underlying data here, the underlying
[12:56] email. Um so basically this is now this hotels table has this fgak applied this column mask on guest email that says null out this data from for for users.
[13:09] Um so the user queries this FGAC table through an engine let's say Starburst and Starburst calls the scan plan API and the Unity catalog again Unity catalog checks the user permissions on
[13:21] the table then based on what the engine passed like the columns it wanted to project or select uh and also the filters it basically applies that kind
[13:34] of select query and the filters onto the underlying data and creates kind of temporary files, filtered files, as you can see here on the one on the left where the guest email column has been
[13:47] nullled out and basically returns a list of files pointing to that data uh and a credential u that only is able to access that left-sided data set making sure that
[14:00] your PII sensitive data is no longer accessible um by these users.
[14:14] So, Unity catalog remains the single source of governance for data. Um, you're able to issue queries to non-FGAC tables and FGAC tables, do joins, whatever. Uh, all in one system.
[14:29] So we have a unified audit data trail for these external engine queries uh provided by Unity catalog because all the all the queries and everything is going through Unity and we no longer have to kind of push
[14:43] policies from one system into another, sync them, keep them in sync. Uh it's no longer necessary for this to work in Starburst, for this to work in Trino, for this to work in Spark. Um it all works just through Unity Catalog.
[15:04] Um so a little bit UC cross engine Aback also what we're calling this uh it's currently in beta. Uh we support both UC manage delta and iceberg tables. Um and so the current engine support we have is
[15:16] with Apache Spark and Delta Spark 4.1. Uh is also with Apache Spark and Iceberg Spark 1.11 recently uh released. um also with Starburst uh integration and then
[15:28] also more to come. So duct DB osso we're hoping to have a lot more engines um be able to connect into Unity catalog using the scan APIs. Um so that again uh you can run any sort of queries against all
[15:42] your tables all going through Unity catalog. Cool. Uh if you want to know more um I suggest um doing maybe a Google search with data bricks and cross engine aback you'll
[15:55] find a lot of information a lot more um talk about this and then I I believe Starburst has a demo booth here um I think it's open tomorrow from 10:30 to 2 p.m. So you can see a live demo of this
[16:07] uh working in action against in our prod environment stuff like that. So um thank you all for coming. That's all I have for today. I am available right after this uh if there are any questions or
[16:21] you want to talk about anything. Uh it was very quick to kind of go through all of that. I skipped over a lot of technical details. Um but if anybody wants to come and chat with me about it, I'm happy to. Thank you for coming.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.