Unity Catalog Fine-Grained Access Controls on External Engines

Summary

• Attribute-based access control: Unity Catalog applies row filters and column masks based on governed tags, automatically enforcing policies on new tables.
• Consistent enforcement everywhere: The same fine-grained access controls work identically whether querying from Databricks serverless SQL or open-source Apache Spark.
• External engine integration: Open-source engines can read and write to UC managed tables with catalog-managed commits, staying fully governed and optimized.

Unity Catalog Fine-Grained Access Controls on External Engines

See how Unity Catalog enforces row filters and column masks identically, whether you're querying from Databricks or from open-source Apache Spark running on a VM outside your workspace. This demo walks through defining ABAC policies once in UC, verifying enforcement on Databricks serverless SQL, then running the same query from OSS Delta-Spark on EC2 to show identical results. Finally, we demonstrate external engines creating and writing to managed tables with catalog-managed commits, fully governed and automatically optimized.
Learn more about Catalog Commits: https://www.databricks.com/blog/convergence-open-table-formats-and-open-catalogs-catalog-commits-generally-available

FAQs

How do attribute-based access control (ABAC) policies work in Unity Catalog?

ABAC policies are applied to schema and match columns by their governed tags rather than by name, so adding a new table with the same tags automatically applies the policies.

Can open-source Apache Spark query Databricks Unity Catalog data?

Yes, open-source Apache Spark running on a VM outside your workspace can query the same Unity Catalog data with identical fine-grained access controls enforced automatically.

fine-grained access control and row-level security on Iceberg tables?

This video demonstrates Unity Catalog fine-grained access controls working with external table formats like Iceberg. Row-level security enables column and row filtering without duplicating data or maintaining separate copies. Organizations can share governed data with multiple external engines while maintaining consistent access policies.

How do open table formats let transactions and analytics read one copy of data?

This video explains how Iceberg and Delta table formats support both transactional consistency for operational workloads and analytical queries on the same data copy. Unity Catalog provides governance across both access patterns. Organizations eliminate data duplication and synchronization complexity.

How can I let engineering and business teams query the same governed data without duplicating access controls?

This video shows how the Databricks Data and AI platform with Unity Catalog enables engineering and business teams to query the same data with role-based access controls. Fine-grained policies automatically apply across both analytics and operational engines. Organizations maintain a single source of truth with centralized governance.

Full transcript

Unity Catalog was designed for the open lakehouse. Previously, data teams were stuck in silos, forced to duplicate data across platforms just to use the tools they wanted. Every new engine meant copying data sets, rebuilding access policies from scratch, and keeping everything in sync. With the UC open APIs, we broke down those silos. Enterprises can keep one copy of data, use any compute engine, and govern everything from one place.

Today, I'll show you that in action. Fine-grained access controls defined once and enforced everywhere, plus external engines that can read and write to UC managed tables. Let's start by looking at our table, Lumina Technologies finance transactions. This is a UC managed table. It uses predictive optimization and liquid clustering to automatically tune data layouts, run compaction and vacuuming, and keeps the statistics fresh.
So, openness meets performance. We've applied governed tags to key columns, such as geo equals region on the region column, and PII equals account number and PII equals amount on sensitive columns. Now, let's look at the ABAC policies. We have three. First, we have a row filter that restricts our demo principal to only see transactions where the region equals US. So, this uses the filter by region function and matches any column tagged with the geo equals region tag.
Second, we have a column mask. On account numbers, it replaces the full number with just the last four digits. This matches the columns tagged with PII equals account number. And lastly, we have a column mask on amounts. It redacts the value entirely by returning null, and this matches columns tagged with PII equals amount. The key here is that these policies are attribute-based. They're applied to the schema and match columns by their governed tags, not by name.
So, add a new table with the same tags and the policies apply automatically. Now, let's see these policies in action. First, we query the table from within Databricks using serverless SQL. This is the baseline. We expect the policies to be enforced here. And they are. Only US region rows are returned. The row filtering is working. Account numbers show as masked values. The column mask is applied. And the amounts are null.
So, the second column mask is hiding the values, too. And this is standard fine-grained access controls or FGAC enforcement on the Databricks platform. But, here's where it gets interesting. Now, we're running the exact same query, but from open-source Apache Spark running on a cloud VM completely outside of Databricks. This is a vanilla Spark session on EC2 using Delta Spark and the UC Spark connector. So, no Databricks runtime anywhere near it.
The critical configuration here is the server-side planning. dot. enabled equals true. This tells the engine to delegate scan planning to Unity Catalog's serverless filtering fleet. That's where FGAC enforcement happens. The filtering fleet applies row filters and column masks server-side, then returns vendored credentials scoped to only the filtered data files. The external engine never sees the unfiltered data. So, let's look at the results.
Identical. Same US only rows, same masked account numbers, same null amounts. The external This is the core message. So, define once, enforce everywhere. Whether you're on Databricks or on any external engine, Unity Catalog enforces the same row and column level policies. The engine does not matter. Governance follows data. So, let's go beyond reads. External engines can also create and write to UC managed tables. Every table they touch continues to be automatically optimized and centrally governed.
So, step one, we're creating a brand new managed table and we'll call it external audit log. It'll be created directly from our external Spark session on EC2. So, no Databricks notebook, no TBR, just open source Spark on a cloud VM talking to UC's open APIs. Now, we insert a few audit log rows from the external engine because every operation flows through UC with catalog managed commits, we get serialized commits that prevent log corruption, complete auditability of every read and write, and continued predictive optimization, compaction, vacuuming, and statistics even on tables written by external engines.
So, step two. All right, let's verify from Databricks. We switch back to the Databricks serverless SQL and query the same table. And there it is. The rows we just wrote from the external engine are immediately visible in Databricks. So, it's one copy of data written from any engine, governed and optimized by Unity Catalog. So, to recap what we've seen, we've got unified governance where ABAC policies defined once in Unity Catalog are enforced identically on Databricks and on external engines.
Row filters and column masks just work regardless of where the query originates. We also saw secure external access where Unity Catalog vends short-lived scoped credentials to external engines on demand. So, no broad static permissions. Your platform team retains full control over which identities can access data externally. And lastly, read and writes where external engines aren't limited to just reads. With catalog manage commits, they can create tables and write data all with fully transactional safety and automatic optimization.
This is the open lakehouse vision. One copy of data, any engine governed from one place. >>

Learn more about the Databricks Data and AI platform.