From Data Portability to Governance Portability: The Future of Cross-Platform Governance

Summary

Open table formats solved data portability across platforms, but governance metadata—tags, classifications, and policies—remains siloed inside each individual data catalog, creating compliance risk and operational overhead for multi-platform enterprises.
Unity Catalog's labels exchange mechanism enables enterprises to carry governance tags and business metadata across catalogs using Apache Iceberg REST Catalog APIs, so a PII tag applied in one catalog travels with the table to another rather than being lost at the boundary.
Cross-platform governance is built on three pillars—performant, expressive, and extensible—enabling fine-grained policy enforcement such as cross-engine ABAC to be delegated to Unity Catalog from platforms like Snowflake or Microsoft Fabric.

From Data Portability to Governance Portability: The Future of Cross-Platform Governance

Enterprises run on multiple data platforms — Unity Catalog, Snowflake, Lake Formation, BigQuery, and many others. PII may be tagged in one catalog, masked in another, and ignored in a third. Open table formats solved data portability, but governance metadata remains siloed inside each data platform. This talk explores how enterprises can unify governance across platforms using labels exchange: carrying governance tags, classifications, and business metadata across catalogs through Apache Iceberg REST Catalog APIs. We'll cover why cross-platform governance is becoming the next federated data pain point, what's emerging in the Apache Iceberg ecosystem, and how teams can operate multi-platform architectures.
Talk By: Alex Jiang, Product Manager, Databricks ; Andrei Tserakhau, Engineering Manager , Databricks ;

Chapters

FAQs

What is governance portability and why does it matter for multi-platform enterprises?

Governance portability means carrying governance tags, classifications, and business metadata across different data catalogs—not just making data accessible across platforms. Enterprises with data in Unity Catalog, Snowflake, and other systems face duplicated or inconsistent governance policies that create compliance risk and operational overhead, which is the problem this video addresses.

How does labels exchange work in Apache Iceberg?

Labels exchange uses Apache Iceberg REST Catalog APIs to transfer governance metadata—such as PII tags and classifications—between catalogs when tables are accessed across platforms. This allows a tag applied in one catalog to travel with the table to another catalog rather than being ignored or manually recreated.

What is cross-engine ABAC and how does it relate to Unity Catalog?

Cross-engine ABAC (attribute-based access control) allows fine-grained policy enforcement to be delegated to Unity Catalog even when a query originates from a different platform such as Snowflake or Microsoft Fabric. This means Unity Catalog acts as the authoritative policy engine across engines without requiring each platform to replicate the policy logic independently.

What are the three pillars of cross-platform governance described in this video?

The three pillars are performant, expressive, and extensible—designed to ensure governance metadata exchange does not degrade query performance, can represent rich policy semantics, and can be extended as new platforms or metadata types emerge. These pillars guide how governance metadata travels between Unity Catalog and systems like Snowflake or Microsoft Fabric.

Full transcript

[00:07] Hey everyone, we're going to get started right here. Uh my name is Alex. I'm a product manager at Databricks and I'm joined by my colleague Andre to talk about the future of cross-platform governance, specifically how the industry is moving from data being
[00:21] portable through open table formats to governance being portable through catalogs and flat ones. As a disclaimer, this content is for informational purposes only. Any
[00:33] information these slides do not represent promises by our company. Uh they're purely for information. And as a reminder, please complete your surveys. Uh all of the input that you'll get through the surveys will help make us
[00:45] make future data and AI summit sessions better. So, we'd really appreciate any feedback you can get on the conference. With that said, quick introduction. As I said before, I'm a product manager at
[00:57] Databricks. I work across Unity Catalog interoperability with other platforms, as well as open table formats, specifically Delta and Iceberg. And if you want to learn about what the best
[01:09] restaurants in SF are to take your team, feel free to talk to me afterwards. I have a lot of great recommendations. >> Yes, and hello. My name is Andre. I'm uh from Europe. I'm uh located in Berlin.
[01:21] I'm um engineering manager. I'm leading table format evolution team. I'm responsible for open source table formats, Iceberg and Delta. And I'm focused on the evolving both of these formats uniformly. And uh I can also
[01:34] talk about food, but mostly in Berlin, so it's not useful here. >> All right, with that said, let's begin. So, to start, I'd love to set context by talking about how the industry converged
[01:46] on data portability. So, the reality is that a lot of enterprises use more than one data platform. It's not just Databricks, it's not just Snowflake, it's often a combination of multiple platforms and
[01:59] tools. But, the issue with this is that data is often fragmented. We see silos in the industry where data is kept in one platform. In order to use that data in
[02:12] other platforms, you might have to duplicate them or duplicate the policies as well. So, this leads to several problems. So, one is that there is data drift from the same data diverging across different platforms because
[02:24] although you're ingesting from the same raw data source, your actual transformations across all the platforms may differ, which can cause data to be inconsistent. Moreover, there's also increased cost from keeping separate copies of data,
[02:38] which can, in this case, three extra cost or even more depending on the number of platforms that your enterprise uses. And finally, as I said before, there's also the issue of fragmented governance where you have specific policies that
[02:50] you might define in one catalog, but the issue is that you'd have to duplicate them across every single catalog or platform in your data estate. So, to help address the issue of
[03:02] platforms keeping data siloed, the industry converged on open table formats, specifically Delta and Iceberg. And with open table formats, the vision is very clear. You can store one copy of your data in either Delta or Iceberg and
[03:15] use any query engine to access your data or any data platform. Moreover, the industry also converged on catalog federation as a way to make data
[03:27] in one platform accessible from another platform. So, on the screen here, you have an example where you might have a table in Unity Catalog that you want to be accessed by Snowflake or Glue. What you can do is you can use open catalog
[03:40] APIs to federate that table into these external platforms such that Snowflake and Glue can access the Unity Catalog table as if the table was in the native platform. Moreover, you can also do the
[03:52] way around. So, let's say you have a table in Snowflake or Glue and you want to bring it into Unity Catalog, you can do the same. The issue here is that although you can bring tables from one platform to
[04:05] another, governance is still fragmented. A lot of the policies defined or the metadata you define remain siloed. And specifically, there's two separate problems. The first of which is for
[04:17] query engines, you can't enforce fine-grained access policies defined in the catalog. And the reason why is because for open source query engines like DuckDB, Spark, and Trino, these engines rely on what's called credential
[04:31] vending to get access to data. So, Unity Catalog, when the engine requests data, will either grant access to the table or it won't. There's no notion of granting partial access to the table. So, that's the challenge on the query engine side.
[04:44] And moreover, when you're looking at data platforms that have native governance capabilities and can enforce policies, these data platforms are often missing critical catalog data required for unified governance. So, you might
[04:56] have governance metadata defined in Unity Catalog, but when you're federating, that metadata remains in Unity Catalog. It doesn't actually get transferred to the other platforms that are you're trying to interoperate with.
[05:09] And to address the first problem when we're talking about query engines and cross-engine governance, so as I said before, the problem is that fine-grained governance is not really extend to these external engines. Um
[05:22] Unity Catalog relies on vending credentials to table, but when you have fine-grained access controls like row filters and column masks, Unity Catalog can't vend credentials because it can't trust these engines to enforce those
[05:34] policies. And to solve this problem, we introduced what's called cross-engine ABAC, which is essentially our way of delegating policy enforcement to the catalog. So, essentially, the catalog will enforce
[05:47] any row filters or column masks on the server side and then only give the external query engines access to the filtered or sanitized data. So, here's how this looks in practice. Let's say you're a user using an external query
[05:59] engine. A user might run a query on the sales table, then the external engine can request a scan plan of the sales table, which then allows Unity Catalog to check permissions on the sales table and
[06:13] sanitize the table according to any fine-grained control policies that apply to the user. Afterwards, Unity Catalog can then give the external engine access to the filtered files that are pre-sanitized
[06:25] and apply the column masks and the row filters. And then finally, the external engine can run the query on the filtered data files and return results to the user. But now, I'll hand it off to Andre to talk about how we're solving the problem
[06:37] of cross-platform governance when we're talking about platforms like Snowflake and Glue that actually have native governance capabilities. >> And with that said, >> Yes. >> So, for the cross-platform governance,
[06:49] there is a different set of the problems. On the engine side, it's uh uh fairly uh limited the impact of the of the of the governance. From the governance perspective for a catalog,
[07:01] you have a source table in your original catalog and you have target table in your federated catalog. We can try to kind of extend the scan API to do the same approach for a consumer catalog to
[07:14] just uh delegate all the decision on the source catalog and read the data from the source catalog itself. But this uh still remain uh the decision power on the source catalog and the target catalog will have no information about
[07:27] this. The The metadata will still remain siloed inside. And this is a key challenges for a for a cross-platform governance. The siloed data uh meta data in a in a source
[07:41] catalog prevent the target catalog to independently operate. Access policies also become fragmented because source source catalog and target catalog, they are
[07:53] can operate with different policies in mind, with different administrative rules, etc. etc. And also you have to you have to expenditure your your stack with external tools like
[08:06] Immuta or something to port your governance between the the two these two systems. And we try to address this problem from the first principles and build this
[08:20] solution on three key key pillar pillars. The first pillar is the cross-platform governance should be performed. This actually means that we can't simply extend a scanner API and delegate all decision to the one of the
[08:32] catalogs and because this will bring the run time dependencies from one from one platform to another. Also, this should be expressive, though the source the source platform and target platform,
[08:45] they have a rich functionality of how you can define the policies. And we don't want to restrict how this functionality is used. We don't want to limit the expressiveness of the source of the target platform. And also it
[08:57] should be extensible for all catalogs in the world, for all platforms in the world. And therefore, it's one model that can fit all the platforms in in on the market. This is our vision that
[09:09] unified governance lives across the different platforms and different catalogs. We have one source of truth with meta information that we share with different platforms. This is our vision
[09:21] of catalog federation. The governing metadata that we use, they're useful not just for fine-grained access control. By fine-grained access control is very important part, but it's also very useful for for such thing as a
[09:34] discovery to find the asset that you're looking for. For the icon text enrichment to genie context for acknowledge graphs and ontologies. These pieces, they usually build on top of a
[09:47] governance metadata. So, the main thing for us that this governance metadata become accessible not just in one platform, but across all platforms. So, your snowflake data
[09:59] So, your snowflake metadata or your glue metadata could be accessible inside of a Databricks for a genie or vice versa. So, you have a your meta information defined in one place and it's travel with your data in the different platform.
[10:11] So, how it works in the practice? We have producing catalog. Let's say on this example at the Unity catalog. On this producing catalog, we define some tags, some meta information and some
[10:24] some policies. Those policies are federated with a consuming catalog. Let's say snowflake and a or fabric. The catalog consume this meta information via iceberg rest catalog open open format of
[10:37] this iceberg rest catalog and exchange the meta information between each other. We use open standard for this and then this this meta information used inside of a
[10:49] consumer catalog to apply the decisions, apply the policies to a query that we execute. So, the decision making actually stays in a catalog itself. The catalog that
[11:02] consuming this meta information still rely on the on this meta data to make a decisions. So, we don't make a decision from a producing catalog. We transfer the meta information that allows you to make a
[11:15] decision on a consumer catalog. So, this actually enables the cross platform governance from a first principle and all of the three pillars are there. The performance part,
[11:29] the performer part Uh, and extensibility. The performance part is rich because we're not replicating everything. We're not replicating all the policies. We're not replicating all identities that you need
[11:42] to replicate from one platform to another. We just replicate the meta information essential to make a decision. Instead of thousands of entities, we just move a dozen of tags, labels, and
[11:55] meta information that travel from one catalog to another. The expressiveness is still there. You still can define a policy on your consumer platform, on a target platform, in a full version of this platform
[12:09] expressiveness. We don't enforce you to narrow down the expressive expression language. You can use the native capabilities of any platform. And extensibility is also achieved because we use a same open source
[12:23] standard. We We rely on Iceberg REST catalog here and driving the open source community to a version of this idea of sharing meta information. And this meta information may become an essential
[12:35] piece for your cross-engine governance, cross-platform governance. And on this, I would like to hand to the questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.