Unity Catalog External Locations: Extending Governance to OneLake and Beyond

Summary

Unity Catalog External Locations connect Databricks to storage across S3, ADLS, GCS, and now Microsoft OneLake in public beta, applying centralized governance without copying data by attaching a credential to a storage path so all assets built on top inherit Unity Catalog policies.
Cross-cloud queries that join data across different storage locations—such as OneLake and S3—are possible through External Locations without ETL pipelines, and the setup reuses existing Fabric Storage Items and Access Connectors for OneLake.
The roadmap extends this pattern to on-premises storage such as HDFS and NFS, positioning Unity Catalog as a single governance layer across an organization's entire data estate rather than just cloud storage.

Unity Catalog External Locations: Extending Governance to OneLake and Beyond

Unity Catalog External Locations are how Databricks connects to storage. Users define a path to the storage, attach a credential, and UC governs everything built on top: managed tables, views and other assets. Currently, it supports S3, ADLS, GCS and Cloudflare R2. We are expanding this functionality to include support for Microsoft OneLake. In this video, we'll show how External Locations provide a consistent, extensible pattern for connecting Databricks to any storage platform — and walk through what it takes to create External Location for Microsoft OneLake. You'll see the architecture, the setup end-to-end, and a demo reading and writing UC-governed assets directly into OneLake storage without needing to setup any ETL pipelines. We will close the session with what’s next: extending the same model to on-premise storage, so Unity Catalog becomes a single governance layer across your data estate.
Talk By: Ljubica Vujovic Boskovic, Senior Product Manager, Databricks ;

Chapters

FAQs

What are Unity Catalog External Locations and how do they work?

External Locations connect Databricks to storage by defining a path and attaching a credential, after which Unity Catalog governs all managed tables, views, and assets built on that storage. This model is supported for S3, ADLS, GCS, Cloudflare R2, and now Microsoft OneLake, providing consistent governance without duplicating data.

What is the new Microsoft OneLake External Location and what can it do?

OneLake External Location support—currently in public beta—allows organizations to read and write Unity Catalog-governed assets directly into Microsoft OneLake storage without setting up ETL pipelines. It reuses existing Fabric Storage Items and Access Connectors, and enables cross-location queries that join OneLake data with data in other clouds such as S3.

Why is copying data to one place not the right answer for multi-cloud governance?

Copying data introduces pipeline maintenance overhead, data latency, drift between copies, and double storage costs. This video explains that the right approach is centralized governance over data that stays in place, using External Locations to provide governed access across clouds and regions without duplication.

What gaps currently exist with OneLake External Locations and what is on the roadmap?

As noted in this video, the current OneLake External Location implementation has gaps around tags and fine-grained permissions. The roadmap includes extending External Locations to on-premises storage such as HDFS and NFS, with the goal of making Unity Catalog the single governance layer across the full data estate.

Full transcript

[00:09] Hello everyone. I'm Yita. I'm product manager in UC and I'm owning the external location um area. And as we are all embracing AI, we heard multiple times today that AI has no intelligence
[00:22] problem but a context problem. But the real question is where is the context? So context is in the data. So now more than ever it is very important that we
[00:34] fundamentally enable access to the data and that is why I'm very excited about like this topic and about external location in general because external locations are the way how you actually
[00:46] are bringing the data to the unity catalog. So let's start. We all know that data is everywhere. But the real question and the real challenge is where
[00:59] is the governance and during the presentation I will try to answer this question and I will try to explain why we think that the winner will have actually centralized governance. So let's take a look at a situation
[01:13] which probably looks familiar to many of you. Most organizations do not have their data in one place. They have it scattered across all clouds and across all infrastructures. This is a typical
[01:26] customer. For example, it has a clickstream data in AWSS3. Uh ads and advertisement data in Google GCS advertising and campaign data in Asia
[01:40] and probably in multiple regions because of data residency. And then we even have certain infrastructure which is sitting on prem. This is not anomaly. This is actually the norm. Multiclouds, multiple
[01:54] regions and even some part of the on-prem architecture. This is happening through many major uh reasons through mergers and acquisitions because independent teams are making different
[02:07] decisions which infrastructure they want to choose because some data is really legacy constrained or because of the compliance it cannot move to any cloud. Uh the problem with this data and with
[02:20] this fragmented approach is what it does to the people that actually need to access the data because you never need to access one data in a separation and in isolation. You need to join this data with each other. You need to enrich
[02:33] customer with clickstream. You need to enrich product with uh ads so you can gather valuable business insight. So your access pattern probably looked like this. And the traditional answer was oh
[02:45] let's copy everything to one place. But we know the problems that copying is bringing. As soon as you copy, you are actually maintaining your pipeline. You're introducing data latency. Your data can drift. You're paying same data
[02:58] twice. We all know the playbook of the copying everything and what that brings. But the worst of all it is bringing is a fragmented governance. So once you start copying you're actually don't have one
[03:11] place where you can understand who can access what where you can see one audit and one lineage and that is where the real challenge is.
[03:23] This is the exactly problem that external locations in unity catalogs are solving. So what is the external location? The external location is definition a unified storage abstraction
[03:36] which is designed to help you govern your data and in the same time it is abstracting all the complexity from the user. What this means in a very very simple example is that we still have all
[03:50] our data all over the place. We have S3, we have Asia, we have GCS and we are able to register our external location in the Unity catalog. For
[04:03] example, ads product is pointing to the Google customer APAC is pointing to your Asure and Clickstream raw is pointing to AWS. Once you have registered your data, your
[04:16] external location becomes a securable object that you can govern with the UC. You can grant permissions on top of these securable objects and in addition to that you can create a values various
[04:30] UC assets. You can create tables, you can create volumes and many more. You're getting unified governance. You're getting one audit and one lineage. But in addition to that, what that means to
[04:44] your for your user is that you abstracted all the complexities away. Your user can write a simple SQL or Spark queries and can create tables on top of your external location. It can
[04:58] query the tables that are actually created on top of external location. It can join them with other uh tables that are created on top of other external location and it can actually solve the problem of this multiple access patterns
[05:12] in behind what is what Unity catalog is doing. Once the query is executed and once a customer execute select it checks whether user has all necessary
[05:24] permissions to access the data including the permission to access the raw files on the storage. And once this is confirmed, it grants a credential to the driver and then driver actually goes to
[05:38] the storage and reads the data. So your user doesn't need to think about cloud identity. It doesn't need to think about pots. It doesn't need to think about many things which seem very complicated.
[05:50] It just needs to execute simple SQL queries and to think about actual business insights. And we are understanding that your data
[06:02] is everywhere. That's why currently we support four major clouds for connecting with external location. We support Asia, ADLS Gen 2, AWS S3, Google GCS and
[06:17] Cloudflare R2. But this is not it. We are aware that data is everywhere and we know that the vendors are constantly expanding. Uh so the the this ecosystem
[06:30] where you can store your data is huge. So today we are actually announcing support for one lake. Uh as of yesterday uh support for one lake external
[06:42] location is in public beta and this is another cloud storage that you can connect to. So let's just briefly tell what is the one lake. uh one lake is a Microsoft fabric unified storage layer.
[06:56] It is a basically a storage that comes with every fabric workspace. What is a little bit different compared to other cloud stoages is that it cannot be provisioned on its own like you can provision ADLS gen 2 you need to
[07:09] actually have fabric workspace and with the fabric workspace you're getting one lake but as soon as this is done uh everything else is a pretty much the same once you register your external
[07:22] location for the one lake you can reuse your external location muscle the knowledge that you built for the previous external location and you as a user don't need to learn anything new as
[07:37] you can see it is the same SQL and it is the same semantics and syntax that I used in a previous example for creation external location this same translates for the one lake for example we can now
[07:52] register product analytics external location which is pointing out to one lake and then on top of that we can create a table product and we can query that table or we can join that table
[08:05] with any other table that is created on top of any other uh external location like clickstream or ads product. So that is uh actually support for a one lake
[08:17] and with that being said we will quickly jump into the demo how everything is created.
[08:30] Okay. So we are currently is it working? Uh we are currently in the fabric workspace where we will create a datab bricks managed item. Uh I will pause here just to explain what is data brick storage item.
[08:42] It is a new type of item that can be created in a fabric workspace which is specifically built for the purpose of this integration. You can think of a datab bricks storage item as an item
[08:54] where you can safely put your uh data bricks unity governed data and it is actually applying all UC governance on that. If we compare that to uh the ADLS
[09:06] gen 2, this is basically what we think of as a bucket in the ADLS gen 2. Once this is done and once our datab bricks storage item is created, we will go to
[09:18] our access connector for Asia datab bricks. For all of you who are aware of the Asia ADLS Gen 2 connection, this is pretty familiar, but for those who are not, uh this is a way to grant managed
[09:31] identity to your workspace and both in fabric and in data bricks. So what this does access connector is an asset which will contain managed identity and this
[09:44] managed identity is actually a key to your cloud storage provider. So why we need this to in order to register external location we need two things. The one is storage credential which
[09:57] actually contains this key and the second one is a path toward your storage which is actually pointing out where your data is. You can think of it as a key and a door that unlocks that
[10:10] particular door. So once we have this access connector for Asia datab bricks we will use this connector to grant uh permission to our workspace. We are
[10:22] basically saying hey this identity can access my workspace where my data brick storage item will be. Uh to do that we need to grant nonviewer role. So it's either contributor me uh member or an
[10:36] admin. Now we have data bricks storage demo item. We have our uh access conctor. So let's go to our datab bricks workspace. We will create a storage credential. And as I mentioned before,
[10:50] storage credential is a a key. We are specifying the key that unlocks your cloud storage. And uh credential can be specified by adding three parameters.
[11:02] One is credential type. In this case, this is Asia managed identity and the second one is name and the third one is the resource ID. This is the basically resource ID which we see in
[11:14] Asia portal. Once this is created, we will go to external location creation wizard and we will create external location. Now external location is a
[11:26] combination of four parameters. One is the name. The second one is a storage type. In this case, this is one lake. The third one is the URL. This is very
[11:38] important parameter. This is basically ABFS path which is pointing out to your storage and specifically to your datab bricks storage item. It contains
[11:50] workspace ID and it contains your uh data brick storage item ID. And the fourth parameter is a parameter that we created in the previous step. It is a storage credential. We are actually
[12:02] pointing out to one lake credential which is containing the keys to your storage. And once we click create, that's it. It is a basically two super simple steps that you need in order to
[12:15] register your data. Now when this is done, you can go to your datab bricks workspace and you can use these external location to create your cataloges to create schemas inside of your cataloges
[12:27] and to create additional tables as we said in the previous example. So now once this is done
[12:39] uh we I mean and this is what I'm very very excited about. We are talking about the future roadmap. Of course one of the priorities is to close the gaps that we
[12:52] have and to actually ship this to be enterprise ready feature. Currently as I mentioned uh one lake support is in public beta. uh we still have some gaps that we need to close. One of the most
[13:05] imminent gaps is that currently data brick storage item doesn't have applied what we call one leg data access roles. What this means is that at the moment
[13:18] any workspace uh principle that has contributor viewer and admin can actually access datab bricks storage item data. That's why even in our documentation at the moment we are
[13:30] advising customers to use workspace fabric workspace that has data bricks data with very small number of principles. This is a current step which we are doing but for the public preview
[13:44] we would like to close this gap. So we enable a fully governed UC experience that we have for a current external location like S3 and like uh ADLS gen two.
[13:57] The second point of this slide is a thing that I'm very very excited about and uh that I think it will come up as we go forward as as we embrace AI even
[14:09] more. uh we are thinking about extending external location to on-prem data and why I mean we have four major clouds that we support aren't we are supposed
[14:22] to migrate all to the clouds that was the story that we heard like previous decade however we notice and we hear that constantly constantly from our customers that there is a data which simply cannot be moved to onrem there is
[14:36] a subset of enterprise customers which are which are highly regulated and due to the compliances they actually cannot move the data but they don't want to limit their insights to the data they
[14:49] can only move they want to analyze it at all so they are making up some solutions which are not scalable and they would like to actually have expanded this support to on-prem uh the second I would
[15:01] say family of the customers that actually has on-prem data is a manufacturer like companies which actually produce a lot of data on prem and for their use case it really doesn't
[15:14] make sense to move it because there is a lot of data on prem and there is like certain economics of the solution we all do need to think about the cost so due to the economics we simply cannot move our uh data to the cloud and therefore
[15:28] we would like to actually have support for this data to analyze it and to bring it to the UC. So as of now and this is actually very active exploration. We're talking to as many customers as they as
[15:41] we can about their onremise state. We want to understand this and we actually want to go forward and to build external location like experience for this particular customers. The little caveat
[15:55] is that with this road map with like the future that we have uh we cannot definitely in this first step support all onrem state like there this market is a very fragmented so we narrowed the
[16:10] our research toward the S3 compatible object storage. What is S3 compatible object storage? Well, this is any storage that implements S3 compatible APIs. You heard probably about Mina,
[16:23] vast, zap like it's a very fragmented uh ecosystem but it is a good that they actually have unified astri compatible storage. So currently uh we even had a
[16:36] certain like scoping and path finding and we are actively exploring to support this in the future uh in the future road map. If you have any use case for onrem
[16:48] I would be the person to talk to. So please remember my name again. I'm Lubita. I will be very very happy to hear about your on-prem use cases. In addition to on-prem use case uh we have but it is a more
[17:01] forward-looking we also are embracing the fact that currently we are very limited to one cloud support. Although we support multiple clouds we support them in a way that you need to have your
[17:14] data bricks workspace in that particular cloud. So for example, if you have data bricks workspace in AWS and you have your storage in AWS, we can do the access. But if you have your workspace
[17:26] in Google and your uh storage in Asia, we cannot do crosscloud external locations. We are aware that multicloud is a super important and this is a forward-looking project that we hope we
[17:39] will cover in a very soon future. But if you also have any crosscloud use case, I am again the person to talk to. And these are like two major items where where we see that the external location
[17:52] will actually move. And this is everything with our joint wish to expand our data estate and all available data that we have as much as we can because we strongly believe that what is a very
[18:05] important like with the AI and especially with this context giving the context to AI is actually to gather as much as the data as we can. We all hear about so many fancy and new uh techs to
[18:19] analyze the data and to actually implement the AI. But we must not forgot what is the fundamental of AI and we must not forgot that that is actually the data that goes inside the AI. With
[18:31] that being said, I would like to thank you for your attention and to leave some time if you have any questions for me.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.