Unify your global data and AI estate across clouds with Global UC

Summary

Global Unity Catalog introduces a single control plane that enables consistent governance, fine-grained access control, and end-to-end observability across AWS, Azure, and GCP from a unified organizational layer.
Users can discover, request, and access data across regions in seconds, while AI workloads can run wherever compute is available without requiring data movement or manual cross-cloud stitching.
Cloudless compute with intelligent routing optimizes performance and cost automatically, and the upcoming multi-cloud account architecture resolves the two-account challenge faced by multinational enterprises.

Unify your global data and AI estate across clouds with Global UC

Modern data estates are fragmented across clouds and regions, making it difficult to govern, access, and use data and AI consistently. Traditional approaches rely on ETL, duplicated policies, and siloed infrastructure that slow teams down and increase risk. In this video, we’ll introduce Global Unity Catalog, a new approach to unifying your entire data and AI estate across clouds. See how a single control plane enables consistent governance, fine-grained access control, and end-to-end observability across AWS, Azure, and GCP. We’ll demonstrate how users can discover, request, and access data across regions in seconds, and how teams can build and run AI workloads wherever compute is available—without moving data. You’ll also get a look at cloudless compute and intelligent routing that optimize performance and cost behind the scenes. Walk away with a new model for eliminating silos, reducing complexity, and enabling true multi-cloud AI at scale.
Talk By: Akash Shah, Senior Director, Product Management, Databricks ; Zeashan Pappa, Staff Product Manager, Databricks ;

Chapters

FAQs

What is Global Unity Catalog?

Global Unity Catalog is a new Databricks capability that unifies an organization's entire data and AI estate across clouds and regions under a single control plane with consistent governance policies. It addresses the discovery, governance, and control plane fragmentation problems that arise when data is spread across multiple accounts and cloud environments.

What problems does Global Unity Catalog solve for multinational enterprises?

Multinational enterprises currently face a two-account challenge where separate Databricks accounts per cloud lead to duplicated governance policies, configuration gaps, and compliance risks. Global UC's organization layer and multi-cloud account architecture enable cross-cloud, cross-region access with unified policy enforcement from a single interface.

Can users access data across regions without moving it?

Yes — Global UC enables cross-region data access by routing queries to the appropriate local or global metastore, avoiding costly cross-region data materialization. Users can discover and access assets across the entire estate in seconds without manual data movement, ETL jobs, or separate Delta Sharing configurations.

How does Global Unity Catalog help AI workloads that span multiple clouds?

With Global UC, AI teams can build and run workloads wherever compute is available — including access to the latest models on any cloud — without being locked into a single environment. Cloudless compute with intelligent routing handles performance and cost optimization automatically, making multi-cloud AI infrastructure a platform concern rather than a team concern.

Full transcript

[00:08] Everyone, uh we got uh some folks trickling in, but uh let's get started. Uh my name's Akash, I'm part of the product team. Um the Databricks DRI for our multi-cloud efforts. >> I'm Zeeshan Papa. Um I'm also part of the product team uh on the UC side for
[00:21] uh what we call global UC. >> All right. Um so uh for those of you in the keynote, you have heard some allusions and references to some of the stuff we're going to talk about. Um but starting out, uh your data is
[00:34] increasingly increasingly becoming global. We're seeing this as a pattern with our customers and hearing use cases that span everything from the basic use case of your employees are globally distributed, uh you just need to go look
[00:47] at the data with them to, you know, you've got to interconnect multiple environments. So, as a base case, multi-region is uh basically table stakes for our leading customers. Now, as we've seen a couple of things
[01:00] evolve, number one, reliability gaps from CSP providers, as well as the growth of AI, uh multi-cloud, we're also seeing becoming uh an increasingly important part of
[01:12] customer landscape. So, you know, you want access to latest and greatest models, you don't want to be boxed out, and you want to be delivering uh resiliency to your customers. Um so, this creates a bunch of problems
[01:25] for you guys. Uh number one, discovery. So, your data scientists, data engineers throughout the organization, difficult to find the data and even access it or use it, furthermore.
[01:37] Uh governance becomes challenging. So, right now you have multiple accounts that you need to create for across clouds. That leads to a bunch of not just inefficiencies, but potential gaps from configuration permissions,
[01:51] um and that leads to compliance and security risks. And finally, from a control plane standpoint, you don't have a way to interconnect these assets together. You have uh one data set in one cloud, you've got another in another, and there's no way
[02:04] to actually make these work together holistically unless you're going to manually stitch together a solution with Delta Sharing and moving data around and spending a lot of money for it. Um so, we're here to talk a little bit
[02:16] about Northstar, and we'll dive a little bit into some of the technical details, and we've got a demo for you guys coming up later. We can walk through some of this working. Um what's our Northstar? We want one unified Databricks across
[02:28] your entire data and AI estate. It should feel like one platform for everyone using it. And your data assets should be connected for your practitioners. Um the management should be unified
[02:40] across all these infrastructure boundaries. And it should be secure with unified governance and permissions everywhere that you're not worried about uh inconsistencies and gaps. So,
[02:52] uh what does this mean? It's basically creating a simple a simple experience everywhere with global discovery, seamless infrastructure across clouds and regions, and creating single sources of truth for our governance and access
[03:06] management. Thanks. So, current state. Unity Catalog is a fantastic product. Sure you guys love it. But, it has some
[03:19] gaps in the sense that uh every single deployment of uh a workspace is tied to a particular region. Every region has a metastore. That metastore becomes a policy
[03:31] boundary. Um it becomes an addressability boundary. Um those metastores have three-level namespaces. If I want to connect data across regions, whether that's like two regions in the same cloud, or let's say a region in AWS and a region in Azure, I have to
[03:44] use Delta Sharing in order to connect them. When I do that, I lose policies. I lose metadata. I I audit. And that becomes a very, very like difficult challenge for you as a as data platform owners and administrators and
[03:57] governance stewards to now replicate policies across multiple regions, figure out like, hey, user in East is going to call this thing XYZ and a user in West is going to call this thing ABC. Like, you know, giving giving your your end
[04:09] customers like a unified way to be able to discover things is challenging in our product today. >>
[04:21] same thing works for the accounts themselves, right? And if I was to look at Databricks in let's say Azure and Azure Databricks versus Databricks in AWS, I've got two different physical accounts. Um and that means that I have disjointed management of those accounts.
[04:34] Um it's essentially siloed. There's disparate infrastructure um as as Kash mentioned, right? Um and also all of the other fundamental primitives that make up the accounts, the things like identity, configuration, uh access,
[04:46] governance, audit, etc. All of those are bound within an account. That's current state today. So, where do we go in the future? So, we have this uh beautiful diagram that we've worked up um that explains what we're building right now. Um so,
[05:00] in the long run, what we're where we're going is each individual Databricks account can spin up resources in any cloud. So, I could have a Databricks account today that's say a GCP account. I could upgrade it to a multi-cloud account and I'll be able to
[05:12] spin up resources, workspaces, and metastores in different uh in different clouds. Um that's great for the customers who hey, I just want to be able to interoperate across two different clouds and I don't need uh you
[05:24] know, any of the delegated administrative capabilities that come with having separate accounts. But for customers who actually want to have separate accounts, like say you're a big multinational, you've got different legal entities, uh and you you know, or different business units and business
[05:36] unit A and business unit B are different cost centers and they don't roll up to the same uh you know, SVP and therefore, you know, they manage Databricks independently. Very common thing we hear at a lot of our large enterprise customers. How does
[05:48] that work in in in Databricks today? Well, in Databricks today you just have two different accounts. You don't have an organization that sits on top of it. One of the things that we're building is an organization that provides a layer of unified identity and a layer of unified management. And with all these
[06:01] primitives in this in this essentially this new substrate, we now have the ability to provide seamless cross account, cross cloud, and cross region access across your entire Databricks estate. And what that means net is that policy
[06:13] is respected wherever your data is accessed. Essentially, I set a policy on an object anywhere in the product and wherever I use that data anywhere else in the product, whether it's in GCP or Azure or AWS or in region A, region B,
[06:25] region C, that exact same policy is enforced. So, how does that actually work, right? So, right now metastores are a three-level namespace. Um and I I you know, if I want to query
[06:37] across metastores, well, logically, we add a fourth-level namespace. And that's actually exactly what we're doing. Um so, now I can directly address the metastores as an addressable entity. Um they become containers essentially, right? They represent uh your
[06:51] cloud regionalized data. Um and those names of those metastores are also unique identifiers across your organization. So, I can name my metastore sort of whatever I want. Um you know, it comes with some cloud region specific names by default out of the box. Um and then I can use that name
[07:06] as part of a four-level namespace query to be able to address anything in my uh Databricks estate. Again, ensuring that policy metadata is defined in one place and simply enforced everywhere.
[07:21] What that also means if you think about it from a UX perspective, and I'll go into this in the demo in a little bit, right? Is that you can actually query across clouds and regions just like it's local data. So, you have the ability to switch UI context across regions and clouds, see all your metastores in one place, have one source of truth for
[07:34] policies, metadata, and lineage, and also directly address all of that data in SQL along with uh intelligent data replication and caching, which we'll talk about in a little bit.
[07:47] >> So, as we've been alluding to, we're we're unifying these infrastructure boundaries under a single account. So, the advantage of this is you have once you're able to create these workspaces, you have your identity, your groups,
[08:00] your service principles, everything is configured in one place. And so, you don't have to go reconfigure across clouds and set it up differently. This allows us to do additional things. So,
[08:12] the cross-cloud lake base that you guys saw in the keynote a little bit earlier, this is the backbone that powers that. We'll be introducing additional features that leverage this multi this account
[08:24] capability to span multiple clouds and power new use cases for Databricks. You may be also wondering, well, for those of you who are currently multi-cloud, you may be wondering what happens with your existing accounts and how we're going to help you with that.
[08:38] So, both for that use case as well as for the use cases where you need to still maintain intentional separation in your organization for BUs or different acquisitions that you may have, you can now you'll be able to unify your
[08:50] accounts into a single organization layer that sits above all accounts. So, administrators at the account level, their experience largely remains the same, but if let's say you need to unify your commit and your contracts and have
[09:03] that roll up to one place, so your finance org that is shared across your business units can see, manage, and handle the payment, you'll be able to do that at the organization level and be able to bring that together.
[09:15] Additional features we're coming out with are on governance, monitoring, be able to roll up usage to the organization level. And another thing we're we are working on that is coming is
[09:27] improved roles at the for administrators at the account and organization level. We've heard loud and clear that our roles are too coarse coarse-grained. So, we're also working that to bring with these account and organization concepts to
[09:39] really give you the flexibility and control that you need to empower people in organization without blocking on single points of contact. Um in terms of usability for the organization, uh these are small things
[09:51] but they add up in terms of the experience for your broader data scientists and people out there using Databricks. The single company domain, uh this is available today. Um you branded URL, memorable URLs, but unified
[10:04] login. So, a problem that we've heard from customers is when you have multiple accounts, right now you need to log in to each of them separately and play tricks to be able to interop with them. You'll be able to log in once and configure all your cloud resources in
[10:17] one place with a single uh path and single journey there. Uh and then global discovery. So, again, for the people who actually need to make use of your data and AI estate, uh for them it's seamless discovery across
[10:31] everything they have provisioned access to and they'll be able to see it at their fingertips and then use it in line in notebooks and SQL queries, which we'll get into in a little bit. >> Awesome. So, uh we're going to run
[10:44] through a demo. So, bear with me while I find it. All right, so here it is. So, Databricks workspace. You guys are all probably relatively familiar with this. Um one of the things that I talked about earlier was the ability to switch
[10:58] contexts and see all of the metastores that are in my uh that are that are available to me now. I'll explain uh some of the details in a later follow-up slide about how the governance model actually works. But for all intents and
[11:10] purposes, right now I'm in a workspace that's in Tokyo and I can see data from a a metastore that's in Canada. Um if I click here and I look at like the web sessions table, I'm actually going and authorizing
[11:22] against this metastore. Now, again, I'll walk through in a later slide and we'll have um some time for follow-up questions on how this actually works under the covers. I also brought some of our uh engineering team here uh for those of you that that want to ask like really detailed questions. But uh long story short, like if I
[11:35] clicked on the permissions tab here, these are the actual permissions that I have on this object that come from the other metastore. So, even though this metastore is not connected to this workspace, I'm able to browse all of the assets in here with my the actual
[11:47] policies on those objects being respected in this experience right now. >>
[12:00] example, a customer data table, I could even request permissions. Do you have to give me access? Do you have your phone out? >> I Uh >> Can you just you're going to get a notification. Hang on. All right.
[12:13] Should have it. Wait a second. All right, there we go. Cool. So, now I just requested access to an object that lives in a different region and I got and I'm now able to see it and I can click on the sample data tab. And
[12:27] presuming that my works my compute was turned off, which I hope it is, I should be able to see it. All right, cool. Great. So, pretty awesome, pretty great example, I think. I'm going to flip over real quick to a demo notebook that I have that
[12:41] actually joins two tables and I'll take a look at this query in detail. So, I'm actually just querying and I'm trying to basically look at store transactions which are stored in Canada. Sorry, web web orders are just stored in
[12:53] Canada with store transactions which are stored in Tokyo. I've got a common common field, a customer ID, and I want to run that query. I can run it. And long story short, like I will get
[13:05] the results. What you'll see here as the results are loading is that I'm actually querying with the metastore name in the in the sequel here, right? So, Global UC Canada was the name of this particular metastore. And since I'm querying from Tokyo, I
[13:18] just address it as Global UC Canada and then I'm able to access the objects that are in that particular metastore. So, there you There you 11 seconds to run 5,000 rows to combine all this data. Pretty cool.
[13:31] Um what I'll show you next is an account console experience that we have for governance hub. So, for those of you that might might have seen some of the the governance hub previews that we've had, we have more of this stuff coming in the very very near
[13:44] future. But, long story short, the you know, the account level experience for governance hub is aiming to provide a very similar um you know, cross metastore, cross account, cross cloud view of all of your
[13:56] assets. So, you'll be able to switch metastores and look into individual metastores from a single pane of glass without having to go log into disparate workspaces in order to get a an understanding of, you know, the assets that you have under management.
[14:10] In addition to that, from a workspace perspective in the actual account console, you can see like the the you know, different workspaces that we have across different clouds, right? So, like this is the the workspace that we were looking at
[14:22] earlier, which is in Tokyo, and then there's another one that we have running in Canada. Um it's in Azure. >>
[14:36] try to explain what we actually just saw. So, for those of you that have been like coming to Databricks events in the past, like you've probably seen the life of a query slide at some point in your life.
[14:49] We use this like life of a query to explain exactly how UC work. In fact, I think I worked on the original version of this slide 5 years ago. Where we wanted the customers to be able to understand what UC was actually offering
[15:02] and why why it was so unique. But, we're upgrading that for the purposes of explaining how four-level namespace queries work. >>
[15:14] Tokyo.pos.transactions, which is my local my local catalog. And then insert that into azurecanada.main.salestransactions. That query, when I run it, I'm running
[15:26] it from Tokyo. The local assets, like the things that are in my local metastore, go to my local metastore. So, any any uh requests for uh you know, authorization or signing of URLs to be able to go and
[15:39] get the data files uh that that represent that particular table will happen locally for the local assets. But, all of the things that are remote, so for example, in Canada, will actually go directly to that Canada metastore
[15:52] across lines, right? Across across the world, authorize locally at source, give those authorization tokens back to the workspace or the compute that's that it initiated it, and then allow it to go directly to storage. And the key there
[16:05] is that it's the exact same access pattern that works for UC. It just now works across metastores. Um, and again, that boundary, whether it's in in your account or across accounts or >>
[16:19] From the perspective this works. That's that's the beauty of the interoperability. >> I actually do want to just like emphasize that point, which we forgot to explain earlier. We mentioned the organizations layer, we're going to link your accounts. Global UC here is going
[16:31] to work across those accounts, and you'll be able to interop. So, not just within an account boundary, but across the full organization. >> So, you might be wondering, okay, this is this sounds great in theory, but like
[16:44] data is siloed for sometimes very good reasons, right? Um, and we recognize that. And because of that, there is intentional friction uh for you to be able to access this data across those infrastructure boundaries in the first
[16:56] place. We call it a two-part control. Um, and the two-part control exists so that you are intentionally granting access only when required. Uh, and it works two different ways. One, there's a grant that is provided to a uh you know,
[17:08] the an end user or group or service principal, um, and that is provided on the metastore itself. So, if you start to to to reason about the The as a container, just like the catalog is a container for schemas. The metastore is now a container for catalogs. And in
[17:22] order to access a metastore from a remote context, you must have access to do so. So, the metastore admin is responsible for assigning a specific privilege called use metastore remotely to users or groups such that those users
[17:34] and groups can now access that metastore from remote context. In addition to that, there's also a boundary control. So, you guys are probably familiar with workspace binding, which is a feature that we have that allows you to take a particular catalog and assign it to
[17:46] specific workspaces. By default, the week the workspaces that you can see or that you can assign to are the workspaces that are assigned to that metastore. We're expanding the aperture there to allow you to assign catalogs to
[17:59] workspaces across the entire organization. So, whether those are workspaces in different accounts, workspaces in different clouds, workspaces in different regions, you'll be able to intentionally bind and explicitly bind those to those to those workspaces, allowing then users
[18:13] of those workspaces who also have this right and this grant to be able to access that data. So, looking forward, I know I talked about this and touched on it like in a flyover in a slide earlier, but we're
[18:25] also offering something very cool here, which is transparent caching. So, the goal here is to get support global access with local efficiency and unified control, but actually do it in a cost-efficient manner because egress fees are really expensive. I think it's
[18:37] something like it costs like I don't know, five times less to store a gigabyte than it does to transfer one time for most clouds and like that's it's pretty expensive, right? So, for those of you that are already working with Delta Sharing today, a lot of you that
[18:49] I've spoken to in the past couple years have told me that you actually materialize a lot of your data and you do it on a regular basis almost just in order to not hit those egress costs on an ongoing manner. So, we recognize that and we think that
[19:01] there's a cool product way that we can solve this. So, essentially the goal here is to reduce your cloud egress costs, accelerate the performance of local queries, and also ensure that hey, when this data is uh cached in a local region, that all of the governance and
[19:14] compliance rules that I've set on it are continued to be followed. And how it works is sort of uh sort of simple, sort of complicated in the sense that um there's a diagram that I have to explain it, but essentially I I have a DBR worker, right? And that DBR worker
[19:27] has a special service in it called an async cache service. Um every time you try to read some remote data, um it's going to go and check the cache to see, hey, does this remote file exist in my local cache? And if it does, it'll go grab it and read it directly from the
[19:39] local cache. If it doesn't, it'll actually spawn an async process to go and hydrate it, and depending on the time it it thinks it's going to take to actually hydrate that cache the uh the cache locally, >>
[19:52] so that it can serve the query faster, or it'll wait for the cache to be hydrated before it serves it from the local cache. Long story short, the goal is to basically reduce the number of times that it takes uh that we have to, you know, cross a regional boundary in
[20:04] order to get access to uh a piece of data. Now, you might be wondering, all right, well, how long does this data stay in local cache? Uh we are going to introduce uh TTL policies. The initial TTL policy will be something around 30 days to, you know, keep you compliant
[20:17] with things like GDPR, but eventually we want to get to a resource-specific uh model where you can specify resource-specific TTL policies on that cache data uh as well. Thank you.
[20:30] >> Um yeah, so uh the organization that we alluded to earlier, um basically the all you need to kind of do is like in order to get this delegated administration, number one, you'll upgrade the you'll get an account
[20:43] upgrade that'll give you access to these multi-cloud, multi-region capabilities. Um within that, as you're upgrading, you'll designate uh sort of a the account that you want to model the org
[20:55] off after, so things like identity and permissions. We what we've seen is typically customers have one account that has the bulk of the usage or higher percentage. Even if there's a a couple, they tend to have similar configurations.
[21:07] Uh and then you can just like add accounts when we come out with this. So you'll be able to be able to sort of create separate accounts for separation. But the hope with this multi-cloud, multi-region account is that we're
[21:20] hopefully eliminating most of the need for you to be able to need to create new accounts. So question many of you will probably have is like when can I start using this? So this is tentative. There's a lot of complexities we're still trying
[21:32] to nail. We want all this to work seamlessly for you guys. But single region in an account Sorry, single cloud across regions within account July private
[21:44] preview. And then single account across clouds we're targeting Q3 time frame. And then multiple accounts across an organization call it towards the end of this year
[21:56] early next year is what we're targeting.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.