Cross-Cloud Delta Sharing: Eliminate Egress cost with Delta Sharing Global Distribution

Summary

Open Sharing (formerly Delta Sharing) has grown to over 300 partners and more than 100% year-over-year growth, and now includes Iceberg interoperability, shareable Genie agents, and a new Global Distribution feature launched in private preview.
Cross-cloud data sharing generates egress costs that grow with recipient query volume because the data provider pays for bytes sent across cloud regions — a scaling challenge that becomes unsustainable for large-scale providers.
Global Distribution solves the egress cost problem by acting as a managed CDN for Delta Sharing data, maintaining in-region replicas so recipients query data locally, eliminating cross-cloud egress charges while preserving freshness guarantees.

Cross-Cloud Delta Sharing: Eliminate Egress cost with Delta Sharing Global Distribution

Cross-cloud and cross-region data sharing introduces egress costs that grow with recipient query volume, creating a scaling challenge for large-scale data providers. this video covers best practices for architecting cost-efficient OpenSharing deployments across clouds and regions, and introduces a new Databricks capability designed to change how providers manage cross-cloud/region delivery at scale.
Talk By: Abhijit Chakankar, Software Engineer, Databricks ; Huey Han, Staff Product Manager, Databricks, Inc. ;

Chapters

FAQs

What is the difference between Delta Sharing and Open Sharing?

Delta Sharing is the original protocol Databricks launched five years ago for sharing structured data across organizations. Open Sharing is the evolved product that has expanded to include volume sharing, notebook and AI model sharing, Iceberg interoperability, and Genie agent sharing — making the name change reflect the broader scope of what can now be shared.

Why is cross-cloud data sharing expensive for large-scale providers?

In a zero-copy sharing model, data stays in the provider's cloud storage and recipients query it directly, which means every query from a recipient in a different cloud or region generates egress charges that the provider pays. As recipient query volume grows, these charges become unpredictable and can scale to prohibitive levels, creating a disincentive to share data widely.

How does Open Sharing Global Distribution eliminate egress costs?

Global Distribution acts as a managed CDN for Delta Sharing data by maintaining in-region replicas of shared datasets in the recipients' cloud regions. When recipients query the data, they read from the local replica rather than the provider's origin, eliminating cross-cloud egress charges while maintaining freshness through a replication architecture with defined consistency guarantees.

What Iceberg interoperability features are now available in Open Sharing?

Open Sharing now supports sharing to any Iceberg client that speaks the standard Iceberg REST Catalog (IRC) protocol, enabling organizations standardized on Iceberg to consume shared data without using Databricks. Additionally, Iceberg tables managed by external systems such as AWS Glue can be federated into Databricks and shared out through Open Sharing, bridging non-Databricks Iceberg estates into the sharing ecosystem.

Full transcript

[00:07] Hi everyone. Thank you for coming. We're going to get started. I'm We'll do a quick round of introduction. Um I'm Huey, uh product manager for open
[00:20] sharing, formerly known as Delta Sharing. And uh we have Abhijit here. >> Abhijit. I'm I'm so I'm an engineer on Delta Sharing. >> Awesome. And thank you guys for being here. And today we're going to talk
[00:33] about cross-cloud data sharing and egress and this new feature we're we're building. So, just a quick overview of what we're going to cover today. Um first, we're going to talk about uh a high-level
[00:45] overview of open sharing, uh what it is, what's new. And then we're going to dive into this new feature that we actually recently just launched in private preview called global distribution, kind of what
[00:57] problem it solved, what's the solutions, and how it works. And then we're going to wrap up with a demo on how exactly it works. And for the remaining time, we're going to open up for a Q&A.
[01:13] So, quick overview. Um we launched Delta Sharing uh 5 years ago at the Data AI Summit. And since then, um you know, Delta Sharing grew tremendously as a ecosystem. And we had 300 plus partners, um over 100%
[01:26] year-over-year growth. But over time, we also added a lot more new capabilities to Delta Sharing, such as, you know, our structure volume sharing, uh notebook AI models,
[01:39] Iceberg interoperabilities, um to a point where we think it's time for us to introduce a new projects, um open sharing, um that's evolved beyond sort of Delta Lake and the
[01:51] original sort of the structure uh table sharing. >>
[02:06] The first is Iceberg interoperabilities and a lot of our customers are, you know, a very big Iceberg customers standardized on Icebergs. So, being able to share two Iceberg clients
[02:19] and also share Iceberg tables are very important for us. So, there are sort of GA-ing or GA coming soon two features very soon for Iceberg. The first is
[02:31] being able to share to any Iceberg clients that speak IRC. And the second is be able to share for Iceberg table that's not managed by Unity Catalog, but let's say Glue for
[02:44] example, that you can federate into Databricks and be able to share using open sharing. So, that's the first pillar, Iceberg interoperability.
[02:57] The second pillar is AI. Increasingly, we see a lot of data providers and our customer want to be able to offer a agentic experience to their users. So, for that, now it's in beta now, so you guys can
[03:09] try it out. You can now share a Genie Agent, formerly known as Genie Space. And what that does is that it allows you to give a curated agentic experience on
[03:21] your data. And your recipients now get an out-of-the-box chat interface to talk to your data. Previously, you know, you need to send some documentation, teach them about the schema, they have
[03:33] to write like select star limit five, try to explore your schema. Now, they can just dive in and start asking question about their data. And last part, which is the focus of this session, is cross-cloud sharing.
[03:46] And for that, we have two features. One is called Secure Connect, which we have another session, one yesterday, and one tomorrow, if you guys are interested. The second is global distribution,
[03:58] which we'll cover today. And the first feature, what it does is that it's simplifying networking, and the second is minimizing egress and improving query performance. So, let's dive in.
[04:11] And customer right now already love open sharing. You know, we have leading AI companies like OpenAI using open sharing, but also traditional enterprises like SAP. So, let's dive into this new feature,
[04:24] global distributions. We're going to talk about what's the problem we're going to solve, what's the solutions, and sort of how it works.
[04:41] Just take a step back. How does open sharing work? One of the key benefit of open sharing is sort of zero copy, where the data doesn't need to move. And the way that it works is as a data providers, you know, you have your tables stored in
[04:55] your storage, and these can be Unity Catalog managed table, or it could be foreign tables that you federate. And as a data provider, you create a share in your Unity Catalog. You add a recipient to the share
[05:08] if they're on Databricks. They just sort of get that share. They can mount, they can query. If they're not on Databricks, they you know, you can create a credential file, and then distribute
[05:20] that credential file to them, and they can use a bearer token to create a share. As a data recipient, what happens is, you know, they mount the share on your Unity Catalog if they are um
[05:32] if they're on Databricks. If they're not on Databricks, they can use a wide range of clients like, you know, Python or Spark to to talk to the share using the bearer token in the credential file we distribute.
[05:44] At access time, what happens is that the client will first talk to the open sharing, formerly known as Delta Sharing Server, to say, "Hey, hey, here's who I am. Can I get access to that shared assets?"
[05:56] We then, on the server side, look up permissions, try to figure out if this guy actually have access. If he does, we then vend them the storage credentials for the client to then access the storage directly. So, that is sort of
[06:09] the zero copy sharing model and sort of how the zero copy works. And this sort of zero copy has a lot of benefits for our customer.
[06:21] The first is minimal cost because you only need to maintain one copy of your share. You don't need to duplicate it and, you know, upload it to a storage that's managed by a vendor that's, you
[06:33] know, The second is operational efficiency because you know only have one piece of storage, you don't need to manage multiple, you know, duplicate infrastructure and multiple different pipelines.
[06:45] And last is single source of truth. So, you have one, you know, copy of your data so that that data is always fresh and, you know, recipient always get the latest
[06:57] data. But there's also sort of certain downside of the, you know, zero copy approach. And specifically, what we hear
[07:09] from a lot of customer is when they share cross region and cross cloud, and usually we hear a few things that either is about egress or about query latency. So, for egress,
[07:23] you know, because this sort of direct storage access model, when recipient issue a query against your data, the bytes will be egressed outside of your bucket. And you, as a data provider,
[07:36] will need to pay for egress. Unfortunately, as a data provider, sometimes, you know, you're collaborating with people from a different company or a different business units, you kind of don't control what kind of query they run or
[07:49] how frequently they run their query, but you're paying the egress bill, which makes the egress charge kind of unpredictable and hard to control. >>
[08:02] their data in the US and they have international partners, let's say, you know, in India, that's halfway around the world. And when you do this sort of zero copy sharing approach where the recipient need to access the storage directly, the
[08:15] query latency can can add up for these sort of cross-region and cross-cloud. And we see a a few customer do a few workarounds today, but none of
[08:28] them really cover the three problem we talked about, sort of predictable query egress, recipient egress visibility, how much each of your recipients incurring, and last part is reduced latency. The first is that
[08:41] we see customer sort of build egress dashboard. You know, if you're on S3, you can use service access log, for example, to piece together, you know, who is incurring how much
[08:53] egress roughly. But then this approach sort of, you know, it give you per recipient egress visibility, but it doesn't really solve the problem of predictable egress and also reduce latencies cuz
[09:06] the bytes after all is still in your storage. And we see some customer use a second approach which is which is storing their data not on S3, ADLS, or GCS, which charge for egress,
[09:18] but store it on Cloudflare R2, which some of you guys might know, they have a sort of a zero egress thing, that's their key selling point. So, for that one, you know, you do get predictable query egress, which is, you
[09:30] know, nine, but you don't get those other things, which is per recipient um and also uh reduce latencies because for cross region cross call they still
[09:42] have to access storage. Last but not least, the last option we see some customer do is that they uh materialize or they actually move the bytes closer to the recipients. Uh for that one, you know, you do get
[09:54] predictable query egress um because now the queries is in regions, but um and you do get reduced latencies cuz, you know, it's in region queries, but uh per each recipient's like egress visibility
[10:07] kind of don't get that unless you build dashboards again. So, none of the workarounds today kind of really solve the the whole problem.
[10:24] Uh we just launched uh global distributions uh a few days ago. Um right now it's in private preview. And the way that you can think about this is kind of a a CDN uh for data that minimize egress and improve query performance.
[10:37] And um the way that it kind of high level works is that uh once you enable it on on the share, Databricks will automatically manage the replications, um and we move the bytes to the
[10:49] recipient region. We'll detect where the recipient is and store it in uh Databricks managed storage, where also known as Databricks default storage. And um and when recipient need to query
[11:01] the data, um we automatically route the uh the recipients to the the replica. And what this does is that one it minimize egress since uh you know, uh it's not cross region cross cloud query
[11:13] anymore, it's in region. Uh second is improve query performance for the same reason. And last part is simple automatic because it's fully managed by Databricks and it's sort of only one click enablement, which we'll do a demo
[11:25] shortly. The experience is really simple. Um you know, we're going to do a demo so that you guys have a better idea. But on a high level, um the way that it works is that, you know, when you create a share
[11:37] at the last step, if we see that you're adding recipient that's outside of your storage region, there will be a button pop up on top right to say, "Hey, do you want to enable global distributions?"
[11:50] And if you opt in, then you're kind of done. Uh we will manage uh the all the underlying infrastructure including running the replication job, moving the bytes, storing the replica, making sure
[12:02] it's fresh. And for your recipient, there's also no change in their experience. Uh on their side, we'll also uh automatically route uh the queries to the local replica.
[12:19] So, with this, what does it mean is that you sort of get the best of both worlds. Um you know, there's still that zero copy options if you want uh that's minimize your cost, uh that increase your operational efficiency, and you have the single
[12:31] source of truth. But now, you also can opt in to this uh global distribution features and that will create a replica on behalf of you in the recipient regions that minimize egress and improve query performance.
[12:49] So, when you use global distributions, um as many of you guys can tell, it does create a separate replica and which means that there will be some cost associated with it and there will be a freshness uh delay with the replica and
[13:03] the original. So, you should use global applications when, you know, the recipient query the same data pretty frequently and incurring a lot of egress. And uh maybe they also have in region uh
[13:15] sort of data processing requirement that require them to query uh the data in in regions. Um but you should skip uh global distributions if, you know, zero copy today works fine for you, uh egress is not a concern
[13:28] uh or recipient always need the freshest data. So, next we'll talk about exactly sort of under the hood, how we're building it, and how it works, and then I will pass it to Abhijit.
[13:42] >> Yeah, thank you, Louie, for the introduction. Now, we'll double click into how we make all of this work.
[13:55] All right, so So, on the left you see, you know, we have a provider who is sharing data. And as if if they enable global distribution, as soon as they start adding recipients to the share, we would start creating and provisioning
[14:09] what we call regional replicas. And these regional replicas, it's it's a whole set of infrastructure that is managed by Databricks. So, the first thing we do is we create a
[14:21] Databricks managed account which has one-to-one mapping with the provider account. So, this immediately gives you isolation. So, you know, nobody else's data can actually mix with your data. It's It's a very high account level
[14:33] isolation. Right? And within the region, we would provision a Databricks managed metastore, a workspace to orchestrate, you know, the the the data copying and data movement, and also storage, which
[14:46] is our cloud back >>
[15:00] Uh some of the properties of this this replica are that, you know, if if, for example, your asset is a a Delta table or an Iceberg table, so this replica would be version compatible with the source.
[15:12] So, what that means is that if if a recipient is querying from the source, let's say they're doing a streaming query or CDF, they would see exactly identical data uh no matter, you know, it is whether we serve it from the source or from the
[15:24] replica. So, you know, if you do a deep a simple deep clone, that doesn't give you this this this capability. And this replica is kept So, the first time you create this replica, obviously
[15:36] the entire you know, table is is needs to be egressed, but from that point on we keep it incrementally up to date. So, then your costs from that point on are incremental and proportional to the change rate that your source table
[15:50] experiences. So, then how do we keep this this this regional replica in sync with the
[16:02] changes that are happening, you know, on the on on the provider side, right? So,
[16:15] their end, we would propagate those to the all the regional replicas associated with that share. So, here are some examples. For example, if the provider adds a new table to the share, we would
[16:27] immediately start replicating it to all the regional replicas where the share has recipients. Similarly, if if the if the provider adds a new recipient,
[16:40] then, you know, it's it's slightly more complex. So, if this is the first recipient for for that for that provider, then that triggers the account creation, where we create a new Databricks account for the
[16:52] associated with the provider account. If it is the first recipient within a region, then that'll provision the regional replica. But if there are already other recipients in the region, that means the
[17:05] the replica already exists, then adding a new recipient to a share is a no-op. >>
[17:17] the provider removes the table from the share, we would eventually garbage collect and I will go into the eventually part. Uh we will eventually garbage collect all the replicas of the table uh that exist.
[17:29] Uh if if a recipient is removed from the share, then again we do the reverse thing. If if this recipient if there are already other recipients still remaining, then it's a no-op. But if this is the last recipient from that
[17:42] region, we will trigger garbage collection for all the assets in that share essentially. And again eventually. And the reason we do this eventually after some some delay is because
[17:55] the provider might be doing some maintenance on there and they might be creating new shares, moving recipients around. You don't want to eagerly delete all your assets that are replicated only to sort of re-egress them out.
[18:07] Because remember the first egress is is the most expensive one. So we give it some delay to make sure that this is really a permanent deletion or permanent removal. And then uh we will garbage collect uh the data associated with the
[18:20] replica. >>
[18:33] querying uh a table that has been replicated uh in DVR. So during query execution, uh DVR asks recipient UC to resolve the
[18:46] table path and get temporary credentials for that path, right? So that request goes to the recipient UC. And recipient UC detects that oh this is really a shared table and the replication is enabled on the
[19:00] share. So it will send two parallel requests. The first one goes to the provider UC, which is, you know, the the provider of the of the table. That's the source table. Because that's the source of truth for
[19:13] auth. We still treat that as source of truth for auth. And the auth there essentially is does the recipient still have access to the share? Does the share owner have access to the assets that were shared? So, that's always checked
[19:25] live. Which gives you, you know, the stay same strong security guarantee that exists today. But for path resolution and credential vending, we send a parallel request to
[19:37] the replica, which hosts, you know, the the table from where the recipient is going to read. And when both of these requests succeed and they execute in parallel, so there's no latency penalty. In fact, the
[19:50] provider call actually is the cross-cloud, cross-region calls, that's the more expensive one. The the call to the replica is a local within the within region call, so that's usually very fast. >>
[20:03] out, the the uh the path and the credentials are sent back to the recipient. And last but not the least, then the recipient reads data. They are now reading from Databricks managed storage.
[20:16] So, that takes away all the networking complexity because Databricks is managing the data access layer. So, the reads are obviously efficient because it's all within the same region. >>
[20:41] asset types. The goal is to cover all the asset classes that we sh- uh support in in sharing. Uh existing shares, we'll open up uh support for those. Those are slightly tricky because it might lead to some
[20:55] unexpected behavior on the recipient side. For example, if they're because the table actually is different. For example, if they're running some streaming workload, if you change the table on them, the recipient queries might start failing.
[21:07] Uh if it's if it's a view etc. then they might because the replica might lag behind the source, they might start seeing, you know, different data. They may they may see that hey,
[21:19] you know, I was seeing the data until this hour, now suddenly I don't. So, there might be some unexpected uh observations from their end. So, we have to work with the provider in order to enable this. We have to let providers probably control it.
[21:32] So, in terms of monitoring and observability, yeah, we plan to provide visibility into your application costs, you know, and break those down into how much compute is being used, how much storage
[21:44] is being used, uh and how much egress uh resulted as as part of replication itself. And we'll also give visibility into how much egress you actually saved because of your reads from the recipient had
[21:57] replication not been there. So, this uh these insights would allow providers to decide whether they should enable So, these insights we'll provide even on non-replicated shares.
[22:09] So, that will give you visibility into whether a replication should be enabled or not. And then, yeah, we'll also uh report the status of the replica, when was it last refreshed, are there any errors and things like that.
[22:22] Uh we also plan to provide uh cost controls in terms of how frequently uh so, that's your freshness and cost trade-off. How frequently it should be uh refreshed uh based on interval, based on, you
[22:34] know, maybe you know, there's a pipeline that runs and you may want to say that oh, start your refresh after at a certain, you know, hour. So, we'll probably provide that as well.
[22:46] And then, we will also support other use cases that are built on top of sharing such as clean rooms because there also we might be processing data that is, you know, cross-region, cross-cloud, and Genie agents uh which are accessing data
[22:59] under the hood. Even that data would be replicated and you would save egress on that >>
[23:11] anywhere uh because we don't control the compute of open recipients. So uh we have to think about how to uh support them. One simple idea is that we'll ask the provider to assign a region to an
[23:24] open recipient. For a data recipient, we know exactly where they're going to access from because we know the workspace, the metastore, etc. But for open recipient, we have to ask the provider. And then we will assign the
[23:36] recipient the provider specified region and then we'll serve it from that replica.
[23:48] And I think this is my last si- slide. With that, we can look at the demo. Now, as you can imagine, replication takes time. It takes about 10 to 15 minutes to replicate, so we cannot do a live demo. So, we just pre-recorded it. All I can say is this is done on production sort
[24:01] of environment, so yeah. It's we're not faking it, so. Yeah. >>
[24:13] on As you can see, the this the provider is on this this shard, uh which is on AWS >>
[24:28] And so, what we will do is we will uh essentially I think Yeah. Create a share in AWS. This is your usual Delta Sharing C U J that you hopefully
[24:41] are all familiar with. And then yeah, it's I think you're going to add a couple of tables. Table one is a toy table. Catalog sales is a table in uh one of
[24:53] our TPCH one terabyte benchmark. We'll use it for some query performance later. >>
[25:09] >> And so here I'm going going to add the recipient. Uh so I'm grabbing the sharing identifier from uh the GCP recipient. And and you'll notice that as soon as we enter a cross region or a cross cloud recipient, uh you this toggle shows up,
[25:23] uh which is what you would use to enable global distribution. And yeah, in private preview we allow you to control the refresh frequency.
[25:35] This defaults to 1 hour, but you can change it. I think 1 hour is the minimum granularity we support right now at private preview.
[25:53] >> And I think that's it from a provider side. Uh that's the CUJ. The only additional thing is that you just enable global distribution. And now as soon as you we do that, Novel would in the background start provisioning that regional replica that
[26:06] that I talked about. And obviously that would take time. Typically it takes
[26:21] Y- yes yes yes yes. Okay, so this is the recipient CUJ. So we are going to mount that catalog on the on the GCP side. And you would notice that none of the tables would show up initially. So our behavior is that until the replica is
[26:34] ready, the recipient doesn't see the table. So they can't query it. They will only see it so I I right now uh the the regional replicas are still getting ready. So the tables don't show up here on the
[26:46] recipient side.
[27:02] >> Now, I think it's uh Hopefully it doesn't wait for 15 minutes.
[27:28] So, So, the question is um can you control which uh tables that I do replicate? Which regions to replicate? So, we detect the recipient regions and replicate just to that recipient region. Um do you have a use case where you want
[27:41] to replicate? I see. Yeah. Yeah. So, since for this private preview we support um data bricks uh recipients, which means that we know where they are
[27:53] and we just replicate to where where they are. Yeah. Okay, so I think uh so >> There's no way to to to limit the regions you want to white list, for
[28:05] example? >>
[28:17] >> Could I just have like a white list of regions that I want the replication to happen in? >> Yeah. >> If I have a new recipient, >> Yeah. I would say not for the share itself. What you can always do is just create
[28:29] another share for the other regions without the global distribution enabled. Um but for this private preview, like you you won't be able to limit to say, "Hey, here's a region I want to
[28:41] replicate to. Other regions, don't replicate." Yeah. >> Can Can we hold on the questions the end, it's just >>
[28:55] >> Almost there. >> Yeah, so now I think the replica is ready and on the recipient side you would see these tables appear. And so on the provider side I'm showing that the table has like seven versions,
[29:07] zero to six. Uh and and I think And then and then on on the provider side you can see that the table path is is S3 essentially,
[29:20] which is you know the table is from AWS is in AWS. And on the recipient on the recipient side you see the exact same versions, which basically means that it is version compatible, but the table is being
[29:33] served from uh GCP essentially. Because the recipient was on Google Cloud Google Cloud South Asia. >>
[29:46] essentially. >> Yeah. >> So yeah, so the table is local, meaning it's being served locally and it is version compatible. >> Okay. >> And I think we'll just do a perf
[29:58] quickly, performance test. >> Oh yeah, so okay. >> And yeah, so in this performance test what is what it did with the first query was a warm up query. This is a synthetic query. All it is doing is it is taking a random weighted sum of all the integer
[30:11] columns of this table or in one of the TPC benchmark. This sort of represents a worst case because you're just it's a read heavy workload essentially. Uh So on on a table that was shared with
[30:23] global distribution enabled, it took about it read about one one and a half billion rows and it took about 30 seconds >>
[30:36] was not where global distribution was not enabled. So again it read the same amount number of rows, but it took about a minute. But this this is a slightly skewed example because usually your queries may vary.
[30:48] And the read the proportion of the lambda is obviously depends upon how what fraction of that query uh spent in actually reading the bytes. So.
[31:01] >> Yeah, I think that's pretty much the end of the day. >> Awesome. I think the key takeaway could take away for recipient experience is that there's no changes. Like they just query SA do today.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.