OpenSharing The New Open Standard for Sharing of Data and AI Assets Across Platforms & Organizations

Summary

OpenSharing is the next evolution of Delta Sharing—being donated to the Linux Foundation—addressing four key problems: no standard for AI asset sharing, cross-platform format fragmentation, on-premises data exclusion, and cross-cloud egress cost and latency.
After five years and 300+ partners, Delta Sharing is graduating to OpenSharing, extending zero-copy governed collaboration from data tables to AI models, agents, Genie Spaces, and unstructured data across any platform or cloud.
New capabilities include secure private endpoint connections for on-premises data, Lakebase CDC replica integration, and cross-regulatory support for environments such as AWS GovCloud and Azure China.

OpenSharing The New Open Standard for Sharing of Data and AI Assets Across Platforms & Organizations

AI is only as good as the data that fuels it. To build and scale models, organizations need a way to move past fragmented data silos and complex ETL. OpenSharing is the next evolution of Delta Sharing, the most widely adopted open protocol for secure zero-copy data sharing We’ll dive deep into: Fueling AI With Openness: How to share live data and AI assets with any recipient—regardless of their platform—to accelerate model training and deployment. Latest Innovations: A first look at enhanced governance, automated provider workflows and expanded support for complex assets like AI assets and unstructured data. Real-World Scale: Real-world use cases showing how organizations use the protocol to unify their data ecosystems and manage AI assets at scale. From live demos to architectural best practices, you’ll walk away with a roadmap to unify your data ecosystem and lead the next wave of open collaboration.
Talk By: Harish Gaur, Product Marketing Director, Databricks, Inc. ; Mengxi Chen, Senior Engineering Manager, Databricks ;

Chapters

FAQs

What is OpenSharing and how does it differ from Delta Sharing?

OpenSharing is the next evolution of Delta Sharing, expanding the open protocol from sharing data tables to sharing AI models, agents, Genie Spaces, and unstructured data across platforms, clouds, and organizations. It is being donated to the Linux Foundation as an open standard and addresses gaps that Delta Sharing did not fully solve, including cross-platform format fragmentation and on-premises data sharing.

What four problems does OpenSharing solve?

OpenSharing addresses: the lack of a standard for sharing AI assets like models and agents; fragmentation caused by walled-garden sharing solutions that break across platforms and formats; on-premises data not being a first-class citizen in existing sharing protocols; and the high cost and latency of cross-cloud data sharing due to egress fees.

How does OpenSharing handle privacy-sensitive or regulated data?

OpenSharing supports clean room-style privacy-centric collaboration and includes a secure connect feature using private endpoints to share on-premises data without punching holes in network configurations. It also supports cross-regulatory environments such as AWS GovCloud and Azure China, making it suitable for highly regulated industries.

What is the real-world scale of the Delta Sharing ecosystem that OpenSharing builds on?

Delta Sharing was launched five years ago as an open protocol for cross-cloud, cross-platform data sharing and has grown to over 300 partners. OpenSharing builds on this foundation to extend the protocol to AI-first assets, with an example from this video showing 400+ airlines across 180 countries sharing data with zero copies.

Full transcript

[00:07] Good morning everyone. Last day of the conference. Thank you for attending and my name is Harish Gaur. I am the director of product marketing for data sharing products and I'm joined here by Mengxi Chen who leads the engineering for all the data sharing
[00:19] products. We're excited to talk to you all here about open sharing, one of our biggest evolution in the last 5 years. So, let's jump in right away.
[00:34] All right. So, 5 years ago companies were sharing files and essentially tables. And now companies want to share AI models, agents, and AI skills.
[00:46] But there is no standardized way of doing so today. If I were to ask you to share an AI skill with me, you would probably send an email. And once the AI skill is updated, you'll probably send me an email again. So,
[00:59] that's not a scalable problem. A scalable solution. So, that's problem number one. Problem number two. Um We are starting to see fragmentation.
[01:11] And that's because there are multiple formats and multiple platforms and multiple companies are taking the approach of keeping a walled garden sharing solution. Uh if I am sharing the same format,
[01:24] Delta Parquet, or Iceberg with you, um and you are on the same platform, sharing works. But as soon as you go cross platform and cross format, sharing
[01:36] breaks down. And in that case, people end up copying the files, creating silos, which is also again not a pretty interesting solution. And on top of that, ton of enterprise data sits on prem
[01:50] still. Not everything is in the cloud. And that is yet not a first class citizen for sharing. And problem number three, cross cloud sharing is costly and
[02:04] pretty slow. Um every time I've seen uh a recipient wants to receive a share from a provider, they have to punch in holes in the network. They have to configure their IP
[02:16] settings just to get the new recipient on boarded. That takes time. And when you're sharing a cross cloud, which happens most of the time when you're sharing externally, when you're sharing a cross cloud or across regions,
[02:29] there is a cost associated with it, the egress cost, which cloud providers charge. And I've seen customers who are surprised and shocked at the end of the month when they see tens of thousands of dollars in the egress cost.
[02:41] And of course, if you're sharing large volumes of data over cloud, there is also latency issue. So, the good news is this is not a problem which has not been addressed. Five years
[02:55] ago, we launched Delta Sharing as the open protocol for sharing data across clouds, regions, and platforms with zero copy. Uh and customers embraced it. So, thank you all.
[03:07] Over the last five years, we have seen approximately 300 partners, thousands of customers, uh and significant adoption of Delta Sharing. Just to highlight two numbers here, 115%
[03:21] year-over-year growth, uh the number of active shares between provider and recipient has grown significantly, and 33% of those shares are external. What I mean by that is 33%
[03:35] of those shares are between Databricks and non-Databricks users, which is a testament that this is an very This is a testament that customers have embraced open and not vendor lock-in.
[03:47] Okay. But we haven't stopped there. Um we are taking the next step. You might have seen the press release last week, and you might have heard about open sharing in the keynote.
[04:00] Open sharing is the governed sharing for the genetic era. We have built it not just to share data, but AI assets. And this is our biggest evolution in the past 5 years of Delta Sharing, and it's
[04:13] built on the same open foundation of Delta Sharing which you all have embraced. So, let's talk about four use cases uh which we have seen customers
[04:25] embrace the sharing for. Share everywhere, AI collaboration, privacy-centric collaboration, and data monetization. So, let's peel the onion one by one.
[04:38] Share everywhere. The idea here is customers could be on any cloud, they could be in using and sharing in any format. So, you could share Customers are sharing whether they are on AWS,
[04:51] Azure, or GCP. Uh and whether they are using Delta Lake, Iceberg, or Parquet. Uh and when the recipient receives the share, they don't have to be on a specific platform, unlike some of the
[05:04] other solutions. They could be using Excel, they could be using Tableau, and still receive the share. And here's an example. Navitaire. It's an Amadeus company which works with
[05:16] 400-plus airlines across 180 countries. They cannot control when they're sharing with someone whether they are on the same cloud or using the same format. That's why they embraced open sharing,
[05:28] and across 400 airlines and 180 countries, never once they have to create a copy of the data while dealing with multiple different formats. All right. Second use case is AI
[05:41] collaboration. Uh this is the new frontier of sharing. We have seen customers now starting to share AI models, genie agents, uh AI skills, uh and and the idea is again simple. We
[05:55] don't want customers to be replicating these AI assets when they're sharing. And one of our customers, Kythera Labs, has already started genie sharing. So, they have built a genie agent on top of
[06:08] their data, and when they share the data, they also share the genie agent, so that customers can ask customers know what questions they should ask with the complicated healthcare data which is
[06:20] sitting underneath. Third use case, privacy-centric collaboration. Uh and this is uh this is when
[06:32] two partners want to share data with each other. They don't want each other to see the raw data, but they want to do the analysis on the joint data. Open sharing along with clean rooms makes that happen.
[06:44] And Intuit is an example uh of how uh they are sharing data with variety of loan providers in a marketplace. And the data being shared is a very sensitive financial data, and
[06:56] they're leveraging clean rooms for doing that. And the last use case is data monetization. Many of our customers realize that their data has value beyond the four walls,
[07:08] and they start to convert data into a product which they can incidentally sell. Through Databricks Marketplace, they can sell this data to about 20,000 Databricks customers. Uh and an example
[07:20] over here is Crisp, who's a data provider, uh which shares the daily store level sales data with Zuru, which is a retailer, and uh Zuru then uses
[07:32] that data for targeted marketing across 800 locations. Again, this is all done using open sharing without any copying of data. So, just to summarize, we talked about
[07:45] that we have evolved from table and volume and file sharing to agentic sharing. Open sharing is the answer for this new era of sharing, and we have already customers like Amadeus and Crisp and
[07:59] Zuru and Intuit who are already benefiting from that. So, we talked about open sharing. Now, let's talk about how this all comes together, what new features we are launching, and what and maybe cool demos. So, I'm going to
[08:12] invite Mengxi. >> Thank you, Harish. Hi, everyone. Yeah, now you have heard so many things about open sharing. I'd like to you to take you through some of the latest
[08:24] innovation we had done at Databricks across open sharing. What is new? What is possible? And what is coming? At Databricks, we took down three pillar
[08:36] in our mission of open sharing. AI native sharing, share any assets with anyone, cross region, cross cloud. And this is our guiding principle to build the best sharing solution for the
[08:49] agentic era. And this year, as we announced Delta Sharing graduated to become open sharing. We have so many feature launches I want to tell you about.
[09:01] So, this might be the slides you want to take your phone out and take a picture. So, let me giving you a quick overview. And on the AI native sharing side, we're really excited to tell you more about
[09:13] Genie agent sharing with data control. And we have many more to come. For example, metric view with agent metadata sharing, sharing of agent skills, sharing of ontology and glossaries.
[09:27] And the second pillar, since the start of Delta Sharing, we have been investing so much to make you to enable you to share any asset with anyone. And in this year, we have many expansion
[09:40] on the source asset you can share, and also on the destination you can reach with open sharing. Manage Iceberg foreign Iceberg, you can share any data governed by the ABAC
[09:52] attribute based access control policy. You can share Lakehouse Federation connection in inside Unity Catalog. And I'll talk about Lakehouse. And for
[10:04] those view sharing users, you can now perform CDF on the share view you have received. And on the destination side, open sharing now allow you to share directly to any Iceberg compatible client.
[10:17] And we also have high performance open client sharing, which I'll be talking about. And for some of those who might be like a government customer or be in a regulatory domain, now we're support
[10:29] direct Databricks to Databricks sharing across those regulatory domain. And I'll also be talking about how you can share with an email invitation. And but not last but not least, as Harish
[10:41] told you about some of the challenges you may have in cross region or cross first cloud sharing, I'm really excited to tell you more about secure connect and global distribution, and how how can they eliminate some of the frictions or
[10:54] some of the challenges in cross region or cross cloud sharing. We have a pretty packed agenda, and I will be talking about different features with demo in between. And then, yeah, so
[11:06] without further ado, let's get started. And the first thing I want to talk about is actually taking a step back and sharing some of our observation on how in the agentic era, how it is
[11:18] revolutionizing some of your data journey. As we know, like in your organization, now everyone is equipped with the agent, and everyone is using them. And
[11:30] now the app or agent actually become a primary data interface to interact with their data. But, as our CEO Ali mentioned, like the AI or the agent, they must understand
[11:42] your data to become actually effective and accurate. So, this is the general challenge we all face in the data-centric era. But, what does it mean for data sharing? Let's put it into context.
[11:55] For example, I'm a data recipient. I'm trying to consume this data as uh I received from a data provider. Now, in the agentic era, everything is powered by AI. And the first thing come
[12:07] to mind is that I would just try to query it using an agent. And here is just an illustrative example you might, you know, like your data consumer might be doing. I received this uh marketing campaign
[12:20] data. I might be just asking, you know, pop on my data agent, could be any agent, and asking a question, "How is this campaign doing? Which one is performing the best?" And this might be an answer I got from the data agent.
[12:33] As you It tells me it seems like it inspected this click table, impression table, and it kind of figured out like to use this landing page URL column to perform a join. And then it's measuring performance
[12:45] based on the average session durations. This sounds pretty cool. It gave me a quick insight within just a few seconds. But, there's a catch. As we look closer
[12:57] into the answer we got from the agent, you may realize does the agent really understand what what I mean by performing the best? Is longer the longer the session actually better or the way that we want
[13:10] to measure it? And different campaign might have different goals. So, that's a question mark. And the second part is that your agent might be lacking data context on the data.
[13:22] It used this this URL as a join key. But, is that actually a unique identifier? Is there a one one-to-one mapping relationship between the two table on this? And what is actually the unit of the
[13:35] session duration? Are they matching against different rows? So this is another question mark. And now, as we understand all of the data context and the business contact behind it, and we look at this answer
[13:48] again. Maybe it's not that accurate. And so, what is the good solution? I want to introduce you about Genie agent sharing. I I'm sure a lot of you have already
[14:01] heard a lot about Genie agent in this summit. It's a great natural language interface to just interact with your data. You know, like the customer you'll get a agentic experience by just asking a
[14:13] question. But with Genie agent sharing, it gives you the capability to package your data with curated contacts about those data together and share them to
[14:25] your recipient. And for those for the recipient, they land it right into a curated Genie Genie agent with all of the contacts and the data packaged together.
[14:37] And on top of that, as the recipient, you can also bring your own data without exposing it back to your uh to your provider. You can customize the instruction, you can add more contacts to the Genie
[14:50] agent, and that's where completely local to you, and this feature is now in beta. And uh let me I'm a engineer, so I want to quickly talk about like a primary
[15:03] architecture. So, as you can see, like your data provider will be packaging their data and all of their curated contacts into the Genie agent, and they will share that agent with you. And on the right hand side is
[15:16] the recipient. They can bring their own data, their own contacts into that same Genie agent, and that become the primary interface for your end user. Oh, enough talking about this. Let's
[15:29] take a look at the demo.
[15:43] Yes, okay. There we go. So, this is a demo uh of a Genie Genie's agent my colleague Tom prepared. It's looking at a bunch of retail sales data. As you can see there's six table packaging in it. I've already given it a general instruction
[15:56] about the context of the data. And with Genie agent it allows you to specify join keys, tell about like different define some of the measure, define some of the SQL query that Genie agent can learn from.
[16:08] This is what we all know about. But this is new. And yes, you click the share button. You can see there's a new external tab. You can start sharing this with an open sharing recipient.
[16:21] It's the exactly same experience of open sharing has today. I just select the recipient and then I click share. Now this Genie's agent is being shared to my recipient. Let me flip to my
[16:34] recipient side and show you how do I consume this Genie agent. As you go to open sharing and you will go to share with me and will find the provider that share this state this
[16:47] Genie's agent with us. And uh we have it and you see there's a Okay, click into it. You will see this Genie's agent actually have six table
[16:59] package within it. And I can just click mark.
[17:12] And yeah, looking into the advanced options, this is where some of the behind the scene magic happens. As the data was packaged with this Genie space, when you click mount, the data is also mounted into a shared catalog in Unity Catalog.
[17:24] And for your consumers, they land right into this Genie space with all of the packaged data already mounted into UC and with all of the contacts the provider have curated already
[17:36] preloaded into this Genie space. And now it's ready for your data consumer. Or you could add more contacts or add more data into this Genie agent that's
[17:49] to further enrich it. So, this is a demo of Genie agent sharing. But, we're not done here yet. I want to talk tell you about a
[18:02] different mode that you can share the Genie agent with data control. But, what does it actually mean? In sometimes, as the data provider, you may not necessarily want to share all of
[18:14] the raw data directly to your recipient. You want to enable them to, you know, derive insight and through a, you know, like agent agentic interface without necessarily sharing all of the raw data
[18:27] to them. Or you may have packaged like secret sauce that in the instructions that you consider them as a IP that you don't want to give away. And this is what data control is aiming
[18:39] to help. With data control, you have the same exactly same curated agentic experience delivered to your recipient, but you are able to hide all of the underlying data. No end end user cannot
[18:52] directly access them outside of this Genie agent. And you get to hide your IPs that might be in the instructions, that might be in the SQL expression, and whatnot. And on top of that, you can also limit
[19:05] the number of rows each answer can return or how many query uh your recipient can run on this Genie on this genie space you shared with them. And on top of that, the recipient
[19:17] can still bring in their own data to to the same genie agent space to do to perform privacy safe collaboration against your data. And we're working very hard on this and
[19:31] we'll have a beta coming soon with this genie agent sharing with the data control mode.
[19:43] All right, the next thing I want to talk about is what Harish already mentioned to you. The silo within formats and platforms. As we all know, Delta and Iceberg has been in a kind of format battle for a while
[19:57] and but now I think it's that concluded. The answer is clear, open table world won. But the problem doesn't really stop stop here. You may be using Iceberg, you may be using Delta in different catalogs.
[20:11] For example, Polaris, AWS Glue, Snowflake. And how do you And when you want to bring them together and you want to share with someone using a different platform? There has not been a good solution
[20:24] because the underlying data is sitting in different catalog or different storage. And then for Delta or Iceberg, they have different catalog APIs that speak different languages and that's not
[20:37] very compatible with each other. So, how as a data provider, if want to What is the best solution for you to bring them all together and share with someone who's either using Databricks or
[20:49] not? Like that may be using any other external platform. So, that's the reason we are we focus a lot of efforts to build Iceberg interoperability with within open sharing.
[21:01] With open sharing and Unity Catalog, you can forget about the table format and open sharing enable you to share any Iceberg source without copying them. You can share managed Icebergs that sit
[21:13] in your in Databricks, or you can share foreign Icebergs that might be sitting in Polaris, Snowflake, AWS Glue, and you can bring them into Unity Catalog through Lake House
[21:26] Federation. And once you've done that, it's the same open sharing experience. One click and you are able to share. And on the destination side, you can reach with open sharing you can reach both the Iceberg and IRC and the Delta
[21:40] ecosystem natively. That is because open sharing implemented both the Delta Sharing and the Iceberg REST Catalog protocol natively. And that will enable you to reach
[21:52] any platform that within those ecosystems natively. And now I want to just show you a demo of actually how it's done.
[22:11] Okay. So, as you can see right now, oh, my bad.
[22:25] Let's try it again. Okay. Cool. Now I'm a I'm a data provider. I have uh I have four table I want to share. Like one of the other event table, as you can see the format is managed Delta, and I have another sensor table,
[22:37] which uh yeah, and by the way this uh we already enabled uniform on this managed Delta table. And we have another sensor reading table, as you can see the format is managed Iceberg. And then I have two views built on top of each tables.
[22:51] Now I want to share them all together to someone one of my recipient who is actually using Snowflake. So, this should be familiar. Like I'm already packaged all of the four table
[23:03] on the share into a into the style of IRC interrupter. Now, I want to add my recipient. Actually, here, I'm going to create a new recipient. And you might notice there's a new option called sharing
[23:15] using email. Which is going to send the invite to this email recipient. Let's take a look. I'm going to name this recipient. I'm going to add the email address in this case. And
[23:28] then, I just add this email recipient into the share. And just a second, I should have Yeah, I already received this email.
[23:40] Uh Okay, this is a better way to show this email. Uh as you can see, I received this invitation from my provider, Marketplace Demo. It shared data with me with the four tables.
[23:53] And now, as a recipient, I got the option to choose where do I come I want to consume this share. I can either access it from Databricks, or I can access outside of Databricks by downloading this token credential.
[24:05] I did it here, and I'm going to this activation page. And um to leverage this nice connector, which I'm going to upload the share credential file I got.
[24:17] And um it's uh has a nice feature that can automatically generate the SQL command I need to use in Snowflake to install this share. So, now I have copied this command.
[24:30] And now inside Snowflake, and within the notebook, I will be um pasting this command in. And I can see there's uh two placeholder in this command I need to replace. Uh the first one is uh the the
[24:43] name of the catalog integration. We're calling it open sharing IRC uh catalog here. Uh and then, I the second man is creating a database, a linked database using this
[24:55] uh catalog file uh catalog integration I created above. So, let's Now, let's run it. As this is running, just a quick overview on what this command is doing.
[25:07] The first command creates this iceberg rest catalog integration using the catalog URI I got from Databricks. And in this case, I'm using bare token to authenticate to this uh
[25:19] rest catalog. And the second command actually create a linked database. And with that, I should have a database in my Snowflake. And uh next up, I will go to the
[25:31] database explorer. Let's just do a quick refresh. As you can see, this open sharing IRC database popped up. And open it, I can see all of the four table I shared from
[25:44] Databricks into Snowflake. And uh Everything looks in good shape. And uh quickly uh looking at the data preview, yeah. You work It worked. With a few
[25:57] clicks, now I have I'm able to share a combination of Delta and iceberg table with views into a different platform using open sharing.
[26:16] I believe that's the end of the demo. All right. Now we have looking at AI native and uh iceberg and availability, I want to start talking about cross-cloud and cross-region. But before I do that,
[26:29] just a quick overview on open sharing and how it works. And open sharing is zero copy by nature. And what do I mean by that? As you have just seen in the demo, like
[26:41] this is the flow. You will create a share, and you will distribute it to your recipient. But when the recipient actually making a query to the data you are sharing with them. After the open sharing
[26:53] authentication and handshake, the data access is actually directly made from the recipient's compute to the provider's storage. And in this case,
[27:06] there's no copy, there's no there's the direct transaction between the recipient's compute and the uh provider's storage. And the benefit of this zero copy fashion is that it minimizes
[27:21] You don't have to maintain a copy, you don't have to maintain a pipeline that maintains that copy. And then for your recipient, it's always they always consume consuming from the single source of truth. They always get the fresher data,
[27:33] there's no versioning, and they're always consistent. But in the case of cross-cloud or cross-region sharing, networking can sometime get into the
[27:46] way. Because when we're talking about the recipient's compute and the provider's storage, and often time or most of the time these resources stand sit behind enterprise firewall
[27:59] or enterprise VPC. And also, there's a problem of disconnected persona. Like as the data owner or the share owner, you might be in the data team. You're managing data, you're managing who you want to share those with.
[28:12] But to actually establish the networking connection, you need to talk to your cloud admin, networking admin to say "Hey, like I need to allow list this recipient's compute address in my
[28:25] storage firewall. Can you help me do that?" And in in real life, it often engage rounds of security review and approval, and it's stopping or is slowing down the
[28:37] connection or the data share. And that's the reason we built open sharing secure connect. Which provides a managed storage proxy
[28:49] to really simplify networking and while enhancing the security. At a very high level, the way secure connect work is through a Databricks managed proxy that sit in the middle of
[29:03] the recipient's compute and the provider's storage. And all of the data sharing requests from multiple recipient will be channeled through this central managed proxy.
[29:16] And with that, for the provider's perspective, all they need to trust or allow is this Databricks managed proxy. And the way they do that is also very easy. They can
[29:28] just allow allow the serverless regional IP range, um, where the this managed proxy actually sit. And for the recipient, if they're using serverless or if or classic or open
[29:43] compute without a VPC policy, it just automatically work for them. And in case it's behind a VPC policy, so for those classic or open with compute, they just need to similarly allow this the Databricks IP
[29:56] of the the managed proxy we just talked about. And at the same time, secure connect deliver private link connectivity through the network connectivity configuration, which you can now attach
[30:10] to your metastore where you manage all of the Delta sharing. And on top of that, since Databricks sit on the both side and sit in front of both side of recipient's compute and the provider's
[30:22] storage, we're able to establish mTLS connection, um, in this case, since we manage both side. Now I have talked about it, let's look at a real demo of cross-cloud sharing
[30:35] with secure connect. I will be sharing a state of provider from Azure, uh, I think US US 2. And I want to share this World Cup flight operation data,
[30:48] which uh as you can see is uh is a managed Delta table with a lot of record about the flight operation uh data that's, you know, going into the
[31:00] World Cup. And as you can see, the storage location here is in uh Azure blob storage. Now, I'm in my Azure portal. I'm going to go to and this is my storage bucket. As you can see, I've already disabled public access mode.
[31:14] And on top of that, I have already provisioned a few private endpoint on top of the storage, which means that any access must go through this private endpoint. Otherwise, it's not going to be allowed. And um
[31:28] in this following step, um yeah, I'm going to I already made this share. I'm going to add my recipient, which in this case as you can see is in AWS uh US West 2. And I have not not the end done anything
[31:40] with secure connect yet, but let me first show you the recipient experience today. Uh on the recipient side uh sitting on AWS, I'm going to mount this share.
[31:57] And we're going to find this uh flight table. I'm going to make a sample query to it. Yeah, this will be the error you will get. Your request failed because it doesn't have access, your computer doesn't have access to the Azure to the
[32:10] private endpoint sitting behind sitting on top of the Azure storage. And now, I'm back on my provider side. I'm going to just click enable secure connect. And behind the scene, I have already done the allow listing I talked
[32:22] about earlier. And um yeah, nothing have changed on my storage bucket. It's still no public access using this private
[32:34] endpoint. Now, I'm go back to my recipient side and let's try that sample query again. And now you got the data. Because the traffic is now flowing
[32:47] through the Secure Connect managed proxy, which the provider have already trusted and able to act on behalf of the provider and accessing uh their private endpoint, which is a
[32:59] network configuration they also attached. So, that's the benefit of uh Secure Connect. But, so in some cases, your cross-region, cross-cloud challenge doesn't really
[33:11] stop here. There's also egress cost. And the data egress charge can quickly add up. And just to give a very concrete example, I might be a data provider. I want I'm
[33:24] sharing with, you know, let's say 10 different recipients sitting in different cloud and regions than me. And um each of those recipients on average make 100 query to my to the data I'm sharing with them.
[33:36] And uh the query might be pulling different side of the data, but when it comes to egress cost, you might be looking at 10 * 100 * the data they they pulled
[33:48] from your shared data and * the rate. Although the egress charge rate is like just a few cents per gigabytes, but in large-scale sharing, it adds up really, really quickly. And we have heard loud
[34:01] and clear on in from some of customer who are got surprised by their cloud storage egress bill when they did uh when they did large-scale cross-cloud sharing. And um
[34:14] and also, like, as the data provider, you don't really want to, you know, just maybe like throttle your recipient. You already entitled them with the data access, and they should be just be able to use it without this egress problem.
[34:27] At the same time, for your recipient, when they receive the shares that's sitting in a different cloud. Every time they query, they incur this geo geo geo latency for the network
[34:39] request or for the data to traverse the traverse through the internet over different cloud and region. So, what is the good solution? In some cases, there there are alternatives that exist.
[34:53] Uh but yeah, we believe open open sharing global distribution is the best solution. So, with global distribution, it actually with one click enablement,
[35:05] it will start to replicating replicate your data from your source storage to the recipient region into a default storage that Databricks managed, which this will be a
[35:17] this will be a one-time full copy and so all of the following update will be done incrementally and also in a version compatible maintained fashion. And once uh we have this
[35:30] replica that's sitting in the middle in the default storage of on your recipient region, every time your recipient make a query, their query automatically routed to this in-region replica, which will significantly improve the query
[35:42] performance. And as at the same time, it's a very simple and fully managed experience. You can just do one click enablement and let Databricks handle the rest. And since the cop the replica is version
[35:56] maintained, which means for your recipient, they don't need to change any code in their query or in their workload. Everything will just continue to work. And let's put this together and I'm let me
[36:09] show you a real demo. Excuse me, can you play the sound? Okay. I I guess I'll I'll just quickly talk us through it. So, right now I'm on the data provider side. I'm using AWS on
[36:22] Canada Central. I have three table I want to share. One marketplace inventory and uh one system house even lost. I want to just quickly show you the history of this table since we'll be looking at it again.
[36:34] As you can see, this uh this table actually have more than 100 version. And I'm going to be sharing another table, this TPCDS scale factor 1,000 uh 1,000, which is 3 GB, pretty decent in
[36:46] the size. And today, the way you would uh do a cross share uh cross cloud share is just, you know, similar to the you how you share with others. And on the recipient side, I've already mounted
[36:58] this demo D2D share with our global distribution, and as you can see, the storage location uh is still using the S3, the provider storage location, which sit in S3.
[37:15] And which means every time you, as a recipient, make a query to this share table, it will incur egress cost to our provider, and the query perform like when the query perform, it actually have to traverse across region and across cloud, which, you know, in some cases,
[37:29] depend on the the region pair we're talking about, it's slow slows the query down. And now, let me show you the true power of global distribution. I'm going to make a new share. Let's call it demo global distribution. And I'm going to find the same three
[37:42] table I'm going to share. And uh continue. I'll just click continue here. Okay. Now, let me add my same recipient, which is in GCP US East 1. And now, you see this.
[37:56] It's telling me that I have one recipient outside of my region and cloud. And I can enable global distribution, which will help me save cost and improve query performance. That sounds great. Let me Let's Let's
[38:08] enable that and give it a try. And uh let's quick refresh here. Okay, I see the three table I wanted to share. And um So, the way global distribution works, the moment you enable it, it will
[38:21] trigger immediate full clone to your to the into the default storage of and that's it on your recipient's region to produce the initial replica. But after that, all
[38:33] of the following update source table update will be will be incrementally updated into the replica on the schedule base. Uh it's probably hard to see, but on the right-hand side you can see there's a refresh frequency which by default every
[38:48] 1 hour. And also the way the replica is the replica will always have all of the version maintained which I'm about to show you quickly next. Okay, so now I'm back on my
[39:01] recipient side on GCP. I've already received this share. Let me quickly mount it. And yeah. As you can see the data is not yet ready because we're in the background
[39:13] performing this full clone. Uh but it should be done fairly quickly. Uh but we did have a one table that was like 3 GB. So Uh let me do a few refresh. Okay.
[39:26] Now you can see it's there. Within the default schema I have all of the three table we're looking at previously. And now it's the true moment. Let me go to deep detail. As you can see in this
[39:38] case the storage location already changed. It's using a GCS bucket uh power backed by Databricks default storage and which in this and that's sitting in the recipient's region. In
[39:50] this case like GCP US West 1. And let me switch to history quickly. And remember this was the table that has like more than 100 version. And it
[40:04] should be here as well once once it load.
[40:17] Yep, there you go. All of the 100 version are fully maintained, which means every any recipient side query or workload does not need to change. If they're doing CDF, doing streaming, it will just continue to be powered by you know, the replica. And yeah, while we're
[40:31] talking, I already made a sample sample query and then you can see it's now served from this in-region replica without incurring egress cost to the data provider and for the recipient, you'll benefit with the in-region query
[40:44] performance. And that's global distribution. Okay. Now let me let me give a quick recap. Oh. Sorry, I won't let this work out. Yeah.
[40:56] Um yeah, this is a you got the benefit of zero copy, but on top of that secure connect bring you simplified networking, private connectivity, egress minimize it and global distribution bring you egress cost minimization and fast query.
[41:10] And uh Yeah. I have a few more slides to talk about, but yeah, unfortunately, I think we're a little bit over time here. So, yeah, quickly
[41:22] yeah, this is high performance open client sharing, which is powered by cloud token sharing, which was was made default in Databricks to Databricks last year. But now we have fully open sourced
[41:34] it within open sharing with a credential vending API, which means within the open sharing ecosystem, either like as a client or as a as a server, they can leverage the
[41:46] vendored credential to perform directory based access. And it's already supported by, you know, SAP Databricks, Snowflake and all a lot of other compatible open sharing server and connectors.
[42:00] And yeah, quickly this is the what's the email sharing feature I showed you earlier. It's a new way of sharing. It will help you stop to the need to ask offline like, "Hey, what where do you want to consume
[42:13] this share, whether some data bricks or off data bricks?" And then, if you want to use data bricks, what's your sharing identifier? And now, as a provider, you just input their email address, and then they as a recipient will just get an
[42:26] invitation, and they can now choose where do they want to consume this share. And it's the same experience as was the same set of security and control open sharing offers. And we have a public preview coming soon on this.
[42:39] And we also have some new feature coming together with lake base. Just quickly talking through them, like, now you can register your lake base within Unity catalog, and then on top of that, you can easily share them with open sharing.
[42:53] And another feature lake base offered is that you can use lake base change data feed, which which will create a fully native CDC replica, and then it's a managed table, and it just work with open sharing
[43:05] natively. And uh yeah. Cross-regulatory domain, we expanded uh the reach of Databricks to Databricks sharing. Now, from any commercial cloud, you can
[43:17] bi-directionally sharing with either AWS GovCloud or Azure China. And all of the connection is still secured and within the Databricks network.
[43:30] Yeah. And to conclude, this is all of the beauty of open sharing. I'm sorry, like, we went a little bit over time today, but yeah, uh we'll continue to make great things happen within open sharing, and we're
[43:43] very much looking forward to your feedback. Thanks for your time. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.