Skip to main content

Simplified Delta Sharing With Network Security

Summary

  • Delta Sharing network security: This video describes a solution that eliminates manual network and storage firewall configuration, which has been a major complexity barrier for enterprise data providers and recipients.
  • Seamless collaboration infrastructure: Learn how new Delta Sharing capabilities enhance user experience and scalability for large-scale providers with strict compliance requirements sharing data across clouds and regions.
  • Databricks Sharing Gateway: Discover the upcoming technical approach that simplifies data collaboration by removing the need for manual security configuration at scale.

Simplified Delta Sharing With Network Security

Watch: Simplified Delta Sharing With Network Security
Talk by Krishna Puttaswamy, Principal Engineer, Databricks ; Samrat Ray, Senior Staff Product Manager, Databricks
Delta Sharing enables cross-domain sharing of data assets for collaboration. A practical concern providers and recipients face in doing so is the need to manually configure network and storage firewalls. This is particularly challenging for large-scale providers and recipients with strict compliance requirements. In this talk, we will describe our solution to fully eliminate these complexities. This enhances user experience, scalability and security, facilitating seamless data collaboration across diverse environments and cloud platforms.
Here’s more to explore:
Open data sharing for data, analytics, and AI: https://www.databricks.com/product/delta-sharing
A new approach to data sharing: https://www.databricks.com/resources/ebook/a-new-approach-to-data-sharing
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What problem does this video solve for enterprise data providers?

This video addresses the significant complexity of manually configuring network and storage firewalls for data sharing, which has been particularly challenging for large-scale providers with multiple recipients and strict compliance requirements.

How does Delta Sharing enable cross-domain collaboration?

Delta Sharing enables cross-domain sharing of data assets for collaboration, allowing organizations to share data across different clouds, regions, and platforms while maintaining security and compliance controls.

What role does the Databricks Sharing Gateway play?

The Databricks Sharing Gateway is a new solution described in this video that handles network and security configuration automatically, eliminating manual firewall setup and enabling seamless data collaboration at scale.

What are the benefits of this enhanced security approach?

This video highlights that automated security configuration enhances scalability, improves user experience for both providers and recipients, and enables seamless collaboration across diverse environments and cloud platforms.

Full transcript

0:00 Introduction to data sharing

Hello everyone. Uh uh thanks for showing up uh this late uh in the day. uh uh it's time to actually I guess we're standing between you and drinks and uh and this is a small group and uh given the size you know we would love to have a very hopefully an interactive session uh because we're going to talk about um a future direction of where we are going and talk about what's just coming versus you know just a set of product announcements.
So it's a bit of different session in that sense. Um so to begin with uh I will just introduce ourselves. Uh I lead u product security uh at data bricks covering uh your serverless data security and AI security of late. And with me is Hello Uh I'm the tech lead for our sharing and collaboration platforms including uh data sharing. And um today uh we will talk about how data sharing actually works uh from a networking standpoint like with the focus on networking because this is uh focused on that aspect the session uh we'll talk about how you can sharely securely share data on data bricks today and we'll talk as I said about where we are going with this like how are we trying to make your life simpler especially if you are uh in a setup which requires stringent network that how are we planning to make your life simpler over the next um few months uh you know weeks and months ahead and with that I'll hand it over to yeah to talk about data sharing so let's start with the basics and you know I'll spend a little bit of time looking at uh what is data sharing how it works uh then we'll dive into u the security controls that we have today some of the challenges and what we are launching soon um data sharing is a uh is an open and flexible way of sharing uh all sorts of assets, data and AI assets uh across different platforms, clouds and regions.

1:43 Overview of Delta sharing

So let's take a little bit of a deeper look. What does it mean? So this picture sort of shows a highle view of uh data sharing. Imagine that you have some data in your warehouse. You have ingested it, clean it up, done some ETL, sort of curated the data. Now you want to share it and collaborate with your colleagues and partners. Uh they may be in the same company, different company, different cloud or may be on a different uh platform altogether.
So this this happens all the time in in in a lot of companies and so um data sharing is our way of sort of enabling all these various forms of collaboration. So um here you see um data in the data warehouse is getting shared from a provider to uh a recipient and data sharing is the open protocol and the API that the two sides will use to communicate and sort of exchange data. Um as I mentioned briefly uh the provider can share all sorts of assets um data AI assets um including notebooks and models and so on.
Um they can share data across clouds. the recipient doesn't need to be on um data bricks to be able to consume the data. Uh they can be using PowerBI, Tableau, um whatever works for them. Um and um we allow sort of sharing you know um across platforms, clouds and the recipients could be in a completely different region as well. And uh the way data sharing works is we sort of uh share access to the live data without making copies or replicating or requiring any sort of ETL on the on the recipient side.
So recently we also started sharing you know if you have data that is in a different place and you federate into unity catalog then you can sort of share that also to your recipients in a different platform. So on top of the data sharing platform we have several of our own data bricks products that we have built. Uh one example is systems table um which is a way of sort of sharing operational data about how customers are using data bricks including things like billing and audit logs and so on.
So that entire system is built on top of data sharing. So we are sharing operational data with thousands of our customers on top of this platform. Um a second example is clean room which is a privacy safe way of collaborating across multiple companies or customers. Um is also something that we built on top of uh data sharing as a platform. Um and finally marketplace where data vendors can come in and sort of provide data to hundreds of people is also powered um on on top of data sharing and sort of we are very excited and happy with the growth we are seeing uh in the last year alone uh the usage in terms of the number of shares has tripled um and 40% of those accesses or shares are D2O these are uh recipients that are out in the open not on our uh datab bricks platform.
Similarly on the marketplace side um we are onboarding a lot of providers. I've sort of listed the logo of some of the recent ones but it's also growing at a very quick pace uh sort of showcasing the power of this open um way of collaborating across platforms. So with that sort of uh introduction now let's sort of look at it from a more technical point of view. How does it actually work? So to get started um this picture sort of shows a um bunch of business units in a company that are um you know distributed across different regions maybe and some of them are on data bricks and some of them are not.

5:42 Technical data flow and security

So here um to share data one of them will sort of become a provider they will sort of create u they'll take um they'll go to unity catalog create what we call as a share add the assets that they want to share into the share and then add the recipients that they want to share with um to that share um if you are on um data bricks on the recipient side which is called as which which is what we call as D2D data bricks to data brick sharing uh all the recipients has to do is the shares automatically show up.
You just have to mount it as a catalog in Unity catalog and then you can start querying the data. On the other hand, if the recipient is open, um then this is what we call as data bricks to open sharing where the provider is on data bricks and the recipient is on open and here the recipient would sort of download a profile file which has link to our you know sharing server with some of the credentials and then they can use any of the open-source data sharing connectors um to sort of query the data on the prov uh from the provider.
So when one of those recipients whether it's you know on on the open or on the data brick side when they actually issue a query um select star from something um these are sort of the this is the flow uh described in this picture. So from the data bricks data plane or from the connector uh in the open uh a request comes up. This API is what we call as query um API. So this goes to um the recipient uh side unity catalog or control plane in D2D and a bunch of checks are done.
So we make sure and check and make sure that you know this particular recipient um is allowed to access um this particular table basic odds and then that request comes to the provider side where we do additional authentication and authorization checks. We make sure that you know this particular recipient is actually added to the share and then the share owner the person who created the share has access to the asset that is being queried and all if all those checks are passed then we sort of um look at the request uh depending on what is being queried identify the set of um files in the in the park files that are in the S3 bucket or storage bucket um and then we sort of generate create a temporary credential to access those files and then we sort of pass those pre-signed URLs for example to the recipient and the connector on the recipient side will use those presigned URLs to access the data from the bucket.
So like I mentioned you can sort of look at the checks that we are doing and classify them into like three different classes. Um user authentication and authorization checks. Um and then um share access check as in you know the asset that is being accessed is part of the share. The share owner has access to that share u to that particular asset and the recipient is allowed access to that share. And then finally the third part is we're issuing um pre-signed URLs uh to the files in the bucket.
So this is where we'll sort of dive deeper and the rest of the talk is going to focus a lot more on on that part. So if you look deeply um so you you have the delta table and delta table let's say is you're on AWS and it's backed by um um S3 bucket. Um usually we have a bunch of network controls you know um S3 access policies um on on the bucket. So in this workflow for the recipient to actually you know consume the pre-signed URLs that are issued the recipient IP address or VPC or something should be allow listed on the S3 bucket and configured on the policy.
Um so this is something that you know the network admins you know when the this person who is sharing is working with the network admins or the storage admins to sort of make sure and configure so the recipients actually have access to the storage. Um similarly if you are on different clouds um you would have to sort of do similar firewall configuration or you know VPC um configurations on AWS, Azure and GCP. They all have same concepts in various forms.
So this one is a little bit challenging. Why is that? So let's sort of take this example. So we have a provider uh we have two personas. So the person who is you know creating managing the share and sharing it with the recipient and then there's a different persona who is sort of in charge of um you know securing the bucket configuring the policies and so on and so forth. So the way it works today um these two personas have to coordinate every time they want to share data with a recipient because uh they want to get you know information like okay your IP address VPCs and so on and then configure it in the storage layer and this sort of scales in the order of the recipient.

10:42 Networking challenges at scale

The more recipient you have you have to do this handshake every time. Similarly, this this is on the provider side, right? Um on the recipient side, there may be similar configurations here. The recipient um I'm getting some share now. Now I want to sort of you know access it in through my connector. Um so I may have egress controls um configured in my compute right. So now when I receive the share I have to understand okay where is this bucket where is this going and then work with my network admin every time I want to consume something I I have to sort of coordinate and sort of configure the network and um you know go through this process for every one of those providers.
So this is a toil I mean it is all secure but there is a lot of toil involved in this collaboration and sometimes this collaboration can takes weeks or months to sort of figure this out. So we have been working on sort of simplifying this and automate all of this and so now Samraat I think is gonna uh describe how it works. Yeah thanks thanks. Uh before I jump into like what we're planning to do a quick show of hands described a challenge from a networking perspective.
How many of you are actually facing this challenge right now or this is a new information for you? good like almost 60 60% of the room is familiar with this and for the rest of you uh is it something that you antic how many of you anticipate that you're going to face this challenge once you start using data sharing forward almost the rest of the room pretty much uh close to that so I think this is a very relevant topic for um many of our enterprise customers because um nothing like no data moves without actually not just securing securing it from an identity perspective but also securing it from a network perspective right so how are we thinking about simplifying this problem going forward so before we jump into a solution let's structure the area to understand like what how we are approaching this so there is two basic variations of data sharing that we support which covered um earlier the first one is when both the provider and the recipient are data bricks are on data bricks so we call that data bricks to data data bricks or D2D sharing and the second one is when the provider is on data bricks but the recipient could be on any platform right so we call that datab bricks to open or D2O sharing within the datab bricks to data bricks sharing itself you can think of the topologies in which we are enabling the sharing the first topology is both the provider and the recipient are in the same region so that's the same region sharing Therefore um if you think of a storage bucket that would be accessible locally or privately in some sense within the same region.

13:25 Future direction and strategy

Then there is the other topology where you are actually sharing across region or even across clouds. In those scenarios the bucket cannot be privately be accessed. If you directly try to access the bucket you're going over a NAT gateway. So sometimes that can be a problem because we may not want to access data overnight, right? Um but that's the cross region sort of pattern that we look at, right? And in the open case because it's um it it can be different topologies, right?
So because it's just the compute could be anywhere. It could be somebody sitting on their laptop for that matter. So it it it doesn't fit in neatly into the set of topologies we talked about. So this is the problem space that we're looking to solve. Like given this problem space, how do we simplify some of the challenges that we talked about? So where are we going? What is our end state that we are aiming for with the uh work that we're doing right now?
The first and important thing is we want to make sure that all this orchestration that described between the um data provider uh and the storage administrator or the infra administrator. We're going to minimize that because that is a big for all of you who have who have been dealing with central IT teams you know that's a and if you have to do that on a very a frequent basis because you want to scale your uh data sharing operation that's going to be a big challenge so how do we minimize that so by so our goal is that you should be able to share data based upon your security princ your security policies that you have defined at the data bricks layer you don't have to for every share go back to the storage layer and have to configure stuff and have to work with your infraadmin to make to enable that share, right?
And this should not just be true if you're working with something like an S3 gateway. This should also be true if you have configured a private link to the storage, which is fairly common especially on Azure. The second aspect that we're looking to support is that from the recipient perspective, if you're on data bricks, you shouldn't have to worry about, oh, what are the buckets, where are the buckets, how do I configure my network to access the buckets.
You should just be able to mount a share on Unity catalog and be able to start consuming the data, right? And the last piece is that from a recipient's perspective, you should always be able to access the data privately from that perspective. And what do we mean by private? It's going over your storage gateway or a private path and it's not necessarily going over your NAT gateway. That's what we mean by private. You should always be able to do that.
So that irrespective of from where the data is shared and this particularly applies to the cross region crosscloud topology. No matter from where the data is being shared, you should be able to access it privately from your perspective. So these are the three sort of main objectives which we are trying to achieve with our solution. So a key component to the solution which we are introducing here today is the data bricks sharing gateway.

17:29 Databricks sharing gateway

This sharing gateway is sits between the uh recipient and the provider's storage bucket and mediates sharing and it is able to uh um uh it is managed effectively the policies on the sharing gateway are managed by the provider and from a recipient perspective they just connect to the gateway if they're in if they're an open recipient if they're a datab bricks recipient then and they're on data bricks then this just happens seamlessly and transparently for you right so what does a typical interaction now look like so let's say I have a storage as a provider um I have a storage account that I want to share and maybe the connection is uh but the requirement is all access to my storage account is over private link or has to be private uh I should have to just onboard the storage account just once and what would you onboard it to the data brick serverless runtime.
So effectively today we support private link connectivity to your storage bucket on Azure or um with S3 gateway on AWS for those who are on AWS but you have to just update your firewall just one time right and now it's connected to the serverless runtime of your local region and the access is private whether it's over a gateway or over private link at this point the data is now available once you have mounted it it's now sorry once you've attached it to that runtime it's now available for the provider to start sharing should they choose to.
Right? Uh how does this work on our platform? Effectively for those who are familiar and those who have started configuring serverless, you might be familiar with something called network connectivity configuration which you have used to configure access to storage. Today you can do that. You can create such a configuration and attach it to workspaces. We will now allow you to attach it to meta stores and shares effectively.
By doing so that data now is from an account admin perspective you have made it available to be shared and now you can delegate the permission to the uh uh catalog admin effectively or metas store admin to then onboard recipients to it. So this is still a onboarding recipient still remains a privileged operation uh within the platform but it's all now within the data bricks landscape right you're not going back to the cloud layer anymore.
So then the second step is the uh the provider can now onboard a recipient. If the recipient is from datab bricks there is no need to get any networking information because it's just a datab bricks is just connecting to the datab bricks component right earlier you had to get the network ID the VPC ID or you had to get the IP range because you had to update a storage firewall which is not familiar with datab bricks concepts.
Here we understand where it is coming from very natively. So you don't need to you just need to say this is the recipient and you just tell us here is the account and here's the recipient actually meta store ID more more than the account just give us that and we know everything else on how to configure this secure connection right um and then uh the last is from a recipient perspective all they have to do is mount the share and that's it because we already understand how to connect and this remains true whether you are in the same region whether you go cross region or cross cloud.
The difference right now is in the same region we can this gateway that you actually see is not actually a physical component. It is a virtual component and we can give you highly optimized uh data access uh patterns. Right? In the cross region kind of scenarios it is does act as a gateway. So there is some minor overhead to having it but it we are able to now secure all connections uh using our primitives whether we have to we can do things like mutual TLS based authentication um tunneling effectively um and even like you know private uh tunnels so if you can build those are all available to us to secure this connection but it's just something you don't have to worry about anymore right so that's the uh In fact uh the key to the core to the solution is through the sharing providing the sharing gateway.
So just to recap some of the benefits that I talked through uh previously one you can now onboard recipients without involving the storage admins. There is zero networking overhead uh for onboarding new recipients. And if you are a uh recipient there's zero networking overhead if you if you are on the datab bricks platform already. So there is no networking configuration required. It's a one-time setup to onboard any new storage bucket to the gateway.
And one of the things which I think I don't think covered but is uh possible through delta sharing is I can start by creating a share that consumes data from two buckets. And then I decide hey my share I've got some new data and it's a different bucket. I can just add that bucket to the share right and this is all transparent from a data sharing protocol. But if you had to manage network firewalls, it's not transparent to you.
But with this aspect, it remain continues to remain transparent to you because the entire storage bucket layer is abstracted away from the recipient. Uh and this solution is not just simpler, it's actually can improve security for you. A couple of places, one is we can always we know where the URL is issued to. So today there are two p there are two steps right uh that described. First is a step that says where should I issue a URL to whom and to where should I issue a URL to that's configured on data bricks and then who can access my data and from where that's connected at the storage bucket now these two things configurations are disconnected you're doing them independently now you do it once and you say this is my recipient this is who I'm sharing it and both the issuance and data sharing are now baked into a single policy that is enforced at both URL issuance and at data access time and in the case of D2D sharing let's take the serverless if your recipient is serverless then the IP address of the network is an insufficient uh control to say do I know for sure this is the right recipient right because the serverless as a network is u uh is a multi-tenant network effectively right now with this solution without having to get into private link and all the complexity that it involves we are able to drive significant security because you can now we can now know that this is recipient this workspace belongs to the recipient.
We know that. So we lock down access to only workspaces that are attached to effectively the metas store to which you have shared which is in the account of the recipient. So network security is baked in at a even finer grain level effectively uh through this approach. So couple of things um to keep in mind uh we said you have to onboard the storage bucket. So the storage bucket has to be in the same region as the meta store in which you create the share effectively right uh and the second thing is uh if you go cross region or DTO today effectively the traffic is still going over the internet between the regions it's not observable by you but that's what is happening but the way you see that effectively is you will see the charges associated with an internet egress for example when you go across region or cross cloud that's um the first iteration will have that limitation.
Going forward, we're actually going to remove that iteration uh that overhead by various mechanisms uh to keep to reduce the uh data transfer or replication cost for both cross region and crosscloud scenarios. So this solution is right now launching to private preview. For those who are interested, uh please uh use this to register your interest and we would love to get your actual feedback when you use this feature and uh you know onboard onboard to it.
We would love to have uh get your feedback so that we can improve it and have you as a design partner. So we talked about what's coming right now, what's in private preview. Let's talk a little bit about um what's coming in the near future. So first is we talked about the simple simpler customer journey. We already talked about that. Um but the storage and the storage firewall configuration automation is part of the private preview as well.

26:13 Roadmap and closing

Right? This will work with both the customer storage and for I don't know how many of you know we announced a data bricks managed storage offering called default storage that works with serverless workspaces. This solution would also work with the default storage capability and it will support all the patterns right like that we have all of this is coming as part of the private preview in this quarter. So what's coming after the private preview? what is it that we are not delivering uh right now but we plan to deliver soon after.
The first is the performance improvements that I just mentioned especially um uh like um around certain uh certain sharing patterns right uh we would have performance improvement. The second is the initial private preview does not support private link to your own storage and that would come right after that. So if you are configuring private link to your storage that would come post Q2. Further today all cost attribution is basically coinrain right like the as a provider all the any networking cost is acred to you and you have to figure it out as to how you're going to do the attribution per recipient.
But here we will the solution will have per recipient cost attribution baked in particularly for the D2D scenario. All right. So we will be you will be able to understand what cost is in incurred by each recipient from the platform itself. And lastly when we do cross region sharing as I mentioned we will be able to optimize your price performance aspect of it. Right? So we will figure out ways to reduce cost and there are different patterns based on are you going from uh you know US east one to US East 2 versus are you going across the Pacific Ocean right so there is not one solution but we are we will provide the optimal solution depending upon what that um scenario entails and that will just be all transparent to you but you'll just uh um see a better price performance as you as we move forward.
So um so this is uh you can this and uh with this uh we'll just uh pause here and see if there are any questions from your side and we would be happy as I said it's a small group we want to keep it interactive and we'd be happy to take questions. I'm going to go back to this signing sign up uh QR code if anyone is interested.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.