Governed Data Sharing & Collaboration at Scale: Deep Dive On Security Best Practices

Summary

This video presents a three-pillar architecture for enterprise-scale Delta Sharing: OIDC token federation for secretless authentication, Secure Connect for simplified network setup, and Clean Rooms for multi-party analytics without exposing raw data.
Presenters demonstrate OIDC federation with Power BI and Entra ID, a single-VPC whitelist via Secure Connect, and cross-cloud Clean Room provisioning between Azure and GCP.
A zero-trust reference architecture backed by Unity Catalog lineage and audit logging shows how organizations can scale to dozens of external recipients without creating additional data copies or one-off network exceptions.

Governed Data Sharing & Collaboration at Scale: Deep Dive On Security Best Practices

Sharing data openly shouldn't mean sharing it insecurely. As enterprises scale Delta Sharing to hundreds of recipients across clouds and platforms, security and governance architectures quickly become the bottleneck. this video dives into how Databricks enables governed, zero‑copy collaboration using open Delta Sharing, Clean Rooms for multi‑party analytics without exposing raw data, and Private Exchanges in Databricks Marketplace for curated partner ecosystems. We'll tackle the three pillars of secure sharing at scale: OIDC token federation for secretless auth with external identity providers, attribute‑based access control (ABAC) for fine‑grained row/column policies, and the Storage Network Gateway for one‑time network setup that eliminates manual firewall/IP allowlisting. You'll see a live zero‑trust reference architecture, backed by Unity Catalog lineage and audit logging, powering cross‑cloud Databricks‑to‑Databricks and open recipients, plus monitoring. Expect code, configs, and field‑tested patterns, including a security checklist, ABAC policy templates, and an OIDC integration guide you can apply Monday morning to scale governed collaboration without replication.
Talk By: Bhavin Kukadia, Work hard, have fun, make money, Databricks ; Krishan Kumar, Sr. Solutions Architect, Databricks ; Michael Saltzman, Specialist Solutions Architect, Databricks ;

Chapters

FAQs

What is Delta Sharing and how does it enable secure external data sharing?

Delta Sharing is an open protocol that lets a Databricks provider share live data and AI assets with recipients outside the organization, regardless of what compute platform the recipient uses. This video shows how it supports both Databricks-to-Databricks sharing and open recipients that consume data through non-Databricks tools.

How does OIDC federation work for Delta Sharing authentication?

OIDC federation replaces long-lived shared credentials by allowing external identity providers, such as Microsoft Entra ID, to issue short-lived tokens that authenticate sharing recipients. This video includes a live demo using Power BI and Entra ID to show how secretless auth works end to end.

What is the Databricks Secure Connect network gateway and how does it simplify firewall setup?

Secure Connect enables a one-time network configuration so that all external sharing recipients can reach data through a single VPC endpoint rather than requiring per-recipient firewall or IP allowlist changes. This video explains that it significantly reduces operational overhead when scaling to many external recipients.

How do Clean Rooms work in Databricks for multi-party analytics?

Clean Rooms allow multiple parties to run analytics on combined datasets without any party seeing the other's raw data—only agreed-upon outputs are shared. This video demonstrates cross-cloud Clean Room provisioning across Azure and GCP, including code execution and output sharing.

Full transcript

[00:08] Good afternoon everyone and thanks for joining us. My name is Krishan Kumar. I'm a solutions architect here at data bicks and today I'm joined by Michael Saltzman in Puff and Cucadia. Over the next 90 minutes we're going to walk through a very practical topic. how to
[00:21] share data outside your organization in a way that is fast enough for the business but still governed enough for security compliance and platform teams to support at enterprise scale. The reason this topic matters is that
[00:33] most organizations no longer struggle with whether they should share data rather they struggle with how to do it without creating more copies more credentials more network exceptions and more one-off operating models for every
[00:46] partner. That is the tension that we'd like to address today. We'll frame the talk around what we are seeing in the field with customers who are trying to scale govern data sharing and collaboration across organizational boundaries.
[01:02] So here's a road map for the session. I'm going to start with a really quick primer on delta sharing so that everyone is aligned on the basic model and vocabulary. Then I'll use a realistic enterprise scenario to show where data sharing starts to break down once you
[01:15] move beyond a handful of recipients. From there, we'll organize the rest of the talk around three pillars. Identity and access, network connectivity, and policy control collaboration. Each pillar corresponds to a real
[01:28] failure mode we see customers run into when they try to scale external sharing under a zero trust mindset. I'll cover the first pillar, identity and access, and then take you into an OIDC federation demo. After that, Michael
[01:41] will pick up the network and data protection story and Pavven will close out with collaboration patterns like clean rooms. The important thing to notice here is that this is one continuous story, not a set of
[01:53] disconnected features. The goal is to show how these pieces work together as an operating model for secure sharing at scale. And of course, we'll leave some time for questions at the end.
[02:08] So, at the foundation of this talk is delta sharing. Um, delta sharing is an open protocol for secure data sharing across organizations regardless of which compute platform the recipient uses. In data bicks, it lets a provider share
[02:20] live data and AI assets with users outside the organization. And in the open sharing model, it lets non-data bicks tools and platforms consume shared data without requiring the provider to copy that data into a separate delivery
[02:33] system first. The key architectural point is that the provider keeps control of the shared asset while the recipient addresses the current version of that asset through the sharing protocol. And that is why
[02:47] this model is so attractive compared to the older export and send pattern. Instead of building a brittle chain of extracts, file drops, and duplicated tables, you expose govern data directly from the platform that already manages
[03:00] it. The result is lower latency, fewer redundant copies, and a cleaner operational model for both the provider and the recipient.
[03:12] The simplest way to understand delta sharing is to keep the provider recipient model in your head. So a provider is the organization that owns the data and decides what gets shared. A recipient is the organization or user
[03:24] that gets access to that share. And a share is the readonly collection of assets that the provider exposes. In the databicks model, the provider creates the recipient object, creates the share, and then grants the recipient
[03:38] access to that share. This sounds pretty simple, but it is the control plane for everything else we're going to discuss today. The reason why this matters is that every security and governance decision in a sharing program lives on one side of that relationship or the
[03:50] other. The provider owns what gets exposed and under what policy. The recipient on the other hand owns how they authenticate and how they consume the data. When teams confuse those responsibilities, that's when you start
[04:03] to see bad designs, shared credentials, unmanaged copies, and unclear ownership of controls. So, as we go through the rest of the talk today, keep asking yourself, is this a provider side control, a recipient side control, or a
[04:15] trust boundary between the two? Once you understand that provider and recipient model, the next decision is which sharing pattern you're actually going to use.
[04:27] If both sides of the equation are on Unity catalog enabled data bricks, the default pattern is data bricks to data bricks sharing. That path gives you the tightest integration with Unity catalog along with native governance, auditing
[04:40] and usage tracking. It also supports capabilities including notebook sharing, unity catalog volume sharing, model sharing and other data bricks native governance and management experiences.
[04:52] If the recipient is outside data bricks, then the right pattern is open sharing. Open sharing is designed for tabular data managed in a unity catalog enabled data bicks workspace and it lets that data be consumed from any computing
[05:05] platform. That's the path you'll use for partners, customers, regulators, external BI tools, or custom applications that are not running on a Unity catalog enabled data bricks workspace.
[05:21] Okay, so now that we've established some foundational context to make our discussion today a little bit more concrete, let's anchor this in a realistic operating model. So imagine a global bank that already has 42 external recipients that live across multiple
[05:34] clouds and now needs to onboard number 43. The recipients include partner banks, regulators, subsidiaries, and vendors. The data being shared includes things like risk data, transaction analytics,
[05:46] and other sensitive operational data sets. The point of this example is not the exact company, but it is the operating pattern. Once you reach the scale, every new recipient becomes a test of whether
[05:58] your sharing model is actually repeatable. And that's a real enterprise challenge. The business wants the next partner alive in days. Meanwhile, your security team wants identity, network, and policy controls that do not weaken posture. And
[06:11] at the same time, platform teams want to avoid copying data and building bespoke plumbing for each new consumer. If your model requires a new credential handoff, a new storage exception, and a new table copy every time a new partner shows up,
[06:25] you don't really have a true sharing platform. What you have is a sequence of custom data sharing products. And what we see in practice in the field is that sharing programs usually break in the same three places. So, let's
[06:37] start with the first challenge, identity and access. Imagine you're this bank and you want to share data with another bank or maybe with a travel company that's a partner. The first issue is that they might not
[06:49] use data bricks that creates immediate integration friction. On top of that, they don't want to handle secrets, tokens or API keys and neither security team wants the burden of managing longive credentials. Most traditional
[07:02] approaches require some form of tokenbased connection and that quickly becomes a problem. Security teams don't want to create longive tokens and they definitely don't want partners holding longive credentials either. So in essence, the real problem is this. How
[07:15] do you make your data sharing easy for external recipients without introducing credential sprawl or weakening your security posture? The second challenge is network connectivity. Customers want to share
[07:28] data but they don't want to copy it. If they want to share it directly from the source from the storage bucket where it already lives, but at the same time that storage layer is often highly locked down. um it should not be able to you know be
[07:42] accessible to the public internet and in many cases the security requirement is that nobody should be able to read from it directly without strong controls. So now you have a real tension here. You want to minimize data duplication and avoid creating more copies of sensitive
[07:55] data but you also need that data to be accessible to approved partners through your data sharing protocol. So the question is how do you preserve a strict network security posture while still enabling direct sharing?
[08:08] And lastly, the third challenge we see is how to collaborate on sensitive data when multiple parties are involved. Think about two banks that compete with one another, but they still might have a shared interest in improving fraud
[08:20] detection. Or think about a bank and a travel partner working together on targeted credit card offers. In both cases, there's clear value in combining signals across organizations, but neither party wants to expose its full
[08:32] underlying data set to the other. So the rest of this talk is really the answer to those three breaks. So firstly, the solution starts with identity and access. Instead of relying on long live secrets, tokens or API
[08:46] keys, the approach is to enable secure external sharing in a way that reduces credential management overhead. For open sharing scenarios, OIDC Federation addresses that problem directly by letting organizations avoid the token
[08:59] sprawl that security teams dislike. And for data bricks to data brick sharing, authentication can be managed directly within data bricks. That gives customers a cleaner, more secure way to connect the right recipients to the right data.
[09:12] Secondly, for network connectivity, the solution is to make direct sharing possible without forcing customers to create unnecessary copies of the data or weaken their security for their storage. Customers can keep data in its existing
[09:25] source location, preserve a lockdown storage posture, and still make that data accessible to approved partners through secure sharing patterns. That means they don't have to choose between governance and efficiency. They can reduce duplication while still enforcing
[09:38] strong controls around how the data is accessed. and we'll show you how that can be done through secure connect. And lastly, for controlled collaboration, the solution is clean rooms. Clean rooms allow multiple parties to contribute
[09:50] sensitive data into a controlled environment where analysis can be run on the overlap without either side exposing its full data set. That creates a model for collaboration that is both useful and tightly
[10:02] governed. Customers get the ability to work together on shared use cases while still controlling exactly what analysis can be run, what workflows are permitted, and what each participant is allowed to see.
[10:14] So, let's start by going deeper on the first pillar of identity and access. The core argument here is that longive tokens do not scale as the foundation for external sharing. They create operational overhead. They increase
[10:26] security risk and they make deprovisioning and life cycle management harder than they need to be. In a mature enterprise model, you want identity to remain with the organization that owns that identity perimeter. So in other
[10:38] words, the recipient's identity provider should be responsible for authentication, token lifetime, MFA, and access life cycle, not the provider minting and distributing longive
[10:50] credentials. And that's the architectural shift that we're making here. We are moving from a shared secret model to a federated trust model. And once you make that shift, the conversation gets much more aligned with how enterprise security teams already
[11:03] think about external access. So here's the important technical detail. OIDC federation is for open sharing. Um
[11:15] if both sides are on unity catalog enabled data bricks, data bricks to data brick sharing already handles identity through the data bricks in unity catalog model. OIDC federation is specifically for the case where the recipient is
[11:28] outside the datab bricks model and you still want a secure modern authentication path. In the OIDC federation model, the recipient uses their own identity
[11:40] provider to authenticate and their IDP issues the JWT. Databicks does not generate or manage that token. Databicks instead validates the presented JWT against the federation policy including
[11:52] claims such as the issuer, the audience and the subject and then grants access if the claims match and the unity catalog's permission allow it. There are a few reasons why this is such a strong model for enterprise sharing. First is
[12:05] that MFA and token lifetime stay within the recipient's IDP. Second, you can use this for both user to machine flows such as BI tools like PowerBI or Tableau and for machine to-
[12:17] machine flows such as applications using Python or Spark clients. And third, no longive data bricks issued credential needs to be exchanged between the provider and the recipient. Now, that means less secret sprawl, clearer
[12:29] ownership of identity controls, and a much cleaner revocation story because disabling the user or the service principle at the IDP shuts down future access at the source. So, the main
[12:41] takeaway here before I show you a demo of this is that with OIDC Federation, identity stays where it belongs with the recipient. The provider is the one that defines the trust policy. The recipient authenticates with their own enterprise
[12:54] identity system and data bricks validates that trust boundary and enforces access to the share.
[13:10] Okay. So now let me show you what this actually looks like in practice by onboarding an external recipient with OIDC federation and then consuming the share from the recipient side. So what I want to do now is show you the user to
[13:22] machine flow end to end following the exact operating pattern many of you will see with an external BI consumer like PowerBI. So the scenario here is straightforward.
[13:46] So let's say the provider wants to share aggregated loan performance data that's been deidentified with an external organization and that recipient wants access to happen in PowerBI using identities managed in their own intra ID
[13:58] tenant rather than through a long live data bricks issued credential. And let's actually start with the recipient side conceptually because this is where the trust model begins.
[14:12] So before the provider configures anything, the recipient needs to identify who should be allowed to access the data from their identity provider.
[14:24] provide to the data provider. So one of those being the tenant ID and the second being a group ID for the group of users who should be authorized to access the share. The tenant ID you would grab from the Entra admin center and the group ID
[14:37] we'd grab from here in the Azure portal. So you'll see that we have various groups here of users that should have access to PowerBI. And what you're looking for is this the group ID.
[14:50] Um and then switching to the provider side I'll show you how to configure the share itself first. So, if we go back to our data bricks workspace and we look at our catalog
[15:02] here, this is our aggregated loan performance data that's been um anonymized. We can go ahead and create a share via open sharing.
[15:14] We'll create a new share with the table.
[15:29] And for now, we don't need to add the recipients just yet, but this will create the share object. Okay, awesome. So, we see that we created the share. Let's go and navigate to it.
[15:44] All right. So, once that basic sharing object model is in place, I'll create a recipient for the share and then I'll create a policy for that recipient.
[15:58] So, right now we see we don't have any recipients yet, but let's go ahead and add one. We could either select an existing recipient or we could create a new one. Let's go ahead and create a new one.
[16:10] For a recipient, let's just call it, you know, PowerBI users. We're going to select an open recipient type because as we talked about earlier, we're trying to share this data set to a
[16:23] consumer that's not on a databicks workspace. And then for authentication, we're going to have the muse OIDC federation. So let's go ahead and create and add that recipient.
[16:49] All right. So now that we have this, let's go ahead and create an access policy for our um recipient. So we'll go here. We'll go ahead and add the policy.
[17:04] And this is where we're going to leverage that information that the recipient had provided to us earlier. So if you recall, I had said that the recipient will provide the tenant ID and then the group ID for the users that
[17:16] should have access to this. So let's go ahead and get that for our users.
[17:43] And now we're going to grab the group ID, which was that object ID we saw in the Azure portal
[18:03] for audiences. is for all PowerBI users. It's actually the same. You'll see that we provide it here in the UI. If you were sharing to something like Tableau, the audience ID would be different. Um, and that's something you can pull from the docs. So, now that we
[18:15] have this, we can go ahead and create this. Awesome.
[18:37] um system has generated an OIDC profile endpoint URL and this is what I'm going to send to the recipient. So let's go ahead and take a look at this OIDC URL that was created.
[18:58] And this is what we're going to use in PowerBI. So let's go to PowerBI where our consumer is. Um so let's say I'm an authorized user in the recipient organization. What I'm going to do is
[19:11] start by creating a new connection using the Delta sharing data source in PowerBI. So we do that by going here. We can search for delta sharing as a source.
[19:31] And the URL that we got earlier is what we'd supply here. Let's go ahead and name this connection. And then here is where you would select that authentication needs to happen with
[19:44] OOTH. And let's go ahead and sign in. And this is really the crit critical part here because authentication is being federated to the recipient's own
[19:57] identity provider. What you'll notice is that I'm not entering a datab bricks credential here. I'm authenticating with organizational credentials that are governed by my own entra ID policies. And if the recipient wants to have MFA
[20:10] enabled, that policy is enabled here as part of the signon flow as well. And that's exactly the behavior you want in an enterprise model because token lifetime and authentication policy remain under the recipient's control
[20:22] rather than being outsourced to a shared secret distributed by the provider. So let's go ahead and create this.
[20:41] And once authentication succeeds, PowerBI can now access the shared data set and I can immediately begin interacting with it as a recipient user. So let's go ahead and see if we can see the data that was shared with me.
[21:15] And voila, we can see that here the external user is quering the shared data through the governed delta sharing path, but the identity assurance is still coming from the recipient's own IDP. So the three things I'm emphasizing here are these. So first, the provider still
[21:28] controls the share and the federation policy. Secondly, the recipient controls the authentication through their own intra ID tenant. And thirdly, the consuming experience in PowerBI is simple for the end user, but underneath
[21:41] it, you still have a modern zero trust identity model with shortlived tokens and enterprise policy enforcement. So that's the practical value of OIDC Federation for open sharing. It gives you a path to onboard external
[21:53] recipients without falling back to long live credentials while still keeping the experience natural for the business user consuming the data.
[22:13] And with that, I've shown the identity side of the story, how an external recipient authenticates using their own IDP and consume shared data through PowerBI. And next, Michael is going to build on that and cover what happens when you scale this beyond identity into
[22:25] the network and storage path itself. >> Hey everyone, uh my name is Michael Saltzman. I'm a specialist on our cloud infra and security team.
[22:39] two which is around network connectivity. So think about a hardcore problem where you have hundreds of recipients
[22:51] crosscloud multicloud that you want to manage and scale long term. So that's when pillar two often breaks
[23:03] down in a lot of enterprise scenarios when it comes to network connectivity. So let's think about a traditional pattern
[23:15] for Azure to AWS uh delta sharing. So in a traditional pattern of Azure to AWS delta sharing you you're sharing data across clouds
[23:29] you have to configure potentially crosscloud firewalls NAT gateways and um to configure the connectivity between Azure and AWS
[23:41] and oftent times if you have lots of lots of different clouds you know Azure GCP cross AWS best connectivity, you're going to have to potentially whitelist tons of IPs,
[23:56] right? Especially if your security team has wants to put very secure policies around your buckets. So that's where we're we announced
[24:11] uh secure connect or actually the new yeah delta sharing secure connect which is in public preview and this is our solution to solving that
[24:25] problem. Instead of whitelisting individual resources, individual IPs, everything goes through our secure
[24:37] connect gateway. And instead of having to whitelist individual IP addresses, you only have to whitelist one set of IP addresses and we secure and take care of
[24:51] the rest. So that's what we're coming out with Delta Sharing Secure Connect where you can have as many recipients as you want and you're going to whitelist one set of
[25:04] IPs on on the provider side.
[25:17] We're also announcing secure connect with NCC sharing or network connectivity configuration sharing where you're going to be able to connect to your provider using VPC
[25:30] endpoints or private link and be able to share that private link that private connectivity to your recipients. That way you can always govern access
[25:43] with your NCC or your network connectivity configuration. You're just whitelisting one VPC endpoint and everything else is taken care for you. You go through our secure connect
[25:57] gateway and everything is taken care of. So that's our goal is to eliminate this manual configuration of buckets
[26:12] throughout our platform. We also announced sharing for lakehouse federation where if you have if you have any data
[26:27] in PowerBI in snowflake in Redshift in SQL you can securely share data from those tables
[26:41] and and you can do share crosscloud to other to other recipients within datab bricks. With this,
[26:53] you know, with this, we're allowing everyone to share data securely. Even when your data is isolated into
[27:05] like a data warehouse like Redshift, you can easily share your data across data bricks. >>
[27:18] we are also enabling we are also announcing tons of features today at data and AI summit and the main features that we're that I
[27:30] want to talk about is number one crossorganization genie agent sharing genie agent sharing is going to allow you to share AI I agents securely share
[27:44] definitions of your agents so others can use them. In addition, we're also ganging soon metric views and agent metadata sharing
[27:58] and GA of MCPs in the datab bricks marketplace which uses delta sharing under the hood to share MCPs to you.
[28:10] So we have a huge marketplace of MCPs created by companies and they can there we are going to be allowing we we have GA support today for
[28:25] MCP sharing from the marketplace also coming soon and part of the keynote announcement was um we're going to have MCP server sharing where you can create your own
[28:38] MCP server and share through delta sharing. And then part of what we talked about at the keynote was the concept of an ontology. So now not only can you share data, but
[28:51] you can share the context around that data, right? You know, think about like a confluence page with all your contextual information in it that tells the AI how to best interact with the
[29:03] data. You can share that contextual information, that ontology information directly in Delta Sharing. And coming soon, we're also going to be coming out
[29:15] with agent skill sharing. And to unlock applications, we are announcing a preview of data bricks app marketplace where companies
[29:29] can create their own apps and share with you also through delta sharing. And so these are to summarize these are the themes of investment that we are
[29:42] working on. On top of this, what we want to enable with delta sharing is to be able to share any asset with anyone.
[29:54] And so to support that, we announced the general availability of managed iceberg sharing and foreign iceberg sharing where where you're going to be able to
[30:06] share data to Snowflake for example um fully fully GA uh foreign iceberg sharing.
[30:19] announcing GA of the Lakehouse Federation sharing that that um that I talked about earlier. And then um and then also um like we talked about in the keynote,
[30:34] Lake Base and your um and your Delta Lake are going to be one unified layer. And to support that, we're going to be allowing um that we already have in
[30:47] preview the sharing of lakebased cataloges and lakebased CDF. Um, and then in addition, and I'm not going to go through all these, but I
[31:00] think one of the most important ones to me and what we're going to be talking about today is the pre is secure connect and global distribution.
[31:18] So to talk about one theme, genie agent sharing. So let's say let's give a scenario where you have a provider that builds a curated natural
[31:30] language interface to their data and shares it. The provider's information stays hidden and instructions, benchmarks and knowledge stores can be
[31:42] shared across using Genie agent sharing. That way you can create your agents and share it across to other customers.
[31:57] And we're announcing the beta of this today. So with Genie agent sharing, you can hide underlying data, hide provider IP addresses, give quotas, rowle limits,
[32:13] and share Genie your Genie agent while protecting the underlying data with fine grain access controls.
[32:25] In addition, we have um and this is a feature close to my heart. We have the private preview of what we call global distribution.
[32:38] Imagine you have data recipients uh across many regions and oftent times let's say you're querying pabytes or terabytes of data
[32:52] from another region. Every single time you are querying data from another region it can increase egress costs. And so what we're announcing with global
[33:06] distribution is going to allow for seamless copies of the data set that you don't have to manage that'll be available in other regions for those use cases where you have large
[33:19] throughput of data queries in other regions. So we're going to copy the data automatically for you in that secondary region so that you can easily query it without
[33:33] without the repeated egress cost every time you're querying the data. So we will take care of the duplication
[33:45] of the data for you. That way, you're not paying any um long-term egress fees. So, that's that's really awesome as
[33:57] well. Now, we're going to be going into a demo on secure connect to show you how this
[34:09] is implemented in practice. So, first I want to start out with my S3 bucket. My security team has a requirement
[34:24] to have very secure S3 bucket policies. And in in the case of secure in the case of a
[34:36] general delta sharing pattern, I'm going to have to whitelist I'm going to maybe have to potentially whitelist other IP addresses or other networks to
[34:50] come into my S3 bucket when I'm sharing it. So what we are enabling with secure connect is the ability to whitelist one set of
[35:04] VPCs or one set of IP addresses if you're cross region. If you're in the same region it's VPCs. You only need to whitelist one set of IPs or one set of
[35:16] VPCs in your bucket and we take care of the rest. The traffic goes through our secure connect gateway from your from your recipient goes through the gateway
[35:29] and then in through the gateway to the provider. The provider only sees one set of IPs or VPCs and that's all you need to whitelist. So
[35:41] the first part is your bucket setup or your storage setup where you're going to whitelist these sets of VPCs or IPs that we will provide to you as part of the
[35:53] documentation. Then now we are going to set up the sharing part.
[36:06] So here, so here I have my provider. It's just a simple delta table that I am that I'm
[36:20] sharing here today. When you share you uh using secure connect, it really only requires one additional step.
[36:34] So, what we're going to do, go into open sharing, shared by me,
[36:47] and I'm going to open up. Actually, let me go back.
[37:40] Okay. So, when you set up the Not sure I'm going to be able to find it because there's so many shares in here, but let me see if I can find it.
[37:52] Oops.
[38:16] So, at a high level, I'm not sure if I'm going to be able to find it, so apologies about that. Um, but at a high level, the way you're going to set it up is when you create your share, you can prefix the share
[38:30] with the share name is going to have a dsng in the name. So that's how you signify um whether a share is um
[38:43] and I think yeah here it is. Here it is. So when you prefix the share, all you need to do is prefix the share with DSNG in the name and that's
[38:56] going to allow it to um to use uh delta sharing secure connect. So that's one option. Or the second option is we're also enabling we're also going to have a UI option where you can uh a toggle
[39:10] where you're going to be able to explicitly um enable um secure connect as well. So, I don't know if that's enabled in my workspace, but essentially all you need to do is enable the um
[39:22] secure connect preview in your workspace and then um and then use this DSNG uh prefix. And then once you put that DSNG prefix, you're enabled um to go
[39:37] through secure connect. Then on the other side, on your recipient side, here's my delta share. So when I set up my delta share,
[39:51] I set it up as normal. But again, on the recipient side, I didn't need to whitelist anything except for the VPC IDs um or IP addresses of
[40:03] the secure connect. And then as long as you see this property here which says delta sharing network gateway enabled that's how you know your delta share is
[40:17] going through the network gateway and at the end of the day the biggest benefit of this is whitelist once
[40:30] whitelist one set of IPs one set of VPC IDs in your buckets and then we take care of the rest through our gateway once you have that set up.
[40:46] All right, so now we're going to go back to the deck. So that's it for the demo and I'm going to pass it on to Bobin in a minute. Um but when you're configuring
[41:01] data sharing you want to keep uh some core principles and best practices in mind. So you can configure IPACL's you can configure what what what we were
[41:13] talking about earlier right with OIDC federation sharing we also have audit logging and network restrictions you can enable on storage accounts that's what we recommend like S3 bucket policies
[41:26] storage policies um and then we also you also can configure logging on the storage account itself um so like for example S3 It has S3 access logs. So you can see every
[41:40] time someone accesses my bucket, I have those logs. So these are the core principles you want to keep in mind anytime anytime you're sharing data to
[41:52] keep your data secure. And now I'm going to pass on to Bobin for a demo of clean rooms.
[42:06] All right. Okay, I got this. I have it.
[42:25] Awesome. All right.
[42:38] brick system tables all all public clouds any workspace any region globally when you get your system tables data that's powered by delta
[42:51] sharing so that is a pretty awesome use case but how does that data get generated we also need to do lot of ETL to get the right data that is powered by
[43:03] Spark declarative pipelines. So the entire feature is built on data packs, right? Awesome. So we're going to top
[43:15] uh this session with clean rooms. How many of you have used clean rooms or have heard of clean rooms? Oh, nice. Uh pretty cool. All right. So what I have in mind is I've got like
[43:27] eight or nine slides. You already seen enough slides and you have already seen lot many announcements and all. So I don't have any announcements. Michael stole the show. He made all the
[43:39] announcements. We're going to do some live uh demo. Hope it works out. And uh I also have a backup if that doesn't work out. U so continuing with the hypothetical
[43:54] use case that we had right that we have some uh different banks they want to collaborate I'm going to piggyback on that u imagine in a bank we have two different teams
[44:07] one is risk analysis and the other one is private client management risk analysis team flags certain transactions these are wire transfers
[44:21] They don't know that the client for whom these wire transfers were flagged is a private client. So they get a different kind of treatment because if there is a
[44:33] transaction that is being flagged, you need to take immediate action based on the client relationship you have. Now given both these teams are separate. How do we handle this? There are different ways to do it. You can use delta sharing
[44:47] or you can have clean rooms. The one of the unique differences you know lot many times people do ask we have delta sharing we have open to data bricks data bricks to data bricks uh
[45:00] what is it what is the difference between clean room and any type of uh data sharing that we do. Uh in its simplest form uh when you have clean rooms you are sharing results.
[45:14] If you were to understand you are not sharing just data. So in our case the risk analysis team is going to share some result that they have bunch of transactions being flagged on the
[45:28] private client side private client relationship management team. They are going to share who that client is and they both come together to produce a result that immediately locks that
[45:41] account because there are bunch of wire transfers. Does it make sense everyone? Perfect. All right. You are at the data and AI summit. Let's let's cheer up a
[45:53] little bit. Right. Awesome. So, just some background. Uh this is my probably 10th summit. I joined 2017. This is my fourth time in a row we are doing this
[46:05] session delta sharing. So after the session if you have any question right now between us three we are part of the platform team any question on any three clouds network security encryption
[46:19] identity access management we are more than happy to address all the nonAI stuff we do so we don't know much about other AI stuff and all but hardcore networking and all that's our strong
[46:32] suit awesome all right So, I'm going to get my eyes. So, I've realized it's not clear in the
[46:44] back. So, I'm going to zoom in a bit once we get into the uh into the uh mid part. Now, what exactly is clean room? It is built on uh open sharing. That's the
[46:58] rebranding of delta sharing. So, if you hear open sharing, that is nothing but delta sharing. Okay. uh uh there are a bunch of things going on over here. First of all, think of clean room as a
[47:11] managed environment that is being provisioned by data bricks between two collaborators. So in our example, who are the two collaborators? We have the risk team and we have the private client management team. They both are on some
[47:25] databris environment. They could be on any cloud. So in our case the risk team is on Azure and the private client management team is on Google. They both are on datab bricks. Right? Perfect. Now
[47:38] they want to collaborate. What we are going to do is create a clean room. So what exactly is a clean room? Clean room is nothing but a barebones headless workspace that data bricks provisions. You don't get to see the UI or anything.
[47:52] It is just an isolated environment. So we create a room between these two collaborators and you can have multiple collaborators. So there is no uh restriction like you can have one collaborator. I think right now the
[48:04] limit is up to 10 and our platform limits keep scaling up. So if you run out of let's say you need more collaborators you want to talk to our product our engineering we can facilitate that conversation. So you
[48:16] create a clean room it's an isolated environment.
[48:28] management team going to bring their data but you can't do much with just data. You also need to have some sort of agreed upon code execution, right? Because you want to run some joints, you want to uh pull some results out, you
[48:43] may want to write some results back, right? So those kind of things. So in the middle the next step that we have is there is a mutually or a common understanding of what code
[48:55] gets executed. So why do we have this? This is uh once you have collaborators into a room, none of them have any super privilege. They all are same. Even the
[49:08] ones who created the clean room, right? So let's say if I were to create a clean room, that doesn't give me any more permission than the collaborator. Everyone has same permission. So the
[49:20] code comes in the in the clean room. There is no direct access between the parties. So earlier Michael was talking about secure connect network connectivity all those things right
[49:32] that assumes that there has to be some connectivity from the recipient's compute to the provider storage in this case everything is shared right and piggybacks on all the things that
[49:44] Michael was talking about earlier right and perfect is the secure foundation okay self-explanatory so with this in mind let us create a clean
[49:59] All right. So here, let us zoom in because I realized I went back and it's not clear enough. I'm going to just refresh it. Okay. So we've got Azure datab bricks
[50:11] workspace and we've got Google datab bricks workspace. Okay. Uh just by show of hands, how many on Azure? How many on Google? AWS?
[50:23] Couple of Azure. Google. Very nice. Very nice. Perfect. All right. So, let's reset this. So, on the Azure side,
[50:37] let me go to catalog and we're going to create a clean room. I already have one,
[50:53] right? And you pick a region where the clean room is going to get created. So you can pick a region. There are a couple of regions that are option available. Right? You got to pick one region. Let's go ahead and use east. Now the sharing identifier. Okay.
[51:15] So we go here.
[51:46] right. So, it's going to invite me and you can lock down the egress. So for example when you are sharing collaborating data if you want to lock down the egress so that you know you don't write something out unknowingly you can do that. So that's that's the
[51:59] option over here. So we're going to keep this or you can have full internet access. So you may ask in what situation you need internet access. Any any guess in what situation you may need? You are sharing data. So in what situation you
[52:13] would need to go over the internet to get something? uh in case if you have to download a library so you you are not when you create a notebook that doesn't mean you
[52:26] run only joints you can have some sort of machine learning model that you both are training together we do have use cases where there are proprietary data on both sides and uh customer wants to
[52:40] create certain models that uses data from both sides and that may require sometimes you have to go like pi pi packaging and download some library. But yes, any any use case that requires
[52:52] you to go over internet but we're going to just keep it simple restricted right and let us create this.
[53:05] So under the hood datab bricks is provisioning the environment for you. It's a serverless workspace. It has serverless compute. It will be in the east US region. And from here onwards we will add couple of tables. Both parties will bring their own tables. We'll have
[53:18] a notebook where we agree upon certain piece of code and we will run that code. The output of that code is shared back to the collaborators. Make sense? Right. Perfect. All right.
[53:33] So this is good. It's creating its environment. So I'm going to go back while it creates. So we have this over here.
[53:50] Yeah, I have some some internet issue. I don't know. It just takes forever to load my my load book. So, while it loads, let's go back to the deck. So, this is what we are doing right now. This middle piece is being provisioned.
[54:04] some of the use cases that I was talking about. Not only you can uh collaborate between your partners but it can enable all types of use cases. There is no limit uh to the use cases over here. Uh
[54:17] some of the known ones of course finance is something that we are discussing right now. Uh advertisement and retail are pretty big and uh health and life sciences. Right. So the architecture we already
[54:30] went through the approval process going to show up immediately. uh right now once the clean room is created and then it is backed by data sharing and we'll wrap it up with certain security aspects right
[54:44] now one unique thing about clean room is it is backed by APIs that means imagine you are building an app and that app needs data from clean room or you are building
[54:58] some sort of UI that provides a a layer on top of your clean room. We do have use cases like this where uh uh third parties they are building apps, they are
[55:11] building data products and they want to have a UI like a co-branding solution. So in that case you can access clean room and all its services through APIs and have an app on top of it and with
[55:24] that you can also have OIDC based access and all. a token exchange what uh Christian was talking about earlier right so again everything is exposed through APIs that you could use
[55:41] right now again the trust level everyone is untrusted in a clean room they need unless both parties approve there is no unilateral execution that if I have created a clean room I can execute whatever code I want both parties need
[55:54] to approve the code and then you also have to designate who can run it, right? So you can have like both sides can run it or you can have one party run that code. So you can do that as well.
[56:07] Okay, now let's go back to the
[56:20] I've realized I refresh it and it works. So let's see. Okay, there it is.
[56:34] So, we got the clean room. Let us go back over here and refresh. And you can see that I have uh clean room 2 on the Google side, right? So far
[56:48] so good. Any any questions? Anything?
[57:01] Yeah, absolutely. Yes. Uh uh specifically in crosscloud. Uh so what Michael was talking about egress cost uh you have use cases where you cannot have crosscloud interconnect. So
[57:14] the question is are there any use cases within a single company same company what are the use cases for clean room right let's say we have and we do have lot many times companies they acquire other companies and by the time you
[57:26] integrate you need to get started right the reason why you have all these acquisitions or you come together is to uh not only consolidate but also utilize
[57:38] and you know uh exploit the data or information that you Clean room is the quickest and uh safest way to get started between uh single company cross cloud same cloud different
[57:51] region or different geos. So sometimes in same cloud you have an international acquisition uh you want to do uh this kind of stuff where you can't uh transfer data. So you create a workspace
[58:04] in a known region, clean room in a known region and you would enable that kind of collaboration because the output is written uh not in any collaborators output is
[58:16] written in a clean room storage and it is shared back. So the rule uh unless you have approved a piece of code that says I'm allowing you to write to some storage by default
[58:30] when you run uh run a clean room and if there is some output it is written into clean room storage and it is delta shared back with both collaborators. That is what is happening in this case.
[58:43] The risk team shares their data. The private client brings their data. They learn some joint jointly approved code and the output that says take immediate action. Don't wait like normal
[58:55] escalation. Don't wait for normal escalation. Block all these transactions. That result is shared back to both. That is what we are doing in this case.
[59:13] Okay. So now this is Google. So I'm going to go back over here. Add assets. And we are going to go over here and here's the uh risk signal and I'll
[59:26] show you the data. It's it's pretty simple. Uh if you go back and so this is how the
[59:45] yeah the moment it comes up we'll see the data. So this is how the uh uh risk signals look like right pretty simple. There are couple of wire transfers that we are flagging. This is a very trivial uh you know data sample but uh it helps
[59:58] us drive home the point that you have couple of wire transfers right that are being flagged to certain countries that this client normally doesn't do any kind of uh transactions. Okay.
[60:13] And similarly on the on the Google side we have
[60:29] the relationship context where it just shows like uh who the client is and uh what's going on right now. Imagine on both sides you have different network, different types of compute, different types of storage, all those
[60:41] settings, right? Right now you are making GCS talk to ADLS that is what is happening. Okay. So here are a couple of now in our case the customer is in platinum segment. So they get a
[60:54] different treatment that is what we're trying to highlight. Okay. So now over here in the Azure I'm going to go ahead Okay, let's go back here first. Let's
[61:07] see what did we add. So, we have the table and I'm going to add a notebook
[61:28] right? And you can pick the runner. So, I'm going to be the runner. It takes few seconds over here in on the private client service management side.
[61:41] Going to go to clean rooms.
[61:59] add one more table that is their side of the story. the relationship so far so good. All right.
[62:16] >> The the Aback right that we have to replicate it is not automatically propagated. So when when we share any data uh in a clean room the assumption is it's a
[62:30] trusted partner with whom you are sharing uh the row and column level security and all and Michael do correct me if I'm wrong right because he is my our aback
[62:42] policy guy so uh so in that case no you have to replicate it right it's good but we are not there yet it it is Good. But we are not there yet.
[62:55] Right. So yeah. So here we have on the Azure side. Let us refresh it because we added the private client relationship uh table. So
[63:08] now you can see so we have uh anyways the question was if we are using row and column level security right so you can have sensitive fields being masked you can have row filters
[63:20] and all would they be propagated across all collaborators right Michael do you want to add anything >> yeah so the reason why we don't replicate them is if you think about a
[63:32] scenario where you have two companies company A and company B and company A who sets up the share um let's say I have um on a column let's
[63:46] say I say for this specific group of users uh a column is masked let's say that group has 10 users in it when I share the data to company B they
[64:01] have no idea who those people are right so so yes you can share share. Um, you can share tables that have
[64:13] attribute-based access controls, but when it gets to the other side, they won't that you won't see those um like encryptions. But if you want to the way to share data
[64:28] the proper way to share um let's say you don't want the other side to see a certain column the best way to do that is to create a view and then share the
[64:40] view of that data. >> See I I I love Michael. He is very calm you know
[64:52] it's like absolute calm. I love that. Yeah. That's why when there is a tough question where we have kind of a product gap, I bring Michael in.
[65:17] needs to approve it. It's not like I just added a notebook and it is accepted. So, we're going to approve it. We're going to run it. We'll summarize and then we'll have some Q&A and then we'll get on with uh I hope is this your
[65:30] last session most of you or there's one more. I love that. Love the fire. All right. So, let's go to GCP
[65:42] and there's the pending message and approve it. Okay, I'm going to review it.
[66:06] Right. Okay. And we have approved it. Now it's the matter of running. But let's take a look at the code. Right. What the code is doing. Actually, it's pretty much just running a join between two records and saying, you know, there's a bit of logic like simple stuff that says yes, this is a platinum
[66:18] account. If there is a uh high-risk transaction immediately we need to escalate it, block it and take some action and that result is written back to a uh to a clean room catalog which is
[66:32] automatically shared. Every time you update the notebook, it needs to be approved and every time the result is written, it is written in a new table so that you have some sort of version
[66:45] control, right? And that is shared back. So that is what this uh notebook is doing now because we have approved runner over here once I run it right it
[66:58] will be returned back the table is written back to a clean so I have I'll show you quickly
[67:12] I already have bunch of runs so I'm going to bring it up I ran and you can see couple of errors because I was trying to fix the catalogs. So when you share uh when you add assets to clean room the uh the
[67:26] prefixes for this for the assets that you have shared changes. Those who have used clean room they may have they may have seen this by that what I mean is when you share any table any schema it
[67:39] has its own catalog. It has its own schema. It has a table in a clean room. The catalog changes, right? Because now it's a clean room catalog. You are sharing it. So that prefix change we need to take care of. It's pretty much
[67:52] automated. But while building this, I made a mistake. So I had to fix it. I it took me three trials to fix that. Right? So that's why you see all these failed runs. But some of the most recent runs are here. Right? During the during the
[68:05] session I ran it and you can see and how the results look like the output which we are hiding back is over here.
[68:21] Okay. Oh sorry not that this one. Here's the output. So every time you run and if there is an output it's written like this. And all this interaction is captured in system tables. You get to see who ran it, what time they ran it, how frequently they ran it. Everything
[68:34] is captured back in system tables and this is uh the output of the clean loop. Right? Okay. So now I can I can get a look at and this is based on
[68:48] the agreed upon code. This is not something automatically generated by data bricks. The notebook's output is captured over here. Now some questions. Uh if my storage is private, how do I
[69:00] access it? You can have private connectivity from a clean room to your storage account. Just so that you know, right? We didn't we didn't dwell into it. So if the risk analysis is stored in
[69:13] ADLS that only accepts private linkbased connectivity, you can have clean room can talk to ADLs through private endpoints. Similarly, you can lock down the egress, right? And because this is
[69:27] delta share, nobody gets their copy. Now once it is shared suddenly the collaborator can run a query and write it to their storage account. We can't prevent that right okay so now I'm going
[69:41] to just quickly jump onto okay yeah I want to highlight this too so if you are using uh datab bricks jobs or workflow clean room can become part
[69:54] of it. So if you have jobs you can integrate clean room as one piece into your jobs workflow. So for example you are doing some ETL there is a step where you need some data from a collaborator
[70:06] you bring the clean room into your jobs workflow you can do that too right uh
[70:18] of course yeah so and I think if a clean room is not used for 30 days it goes away there is a is that there is nothing like that okay Michael says don't don't get into that so so forget that whatever
[70:30] I Right.
[70:45] going through it. If there's something that I haven't discussed I want to bring it up but I think we are almost done with clean rooms. Um I want to just summarize. There are a couple of things that are that are in
[70:57] the works like bunch of private preview features right I think all this deck is going to be accessible eventually before we close this session I want to summarize what we started with and all
[71:09] so we started with open to data brick sharing one of the key tenants IDP integration specifically token exchange right so earlier versions we used to use platform tokens those who have used pad
[71:21] tokens you remember someone generates it forever and nothing lasts forever and uh you know that those are like security risk. Now everything is oath based and we have this whole OIDC based token exchange.
[71:35] Highly encourage you to use that. We do have all types of tools available now automated tools that can flag if you are using platform access tokens or pat tokens. So if you are using it I
[71:47] recommend you run it on your workspace. It would flag if anyone has ever generated which you may have missed and you can revoke it and go ahead with all this you know different oath based features that we have added right the
[71:59] second one the whole network stack a pretty awesome any cloud all three public clouds that we support it also includes open sharing so if you are using open to datab bricks or datab bricks to open we can help you uh the
[72:14] challenge is uh if you want to have end to end private connectivity it may come at a cost because you are using you know private access so that's going to be a bit costly and that is that cost goes to
[72:27] the cloud data bricks has no role to play uh our architecture is shared deployment model we you we utilize your network whatever you have put in place right the secure connect is changing
[72:41] things a bit but lot of customers they are still uh pretty much on classic architecture where you deploy workspace in your network and all. So we are going to inherit good and bad whatever you have put in place we are going to inherit. Okay. And the final piece is
[72:55] the clean room. Uh with that I'm going to rest. Is there anything I missed Christian Michael? Cool. Perfect.
[73:07] >> Yes. Uh it's it's open and we appreciate you all coming together over here. Right. uh largest audience I've seen for data sharing because a lot many times
[73:19] people go to AI and all that stuff.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.