Apache Iceberg and Open Sharing: Multi-Platform Data Governance Without Lock-In

Summary

Databricks Open Sharing now provides first-class support for Apache Iceberg, allowing organizations to share data as native Iceberg tables to consumers on Snowflake, Trino, Flink, and Spark without creating copies or requiring format conversions.
FootLocker, operating 2,400 stores across 20 regions, eliminated data duplication and reduced pipelines by 50% by implementing open format sharing between Databricks and Snowflake, replacing manual SFTP-based data syncs with a governed zero-copy architecture.
Unity Catalog's governance layer unifies foreign Iceberg tables, managed Iceberg tables, and Delta tables through the Uniform format, maintaining one security model and one source of truth across diverse table formats and platforms.

Apache Iceberg and Open Sharing: Multi-Platform Data Governance Without Lock-In

Multi-platform data sharing is now practical. Databricks' Open Sharing (the graduated Delta Sharing protocol) now provides first-class support for Apache Iceberg, allowing organizations to share data as native tables to Snowflake, Trino, Flink, and Spark without copies or conversions. Unity Catalog's governance layer unifies foreign Iceberg tables, managed Iceberg tables, and Delta tables with Uniform format, supporting diverse table formats while maintaining one security model and one source of truth.
FootLocker operates 2,400 stores across 20 regions, generating terabytes of data across Databricks and Snowflake. By implementing open format sharing, they eliminated data duplication, reduced pipelines by 50 percent, and enabled real-time pricing recommendations and allocation intelligence from external vendors. Learn the architecture patterns, federation strategies, and business impact of moving from manual data syncs to zero-copy, governed data sharing.
🤝

Chapters

FAQs

What is the difference between Delta Sharing and Databricks Open Sharing?

Delta Sharing was the original protocol focused on sharing Delta tables, launched approximately five years ago. Open Sharing is its evolution, rebranded to reflect a broader mission of sharing any data asset—including Apache Iceberg tables, volumes, AI models, and notebooks—with recipients on any platform anywhere, including Snowflake, Trino, Flink, and Spark.

How does FootLocker use Databricks Open Sharing in practice?

FootLocker operates 2,400 stores across 20 regions and previously relied on SFTP-based pipelines and manual data syncs to share data between Databricks and Snowflake, leading to duplication and high pipeline maintenance costs. By implementing open format sharing with a single-hub architecture, FootLocker eliminated duplication and reduced their pipeline count by 50%, while enabling real-time pricing recommendations and allocation intelligence.

What is the Uniform format and how does it enable interoperability?

Uniform is a format feature in Databricks that enables Delta tables to be read as Apache Iceberg tables by external systems, eliminating the need for format conversion or data copying. This video explains how Unity Catalog uses Uniform alongside native Iceberg table management and foreign catalog federation to present a single governance model across multiple table formats.

What are the common enterprise use cases for Open Sharing?

This video describes four common use cases: peer-to-peer sharing with a specific recipient, publishing data products to the Databricks Marketplace for one-to-many distribution, cross-region and cross-business-unit collaboration with governance applied consistently, and federating system-of-record SaaS applications such as Salesforce CRM or unstructured content from Confluence.

Full transcript

[00:08] Awesome. So, I know it's the last day of the Data AI Summit, and you guys are probably all pretty tired. Maybe you want to go home at this point, but just bear with us for one more session. And I hope you're all in the right place. We're going to talk about iceberg interoperability with open sharing today. My name is Tia, and I'm a product manager at Databricks.
[00:24] This is Balaji, technology leader at Foot Locker, a data analytics space. And let us connect this clicker real quick. Um awesome. So, let's just get right into it. Um I'm assuming that many of you guys are here because maybe you need
[00:39] to share some sort of data or even AI assets within your organization or within your everyday work. Um so, let's maybe just walk through a couple of usual use cases that we see that are super common in in the in the enterprises that we work with. The first one is simple peer-to-peer sharing. So,
[00:56] this is the most classic case of open sharing where I say, "I'm trying to share it to a customer or I'm trying to share to one given recipient, um and I need to initiate this open share with them, right?" Um the second thing is maybe you also have some sort of data
[01:12] products that you are able to monetize that you list on our Databricks Marketplace, for example, right? So, this is instead of one-to-one point-to-point share, this is actually a one-to-many type of share, right? So, that's also a pattern that we see very commonly. The third one is maybe you actually want
[01:28] to collaborate across different business units, different regions, uh different accounts, for example. Um and this will really you have to then start thinking about, "Hey, maybe I have egress costs that I need to think about. Maybe I have replication costs that I need to think about, um
[01:44] when I'm managing governance across different regions, right? I need to make sure that I have the same attributes being applied across all my tables, even regardless of if they're sitting in one BU or another BU." And then the last thing is maybe you also have system of records that are SaaS applications, right? So, maybe you
[02:01] have a CRM application in Salesforce that you need to manage. Maybe you have things sitting in Confluence that are unstructured data that you need to manage, right? So, these are all things that are part of your regular workflow that Open Sharing at least aim to support.
[02:17] Um so, next slide, please. Oh, perfect. it. Cool. Yeah, so let's talk a little bit about what Open Sharing is. And maybe some of you are like, "Oh, I've actually haven't heard of that." Maybe you've heard of you've heard of Delta Sharing instead. Um so, as of last week, we
[02:33] actually announced that Delta Sharing has officially graduated to Open Sharing. And this comes with the selection of open features as well as capabilities that we've added. The first one that I want to call out is that this is really the open protocol for the agentic era. And we've actually also
[02:48] donated this protocol to the Linux Foundation. Um so, you can see on the left-hand side here, we've added a couple of different AI assets that are now supported, of which they include AI skills, they include AI models, and then they also include agent Genie agent sharing.
[03:05] And on top of that, for the Genie agent sharing, what that kind of encompasses, in case you guys haven't really heard of it yet, is you can actually share a subset of your table within, say, like a space, and then you can actually also give pre-sample queries, or you can put
[03:20] instructions that really helps shape the exact context and metadata that you want to share with the recipient. And you can also put data controls on top of that, right? So, if you want to limit how many tokens can actually go through that Genie agent, or if you want to also
[03:36] control, "Hey, I don't actually want my recipient to see the underlying data." That's also something that you can actually uh basically control and and hide restrictions from. The second thing, and the thing that we're all here to talk about today, is sharing to the IRC kind of ecosystem and
[03:53] supporting Iceberg as a first class citizen in open sharing. So what this includes is you can actually now share to a wide variety of different clients and of which includes things like Snowflake, Trino, Flink, Spark, and more. And
[04:08] obviously we'll talk about that for the rest of the session today. Um and then one last call out here is instead of just being able to share to and from clouds on premises is actually also supported now. So we know a lot of enterprises, you know, still live in where you have your data sitting in those type of environments. Um so we
[04:25] want to really open the ecosystem up to support all types of use cases that you might have. Next slide, please. Awesome. Yeah, so this is just flashing some of the logos and brands that have worked with us as
[04:41] part of this open sharing ecosystem. And you probably recognize a lot of them. Right? So this really spans across all different types of industries, whether it's finance with FactSet, LSEG, um S&P Global, or maybe it's healthcare, right? So you'll see IQVIA,
[04:58] uh you might see HealthVerity, you might see Kythera. And then there's also a lot in the advertising space. Um so you might see there's Aviva, maybe there's uh Imperva, Epsilon. So it really it kind of varies from these big brands to all the way to boutique more
[05:14] specialty providers as well. And this type of breadth is also what comes with open sharing ecosystem. So if you see any of these logos or beyond that you're like, "Oh, I actually my company is already sharing data with them or is already receiving data from them." Open
[05:30] sharing could be a protocol and a solution that you guys could consider. Cool. So now let's talk about Iceberg interoperability as a whole. And what exactly was the problem that kind of got us all here today. Right? So we know that of course open
[05:47] lakehouse is an important concept. But, if you look a little bit deeper into that, there's actually multiple different types of table formats within that. And of which the most two popular ones are Delta as well as Iceberg. Um back then, right?
[06:03] We actually found that there's a pretty equal split between Iceberg and Delta tables, but they conventionally don't interact or talk or mix mix very well with one another. They operate off of completely different catalog APIs. Um if you want to copy, if you want to
[06:20] basically one Delta table to another Iceberg table, you have to make a copy, you have to convert them. And that's a lot of storage costs, a lot of overhead, and a lot of uh just having to manage those governance that you have to think about.
[06:35] On top of that, we also see that you now as enterprises begin to grow more and more into this data sharing pattern, you may have data spread across a variety of different engines. You might have, say, your analytic work done in Snowflake, but you might do your AI work in
[06:51] Databricks, and then you might do something else in another engine. Um and having to manage all those different copies of data is just simply hectic and unreasonable. And we find that one last thing that we see is customers are forced into this false
[07:06] choice, right? Or a false sense of choice that if I have already centralized on Iceberg tables, that's what I'm forever stuck with. Um and actually, that's not quite true. So,
[07:22] that's why we're actually super excited to announce that Iceberg both from sharing foreign Iceberg tables as well as sharing two Iceberg clients are both fully in general availability today, and you can directly use them in the product however you want. So, let's kind of walk a little bit through of the history of
[07:38] how we got here. All right. So, first we acquired Tabular a few years ago, and they're the company that were founded by the original creators of Apache Iceberg and we brought that expertise in house. Second, we actually introduced something called uniform or universal format. And
[07:55] what happens here is uniform actually lets a single physical table be read as both Delta as well as native Iceberg and it just binds together that metadata layer and underlying it is still the same parquet file, right? You're not replicating it, you're not converting
[08:11] it, you're simply just having that additional metadata on top that governs both the stable formats across the same singular copy. On top of that, Unity Catalog, so you see that in the middle, actually now governs Iceberg in three different ways.
[08:27] Maybe you'll hear of foreign Iceberg tables, manage Iceberg tables or even Delta tables as native Iceberg tables through uniform, right? So these are all choices that you also have when you start thinking about, "Hey, how do I want to want to be able to govern and centralize my entire data estate?"
[08:43] Right, these are all data formats and types of tables that are now supported. And basically what that means is this brings together one centralized governed lakehouse where the format no longer matters and it's no longer a lock-in decision.
[08:58] And as people like to say, the format war is officially over. So let's just take a quick look at this diagram. Let's start from the left real quick. So you can see we have foreign catalogs which includes, let's say, common ones that we hear about, Snowflake, AWS Glue, maybe you have Hive
[09:14] Metastore, maybe you have Google. Right, so you can actually federate that into UC and we'll also show a demo of this later, so don't worry. And you bring it together where you can put additional permissions on top. Maybe you want to apply attribute-based access controls or ABAC which can govern at the
[09:31] row level, right? Maybe you want to actually track the lineage and also audit, "Hey, which specific users are reading my tables?" And all of that you can do through your Unity Catalog. And then on the right side, you can see there's actually three options or three buckets for where you
[09:46] can share your data to. The first one is what we just talked about, so the new RSC ecosystem. Um so, basically any open engine that supports this open table format through the Iceberg REST protocol API, you'll be able to automatically read live tables from Unity.
[10:04] Second one, this is our classic Databricks to Databricks sharing format. And then the third one is you can actually also share to open Delta clients as well, even if you're sharing a managed Iceberg, foreign Iceberg, or any other type of table. And these include things like Adobe, Oracle, SAP,
[10:20] or maybe you have BI tools such as Tableau or Power BI, and more. So, now that we have an idea of the overall arching kind of ecosystem and how that of things how how things flow,
[10:37] let's take a look a little bit at the more technical level, right? Like, what exactly is happening under the hood? So, let's start from the direction of if I'm taking data sitting outside of Databricks. So, if I have foreign Iceberg tables, I want to actually bring it into UC, and then I want to use open sharing to share it to my external
[10:54] uh open sharing ecosystem. Like, what happens, right? So, the old way, and maybe some of you might heard of this, is provider-side materialization or what we call PSM. What happens here is each share actually generates a temporary Delta table under the hood, and then you serve it through
[11:11] pre-signed URLs. But, this kind of degrades the core performance sometimes because you need to materialize every single table every time. That takes some time to materialize. You have to then worry about storage costs when it comes to storing those materialized copies.
[11:28] Um and it honestly it works, but what happens here is it's not super scalable, especially as you scale up to thousands or tens of thousands of tables. Um and so we introduce something called cloud token sharing. So this is the new path. We actually reconcile two storage
[11:45] locations to be exact. The first one is the source location of where your actual iceberg metadata is living. The second one is what we have done with uniform, which creates this Delta metadata layer that also gets shared to the recipient.
[12:01] And our query time UC will then short-lived specific scoped credentials to both of these metadata. And the recipient's runtime simply reads the parquet exactly from where the source is. And both credentials, by the way, are read only. And this is a very
[12:16] deliberate security decision to make sure that hey, you can't ever write back to wherever the source is. And if you then want to run, say, downstream use cases, you want to put a model serving endpoint, for example, and run some agents on top of that, you can. Um but by making a copy of it in your own local
[12:32] environment at that point. There's also one more thing that I want to point out, and it's actually the freshness piece. Um so under the hood, we actually also do auto metadata refresh to make sure that there's no drift between the schemas of where your foreign iceberg table is sitting and then how it's
[12:49] registered in UC as well. Um so I guess a little one caveat that I might have to say here is this does today only work for D-to-D sharing scenarios. So if you're trying to share a foreign iceberg table to an outside open ecosystem, that still uses PSM, but
[13:05] that's something we're also working on that we're looking to improve continuously. So let's look at the other direction now. What happens if I have data sitting in UC that then I want to share out to IRC clients such as Snowflake, Trino, Spark, uh Flink, and Spark, and more.
[13:23] Like how does these open connectors read the data living in my UC without having custom connectors or custom logic happening under the hood. So, we have actually implemented something called the IRC. You might have heard that all day. IRC IRC IRC, what does that stand for? So, that stands for
[13:40] the Iceberg REST uh Iceberg REST catalog. Uh and what they do is actually they support it's an open protocol that like anybody can implement, by the way. So, this is not specific to Databricks. But, this uses standardized REST API endpoints to basically access Iceberg
[13:57] open table formats. So, this allows you to have a standard set of APIs and a standard set of schema and a standard set of metadata that you can then use to access and read across all Iceberg tables. So, what happens here is actually when
[14:14] you create an open share, open sharing will will create this something called a credential file. And what does this include are bearer tokens, the URI to point out where your source file is living, as well as a time scope basically time of where and where this
[14:30] token was generated. So, that when you import that and when you run that query directly in Snowflake, let's say, they know exactly how to authenticate against the source and also where to authenticate to. Uh and once again, everything is live.
[14:45] Everything There's no copies being made. Um and then we'll show a demo of that directly. But, before we go to the demo, I also wanted to talk about one more thing. And it's that Iceberg interoperability is not an isolated thing. It is a part of the rest of the entire open sharing
[15:01] ecosystem and it comes with all these different additional features, as well as governance that you might have already relied on today. The first one is view sharing. So, we hear that a lot and this is actually a feature request that came in after we launched our initial private preview that we have added to the public preview
[15:17] and then also to GA, as well. So, like, hey, maybe I only want to share a specific subset of my table, right? I don't necessarily always want to share the entire table or the entire schema to my recipients. So, we've added view sharing capabilities to do that. The second thing, and also something
[15:33] that we've added after our private preview, is OIDC authentication. For a while, we only had bearer tokens, but we hear that a lot of companies, "Hey, I actually want to be able to use my own recipient's IDP to authenticate." Right? So, Databricks now also will have
[15:48] these uh short-lived scope tokens that were issued by the recipient, and they can authenticate into Databricks. The third and fourth thing are actually things we just announced this week. So, the first one is global distribution. And as we talked about earlier today, we
[16:04] said, "When I have to share across different BUs or different regions, I have to think about replication costs. I have to think about egress costs as well." Traditionally, if I'm trying to share a share to, say, five different recipients in the EU and I'm in the US, then every
[16:20] single share I create, I have to pay egress costs every single time. And that gets pretty costly, especially for a large enterprise with a lot of shares going out and a lot of tables. So, what global distribution is is it allows you to replicate your data once
[16:35] to every single region you have recipients in. And from then on out, every single share you do, you don't pay egress, right? Cuz you're sharing within the same region at that point. That's something we're super excited about. The second thing is secure connect. So, networking is also a huge thing that we
[16:51] say, we hear, "Hey, like sharing is good and all, but there's actually still a lot of friction, right? I still have to do a lot of configuration. It's actually still pretty hard to use." And so, we've introduced secure connect, which what it does is you can think of it as a gateway, in a way, where you set up your
[17:07] configuration to this gateway once, and all that configuration propagates down to your recipients down the line, so that you don't have to set up network config every single time to every recipient.
[17:22] Awesome. So, now that you've gotten basically this overview of the Databricks Open Sharing architecture, how it works under the hood, and what Iceberg interoperability means, let's actually ground it in an actual business use case. So, I'm going to pass it on to Balaji to talk about how they've implemented this at Foot Locker.
[17:38] Thank you, Dia. How many of you have shopped at Foot Locker before? All right, then you know us, right? So, we are a global sneaker athletic retailer. We operate 2,400 plus stores in 20 plus regions, right? That means a
[17:55] lot of data, So, before I hop into the architecture, right? Let me show how our landscape looks like, right? So, we operate in Azure is our foundation. We are Azure workshop, so So, on top of Azure, we have a single data lake storage.
[18:11] That sits on top of it, right? And we have both the platforms, Databricks and Snowflake, operating producing and consuming the data from the same storage layer, right? So, before we started our journey, like 7 years before, we were on on-prem. So, we
[18:27] modernized our platform from on-prem to cloud. We brought in Databricks as our primary vendor to do that analytical reasoning, right? So, today, the primary ALML analytical workloads runs on top of Databricks. Plus, we also use it for our open sharing. Before it was Delta Sharing, maybe
[18:44] last week. Uh So, uh from coming to Snowflake, so we I have my data foundation layer producing the platinum curated data sitting in, but writes data back into the into Azure Data Lake. So, then I do have Power BI on which we build our visuals. I have
[18:59] 5,000 plus business users who use dashboards on every day to drive insights and matrixes, right? So, Databricks uh not only does the analytical engineering part, so we also use it to power our data sharing internally and externally, right? You
[19:15] can see some of the ones that I have mentioned below, the six of them, right? So, this is how we we power our Foot Locker enterprise intelligence platform, I would call that. So, uh the customer CDP identity resolution,
[19:30] pricing intelligence, allocation intelligence, and sharing the data between our internal external vendors happens through that, right? So, this is a This this is This is all in production, actually, as of today, right? This really contributes directly to our top line and bottom line when it comes to
[19:46] that. So, our strategy was very simple. Bring your platform to the data, not your data to the platform. That's where Open Sharing plays a critical role for us. So, what was the problem that we are actually solving, right?
[20:05] Yeah, thanks, Ria. Um So, how many of you today uh live in the world of SFTPs to transfer the data to your vendors or your partners? Okay, I see you. Right? So, how many of you had a failure in the 2:00 in the morning when you had your
[20:21] sync pipeline failing and you had to go fix it? Okay, I see you. Few hands up. So, this is exactly where I was 3 years back. I lived that world. I know the pain. So, I'm going to show you how we can solve it, right? So, if you if you could see in the slide, the left side, right? Where we
[20:37] where I produce my Snowflake data, a customer, our pricing, allocation, right? All the data has to be copied over into Databricks platform. Uh for So, we build two type of products. I didn't I should have led with that, right? We build analytics as a product and data as a product. So, data as a product is where we share data
[20:54] to internal external consumers, right? Through which we power business. So, now I need to I have extracted and copied the data from Databricks from Snowflake into Databricks, another copy, a pipeline. I have to sync the data back into the pipeline because Snowflake jobs will run, it will create the data. Now I
[21:10] need to make sure that the data is syncing Databricks environment as well, right? Now then I need to produce the extract through which I need to send it back to the clients to SFTP. Um now again, if there any transformation I do on Databricks side, I need to sync it back into source. So
[21:25] tomorrow when I sync it back, it is not in two different versions of truth, right? So this is not just engineering cost, right? There is a huge governance overhead if you look at the data sitting in two different platforms. Right? So when it And also security risk,
[21:40] right? You need to encrypt the data, tokenize the data because it lives in both the platforms. You need to maintain both levels of security integrations to be done. Now you need to also start validating the data. Okay, what I have in Databricks and Snowflake are the same or is it two different versions of truth living there?
[21:56] So every copy is getting multiplied. Right? The storage latency, right? At terabit scale, this problem compounds really fast. So in the hindsight, we were thinking, "Hey, what is the problem? What we should better, right?"
[22:12] When you look at the problem, the problem is not about transporting the data from to externally or internally vendors, right? The problem is where we are duplicating the data only for the purpose of copying, right? That's where we put together strategy. We should produce a one single copy of
[22:27] data in open format um in consistent schema where people can come and grab the data. Zero touch self-serve. With that particular strategy, that's where we brought in open sharing, the open format.
[22:44] So this is the architecture that actually solves the problem for us, okay? This is architecture that is live in production and this is not a roadmap slide. This is actually in production. We use it on day-to-day basis. Right? Um if you see on the almost on the right side, so I have the Snowflake
[23:01] DBT pipelines powering the platinum curated data through which we power our business insights, right? So, Databricks also produces the data for from our ML outputs like base connectors, writes the data into data lake in open format, on universal
[23:17] format, right? Through which, right? Uh now the data lands as Iceberg data in data lake. So, write once, I federate the metadata into Databricks environment where I apply the governance through which
[23:32] one single hub of all governance layer that sits in Databricks, right? So, from which we share the data to external leading CDP provider to Databricks to Databricks, and we have identity resolution provider through which do Delta to open, and we do have a prescriptive and ML
[23:48] analytics provider through which we share in Iceberg format and to our parent company Dicks, we share both the protocols Dicks Databricks to Databricks also Iceberg open data share, right? So, what is take away from this slide, right? Write once, federate metadata, share from one single
[24:05] hub, right? So, one physical data copy layer, there is no duplicate copies happening on platform on platform. So, it's totally we avoided that, right? Um so, before I move into the business use case how what we power, I want to pass it on to Tia. She will show us how
[24:21] simple this particular sharing is. Awesome. So, this demo that I'm going to show today breaks it off into two different scenarios. Let's start with the first one. The first one is actually I want to federate an Iceberg table. So, I have an
[24:37] Iceberg foreign catalog sitting in AWS Glue. I want to bring into UC and then use open sharing to serve it and share share it to an end Databricks recipient. So, let's see that in action. So, I come here. This is my Unity catalog page,
[24:53] and then I want to create a new catalog actually so I can register my foreign catalog. And here we'll just say foreign dice. And I choose okay, great. I want to create a foreign catalog. And I've actually already set up a connection beforehand, but this is something that
[25:09] you guys will need to configure as well. And it's a glue test with the authorized path, and this is the storage location. And what these two are is just one is what we talked about to the Delta metadata layer. The other one is to the Iceberg metadata. So that's what the two paths are for.
[25:25] So I create this and I can view the catalog. And directly here I can even see already this is a foreign catalog that has this connection to AWS Glue source. And then if I come here once again, yeah, a foreign catalog. And eventually
[25:42] we'll even be able to see once it loads that this database is in the Iceberg format. So it'll take a little bit. But now okay, great. Now that I've federated this foreign Iceberg table into Databricks, what do I do next? I want to
[25:57] add it to a share. So let's do that. And yeah, see right here we we see that it's a foreign Iceberg table. So to add it to a share I click on share, and then I say I want to share via open sharing. And I actually want to create a new share. So dice, let's do Iceberg share.
[26:15] And for my recipient, I want to share actually just to this other Databricks store. So I copy the sharing ID and I come back to the provider experience and I paste it in.
[26:32] And I can see that oh, this recipient actually has not been created. So I'll go in and add it I'll add it later. So that's no big deal. But for now, let me just share it. So I've at least now added the table to a share. Then I can go to the share and under the recipients page, I can add the recipient, right? So here I can create a
[26:48] recipient. Uh let's call this Iceberg Databricks, put in the identifier, and boom, create the recipient. Oh. And for some reason, doesn't work. But I
[27:03] think I know why, and that's because I'm trying to share from a dev environment to a staging environment. As you guys probably all know, that's a no-no, so let's actually throw it into a different one. Let's do
[27:18] this Marketplace sandbox environment. So let's add it in here, and we can see, okay, great. The share is successful, and I've also successfully added a recipient to that foreign Iceberg share. So if we come here, and I actually go to my Marketplace sandbox environment, and I just simply
[27:35] log in. Right? So this is just the usual Databricks flow. Say, great. Yes, yes, this is me.
[27:51] Hm. It's the wrong link. And I come back here, and I realize maybe I've clicked on the wrong one. So there we go, this is the right one. So I go into catalog, and I click on
[28:07] open sharing, and I actually see, okay, what was shared with me earlier. Right? So this was from this other catalog. Let me just get the metastore ID as well, so I know exactly where I'm sharing from, so I can search that up.
[28:27] And I go back here, and I see, great. I have a share that I can now mount to my catalog. Remember, this is the one that we just shared. Right? So I can directly mount this to my UC catalog, and then I can directly start querying that foreign Iceberg table I just got from an AWS foreign catalog. Now you might be wondering, okay, what
[28:43] if I want to do the opposite direction? What if I want to share a managed Delta table that already sits in my UC to an open Iceberg recipient? In this case, let's do it in Snowflake. So, let's start. Let's go back to This is also, once again, just my UC, and I actually want
[28:59] to share the CRM account table. And let's add it to a open share, once again. And we can create This is going to be Snowflake open dice. All right. Once again, we've created the share. Let's go add the recipient. In this
[29:16] case, because it's an open recipient, so this is going to be Snowflake open dice. And here you can see this is what we talked about earlier, right? You can either authenticate via a bearer token, or you can authenticate via OIDC. In this case, let's just do a bearer token, cuz it's
[29:32] simplest. And then you can also set a token lifetime. If you want to rotate the token whenever, you can also do that. So, this is all meant to add security to your share. And now I see that, okay, I actually have the URL for the activation link that contains the credential file to the
[29:48] bearer token that we talked about earlier, to the URI that we talked about earlier. So, what usually happens here is I'll send this URL actually to my recipient. So, let's pretend I did that, and let's open this in a new browser. I can see that, oh, I actually see a set of open connectors that I can access
[30:04] this share I just created with. So, I'm going to download the credential file. I'm going to click on Snowflake, because that's where I want to ingest this data from and read this data from. And I want to then import it in.
[30:19] So, I just Yeah, this one. And I can see, great, I actually have this share that we've just created together. And then I want to generate the SQL. So, what does mean? This is meant to be a super easy copy and paste type of thing, where you create the
[30:35] catalog integration, you see this is where uh the source URI is, the bearer token of how it can then authenticate to that source, the refresh interval so we make sure that the metadata is fresh and not stale all times. And then finally, we create a link database. And what happens
[30:51] here is if I then at the source want to add in a new schema or a new table, it also just reflects live on the Snowflake side. So, I'm going to go ahead and copy this SQL query, jump over to my Snowflake notebook or SQL, paste it in.
[31:06] Now, there's a few things I need to change, right? You can see there's a couple of placeholders. And this is simply what do I want to name my catalog? Let's just call it dice demo. And then if I want to then also set a new database name, we can call it dice
[31:22] database. And then same thing, it should be called the same name, right? It was the same catalog name. So, now great. I can actually run this.
[31:39] Line 22, there's a syntax error. Let's get it together. So, I see that maybe I cannot call it database. Maybe. Underscore. Thank you, guys. Great. Nope.
[32:06] There we go. Nope. Oh, okay. One more thing. And this is a great reminder. So, what we also need to do is what we talked about is uniform. We have to turn on universal format. And what happens here is you basically create that Iceberg metadata layer on top of your Delta table. All right. So,
[32:22] here I come in. I've actually already done this beforehand. So, you can see that okay, I need to turn on uh column mapping. I've enabled the Iceberg compatibility to be true. And then I've also turned on uh universal format so I know that hey, I actually need to create this Iceberg
[32:38] metadata layer. So, I'm going to run this. And it's going to tell me, "Great. Okay, I actually need to now go back and we can see it's actually
[32:57] You'll see underneath the details page of the UC that it's actually now an Iceberg uniform enabled table now. Great. So, if we come back here, this, in theory, should now work. Great. So, we can see that the integration was successfully created.
[33:15] And if I want to just show, very quickly, let's say select star from this table that we just created, so diced demo {{}dot} default {{}dot} let's say CRM accounts.
[33:32] Hopefully, that loads. Select the select the statement. Just select the statement and run it. Fully, including select. Yeah, run it. There we go. No, no, no, no, no. Nope.
[33:49] Select 27 line, then click that. Yeah, do it. Yeah, yeah. Yeah, there you go. There we go. Okay. Accept. No worries, no worries.
[34:22] Technical difficulties, guys. There you go. Maybe this will work. Select the statement. Yeah.
[34:48] What? Everyone saw it. Well, we can actually come here. Here's an alternative.
[35:04] Let's see the catalog that was just created. Let me see if it actually is there. Maybe I'm just bad at writing SQL. So, we come here. We can see Oh, there is actually this database dice database that I've created. And in the schemas, there's the default
[35:20] catalog. And then there there's a table called camera account, which is one that we just shared. Right, so I can see actually here that I've directly shared from a managed Delta table living in UC, created a uniform layer on top, so this creates that iceberg metadata layer that
[35:37] makes it easy to be readable through Snowflake, and then I've queried it and now actually lives as a native iceberg table sitting in Snowflake. Awesome. So, let's go back to the slides. And let's pass it back to Balaji to talk about the actual business use cases.
[35:53] Thanks, Jia. Um so, we have seen the demo. It's It's really simple, right? Once you produce the data in open format with a few clicks off the button, you will be able to share the data to external vendor. They can consume from different uh platforms, right? It could be AWS, it
[36:10] could be Snowflake, or Trino, etc., whatever that could be, right? So, uh what did we actually achieve when it comes to business use case, right? So, this is where the what it gets real, right? So, we had architecture. We said, "Okay, write it once, federate it. It works, right?"
[36:27] But, all of good in whiteboard. So, but what did we actually enable for the business, right? That's what I'm here to explain what we did, right? I took I'm taking six things right from the start, CDP platform. We are powering our CDP platform, right, for our business using Databricks to
[36:43] Databricks in that to sharing. Now, business is able to do market or marketing orchestration, segmentation, and personalization use cases because we are able to do the share through open sharing. Second, uh in when we do did it, right? We are not duplicating the data. We are actually
[36:58] sharing the data to the vendor. Now, we are able to enable the business use case without copying with with one single governance on top of it. Now, coming to the idea, right? Now, we are doing cross-entity channel mapping. So, we have identity, uh we have store presence, and we have
[37:14] online presence. We have customers living coming through both the platforms. Uh now, using IDR, we we were able to resolve the identities and share the data back to CDP through which we are able to enable IDR. Now, there is no big data getting
[37:29] complicated just because we want to build identity resolution on top of our customer data sets. It's getting easily shared through open sharing. Uh now, we share our data between our both my parent company and back to Foot Locker, DSG and Foot Locker. We use both
[37:45] Databricks and Databricks and Iceberg data sets that we share with them. We get it back. We enable our internal use cases. Right? So, the fourth one is a very big win uh for us. Uh it's the markdown and promotion optimization. So, we share our data to our prescriptive
[38:00] analytics vendor. Now, they share the data back to us. Now, they'll be giving us the data uh pricing real-time pricing recommendations at store level, right? Previously, we used to price the products across the same across all the stores. Now, we are able to price the
[38:16] product by clusters and by size. It's a huge win for business. It drives our top line. Right? That's through iceberg sharing. So now the next use case is allocation intelligence, right? We share our DC by quantities to them. So we get back the
[38:32] PO placements at size curves. So this is also iceberg sharing. So now because of that we are able we are not overstocking the same product in the same store no more, right? It is reducing our overstock. So we have highest sell through. This means we are lifting our margins and top line bottom line. The
[38:49] last one is, right? We are able to repeat this pattern and share our data across different uh vendors internally and also externally. So by doing this, what did we enable us? So we were duplicating the data before.
[39:05] So now we are reaping the benefits of open sharing, right? So if I look at how do I measure what did we actually benefit out of it, right? I I take five different dimensions to measure it. Number one, storage. We are no longer duplicating the data. We had terabytes of data that is getting duplicated. We
[39:21] are storing twice. It's no longer happening. So we have pipelines that is actually copying the data. We have to run computer additional. That is no longer happening. Three. Now this is where it gets really So we are all talking about cloud saving, storage and technical stuff, right? So
[39:36] business is able to do the pricing impact, right? Having the right product at the right time in the right place drives retail. So we are able to do that with real time pricing recommendations from That's a pricing impact. Now the allocation, right? So we are able to have the right amount of inventory present in the right
[39:52] store to drive growth. So that is getting enabled from business perspective. Four. Previously, we have to take multiple sprints to produce a feed, sync the data, and deliver the value to business. It takes days, weeks. Now we can do it in few days or few hours based upon what
[40:08] data we want to share. It's a huge impact to our organization. That's how we I measure the ROI. Storage, pipeline, and the business value all together we could able to see what is the uh what is the ROI that we are getting out of it.
[40:25] Uh so now if you if you are thinking about, right? Okay, you want to do something back take it from this session back to your organization, right? Here is the playbook, right? This is how you have to play. First design. The design is right it's a data plane and the sharing plane. First you have to
[40:41] decide both are separate they are not the same, right? You have the data first created then you can use it for sharing. Okay. The second you have to who am I going to share the data with? What type of data they really need? You have to decide that, okay. Do they want delta format? Do they want open format? Or they want
[40:57] iceberg format, right? Based upon that you need to ensure that you have that design in place, right? Use one single storage layer, right? My case it's ADLS, maybe your case could be AWS S3 or could be Google storage, right?
[41:12] Now when you're running pipe workloads in two different platforms, decide what will run in which platform. Do not assume that okay, let's run eventually it will automatically work. It won't work. That's how we have multiple copies of data present in both the platforms, right? So that's how you have to design
[41:28] not do's and don'ts. Now when you start building this capability, right? Understand the platform limitation, right? For example, if you partition the data the partition data works a little bit different when you share the data out. The CD CDF has its own the change data capture has
[41:45] its own quirks when it comes to sharing. Understand the limitations. Previously iceberg for an iceberg tables on which you can create a view. I think this is that was last time. Last this week it's no longer problem. So it keeps evolving the product as well. So keep tracking that. What works, what won't work so you can
[42:01] plan it accordingly and have work around around it. Right? So and also understand the ADLS compatibility issue, right? So DFS versus blob store, uh it won't work. Nobody said me it won't work. I I lost a week to it when I was setting this up. So, I want to share
[42:17] this learning from you to you so you know we have to set it up DFFS. If you're going to use blob store, make sure the permission set up in Snowflake is right when you integrate it. Right? So, you have descended Now, you know how to build it. Now, the third is we need to measure it, right? So, what How do you measure?
[42:33] First, baseline A, how much terabytes of data that I'm going to be removing or not duplicating? How many pipelines am I going to retire, right? Two. The third thing is most important thing, what leaders leadership always ask, "Hey, what is the impact or what is the value that we are going to create?"
[42:49] In my case, it was pricing and allocation, right? Some of it. So, similarly, you will have your own business case business case where you can uh uh understand that. So, you can give back the ROI back to the business what it really means. Uh So, So, when we started our journey, I had
[43:06] the same sync pipelines sharing the data through SFTP, right? Few years back. So, now everything has been We are not living in that world. We are using open sharing. Three different type of formats, protocols we follow, governing to one
[43:21] single hub, we are achieving the business value, which used to take weeks and months. Now, it is in days. That's the power of open sharing and open format. Thank you, team. Uh we are open to take questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.