How Danone Enhanced Global Data Sharing with Delta Sharing

Summary

• Hub-and-spoke to mesh: Danone migrated from traditional hub-and-spoke data architecture to a more efficient mesh model using Delta Sharing.
• Cross-region sharing: Implement in-region and cross-region data sharing with fine-grained access control and data contracts.
• Governed collaboration: Centralized governance and standardization reduce data silos and replication costs across 120+ countries.

How Danone Enhanced Global Data Sharing with Delta Sharing

Talk by BASELTO Yohan, Lead Data Engineer, Danone ; Gergő Pásztor, Resident Solution Architect, Databricks
Learn how Danone, a global leader in the food industry, improved its data-sharing processes using Delta Sharing, an open protocol developed by Databricks. this video will explore how Danone migrated from a traditional hub-and-spoke model to a more efficient and scalable data-sharing approach that works seamlessly across regions and platforms. We’ll discuss practical concepts such as in-region and cross-region data sharing, fine-grained access control, data discovery, and the implementation of data contracts. You’ll also hear about the strategies Danone uses to deliver governed data efficiently while maintaining compliance with global regulations. Additionally, we’ll discuss a cost comparison between direct data access and replication. Finally, we’ll share insights into the challenges faced by global organizations in managing data sharing at scale and how Danone addressed these issues. Attendees will gain practical knowledge on building a reliable and secure data-sharing framework for international collaboration.
Here’s more to explore:
Open data sharing for data, analytics, and AI: https://www.databricks.com/product/delta-sharing
A new approach to data sharing: https://www.databricks.com/resources/ebook/a-new-approach-to-data-sharing
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is a data mesh architecture?

A data mesh moves away from centralized hub-and-spoke models toward distributed, domain-oriented data ownership. Danone uses this approach with Delta Sharing to enable regional teams to share data independently while maintaining central governance through the Databricks Data and AI platform.

How does data sharing work across regions?

Delta Sharing enables in-region and cross-region data sharing with governed access. This allows Danone's teams across 120+ countries to collaborate securely, with data contracts ensuring consistency and fine-grained access controls protecting sensitive information.

How do I scale custom AI agents across teams and business units?

Scaling custom AI agents across teams and business units requires a data sharing mechanism that allows different teams to maintain autonomous agent development while ensuring consistent data access governance and preventing duplicative work. Delta Sharing enables organizations to share datasets with governed access across teams without copying data, allowing each team to build agents tailored to their business context while accessing the same authoritative data sources. This approach scales agent development teams from individual contributors to coordinated networks while maintaining data integrity and compliance.

What are data contracts in data sharing?

Data contracts define the schema, quality expectations, and usage rights for shared datasets. Danone uses them to ensure data consistency across regions and reduce misalignment between data producers and consumers.

Full transcript

So thank you all for joining in our session today. We will start with an overview of global data platform and then we we will take a deep dive on how we leverage data sharing and data bricks to simplify data sharing and governance both locally and globally. I'm Yuan Bazelto, lead data engineer for Danon global data platform with over eight years of experience as a data engineer and my co-speaker GGO. Welcome everybody.
I'm a resident solution architect at data bricks. I'm part of the professional services team which goal is to help customers like Danon to plan and implement their next generational data platform. But now let's talk about why Danon shares data. So, Danon is a leading global food and beverage retailer and at Danon our mission is to bring health for food to as many people as possible. Guided by our vision, one planet, one elf.
And to achieve this mission, we produce healthy products adapted to local markets utilizing sustainable distribution processes. Those products fall into three air focus business lines with our essential dairy and plant-based. So with fresh dairy products and plant-based products and also our IO protein option like our I pro brand perfect for issuing after good workout and specialized nutrition for little ones and medical nutrition and also waters with our well-known brands like Evian.
So with products available in over 120 countries with more than 89,000 employees in 55 countries around the globe. The non global presence necessitates distributed IT OBS and that's why a scalable and adaptable data platform is critical to ensure that we meet um diverse regional needs and regulation and seamlessly share data driving improvements for our production and distribution processes. So like us and everyone in this phone our data platform must constantly evolve to meet the ever data demand.
So in our previous architecture we relied on Informatica for ingestion, Informatica ADF databicks for ETL alongside snowflake for BI and as user interface. And as you can see, we utilize a diverse set of tools to fulfill our data request. And this complexity led us to three major challenges. Firstly, multiple solution driven by individual business needs created a fragmented landscape and reduce our operational effectiveness.
Secondly, the absence of centralized governance and standardization led to data silos and data replication. Thirdly, this overall inefficiency drove up platform maintenance and processing costs. So, this approach was complex and hindered our ability to scale and innovate. That's why we searched for we sought for a solution to consolidate our system and establish a unified data strategy. So for our upgrades in one source 2.
0 O we sought for a solution to consolidate our system into a unified platform, a unified governance for easy access, discoverability and reusability of our govern assets and standardization of our processes for scalability and accelerate time to value through self-service. So we transitioned from a siloed fragmented landscape to a centralized unified strategy centered around automation and reusable components. So for our new So for our new architecture we centers around the Azure data bricks for all type of processes from ingestion ETL BI and data science with at its core unity catalog as our data governance solution.
So we have our lakehouse one house based on the typical medallion architecture and accompanied by the user workspaces that we call DT use data transformation units that are personalizable and will will be able to offer different sets of capabilities to our users with those it empowers uh the business to reach the power of data bricks across all type of pipelines. So we we standardize and replicate our infrastructure in each our in our reg present regents with our central lakehouse and the different workspaces for our users.
Now let's take a deep dive on how we use we share our data go and learn how our environments communicate with each other. So let's take a deep dive how shares data in their platform. Denon uses a central publishing model. In this hub and spoke model, each region has its own hub and shares its data with the others with data sharing. Uh these regional hubs are all managed centrally by the central team and they have different spokes.
These spokes are different business units that separated from each other. Each business unit can only share data with their own regional hub. Then this platform architecture enables Denon to create additional platform features across the world data platform. For example, they can ingest uh common data in the organization, run some ET processes on it and provide it to the business units. They can also create standardized features like data quality control and monitoring.
The advantages of this architecture also is that um the control of the data sharing is centralized and they can't share anything without the knowledge of the central team. We use a data product model here where the different business units publish the data product with the data contract to enable effective data governance. Danon created a data steward role. This data role um is the owner of the data products itself and they set the permissions.
So who can access or can't access the data products. So when a business unit want to publish something want to publish a data product they need to go to the data steward first to certify their data product and then when the certification is done they can publish the data product to a common schema. Then other business units can read the metadata or all of the published um data products and then these data products they can request access to from the data stewards.
The key point here is to have this data contract. Data contracts is a crucial part that enables us to implement this very complex architecture. Data contracts is basically a written agreement between different stakeholders. These stakeholders are both business and IT stakeholders. Um these written contract defines both business and technical properties. Business properties like service level agreements or for example a classification of the data and technical properties like the storage location.
Some of these properties are purely businessoriented. For example, how a business is mapping this data product to a business KPI or goal that's purely business oriented and it's not enforced by the data platform itself. Some other technical aspects can be enforced by the data product itself. For example, ownership or permissions. To be able to um write this data contract in a way that is consumable both by humans and by machines, we usually use something that can be consumed by both.
So it's usually in a format of JSON or ML. And one um good example for this data contract is the open data contract standard that you can find on GitHub if you're interested how this looks like. So now let's look at how in region data sharing works at the case of Danon. You now see the setup in Unity catalog. You can see that there is one regional meta store and then below it there is three different kind of cataloges.
One cataloges is the business unit cataloges. Each business unit has their own catalog and they can manage their own schemas and objects below it. Then for each region there is two additional cataloges. One for tables and one for every other object types. I will go in details a little bit later where why we have this separation. In the case of the regional table catalog, we only store tables and every other object types like views, ML models um or uh volumes are going to the other catalog.
You can also see that in case of views um each reference to a view it can come from either the regional part or from the business unit manage part but they can can't cross this border. So a business unit uh view can't uh reference something that is coming from the regional catalog and most importantly the regional catalog can't reference something that is coming from the business unit. This separation is crucial because it enables us that the data product in the regional catalog are actually independent from what the business unit has.
So when a business unit want to publish uh a data product to the regional catalog, this is how they are doing it. There is um based on the object type there is different methods and responsibilities. For example, in case of tables and volumes, they can just copy the data from the business unit catalog to the regional catalog. However, in the case of ML models and uh and views, there is a different method. In case of views, they will put the wall view definition into the data contract and then a central team CI/CD um job picks this up and automatically creating the same view definition in the regional catalog.
But in this case on top of the regional data and not on the business business catalog data. In case of an ML models they define the ML model version in the data contract and then again a CI/CD jobs that is managed by the central team pick this up and copy this version of the uh ML model to the um regional catalog. This is crucial again to have separation between the regional data and the business unit data. Then let's look at how this can be done in a cross regional way.
So when we have multiple regions then of course we have multiple metas stores too. Unity catalog require us to create a different meta store in each region. Then below that there is also the regional table cataloges and the uh data product schema. So here again as previously the data products are represented by a schema. Then we share the data the tables from one region to another with the datab bricks to databicks data sharing.
But we are not just doing that. We also applying some additional metadata. Databris to databicks um data sharing supports column and table comments and also primary case but they are not supporting all of the metadata. for example, it not provides us a way to mitigate uh to uh copy over the tax on those tables. So we have a different process that is doing this synchronization tax synchronization for us. Then there is a question if we use data sharing then it data share will give access to the other region to remotely query the data but then this will incur some aggress and the question arises what kind of agress cost that we're looking after.
So we did some experiment about how the query cost changes when we query the data from a different region. On the left side you see the results on a bigger table with three different kind of queries. The first query in the first column is a remote query. So it's just using a data sharing table to remotely query uh the table itself. The second query is a local query and it's running on the data that is copied over um from the other regions.
So it's basically doing a full copy and then run the query on it. And the third query is very similar to the second query, but it's doing an incremental copy instead of a full copy. If we look at the bottom where we see the numbers, uh yeah, maybe that's not not seen because of the subtitles, but if you can see that there is almost uh €7,000 on the left side. So a remote query is very expensive. It's almost 10 times more expensive than uh the incremental and full copies.
And also there is a big difference between the full and the incremental copy. The incremental copies as cost around half than the full copy. So we see that based on the different um query type we have very much big uh price difference. If you look at the right side of the table we see the same queries but on a much smaller table. In this case, the remote query is actually much cheaper than a full or um uh incremental copy.
And the difference between full and incremental copy is basically non-existent. They have the same price. So we see that based on the data sizes there is totally different characteristics and prices. And based on this we see that we need to choose our sharing strategy based on what kind of data that we share and what the producers requirements are. So when we doing cross regional data sharing then we came up with a new concept.
We have two cataloges now. We have the catalog that contains the shared data that is coming from data sharing and we have an additional catalog where the end users actually have access to. So the end users here the business unit region B can only have access to uh this second catalog and not on the shared catalog itself. And then we mapping the tables from the shared catalog to the end user catalog in some way of form.
And this way of form can be defined by a share specification. This shared specification has two inputs. One input is the data contract itself that is defining that what kind of SLAs and requirements that the business has for this data set and the other input is coming from the central team itself. The central team can monitor how the end users using this table and based on the usage they can decide if they want to create for example a local copy to improve query performance or just create a much cheaper setup for themselves.
So they can have a cost control transparently and the end users doesn't see anything from these changes because they always query uh the uh original table catalog on the left side. So this enables us to create something where we can change the data sharing strategy transparently to the end user. This is how the full process looks like now put it together. So we have uh data sharing in the middle. We have two catalog on the right.
But there is more complexity. This is works with tables because table has a concept of a view. A view is basically kind of a link that can create uh a link to the other to the share table if you want without actually copying data. But we doesn't have a similar concept in case of a volume or an ML model or something other object types. So in those cases we actually create a separate regional shared catalog. That's what I mentioned before that I will tell you why we created two cataloges there because and that's why we separate between the object types because those can be uh have the similar setup that you see here.
So those we just give access to the end user directly on the shared tables and on the shared volumes for example and if they need to create a local copy then they can do it themselves but because when they change from a local copy to a shared uh volume they need to change also their code. So that's not transparent for them. Now let's zoom in uh in this data sharing strategies there is four different kind of strategies.
There is copying data that can be full copy or incremental copy as I already mentioned in data rigs that this can be implemented with deep clone or materialized view. The second uh um strategy that I already mentioned is the remote access strategy. This is usually done by data sharing on data bricks and you can also use data uniform to support more than one data data format. Then there is remote processing. Remote processing is a little bit special case because here usually there is actually two compute involved.
There is one compute involved from the third party side and one on datab bricks. This can be implemented by GDBC or DBC connection by lakehouse federation or con using a third party connection. Then the fourth u category is data synchronization. Under data synchronization I mean change data capture. Usually it can be one way or two way. There is not much different between one way and two-way. Basically just the number of pipelines that you're setting up in databicks.
You can implement this with DIT CDC or with lakeflow connect or if you want your own implementation you can use that change data feed for this. So how this sharing strategies when to choose which sharing strategy? If you have a full copy then the latency will be ours latency on the data freshness. So how fresh is the data that you're actually querying? Um and there will be the query latency will be minimal because you will query local data but the processing cost and the eress cost will be moderate as every time you will basically copy over the wall data set.
Oh sorry it's no going to the direction. Can you go back? Yeah. So in case of a remote access then um the data freshness is of course as we are querying the remote data itself it will be always up to date but then the uh agress cost will be high. On case of remote processing the characteristics will be very different from the others because here uh there will be two processing so the processing cost will be double. However, the aggress cost will depend on the result set of the query and not on the size of the data set itself.
Then there is synchronization. Synchronization will give you the best um possibilities in across all of these properties except one the processing cost because uh synchronization is something that you're constantly running that means that um you need to have a constantly running compute. So now we talked about in region and cross region data sharing and also data sharing strategies. Let's talk about what happens when we want to share just partial data.
So let's talk about fine grain access control. In fine grain access control how everybody starts to implement this is just to creating views or materials used. When you create a use or material is used based on how you want to filter. You create a new view for every kind of filtering scenarios. Oh, this business news user group want this filtering by this country. That user group want another filtering. So you create multiple views.
But then when you data share this across region then you duplicate all of these views. You can easily see that there will be easily hundreds or thousands of these views laying around what is kind of unmanageable. So we need a different solution. The different solution in database can be twofold. You can either use a dynamic Q. The dynamic Q is basically a special view that is able to um define all of these filtering rules in one view.
So it's just creating one more objects not multiple ones. Or you can use uh role level security. What it means is role level filtering and column masking that is applied directly on the table itself. So there doesn't creating new objects at all and it's using a separate SQL function uh for filtering. So you can reuse this filter on other tables. But then what happens when you want to share this uh in another region then role level security is not supported.
You can't use role level security. You need to use dynamic use. But then when you share data with another region in dynamic use, you need to use the current recipient um function SQL function to define what kind of user groups that you want to allow and how to filter. But these current recipients are basically recipients of the data share that's basically mapped to each region. So you can only filtering on different regions.
However, in the case of Danon, we don't wanted to filter just on regions but much smaller groups. That means user groups. So that was not really a good fit for us because here even if we create the dynamic cues and share those we can only filter on groups on uh regions. Then we came up with this solution that will just use a shared table and then apply the same dynamic Q definition on in each region. So basically we'll have one common dynamic Q definition and then in each region we'll create the same dynamic Q on top of a shared table.
In data weeks we are working on a solution to enable attributes based access control and this attributes based access control rules will enable us to apply role level security uh in shared tables both on the recipient and the provider end. So in uh some months from now or some time from now we will have a possibility to not actually use dynamic use but actually apply the RS rules on the tables in each region itself. Okay.
Now let's redirect our attention on tech technical aspects to people and processes which are equally or even more important. When will my data be updated? when when will will it be available in my region? Is it legally compliant within my region? All those question as platform owners we need clear answers to those question to those questions. Furthermore, we need assurance that the data commitments will be upheld and um sol and addressing this concern requires solving or organizational issue challenges connecting departments and entities and establishing clear processes between business and IT.
So firstly we need to establish like GGO said robust data contracts and those data contracts will address key data governance aspects. First data cataloging that will map our business catalog our data to our business catalog in our case our domain driven structure. Secondly, data ownership to establish key responsibilities for business and technical owners. Thirdly, the data classification to set the sensitivity level of our data assets and fulfill legal requirements like um GDPR in Europe and automate automate our sharing our sharing and lastly data certification to build trust on our data assets, manage your life life cycle and automate steps like monitoring and quality checks.
After our data platform is also valuable only if the users can easily find and share their data and prior prioritizing data discovery is essential for users to for user engagement uh platform adoption and foster collaboration. Increased collaboration will reduce your data duplication and processing cost leading to significant savings. Data discovery can be governed by three different permission permission models. First, share nothing.
It's ultra secure but expect a lot of duplicates. Then metadata only like uh with the browse permission in datab bricks where you can you can share only the metadata on your data sets. And lastly, the read access by default where you give access to all of your data with minimal administration but it's not secure. At Denon we have adopt the metadata only permission model balancing between security with accessibility and finding data in a data platform is challenging in everybody search in different ways.
So it's essential to provide a lot of documentation or your data assets and that's need to be enforced. Documentation cannot be an an afterthought it needs to be forced otherwise it will never happen. And also if uh some documentation description are not available on our data sets, we can leverage the built-in AI capabilities from databicks to get a first uh a first description on our on our data assets. And also finding the data is key and uh thanks to the Unity catalog semantic search, our users can easily find their data.
Also uh we recommend to our users to add personalized tags to their data sets like their project nickname or JR project for others to easily find their data products in um in the global platform. Then on a as a platform team, our main goal is to provide a good user experience and this user experience is mainly based on how easy is it to manage their environment and cooperate with which with others sorry and this is decided by the speed and complexity of our processes like publishing on or consuming a new data product.
Speeds can be achieved through automation which is key to quickly unlock our users and um and ease of use is achieved through self service to boost the speed and lower the complexity for our users. So as you can see in this simplified sharing sharing process diagram, our users can define their sharing um settings in their data contract. Then through uh with the stakeholders through pull request we will validate this uh new data contract in git and then in the in the automation we'll be able to publish create the new objects and automatically create and the different replication pipeline if necessary and automation.
Thanks to automation, the central team can provide further services like uh ALS, masking, data quality monitoring and metadata management. So now let's talk about what we actually learned from this project. What was the lessons learned? The first one is data sharing is just partially uh about it and technology. If you think about a little bit deeply, data sharing is all about your organization. how your organization works and how the business processes are defined in this organization.
So you need to think also about your organization itself. It's not enough to just think about it and technologies and it's not enough to just try to enforce all rules with technology itself. You need to also think about your business processes. The second one um learning that we had is this is an interval about what you can choose from. In one part of the interval is the onetoone sharing where every data product can be shared by their owner itself.
It's kind of a data mesh and on the other end of this interval is the very centralized setup where everything is managed by a central team. The onetoone sharing is creating chaos while the central team is creating overhead and you need to choose based on your organization how in this interval uh where the in this interval you are at. Then the third one is a very important one. When you try to think about what kind of data that you actually want to share, you will hear things from business that oh we will never share this data. we will only share these two tables but the others are all secret and we will never share those.
This is true at the moment but later you will see that every data in an organization will be shared sooner or later especially if you allow it. So be prepared for this and be enable them to be able to share the data securely when they need it. And the first one is as I already mentioned it that your data sharing strategy depends highly on your cost and performance attributions. So you need to have a flexible architecture that is able to support all of their needs.
And this will you will not create a onefits all solution. You need to create a flexible architecture that can fit multiple different solutions uh for the same problem based on the different requires. Also to have complete data products you cannot only share table of views. You need to handle all type of objects like models and unstructured data. Of course data contracts foster collaboration and safety in your organization.
And for good user experience, a high level of automation and self-service is vital for your platform. I would like to end this session with a quote from our director that with the many lesson learned a more flexible, unified and easy to maintain architecture in one source 2. 0, O we expect to reduce the data to decision time by up to 30% helping as a company read to market change. So thank you for listening to our session.

Learn more about the Databricks Data and AI platform.