Breaking Silos: Enabling Databricks-Snowflake Interoperability With Iceberg and Unity Catalog

Summary

• Cross-Platform Interoperability: Enable seamless Databricks-Snowflake integration through Unity Catalog and the Iceberg REST API, making the Databricks Data and AI platform a central hub for cross-platform data operations.
• Multi-Engine Architecture: Build a flexible, vendor-agnostic architecture that bridges Databricks and Snowflake without compromising performance or security using Uniform, Delta, and Iceberg.
• Governance and Efficiency: Implement data sharing and access control patterns that maintain governance and efficiency across multiple engines on the Databricks Data and AI platform.

Breaking Silos: Enabling Databricks-Snowflake Interoperability With Iceberg and Unity Catalog

Talk by Geoffrey Freeman, Director of Data Engineering, T-Mobile ; Mohit Kumar, Member of Technical Staff, Sol Arch, T-Mobile
As data ecosystems grow more complex, organizations often struggle with siloed platforms and fragmented governance. In this video, we’ll explore how our team made Databricks the central hub for cross-platform interoperability, enabling seamless Snowflake integration through Unity Catalog and the Iceberg REST API. We’ll cover: Why interoperability matters and the business drivers behind our approach How Unity Catalog and Uniform simplify interoperability, allowing Databricks to expose an Iceberg REST API for external consumption Technical deep dive into data sharing, query performance, and access control across Databricks and Snowflake Lessons learned and best practices for building a multi-engine architecture while maintaining governance and efficiency By leveraging Uniform, Delta, and Iceberg, we created a flexible, vendor-agnostic architecture that bridges Databricks and Snowflake without compromising performance or security.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction
• 0:54 TMobile Data Silos
• 1:53 Multicloud is the norm
• 2:45 The cost
• 3:38 Interoperability
• 7:03 Minimum Bar
• 10:22 Functional Flow
• 19:38 Challenges Learnings
• 21:59 Architecture Considerations
• 27:10 Questions

FAQs

Why does interoperability between Databricks and Snowflake matter?

According to this video, as data ecosystems grow complex with siloed platforms and fragmented governance, interoperability enables organizations to leverage both platforms' strengths while maintaining unified governance.

How does Unity Catalog simplify interoperability?

This video explains that Unity Catalog, combined with the Iceberg REST API, enables seamless data sharing and access control across Databricks and Snowflake without requiring separate governance solutions.

What is the Iceberg REST API?

The video covers how the Iceberg REST API allows Databricks to expose data for consumption by Snowflake and other systems, enabling cross-platform data sharing on the Databricks Data and AI platform.

How do I build a multi-engine architecture?

This video provides lessons learned and best practices for building a flexible, vendor-agnostic architecture using Uniform, Delta, and Iceberg that bridges Databricks and Snowflake while maintaining performance and security.

Full transcript

0:00 Introduction

All right, everybody. This is uh breaking silos, unlocking enterprise value through data interoperability. Uh if this is not your destination, now is the time to leave the plane. Um. I am the director of data engineering for T-Mobile's finance division. Um I've been with T-Mobile for about five years. Before that I spent 13 years at Microsoft ending up on Azure and I spent a couple years at GM Cruz. Um and I'm Mohit Kumar, technical staff solution architecture at T-Mobile.
I have been with T-Mobile for over nine years now. Uh before T-Mobile, I was consulting at Microsoft and um just just working on data analytics and solutions. So, we're going to talk about a little journey that we've had at Microsoft of of trying to figure out how to centralize and and standardize our data platforms and ending up with starcross lovers of platforms. So, T-Mobile when I joined about 5 years ago had something on the order of 18,000 separate databases that didn't talk to each other.

0:54 TMobile Data Silos

They were separate standalone databases. Some of them were on Azure, some of them were in AWS, some of them were on on premise, some of them were under people's desks, and none of them really talked to each other unless you had a a a custom pipeline. And T-Mobile recognized that this was a problem. So, they had uh several different efforts to try to uh standardize and govern their data in a more uh centralized way. Um, but some teams went to best-of-class snowflake and some teams went to best-in-class data bricks.
And it's way better to have two silos than 18,000, but it's still a problem, right? So, this is not standard. This is not just a problem with T-Mobile. This is a problem across all enterprises today. Um, multicloud is the norm, not the exception. Um, it's it's totally normal for us to see companies that have AWS, uh, GCP, uh, onrem, Azure, uh, spread all across a place. And it's totally normal for us to see companies that are trying to standardize and centralize, but still end up with a a bifurcation because they're trying to go with best of breed.

1:53 Multicloud is the norm

Um, but different people have different ideas of what's best of breed for uh sometimes it's their use case or sometimes it's their company, but it's it's not unnormal for there to be uh different pieces that don't talk to each other. Um, fragmentation. So like this that's a normal thing, but it does come with a huge cost. It comes with a huge overhead. Um there's a lot of lost value that is is not captured and there's a lot of extra cost in maintenance governance uh like not understanding where things are and a lot of like spun wheels as people try to get this data to talk.

2:45 The cost

Um, it becomes really important for companies to try to make not just centralized efforts but to try to figure out ways to uh allow them to talk where they don't require centralization and that's what we've really been trying to drive. Thanks for defining the problem space. So let's talk solution and uh solution to us is interoper interoperability and we feel interoperability is the key enabler for crossplatform harmony.

3:38 Interoperability

Data mesh as a concept is a decentralized architecture where you know each domain manages their data product in the technology that they know best. um we we feel about it a bit differently and um believe that a federated architecture is what makes data mesh interoperable and scale better. Um we believe the tree true interoperability state is when you know the providers and consumers of data can collaborate effectively. um they have less data hops between the provider and the consumer of data essentially cutting down the data proliferation.
Uh they have the choice of bringing their own compute uh because each compute has their own USB. So how how can you use it to better organize or run your workload? And obviously the goal is to prevent more data silos. And uh with this goal, if you think about the centralized components like the semantics, syntax and governance, this is what defines interoperability. Think of syntax as the foundational building blocks. you know like which table format to use, how are we storing the data, where we are storing the data, where we are cataloging it.
Those are some of the foundational blocks that you need to go get right. Semantics is more about the enrichment. How do you add context and meaning to the data you have like the um comments that describe your table, your columns, the taxonomy of the data, the tribal knowledge which is associated. How do you embed more of that information into the data states that you are building? Um, and then there's governance which essentially involves all the policies, all the rules and processes around which we all need to kind of align.
That's what ensures the trust and accountability is there for that data use. And if we do it right, I mean there is a lot of business value to unlock like we can meet the users where they are. we we don't have to ask the user to move to a platform but rather enable users to bring their own compute and use it. um we can make architecture composible which is you know just use a modular best of breed technology stack you don't have to centralize on on one aspect like was mentioning before uh you can support your data minimization efforts uh because obviously with the grow you need to have the guard rails where you can control it and obviously when you simplify a lot of these you'll be able to achieve lower costs um with less data hops obviously your process processes and data latency will go down and and you know there is a world where the transactional and analytical needs are converging.
This architecture is is going to uh help the interoperability aspect is going to help be us ready for that architecture and to talk more about the solution. So uh a key aspect of what we were trying to achieve when we set out our architecture was to minimize the bar keep the bar as low as possible in order to enable interoperability in order to empower that concept of a modular uh system in order to empower people to bring their own compute to bring whatever uh new innovations that they come through from an architectural standpoint. what's the minimum bar that we need in order to make it possible for us to have interoperability and for us that came down to these few foundational blocks.

7:03 Minimum Bar

First, data is managed for us in ADLS. That's that's not uh a hard rule. Um people can have data in in other locations, but data has a center of gravity. data has like there is importance in locality and so for new things that are going to be done um we preference strongly towards ADLS not because we have a uh because we both worked at Microsoft but because that's where there was already a ton of data at T-Mobile um data stored in open table formats this is actually way more important than the uh data stored in ADLS but open table formats iceberg delta hoodie we don't actually care um what open table format we you use.
We have a preference for delta with uniform. Um but bring an open table format and we can figure out how to make it work. But this is the real key is you need to have your data stored in a cloud storage if possible um in an open table format and we need it registered with a programmaticly accessible catalog. So for us we chose Unity catalog at this point. It is by far the most fullfeatured. um we already had a ton of uh teams that were had already moved aggressively onto Unity catalog and it made a lot of sense for us to centralize there.
But um the foundational piece that you need is a programmatically accessible catalog so that you can have things like governance processes that are going to be able to go find your data and scan it, right? Um, and that leads us directly into the cross functional governance practices, which is you need to plan for there to be some sort of centralized processes that are going to make sure your data is being properly secured, make sure it's being properly classified, make sure you have, you know, whatever sort of like PII or or sensitive data scanning available.
We don't really know what all of those practices are going to be in the future. we have a good idea of what's what we do now, but you need to make sure that you're future proofing by making by trying to get all of your data so that it is accessible programmatically. And then finally, the open rest interfaces. This like fits really neatly with the cross functional governance, which is we needed something to be make sure that uh as new security or governance requirements, as new scanning requirements uh come about that we were going to be extensible there.
So, um, we feel like that's a really low bar for teams to say, "Hey, you like use whichever system you want as long as it's storing data in open table formats in cloud storage and is able to plug into our centralized catalog. Thanks. And to talk about the functional flow and if you keep that minimum bar you know if we talk it from a functional view perspective let's take the first scenario of you know data that is managed by data bricks and some and and is read by a user who is on snowflake right so obviously when the data bricks is managing uh the data is stored in adls and we are keeping it in a uniform format um so the first time when the user is on snowflake compute and is trying to make connection or is trying to read a table.

10:22 Functional Flow

What essentially happens is Snowflake is redirecting those metadata request through the open rest API that just talked about that goes to the Unity catalog retrieves the metadata location and then passes on to Snowflake. Snowflake has the same context of the ADLS where the data is being stored and given it can understand the iceberg component of uniform metadata it's able to process and render the data to the user. The same is true even the other way around when data is ma data is managed by snowflake but read by a user on datab bricks the only difference is you know what the bare minimum file format uh or table format is is guaranteed as so in the case of data bricks managing it it's uniform in the case of snowflake managing it it's it's iceberg and those are the only two requirements we have for this interoperability to work and the beauty of it is you don't need to know the data path You don't need to worry about when a data path is getting updated, which version of metadata is getting picked up.
It's always the latest that it is tracking through. Obviously, there is like a latency and that latency is just 30 second because it's refreshing that metadata for you behind the scene. But compared to the benefits that we are getting, that 30-cond refresh is um like a minimum constraint that we can work with. Um, so we have a a demo and uh I'll just get right up. So we were hoping to do a live demo. Um, but uh we weren't sure.
Uh, T-Mobile Cyber Security. Everybody knows live demos are kind of uh you're really rolling the dice, so we pre-recorded it. And eagleeyed amongst you might notice that we uh lightly edited to skip the 30 second refresh. Yeah. But um and see interoperability in action between datab bricks and snowflake. For the first scenario where table is managed by datab bricks and read by snowflake we need a uniform table that is stored either externally on adls or on a managed catalog.
For this you would need datab bricks access connector with at least storage blob data contributor privileges on ADLS appropriate datab bricks vnet subnet service endpoint by listing in case ADLS has network restrictions. We must ensure datab bricks compute has open network paths to ADLS and external location listed within unity catalog. We also need a service principle which will be used for catalog integration on snowflake with at least read only permission on unity catalog table and as a registered user within the workspace.
We also need the snowflake application with at least storage blob data reader and delegator permissions on ADLS. We also need appropriate snowflake vet subnet service endpoint wide listing on ADLS in case ADLS has network restriction. And we must ensure SF compute has open network path to ADLS. Lastly, the external volume must be listed within Snowflake. To see this in action, we have a sample uniform table with support for iceberg metadata that we'll be using.
First thing first, here we have the external location which we'll be using to store the table. As can be seen, it's leveraging a access connector connecting to the storage account for listing this external location within Unity catalog. Next, we have the sample uniform table which as shown previously is already created in my case. Important to note is the syntax with respect to table properties that ensures the compatible reader and writer version of delta are used.
Now let's do a record count for this table before we attempt to read this in Snowflake. And here we have 180,420 records. Now on Snowflake side, we need to have a catalog integration. And as the name specifies, it is per catalog designated using the warehouse or catalog attribute in the syntax. You must also list the Unity catalog, iceberg rest endpoint and configure OOTH using OIDC endpoint. using the service principle client ID and secret.
Alternatively, a personal access token can also be used. Next, we need to create the external volume for which you need the container URI and Azure tenant ID. Upon initialization, you'll get a Azure consent URL to register the snowflake application for first time registration post which appropriate roll back access control can be granted on ADLS. Now let us initialize the iceberg table leveraging catalog integration and external volume.
Now we can do a record count check on this table and uh this should match the number we got from datab bricks and it's the same 180,420 records that we got. Now let us attempt to insert new records on datab bricks side and uh then come back to see uh whether those changes reflect or not. So here we are going to insert 10,000 records into the same table. Now once the insert completes, let's take a new record count check on the table.
And as can be seen, we have 190,420 records. So the 10,000 records were added. Going on to the snowflake side. If we take the count again, this should match the number we got which is 190,428 records. Now for the second scenario where table is managed by snowflake and read by datab bricks, we need an iceberg table managed by snowflake and stored externally on ADLS. For this you would need a snowflake application with least storage blob data contributor privileges on ADLS appropriate snowflake vet listing on ADLS in case ADLS has network restriction.
We must ensure the snowflake compute has open network paths to ADLS and the external volume to be listed within Snowflake. We also need a resource account or an identity with read only and owner privileges on iceberg table. This will be used for catalog federation on datab bricks side. And lastly, we need the datab bricks access connector with at least storage block data reader privileges on ADLS um and appropriate data bricks vet subnet service endpoint whitelisting.
We must ensure the datab bricks compute has open network path to ADLS and that would mean we have the capability to list the external location within unity catalog. So on snowflake side assuming you have the external volume created already all we need to do is create a new iceberg table and then let us take a baseline record count which is a little over 600,000 records here on data brick side to create the catalog federation we need to create a snowflake connection first this can be done using the resource identity account leveraging key pair or username password authentication once the connection is created we can create a federated catalog and add the authorized path.
The authorized path essentially signify the root container where iceberg tables are stored. Once that is done, datab bricks automatically does the schema level and table level metadata sync. And uh if we query the same table and take a baseline count here, it should match what we got on snowflake which is again a little over 600,000 records. Now let us attempt to insert net new records on the snowflake side and uh see the results on data brick side.
So we can see a little over 190,000 records were inserted. And if we take a new record count, that's approximately 790,000 92,000 records. If we go and check the record values again on data brick side, we should be able to see the same value. And there we have it. The same 792,000 records are available on data brick site. I hope you found the short demo useful. It showcases some of the promises we see with interoperability between data bricks and snowflake.
Thank you. Now, my man is is is pretty understated in his delivery, but I can't I I you might tell from my body language like I still get really excited about this. How many people have where they work both Snowflake and data bricks? Yeah. Like this is a big deal. You don't have to copy data anymore. You just People have their processes that are running in Snowflake and people have their processes that are running in data bricks and you don't care.

19:38 Challenges Learnings

They don't care. They don't they don't even know. They just access the data where it is. It's just there. Like I don't know for me this I'm pretty excited. Nice job. Thank you. Um there are some challenges and learnings that we've had. Uh there are a lot of like gotchas and speed bumps that we found. Um one of the biggest for us is because we have such a a diverse set of compute and like systems and stuff that's already in place.
Um has been backwards compatibility. Um we have a significant number of people who still use Synapse um and are using the Synapse serverless SQL uh reader and that is not compatible with Delta Reader V2. So um we are actively migrating those people off and we're actively like we have a plan to decommission that but but in order to write your data in data bricks using uniform um it has to it can only be read by delta reader v2.
So that means that like synapse is not going to work. So um compatibility is something that we had to consider. And if you're anything like us where you know your workspaces are uh behind v-nets and the firewalls of the world then that's again one gotcha because the iceberg endpoint is something which you need to have publicly available for snowflake to come in and when I say publicly it's it's not like you open up to the internet but at least for the IP space from which the snowflake compute is running and that way snowflake compute has a network path going into your unity catalog being able to retrieve the metadata and this connectivity is only for the metadata.
There is no data transfer happening here. Um it's just a request response that happens through the eyes per catalog but but still a consideration to have. And talking about some of the architecture consideration and this essentially goes into defining the semantics and the governance of the world. you know like different teams have different practices and uh one common practice that we have seen especially for you know teams who have been in the SQL of the world like you interface through views not through the tables because you want to have that abstraction from the user the no lock behavior and whatnot and some of those patterns have inherited into what we do today.

21:59 Architecture Considerations

Um, so that's one consideration to have because interoperability is on the table. It's on your core data. It's it's not on your definition. So if you want to manage data through views, then you have to do that additional work of keeping the views in sync across two different platforms. Um, so that's that's one architecture consideration to have. Um, data security like encryption versus masking. This plays directly into what we were just talking about with tables versus views.
Um, a really common practice is to mask data using rowle security, column level security in a view. That's obviously not going to apply here. So, you have to think about, okay, am I going to use encryption um in order to encrypt the entire table or something along those lines. Um, whatever encryption you use needs to be compatible with both engines. So you need to make sure that if you're using some sort of encryption uh tool that it it works both in Snowflake or in data bricks or whatever other tool that you're going to use whether that's DMO or or whatever.
Um uh uh there are obviously things that you can do like Mo had just mentioned. You could use views and keep them in sync either you know manually or you could use a tool like Ammuda or Privaca or um or or you could tokenize your data. That's that's a a practice where instead of saying like, hey, I'm going to store my my data in plain text and then encrypt that table or or use row level, column level, you actually just tokenize it using an external service before you even put it in the table.
It's an option. Um, but these are things that you have to think about when you're when you're talking about how you're going to make things interoperable is how are we going to figure out those those requirements and push them down as far as we can. And one such consideration is also around u external tables and managed tables. With managed tables obviously if you are on data bricks you already know you you get a lot of benefits like liquid clustering auto partitioning and whatnot.
But if you have you know um a world where you have things like fabric or you know any other compute platform which doesn't like the grid aspect of how managed tables are managed um then it's it's definitely something that you want to have control over wherein if you define external tables you have the control of which exact path on ADLS it's going to manage that data for you and if you have that control you can work with technologies like fabric and do storage integration.
So um considerations like that are also important uh especially when you are organizing data either as external tables or as managed tables. This is actually an area where Mo hit and I disagree a little bit and that like he's he's more on the external side and I'm much more on the like the if you have a catalog the catalog should be able to abstract that from you and um we shouldn't allow that to stop us. But I mean these are this the conversations that you have to have when you're planning okay how are we going to make our stuff interoperable and um two other things conventions are always um like I know a lot of people can have all day conversation about conventions but it's it's important to really have a bare minimum conventions especially when you know the data products are hosted crossplatform it's important to have the ability to automate things it's important to have the ability ability to have less of tribal knowledge and more of self-s servability.
Uh it's important to build your data contracts and whatnot. So what what we feel is align on a bare set of uh conventions and automate and scaffold your processes as much as as possible. So where where do we go from here and um like for us the road doesn't end here like we are exploring and actively working on making interoperability work between data bricks and fabric um data bricks and GCP or uh the bigquery instance and so on.
So it's it's a process and it's a journey that we have started on and we are continuing to build more on it and that's because we truly believe you know we are we need composible data ecosystem. We we just cannot have one centralized technology stack u because the AI and the MLOF world that workload is uh something that demands us to be interoperable more than ever. Um there are standard bodies who are investing in it creating open blueprints and so on.
And that goes on to define why interoperability is needed more than ever. That's our talk. Uh we'll actually open it up for questions. Yeah, we save a few minutes. If you if you have any questions, please come forward to one of the mics. Um or if you're just really loud, I mean you can take take your shot. Um thanks everybody for coming. Thank you.

Learn more about the Databricks Data and AI platform.