How to Build an Open Lakehouse: Best Practices for Interoperability

Summary

• Open lakehouse reference architectures: Design interoperable lakehouse deployments across AWS, Google Cloud, Azure and integrate with Snowflake, BigQuery, and Microsoft Fabric on the Databricks Data and AI platform.
• Open table formats and governance: Leverage Delta Lake with open table format standards and Unity Catalog to ensure cross-platform data access and unified governance across your entire ecosystem.
• Future-ready interoperability: Build data architectures that remain flexible and vendor-agnostic, allowing seamless data movement and access without vendor lock-in on the Databricks Data and AI platform.

How to Build an Open Lakehouse: Best Practices for Interoperability

Talk by Aniruth Narayanan, Product Manager, Databricks ; James Malone, Product Manager, Databricks
Building an open data lakehouse? Start with the right blueprint. this video walks through common reference architectures for interoperable lakehouse deployments across AWS, Google Cloud, Azure and tools like Snowflake, BigQuery and Microsoft Fabric. Learn how to design for cross-platform data access, unify governance with Unity Catalog and ensure your stack is future-ready — no matter where your data lives.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and agenda
• 2:08 Defining the open lakehouse
• 4:26 Open table formats
• 6:52 Unified governance via Unity
• 10:08 Interoperability architecture
• 13:29 Guiding principles recap
• 15:51 Practical implementation
• 30:44 Hyperscaler architectures
• 35:12 Conclusion and next steps

FAQs

What does 'open lakehouse' mean?

An open lakehouse uses open table formats (Delta, Iceberg, Hudi), unified governance (Unity Catalog), and vendor-neutral architecture to enable interoperability across cloud providers and tools.

Can we integrate with Snowflake and BigQuery?

Yes, open lakehouse architectures support seamless integration with Snowflake, BigQuery, and Microsoft Fabric while maintaining central governance through Unity Catalog.

How can AI systems understand data across different platforms?

Implement unified metadata governance through open standards like Apache Iceberg and Unity Catalog on the Databricks Data and AI platform. This enables AI systems to discover, understand, and access data semantics consistently whether data is stored in Databricks, Snowflake, or BigQuery. Open table formats ensure data portability and prevent AI models from becoming locked to specific platforms.

What is the vision for open lakehouses in 2026?

The open lakehouse paradigm emphasizes interoperability, avoiding single-vendor dependencies while maintaining governance and performance. The Databricks Data and AI platform leads this movement through open formats like Iceberg and Delta, standardized APIs, and partnerships across the data ecosystem.

What is the best table format for building a new open lakehouse?

Apache Iceberg is emerging as the gold standard for new lakehouse builds in 2026, providing read compatibility across Databricks, Snowflake, Trino, and other platforms. Iceberg's specification-first approach ensures true interoperability. On the Databricks Data and AI platform, Delta Lake offers tight integration with Spark and governance, while Iceberg enables multi-platform deployments.

Full transcript

0:00 Introduction and agenda

Good morning everyone. Hopefully I didn't startle anyone. This mic's a little weird for me to get used to. Hope everyone had a good time this morning at our keynote here at the data bricks data and AI summit. We are very excited to talk to everyone about how to build an open lakehouse. If you are here and you are not interested in how to build an open lakehouse, unfortunately we will talk anyway about how to build an open lakehouse.
With that being said, before we talk about all the great things that we've been working on, just like every airline flight that you've ever been on, we have a list of and other things to go through. Please take a picture of this slide with your phone and chat GBT so you don't actually have to read it. The whole point here is just that we're making a lot of future-looking statements, not all of which may be true, but most of them might be.
We also encourage everyone to complete the surveys. I personally ask you to complete the surveys. Apparently, this is reflected in my performance reviews. So, please mark that I did everything great. Uh do appreciate it. With those out of the way, we do want to talk a little bit about how to build an open lakehouse. What are our best practices for interoperability and how can we achieve this data architecture in all of ours going forward?.
I'm a product manager here on the storage team. I'm joined by who it's interesting to see you here. I feel like normally I've seen you at Musone before. I have been, but usually the week before, not this week, in like a different costume with more blue. That's true. I came somewhat recently from Snowflake. It's good to be back. My uh last Spark uh conference was the Spark Summit in 2017. I recently joined Data Bricks specifically to get back to Spark because it's something that's near and dear to me.
Well, it's good to have you here. Nice to be here. Uh with that, we want to talk a little bit about three things. We want to start with some guiding principles for what we think an open lakehouse should be. We then want to go into some example open lakehouse architectures across the clouds and various tools that you may use in your platforms. And then finally, we want to touch on how interoperability works across not only engines but across tools and entire ecosystems that you may want to use.
Okay. To do this, we want to try to make it real. So, we're going to run through a hypothetical example using a theoretical company, the San Francisco Paper Company. And the goal here is we want to speak to the pains that either you are feeling or people in your organizations are feeling which is why we're using this scenario. So let's meet the team behind this fictional paper company. We have a data engineer. is dealing with the pain of having duplicated data essentially in data silos.

2:08 Defining the open lakehouse

There's data everywhere between different systems. There may be duplicate data. It's just kind of a mess. Then we have our platform architect. is dealing with the pain that governance is spread across different systems. Essentially, each different system wants to be its own catalog, wants to enforce its own governance. So, trying to create a single pane of glass or a single picture of governance is really problematic.
Third, we have our business analyst. needs to get access to different data sets, but doesn't necessarily know where to go, uh, what to look for, or what system has what. So Blake's job, a lot of it is just spent trying to find the right data and quering across platforms. Our goal here is to kind of give you the IKEA manual for building an open and interoperable lakehouse. You're going to see a lot of kind of generic diagrams, a lot of kind of abstract principles.
We're trying to give you the building blocks to go build an open interoperable lakehouse. So let's look at the pains that these three personas are feeling. We have a lot of duplicated data. Well, what's the solution for that? A single source of truth. Keeping one copy of data that multiple systems can access rather than having fragmented governance split across many different systems. What's the solution for that pain?
Trying to unify that so you have a single pane of glass for governance, control, access to data, and a way to understand and find your data. You don't want to spend all of your time in navigating these different platform silos. So what's the solution for that? Well, having an open, interoperable, and pluggable data architecture. So let's put together the tools, capabilities, APIs, governance, and federation that you need to build an open lakehouse.
So let's start with the base layer of our lakehouse, the stickiest part and potentially the most concerning part, the data format that we're going to use. And this was alluded to this morning in the keynote that mentioned. We have this these two open formats called delta and iceberg which are pretty important. All these are is just a bunch of parquet files or data files that are tracked by various metadata and these metadata is slightly different between the formats but they remarkably similar and they provide a host of benefits.

4:26 Open table formats

In fact, they are what allow us to build the lakehouse architecture because these are open so any engine can understand and work with them. You also have vendor neutrality. So there's no ability for one vendor to make a decision on an open table format and lock you in or force you to use that. There's also high performance. A lot of the efforts with open table formats, especially with the new Delta 4. 0 release and the iceberg v3 release in the past few weeks, are really focused on taking a lot of the proprietary format benefits and bringing them to the open table formats.
We also benefit from the open source communities where again similar to vendor neutrality, there's a lot of folks that are involved in these, right? And it's not just vendors. A lot of the big tech companies and other folks that actually contribute to the formats are able to weigh in and make sure that we're making the right decisions. Has anyone here been through a Vertica, Natiza or any of the other various data warehousing onrem?
Oracle is a very common one. I said Oracle. I should see every hand go up in the room. I know everyone here has Oracle in some way, shape or form. And the whole point of this is to avoid those kinds of migrations, right? where there's no issues with the kinds of formats that you have and you're able to futureproof your data architecture going forward. The only thing though is that lakeouses go beyond the formats. So we've been spending a lot of time in the communities talking about Delta talking about iceberg but we also have a whole variety of other things.
You might have unstructured data, you might have models, you might have files, you might have notebooks and all of these need the same governance that you need across the board. In fact, you need something that can track all of these things just like you would all the various formats. In fact, this is what the data engineer says, right? Not only does she have all those various assets I listed out, even at the actual table level, it's not just Delta and iceberg.
Now, we're talking about JSON, parquet, Abro, OC format, maybe even more. And our solution here is to try to find some cataloging solution that can track all of it. And that's where we really try to position Unity catalog where we think beyond just the format, just the data that you may have to all of the assets that you have inside your lakehouse. Okay, so let's unpack that. Let's look at unified governance in a catalog specifically Unity catalog.

6:52 Unified governance via Unity

So again, we have Po, our platform architect. He's trying to ensure governance across many different systems. The problem is is all of these different systems and many of you have probably felt this pain have inconsistent and different governance solutions systems. Some may have great solutions. Some may have very weak or no solutions. That ultimately means that you have a bunch of different systems. Each one may have its own catalog.
Each one has its own way of doing governance of role-based access control of controlling access to data. So if you're a platform architect, you're sitting there thinking how do I wire all of these together? Otherwise you have to go replicate your governance and your security and any number of different systems and it becomes very very time consuming. Then you have the business analyst who's really just trying to get access to data.
So Blake's main problem is I know that somewhere there is this data set. I don't necessarily know where it is. I don't know how to get access to it and I don't know how good it is. So he faces discoverability barriers to actually make use of the data in the first place. What if hypothetically there was a way to try to solve both of these problems at once? And there is. It's through a unified catalog. It's a way to solve both the governance problems.
You can have one system control access to data and enforce governance. You can also have one system surface that data. make it really easy to find what you need, what you're looking for, and how to actually make use of that data. So, Unity Catalog solves both of these governance problems. First, Unity Catalog can centralize governance between all of these different systems. So, in this example, Unity Catalog could tell somebody or a system where is the actual data stored and then vend credentials to actually go make use of that data.
So when any particular person or system wants to go make use of data, Unity catalog is essentially telling you where it is and also giving you the provision to be able to go make use of that data. So you're going from many different systems controlling access to data down to one system. You then also solve the data discovery problem. You can have one system provide access and insights on where the data actually is and also provide lineage on top of that data so you know who's using it and potentially for what purpose and what they're doing with the data.
In this case, you could have my SQL, Postgress all surfaced inside of Unity catalog. This is really important because you can have one catalog project its capabilities over other systems. So it's not necessarily data that's just in one system. this Unity catalog layer is being applied across different systems. So it's a really powerful way to tie that governance together. So great, now the platform architect is happy because across all of these different systems, you have one mechanism that's providing a consistent way to enforce control and access to data.
And you have one system that's providing a universal way to be able to find data regardless of what platform or system it's in. So let's talk about interoperability. Our final principle, opening up access to any engine or platform. This is really the holy grail of what we're trying to achieve and sort of the motivating reason that we're trying to go after this open lakehouse architecture. So we start out with this idea of the catalog.

10:08 Interoperability architecture

It's an inventory of all the data assets that you have. But it's not enough to just be able to track it in one system. You want this catalog to work across your systems. Specifically, we have all of our delta engines. We have our iceberg engines which both make sense, right? We can use some sort of API or agreed upon interface to go between those engines and the catalog. We also have our data sharing use case which is very commonly interorganization where now you're thinking beyond just teams within your company and now perhaps you curate some data products or you produce some final outputs that typically you sell or share with another entity.
But there are also some engines missing from this picture. Right? If you notice there's Delta engines, there's iceberg engines. But maybe what if wants to write from Athena, right? He wants to write some table from the Athena engine in the AWS ecosystem, but he somehow wants to be able to use Unity catalog to leverage the same data, right? Achieve that open lakehouse. So we have Athena set up over there and we have Unity catalog.
But the only problem here is that if you go into Athena and you try to set up Unity catalog, it just won't work. Right? This is a very common problem that we face, right? where an engine for some reason is gated behind some specific catalog or some specific entity and in this world we say okay Athena will go ahead and write to glue and we can use glue in the AWS ecosystem but now we start thinking between cataloges right and so we have this world that we call catalog federation in which unity catalog goes and gets the underlying table locations from another catalog allowing engines that go through unity catalog to execute directly on table files and this is very very important The distinction that I want to make here is a system where you have to incur Athena computes can talk to each other to figure that out.
And as long as that happens, we can now access the data. So let's go through an animation of what that would look like. So let's say you have the data bricks engine connected to Unity catalog and now you're trying to figure out the right metadata for this table. So on every single read, data bricks will go to Unity catalog and ask, "Hey, where's this table that Athena wrote?" Unity catalog then goes to Glue and then gets the metadata path from Glue.
So whatever Athena has written, you can be sure that you're going to get the freshest, latest data. And then finally, data bricks will directly execute those queries on the underlying table files. And because we've been using open table formats this whole time, there's no need to invoke Athena at all in this query path, right? It's just data bricks occupying on another table as though the table was directly in unity catalog.
And therefore we can complete this diagram by actually adding in this point of interoperating with other cataloges right where unity catalog supports working with HMS iceberg rest catalog glue and snowflake horizon catalog. In fact it's fully interoperable across all of them where we have the APIs across the delta ecosystem and the iceberg ecosystem. We have delta sharing which despite the is a generalized open sharing format.
And then finally we have federation which we just discussed when working with other cataloges. Okay. So we've gone through the three guiding principles. And I just want to say at the outset uh we're here providing a lot of kind of prescriptive guidance and architectures. If you want to see a real example of this in practice, I would encourage all of you to catch the keynote tomorrow. There is going to be a very interesting demo of this interoperability in practice.

13:29 Guiding principles recap

So, just as a teaser, I'm not going to spill any more beans. Uh, but going to be providing just what in my view is an awesome demo because it combines uh one of my favorite non-Spark query engines uh with an interoperable catalog. So, just as a quick shout out. Uh, but let's recap the three layers that we have. We have the single source of truth that's founded on open table formats. And if you saw the keynote, the data bricks keynote this morning, you could choose iceberg, you could choose delta.
Even if you're not using data bricks, I'd still advocate you use an open table format with your data because you're going to have the foundation of a single data set that is understandable across different systems. If you if you don't do anything else, if you don't remember anything else uh from this talk, I think that's the foundational piece that I would really encourage everybody to go chase. On top of that, you have a single unified governance layer.
And that single unified governance layer enables pluggable interoperability between different engines and different tools. And that uh interoperability layer should use potentially different APIs. And we'll show this in practice, but there's going to be different APIs that are used so different engines can essentially speak different things to the underlying Unity catalog. We've not actually shown these principles really how they're they're applied.
So we're going to slowly start making that transition. This is another way of looking at the three principles blended a little bit with an architecture slide. So you can see that you have that interoperable single source of truth at the very bottom. It's the basis on which everything is derived. You then have a lakehouse catalog, Unity catalog sitting on top of that data and that creates a surface area for governance and discoverability that different use cases can use for that data whether it's ingestion, transformation or consumption.
So we're going to unpack all three of these different areas in in depth so you can look at how does ingestion work on an interoperable lakehouse. How does transformation look like and what does consumption look like across that lakehouse. So let's put these principles into practice. Whether you're using ingestion, transformation, or consumption, they're all talking to Unity catalog underneath. Why? Well, you want that single source of governance.

15:51 Practical implementation

You want the single source of control. In the examples that we're going to show, the architectural diagrams we're going to show, just as a disclaimer, these are not exhaustive. Uh and there's going to be uh in some cases an overwhelming number of different icons. The reason that we're showing so much detail is different customers, different people use different things. We want to show you something that essentially everybody can relate to.
So nothing in one of these slides is going to none of this is going to apply to everybody, but some of this will apply to all of you, if that makes sense. So first, let's look at ingestion. So let's say we have, our data engineer, and she works at the San Francisco paper company, and she's trying to understand how to use Kafka. And in this scenario, she ingests from a number of sources, maybe some streaming sources, some APIs, maybe some files.
And she's trying to understand how she can use her existing Kafka info today where perhaps she has some message bus set up to ingest from all these sources, but she wants to write it and persist it in some format. So she uses data brick structured streaming as a sync here, maybe some light ETL on these processes, and then we write to the Unity REST APIs that Unity catalog exposes from the data bricks engine. Of course, there's a question with ingestion that frequently comes up.
How does your data land? And in this scenario, because we're writing to the Unity REST APIs, which is the Delta ecosystem, our data lands as the Delta format in the bronze layer. And so already we've started to see that we're this is a very common pattern. We noticed this may be a pattern that many of you are familiar with. But now let's start to make it a little bit more complicated. Let's start saying that instead asks a question about using the same Kafka infrastructure, but now landing the data in the iceberg format instead of in the delta format.
So there's a couple interesting nuances here, right? So perhaps there's this open source Kafka connect that she can use, right? It's a different sync instead of using data brick structured streaming. And Kafka connect is great because it'll write into the iceberg rest APIs. And so in this world, Unity catalog exposes the same kind of ecosystem where you can now just plug into a different set of APIs and the same Kafka infrastructure can now be leveraged to write to Unity catalog.
But there's one very interesting note here that I want to point out. Right? If you notice the first question we started out with on this slide, it's now about ingesting into iceberg. So what happens to all the delta stuff that exists, right? Does it matter if she has two formats? And the answer is no. This is a very important point that I want to make sure we emphasize. As long as you use either of the open formats, you're fine.
In reality, they are very, very similar. It's the same paret file shared across both formats. It's slightly different metadata, but the great thing about Unity catalog is it acts as this middleman, right? Where we can expose those tables on either set of endpoints that you want. So you get all the engines you need and you can also have whatever tables you want. So there's no need where for you to ever be concerned about having delta or iceberg or some combination of both.
In fact, it's a very common pattern. And you'll see that on a lot of the slides, we just have both because we assume that it's a solved problem, right? Where you can have your data on delta, you can have your data on iceberg and things work. And that's why our first principle was so important, right? If you notice, it's not that we're saying you should have your data duplicated across formats. It's that you do have a single source of truth as long as you use the open formats.
But finally, has this question about going beyond just the event streaming and APIs and files that she was using with Kafka. Maybe for example, she wants to use Workday, HubSpot, or Google Analytics. Maybe something that doesn't quite have a great connector into something that she can use today. And for this data bricks for the ingestion piece has a great partner ecosystem for example with fiverr where fiverr can connect to all of these sources right to the unity rest api that unity catalog exposes and again a very similar pattern where we have that single source truth in the open table format on the bronze layer.
If you notice there are a couple patterns here that we've been emphasizing and we're going to continue emphasizing them throughout the presentation. So keep an eye out and see if you can find what those are. So let's look at transformation. You've ingested the data into the single source of truth. You need to transform it to make it usable for any downstream use case. Brought a data bricks event. I would be remiss not to say first you could use data bricks to transform your data.
So you could use data bricks spark use data bricks SQL for etling this data. In this case, both tools are going to talk to the Unity REST API to work with Unity catalog, get access to the underlying data, mutate the data, do whatever transforms are required. One of the interesting things to note is Unity catalog is also going to use liquid clustering. So, it's going to autooptimize the data underneath. It's going to autocluster, find the right patterns and keys for the data.
So the catalog itself is automatically optimizing the data underneath. What's the end result? Well, you're in the data bricks ecosystem. It will make your ETL more performant. You're going to get better price perf per performance. So an interesting benefit of Unity catalog. But what if we want to use Amazon EMR for the transformations? It's a Spark runtime, so it makes a ton of sense. Sure, use EMR for your transformation.
In this case, Spark on EMR would also be able to talk to the Unity catalog REST APIs. You could do the same transformations in EMR. There's absolutely no reason you couldn't. You have the single source of truth. You have unified governance layer. You want to use a different Spark runtime. Awesome. One interesting benefit here though is since the table optimizations in this case the liquid clustering is done at the catalog level those performance optimizations aren't specific to the data bricks ecosystem.
So the liquid clustering that made data bricks faster is actually also going to benefit Spark in this uh use case on EMR because all of that optimization is done at the catalog level and not the Spark runtime level. What if you want to use Google Cloud Data Proc? Cloud Data Proc is near and dear to my heart because it was the first product I launched as a product manager uh when I was at Google which uh got me into the Spark ecosystem and part of why I'm here today.
Uh but what's interesting here is kind of two things. First, you're multicloud now. You could you be using uh go from AWS maybe you're running data bricks and unity catalog and inject a Google cloud spark runtime into the mix. In this case data proc is going to talk to the iceberg rest APIs that are on unity catalog. So you can see how the catalog having multiple API surface areas allows for a greater net of interoperability because different platforms tools runtimes and engines can talk to these two different APIs.
So, it's like having uh multiple front doors on your business and different people, you know, trucks may want to use the loading dock and people walking in may want to use uh what looks to be just a normal uh front door. The other interesting thing about this is those liquid clustering optimizations in Unity catalog that worked for data bricks that worked for EMR same thing they also apply to data proc because the optimizations are at the catalog level.
What if you want to use Snowflake? Cool. You should absolutely use Snowflake if you already have it and you maybe you have a bunch of ETL that you've already built in Snowflake or you want to use Snowflake for a specific use case. Again, Snowflake just like data proc can talk to the iceberg rest APIs. You can do all of your transformations in Snowflake. You still have the single source of truth. You still have a unified governance layer.
And in this case, Unity catalog is going to tell Snowflake, here's where the data is, provide all of the metadata needed to access the table and also provision uh keys or credentials to be able to go interoperate with the data set. So again, if you want to go use Snowflake, no problem. And again, since the optimization is done at the catalog level for something like liquid clustering, it applies to Snowflake as well.
Wait, hold on. Hold on. I've I've seen a variation of this slide before that shows the snowflake horizon catalog and unity catalog and has arrows between them and then there's arrows to the snowflake storage and arrows to the snowflake compute and then arrows to the data bricks compute and then arrows to the data bricks storage. I might be forgetting a few arrows here. Why? What happened? Why is this so you must be seeing arrows in your sleep, man.
Uh what changed? Uh what's interesting is Snowflake just announced support and to their credit they announced support for not only reading but also writing to foreign iceberg cataloges which is great as people standardize on the iceberg rest API it makes life much simpler. So instead of having to tinker about with all of these arrows and spending uh time thinking about how to wire everything together uh Snowflake can actually read and write against the Unity catalog iceberg rest API.
So you don't need to think about using multiple cataloges and injecting multiple cataloges into the stream. Everything can just work against the single iceberg rest API that's exposed inside of Unity catalog. So uh makes things much much uh simpler. And now that we've etlled our data, and, our business analyst and our platform architect want to make sure that we have the right consumption tools set up. And similar to last time, data bricks has DBSQL, one of the fastest growing products that data bricks has ever introduced for warehousing.
So perhaps we can connect our DBSQL warehouse for consumption to Unity Rest. But perhaps wants to use PowerBI and AIBI. In other words, they want to set up some form of dashboarding. Maybe their company already uses a lot of PowerBI. And the great thing here is that DBSQL warehouses are extremely flexible. If you're using Azure data bricks, for example, you can use PowerBI that's powered by your DBSQL warehouses. And the same goes for your AIBI dashboards.
It really is about giving you the flexibility where you make sure that data bricks has to win every single workload, right? Every single thing that you use with data bricks, we have to keep fighting for and making sure that you want to use it. At any point in time, you can always go to the catalog and use something else instead. And so been a recurring theme throughout this talk. In fact, this is our third principle here really being applied, right? have interoperable connectivity not only directly from the catalog but even within other services inside the data bricks ecosystem as well and we also have the same governance principles applied where the identities that are used with DBSQL warehouse can still apply in other systems that leverage that form of compute but perhaps you want to use Snowflake for warehousing and in this setup you have maybe some are you actually telling people to use Snowflake you know we're at the data bricks conference like to be fair to be fair I we we do realize that a lot companies have existing Snowflake, right?
Snowflake was one of the first cloud data warehouses and there is a lot of existing usage and it's hard to rip that out, right? It's one thing to say we're going to do an entire lift and shift migration versus saying maybe we can plug in data for some of our ETL workloads and keep some of our analysts on Snowflake. It's a very common coexistence pattern. So no on the surveys you do not need to to emphasize that either.
What about the Delta Direct support? because I'd seen that snowflake may have had a hat and builtin in the past that snowflake has delta support but that's not reflected here it's all iceberg why that's a great question right so if you look snowflake here is connected to the iceberg rest API is very similar to what we had with the ETL scenario and so the question is well snowflake can also create iceberg tables from delta files and object storage that's what they call it on their documentation so why can't you just use that right like you already have delta set up with data bricks just keep doing that and the problem is it violates our second tenant, right?
Where the second tenant was about leveraging governance, right? We want to make sure that the catalog is involved with query access patterns to your data, right? The whole reason is imagine a world in which this doesn't happen. Now, you have to govern Snowflake's direct access to whatever object storage you're using as well as connect snowflake to Unity catalog to get access to which table location you have. And the world that we want is one where the catalog does both of those things, right? where we vend the credential that talked about earlier both in terms of the storage location and the access credential to access the actual files and that is only possible if you're going through the ice for rest APIs and not around the catalog directly to object storage and of course our third principle is also in effect here as well as the first principle right we're using open formats and we're getting connectivity via these APIs throughout as well or perhaps you want to use Redshift and Athena for warehousing and you want to talk a little bit about the AWS ecosystem them.
So this scenario is a little different, right? We talked in the beginning slides about the scenario where Athena was writing a table and he wanted to access the table inside Unity catalog. And a very similar problem exists here where Redshift and Athena want to talk to Glue, right? They don't want to talk to Unity catalog for whatever reason. And so in that scenario, we can still use Redshift and Athena as our warehousing engines.
We just need to include Glue here as this intermediary catalog, right? So this goes back to this idea that for many cases there may be worlds in which you have multiple cataloges at play. The important thing is that the cataloges all talk to each other over the same APIs that the engines would use as well. And so in this world it's a very similar pattern where instead of now the data bricks engine reading the files that are managed by Glue, now the AWS engines are reading the files that are managed by Unity catalog. of course made possible by the fact that we have opent formats and that we're able to plug into glue across the entire interoperability ecosystem.
So really the way that we're thinking about it is it's not just engines that are being connected but it's entire platforms that are being connected via the APIs. And finally this wouldn't be a talk at data bricks if we didn't talk about ML somehow some way. So we do want to throw in a slide here that oversimplifies mosaic AI to a great great deal. There's a lot more things that it can do, but the high level idea is you have a lot of data that Unity catalog has and now you want to use this data to train machine learning models and of course you may want to serve them as well and that's totally possible in data bricks today but we also have plenty of customers that may want to use SageMaker a different cloud provider platform and it's a very similar pattern as can already guess where you're just going to use glue where very similar to the Athena and Redshift stuff that I just showed you'll now just use glue to go ahead and connect in as a foreign catalog to the iceberg rest APIs and then connect SageMaker.
One thing that I've left out of this slide is about serving, which we just talked about. The reason why is because there's a whole other talk dedicated to just AI gateway and how all the serving stuff works. But I do want to mention that there are tools as well within Unity Catalog to extend your governance at the serving layer. So it's possible to use something like AWS Bedrock or Azure OpenAI to serve your models with the same governance that you have from Unity Catalog and leverage the same data that you have within the catalog today.
So it really is possible to get all of your across the board via the Unity catalog interoperability, open formats and unified governance. And I want to just reinforce as we kind of look at how this might work in hyperscalers a really important point. What we're really trying to tell you here is uh two things. One, don't give up on a single source of truth and a unified governance. Two, data bricks should fight for your individual workloads.

30:44 Hyperscaler architectures

With a pluggable architecture, the proof should be in each job, each workload, each use case. Uh we wouldn't be up here talking about interoperability, encouraging you to go down the rabbit hole of interoperability if a we weren't confident that we could win, and we won't win everything to be honest. That's just the reality. But b uh if we didn't think it was the right thing that you have the flexibility to choose the right tool for the job.
So I just want to say the interoperability piece here is not lip service. It is from a place that data bricks really should be fighting for each individual workload. It'd be much easier to just say you know you have to buy something you have if you want to use a small thing you have to buy a very large heavyweight platform. Uh we're very much trying to advocate uh against that. So you can take a picture of this slide.
Uh there's going to be a couple you don't have to take a picture but you know people love taking pictures. Um, there's a lot of diagrams in this slide and the following slides. And again, this this falls into the category of we're showing you more than probably any one person or anyone company uses, but we want to show roughly how the different pieces that most people use in one way or another kind of fit together. So, uh, in this case, you can see that we have that single source of truth.
There's open table formats in AWS S3. We have Unity catalog that's federating to Glue and we understand from AWS that additional federation with glue will be coming this summer which is great. Uh that means that glue will be able to really interoperate and federate with Unity catalog. We have elastic map produce talking to the Unity REST APIs. Uh what's interesting is because you're running Spark just as a side note you could go use the iceberg rest APIs as well.
We're showing one set but uh there's flexibility here on which APIs you choose to use and on the consumption layer you can choose whatever you want to consume the data in this architecture. This is uh creates a lot of flexibility for you but the unity catalog is that unifying piece that allows many different systems to talk uh together and have a single source of truth and unified governance layer using open APIs. If you're an Azure customer, here is a eye chart for you.
Uh there's a couple of really important pieces here. First, uh and this was announced earlier this year, there's an SAP integration with data bricks and a SAP data bricks product and that's using heavily using delta sharing which is uh a really neat use case for delta sharing. We announced this somewhat recently. Not enough people know about it. I'm going to tell you uh you can orchestrate data bricks jobs from Azure data factory and then you can use data brick SQL to power uh PowerBI use cases and again this fits into test it out if you want the best price perf performance for powering powerbi you could use data bricks to go power that use case if you want it's all powered by the interoperable lakehouse pattern and finally Google cloud something that's near and dear to uh to my past.
Uh again, you have that single source of truth, multiple open table formats in Google Cloud Storage. You then have Unity catalog sitting on top of that data offering you a single source of truth uh and governance on top of it. And then you have data proc interoperating with Unity catalog. Again, it's a Spark runtime. So you could use the iceberg rest APIs, you could use Unity catalog APIs. uh BigQuery doesn't federate yet with other iceberg rest cataloges.
If you are a Google customer and you're using BigQuery, uh reach out to your account teams and uh on the data brick side so we can find the right solution to you. Uh we've seen Google incrementally taking steps towards iceberg. So if anybody here is from Google or you use Google, just continue uh putting wind in the iceberg sale because the that interoperable open API helps everybody interoperate. It's a good thing for everybody.
So, uh, reach out if you're using BigQuery. To recap, the three guiding principles we have with the single source of truth, open table formats. And remember the keynote this morning, you could choose iceberg, you choose delta. The table formats are actually very similar at this point. Uh, there's really no wrong answer. You then have a unified governance layer on top of that data. It's telling you where the data is. It's provisioning access to that data.

35:12 Conclusion and next steps

So for example for Snowflake it's providing a location of the data and handing brokering uh security tokens. So in this case snowflake can access the data and then you have interoperable connectivity on top that's made possible because the catalog itself is using a set of open APIs that then point to data that's using an open table format. Ultimately that means that our uh data engineer is happy. there's a single source of truth.
Our platform architect is happier because now there's one place to do governance and people using the data or the business analyst is happy because there's one place they can go to find all of the data assets that they may have. Thank you. I think this is the last handoff, so it's great that we haven't dropped this. When we were practicing, we had no idea there'd be a clicker involved. The overarching part that we were making with Unity catalog at the very beginning was that we really see it as this lakehouse catalog as it relates to those guiding principles.
So everything that just mentioned is greatly supported by UC where we have support for those open formats and all the various asset types that you may have in your lakehouse. We have unified governance where we can integrate governance with your data access and management. Really taking the catalog beyond just a catalog into this whole realm of doing things that a governance catalog and a business catalog will do. And then finally, we also have UC interoperating with other entire systems and engines, right?
Where we can expose the relevant table information that you need to connect whatever engine, whatever system you want to all of your data in however way you may need it. And really, when we've been talking about Unity Catalog, we've been talking about it like this, where it's just some large thing that can perhaps do some stuff with your Delta, your iceberg, your parquet files. But in reality, there's so much that we've left out that the other Unity catalog sessions are going to go through, right?
It's really just an enterprise ready solution that does so much across so many different things that we haven't even talked about like discovery, quality monitoring, business semantics, cost controls, data quality. There's a very very long list and also all the various things that it has going forward on the openest side and the connectivity pieces. I will pause here. This is a great slide to take a picture of. I do know I have it screenshotted in quite a few of my decks.
Uh, and really the whole point I've been trying to make throughout this entire thing is that you need some system, some basis to base your entire lakehouse architecture of. And we really think Unity catalog solves those problems for our customers today. With that, we've talked a lot about how to build an open lakehouse, but we want to make sure that you can continue your learning journey. I don't really know much about this slide other than they told me a few times to add it in.
It looks like there's self-paced options, instructorled training, and certifications. We do have a QR code if anyone wants to scan it here to go through some of these options. But that wraps up our talk here today on how to build an open lakehouse. I hope you all enjoyed. and I will be around to answer any questions as needed. Thank you. Thanks.

Learn more about the Databricks Data and AI platform.