The Missing Link Between the Lakehouse and Data Intelligence

Summary

• Unity Catalog as lakehouse linchpin: Unity Catalog bridges the lakehouse and data intelligence by providing governance, discovery, and interoperability that operational/business catalogs cannot.
• Cross-platform interoperability: Unity Catalog enables data sharing and collaboration across diverse data systems, cloud platforms, and organizational boundaries.
• Shared governance model: A unified governance layer simplifies managing security, compliance, and data quality across the entire data and AI stack.

The Missing Link Between the Lakehouse and Data Intelligence

Talk by Jason Reid, Director, Product, Databricks ; Michelle Leon, Staff Product Manager, Databricks
What connects your lakehouse to real data intelligence? The answer: the catalog. But not just any catalog. In this video, we break down why Unity Catalog is purpose-built for the lakehouse, and how it goes beyond operational or business catalogs to deliver cross-platform interoperability and a shared understanding of your data. You’ll walk away with a clear view of how the right data foundation unlocks smarter decisions and trusted AI.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why isn't a traditional business catalog sufficient for lakehouses?

Traditional catalogs track metadata but don't enforce governance, manage access control, or ensure data quality at scale—Unity Catalog adds enforcement, security, and active monitoring.

How does Unity Catalog enable cross-platform data sharing?

Through Delta Sharing and open standards, Unity Catalog allows secure data sharing across cloud providers, data platforms, and organizational partners while maintaining governance.

What are the differences between Unity Catalog, Iceberg REST, and Hive Metastore?

This video explores the missing link between the lakehouse and data intelligence on the Databricks Data and AI platform. It explains governance differences: Hive Metastore provides basic metadata management, Iceberg REST offers catalog compatibility across platforms, and Unity Catalog provides comprehensive governance across all data types and AI assets. Each serves different architectural needs in modern data systems.

Who improves interoperability with open data formats?

This video demonstrates how the Databricks Data and AI platform improves interoperability through open formats like Apache Iceberg and open APIs like REST catalog protocols. By supporting multiple catalog standards and open table formats, Databricks enables seamless integration with other platforms while maintaining the governance advantages of Unity Catalog. This open approach bridges the lakehouse and intelligence layers.

What's the difference between operational, business, and Unity Catalog?

Operational catalogs track technical metadata; business catalogs provide business context; Unity Catalog provides all of this plus access control enforcement, data quality guarantees, and audit trails.

Full transcript

Hi everyone. Hello. Hello. Hi. Okay. Uh let's see. Let's get started. I first maybe wanted to start off introducing ourselves. We're here to talk about what is a catalog, why do you need one, what's the missing link between Lakehouse and data intelligence. I'm a product manager here at Data Bricks. I work on our open lakehouse offerings. I'm based in San Francisco and you can talk to me about Unity catalog interoperability, open table formats, and then where to find the best burritos in the mission neighborhood.
Very good. I'm gonna I'm gonna follow up with you on that. Uh so I'm a director of product here at Data Bricks as well. I came last year by way of the Tabular acquisition. So I was a co-founder at Tabular. We were the iceberg based company. mentioned his keynote this morning that you know now data bricks full interoperability with iceberg as a result of the acquisition. We've been working all year on that. Been working closely with and our team to get that done.
Um and you can also talk to me about data engineering at scale. I've done a lot of data engineer engineering in my life. I ran data engineering at Netflix for many years before doing the startup. I spent a lot of time because I was at Netflix uh and I used to talk to vendors like data bricks and snowflake and Amazon and so forth. Spent a lot of time thinking about build versus buy decisions at Netflix. So, that's an interesting topic that I have a lot of experience with.
And I can talk about the best beers in Mountain View. So, we can compare notes. Compare notes. Okay. Well, I'm I'm even know they had burritos in Mountain View. Oh my goodness. Okay. All right. We'll have to we will have to follow up. We will have to follow up on that. Um All right. Well, I'm excited to get started. So, we have a packed agenda. I think uh like say like 35 more minutes together. Uh so, we're going to start off maybe talking a little bit about what is a lakehouse catalog.
I mean, I think that catalog is almost a little bit of a overused, overly used term right now. There's a lot of uh a lot of different types of cataloges out there. So, we're going to uh talk a little bit about what is that catalog and why do you need one? Uh talk about some of the challenges today and then we'll talk about how Unity Catalog addresses some of those challenges. And maybe to start off just with like a quick warm-up, wanted to get to know all of you a little bit better.
So, we have a little interactive quiz. It's not a quiz. It's like more like a It's not really a quiz, I would say. Uh maybe some a quick survey. You won't be graded. There's not going to be no prizes. There won't be We won't be graded. knows I like to do these internally as well. And there are grades for those and sometimes prizes. And sometimes prizes. It's true. Not today. Sorry. We should have had a prize at some t-shirts.
I know. All right. So, is everyone hopefully everyone is in? And then I think we can go to the if we go to the other tab results coming in live. Live. Let's do it live. real time data. Oh, that's really cool. Wait. Oh, you're able to see. That's awesome. Okay. Well, okay. Well, what technical catalog do you use today? All right. So, we have zero people using Hive Metas store, which is honestly surprising. Okay. We have a I'm very glad to hear that person.
Yeah. Yeah. Uh, and then a bunch of folks using Unity. Amazing. Glue. Um, iceberg rest catalog. Unity is an iceberg rest catalog, so that's almost duplicative. And then um one other and then a few folks asking what is a technical catalog? Well, great. We're actually gonna talk about that today. Good question. Fair, very fair question by the way. Yeah. Yeah. And then uh let's do this next one. Are you able to Does the quiz Does this immediately work if we just move?
Are folks able to start answering this one or no? If not, then I can um I can fix that. I don't know what shows up for people's phones. Are you guys only getting one question on the quiz? Yeah, I know. Oh, sad days. All right. Well, here now. Is it working? All right. Real time. We're figuring this out in real time. So, this is what business catalog does your organization use today? Okay. Unity, Elation, Calibbra, Data Hub.
What is a business catalog? That's also an excellent question. I only recently found that out myself. Other. I ran out of room. I did run out of room on this. There's a lot of business cataloges out there. For the folks that said other, I'm just curious what what was what was I missing. Yeah. Yeah. I think Yeah, absolutely. I was missing Informatica. Yeah. There's a lot I think that I was probably Atlin I think I think I'm missing from here too.
Yeah. Okay. Awesome. And then what is a business catalog? There's a few fair amount of folks that ask that. Okay. Let's look at this next one. What engines do you use today? This is also also multiple selection by the way because you can have multiple cataloges and many folks will have multiple engines in their environment. I think everybody does. If you refresh it, I think you can see. Okay. A bunch of Spark. Well, that makes sense.
Trino, Snowflake, CFKA. Okay. A bunch of CFKA. Other. What was the other? What was the other that I was missing? Just shout it out. And then last one. What open table formats do you use today? I got one person who said my data is 99% CSV. Two people actually said that. At least they're honest. There's no Excel option. It's true. At least they are honest. Okay. A bunch of parquet, a bunch of CSV, which I think is actually I mean this is like a real answer by the way.
A lot of people actually have a lot of CSV data. Delta Lake and then iceberg. All right. Awesome. Okay. This is helpful. I mean, here's what here's my takeaway actually from some of this is that first I think it'll be helpful to talk about what is a technical catalog, what is a business catalog, both of which I'm going to maybe talk a little bit about right now. And then also what I'm also learning is that many folks have diverse uh environments where they'll have multiple table formats, multiple engines, maybe not anything in an open table format.
It's a CSV file, still a bunch of that data dumped into object storage. And so, uh, whatever solution that you have for your lakehouse for organizing it needs to be able to support all of those things. Okay. So, I'm gonna start off here maybe with like a short history lesson on, you know, the evolution of lakehouse and lakehouse cataloges. So, lakehouse architecture. Many folks will have seen this um because I think maybe even showed a diagram that's a little bit more abstracted version of this uh earlier today during the keynote.
But basically it's the separation of storage and compute defined most importantly by having this sort of flexible compute layer where you can bring CFKA, you can bring Spark, you can bring DB um to all of your data that's stored in open cloud storage. And so that can just be CSVs and that can also be in open table formats like Delta Lake and iceberg. And the missing link between the two is the catalog. And so that's what we're going to talk about today is we're going to talk about this box here in the middle, the lakehouse catalog, which sort of is this connection layer between compute and between your data.
And so early cataloges, they were very basic. Looks like a lot of folks here weren't using Hive Meta, but there's many folks outside of this room who are using Hive Metasaur still. And so this is sort of the I would say like kind of the leading technical catalog standard for many many many years. Uh it's now showing maybe a little bit of its age. And early catalog are very basic. It was basically saying, hey, I have Spark.
I want it to be able to work with my data that's stored in object storage, and I just need something in between that can tell the engine where is for this table name, you know, my uh image capture logs, where is that in storage, what's the path to it, the location, maybe the format, and then maybe the schema. So really, really, really basic information. uh this is what you can call like a technical catalog. So that's what uh hive metas store is.
I think it's uh basically at its core it's about helping the engine know important things about uh the table or asset that's in storage and help resolve that for them. So that's a technical catalog. All right. So now your needs are evolving, right? So we started off really basic uh saying we just we just had Spark, maybe we just had MLflow, but now you want to bring in more engines. Now you uh want to start doing things like access control.
You want to be able to govern these data sets. Not everyone should just be able to mess around with any file um that's in your bucket. Uh you maybe you want to know more things about observability like who is accessing the data, what is accessed, how is it accessed. You want to know things like lineage, where did this data come from, what applications maybe ingested data and brought it into this table and then also downstream who's using this table.
And so back in 2021, we built Unity catalog to address these problems. And so this is really an evolution. I mean for folks who were data bricks users back then, you know that we had uh built our architecture leveraging hive meta store because that was again the premier standard at the time. Um Unity catalog didn't exist. Uh mate, our brilliant co-founder um said, "Hey, we need a better solution than this." And that's where Unity catalog was born.
And so since then, needs have really kept evolving. I talked about basic things like access control and observability and lineage but now we're seeing a lot more right a lot more engines a lot more compute options um really advanced applications that are coming in um this need for AI governance you want to actually govern your tabular data sets as well as your unstructured data sets and your AI models and functions and stuff like that all together um maybe your agents actually need to start accessing tabular data sets I mean that's something that we're um really seeing right agents are actually now creating data and producing data themselves uh in addition to consuming it.
And then of course also you're seeing multiple table formats emerging and with these multiple table formats then this started creating almost this concept of like metadata silos where some cataloges only support or provide better support for certain table formats and then of course um like I mentioned before there's multiple engines and so now you're seeing that the lakehouse architecture is becoming more complex and continuing to get even more complex because now you have these discovery problems and then you end up in this world where oh yeah I do a talk a little bit about business catalog here where it's like you know once I have all this unstructured data this AI assets this tabular data in my organization discovery is going to become really really important when you're thinking about scale across thousands of users applications and now when now you're thinking about even agents being a persona um also how do you want to make it easy for business personas to find insert insights and answers you have a bunch of bunch of bunch a bunch of data now in your data lakehouse how do you uh category ize this, tag metadata about it, build up like a semantic understanding about that data.
Um, and many folks call this type of catalog a business catalog, by the way, or reference catalog, I think, is another name that it goes by. Yeah. And so there's like that piece about the technical catalog and then this piece about the business catalog. And so now you end up in this world where this might look familiar to many folks in this room. This world where you have multiple cataloges. You almost have this problem have too many cataloges.
Now you have a new problem. You have to keep their metadata in sync. And because you had all these specialized solutions, you needed to stitch them together. And now you have to sync between them. And now you need a solution to sync between them. And so I think this cat this problem has almost become recently exacerbated by this sort of uh explosion of cataloges. And so what we've seen is that really there's three core problems and challenges that have emerged.
Now that needs have evolved, the lakehouse architecture has evolved and become more advanced. Now we see that there's three core problems um with catalog interoperability that has emerged which is that data and AI assets are siloed into different platforms. The second is that uh there's fragmented governance because now that you have these multiple different platforms well now your governance and administrative uh burden is sort of fragmented and disjointed across them.
And then the third is that in many cases many of these platforms maybe still aren't using open table formats or open metadata. And so you'll still get these kind of increased costs from redundant compute um or storage uh if you're copying data around. All right. And so these problems are only multiplied by that recent explosion of cataloges I think I mentioned. And I have a another slide which is like the all the logos.
There's so many different t cataloges and so many different specialized solutions and many of these will look familiar to folks in the room and there's many new offerings that have just come on the market that again even even the same vendor will actually have multiple catalog offerings now that you need to stitch between. Um and so and then again I threw in the ears catalog retail catalog joke. I guess that made the final cut on the slides.
I mean I can appreciate it because I used to get serious cataloges but I'm an old guy. Yeah. And you get burritos in Mountain View too. Yeah. Well, yeah. Okay. So, how are we doing? Amazing. Okay. So, uh now we're going to talk about well, how has Unity catalog evolved with those needs? Right. I said we we built it first in 2021 recognizing some of those limitations of hive meta store, but now let's talk about well how has as those needs have evolved, as our customers needs have evolved and the market has evolved, how has Unity catalog evolved with it?
Um, and so I think I have like a funny a funny joke here about we built the first version of Unity Cal 2021 and then I think I put Mate's face on Jobs and then since then of course Unity catalog has become the standard. Um, it's adopted by over 70% of our customers. Um, actually more than over 70% of our customers but then also many workloads also run on UC. And then now we're going to talk about the next generation of Unity catalog which has well a lot of new fancy apps including first class iceberg support first.
Yep. Exactly. And then also evolving to become a reference or business catalog. So what we were talking about earlier and then now I'll pass it over to talk a little bit about this. Awesome. Thank you. Um so as mentioned you know 2021 data bricks announced unity catalog. That was the same year that we founded Tabular. And one of the things that we did when we founded Tabular, which sort of this iceberg based lakehouse, was we um built out a a REST catalog implementation, right?
So like we saw a lot of the same challenges I think the data brick saw with Delta Lake. Not surprisingly, those same challenges exist for companies who are trying to adopt Apache iceberg tables. And so tabular was an implementation of this iceberg rest catalog interface. We donated to the Apache iceberg project. And now we actually see implementations of that open source implementations, vendor implementations. Unity catalog also is an implementation of that specification which is fantastic.
That's where we get the iceberg interoperability from. Uh but that was like really just the beginning and uh it's really exciting now that we've joined forces with data bricks and join forces with unity catalog how we can really extend uh and and this is like the three sort of core tenants when we think about uh why unity catalog what should it be responsible for? What are the guiding principles for how we're building it out?
These are the three things. So unified ali spent some time on this in the keynote this morning which is fantastic. And he said it's not just about tables. He talked about a lot of the cataloges in the marketplace today sort of only focus on that one side where it's like tables and access controls and that's what they think is enough for your catalog and for your governance layer and it's really not right. It's not just tables absolutely you have to conclude your AI models in there.
You have to include your unstructured data, your documents, images etc. uh as well as dashboards, right? He mentioned how dashboards is like if you have governance split off and you have your BI tool and it's got its own governance and access controls models that are different from what's in your actual lakehouse. Well, that's an easy way for people to get access to data they really shouldn't have and Xfiltrates and it gets out the door.
So, you have to be able to include that entire ecosystem of your data estate and have governance, consistent governance on top of that. We also even saw how the apps data bricks apps that same governance model from Unity catalog applies all the way to apps. We're going to see how it applies to our lakebased tables, right? So, this has become even a more and more powerful construct that everything has to be unified under this governance umbrella.
Uh the other thing that we have done is as mentioned, look, you don't have just one catalog. That's the reality in most of the companies that we talk to and customers we talk to. And so, we have the concept of federation where you can actually bring your other cataloges, federate them through Unity catalog and then still apply that singular governance model. So even if it's not tables that are directly managed or assets directly managed within UC, we can still get that unified governance.
That's really important for that single pane of glass uh view of your estate and and consistent governance uh and that endto-end lineage. So you can see all the way from where data is being sourced from coming in from those Kafka streams all the way through to the dashboard and the reports that and tie that whole thing together. So that's pier number one. Uh sort of principle number two is that it's open. said open many many times this morning.
It's like a core tenant of data bricks the company and that obviously uh extends to our our catalog story. So yes, open- source table formats, Delta Lake, Apache, Iceberg being the primary ones, but not just that, not just the storage layer, that's really really important, the storage layer, but this the APIs, those catalog APIs with which we interface to that data also have to be open, right? For you to be able to come in and plug whatever comput system you want and for you to be able to adopt Unity catalog and not feel like you're just locking yourself into the next, you know, vendor problem, right?
It's like okay my sure my data is in open format but now I have this catalog and if I'm using Unity catalog aren't I just locking myself into data bricks is how is that different than where I was locked into my previous data warehouse vendor and it is because even those catalog APIs Unity catalog APIs that you're interacting with the data are all open source as well right and they're not just implemented on Unity they're implemented in many other cataloges and then all the engines use those APIs to talk to your data so they're not you know using uh proprietary APIs from data bricks or there's no special code from data bricks they have to get to interact with cataloges.
It's all open source. Uh and then a unique offering here which is very unique to our Unity catalog experience is that because they're open formats and because they're open APIs, we actually allow any compute engine to read and write data to those tables. So you can simultaneously stream data in from Kafka to a table in Unity Catalog. You can then make updates to that table using DBR and then you can read that data out using Trino or DuctTV or whatever you want. all those tools operating on the same single source of truth data with one unified governance layer that's that's you know that's actually enforcing governance for Kafka for data bricks and for say ductb or train or what have you snowflake okay that's really unique in the marketplace and a powerful construct uh and then the last bit is that it's intelligence right so you know data plus intelligence that's kind of the whole theme of the week has been the theme over the past year for us and what does that mean for that catalog layer uh the one piece that shows up which isn't be apparent is that we use the intelligence that we get from how you all create and use your data, right?
Because the catalog is involved in all these operations. We we use that intelligence to then optimize how we uh physically lay out your data on in the object store, right? So this is all within the open source table format umbrella. This is still data you control, right? It's not in our storage. It's in your storage. But we can even though it's in your storage, we can help you optimize it. And so you can get great performance.
We can we can beat proprietary storage formats and engines using open source tools because we have this intelligence, right? We look at query patterns. What filters are you using on this table? What are the commonly used filters? What are the commonly joined tables? Can we make sure that we organize data so when you run those queries, the next time you run your queries, they're faster. You didn't you did nothing. You just kept running your queries and tomorrow your queries are faster than they were yesterday just because we were like laid out the desk the data on disk more effectively and that's that intelligence built in and then also adding in the semantic modeling on top of this like core data sets right so that your business users can come and use the natural language right and they can use the genai and LLM to uh discover and answer questions of the business you need that semantic modeling that really helps those LLMs you not be general general but actually answer questions about your data right in your in your domain.
Uh and then obviously we can you know generate some of that metadata as well, right? So I would guess if you go out to your catalog system, Unity catalog for many of you and you inventoried how many of your your data assets in your Unity catalog have descriptions, table descriptions, have column descriptions, uh have tags, you know, probably a small percentage, right? Because historically this was like a manual process.
Uh it was like documentation. uh it never got done all the way and lots of tables get created that don't ever get documented with metadata. So we can actually use uh genai to help automatically fill it and we look at the name of the column we look how that table is being used and then we can generate descriptions uh and tags and such. Okay so those are the core pillars and we're going to dig into each one of them. Um cool.
So the first one is that unified governance that I talked about and just to reiterate right so this is uh not just not just your tables but your models and files as well and then it's not just access controls so the other thing I want to bring to light here is when we have this unified story we also get collaboration this is the discovery aspect how do you find the tables that you're looking for not just within your company collaboration but also collaboration with your partners and your customers through through things like delta sharing and delta sharing works even for iceberg tables now so you can delta share everything in unity catalog and it's actually it's not just tables.
You can delta share things like models as well. So you can share all all these assets that are within the Unity catalog. You can share them internally across business units but also with your partners externally. You get that end to end lineage that we talked about earlier. Uh quality monitoring, right? Data quality, right? Garbage in garbage out. We talked about how important good data foundations are for all of our AI efforts and applications.
And so there's like built-in quality moni monitoring uh and anomaly detection, right? So we can proactively tell you, hey, this data looks off for any number of reasons, right? And then you can go investigate that. Uh as well as the semantic layers and one that maybe like uh often gets forgotten but is like super important and for customers that I talk to cost controls. Like we we need good cost controls. We need to understand like where is the money being spent?
Which data sets are the ones that are actually expensive? Which dashboards are the one that are expensive? And is that are they actually being used? Right? to know now with Unity catalog where it's end to end lineage and having everything from source all the way to dashboard we can actually tell you that dashboard what it's costing you and not just what it's costing you to serve the dashboard everything from source data all the transformations all the way through serving up that dashboard and we can also tell you how often is it being used well that's really powerful information you can actually finally get rid of the 5,000 out of your 10,000 dashboards that no one ever looks at but that you're still paying for right it's just cleaning up the waste in the in the system okay so that's our unified governance And then this is the secondary piece I talked about earlier around catalog federation, right?
It's not just the stuff that's within Unity catalog, you have data sets in these other systems. Uh and we want to make sure that when you're using these these external compute environments that your governance model is still applied. Uh we do that through this this concept called vended credentials. And so if I'm coming in and I'm I'm going to be connecting to Unity Catalog, say from EMR and I have a I have access control set up on my assets in Unity Catalog.
I want to make sure that EMR that compute engine is respecting the governance I have set up in Unity Catalog. So we do this this notion of vended credentials. EMR connects to the catalog. It provides some authentication identity right with access tokens or OOTH and then Unity catalog checks the privileges and then dynamically it says like hey you have read access to that table. let me dynamically create some tokens for you to go to your object store to S3 that are scoped down to just the files that are in that table and you have read access to those objects in S3 and that's it and it's a it's a temporary token that will expire when your job is over right so the EMR that compute system doesn't have to have any direct access to underlying storage it gets that dynamically through the catalog and that's true across the board all of these different engines even though they're very different uh compute engines different languages for different purposes streaming batch they still all follow this governance model through this venic credentials mechanism and it works across all the clouds, all of your object stores.
And then the last bit is federation. So you may have data sets in these other cataloges whether that's from legacy reasons, maybe you're doing you're through through acquisitions in your company, you're combining business units. I have my data is over here in Snowflake or my data is over here in Glue, but I still want to be able to bring that into Unity catalog, be able to see it all in a single pane of glass and have that governance that I just talked about, right?
So we can actually provide governance even on assets that are coming through other cataloges. All right. So that's sort of the unified governance story. The next pillar that we talked about and sort of our our three practices around Unity catalog is is open. Right. So uh you know we alluded this earlier in the talk that you know uh over the last year we've we've gone in and really supported Apache iceberg as a first-class storage citizen within data bricks.
But beyond that, we're actually doing a lot of work in the open source communities. We're not one of these companies, right, that just takes open source, packages up, and tries to resell it as, you know, some sort of service, right? Like these technologies that alluded to in the keynote, Spark, uh, Delta Lake, MLFlow, Apache, Iceberg, the people who actually wrote that software are the main contributors in open source to these things.
We spend a lot of R&D dollars at data bricks driving these projects forward. So, we're not just taking what's in these communities in these open source. we're providing, we're giving back a lot. Making sure that what happens in those open source projects actually meets the needs of our customer base, right? And one of the things that our customer said was like, we don't like to have this decision between the two formats.
Can we get rid of this? And so we've been working in the last year and actually what we call co-evolution, right? So we are we are adding things to both projects that actually bring them closer together. That makes our life easier. we can support both formats more simply but also is better for our customers because they don't have to decide like oh if I have data that's in iceberg and snowflake is it going to be able to do all the things that delta tables can do inside data bricks and the answer is uh increasingly yes um so these were recently announced delta lake 4.
0 just dropped this week last week brand new delta lake 4. 0 So uh we're introducing catalog based commits into Delta Lake. That's how we bring Delta Lake tables into this whole catalog based world and making sure we can do all that intelligent work with Unity catalog. That's a concept that very much comes from the iceberg community. Uh on the iceberg side, we introduced deletion vectors. This is a a concept that comes from Delta Lake.
How we encode our our rowle changes to tables uh in a very efficient way, right? So we can making small changes to tables. We don't have to rewrite big partitions or big f paret files. And then both table formats uh we introduced a variant data type that is compatible across both of them. Variants are for storing your semi-structured data. So you have your JSON strings or semiructured data or column or column schemas that evolve or need to evolve rapidly.
Variant type is a way we can actually uh effectively store that data in a binary format. So you get better storage, better performance. This is how we're you know beating proprietary systems by continuing to evolve these open formats. And then this is sort of the final picture that we're trying to get to where regardless if you're coming in through a delta client or an iceberg client, you come in through that Unity catalog, those open source APIs that we talked about earlier.
Underneath you have these manage tables from Unity Catalog and you don't have to worry about what the underlying storage format is. You don't you don't have to worry about optimizations or management. Uh Unity catalog and data is going to take care of all that for you and you can just you can come in with any engine that you want, right? like that. Yes, obviously data compute works great in this architecture, but so does like any other comput system or combination of comput systems that your teams need to use to get your jobs done.
All right. And we talked about this a bit already. I think I I already have alluded to a lot of this, so we'll skip over this side. But yeah, open APIs, vended credentials. This is what allows us this open ecosystem, interconnectivity. There's no lock in here whatsoever. uh you're free to like take these tables and move them to other cataloges. You're free to swap engines, right? This is really futurep proofing your data architecture, right?
You can make these choices now and feel very confident that no matter what happens next year, the year after that, when all your agents start coding and now they need to bring their own tools in this ecosystem that we're going to be able to plug all this stuff in. And then the last bit that we alluded to earlier was the Delta sharing protocol. So this is a way for you to get uh your access to the assets in the Unity catalog in a totally secure way.
Uh it uses a slightly different mechanism. Vended credentials we talked about earlier sort of dynamic access to storage layers for comput engines. Delta sharing takes a even more restrictive approach because usually we're sharing data across companies. Uh and so it's a it's a highly secure way to share uh information from Unity catalog with your partners and with your customers. And again, they can be using any number of tool sets on their side, right?
You can share to your partners and they may not be using data bricks or they may not want to use data bricks as the consumption layer. That's totally okay. There's no this is an open protocol. So many tools know how to understand and consume from delta sharing. And so they don't have to be using data bricks to take advantage of that sharing capability. So that's also a unique offering terms of like a data marketplace that is also totally open, right?
It's not bound to a particular uh set of computer infrastructure. Yeah. And one thing on that too is that in preview uh now you can not only uh delta share iceberg tables but you can also delta share two iceberg clients. Yes, that's right. We added an iceberg iceberg rest catalog protocol as part of delta sharing. So something like uh your snowflake compute engine which speaks iceberg rest catalog can actually consume delta shares.
Right? So again just really open ecosystem. You don't have to copy data to snowflake if you want to share it to a snowflake consumer. just put it as a delta share. Snake can connect directly to it. Now we're consuming that data directly. So no more data copies. All right. And the last pillar that we sort of talked about earlier was uh was intelligence. And so we'll spend a little more time digging on here. The way that we describe this is through this Unity catalog managed tables.
So you can have a variety of assets within Unity. The ones that get all this intelligence, that get all this optimization where we're rearranging the layout of storage on disk. uh that's available through this manage tables construct. These can be delta lake or iceberg tables. Uh and here's where you know we see we can actually reduce your storage by up to 50%. So not only are your queries getting faster like I mentioned, right?
We're choosing uh you know join keys and filtering columns and we're laying out data appropriately but we're also doing things like choosing the right compression codecs uh within your within your parquet files like should you be zstandard should be snappy or gzip right and depends on the data set is the answer and we use intelligence to figure out like what should the right compression codeex be how should the how should the clustering work and so yeah this reduces your your storage bill and because you know we don't make any money on storage right the storage bill completely goes to the cloud vendor everybody benefits at least in this room, right?
Maybe not your cloud provider, but everyone else, but like we don't want to spend money on storage that we don't have to. So, we can save money on storage. And because most of our queries are IObound, right? The performance of most of our queries is is dependent on how much data do I have to actually go and load from my object store. When we have less uh when we actually are able to compress that data better, the IO is better and your query speed up.
So, we load less data, we load it more efficiently, and that's why you get, you know, big big improvements on query performance in a lot of cases up to 20x, which is, you know, amazing. which is just less dollars, right? It's all a consumption based model. So that's just less expensive for you. Uh the other things that uh I didn't allude to earlier but are really cool and will have I think more talks about this later in the week.
So if you want to hear learn more about these topics, there are more detailed talks, but something like uh multi-table and multi-statement transactions. So these are things that were pretty standard in data warehousing have been for a long time, but have never had yet been done on open formats like in an against an open catalog APIs. And so we're pretty excited about bringing sort of multi-statement transactions to this world.
Lots of use cases uh require that you have consistency, right? Uh across across multiple tables at the same time. I want to update this table and all the foreign keys and my relational model, you know, atomically to make sure that my joins are always successful, things of that nature. Um and then we're also able to do automatic table feature upgrades because we can see all of the usage of your tables. We know which versions of DBR are you using. we know which external tools you're using, right? then we know which features we can automatically enable that actually have uh support for the tool set that you're using with your data set because the last thing you want to do is upgrade a table to guide a new feature and then something breaks and so it makes everybody very nervous about like oh do I want to add that I don't know I don't know what's going to break but we do right we understand all the dependencies we understand how the data is being used so we can automatically upgrade your table so you can get the latest features best performance uh confidently knowing that none of your workloads are going to break in the process uh and that's something we've heard a lot of pain points from customers and we're excited about And then the other big one that we're working on the second half of this year is disaster recovery.
Right? Again, uh because we see all the activity against your data in in the catalog, uh we can take that information and then replicate it to other uh regions within your cloud, potentially even across clouds, right? So you get really good DR properties and we've, you know, really hampered on the fact that these things are all fully open and interoperable. There's no lock in involved. So this is sort of like the final big picture.
Uh this is like really the drilledin view for maybe the uh the high level picture that showed in his keynote. All the different engines, you know, this is this is only a small subset of the things that actually work against this ecosystem. Any of the cataloges that are out there in the ecosystem, you can get those federated in as well. Make sure you can see all your data and make sure you have unified governance. And that really like can't be understated.
I, you know, we hear from customers a lot, you know, I'm a data scientist. I'm I primarily work inside data bricks. I I want to build a model and obviously the when I start any new project or I want to answer any new question for the business it starts with like well where is the data where is the data set that I need to answer that question and it's often not in the you know existing data bricks tool oh that data is actually in snowflake but with federation you just plug all these things in and now your you know your data scientists don't have to go around f like chasing this stuff right it's all discoverable in their in their environment and it's governed right they have access to the things they should have access to so it's a really powerful like in terms of increasing the productivity of our data experts uh and of our agents, right?
Our agents now can also have access and see all that stuff which means that you know our natural language uh questions of our data also get more powerful not just access controls all those other things we talked about in Unity catalog how important that stuff is. Uh so we got the unified governance layer and that built-in intelligence right like you really shouldn't have your data engineering teams spending a lot of time like hand tuning hand optimizing what partition strategy should I use how big should my file by parquet files be should they be should they be 16 megabytes or should be 128 megabytes what compression codecs all these things matter a lot they matter a lot to your bottom line the cost that you have for your data architecture those things matter a ton but you don't want to spend a lot of human engineering resources maximizing, optimizing those things, the things that machines are actually much better at than humans and your, you know, your data experts have um lots of other more valuable things they can do for your business and chase these optimizations.
And so we built that intelligence into Unity Catalog. All right. All right. We're at the end. Yeah. Yeah. I think uh thank you. I think uh I guess just to just to wrap up as well even going to that previous slide you know the session was about what's that missing link between the lakehouse and data intelligence and I think hopefully we've illustrated how the catalog plays this essential role as this sort of like layer in between all your compute your AI agents your tools and of course like your governed data and AI assets um underneath that and then provide a lot of these extra like intelligent capabilities on top of that by having all the observability and uh lineage and governance capabilities all coming through this interface.
And uh that wraps it up. Don't forget to attend these other sessions that are coming up. Um actually there's one if you're interested in multi statement transactions there that's coming up in just a little bit. And then also if you want to learn more about managed tables and the optimizations that we do there a deep dive into that um as well as Unity catalog open APIs. I'm actually gonna be doing that session later today at 410.
Um demoing actually I'm gonna be demoing live using Spark, Duct DB, Starburst, Trino and uh Daft a multimodal query engine all with data sets and volumes in Unity. Awesome. And then um also check out tomorrow the future of open table formats uh as well. So talking a little bit more about that co-evolution piece that was alluding to. Yeah. So final call to action. If you're not already using Unity catalog, I think most of you already are, but uh start or double down, right?
Finish off those unique catalog migrations. There's a lot of value you're unlocking for your business by getting there. If you're not using manage tables and taking advantage of the intelligence, the optimizations, start or like accelerate those migrations to manage tables that are going to save you a lot of money and make your life easier for your data teams. And thank you for coming. Appreciate it. Thank you so much.

Learn more about the Databricks Data and AI platform.