Skip to main content

Open for Business: Building the Open Lakehouse for the Agentic Age with Unity Catalog

Summary

  • Unity Catalog OSS enables a fully interoperable open lakehouse stack where Spark, DataFusion, DuckDB, and Flight SQL can query managed tables through coordinated commits and credential vending introduced in v0.5.
  • The new UC Delta APIs enforce access policy at the catalog layer through down-scoped token vending and server-side planning, closing the governance gap that traditional per-engine access controls leave open.
  • The OpenSharing protocol extends open governance beyond tables to agent skills and semantic pages, addressing the emerging gap where agents read governed data and then act in ways the catalog never authorized.

Open for Business: Building the Open Lakehouse for the Agentic Age with Unity Catalog

Watch: Open for Business: Building the Open Lakehouse for the Agentic Age with Unity Catalog
You can build a serious data-and-AI platform almost entirely from open source today. We'll grow one alongside a business — open storage and Delta, Unity Catalog OSS, multi-engine access on the same managed tables via the new UC Delta APIs, MLflow, lineage. The Open Lakehouse is real — and it already governs data well: credential vending, server-side planning, trusted compute. What those patterns share is an assumption — every request comes from a human you can hold accountable — and agents open a gap in it, reading what they're allowed to then acting in ways no one authorized, while the catalog that governed the read never sees the act. We'll show today's open governance patterns, where agents and cross-org sharing strain them, and why closing the gap — including with the new OpenSharing protocol — means extending governance to the platform, in the open. Context is the new data.
Talk By: Michelle Leon, Senior Staff Product Manager, Databricks ; Robert Pack, Staff Developer Advocate, Databricks ;

Chapters

FAQs

What makes Unity Catalog OSS different from other open-source catalog projects?

Unity Catalog OSS is designed around three pillars—universal access, multimodal support, and unified governance—making it an open, interoperable catalog rather than one tied to a specific engine or storage format. With over 2 million downloads and two years of development, v0.5 adds coordinated commits, credential vending, and UC Delta APIs for multi-engine access.

How does credential vending in UC Delta APIs work?

UC Delta APIs issue down-scoped tokens at query time, enforcing the catalog's access policy before any engine—Spark, DuckDB, DataFusion, or Flight SQL—reads data. This server-side planning approach means policy is enforced consistently regardless of which compute engine submits the query.

What are metric views in Unity Catalog and why do they matter?

Metric views are governed business semantics defined in YAML within Unity Catalog, providing a single authoritative definition of business metrics that any engine or agent can reference. They ensure that different tools and users are always working from the same definitions rather than divergent ad hoc calculations.

What is the OpenSharing protocol and what problem does it solve?

OpenSharing is a new open-source protocol that extends governance beyond tables to agent skills, tables, and semantic pages shared across organizations. It addresses the governance gap that arises when agents read data they are authorized to access and then take actions—such as calling external APIs or sharing outputs—that the catalog governing the read never sees.

Full transcript

[00:09] Okay, hi everyone. Hi. And thank you. Thank you for saying hello back. >>
[00:22] open lakehouse stuff. So, that's uh Delta Lake, Iceberg, Unity Catalog, uh and then now also we're really excited to talk a little bit more about open sharing, which is a new open source project. Uh and I'm joined by Robert.
[00:34] >> Yes. Hi, I'm Robert also at Databricks. Uh I'm with DevRel. Essentially, everything Michelle just said. So, I try to help and work with the people who drive and build these open source projects out there.
[00:46] And um yeah. >> Yeah. And so, I'm I'm really excited cuz today what we're going to do is we're going to actually walk through how using all of the uh you know, various technologies in the open source ecosystem
[00:58] um that we're really excited to be a part of, how you can leverage all these to actually build like a fully interoperable lakehouse stack. And so, uh we're going to start off with first talking a little bit more about
[01:11] what makes a great lakehouse catalog. We're going to dive into that, talk a little bit about Unity Catalog, and then we're actually getting into some really cool stuff um talking about a little mini business that a side project actually that me and Robert are going to start up, and then walking through uh
[01:24] with some cool demos about how you can leverage um all these various different open source technologies in the ecosystem. So, without further ado, let's jump into it. Uh there's sort of three challenges that we kind of observe
[01:36] today in the broader data and AI ecosystem. You know, you heard Ali and other folks talk about it this morning, and probably you're going to hear about it all the rest of this summit, uh which is around how, you know, historically what we've seen is that data and AI
[01:48] assets, they're very siloed into different platforms. Um this can cause uh downstream problems with fragmented governance, increased administrative burden, um and then also increased costs because
[02:00] you have redundant compute, redundant storage, you know, historically a lot of these different silos between systems or even between things like, you know, OLAP and OLTP worlds and stuff like that, like the analytical and
[02:12] operational estates have been siloed, and now they're all coming together. And so, our goal is that we want to give users the flexibility to use these data and AI assets with this open and interoperable ecosystem without
[02:25] sacrificing security. So, that's our goal and statement today. Unity Catalog helps you accomplish this vision. It has sort of like three kind of core pillars, which is that provides
[02:37] universal access to all of the different tables and data assets in your estate. It's multimodal, supports unstructured data, your AI assets like models, functions,
[02:49] volumes. There's some other really cool stuff that's in the works as well that some of my other colleagues will be talking about throughout the summit. And then also provides unified governance. And so, here's a little bit more about that. So, what we're going to focus on
[03:01] today is this kind of middle box, which is how do you get unified governance with these open APIs and open sharing to govern all of your data and AI assets in your estate and
[03:13] actually open up access securely to this really broad ecosystem of various engines. Okay. So, now let's get into the fun stuff. What can you do with UC and open
[03:25] sharing? And now, I'll pass it over to Robert to tee us up a little bit. >> Excellent. So, probably an obvious question, who of you has used an app before to order food?
[03:37] Show of hands. Yeah, expected that. So, who of you knows what a ghost or sometimes called a dark kitchen is? All right. So, sometimes when you order food from these places from your favorite restaurant, it may
[03:50] not actually be prepared within that restaurant, right? So, a business may rent out a kitchen or may rent kitchen space without any seating, without anything, just to prepare orders so that they can pick up or can be picked up by the delivery guys and and uh
[04:03] brought to your home, right? So, and this is ghost kitchen, and Michelle and I >>
[04:16] couple of the problems that we encounter along the way, right? >>
[04:29] the team that interact with the data lake. Uh we have the vendors that rent our kitchen space that we're interacting with. We have the customers who actually do the ordering, and we have the delivery operators who who do all of
[04:41] this. And we don't want to be just landlords, right? We want to be kind of the uh middleman, uh facilitate all of this kind of business, because usually that's where um the money sits. So, what we need to
[04:53] do is we need to have like a constant stream and flow of data through this platform. And this is a um it's surprisingly data-heavy business. And um yeah, I'll let Michelle maybe
[05:06] raise some of the first questions that uh arise within the business. >> Yeah, well, as as you know, the the PM and the the really the mini CEO of our ghost kitchen business, I care about a few different things. The first of them
[05:19] is are we making money? So, that was really important cuz we need to be able to pay ourselves and, you know, we're ultimately trying to run a profitable business here. So, are we making money? Why? Where's that money coming from? I want to learn more about the user
[05:31] experience for our customers, cuz I want to all iterate and build a better product. Um I also want to know which vendors are driving the most usage. So, you know, we are like, you know, almost this uh middleman layer, so we're working with a bunch of different vendors who are
[05:43] helping us do this food delivery and app delivery and stuff like that. I want to know what are my most effective channels and then I also want to know what are our most successful products cuz I want to be able to continue, well, making money. Ultimately, that's the end goal.
[05:55] I'm very uh capitalist. >>
[06:07] I would say um this already is a lakehouse, right? You have a storage account, you choose an open table format, um and you already have a lakehouse. And I think one thing um there is a dif- distinction between, let's say, the the the traditional
[06:19] lakehouse and the modern lakehouse. Um originally, the choice of query engine was the key decision that you needed to make when you build your lakehouse because you needed to build this out and ultimately, that choice
[06:32] scaled your entire >>
[06:45] hardware, modern like out-of-core compute and things like that really made it really easy just to choose any engine, point it at the object store, and you kind of have your mini lakehouse
[06:57] already going. But, we also learned um I have a background with a lot of thermodynamics, so entropy is not good. How do we find How do we find entropy?
[07:09] We introduce a catalog so that we have sort of a baseline of discovery um within this entire system. Right? So, uh we want to know who creates which table um and at least kind of have some sort
[07:22] of order and traceability in there, right? And this is kind of what we said at the beginning and um let's just do a quick demo what this looks like. So, here um these days you vibe
[07:35] >>
[07:47] Catalog UI. >> Yes. >> Yeah, or a version of it that you've branched. >> Exactly. And the important part here that you should take note of is that this is both Delta and managed.
[08:01] This is an important distinction when I say we can query it because um querying something like say raw on storage uh versus the managed table is um
[08:13] pretty different, right? And based on this, we also just kind of build a little dashboard uh that shows, I don't know, >> This is awesome. These are the answers to my questions, right? >> Yes, yeah, hopefully. So, you see, I
[08:26] don't know, uh all the volume, you see uh the heat map for when something is happening. I did maybe generate some data while I'm coding. I'm originally from Germany and the I decided to to generate the data in
[08:40] Germany, but
[08:52] answered now. >> This is awesome. And so, what we're seeing is being able to leverage That was Apache Spark, right? >> That was about Well, some of it is Apache Spark. I mean, that was a multi-engine demo. Um so, in the catalog, I I I forgot to show
[09:06] this. In this catalog, we can submit queries. This flows through data fusion. In the notebook, we have some DuckDB in there. Um just to make it more complex, we can also go via Flight SQL uh onto this data
[09:19] and your trust Spark uh knows how to do this. >> So, that was all of those engines, Data Fusion, Spark, a little bit of DuckDB, Flight SQL, all of them interacting with Unity Catalog managed tables.
[09:31] >> Exactly. Exactly. >> Amazing. Okay, and then I also got all of business insights. Well, let's see. My my uh needs don't stop there or the needs of the business don't stop there cuz now we need to actually go beyond some of these
[09:43] foundations. Um and now we actually are growing our team. So, we've run a fairly successful business in Germany uh and now the Casper's team is growing. We want to build out actually like our dedicated
[09:55] finance operations teams and then we actually want to go into some new verticals. Uh and we're really excited about, you know, expanding some of our partnerships with some of our vendors. And so now what I need to do is I need to be able to think about how do I scale
[10:08] securely uh my governance to all of these new roles, functions, and also to these partners that I'm working with. And so Robert, how can we do that? >> Um yeah. So,
[10:21] some of the growing pains Michelle mentioned um So, we have our architecture. We do need to get a little more complex. Um I'll do like a little it's not pip install docker pull um update to V5.
[10:34] And um we're good to go. >> It's that simple as that. Okay, so now I'm able to onboard all the finance operations teams and be able to extend uh to all of these vendors and partners
[10:46] with a simple version bump. >> At least a simple version bump. Maybe we elaborate a little bit on why uh that is. So, in V5 of Uni Catalog OSS, the UC Delta APIs are being introduced. Uh UC Delta APIs, so those of you who
[10:59] know uh Iceberg Rest uh catalog probably know kind of what direction this is going in. This is an API in a similar spirit but tailored to the needs of Delta. So, it speaks native Delta. We
[11:13] looked at adopting RC but ultimately um couldn't kind of cover the entire feature set and needed kind of this this explicit Deltaness in there. So, this is first of all, it can coordinate
[11:26] commits. There is uh credential vending built in there and this This kind of the piece that enables a lot of governance, so credential vending will will will cover a lot later. It um has kind of versioning, so we had
[11:39] a lot of learnings in the past of what versions and features and kind of stuff like this. But there is one important piece. So it's it's fairly easy to corrupt a table um and that is
[11:52] Delta has a bunch of features and if you enable uh like a higher level feature, you need to have multiple sub features like column mapping, deletion vectors enabled and all of this should be
[12:04] consistent. It's defined in the protocol, so if you implement this yourself, everything should be correct, but this is hard work. So we let the catalog kind of deal with all of this stuff as well so that you can have like an intent-based API rather than dealing
[12:17] with all of these low-level features that you need to activate um individually. >> Yeah, and one thing that's important there too as well is that in an extension of kind of the multiple engines that you were showing earlier, if you have all of these different
[12:29] engines interacting with your catalog with the Lakehouse with these tables, you want to make sure that you have this kind of governed consistent experience and the catalog actually helps to manage that through these new UC Delta APIs. >> Yes, absolutely. So these Delta APIs
[12:43] live there. Um I apologize for the naming, so those of you who are into zero trust architectures we we have simplified it a bit so the trusted system equals catalog is Unity Catalog.
[12:55] PEP, PDP, PIP, that's policy decision enforcement and information point. We're not going to go deep into this, but we'll look at that green box a little bit more. So what does it look like in credential vending? Credential vending again, the
[13:08] client requests a table from the catalog. First you acquire a token from your identity provider usually or have some other form of credential to authorize against the catalog. You then kind of the the the catalog validates
[13:21] this credential. If it's feels like yes, you are the person you're you're uh supposed to be and you have permissions to read the table, I'll return a credential to you to the client, and the client then also will can access that uh
[13:36] the the storage with this down-scoped credential that's kind of targeted exactly to um the reads you need to make for this, and that's short-lived. I can maybe share from my personal experience. Back in the day when we
[13:50] needed to decide on my query engine, we just handed out storage keys uh to everybody. Identity made things a little bit better, but trying to manage a complex data estate just by access control lists on storage
[14:03] is a nightmare, and will well, your admins will will will will hate it. And credential vending is a wonderful mechanism, deceptively simple,
[14:15] but um also very powerful. So, with this, I think we can uh serve some of your needs. Can Can we release some of your products? Um have our vendors uh have their dashboards. >> Amazing.
[14:28] All right, that's great. Okay. So, now the business is continuing to grow. We're expanding across more cities. We're expanding beyond Germany now. Um and we actually also are starting to
[14:40] onboard some of the non-technical uh users in the company. So, uh with like internal BI tools. And then, we're actually also exploring now in genetic tools for being able to explore the company's knowledge um and really unlock
[14:54] more value for all of the people at our Casper's business. Uh there's some growing pains that come with this, though. Uh so, what we've started to see is that now with empowering all these business users to be able to answer some of these questions themselves,
[15:06] including, you know, myself from earlier in the presentation, uh now we have a bigger problem, which is this sprawl of the diverging definitions of KPIs. Um the business semantics are not really
[15:18] codified anymore. Now, we're seeing that, especially because it's so easy with agents to like generate new code, new standards. We're seeing now that logic is being duplicated in multiple places and then that actually can come
[15:31] with bugs and errors and those get out of sync. And then now this is causing problems because my agentic assistant is now making erroneous assumptions about business terms and it now it's actually starting to
[15:43] become a little bit of a mess. And so how can we get some of this under control, Robert? >> Well, the good news is we we we kind of already did. So we did the update to V5. >> Oh, great. >> Yes, V5 also supports metric views. Um
[15:57] metric views are just a way kind of a structured way to define business semantics. Right? So you have kind of a source table. You can define
[16:09] kind of the source table can be a bit more complex. I think you can define filters, stuff like this. On top of this source table, we kept it simple here. Um and then you can define um metrics or
[16:21] measures. Kind of these are the things that the KPIs you want to compute. And um you can define dimensions on top of this. So kind of the things you can slice and dice by. Um if you want to do
[16:33] it like this, right? So And why is that good? So first of all, we have now this entity and to be honest, it's just some some YAML. But it's governed YAML. That's important, right? It's a structured format and it's governed and
[16:45] there is a single source of truth that everybody can rely on and we can evolve this over time. And any engine that can understand this will also evolve as our understanding of the business evolves, right? So I think this is quite powerful
[16:58] and very important. And we also kind of decouple these definitions from the query formulation. And the good thing is one thing LLMs are good at is reading YAML.
[17:10] They're they're the text-based. Right? So they can understand this and they have much less room for error when it comes to this. So, feeding this context, feeding this information into a session actually helps the LLM reason much, much
[17:24] more about this. So, we built kind of a little demo around this. And this demo, to be honest, it's it's illustrative in in in many ways. Let me explain a little bit what's what's presiding here. This is just kind of showcasing, okay, we have some different
[17:37] KPIs. Here, we have some measures and dimensions. Ooh, now it's great. The important part and um this is it's not so hard to leverage this
[17:51] um to build, to integrate this kind of whatever engine you want, right? You just need to understand the YAML. My hope is of course, and as DevRel I have the opportunity to work with the community to integrate this into
[18:03] multiple engines. And post I is this is one of the things I'll try to do, but it's not terribly hard and I did this kind of behind this notebook to to understand this YAML and we can build interactive stuff on this, right? So, I
[18:16] mean we slice and dice by different dimensions. Um Hopefully it'll update now. It should. It doesn't. Well, you have to trust me that it did
[18:28] once. >>
[18:42] But this is an interesting question. What's on time or what's not on time, right? So, are we considering anything that's 10% below our estimate or above our estimate on time? Or is it 5%? We can change this
[18:55] definition. The Deutsche Bahn, if you ever in Germany, does this for instance, change the definition what's on time means. Um we can change this definition and the entire company kind of just follows suit.
[19:07] Right? So, um this is metric views. Uh we have anything else we need to do? >> That's exciting. Really cool. Okay, so I I love that. Shout out to how I guess there's a what
[19:20] some board in Germany that changes the definition of on time. >> Yes. Yeah, well, the the the our train essentially they had very bad metrics on being punctual. So they just kind of
[19:33] changed and said, "Yeah, rather than 3 minutes too late, now 5 minutes too late is still on time." >> You got cuz you got some chuckles in the audience. So we got that joke. That's pretty good. Okay, well, that's really exciting. So
[19:45] now Castor's business, I mean, we're growing like wildfire. So there's more cities, there's more data, more pipelines. I need to be able to scale up. But now I have some serious growing pains.
[19:57] Some of the PII data from customers is starting to be leaked into these vendor dashboards. I got my first GDPR request. I mean, I should have probably got my first GDPR request earlier by starting my business in Germany. But
[20:09] yeah, you know, I got my first GDPR request came in. And then also updates to existing tables became a lot riskier. And so now I have to deal with all the compliance issues that come with running this well, really highly scaled international business.
[20:22] And so, Robert, how can we address some of these problems? >> Well, so this is a problem and I like to distinguish kind of between what's catalog responsibility and what's a platform responsibility. The answer here
[20:36] is lineage. So lineage can help us solve these problems, right? We know where data propagates. If you use Databricks before and many of you probably have, kind of lineage is
[20:49] the basis for ABAC and things like that as well within the Databricks platform. So if you're building your own lakehouse and you want to build kind of some of these advanced features on top of this, you need lineage. And engines can emit
[21:01] lineage. There's existing integrations out there. So open lineage. We actually build like a little custom Open Lineage so server which maybe we shouldn't have because it's quite slow, but well well
[21:14] we'll see that in a second. Um and we we then integrate Open Lineage into Spark. We also integrated it into Data Fusion. Um and we can use that to
[21:26] make some of these decisions. We can use that okay, how does PII propagate through our platform? So if there is a GDPR request, what do we need to delete? Which tables consumed all of this data rather than
[21:38] guessing who built what with which queries and manually inspecting inspecting your entire data estate if there is something in there or do it like I don't know there isn't like a select all tables query, but
[21:52] you you you can imagine kind of how how how annoying that that might be if you try to do this. So how does this work? Um for this platform we built here, we used Data Fusion uh
[22:04] easily my my favorite open source engine or composable data system if you want. And within that you can kind of hook into the logical query planning, inspect the plan, have the plan emit lineage,
[22:17] have the plan pull information, right? You remember the PIP earlier? Policy Information Point. So this is information we can we can pull information out of Unity to make policy decisions within that plan, and
[22:30] we can emit data to something like anything Open Telemetry. In this case, we've been using Emma Flow um for Open Telemetry because it just happens to expose an Open Telemetry endpoint which is convenient. I'm not
[22:43] saying that if you do massive scale log collection that necessarily Emma Flow is the right tool. It's more agentic, but still. So um we did this and what does it look like? Um let's have a quick look.
[22:57] So um this is lineage we collected from running the queries to build up our data estate, right? Across bronze, silver, gold, what have you. Um
[23:09] it took a little bit of work, but not too much work um to do this. For Spark, it takes Well, actually, for Spark for me it was almost harder because I hate juggling jars, but ultimately you just need to install the right jar and and load it
[23:22] into Spark. Um do a little bit of configuration and you're emitting lineage. Datafusion was more fun. Um took a little bit longer, but to be honest um and when it comes to Open Telemetry
[23:35] this is kind of some of the things you can see here. This is not a Gentic stuff here. We just instrumented our query engine to learn a little bit kind of what's going on on the inside um of that query engine.
[23:47] I think we're good on time, so I'll just see if I can find one specific interesting one because we talked about you want to make your platform great.
[23:59] I should have just uh remembered which traces do what. Um I'll find Yeah, this one is interesting. So, we talked about governance if and
[24:12] and that you need lineage to build governance, but let's say you want something like um predictive optimization that Databricks offers. Right, I'm not saying if you emit traces you have predictive optimization.
[24:24] But one thing we did here so we can see the entire query plan. Um this is a messy query plan because there's a bunch of uh files that we need to load. We can
[24:36] also see some intermediate data. We have all of this trace, right? So, we can see some samples of the data as it flows through the query plan. This is very very deep insights you can gather and then you can actually use an
[24:50] agent. I never tried this, but you could try to just point an agent at those traces and say how should I partition my table based on this load that I'm dynamically collecting? This is not going to give you predictive optimization as it is built within
[25:04] Databricks, but the the the message is you you you need to gather kind of deep insights, and using open source technologies you can, it's actually quite straightforward to do so. Um and you get some nice colorful um
[25:18] plots as well. So, >> that's really cool. I mean, what you've really just shown is how powerful and extensible the and and quite rich actually the existing ecosystem is for, you know, how you can actually build a
[25:30] lot of these things with existing uh lakehouse and open source technologies that are out there. Like, this stuff is, you know, it's it's it's within um it's within your grasp actually to build and extend a lot of the ecosystem. So,
[25:42] that's really awesome. Um and so now, uh my business is continuing to grow, and now we're actually starting to get more into building agentic data products. So, now Casper's has actually become a large conglomerate, and we're
[25:54] going beyond just the ghost kitchen business. Now, we're starting to sell agentic data products. Uh and now, I want to uh and I'm starting to work with um uh other partners, and my growing pains
[26:06] around this is, you know, well, how do we start shipping this? How do we keep the data agent in sync? Um and so, let's see, Robert, how can we accomplish some of these problems? >> Well, good news is um we don't have to
[26:18] do much. Let's use open sharing. Open sharing was announced this morning. Those of you who know Delta sharing, Delta sharing is uh a protocol that shares Delta tables and
[26:30] Iceberg tables, I think models are part of it as well. So, this is being extended to, well, share models and volumes. We know that, but also agent skills and models. So,
[26:42] agent skills um well, you know, these kind of folder structures that instruct agents. There is community proposals for the actual agents and pages.
[26:54] Page sounds a little bit boring, but it is not. It is absolutely not. Some of you maybe use Obsidian on on OS X before? So, Obsidian is kind of this program
[27:07] that essentially is just a bunch of markdown files, but you can create links kind of within the between different entities. Right? And pages are just that. Pages are the description of an entity that
[27:20] you can link to other entities. So, if this sounds like a graph to you, if this sounds like an ontology to you, well, that's what it is. And we heard in the in the keynote this morning
[27:33] I'm I'm not saying this is ontologies, but this is rich contextual information that agent can traverse that you can kind of ship alongside
[27:45] your your data products, right? So, when I say when we say agentic data product, it's okay, here's your data, here are instructions for your agent on how to use it. Here is semantic context, entities that
[27:57] live within this data set that may even relate to your own entities, right? You you decide that, you build that out. And then and this is roadmap stuff. So, the the community proposals are actually proposals. There is open questions we
[28:10] need to answer before we can kind of flesh out the spec, but it's pretty far along. The roadmap items, that's further down the road, but we also want to have evil data sets, synthetic data sets, vector indices to kind of finish that agentic stack, to
[28:24] build this this this out, because kind of people may not just want to be consumed, they may want to build on top of your data, and this stuff is important for this. >> Yeah, so one thing you're saying basically is that so say we're running,
[28:36] right, our task for our ghost kitchen business, and we have all these vendors that we're working with, and we want actually want to give them information about you know, the orders that are flowing through, how effective their distribution channel is, the business that they're seeing from their partnership with us. And so now actually
[28:50] leveraging open sharing, I can go beyond just sharing the tables and the dashboards. And now you're saying you can actually give their agents context, enrich their own, you know, LLM wikis and ontologies and stuff like that with
[29:03] this protocol. >> Absolutely. That's exactly kind of where this is going. And the good thing is we already learned about credential vending. All of this is essentially file-based things, right? So, everything is
[29:16] underpinned by credential vending, which is in and of itself a very nice, secure, proven pattern that's also kind of clients already know how to do all this kind of stuff, right?
[29:28] So, it's not we need to learn some new complex governance mechanics. So, it's really an evolution of existing patterns, but this makes kind of permeation in the ecosystem uh really easy.
[29:40] We encourage you to join the discussion. If you have ideas, if you have other assets that you want to share out there, right? Feel free to propose it. Uh open a GitHub issue. Um go ahead. Yeah, please do.
[29:53] We also uh want to kind of share there is an existing Sorry, just in case we well uh there is an existing ecosystem out there of people who want to adopt this, of people who have integrations. Um
[30:05] One thing that is kind of very important, there is on-prem storage providers who um also participate kind of in this and and are committed to implementing and supplying this.
[30:17] Because I mean uh a project like this is only as powerful as its ecosystem, as its community. Delta sharing, there is a lot of ecosystem around that, and we're hoping for open sharing
[30:29] it'll be even more. >> Yeah. Really good. >> All right. >> Amazing. Okay, thank you, Robert. I mean, you walked us through like how our Casper's business now is is uh thriving.
[30:41] Um and we were able to accomplish all those goals that we talked about in the beginning, right? So, what we covered just now is we first started talking about authorization patterns, how credential vending within Unity Catalog
[30:53] helps you, uh, manage access and authorization across all these different engines that might be in your ecosystem and your and your lakehouse. We talked a little bit about Unity Catalog, uh, manage Delta tables that with the power
[31:05] of catalog commits and these new commit APIs, uh, they can actually enable better transaction management, um, especially when you're working with, for example, concurrent workloads across these multiple different engines. And then,
[31:17] uh, we also talked about how to extend Unity Catalog with things like open lineage, with open telemetry, um, and being able to tap into some of that to actually unlock even more powerful workflows. And then, uh, how
[31:31] you can use Unity Catalog as your context layer for agents with metric views. And then, finally, we also just talked about and shared how you can now share tables in any format, all of your AI assets, models, tools, your MCP
[31:45] skills, uh, these wiki pages, these these, uh, markdown files essentially in an open format, um, that now can unlock a whole new class of use cases,
[31:57] um, and we're really excited about growing the open sharing project. So, uh, I also wanted to give a happy second birthday to Unity Catalog, um, open source. So, this is something that if you recall, if you were here at Data
[32:09] AI Summit 2 years ago, uh, I think on this day, uh, exactly, that's when Matei actually went live with the open source Unity Catalog project. Uh, we're really excited. It's now actually, you know, continued to grow. There's tons of new,
[32:22] um, packages. There's actually even more that I haven't even added here and stuff like that, but, uh, we've grown to over 2 million downloads per month. This is actually underselling it. I think we're actually almost like we're at like 3 million downloads a month. I just These
[32:34] are older numbers. And then, over 100 contributors. Really excited to see the community growing. um, Um, a little bit more about the 0.5 release. So, that's what you just saw us upgrade uh to in the unit in the Casper's business. But,
[32:47] this is where a lot of those things that Robert just covered are in. So, the UC Delta APIs, metric views, um some really awesome new stuff to harden uh the production readiness, especially of clients and stuff like that. Um, and
[32:59] also supporting uh credential vending and and things like that to make it easier for engines and integrations like that to be able to plug in, rather than needing to reinvent the wheel on some of those pieces. We now have an out-of-the-box client.
[33:12] Cool. And so, what's coming up ahead? Uh some of the things for that are priorities for us, and then we actually really want to hear from you all what you would be interested in seeing more of in the um Unity Catalog open-source
[33:24] roadmap. Uh but, here's the things that we have uh so far that are high priority. So, views uh support um and taking metric views, for example, it's already experimental today, but taking that um further in the next release. Um,
[33:37] of course, open sharing and implementing like the full uh protocol around share, provider, and even recipient's protocol within Unity Catalog. Um, full Iceberg rest catalog API support. Uh we already
[33:49] support the uh uh read APIs, but now actually adding the create and write APIs. And then, S3 compatible storage. And then, just making some of these just the developer experience a little bit better. So,
[34:01] things like deployment, and security convenience, um investing more in like the Docker container and stuff like that, and making authorization more flexible. There's quite a rich roadmap. We're always looking for more people to join the community, um help us build
[34:13] some of this and and unlock greater interoperability um in the open lakehouse. And so, with that, um I think we're uh at time. And so, you know, thank you so much for
[34:26] joining us. >> I think we have about 6 minutes for questions. >> Okay, yeah. >> Thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.