How to Securely Access data with Lakehouse Federation

Summary

Lakehouse Federation in Unity Catalog enables organizations to query data across Snowflake, BigQuery, AWS Glue, and other sources without copying data, with auto-applied governance policies on every federated table.
The session covers choice, context, and control — the three pillars of federation — and demonstrates how teams on multiple platforms can be unified under a single catalog while retaining their existing workflows.
New capabilities include Iceberg federation with cross-cloud query pushdown, Genie natural language queries on federated data, and bidirectional Delta Sharing with BigQuery for disaster recovery.

How to Securely Access data with Lakehouse Federation

Your data can be spread across multiple data vendors: AWS Glue, Snowflake, BigQuery, Postgres, etc. This is the session that provides you insight on how to govern, secure, and query it across different teams without having to copy data. In this video we will cover Lakehouse Federation and how Unity Catalog can be the catalog of catalogs for your entire estate. We'll demo how federation works, how customers are using it at scale in production, and talk about how the Iceberg ecosystem is evolving and what you should plan ahead for next. We'll show how you can use Databricks' latest AI features like Genie on your open lakehouse and how you can upgrade your lakehouse over time to benefit from everything the platform has to offer.
Talk By: Dmitar Redzic, Engineering Manager, Databricks ; John Spencer, Sr Product Manager, Databricks ;

Chapters

FAQs

What is Lakehouse Federation and what data sources does it support?

Lakehouse Federation is a capability in the Databricks Data and AI platform that allows teams to connect, query, and govern data stored in external systems — including Snowflake, BigQuery, AWS Glue, and Microsoft Fabric — without moving or copying the data. Unity Catalog acts as a single catalog of catalogs across the entire data estate.

How does Lakehouse Federation handle governance across different platforms?

Auto-applied governance policies follow the data regardless of where it lives, ensuring that access controls, masking rules, and audit trails are consistent across federated sources. This means organizations do not need separate governance implementations for each platform their teams use.

Can Genie run natural language queries on federated data?

Yes — this video demonstrates Genie running natural language queries on federated inventory data, grounding AI with a semantic layer that spans sources outside Databricks. The AI semantic layer provides the context Genie needs to understand federated schemas and generate accurate SQL.

What is Iceberg federation and how does cross-cloud query pushdown work?

Iceberg federation allows Databricks to connect to Iceberg tables managed by external systems and push query execution down to those sources, reducing data movement and improving performance. This enables organizations standardized on Iceberg to participate in the open lakehouse ecosystem without migrating their tables to Databricks.

Full transcript

[00:07] Hi everyone, thanks for joining us today. Today we'll be talking about securely accessing data with Lakehouse Federation. Um, as you can see, we talk about a few sources today. Um, Glue, Snowflake, BigQuery, Fabric, and how you
[00:21] can innovate faster across your data estate because we recognize a lot of you guys have teams that work across different platforms and tools. So as a brief introduction, hi I'm John. I work on the product team. I work on
[00:33] federation. I work on iceberg. I work on multiple things at data bricks. Uh my name is Mitar. I'm engineering manager um leading the lighthouse federation team. >> Great.
[00:45] Uh you probably have seen this since you've been here a couple days, but forward-looking statement. We'll talk about some things that are newly released, some things that are coming. So just take some of the things that are coming with a grain of salt. Um please complete your surveys.
[00:58] But yeah, let's jump right into it, which is what is the goal for the session today. What we hope is that by the end of this session, you have learned u how to securely connect your data state without friction. That's
[01:10] really the goal of Lakehouse Federation as a product suite at data bricks. We'll start with the problem area which is you may have teams using multiple platforms. Uh we'll talk a little bit about how we are solving it. Uh some of the new
[01:23] federation sources we've released. uh some new features that we have coming and then what's next. So we'll just give you a sneak peek of what we released and what we plan to release in the rest of the year.
[01:38] So this is something that we've talked about at the keynote and we're going to continue talking about it today which is we really feel that AI doesn't necessarily have an intelligence problem. It has a context problem. And so three things we're going to hammer on
[01:50] is choice, context, and control. How can you bring these really smart models to all of your data? So, the first thing is choice. How can we eliminate the silos where you may have a team that's been
[02:02] using a platform for a really long time, you can't afford the migration overnight, but you want to get them all together so that you can start using this really cool technology that's being built in terms of fragmented governance.
[02:16] Just accessing the data is one step, but then you have to be like, hey, how can I secure all these users from all these different platforms? using things like agents. So that's choice and that's actually honestly a lot of what we'll
[02:28] focus on today. But we'll also talk about context, which is bridging the meaning gap. Sure, you can bring in some legacy tables, but you know, if you just have Claude try to read through all those tables and give you a good answer,
[02:40] it's going to be super expensive and it's going to take a really long time. So how can you start building the semantic layer in data bricks that gives the models context? How can you ground the AI agents with that context? And
[02:53] then the other thing we'll touch on today is control. So whether it's data that's managed within Unity catalog and data bricks or outside of the four walls of the datab bricks platform, how can you have precise access constraints on
[03:06] that data and how can you automate the governance at scale? Because let's be realistic, you're not going to be able to do everything by hand in this new era. So, this is um a meme that I stole from
[03:19] my friend Raj, uh which is a Star Wars meme about like, "Hey, you know, I've governed our lakehouse. I've done it. I've connected everything." And then you may have a business stakeholder that's really jazzed up on everything we're talking in the keynote and they're like,
[03:31] "Okay, so you're ready for all these autonomous agents we're going to unleash. We want the context. We want the guardrails." And then it kind of becomes, you know, like a an awkward conversation because it's not an easy thing to do. Like, it's pretty
[03:43] difficult. So today we're going to talk about two things. We'll start with lakehouse federation. We built this to unify your data. And if there's like one highle diagram for this
[03:55] talk, I think this is one of them to really take away from this talk which is on the left hand side you'll see a bunch of logos. You'll see traditional databases and warehouses, terod data, Oracle etc. On the right hand side
[04:09] you'll see the open format ecosystem. This is where you might see iceberg, Delta Lake, and there's a bunch of logos there now too. And so in the middle with federation, what you are supposed to be able to do is bring it all into Unity
[04:23] catalog without migration and then start asking these natural language questions to your data such as which campaigns drove the most return on investments last quarter. So we'll actually kind of demo this. That's the goal for today.
[04:36] Um, but that is really what we're trying to solve here. And it's worth calling out that lakehouse federation is part of this greater stack. Right? So at the bottom of this diagram, you'll see choice in
[04:49] terms of the open table formats, any cloud, any model, any data. Then when it comes to the context for that uh data, we have things like lakeflow for ingestion. Today we're talking about
[05:01] lakehouse federation. We have lakebase. Then on top of that, Unity catalog is the platform that uh controls the access to all of that context. And then there's a lot of things above it, right? You're
[05:14] doing discovery, you're building out lineage, you're building out cost controls, and so yes, this is more of the L200, L300 on Lakehouse Federation, but it's always good to keep in mind that it's part of this greater platform.
[05:27] So now that I've given you a little bit about the problem statement, um I'm going to pass it over to Demeer who's going to actually show you a demo of what it looks like in practice. >> All right. Thanks, John.
[05:45] Okay, so uh as John mentioned, um we have now a number of data sources that we support and that we introduced since the last summit. uh overall lakehouse federation supports 15 plus data sources that are wellnown u so let me show you
[05:58] quickly how you can connect to your uh foreign data so uh typically it's a two-step process in consists of creating a connection first to your uh foreign system and then creating a foreign catalog on top of this connection um for
[06:11] the purpose of the demo uh I've pre-created one connection uh which is uh a connection to Google lighthouse this is one of the data sources that we most recently introduced. Uh it enables you to read iceberg data from the Google
[06:25] lighthouse. We can see that it's connecting to the iceberg uh rest catalog of the Google lighthouse. Uh let's also check out the catalog that is that is created on on top of this connection. Um we can see that the type
[06:39] of the catalog is um yeah we are supposed to see it but uh so we we can we let's check out some some of the data in this catalog. Um for
[06:54] example this marketing campaign table.
[07:07] Okay. So uh the the data is in iceberg format. Um I think we have some technical difficulties. So we cannot actually see the table format uh here. Oh yeah actually it's it's shown here. Um we can see that table format is foreign. Um so next to this catalog we
[07:21] have also created a catalog that connects to a bigquery data that actually stores the data in the uh bigquery's native format. And uh I wanted to quickly illustrate why um
[07:35] illustrate how we use two access patterns in the leos federation uh space. So uh the catalog that connects to the bigquery uses so-called query federation pattern um while the while the uh the
[07:49] other catalog that connects to the iceberg uh rest endpoint uses so-called catalog federation uh pattern. So what is the difference? U in the query federation data is typically read through a remote compute uh and we try
[08:03] to push down parts of the query or or the full query to the remote compute. So this pattern uh we usually implement in the cases where we don't have direct access to the remote storage or the data is stored in a proprietary format. Um
[08:17] the other pattern that I mentioned is catalog federation. So in cases where we have direct access to the remote storage and data is stored in the open format, we typically implement catalog federation which is going to be
[08:30] typically more performant as it bypasses the remote compute and overall result in less costs uh overall cost for your queries. Uh I just wanted to quickly illustrate what the query plans look for
[08:43] these two different uh patterns that we have in the system. So um we have a same same data set that that we are quering through the GCP layhouse and we can see that uh data is being
[08:55] scanned directly from the perkade table. Um let's look at the the other catalog. So it's a relatively simple query. it it has two filters and we can see that in case of BigQuery catalog it um creates
[09:10] this scan node which actually um which actually pushes down some of the computes to the remote uh data source and uh with some help from the genie I managed to decipher uh this um query
[09:22] plan and we can actually see that uh filter predicates are getting pushed down uh to bequery okay so I think this is it what we had for the first demo
[09:34] So just to re-emphasize like what was going on in that demo um there's like two ways we can access your data. So there's query federation on the lefth hand side and in that case we are pushing down the query to the external
[09:46] database and that's where you're using native tables or proprietary tables that not every engine understands but de meter's team has actually done a lot of great work. So we try to uh be as performant as possible when we're
[09:58] passing the data over the network. And then on the right hand side you have catalog federation. And this is really cool because now we can start running the query against file storage directly because it's in something like iceberg.
[10:10] Um and no remote compute is necessary here. So there's no double compute. And just to go through what that looks like from um a query standpoint, I'm going to try and use both screens. Give both sides some love. We have a very simple
[10:22] query on the right hand side. um the query optimizer within data bricks is going to try and uh translate that to the supported operations whether it's Oracle SQL server whatever and then we'll push down the filters and the
[10:35] predicates when it comes to catalog federation if your data is in something like iceberg the same exact query looks a little different all we have to do in this case is sync the metadata essentially hey
[10:47] where is this parquet file in object storage from whatever platform you're using in this case it was Google lighthouse And then we can execute on a single engine because everyone talks iceberg. And so you know one big takeaway from
[11:01] today and you've probably heard this in other um you know sessions throughout the week is that the ecosystem as a whole is adopting these open table formats and that's pretty exciting. Um it gives you a lot of power in your
[11:14] hands now for connecting multiple platforms when everyone speaks the same language. And so here are just some of the logos out there. Um, and what's also really cool about this and is not like super uh easy to tell is that these
[11:28] arrows have both directions, right? So like not only do we read data from that platform, but they read data from us, which is also really nice because sometimes you need that flexibility as a customer.
[11:42] Now, um, just to give a little architecture overview, uh, this is a highle architecture of like what's going on. Say that you are in BigQuery and you have producers and consumers that are using BigQuery tables. When you are
[11:55] managing that table in BigQuery, you may be reading and writing to it. And if it's iceberg, it may be stored in something like Google Cloud Storage. Same thing in AWS with Glue. Same thing in Azure with fabric and one lake. Now
[12:07] on the right hand side what we're essentially doing is we're creating a foreign table which is what that connection de meter was showing you was and it's a readonly metadata copy of that table. The data stays in object storage. It doesn't get moved
[12:20] necessarily but now you can have consumers in Unity catalog access that table with fresh reads. And we're not going to go deep into the
[12:32] access control framework in Unity catalog in this session. We're a bit more focused on the federation side of the house. But just to emphasize, you know, we have aback policies. We have role-based access control. All that stuff can work on these foreign tables
[12:45] too, right? So that's the goal is like whether it's managed or it's foreign, you can apply a single set of policies across your data estate and then have that observability on top of it.
[12:58] So with that, I'm going to hand it back off to Deer who's going to show you some new functionality on how we build this context layer to make Genie really accurate. Right. Um
[13:10] before we get to the Genie demo, I actually wanted to quickly show you u how we can apply some of the advanced governance features in UC uh on top of the foreign tables. So u we actually
[13:23] have a data set that contains uh HR data. uh it it contains user sensitive information such as such as social security number and also um compensation data. So we're going to apply um these
[13:35] aug policies on top of the foreign catalog and we're going to show you how uh it works in practice. Um so let's quickly check what is the data
[13:49] inside the table.
[14:06] you're having like all the fun live demo stuff, but now you know it's not fake. >> Doesn't want to show you anything, >> right? It was working
[14:19] you need some narration assistance >> but let's see if it gives you anything. >> Yeah. How weird.
[14:33] >> Let's hit the serverless cluster. Maybe this will help.
[15:04] Do you want to stay on it or >> apologies for this? I I think it's >> No, no worries. Yeah, there's a >> All right. So, I think we can walk through quickly. So, so basically let's let's walk through the demo. So, basically
[15:16] >> the process of setting up a book policies on top of your foreign catalog consists of several steps. Um, as a first step um you need to set up the governance tags on some of the sensitive columns that you have. This step can be
[15:29] also automated by using automatic data classification. Uh so in this example here we have set governance policies on the employees u social security number seller information uh and also like
[15:42] other uh composition uh sensitive data. Um as a next step we created a masking function which which actually allows you to which actually masks the data for the users that are going to access this
[15:55] catalog. And um uh basically as a last step we create policies uh that are going to apply uh these masking rules to our foreign catalog. What is important calling out
[16:08] here is these policies will automatically apply to all the tables that show up in your foreign catalog. And if you if you set up automatic data classification, these policies are also going to apply to any u uh any new
[16:22] tables with the sensitive data that show up in your catalog. Okay. Um so I think for the next part we can um we can show the genie demo. So um we
[16:35] have actually a retail data set that is spread across three different data sources. So we have some data um in the snowflake which contains customers and orders information. Then we have some
[16:47] data in the uh databris manage tables which contains uh product catalog uh information. And finally, we also have some data in the BigQuery which which contains marketing campaigns
[17:00] information. Um, it's worth calling out here that now we can also see comments and descriptions on the foreign um data. Uh, as we are as of this week, we are actually announcing beta of this
[17:13] feature. So all the comments and descriptions of the columns and table levels are now visible in the uh foreign um foreign data sources. Um so
[17:25] for the next part we have actually created a genie space that um that combines all these data sets from the uh from the different foreign sources and we we have run um analytics prompt here
[17:39] actually which attempts to get all the campaigns that drove the the most ROI last quarter looking at the data from the from these diverse data sources. Um so we can see that uh Genie
[17:52] produced a nice nice graph for us um ranking the the campaigns that had the highest ROI and also we can see that it created created a nice plot uh with a total revenue and campaign budget um on
[18:06] the x uh axis. So let's take a look look closely at what happened. Um, if we check the code that Genie generated, we can see that it actually
[18:18] does join between the BigQuery and the Snowflake catalog. And what's really cool here is like the comments, they weren't created in data bricks, right? These were like BigQuery comments that were at the column and
[18:30] table level. Genie was actually able to look into those comments and then understand what the table was. So that's a huge benefit of having your context layer in the same place is that Genie
[18:42] would not really give an accurate or any model wouldn't really give an accurate response if it didn't know what was act was happening in the table. So um it's cool to see that in the query plan. Yeah. Okay. On to the next. Let's do it.
[18:58] So um we kind of just showed what this chart looks like in the UI, right? We had multiple data sets in different places. It all came together for which campaigns drove the most return on investment.
[19:11] Now um just to highlight um what Genie's uh architecture looks like at the bottom of the diagram you'll see all the potential places you can grab data from right so it's all the clouds and it's
[19:23] all the major data platforms then on top of that you can start building things like metrics these are the reusable KPIs that your business uses. So in this case, if you had something like return on investment, you may have 20 people in
[19:36] your organization defining return on investment in different ways with different data sets. And so being able to standardize that is very helpful. Using UC metrics, you can start creating glosseries and domains that just help
[19:49] your business users access this data, which may be data they haven't traditionally interacted with because it only used to be analysts interacting with it. And then the ontology layer will pull in comments from your foreign
[20:02] uh sources and it will autolearn your enterprise context. Um and then on top of all of this is where you can start building dashboards. You can query Genie with an MCP. So again like if you just
[20:14] wanted to connect all your data estate and expose it through an MCP, you could do that with what we showed you today. Um or you can use Genie within data bricks. And what was awesome about this is that
[20:27] we were able to meet your data where it was. Um, so in this case, the external platform was Google. But one thing that we are excited about is that we also do give you the ability if you want a more hands-off experience to incrementally
[20:41] upgrade these to manage tables. And it's just one simple command. It's alter table set managed. And what that does is it means that you can start producing on that table now also in data bricks. But it's important to keep in mind that for
[20:53] open table formats like iceberg or delta, it's really a catalog commit-based format. So it's helpful to have these explicit commands. If you want to start migrating between platforms, this is one way you can do it
[21:06] where you can start with a foreign catalog, test out things like Genie, test out consuming workflows and data bricks and then you can use simple commands like this if you want to start producing data on that same table using
[21:18] things uh like DBR. And um what is a benefit of this too is it allows you to do things like vacuuming and compacting using predictive optimization, right?
[21:30] Like once you run this command now you don't have to worry about the manual overhead of making sure that table is as managed as well as possible. And so I'll hand it back off to Deer just to show
[21:42] you um what that looks like in practice for our third demo. All right. Hopefully this one goes fine.
[21:55] incremental table migrations from Glue to Unity catalog. So um let's quickly illustrate uh how how it's done in the practice what John just described. Um so
[22:07] we have a table that is in glue in the perk format. Um it contains some wine inventory data. Um as a first step we're going to
[22:20] migrate or actually convert the table to um external table. Uh so what this enables is as a first step it enables dual rights or or actually dual engine access on on this table. So both data
[22:33] bricks and external engines uh from this point on can access the table both reads uh and writes. Um also this step is reversible. So once you move to external
[22:46] table you can always go back to the foreign table uh without friction. Um as the next step the table is converted to delta. U now on top of the perk we get a
[22:58] transaction log delta transaction log um and all the benefits of delta such as AC transactions time travel etc. Um as the next step we can actually now
[23:11] migrate table to managed. Um by running this uh set manage command. What happens is once the table migrates to managed it gets full full ownership
[23:23] is being transferred to unity catalog making the table a first class citizen in the unity catalog and also it unlocks all the benefits of managed tables such as liquid clustering predictive
[23:36] optimization multi-statement transactions etc. Um if we check the table again we can see that the schema remains intact and also all the rows are there. Um
[23:49] yeah and also checking the type of the table we can see uh now that the table is uh managed. Um so what's worth calling out here on top of the set external and set manage
[24:03] commands we also have a uh additional uh migration features. uh for example you can now create uh new manage tables in foreign catalog directly. So if you have workloads that are um that are created
[24:15] on data bricks they can write to foreign catalog so you have more flexibility in organizing your your data. Uh also you can convert views uh to from from glue to uh UC views but this also works for
[24:28] hive metas store federation. Um and as a last step once you're done with with your migration once you have converted all the tables of interest u you can um actually drop the disconnect your
[24:40] catalog from from glue and from that point on catalog is treated as a standard UC catalog. >> Cool. Thanks. >> Okay, back to you John. >> Yeah. Um so just to recap what we went
[24:54] through there. Um in the first step here all we did was create that connection and federate uh to your table. So in this case it was even a hive table. Um and what's really cool about this is
[25:07] like on day one you can start asking natural language questions of your data like the uh example we've been using today. Which campaigns drove the most return on investment? And so this is one
[25:19] step that really doesn't really disrupt anything in the existing like hive setup. Right? if you had consumers and producers, if you had thousands of EMR jobs, nothing would change at this point in time. Um, but if you wanted to
[25:32] upgrade incrementally, what he showed in the notebook looks very simple, but it's also awesome that you can do it through like SQL commands, right? So, he converted from a Hive table to Delta in this case. So, now you get things like
[25:44] asset transactions and then he also set uh Unity catalog as the managing catalog. Um, and it all again just stays in your bucket and object storage, but it just makes that transition, which can
[25:56] be very daunting for a lot of teams, a little bit more seamless. And then when it comes to what we see with managed tables, um, we do see customers benefit a lot from the cost savings and the faster queries just
[26:08] because you do not have a million people on your team to always be running things like vacuum and compactions. So for the tables that may be in the long tail, you see a lot of savings on these tables
[26:20] once the catalog starts looking at your query history and doing things on your behalf intelligently. Um, another thing to call out here too is that consumers do not have to be within the data bricks platform. They
[26:33] can use external access to um, you know, read and write to those managed tables, which is pretty cool. Um so we'll we'll show a slide on that but um whether it's through our UC endpoints like our IRC
[26:45] endpoint or whether it's through external platforms like Snowflake, Glue, BigQuery that federate to UC. Um these managed tables we try to make as accessible as possible to all your tools.
[26:58] So uh let's keep talking about what we think are some of the really differentiating uh features for uh Lakehouse Federation. One of which is external access. So um I'll kind of break down this
[27:12] diagram from um the top to bottom. So on the top you'll see Delta clients and iceberg clients. Uh this could be Trino, this could be open source spark that want to read and write to your data. And so using the REST APIs that we expose,
[27:26] you'll be able to do that in Unity catalog. This could be a manage table. This could also be a foreign table. On the bottom of the diagram, you'll see all the federation connectors. So, we didn't add all the logos, but you you
[27:38] get the deal. And this kind of boils down to like three major things here, right? Um on the right hand side, you'll see we want you to be able to discover and govern all your data. Um so, you can access foreign tables using the REST
[27:52] APIs, which is pretty cool. Um we do see a lot of customers now experimenting with this paradigm of a catalog of cataloges. Um, and so we do have people that bring in a federation connection
[28:04] and then just expose that to an open source engine. That's totally a viable use case if that's if that's something that you want to do because you can at least now secure it and use all the features on top of it like Genie. When
[28:17] it comes to open sharing, this was formerly called Delta Sharing. Um, but this week we're calling it open sharing uh with a bunch of new features. One of the really cool features is that you can share foreign tables too. So say that
[28:30] you have like a legacy data estate that you want to share with some data recipients that might be in a different business domain. They're just in a different isolation layer than what you work in. Now you can use sharing plus
[28:43] federation to get data to them without copying the data which is a very powerful thing. And then um when it comes to using any client, you can um create, write and read uh from manage
[28:57] tables from DBR or any external delta or iceberg client. And one of the things that um de meter was showing off was like hey this conversion story of I had this foreign catalog and I set managed.
[29:09] You can also create managed tables in that same catalog. Um, so in terms of like your architecture of the catalog, we've given you a lot of flexibility on how you would want to do that over time incrementally.
[29:25] >> So this is one slide that's just showing off more about open sharing. Um, so just wanted to highlight uh Lakehouse Federation as one of the data assets. I think there's a little bit more that's supposed to be rendered on the left, but it it didn't get rendered. Um but yeah,
[29:38] this protocol is actually something that's like really widely um accepted in terms of pandas, spark, tableau, um snowflake also implemented it. So it's a really nice protocol for creating data
[29:51] shares and then having data recipients accept those shares. Um it makes the handoff very easy between the two and you can do this also for your foreign assets.
[30:09] Uh, okay. Next. Um, so yeah, now we're going to talk a little bit about what we released and um what's coming up. So there's like a lot of text in this slide. I might just go through each one and kind of highlight it so you have visibility on it. Um, starting on the
[30:22] left hand side, what did we release since we last talked to you at the summit? Well, one of the things we did is release custom JDBC connections. So, if you're using a data source that might be in like the long tail of data sources
[30:34] that doesn't have a first-party connector yet, you can actually upload that data source into Unity catalog and secure it. So, it's like a very like bring your own to UC and that just gives
[30:46] a customer like complete flexibility on what you want to connect to. Uh, we connect to Palunteer who adopted iceberg recently. Uh, we have a great working relationship with their team. Azure OneLink uh went into preview but it also
[31:00] went into G so it's kind of mentioned two times. Um remote query function. This is one for maybe some of your more technical users that really want control over the SQL that we push down to a
[31:13] foreign database. This is how you can um now use that in preview. We already talked a little bit about set managed for JDBC's. We also um improved the filters and the predicates we can push
[31:25] down with um joins and then we continue to expand like the security posture of federation. So if you're a company that needs to use OOTH because that's what your security team requires now in these
[31:37] connections you can start using OOTH too. So that's what we released last semester after we talked to you at the summit. This year we've released even more things. We went GA with iceberg v3 not only for managed tables but also for
[31:50] foreign tables. So that v3 feature set that's kind of unifying the formats between delta and iceberg. You may have seen that in the keynote where Ali brought uh Ryan blue onto the stage. That is supported not only for data
[32:04] bricks tables but also tables that were created elsewhere. Open sharing is what I just talked a little bit about. Federated comments we're actually very excited about. We've seen a ton of customers join our preview for this because a lot of these data
[32:18] sources like Postgress and and just like legacy warehouses, people have not really thought ambitiously about doing natural language questions on top of these data sources. But now that we're
[32:30] able to bring in the semantic understanding from those tables, you actually can ask things like how much did we make from the last quarter? what was the highest return on investment for iceberg partition evolution. That's
[32:43] a feature that has been supported in the the open ecosystem for a long time. We added support for that in federation. Um more IM role um improvements for Redshift, Postgress, MySQL, Google
[32:57] Lakehouse. Um this was a pretty special partnership because BigQuery was not one of the sources that had historically supported Open Table formats. In April at Google Next, we were actually the
[33:10] first one to announce a birectional relationship with Google where now you can read tables that were written to in BigQuery from data bricks without data copy because it's iceberg and also
[33:23] BigQuery can read UC managed tables. What's really cool about this is the UC managed tables don't even need to be in Google. They can be in AWS. They can be in Azure. like the the possibilities for multicloud and multi-gine are like
[33:37] expanding rapidly. Um and so we're pretty excited about being like a launch partner with Google for that. Then uh for Azure one lake we're announcing GA this uh week. So now it's in in GA for
[33:50] reading one lake data without data copy from data bricks. uh UC secrets is a really nice feature for again those securityoriented teams where you want your connection information like the
[34:03] authentication that you use to talk between platforms to be stored in a secret and rotated regularly. Then um for the road map the top item that we just added in actually this morning is Raiden on foreign tables. So,
[34:17] at the keynotes, we talked about how we're trying to increase um how fast you can use your warehouse and your lakehouse now with the Raiden engine. We're actually bringing all that to foreign tables, too. Um and so, Deer's
[34:30] team has like a lot of work cut out for them because that's not an easy task. But like the whole goal here is that it's a first class uh experience within data bricks. And so we're bringing everything that we've done to get those
[34:42] like millisecond level queries into also foreign tables where it's possible. Uh schema filtering. This allows you to control what you can bring into the foreign catalog. Register iceberg. The
[34:54] iceberg um API set has always had this register command which means you can set the catalog that owns your table without any data copy. You just keep the parquet files where it is. We're bringing that to Unity catalog. Disaster recovery.
[35:08] This one is pretty ambitious too. How can we uh actually give you disaster recovery for tables that are outside of data bricks? This is an expansion of the disaster recovery solution that we're
[35:21] going into public preview this week where not only can you back up your data, you can back up your jobs, your notebooks, and all that semantic and connection information that you put in UC to talk to other databases. So the
[35:35] disaster recovery story can actually now communicate not only between data bricks but also other platforms where you set up disaster recovery such as AWS, Azure, GCS. um automated refresh improves um how
[35:48] much control you have over the refresh times of metadata, custom JDBC foreign cataloges, and then lastly, but very importantly, we're just rapidly expanding the number of um IRC catalog
[36:01] connections that we can access from data bricks. Um if you're using tools, you may see that they've recently announced support for iceberg. it's really popular and so we want to be able to talk to all those different endpoints so that you
[36:14] can federate to them in UC. And I think that is everything for today. Um, thank you all so much for joining. Uh, we would love to hear any questions or comments after the
[36:26] presentation. Um, but I think we'll maybe just wrap it up here. Thank you so much for coming guys. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.