Skip to main content

Lakehouse Federation: Govern Snowflake, Glue and BigQuery Data with Unity Catalog

Summary

  • Catalog of catalogs: Lakehouse Federation makes Unity Catalog connect to Snowflake, BigQuery, AWS Glue, OneLake and Postgres with unified governance policies applied to all sources.
  • Federated governance: Lineage tracking, row filters, column masks and ABAC policies apply consistently to federated sources, enabling secure query and governance before any data migration.
  • Incremental migration: ALTER TABLE SET EXTERNAL, CONVERT TO DELTA and ALTER TABLE SET MANAGED move federated tables into Databricks managed tables while data remains in its original location.

Lakehouse Federation: Govern Snowflake, Glue and BigQuery Data with Unity Catalog

Watch: Lakehouse Federation: Govern Snowflake, Glue and BigQuery Data with Unity Catalog
Talk by Liran Bareket, Lead Product Specialist (Lakebase), Databricks ; Michael Saltzman, Specialist Solutions Architect, Databricks
Lakehouse Federation makes Unity Catalog a catalog of catalogs: connect Snowflake, BigQuery, AWS Glue, OneLake or Postgres and govern them with the same policies as your lakehouse tables, so you can query and secure data before migrating any of it. This video shows two-way federation and open sharing, lineage, row filters, column masks and ABAC policies applied to federated sources, a Genie demo answering natural-language questions across AWS Glue and Databricks data at once, and the in-place migration path using ALTER TABLE SET EXTERNAL, CONVERT TO DELTA and ALTER TABLE SET MANAGED.
You will see lineage, row filters, column masks and ABAC policies applied to federated sources, plus a Genie demo asking natural language questions across AWS Glue and Databricks data at once. The incremental migration flow uses ALTER TABLE SET EXTERNAL, CONVERT TO DELTA and ALTER TABLE SET MANAGED to move Glue tables into Databricks managed tables, unlocking liquid clustering and predictive optimization while the data stays in your S3 bucket. The video also walks through creating a federated connection and reviews the roadmap, including custom JDBC connections and Iceberg support.

Chapters

FAQs

What is Lakehouse Federation and what does it enable?

Lakehouse Federation makes Unity Catalog a catalog of catalogs that can connect to multiple external data sources including Snowflake, BigQuery, AWS Glue, OneLake and Postgres. This allows you to query and govern data from these sources using the same policies as your lakehouse tables without migrating the data first.

How can you apply governance policies to federated data sources?

Unity Catalog applies the same governance policies to federated sources as it does to lakehouse tables, including lineage tracking, row-level filters, column-level masks and attribute-based access control (ABAC). These policies are enforced consistently across all data sources.

What does two-way federation mean in Lakehouse Federation?

Two-way federation means you can both read data from external catalogs into Unity Catalog and also allow external catalogs to read data from your Databricks lakehouse. This enables flexible data sharing and integration patterns across your data platform.

How do you migrate federated tables into Databricks managed tables?

The incremental migration path uses three ALTER TABLE commands: ALTER TABLE SET EXTERNAL, CONVERT TO DELTA, and ALTER TABLE SET MANAGED. These commands allow you to convert external tables to Delta format and move them into Databricks-managed tables while the data stays in your S3 bucket.

Full transcript

00:00 Introduction: Unity Catalog as the catalog of catalogs

Congratulations. Congratulations. This session is dedicated to managing each platform. Too loud? Good. Manage each platform using Unity Catalog as a catalog of catalogs. I am Leron. This is me—Michael Saltzman. We are both product specialists. I'm a product specialist on the Lakehouse side, and, uh... I'm a specialist, a solutions architect on the cloud infrastructure and security team. OK. Hey, okay. This is the reality we all live in.
We have multiple data sources with many catalogs or centers of gravity within our organization. Sometimes up to three different platforms. Your data may be in PostgreSQL, Snowflake, or BigQuery—data that is not yet ready to be moved to Databricks or your Lakehouse, or data that you are currently moving to your Lakehouse. Your needs or requirements can't wait until you complete the migration to move data from legacy systems to Databricks , and that's what we're going to talk about today.

01:33 How Lakehouse Federation works

The idea of ​​Unity Catalog as a catalog of catalogs is that it allows, through federation and open exchange, to use any client that can connect directly to Unity Catalog, both your Databricks clients and others, like... like, uh... sorry. Like, for example, your open-source ... Spark or others that directly connect to all your sources. So, the idea is that Unity Catalog will sit between you, your other catalogs, and your client, providing managed access to all other catalogs.
This way, you can use your various APIs and access federated sources directly. With unified management, the same rules you apply to your Lakehouse tables can be used to control access to other sources. You get all the benefits of lineage tracking and can even apply row-level filters and column-level masks to these sources. So you have one policy and you can apply it everywhere, whether your data is at Lakehouse or, God forbid, it's still at Snowflake.

02:36 Unified governance for federated sources: lineage, row filters, masks

On the other hand, we allow you to share this data with many clients. All of these clients now support the Unity Catalog API, and accessing data from any of these clients provides the same level of control as when working with Databricks. The Federation works in both directions. And by "both ways" we mean that you can read data from external catalogs, and external catalogs can read data from the Unity Catalog. This is an up-to-date map of their connections, and these are the most popular directories we see on the market.

03:41 Two-way federation with Snowflake, Glue and Salesforce

So, for example, with Snowflake we can merge catalogs, so the Iceberg tables in your Unity Catalog can be read directly from Snowflake, following all the governance rules, and of course we can read what's in your Snowflake catalog directly from Unity Catalog. The same goes for AWS Glue, Salesforce, and all the others mentioned here. It is important to know that the responsibility for reading lies with the directory that is reading the data, and typically you write to one directory and read from another.
Cases where both directories write to the same resources are rare, and doing so is not recommended. Next, we will demonstrate how we perform a " transition in place". How can you take a table that you have joined from one of your catalogs, and after converting it to an external table, we convert it to a managed table. We'll touch on why it's important to convert a table to managed, but the idea is that we provide a complete path to get resources from an external catalog all the way to becoming full-fledged objects in the Unity Catalog without having to perform ETL or other cumbersome and expensive processes.

05:51 Why managed tables: liquid clustering and predictive optimization

The reason you want to convert tables to managed is primarily because it reduces costs and provides better performance. This is actually the same thing . With managed tables, when you move them to Databricks , we give you Liquid Clustering, which is the best way to cluster or partition data, as well as predictive optimization. Because the table is in, because we store the tables inside Databricks, we can give you predictive optimization.
We optimize tables based on your query patterns and their performance. We will reduce operational work by compactly organizing tables for you. We will remove them from the repository when you delete them. We give you the option to restore them within 30 days or within a predetermined time. Finally, we give you advanced capabilities whether you use Databricks natively or through one of our clients. Advanced capabilities include catalog-driven commits, which prevent any locks or conflicts in the repository, and allow you to use one of the new features introduced last year — multi-statement transactions , similar to those found in databases, where you can combine commits from multiple tables and roll them back as needed.
And finally, it's better control. You won't have metadata rejection or unauthorized data access because we control the storage with managed tables. And now we will do a demonstration. Perfectly. Thank you, Theron. So, you hear me, right? Perfectly. Thank you. Um, well , just a quick show of hands before we get to the demonstration. Who here has used Lakehouse Federation before? Have you used it before? Good. Good. Raise your hands.
Who does this with something on AWS? Something on Azure? GCP? Do you want to start with a notebook or Um, so yeah, we're going to move on to demonstrating how you can integrate federated data with the data that's already in UC. So we'll start with a Genie-based demonstration. So, just a quick demonstration. Here in my Genie space, I have already set up a connection to the Glue directory. And in my Genie space, I also have a connection to my federated data in Glue.

08:20 Demo: Genie across federated AWS Glue and Databricks data

And what can I do when I set up my Genie space, let me see if I can find my Genie space settings. It's a little hard to see. But essentially what I've set up here is a connection to my external Glue tables, as well as the tables I manage in Databricks. And here, what I can do is just a quick demonstration based on the wine data with some histograms, line graphs, scatter plots, heat maps, right? a little, okay? Of course.
And, you know, I can ask all these questions. I can ask all these questions at once. And then I can start getting answers both about my federated data and about the data that's already in the Unity Catalog. So, here are some histograms. You know, a few more histograms, line graphs, pie charts, right? So you can take data that's already in your Redshift, or your Glue, or any source into Snowflake, right? And you can combine this data with the data already in Databricks and start asking natural language questions about this data without even migrating it .
Now, what we're going to talk about as we move on to the next demo is how do we take data that's in external tables, let's say in Redshift? And we'll mostly be talking about Glue: how do we take this data and start converting it into Databricks managed tables ? So here I have the data, as I showed earlier, this is my wine data that I have stored in my Glue catalog. And here I just do a "select star" on this data. And when I scroll down, there will be this command here, which is the "alter table set external" command.

10:13 Demo: SET EXTERNAL, CONVERT TO DELTA, SET MANAGED

It takes my external table and turns it into a table. In general, when your data is in Glue, it is already in an S3 bucket “under the hood”. So, we just quickly convert and change the table to an external one. And this feature became generally available (GA) starting last month. You can take external tables in Glue and set them as " external". Now this table used to be a parquet table. It doesn't fit the optimal format we know and love, like Delta.
So now we can do this: once it is set as external, my external table becomes available for both reading and writing. I can both read data and write to it. External tables are usually read-only. But with the command "alter table set external". This allows you to make the table readable and writable. So, once you make the table readable and writable, you can run the "convert to Delta" command. The “convert to Delta” command is available for any existing parquet tables you have now, you can now convert them to Delta and prepare them for Databricks.
Once I converted it to Delta, I can now run my magic command, which sounds like "alter table set managed". I can take my external Delta table and immediately turn it into a managed Delta table managed by Databricks. It still remains in your S3 bucket storage, under your control, but now you can immediately take advantage of all the optimization benefits. So, you can immediately use sparse clustering, predictive optimization, all those optimizations that Liaran talked about earlier.
You will receive them immediately. This allows you to set it to "managed". So, through this gradual process, which we call the incremental migration flow, where we first set "external", convert to Delta, and then set " managed". This allows us to perform an incremental migration to Databricks in stages and immediately enjoy the benefits of migrating to Databricks. So, that was a demonstration in general terms. Um, I wanted to show you something else if there's time, but how much time do we have left?
Do we have 7 minutes? We have 7 minutes. If you move on to, well, let's move on to notebooks. Good. Do you understand what I mean? Go to... Oh, wait. Notepad? So, can we move on to notebooks? Oh, yes. What do you want to know? I wanted to... Actually, maybe you don't have this notebook on your computer. Never mind. Um, okay. So, yes, we can explore the catalog level itself . So, how do I actually set up a federated connection?

14:17 How to create a federated connection

Basically, all you need to do is go to settings. Then you go to connections and click "create connection". When creating a connection, you will see a variety of available types. We have Redshift, Databricks, Google Drive, Hive Metastore. In the case of Glue Catalog, I simply select Hive Metastore and specify AWS Glue as the source. Then you go to the next page and start configuring the connection settings. So I'll just type in a random name.
Then you specify your AWS region and simply go through these steps to get set up. It's a fairly simple interface where everything is done with clicks. We provide the same interface for other connections, like Snowflake and all the other connectors we talked about today. We support these federated connections. We make it easy for you to access it directly in Databricks. And for these federated directories, you can also configure access control.

15:32 Access control and ABAC on federated catalogs

So if I go here, I can go into the permissions of my federated directories and configure the permissions just like you do today in Databricks for managed tables. You can configure permissions, ABAC policies, column-level masking—everything you know and love about the Databricks Unity Catalog will be available for both managed and federated tables . Now let's talk briefly about the roadmap. Last year, we released our own JDBC connections in preview.

16:08 Lakehouse Federation roadmap

This allows you to create your own JDBC connection directly to Lakehouse Federation instead of using the standard connections, so if you have a JDBC that is not on our list, you can use it right away. We have a preview of Palantir released last year, a preview of remote queries and set management that I just showed you. As well as a push-down preview for JDBC join and the generally available (GA) versions of OAuth and Snowflake.
This year we released public federation for Iceberg V3. Regarding external Delta Sharing tables—you can use what is called Lakehouse federation sharing, where you can directly share federated tables with clients without giving them access to your main Redshift. So, we do some data materialization to give them access to this external dataset. We also have a preview of federated comments and the evolution of Iceberg sections.
Previously, we couldn't read Iceberg partitioned tables. Now we can. We also support authentication through IAM roles for Redshift, Postgres, and MySQL. These are some of the other previews we released this year. As for the plan for next year, we will pay more attention to schema filtering and converting external tables to managed ones for Delta. We will create a new separate team for Iceberg as well . With it, you can easily convert external Iceberg tables into managed Iceberg tables.
However, if you want to do this today, you can simply use the CTAS command, which is " create table as select", where you simply create a table by selecting the current data set. So, you can do this today with Iceberg tables, but we'll make it even easier with the convert command. Honestly, this command works pretty much the same under the hood as CTAS or similar solutions. But we will try to make the process more convenient for you.
We will also have disaster recovery tools for external tables, automated updates, as well as extensions for custom JDBC catalogs and more IRC endpoints. We've already released Palantir, an IRC federation for BigQuery, and we're working on more new features. So it will be even more interesting from now on. Thanks, I think that's all about the slides, right? Em. Yes. So: first federation, then management in Unity, and then migration.
That is our goal. So, Unity— you use UC as a directory of directories; No matter where your data is located : in Snowflake, BigQuery, OneLake, or anywhere else, you can bring it directly into Databricks. So, yes, we are open to any questions right now. Yes. We are ready to listen to them, let them come to us with questions. If you have any questions, feel free to reach out. We would be happy to talk to you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.