Migrate from Foreign Parquet to Managed Delta in 3 simple steps
Summary
- Foreign Parquet to Delta Migration: Convert external Parquet tables from AWS Glue into managed Delta tables on Databricks in three steps with full management and optimization.
- Federated Query Access: Use Unity Catalog connections to query Glue catalogs as foreign catalogs, enabling read-only access to external data before migrating to managed Delta.
- Schema Evolution and Clustering: Once converted to managed Delta, unlock Databricks features like optimized clustering and schema evolution that aren't available on external Parquet tables.
Migrate from Foreign Parquet to Managed Delta in 3 simple steps

Unlock the secret to transforming your data management with a step-by-step guide to converting foreign parquet tables into managed Delta tables on Databricks. Discover how to connect AWS Glue with Databricks, convert Hive-based tables, and enable features like optimized clustering and schema evolution. Ryan and Michael provide practical steps to go from read-only access to full management, ensuring minimal disruption during migration. Perfect for data engineers and architects, this episode offers techniques to modernize your lakehouse and scale your data lake efficiently. Don't miss out on mastering the transition from external to managed tables—your data’s future depends on it.
*Make sure to test in a dev environment before deploying in prod.
⏱️ CHAPTERS:
Chapters
0:00Introducing Michael Saltzman and Ryan Cicak0:36Creating a Foreign Catalog to Glue with Unity Catalog1:40Creating the Glue Connection in Databricks4:12Granting Privileges and Viewing Federated Tables7:20Converting Foreign Parquet Table to Delta Format8:29Converting to Managed Delta Table and Benefits
FAQs
Why would you convert foreign Parquet tables to managed Delta?
Managed Delta tables offer schema evolution, optimized clustering, and full data management capabilities that aren't available on external Parquet tables. This enables better performance and more advanced data operations.
What is a foreign catalog in Unity Catalog?
A foreign catalog is a read-only reference to an external catalog (like AWS Glue) created through a Unity Catalog connection. It allows you to query external data before deciding to migrate it to managed Delta tables.
How does Delta improve on Parquet for AI workloads?
Delta tables on the Databricks Data and AI platform enable multiple versions to coexist for point-in-time reads, critical for AI agents that need to read and write to the same table. Delta provides ACID transactions, schema enforcement, and optimized clustering that Parquet cannot deliver, making it the preferred format for dynamic ML and AI workflows.
How can I recover a Delta table after a failed write operation?
Delta Lake maintains transaction logs and versioning, allowing you to restore tables to previous states using time-travel queries. This capability on the Databricks Data and AI platform enables quick recovery from failed writes without manual intervention or data inconsistency.
When should I choose AWS Glue versus EMR for Spark ETL?
AWS Glue provides managed ETL with simpler setup but less control, while EMR offers greater customization and Spark tunability. For modern architectures, the Databricks Data and AI platform combines managed simplicity with full control, unified governance through Unity Catalog, and optimization for your specific workloads.
Full transcript
0:00 Introducing Michael Saltzman and Ryan Cicak
Hey everyone. Uh. I'm a specialist solutions architect uh with datab bricks and uh today uh we're going to be talking to you about how to convert a foreign uh parquet table in glue to a managed delta table all the way in datab bricks. Uh Ryan, you want to introduce yourself? >> Yes, thank you. , solution architect at Datab Bricks as well. And Michael and I are going to switch on and off of uh of actually doing this conversion.
The very first thing I'm going to do is I'm going to create a foreign catalog to glue with inside of UC. Uh let's do that together. I'm going to go into connections. And before that, I just want to make it clear that we are actually connecting to Glue catalog. And locally on my laptop, I'm using OSS Spark. And you can see with OSS Spark, this is actually connecting to this glue catalog under the database federation demo.
0:36 Creating a Foreign Catalog to Glue with Unity Catalog
And there are a few tables here. Today for this demo, we're going to focus on the customers table. If we do a quick select from customers, we can see the results of this table on Spark OSS. And then this is going to return five results. And this is exactly the glue catalog we're federating to. We're going to create this connection. We're going to call it glue connection. It is of type hive metas store which is AWS glue.
1:40 Creating the Glue Connection in Databricks
We're going to be connecting to US East1. All right. Here is the AWS account and then we have to select a credential. In this case, we already created a credential and in order to create a credential, you just have to create the right IM role and create the right external ID uh to make sure that you have the right permissions for that credential. That will create a successful connection. If I go back into my catalog, I can do everything here.
I can continue creating a catalog, a foreign catalog from this connection here. But I like to just verify everything is set up. I'm going to do that here. So, we created a connection. Now, we want to actually create the catalog. In order to do that, we're going to create the plus button and click on create a catalog. The catalog name that we're going to use today is Federation Glue demo. And this is a foreign catalog again to glue catalog.
And then we're going to use the connection we just created which is glue connection. And then we are going to have to do the authorized path selection. And we already did this ahead of time. The idea here is you're going to select a path that this role is going to be able to access the S3 buckets of your tables. You can see we have the federation demo external location. And then for the storage location, this is where the metadata is going to actually get stored.
It's best practice not to use, as will tell you, not to use the same IM role in this case. And this way, we're going to use the Federation demo storage location. Again, these are separate. And then this is going to read from the same S3 bucket that our customers table is going to use. So this is going to write for the storage location to the glue federation demo storage location. We're going to create and we're going to go to uh configure the catalog and we're going to make sure myself and have the proper privileges and we're going to grant those privileges.
4:12 Granting Privileges and Viewing Federated Tables
Uh so I'll do and myself and we are going to just grant all privileges for this demo. And then that's it. Save. And then you can see the Federation Glue demo has pulled that database that's pulling from glue catalog. And that's federation demo. We should see the same tables as we do now. You can see customers and flights. We're focusing on customers. If we click on customers, you can see Unity catalog has picked up the columns.
If we go to sample data, this will sample uh a limit of 100 automatically as you can see here. And again, this is reading uh read only from glue catalog and there's no data being copied over. And I'm going to pass it over to who's going to continue the conversion process as right now we're in readon. Thanks for passing that over to me. So now I'm going to show on the table that created in the foreign catalog. I'm going to show you today how to convert that table from a foreign table to external and then at the end converting it for all the way from a foreign parquet table to a manage delta table.
So at first I'm going to select from the table just to make sure that I can read it. So here I can read the table. Now, if I try to convert the table to external right away, it's going to give me an error that it's an unsupported provider hive. And that's because the table that we are working with has has uh is a SARD table and we just need to add some properties to it to allow us to convert it. So under the table property section, you're going to want to add a key called spark. sql. sources provider and put uh parquet as the value.
And then you may also need to add a a a value called spark. sql. sources. sema, which I'm just copying from another table. and then put your uh table into a JSON format so that uh so that now we can read it as a regular parquet table. So now my table is a regular parquet table and I'm just going to go back to um to try and convert the table again. Yeah. All right. So now we're going to run the alter table set external command so that we can now read and write from the table.
7:20 Converting Foreign Parquet Table to Delta Format
Awesome. So I was able to run the set external command. Once you run the set external command, then you're going to be able to read and write from that table just like you would from glue. So now I'm going to insert some data into there. So before I only had 500 rows of data and now you can see I'm able to insert data because I've converted it to an external table. And I'm just going to do a select SAR just to show you that we were able to insert the data.
And now there's 510 rows um in the data set. Then I can run the convert delta command to convert my foreign uh parquet table from parquet to delta. So that's just going to take a couple seconds to run. And then I can then alter the table one more time to alter the table from a foreign delta table to now a managed uh delta table. And once I've converted that to a managed delta table, I'm going to get all of the benefits of managed tables, including liquid clustering and predictive optimization.
8:29 Converting to Managed Delta Table and Benefits
So that's just going to take a few seconds to run. All right. So awesome. So now we have officially converted our foreign our foreign parquet table from uh to a managed delta table. And once you have it as a now keep in mind once you convert to a managed delta table your data set is no longer in the same location as its original uh place. So you will need to migrate your producers and consumers to the new S3 path. Now with this uh uh foreign to external and then foreign to finally managed flow, we make it easy for you to keep your existing uh your existing table structure in in your glue catalog.
That way you can do an incremental migration path so that now your table is both read and writes. So you can make uh read and write changes and you can and then after you uh migrate your producers and consumers to Unity catalog then you can convert uh then you can convert to a manage table so that you can take advantage of all the full benefits of data bricks. All right, that's it with our demo. Thanks for watching. >>
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.