Skip to main content

Unity Catalog Managed Tables: Faster Queries, Lower Costs, Effortless Data Management

Summary

  • UC managed tables benefits: Unity Catalog managed tables slash storage costs, supercharge query speeds, and automate optimizations with AI on the Databricks Data and AI platform.
  • Seamless interoperability: Third-party clients integrate smoothly with UC managed tables while maintaining the performance gains of managed storage and automatic optimization.
  • Effortless migration: New tools make moving to Unity Catalog managed tables straightforward—enabling teams to realize cost and performance benefits without complex data movements.

Unity Catalog Managed Tables: Faster Queries, Lower Costs, Effortless Data Management

Watch: Unity Catalog Managed Tables: Faster Queries, Lower Costs, Effortless Data Management
Talk by Elizabeth Bowman, Product Manager, UC Managed Tables , Databricks ; Sirui Sun, Sr. Manager, Product Management, Databricks
What if you could simplify data management, boost performance, and cut costs-all at once? Join us to discover how Unity Catalog managed tables can slash your storage costs, supercharge query speeds, and automate optimizations with AI on the Data Intelligence Platform. Experience seamless interoperability with third-party clients, and be among the first to preview our new game-changing tool that makes moving to UC managed tables effortless. Don’t miss this exciting session that will redefine your data strategy!
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What are Unity Catalog managed tables?

UC managed tables are tables on the Databricks Data and AI platform where Databricks manages the storage and optimization—enabling automatic performance tuning and cost reduction.

How much can we save on storage?

Customers typically see significant storage cost reductions through compression, tiering, and intelligent partitioning—specific savings depend on workload characteristics.

Do third-party tools still work?

Yes, UC managed tables maintain interoperability with third-party SQL clients and analytics tools, so you don't sacrifice flexibility for performance gains.

How do we migrate existing tables?

New tools on the Databricks Data and AI platform streamline migration to UC managed tables, making the process simple and low-risk for existing deployments.

Full transcript

All right, awesome. Well, hey everyone. Let's get started. Welcome to Unity Catalog Manage Tables. Uh, faster queries, lower costs, effortless data management. First, some boilerplate forward-looking statement. TLDDR: Everything here isformational. Please don't sue us if our future roadmap doesn't come true exactly when we say it does. Uh, next. Okay. Who am I and who who is she? Uh I'm uh si uh I'm the product manager heading up a team that works on storage at data bricks.
So manage tables everything that makes them great disaster recovery so on and so forth. And Hi everyone. I'm the product manager for Unity catalog managed tables. You can also come talk to me about about converting your tables to be UC managed as well as table level observability. I'll send it off to Siri to get started with this presentation today. All right, let's do it. So I'm going to kick us off. By the way, all the art you see uh today is AI generated.
Pretty pretty cool. Uh so I'm going to kick us off with a with a trade-off that is as old as time, I think, which is the fork in the road between great performance and openness. So by performance, I mean that in the broadest sense of the word. How good is something? Right? Is it fast? Is it easy to use? And by openness, I mean, is the thing interoperable? Is it modular? Can you swap it out? Are you locked in? Are you not locked in?
Uh and so usually uh you can only have one or the other, right? It's a fork in the road. Great performance or openness. And so uh maybe to take a household example, Apple iOS versus Android, right? If you want the smoothest, best performing operating system. This is slightly controversial, but iOS I think is the way to go. It's uh amazing, but you have to use Apple hardware, right? But separately, if you want a operating system that's open source and that it's modular and you can swap it out and you can use it on any hardware, uh you got to go with Android, but you know, it's a little slower.
It's a little less easy to use. Closer to home, I think we have proprietary databases versus open data platforms. So, this has historically been the trade-off in the space, right, for years and years and years before the lakehouse. So, proprietary databases for a long time have promised you really fast performance. They've done automatic optimizations underneath the hood. They give you a really nice polished user experience, everything else under the sun.
And they've been doing that for a while now, but only so long as you keep the data in their systems in a proprietary data format. And if you want to get to that data, you have to go through their compute, right? You have to go pay them and uh run queries through their uh through their engine. And on the other hand, you could cobble the systems yourself using open source technologies. Uh, and that'll give you interoperable or give you modular, but it comes at a cost, right?
Either you hire a team of really great data engineers or uh, you know, you might suffer in terms of price performance uh, and have a number of other trade-offs as well. So, fork in the road still still exists, right? Proprietary database systems versus open data platforms. And at this point, you might uh some of you might ask, well, you know, haven't the proprietary database vendors announced open format support, right?
And I won't name their names, but there's a few of them, and they're all uh I think they're all blue for some reason. Uh but the question is, well, you know, they announced iceberg support recently. So, so isn't this trade-off done for good? Aren't we aren't we past this? Well, it turns out these vendors, well, they might claim to have uh iceberg support or open table format support, but in reality, if you do some digging, it turns out, well, they still fundamentally present you with the same choices within their platform.
So, uh with these platforms, you can either go with their proprietary format flavor, which they've had for a long time now, but of course, in that case, you uh it's the same trade-off from before. You can't get external reads and writes, at least not straight to the storage. Um but of course you get the best performance there right at least relative to their own offerings. Uh you'll get their automated uh query speed up automated storage optimizations all that good stuff.
And then separately they rec recently announced this open format capability. And one thing to note here is that actually all of these vendors this is true for all of these vendors they allow thirdparty clients uh thirdparty tools to read from their data but none of them offer thirdparty rights. So even in this sort of openness dimension, it's it's kind of a partial credit. Uh and then on the price perf dimension, if you're going to use those external tools, they don't like that.
And so they're actually going to they're not going to provide all of their performance optimizations there. So things like their search optimization service or other query optimization services, they're not going to apply to these third-party tools. So you can sort of use them, but the fork in the road critically still exists, right? You can either get blazing fast price performance relative to the other offerings or you can get full openness and you have to choose.
And the core thesis of this talk, the thing that we're going to talk about today is that with managed tables, we've built them from the ground up to not have this choice, right? There's only one uh managed table offering. There's no fork in the road and you get both, right? You get really great performance and you get openness. And so specifically, we'll talk about why we have customers like Zeppto who say that with our optimi uh automatic optimizations, we've been able to save them over a million dollars in storage costs annually.
At the same time, we have customers like Rivian who talk about how manage tables are super open. This is the same offering, no fork in the road, how managed tables are open and have open standards and there's a wide ecosystem of tools uh that they can use. And so specifically uh in a moment will come on we'll break down what our manage tables. So we'll start from square zero and then we'll dive into all the amazing benefits and then from there uh we'll we'll go a little future facing and talk about one a really exciting announcement on an easy conversion command to take your external tables and make them managed and then two as well our upcoming roadmap because we're constantly investing more and more into these managed tables and along the way we'll also talk about a bunch of the new announce uh new announcements that we're launching uh immediately at the data and summit.
Okay, so let's dive in. Uh, no more abstract red and green houses. Let's talk about specifically the problems we're trying to solve. So, first we're gonna center this talk around, our hypothetical head of data engineering. Uh, we're going to try and solve three problems for. The first is great price performance, right? Uh, and specifically, is worried about, well, what's helping to make my tables the most uh, price performant that they can be, right?
Are people getting actionable insights? Are my jobs completing within SLA? Or is everything slowing down to a halt? At the same time, uh, also worried about overspending. Uh, am I overinvesting in optimizations? Am I overdoing it? Uh, as well so could things be, uh, cheaper? Secondly, is worried about vendor lock in and interoperability. So here specifically, am I able to use open source formats? Do I have the flexibility to fire my vendors and go take all of my data somewhere else without having to copy all my data out?
Uh, and third, uh, making the move. Uh so even if you're convinced of the first two bullet points uh are you able to actually take your external tables which often make the uh majority of your data state and make the manage tables easily. So from here hand it off to. So I'm going to tell you about all the benefits that Unity catalog managed tables have to offer for the data engineer and for all of you here. But first, I want to talk a little bit more about what are UC manage tables for any of you any of you who may not know.
In addition, for those of you who do know, I'm going to share some new information as we go along that you actually haven't heard before. So, first, what are UC managed tables? How are they different than other table types? The most important thing to know is that with UC managed tables, Unity catalog is actually the source of truth. In comparison with UC external tables, you can read and write to the cloud storage system, but Unity catalog can't actually see what's going on.
In addition, Unity catalog managed tables provide better performance as well as cost savings as compared to these other table types. So that's why they're so great. And actually that we we've found that of all Unity catalog managed table t tables plus UC external tables we actually see that the majority of Unity catalog table types are managed today. So you know you don't want to be left behind but we're going to share some new tools to help you on your journey.
So that's great. I talked about some optimizations but you're probably wondering what actually makes Unity Catalog manage table so powerful. Tell me more about the optimizations. Let's dive into it. But what's most important to know about Unity catalog manage tables and how they're actually optimized is the fact that unlike UC external tables, UC managed tables actually intelligently learn from all your reads and rights.
So here on the left you can see with external tables, the way that they're actually created is that you say the create the table and you specify the path. Whereas with UC manage tables, Unity catalog will create the path for you. However, you can define the catalog and the schema where that managed table resides as well as the storage location for that catalog and schema. Why is this so important? With external tables, reads and writes can actually bypass Unity catalog access controls.
So, it's less secure and we can't see what's going on. Whereas with manage tables, all access actually goes through Unity catalog. And this structure is so important is because that's what enables fully automatic optimizations because we can see your reads and writes. We can see your query patterns, what predicates you are using, how frequently that you are querying those tables. That's how we're able to come up with automatic optimizations for your specific needs.
So I talked about managed tables, right? Managed, but are they open and interoperable? The answer is yes. They're fully open and interoperable. You can see all of the thirdparty tools here that we offer for you. In addition, with Delta sharing, you can also share data between different organizations using a single stored copy of your data rather than multiple. So, no need for data duplication. In addition, we offer modern open table formats and open catalog APIs.
So you can use something called credential vending to get thirdparty access to your UC managed tables. So you can access your UC managed tables with thirdparty tools outside of data bricks. So think about the data engineer again, right? He doesn't want to be locked in. He wants to choose the best tool to get his job done. And with Unity catalog manage tables, you can do that. So when I say access from thirdparty tools, what do I actually mean here?
First, I'm talking about reads. So you can read a UC manage table from a thirdparty client, but as well you can also write to a UC manage table today using a third-party client. So this is something that's new today and that's being announced at this summit. Unique catalog is again the only catalog that actually supports reason rights from third-party clients and with governance. So that's really what makes data bricks so special.
Our tables are fully open and interoperable. And as a sneak peek of some things I'm going to tell you tell you next. We actually optimize your tables for you. And what does that mean in terms of thirdparty access as well as access from data bricks? That means that you get actually faster reads and writes whether you're reading or writing from a data bricks client or from an external client as well. So that's really why Unity catalog manage tables are so powerful here.
So time for a quiz. Let's see who's been paying attention and who's awake so early in the morning. Which of the following is true? So thumbs up if you think it's true, thumbs down if you think it's false. If you don't know, you can do this. You see managed tables live in customerowned storage. True or false? All right, looks like most of you guys got it right. It's true. Yes, customer you see managed tables. They're managed by unique catalog, but they're in customerowned storage, so you have full control of your data.
Number two, you can save up to 20% on storage costs using UC manage tables. True or false? Or I don't know. Can save up to 20%. I got some I don't know, some yeses. I don't see any nos. False. You can save up to 50% in storage cost using Unity catalog manage tables. So definitely a lot of benefits here. Last question. This is a trick question. It's a tricky question. A lot of our customers don't really know the answer.
So let's let's see who knows the right answer. Predictive optimization is available with UC external tables. True. False. I don't know. I see a lot of nos. I see some yeses. I see some uh some other things going on here. False. So, predictive optimization is actually not available with UC external tables, but because we can't see all the reads and writes that you're making to your table. And predictive optimization, what it does is it learns from how you're reading and writing to your table.
It learns from your usage patterns. But with UC external tables, we can't actually see everything that's going on. Whereas with UC manage tables, remember UC is the source of truth or Unity catalog is the source of truth. And so it can see everything that's going on whether you're reading or writing from data bricks, whether you're reading or writing from other thirdparty tools. And that's how it's able to do all of these optimizations.
All right. So now we talked about an intro of UC manage tables. We talked a little bit about what they are, how Unity Catalog is actually the source of truth. We talked about how the they're the most open and interoperable in the industry, and we talked about some of the optimizations, the automatic optimizations that they provide to you. Let's dive a little bit better a little bit deeper into the benefits. But before I do, I did want to mention one thing to you, which is what's great about UC manage tables is you you can always know that you you're using the latest and greatest features.
You don't have to pay attention to, you know, what's coming out. Maybe I should update my tables. We will do that automatically for you and when it's safe to do so. As you can see here with deletion vectors for any um new tables that you create, this is automatically enabled for you, which makes u merge and table join operations much more efficient. So again, you're always up to date with UC managed tables. So now let's dive into the most important feature available with Unity catalog managed tables that gives you a lot of these benefits and that is predictive optimization.
It gives you 20 times faster query speeds as well as up to 50% lower costs which I hope all of you guys remember from the quiz earlier today. Predictive optimization has been widely adopted. Again, it's saving customers like Zeppto $1 million annually per year. So that's every year they're saving a million dollars. It's a lot of money. And if you're even a larger customer, then you'd save even more. So I said predictive optimization.
You might be wondering, so how exactly does that work? We're talking about predictive optimization. You know, what is it actually predicting? I'm going to walk through that with you here. So what predictive optimization does is it automatically optimizes your layout using artificial intelligence. It will look at the table data, how you're querying from that table. So you know how many bytes are you reading, what are the query predicates, how frequently are you querying that table, all that kind of information it will look at and then a model is trained based on that data to determine the best data layout.
But as it's doing this, it also does the math to see are the optimizations actually worth doing. Let's say that a table is actually like rarely ever queried by anybody. Then does it make sense to optimize it? Probably not. As opposed to if you have a table that's very frequently used, then you definitely are going to want to optimize that for faster query times. So we do that math for you and make sure that the benefits outweigh the costs of actually doing the optimizations.
Those optimizations are then run automatically for you using predictive optimization. We'll automatically size the cluster for you. We'll binack jobs for maximum efficiency. And we'll also log the results to actually show you what happened. And we have a new predictive optimization return on investment dashboard available in beta today. So take note of that. Check it out. That's something new for you. So, we talked a little bit about predictive optimization and how it's actually working.
One more point I did want to point out in terms of the difference between UC external tables and UC manage tables is that with UC external tables, there's a set of commands that you can run to help optimize your tables. But you have to do these manually. These are the ones on this screen. However, with UC manage tables, these are done automatically for you. And yes, I see people taking pictures. That's great. These are good to know about.
UC manage tables, however, will run these all automatically for you. So, as long as you're using a UC manage table, you don't have to worry about these things. So, what's actually going on is that UC manage tables is clustering your data so that it can be queried in a more performant way. It will compact your file sizes to help you save on storage costs. These are optimized commands. But Daniel, the data engineer, for example, he may get really excited about optimizations.
He's running optimize on a regular job on all of his tables. He thinks he's doing great, but he may be overoptimizing his tables when he actually doesn't need to be. Maybe some of them, like I said earlier, aren't actually queried so often. But you see, manage tables will do all that math for you to actually determine when the best time is to run optimize. And then secondly, we've got the uh or thirdly, we've got the deleting unneeded files with the vacuum command.
This is a very important command that helps you save on storage costs. And you might run this with UC external tables or like some of our customers unfortunately they forget to run this and so they're leaving a lot of money on the table unfortunately but with UC manage tables this is run automatically for you so you can make sure that you're saving that money. Lastly there's a command called analyze with UC external tables which UC manage tables automates for you.
I'm many of you probably don't even know what this is. Customers don't know. So we run this automatically for you. What it does is it collects data skipping statistics to make queries run faster. And I'm going to dive a little bit deeper into that specifically for UC manage tables. So I gave you an overview of one of our most exciting features available with UC manage tables called predictive optimization. But I also wanted to dive a little bit deeper into each feature, you know, how it works so you can really understand it and go home and say, you know, I'm an expert on UC manage tables.
So here's a list of everything you get. Some of them you probably know, some of them you might not know. We've got automatic statistics collection. We've got automatic liquid clustering. We've got automatic vacuum with predictive optimization on Unity Catalog managed tables. So again, they're all automatically done for you. We've got file cleaning for drop tables. We've got metadata caching, which I'll tell you some more about.
And we have file size optimizations as well as more features to come. So let's dive into it. Automatic statistics collection. You're probably wondering what what is this? How does it work? What we do is we collect statistics on your tables for particular columns so that we know which data we can skip. So for example here, let's say you're you know you're querying for number four. Um you're looking at column two. Here with file one we can see the minimum is one, the maximum is two.
Whereas for file two, the minimum is 50, the maximum is 100. Those are statistics as some of the examples of ones that we collect. And so you can see we can just skip file two entirely when we're c querying that table. And so that's how we're able to speed up query times because we've collected these statistics. With UC external tables, uh by default you could collect you could run the analyze command and collect some stat statistics for the first 32 columns.
But with UC manage tables, it will optim optimally determine which of those columns it may want to collect the statistics for and it does that all automatically for you which helps increase query speeds. Number two, we've got automatic liquid clustering. So not just liquid clustering, it's automatic liquid clustering. And so just to explain what's going on here conceptually. Let's say I have a query select data from the table where it's February 5th.
Uh customer ID is B. As you can see here on the left side, as data is ingested, it just is ordered by date. So that's great. But when you're querying that data, you're only able to skip 50% of the files. Okay? So it's not in February 6th, it's February 5th. So I'm going to look in these and then I'm going to find you know customer B. You can skip 50th% of the files. Whereas in the second case with liquid clustering in our case automatic liquid clustering because we do that automatically for you.
Here we're clustering the data by customer ID and date. So you can see here if we cluster in a different way we can actually skip more of the files leading to faster query speeds. So when we're talking about external tables and UC manage tables. So what's really the difference here with UC external tables? You have to do manual liquid clustering. You have to choose which columns to cluster by. You have to know it's customer ID uh you know is is B.
So I want to cluster by customer ID as well as date. But imagine you know I have my tables and I'm trying to think like okay who are all the people that are reading my table? What query predicates are they using? I have no clue. I don't even know and I have so many tables. how how would I know how to optimally cluster my data? Well, you can do that. You can try, but that's going to take a lot of time. Instead, with UC manage tables, we do that automatically for you.
We determine which clustering keys will will make the most sense. So, you can just specify cluster by auto. So, you have the option to do so. So, again, you have the optionality with UC manage tables to better cluster your your data and improve query speeds. And so again what you know what's the difference here with UC external and UC manage like I talked about before the thing with manage tables is it can learn from your read and write patterns like I talked about earlier it's looking at how are you querying from that table and based on that it's determining what's the best data layout that we should do.
So I talked about the first two features available with predictive optimization. Again only on UC managed tables that quiz question I talked to you about earlier only available on UC managed tables. Um the third one is automatic vacuum. And so here what's going on is we're deleting data that's no longer referenced by any active table. But as you can see here you can still time travel on the table. You can still go historically to past versions.
But again there may be some data that we just it's lying around. we don't need it. So, we're going to vacuum that. And UC manage tables will actually do that automatically for you to help you save time, to save on costs. Uh please, if you go home, make sure you're either running vacuum or you're using UC manage tables to automatically run vacuum because please don't leave so much money on the table. In addition, something that's um kind of new with UC manage tables specifically is we have something called light vacuum.
And what that is is if you actually have a really large table, some of our customers have had trouble, you know, running vacuum on those really large tables. But with UC manage tables and light vacuum, we'll cache the transaction log information and so you can kind of pull that information more quickly and um then do that vacuum process uh better on those large tables. So that's something available with UC manage tables.
So we talked about those three features. Um, now let's talk about some features that are specific use to UC manage tables but not necessarily part of predictive optimization. We've got automatic file cleanup for dropped tables. So let's say if I run drop table my table, you know, I want to delete it. I want to get rid of it with an external table, the table metadata, that's deleted. But the actual table data, you actually have to go back into cloud storage to get rid of that.
And so some of our customers have had trouble because you know the developers have just they forgot to go and delete everything. So you have a bunch of files sitting around that you know you didn't want those over there. In comparison with UC manage tables you have the option to automatically delete all that data right away. So you can delete the table metadata as well as the table data. So you don't have anything lying around that's going to cost more costs.
But we also have an undrop period that's going to be configurable. And so again, you're not leaving any money on the table. You're making sure when you actually delete a table, it's, you know, going to be deleted. So we talked about that and now let's talk about the last two features. This one, metadata caching, you actually might not know about. A lot of our customers don't know about it. I don't even know if we've like publicly shared it anywhere, but this is something that's good to know about UC manage tables is that in this case when you actually run a query, what we we do is we cache the transaction log.
And so when I say metadata caching, you're probably thinking like you know the column names or maybe the number of rows of the table. It's not that. We're actually caching the entire transaction log where the transaction log tells you the change history, whether you inserted a row in the table, whether you deleted something, all that information, we're actually caching that. So instead of having to actually go to cloud storage, having repeated API calls to cloud storage, which could cost additional money, we're caching that for you.
Um, so we can pull that more quickly. And this is also important with UC manage tables because here whenever we write to that table we're actually able to refresh that cache. And since Unity catalog can actually see everything it can keep an upto-date cache whereas with UC external tables that could be out of date. So again that's why this feature is only available with UC managed tables. So again this helps speed uh query times.
Lastly, I know you guys are like, "Wow, there's so many features there. I can't believe there's another one." This is the last one I'll talk about today. So, we actually automate automatically um optimize the file sizes for you, compacting the files just so they're they're the right size to help you save on storage costs. Um this leverages learnings from thousands of production deployments uh in order to do this. So in summary, UC manage tables give you faster queries, lower storage costs, and all of this is automatically done for you.
So again, the word automatic, that's key. So everyone who's fallen asleep, please stand up for a stretch break. Everyone stand up. Stretch. Stretch. Stretch. I'm going to pass it to Siri. He's gonna share some exciting info. All right, y'all. Y'all can sit down. Good stretch break. All right, cool. So, um, in this section, I'm going to be talking about, well, what's coming up next because we're investing heavily into making manage tables, uh, as always, even more interoperable and also faster and cheaper and easier to use.
So, uh, I'm going to hit you all with a flurry of announcements, uh, of things that are coming and things that are currently uh, released. The first one uh, a release that is a DICE announcement. It's available now. Multi-statement transactions on managed tables. So uh multi-statement transaction I'm sure most of you know what these are is the ability to execute a transaction across multiple tables or statements. So in this particular example query we are merging into a uh customer dimension table as well as a fact table for uh sales at the same time.
And so these two statements across these two tables are either going to atomically succeed or they're both going to fail. Uh and readers are going to see a consistent view of the universe no matter what. So these are a staple of traditional database systems as I'm sure many of you know and it's super helpful for when you're especially when you're migrating to data bricks from one of those systems where you have code like this.
Uh and they are coming in uh they're available in private preview now. Uh and they take advantage of the fact that for managed tables Unity catalog is the source of truth. So there's a catalog that sits above uh the table which is required to actually coordinate transactions between multiple statements and multiple tables. Uh and so try that out. Oh, you can't even see the text down there, but it says now in private preview.
So, uh contact your account team. Uh if you want to try this out next, uh this is something we talk a lot uh about to customers is a lot of y'all we know have external tables. Uh you created them when uh maybe back before manage tables were as good as they are now or maybe they're as part of a legacy deployment. And so, one thing that you'd like to do is move them to manage tables, assuming we've we've sold you today.
Uh and so uh what we've built oh before that let's talk about the problems with uh moving from an external table to a managed table. So uh going back to uh a detailoriented data engineer and so he thinks about the process of moving from external to managed and there's some there's some challenges here right because uh what about preserving table history right uh if you just move a table today and you just copy it straight from an external to manage you're going to lose that table history no matter what you do maybe you don't want that what about preserving the name and the settings and the permissions usually those get wiped as well so you have to reconfigure them readjust your code to uh point to the new table, things like that.
What about uh reducing reader and writer downtime? Because many of you have production systems, many many users, they're all uh expecting these tables to stay uh queryable and you can't really afford long maintenance windows. And then finally, what about handling rollbacks? So what if something goes wrong and you need to undo it? So all of this is not impossible to solve, but it's just elbow grease and maybe you don't want to uh not able to invest in in doing this.
Maybe it's a little challenging. And so that's why we've built this capability uh to basically allow you to easily convert from an external table to a managed table with a single simple command single simple command. Alter table set managed. Okay. So you run this command and it immediately makes this table a managed table. Everything that talked about in terms of predictive optimization, the delta metadata cache, file write sizing, everything like that, it immediately kicks in for this table once this command succeeds.
And um among the benefits is it really aims to solve the biggest challenges today, the ones that just thought about. So first of all, it retains table history. So the new t the new table is a managed table, but retains all of the history from before. Secondly, it retains the same table configurations as well. So the table keeps its name, its permissions and everything like that. And so already you can start to imagine how this uh conversion process might start to be a little bit lower friction.
It also eases reader and writer downtime. And so part of this is just I don't have to go and reconfigure my code to talk to a different table, but also what it's doing is it's doing a lot of the configuration in the background and then it's only swapping clients over once it's finished that work. And so you're not actually waiting in the middle and incurring that downtime. And it's even smart enough to handle things like concurrent writes during the conversion process.
So it's doing a bit of work to do the conversion. During that time, the world might shift out from underneath you, but it's doing uh it's smart enough to reconcile those changes and apply them over once the migration is complete. And so it's really designed so that you can just hit go and really not worry about it. But of course, you might still worry as a proper data engineer, a proper detail oriented one. And so we've also built a roll back capability.
So, uh, for a number of weeks after you run this command, you can actually say unset managed and you'll roll back to the state that you were in before. And our goal is to make it as easy as possible and al also to make it uh a two-way door as well, so you're not walking through any uh anything that you can't undo. This is also available uh in public preview today, and we'll show a QR code in a second for how you can try this out.
Okay, so uh what else is on our road map? But we've already seen about multi-statement transactions. We've already seen about easy conversion. There's another slew of announcements coming up in the keynotes which I'm not allowed to talk about because they are embargoed. Uh but they definitely center around interoperability and the story there around connecting to manage tables from external tools read and write. So stay tuned for that.
Uh I think on Thursday is the keynote. But let's uh lightning round through a bunch of the stuff that's coming after the data and AI summit. So the first one is configurable undrop periods. So as mentioned when you drop a managed table today we will go ahead and delete the underlying files. Today we actually give you seven days in which to undo them. So you can say undrop within seven days and we'll just restore the table immediately.
No questions asked. One thing we're doing is we're just making that a little bit more configurable. So now you can say you know keep this table around even after I drop it. Keep the table around for up to 90 days or 100 days or 30 days or don't keep the table around for seven days. And then you can also sort of empty your recycle bin as well. That's a command below. Uh where you can just say, "Hey, I dropped it. Delete the files immediately because I don't want to pay for storage anymore."
Uh next, we talked about the easy conversion command from external tables. We're also working on one for foreign tables as well. Uh so foreign tables of course ones where there's another catalog uh a foreign catalog that we're federating to and so converting those tables to managed uh easily changing a catalog and schema's location. So, this one's kind of interesting. When you create a manage table, the storage location is in a storage uh bucket that you control.
And the way that we decide which storage location to use is we look at the catalog and the schema and then you can set a storage location for cataloges and schemas. Uh historically, you haven't been able to change them though, and sometimes people have just set it to the default and forgotten about it. But now, they want to sort of clean house and make sure your storage locations align with your catalog and schema. So we're introducing the ability to actually set that after you created it.
Seems obvious but took a bit of work. Uh okay. Uh among other things improve table observability. So this is a really common ask that we get as well around I have a bunch of tables but you know how big are they and how good is their data layout? And we want to make it really easy to observe those properties of tables. In fact not only just of manage tables but also of external tables. And so we're working on a out of the box system table that you can query to get that kind of information about all of your tables.
Okay. And then the last one on this list for the second half is going to be a busy second half of the year for us is auto time to live for your tables. So this is a really common use case we hear where for example for GDPR purposes you might want to say hey for a table delete every row that's older than 60 days. Right? And you could do this today. You can basically write a job that runs all the time and says, "Hey, if this row is older than 60 days, delete it and then later vacuum it."
But we just want to make make that really easy for you. Uh kind of a set and forget thing. So you can say something like alter table set ttl 60 days using this particular column name. And so every row in that column which is older than 60 days uh just automatically gets deleted and vacuumed by predictive optimization. Okay, two more two more things. One is in the second half of 2026 more predictive optimizations. We're just going to be constantly cranking out more and more optimizations.
Our our whole internal process is figuring out that the knobs that data engineers turn and trying to see, hey, can we actually turn those for you at least for a subset of cases where maybe you don't want to worry about them. Um and then finally disaster recovery. Uh really big effort for us. A big lift but also a super important piece of work. We want it so to make it so that for a UC managed table you can tell us to basically replicate that to another region and that we'll do so within 15 minutes uh regardless of the size and the scale of the table.
So super important for us but uh and coming in the first half of the next calendar year. So busy busy road map but to close I hope our talk today has illustrated how manage tables are the best of both worlds right you get openness you get thirdparty reads and writes that's the only managed table offering uh in the market for that and you also get performance and so you know instead of worrying about forks in the road you can uh you know march with confidence down the pave path towards this uh managed tables offering that has both and in fact with the conversion command we make that uh uh the ability to do that even easier and So, I think we've solved all of challenges that we set out to solve, and that makes a happy camper.
All right, so that's concludes our talk today. A few things before we go. One is the QR code for signing up for that conversion to manage tables command is right there. Even if you miss it now, uh you can just look in our docs and there's a blog post coming out in a week. Uh two is uh sessions. So if you're interested in some of the uh in diving into the topics that we talked about today, there's actually deep dive topics that you can see there.
Uh one on uh diving into the conversion command, one on diving into predictive optimization and how it works. There's another one on liquid, not pictured here, and then the last one on well how do you actually uh interoperate with managed tables using external clients. So definitely be sure to check these out. And finally uh be sure to leave uh session survey results for us that helps us improve the talk. And with that, thank you very much.
Enjoy the rest of your conference.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.