Sponsored by: MinIO | On-Prem to Databricks: Zero-Copy Analytics with Native Delta Sharing

Summary

MinIO's AIStor Table Sharing embeds the open Delta Sharing protocol natively into the AIStor binary, enabling Databricks to query on-premises Delta Lake and Apache Iceberg tables directly without copying data, building pipelines, or moving data to the cloud.
This video reports a benchmark showing a 4-second WAN query using AIStor versus 35 seconds with conventional replication, illustrating performance gains that benefit industries with data sovereignty or latency requirements such as retail, medical, and manufacturing.
AIStor integrates with Unity Catalog for authentication and governance using bearer tokens and OAuth2/OIDC, supports bare-metal, VM, and Kubernetes deployments, and provides zero-downtime share, schema, and table administration.

Chapters

FAQs

What is zero-copy analytics and why does it matter for on-premises data?

Zero-copy analytics means Databricks can query data stored on-premises directly without replicating it to cloud storage first. This matters because copying data introduces duplication, pipeline complexity, egress costs, and latency, and may violate data sovereignty requirements in industries like healthcare or manufacturing where regulations prevent moving certain data to the cloud.

How does MinIO AIStor Table Sharing integrate with Unity Catalog?

AIStor Table Sharing implements the open Delta Sharing protocol and authenticates consumers such as Databricks via bearer tokens and OAuth2/OIDC. This allows Unity Catalog to govern access to on-premises tables using its standard access control model, so data shared from AIStor is subject to the same governance framework as data stored natively in the Databricks Data and AI platform.

What industries benefit most from keeping data on-premises while still using Databricks?

This video highlights retail point-of-sale, medical imaging, and manufacturing as industries where data sovereignty or latency requirements prevent cloud movement. In these cases, AIStor Table Sharing allows organizations to keep data on-premises under their control while still giving Databricks read access for analytics and machine learning workloads.

What deployment options does MinIO AIStor support?

AIStor supports bare-metal, virtual machine, and Kubernetes (containerized) deployments, as well as hybrid configurations. This flexibility allows organizations to deploy AIStor wherever their on-premises data lives—whether in a traditional data center or a modern containerized infrastructure—without needing to redesign their storage architecture.

Full transcript

[00:07] Thanks for being here. My name is Dwight Evers. I'm from Minio. I'm a senior field architect and we work with a lot of companies to build high performance object storage that scales to meet the enterprise AI requirements of today's
[00:19] market. So what I'm going to be showing to you today is how we've made it possible for data bricks to query your on-prem data directly without ever having to move it. And this is the whole
[00:31] idea behind a zero copy process. And this avoids having to set set up complex pipelines and having to build uh infrastructure around that. I'll show
[00:43] you how this works in our environments, but first let's talk about where this matters and how you can get started with it. Before I dig too deep, I first want to say how uh honored we are to be selected
[00:59] as ISV partner of the year by datab bricks for the this new storage tier. As a brick builder, we're very very appreciative of the work that we've had both working with data bricks as well as
[01:11] with our customers on this. We started this process back in October of 2025 and today we're here on at at Summit explaining this to everyone here.
[01:29] So at MIO, we're object pure play at heart. We started out about 10 years ago working with customers to be able to pro provide the the high performance object storage that meets any type of demand. Uh our founders saw that there was going
[01:41] to be at some point an exoscale type of challenge that customers were going to have to be able to overcome and we wanted to be able to help them. So we're actually on all seven continents. Yes, somebody's downloading and using us in
[01:54] Antarctica. Hopefully they're staying warm, but we're here working with customers like yourselves to make sure that there's an efficient pipeline in working with data to data bricks.
[02:10] So, everyone in this room is invested in data bricks. You're running analytics and machine learning, maybe a handful of different AI workloads, but the reality is that a huge percentage of the data is still sitting where it's not easily or
[02:22] accessible via data bricks. on prem. So if you're in retail or point of sale, if you're in medical, you have imaging information, all these are different challenges that hey, how do I get this up to data bricks to be able to work
[02:35] more efficiently with it? And it's not that data bicks is doing anything wrong. They're doing everything right. And that one of the things that they've done is improve the ability for companies and and organizations to share data using
[02:48] the what's known as the open sharing protocol. So one of the things that we do is work with customers to avoid these the idea around data silos. It's increasingly becoming a challenge.
[03:01] Hey, I have I accumulate my data in all these different stove pipes and then hope that I can work with them more efficiently later on. That becomes more of a problem because that creates data duplication. When it comes to analytics, I don't know
[03:14] of an analyst in this room that doesn't already want to be able to work with more data more efficiently and without having to wait for it to be copied. Replication to them is a four-letter word. And then infrastructure, there's a
[03:28] lot of times with infrastructure has to be able to keep up with the changes that are happening in an environment in an enterprise. And of course, one of the biggest challenges is cost. How do I keep the billing under control? How do I
[03:40] make sure that my pipelines are staying fed? So, what we did is we are the first truly native open sharing method uh solution that's out there that's
[03:52] actually implemented at the storage layer. There's no bolt-on. There's no extra service you have to deploy. It's directly integrated into the AI store 200 megabyte binary that anyone can
[04:05] download and deploy. The whole idea is to make it native, but it works with both iceberg and delta tables. We don't discriminate. Originally, the delta sharing protocol only worked with delta tables. In
[04:18] working with data bricks, we've expanded on that and brought that full scale to include a rest catalog that we provide for iceberg tables as well. And of course, it's read only by design.
[04:32] The reason for that is it's sharing not giving. If you have dogs or kids, you understand the difference between sharing and giving. So they don't get to keep the data, but
[04:45] they get to work with it. That's the whole premise of of what we've been doing working with data bricks going forward. How it works under the covers is this is that you have py whether it might be
[04:58] python spark trino whatever is generating the information on prem you can then go ahead and store that in a store with with delta tables it's bucket and path style access to the to the uh
[05:10] delta logs with iceberg we provide an easily accessible scalable and secure rest catalog which will then we can then share that information, generate data
[05:22] bricks ready credentials and then be able to have Unity catalog mount that directly at that point once it's in Unity catalog you can query it from any service that
[05:34] it's already integrated with UC that has been the whole premise of us being able to work quickly with data bricks on this.
[05:46] So for delta tables very straightforward you have direct access to those tables wherever they sit with an AI store. The bucket and pastel access is easy for everyone to understand today. And of course there's you can logically group things across different uh table
[05:58] structures and but still preserve all those table predicates that are there including dynamic updates of those tables. If I increase the number of rows if I have to drop or truncate a table
[06:10] all that's still maintained. with iceberg tables. It's fully integrated into the AI store table catalog. It's a REST catalog that understands uniform uh formats like
[06:24] iceberg. It's highly efficient from the standpoint that nothing I have nothing I do requires additional pipelining. It's on the-fly conversion from the data types that are native to Delft tables
[06:36] for iceberg tables. That's how we fit across the protocol. The two formats are similar enough that we don't have to do anything extra and it preserves things such as time travel and snapshots that are taking
[06:49] place behind the scenes and of course maintains the predicate push down for for our data engineers to work with.
[07:02] So you can work with us either sharing locally using for example Python uh Jupyter notebooks or within data bricks itself using your workspaces in the SQL editors the data will look the same exactly the same way and there's never a
[07:15] challenge to hey is my data being cached data bricks will stay will still maintain a cache of the schema and the data types but it won't store the data
[07:27] unnecessarily this allows you to update to have fresh fresh access to data whenever you need it. Tomorrow we'll show that in a demo, by the way.
[07:39] But it's all about accelerating time to insights. The proof is in the pudding. Rubber's got to meet the road. Show me how fast this goes, Dwight. And this is one of the biggest challenges. A lot of
[07:51] people are like, seriously, you're going to access data on prem from the cloud? >> Well, as my customer told me, It's 2026, not 1986. We're not coming over 300 baud
[08:04] dialup modems. We have fast access. So when we looked at deploying AI store directly in the public cloud sitting right alongside datab bricks, we saw that it was about a 1.2 second for a
[08:17] TPCH query to run. When we looked at what it took to read that same amount of data from on prem over a two gig pipe across the WAN, it
[08:30] was only 4 seconds. It's a little slower. We're not going to lie about it. Compare that to normal traditional methods where you have to replicate it. That same uh terabyte data set took
[08:44] under the best conditions 35 seconds to replicate to the cloud. And then you had to go through the ETL process. One of the engineers that visited me at the booth recently said ETL is a new
[08:56] four-letter word. And then of course there's CDC pipelines. They have a they have their place within an organization. When you have large amounts of data that's changing very very rapidly, that may not
[09:10] be the best suit method to go through. And we saw that during our proof of concept that the time for all the data to quies was over 94 seconds before you could have accurate results. With AI store, you're able to get that
[09:23] within 4 seconds. How would you go ahead and deploy this? Well, we don't discriminate. We're not a hardware company. We understand that you
[09:36] may have infrastructure already there. This is a starting point that I have lined up here. Whether it's bare metal, some storage admins just give me a binary and I'll drop it in and deploy it using anible. For others that are have
[09:49] more of a private cloud type of approach, we fully support the use of containers and Kubernetes. We have several very large production customers that are on that today. But it gives you either resource isolation or maximum IO.
[10:02] You can you can pick and choose what makes the most sense for your organization. As far as a starting point, look, security, security, security. We definitely want to walk through that with your teams at some point. But the
[10:14] idea as far as hardware, we understand a lot of companies want to start out with something as a proof of concept as just show us where we can begin to look at things, for example, like a VM. So, we do have customers on that. As far as the
[10:27] software, it's pretty skinny. It's a 200 megabyte binary. Drop it on either Linux or Mac, you're ready to go. As far as networking though, I'm not going to lie, if you have a lot of data,
[10:40] two gig may not be enough. If you're talking hundreds of pabytes of data, this is where we would want to go through some type of sizing exercise with your team. But for our benchmark with several terabytes, two gig was enough to do a 4-se secondond response.
[10:58] So for the business, this changes the math on how on four different fronts. You get faster time to insights because you're querying the live data. There's no stale copies to have to trip over. You lower cost because you're not having
[11:10] to duplicate storage repeatedly or having to maintain complex synchronization pipelines. And then you also get to reduce the the operational risk because there are fewer systems to manage that are going to potentially
[11:24] create a single point of failure. And because it's open protocols and open formats, you're not locked in. You have options in front of you that you can work with. So today we're focused on
[11:36] tabular information. That's iceberg and delta tables. Today we're working very closely with data bricks to bring other things to the table as well. For example, tagging from the storage side
[11:48] that's then predicated all the way through the pipeline. That's around the idea of around governance. The next piece we want to work with them on is unstructured data. Your JSON navro files that can be sitting out there in
[12:00] buckets, your CSV files, but also images and video. There's a lot of there's a lot of things that we're working with data bricks directly on to make that sure that's part of the upcoming protocol updates.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.