Petabytes in Plain Sight: Extending Databricks to the On-Prem Data You Already Have

Summary

MinIO CTO Ugur Tigli explains how MinIO and Databricks use open protocols—Delta Sharing and the Apache Iceberg REST API—to let Databricks query on-premises data in place without moving it, directly addressing HIPAA, GDPR, and data sovereignty constraints.
MinIO is a 10-year-old, S3-compatible object storage system capable of running from exabyte-scale deployments down to edge devices, optimized for performance through SIMD instruction sets and erasure coding.
The zero-copy architecture keeps data under the governance of the on-premises data owner while making petabytes of production data available as first-class assets in Databricks analytics and AI workflows.

Petabytes in Plain Sight: Extending Databricks to the On-Prem Data You Already Have

Every enterprise building a lakehouse hits the same wall: the most valuable data lives on-premises. You can mask it. Tokenize it. Replicate a fraction. But data bound by HIPAA, GDPR, or sovereignty mandates cannot leave at all. Either way, Databricks never sees the full picture.
Until now.
MinIO and Databricks have solved this by embedding open sharing protocols directly into the data foundation. When your data foundation natively speaks open sharing protocols, data movement becomes unnecessary. Databricks queries on-prem tables directly, in place with full governance retained by the data owner.
MinIO CTO Ugur Tigli shows how customer demand drove MinIO and Databricks to run analytics on full-fidelity production data on-prem through open protocols like Delta Sharing and the Apache Iceberg™ REST API. He covers what happens to your security posture when pipelines disappear, what it means when you unlock petabytes to become first-class Databricks assets.
Talk By: Ugur Tigli, CTO, MinIO ;

Chapters

FAQs

How does MinIO allow Databricks to query on-premises data without moving it?

MinIO embeds open sharing protocols—Delta Sharing and the Apache Iceberg REST API—directly into the on-premises data foundation. When the data layer natively speaks these protocols, Databricks can execute queries against on-prem tables in place, with governance retained by the on-premises data owner and no replication required.

What compliance challenges does the MinIO and Databricks integration address?

Enterprises in healthcare, financial services, and GDPR-regulated environments often cannot move sensitive data to the cloud at all. The MinIO and Databricks integration solves this by enabling full-fidelity analytics on production on-premises data while keeping that data physically in place under the organization's own governance controls.

What is MinIO and what makes it performant at scale?

MinIO is a 10-year-old open-source-rooted object storage system that implements S3-compatible APIs and can run on any hardware—from exabyte-scale commercial deployments to edge devices like Raspberry Pi. Its performance comes from SIMD instruction sets and erasure coding, which offloads data persistence work for best-in-class throughput.

What capabilities are shown in this video beyond standard Delta Sharing?

This video covers MinIO's Iceberg v3 with REST catalog as an on-premises equivalent to S3 Tables, a NVIDIA collaboration for KV-cache memory offloading for GPU inference, and the broader vision of making both structured and unstructured on-premises data available as AI-ready assets within the Databricks Data and AI platform.

Full transcript

[00:08] Good morning and welcome to the session. Um I'm already 30 seconds in so I guess we could get started. Um quite early morning session. So hi everyone. Uh my name is Ur Tigley. Uh I'm the CTO at Min.io and today I'm going to talk about
[00:21] three things. First who we are, what Minio does and then the nine months of last nine months we've been partnering with data bricks. It's one of the best partnerships that I ever worked on. Um,
[00:33] great people, great technology and we find we found a really good area that we can work together and change the dynamics in the on-prem data being available for data bricks um tool set.
[00:47] So I'm going to talk a little bit about that and then the last piece I'm going to talk about how this partnership that we have with data bricks can help for the AI native or AI readiness for a lot
[00:59] of the enterprises and I'll get to it uh towards the end and also leave a couple of minutes for uh questions that you may have as I said u I'm the CTO at minio we've been uh at uh our company has been
[01:14] around for about 10 years we have been um doing um object storage from the start it was our roots are in the open source and what a little bit of the
[01:26] legal stuff I'm going to pass and I'm going to just jump into the meat of the slides um what we have been doing is what we implemented was the data persistency just like what AWS S3 has
[01:39] done 20 years ago 25 years ago using the S3 APIs and the S3 services in a software package that you can run anywhere. We are agnostic to any hardware. We can run on different
[01:52] chipsets. We started working with um x86 and ARM and we have implemented certain things within our code that makes our software the fastest in the market. We use something called CIMD instruction
[02:04] set to get our performance. We offload all of the data persistency work which is called erasure coding in the industry. If you are familiar with block and file type of legacy storage systems,
[02:16] they use RAID and RAID is a subsystem of erasure erasure coding essentially. So we use that and you can run us from exabytes scale into the smallest Raspberry Pi uh from our community.
[02:29] They've been doing this in drones and other things. So that's what we've been doing for about 10 um 10 years or so. In the last one and a half year or so, we've been focusing on purely on
[02:41] commercial customers and uh we have went into a different path. We took it downstream of our uh code and made it commercially available to all of our customers. And as you can see because of
[02:54] our open source roots, we are being used everywhere that's outside the S3 or AWS S3 kind of framework. Everybody has been using us as the de facto standard for S3
[03:07] integration applications. So that's what we've been doing. But over the years, we have also seen a lot of the world's data has been unstructured.
[03:20] We are leading the efforts on unstructured because object storage tends to be starting with unstructured data. But over many years working with enterprise customers, we've seen that
[03:32] they have best of their data are always in a structure format whether from the Hadoop days or in a database format they have been in that kind of a setup and also with the table open table wars if
[03:46] you will that has been very obvious and clearly iceberg is the de facto standard now that everybody kind of agrees on that. What we have done is since we have
[03:58] all of the unstructured data for our customers large scale like one of our customers has three 3.3 xabyte right now and they keep on adding one exabyte a quarter type of a growth. So
[04:10] unstructured data we have been serving very well but the structured data with icework v3 and AWS S3 has done this too with their S3 tables. We wanted to implement AI store tables which is the
[04:24] equivalent of S3 tables in the cloud or in what AWS offers. We have done that with icework v3 and the rest API catalog. So with that a lot of our customers
[04:37] wanted to use it for various reasons. Um and also we have been quite successful replacing HDFS with our customers because HDFS takes three copies. With our erasure coding, we can eliminate that three copy issue and save tons of
[04:52] money for our customers because of the S3A integration with Hadoop and HDFS. So all of those things coming together in the last one and a half years we've been building AI store tables. That's iceberg
[05:04] v3 with the rest catalog. That base building block allowed us to basically work with data bricks and that's kind of the key point here and we
[05:16] use that as a way to look at delta sharing now data bricks calls it open sharing and we use that transfer protocol to use our iceberg v3 tables and all the data there as well as delta
[05:29] if you will if as an object but in iceberg case we take that share with delta sharing or open sharing and unity catalog sees system right in there. So, no copy of the data. One of our
[05:41] customers needed this desperately because they didn't want to deal with copying of data, operational copy issues and all of the things that comes with it. And a ton of customers are waiting
[05:53] like financial services, healthcare, um, European GDPR type of uh, regulations and customers there, Middle East, they all want to keep their data on prem
[06:05] within their four walls or in a colo within their control and then just send the pieces that they need to do the processing for data bricks and we took that and made it into a product and working in our partnership. And last
[06:17] point in this slide I want to mention is we are also working with Nvidia to address a few of the memory issues which I'm going to get to it in the last couple of minutes of our talk here. How we address a agent memory and how we
[06:30] address KV memory. Essentially the amount of data in the GPUs like the amount of data agents create is not fit for the GPU memory. So you need an
[06:42] offloading engine. It's like the old days when we have the laptops and you have a me 8 gig memory and then swap needs to be the supporter and Nvidia figured out that the GPU memory and
[06:54] local memory is not enough when you start generating a lot of agentic data and it doesn't fit into each GPU's memory. So you need to offload it. It's called KV offloading or KV cache offload
[07:06] and we have written into that and Jansen announced this in the beginning of the year as G3.5 layer. It was a non-existing layer and they're trying to address that with MVME type of disk and offloading from Dynamo and using Nixel
[07:20] libraries to transfer that data. So we do all of them in one platform software only. It's about 200 megabytes or binary and we have been very successful in large deployments large scale
[07:33] deployments with hypers scale customers. The gist of the problem that we have seen with our customers were they always wanted to keep their data data bricks
[07:45] has done a tremendous job trying to convince them to move their data onto cloud and then get the analytics going within the cloud framework. But still according to many studies most of the
[07:58] customers still have their core business value or revenue making data within their four walls and this statistics is about all about that. One in five enterprises plan to move their sensitive
[08:10] data to the cloud is very low still. They still most of the customers that we work because we are kind of the on-prem equivalent of this cloud solutions. uh what we work with is always we want to
[08:24] keep our data because it's for many reasons it can be security it can be governance compliance it's our core data it's our competitive edge we don't want to it's most of the time control if you
[08:37] look at underneath like I used to work for bank of America running their infrastructure for server storage and data center my data colleagues in uh bank of America used to build cloud 10 years a I mean Hadoop clusters 10 years
[08:51] ago So 300 400 their CPU utilization was 2% 3%. They were just buying HDDs hard disk drives just to keep the Hadoop and HDFS going and if you look at their data
[09:03] even though it's the highly regulated financial services firm 80% of the data could go into cloud for data analytics but that 20% would always stop them to blended that 80% like PII data and other
[09:16] stuff uh together and they would just not move. So that's kind of the most valuable data. It still has the gravity. It still holds many people and they don't want to move, but they're okay to
[09:29] do something like what we worked on with data bricks using delta sharing as a transfer protocol as a bridge to move your data for processing. And then if they know how it works, it's throughout
[09:41] the session you run a query. So the way it works is you basically share the tables customer table main customer table and then main transaction table both of them are shared it sits on
[09:53] iceberg v3 on minio let's say it gets shared with delta sharing open sharing to unity catalog unity catalog sees them all together and you can run a query the
[10:05] query and processing is done by data bricks on the cloud but the data stays there and you have methods and cleansing meth methods and tools that you can use truncate the table and then everything always stays there it's one way it's
[10:18] read only so it's quite beautiful actually and people say okay I can manage that if I know how mechanics of that works from a regulatory and compliance perspective
[10:31] um this talks about all don't replicate just keep it where it is and then analyze it and when you do these techniques because the alternative of this is you have to mask the data, you
[10:43] have to tokenize data, you have to take the value of the data, reduce the fidelity of the data and then push the data out. At the end, you don't get the same results. You want full fidelity, you want the full data set. And what we
[10:57] have built with data bricks allows you to do that. And these are the same points that I made like regulatory reasons in Europe. We always get GDPR question and that's kind of the restriction why they don't want to move
[11:09] to cloud. and they blend it or bundle it together with one issue like 20% of the data can't move the 80% stays as well. So we really as an industry we really have to solve that problem and with data
[11:23] bricks working we address that and the reason we were the first one and we got the is award partner award with data bricks is all storage industry was looking at this problem wrong they all
[11:35] wanted to have SDKs or APIs into unit catalog and that's that's not how data bricks works essentially and waiting for that and trying to implement something into the catalog was not the right way
[11:48] you keep your data on prem. You keep your data on in your control. you share it to data bricks for analytics work and for compute and that's was the real opening for both data bricks side and us
[12:01] to say okay this is a much better approach cleaner approach you don't have to deal with copy and we have some data on this as well on a separate slide if you're interested and I have the highle numbers later but you spend so much time
[12:15] trying to do ETL pipeline and then replicate and replication issues with that data and if the amount increases is that becomes a operational problem and then by the time it gets to data bricks
[12:27] for analytics it's just a huge infrastructure mess that you have created already trying to just get data there with our solution you just do open sharing no copy and the processing the
[12:40] the real issue is you don't really send the data there's ephemeral or trans transitionary data that goes in for processing but there are methods you can just say truncate the
[12:52] clean the work session and then you don't really have because data bricks doesn't make an effort to do persistence of that data. Let's say in my example of customer master table and then transactions master table goes shares
[13:05] into unity catalog. You run a SQL query on it and then you get the results only that results of that query is in that session and you can clean that session. the real data stays on prem and that's
[13:17] the beauty of this solution and that's why last yesterday last two days normally in these kind of conferences we get 200 people stopping by our at our desk we got 554 yesterday so it's really
[13:31] easy problem to understand and it is a fix to the that problem and that's how this works basically that's I've been sharing only thing you need is the using our AI store tables essentially Iceberg
[13:45] V3. Whatever database, whatever source data you have, you just put it into that data to format, open table format. Once you're in iceberg v3, we do the rest of the work through a guey. You can just
[13:58] share the table and Unity catalog will recognize it. You run the queries, everything done. Oneway street, read only. And it's pretty low key. Like, it's really easy to understand. It's no
[14:11] different than sharing a file on any um from a Dropbox to sharing it on an S3 UI with a signature that you're doing um on on the cloud. So straightforward
[14:25] and the the beauty of it is what I was mentioning the full fidelity aspect of it because any other alternative that I've seen in the industry for the last 10 years it's always been mass PPI and
[14:37] no do the subset of it or do dummy data and all those kind of things all goes away. You control your data. You keep it where it is and you don't have to worry about the whole copy issues. And this is
[14:52] just the one example we took. It was 94 seconds for a full table synchronization in a test we have done and it goes down to four seconds like it's a dramatic it's a magnitude difference in terms of
[15:05] how you can reduce this effort. Yes, of course, your query is running from the cloud and the data is coming, but it's portions portions that you need and it's quite fast compared to doing the full copy of a table
[15:19] and it's just very sleek in terms of how it works and how easy it is even at a P like in our PC's we need to set up a full cluster of our system because of resiliency and things like that. This
[15:31] can be done in a very simple P as a full functionality on a single node, single VM, single node or any kind of a setup that you can just focus on to the um functionality.
[15:44] And these are all the things that we've been talking about. I want to spend one minute on the future of this. The data is already there. Working with minio and data bricks, you can use data bricks to
[15:57] analyze that data at full fidelity is the bottom line. No copy, no setup and no operational issues of doing ETL or change transformation. All you need is to get it on minio tables which is
[16:10] iceberg v3 and nowadays it's so easy. Everybody has tools to figure out a way to put it in that format. Once it's there, data bricks can use full fidelity of that data. What's important is
[16:22] unstructured and structured are coming together in one platform under minio AI store and that's very powerful for enterprise who wants to get ready for AI. The next step of this is agents work
[16:35] different than um a lot of the data platforms. So they create PDFs, they create HTML, they use HTML, they use data in different formats and they
[16:47] create memory because their context is very different than what we are used to unstructured and structured. When I do cloud code myself, I tell my skills that remember this phase because I may lose
[16:59] connection because I have two three different computers. I'm jumping from another. They create PDFs, HTMLs and the intermediary data they create is unstructured and they can use structured as well. So putting it all together,
[17:13] this is going to be quite powerful for people enterprise who wants to do AI. We can provide them unstructured and structured at the same time and they can send it to data bricks for analytics or they can use it for any other AI agent
[17:26] kind of a framework that they're building. So it will be quite powerful to have one platform addressing structured and unstructured and sharing with data bricks.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.