Unlocking the potential of multimodal data at scale

Summary

The FILE data type is a new feature being built into Delta and Apache Iceberg that unifies large binary assets — video, images, audio — with structured metadata in lakehouse tables, eliminating large-binary performance pitfalls, zombie file references, and fragmented permission models.
Unity Catalog governs file-typed columns with the same lineage, access control, and permissions as any structured column, enabling a true multimodal lakehouse without requiring separate governance systems for unstructured files stored in S3 or SaaS apps.
This video demonstrates how an AI query function extracts scene metadata from dashcam video frames in a Delta table, enabling real-time SQL analytics on streaming multimodal data from autonomous vehicle footage.

Unlocking the potential of multimodal data at scale

AI is becoming a very powerful commodity tool to turn unstructured data into structured data. This has greatly improved the importance of unstructured data and is posing new challenges in the world of data analytics: how to govern the data, how to store the data, how to access the data efficiently, and how to make the data available to ML models. In this talk, we will discuss how we're approaching these problems given the state of the world with open table formats such as Delta Lake and Apache Iceberg, as well as new file formats such as LanceDB.
Talk By: Dejan Krakovic, Staff Software Engineer, Databricks ; John Spencer, Sr Product Manager, Databricks ;

Chapters

FAQs

What is the FILE data type in Databricks and what problems does it solve?

The FILE data type allows large binary assets such as video, images, and audio to be stored as references within Delta or Iceberg tables alongside structured metadata, rather than embedding raw bytes or managing files separately. This solves large-binary write performance degradation, zombie file references after row deletes, and fragmented permission models where files and metadata are governed in different systems.

What is a multimodal lakehouse and how does this video propose to build one?

A multimodal lakehouse combines structured metadata and unstructured files in a single governed table using the FILE data type, with Unity Catalog providing unified lineage and access control across both. This video proposes embedding the FILE type functionality into Parquet and Iceberg directly so it is available to all engines your teams already use.

How does Unity Catalog govern multimodal data with the FILE type?

When a column uses the FILE data type, Unity Catalog applies the same governance model — lineage tracking, access control, and permissions — to the file reference as it does to any structured column in the table. This replaces the fragmented model where unstructured files in S3 or SaaS apps like SharePoint are governed entirely separately from the tables that reference them.

What does the autonomous vehicle demo in this video show?

The demo shows a Delta table containing dashcam video footage from self-driving vehicles, with an AI query function applied to extract scene metadata such as objects and events from individual video frames. This enables analysts to run SQL queries over the extracted scene metadata in real time as new video data streams into the multimodal table.

Full transcript

[00:08] Thanks everyone for coming. Uh we're really excited to present some new stuff today. Uh my name is John. I'm on the product team. I work on iceberg and uh multimodal storage. Hey everyone, thanks for joining. Uh my
[00:20] name is Dan. I'm a software engineer. I work in SQL team but also on building like the best possible engine for AI and unstructured data which we'll present today.
[00:32] >> Awesome. So the title for today is unlocking the potential of multimmodal data at scale. We'll be talking about file type which is new functionality for unifying unstructured and structured data. And we really want to create what
[00:45] we are describing as a multimodal lakehouse. Just the disclaimer here. Um forward-looking statement. So, uh some of the stuff we're talking about is
[00:57] released, some of it's not released. So, just take that with a grain of salt. Uh call out to complete your surveys. It's Thursday. You've probably seen this slide a lot. Uh so,
[01:10] Um, and then yeah, one thing is we have a a smaller room today, so we'd love to get questions, but we'll try to go through all of the presentation first before we start taking questions. So the goal for this session really is
[01:23] by the end of it, you'll have learned how to reduce the friction working with multimodal data of any size. So that's where we're talking about large files that have been traditionally very hard to work with in a warehouse uh using any
[01:37] engine. So this is pretty exciting because we know a lot of you guys work within data bricks and outside of data bicks and a lot of the functionality that we're talking about today we're actually putting into parquet and iceberg. So we'll get more into that. Um
[01:49] but that's one of the goals for working with multimodal data is bringing it to all the tools that your teams are using to work with unstructured data. And then lastly, stored in any location because I
[02:01] think a lot of people's data states might be a little bit more um defined for their tabular data. But when it comes to your unstructured data, it could be in S3, it can be in SAS apps like SharePoint and G Drive. So a big
[02:15] goal of this is meeting your data where it is from any location. Just to quickly go through the agenda, we'll start with the problem area, what it looks like storing unstructured and
[02:27] structured data together. Then we'll talk a little bit on how we are solving it. Uh this is where we'll be able to introduce file type and talk about uh a few cool demos. Uh one will be around autonomous driving and another will be
[02:40] around auditing. And then we'll talk about what's next at the end. um what we are trying to build with this new type and uh joining the preview
[02:55] and uh one more slide before I hand it off to Dan which is just framing the problem statement here right which is warehouses were traditionally built for structured data things like timestamps and strings but when you want to start working with unstructured data that's
[03:08] images PDFs videos that can get into the gigabytes that's where things start scaling uh very quickly. Um now your structured rows aren't in the kilobytes, but they're in the megabytes and
[03:21] gigabytes. And that's where things like um parquet and iceberg weren't like traditionally built for these type of things. So we probably won't go into the weeds today, but if you think about how we are compressing that data, how we're
[03:33] creating row groups to pass it on to an engine like Spark, all that stuff was not built for passing videos through uh through a row in a table. So that's what we're trying to solve with this. And
[03:45] with that um I will hand it off to Dan. >> Thank you John. So as everyone knows AI has recently become very powerful tool in transforming unstructured data into
[03:57] structured data. So um you can get more insights from a vast variety of multimodel data and basically run your analytics. So uh today we're going to show you how we are enabling these
[04:10] scenarios by by making it easier to store and manage this data. And we're going to start simple from a real life example to basically um resonate with everyone hopefully. So let's say we have
[04:23] a self-driving um car company uh that um is doing its best to kind of satisfy its customers. uh but the CEO is looking how how to improve what are the most burning problems and questions. So
[04:37] one of those questions might be how to identify situations where the car suddenly stopped for no reason because of course there's some skepticism uh around passengers it can damage the reputation. So now we'll go through the
[04:51] this uh real life example how how to get these insights. So uh imagine that uh every autonomous driving vehicle has an onboard dash camera that records all the
[05:04] time and these videos can grow very large in size. So uh in this example you can see like a sample of those videos and then one of them kind of shows that this situation happened here that the
[05:17] car stopped without reason. So we want to identify that one out of large variety of different videos. Uh and to do that like we have the video data set
[05:30] uh that is um inserted into a table right and um basically some metadata explaining this. So now the challenge is how to process this um huge videos that
[05:42] can be gigabytes or or tens of gigabytes in size right and to uh extract insights from this unstructured data. So AI is really good at this, right? But we need
[05:54] to feed it like the dash cam videos so to get like structured or semistructured scene description which is like humanly readable and can be parsed or or acted upon. Right? So um uh then once we do
[06:10] that it's much easier to extract additional insights from this scene description column which is kind of plain text or descriptive and extract really enumerated like um uh verdict
[06:22] whether there was any real hazard like emergency vehicle or hidden site or there was none which in this case we're looking for. So what we're really looking for is to extract this third and final row out of the uh wide variety of
[06:36] rows which show different situations. So now um we can also do some uh additional analytics or embedding. So we have a multimodel table that has both
[06:48] structured and unstructured data. So the challenge we're trying to solve is how to efficiently work on top of this like how to be very uh um efficient and uh to unlock the potential of AI.
[07:03] So back to you John to explain the challenges of the current approach and how we help out. >> Thank you. Yeah. And Dan's had to work with new animations. We did not uh
[07:15] a new slide it only hides part of the the table, but bear with us. So, we're going to we're going to start with uh what it would look like storing binary in line. Um because this is actually something we've seen both internally and
[07:27] externally with data bricks. A lot of customers start here, especially when you're in the range of a of a file that might be able to make it into a table. Maybe it's in the kilobytes or small megabytes. But one of the key things here is that it just hurts performance,
[07:40] right? when you are trying to put a large binary by value into a table, everything downstream of it now has to be uh passing that large binary and that means all of the compute on top of it
[07:53] needs to make sure it has the right memory size to handle that binary. But a lot of times you don't need to work with the file, right? You might be doing analytics on the table that have nothing to do with the underlying file. And that's why it's not very efficient to be
[08:07] storing things as binary even though it does integrate it really well with the table. I think a lot of people would now store strings in your table, right? It's a pretty like natural thing that you will do. Um so in this case uh on the right
[08:21] hand side you have your like object storage S3 and then you have strings pointing to what would be these dash cam videos in S3. And it now at least you're not passing this like large binary by uh
[08:34] value. You have a reference to the the the binary which is very helpful. But what's tough is how do you coordinate now especially multiple teams working on this data. So in this case we deleted
[08:48] one of the files in object storage and the life cycle policy has nothing to do with the table anymore, right? Like you were just storing a string in the table and so we have a zombie reference. uh that's that's what we're trying to
[09:00] illustrate there um in the red cell and and that's just like difficult to coordinate like it it takes manual overhead to coordinate this. Another thing that we've um seen with customers is it complicates the
[09:13] governance, right? So um when you have analysts that are maybe working in Python or SQL on top of the table, they're used to be doing things like fine grain governance. They might be using role-based access control,
[09:25] attribute uh based access control on that table that's not just impacting the entire bucket of files, but it might be impacting specific rows or columns within this multimodal data set. Well,
[09:38] in this case, the files that are pointed to as a string are governed by a whole different system. This is like broad permissions in uh AWS, for example, where you're like handling them on the
[09:50] bucket level. So now that we've kind of set up some of the difficulties we've seen with customers interacting with these multimodal tables, we'll talk about file type um which is really in the simplest
[10:04] terms the first class uh type for unstructured data. Um, we've seen a lot of this in the community on people trying to improve this problem and so we are really like continuing that conversation and at the end of the day
[10:16] what we're doing is storing the large binary by reference just like you would with a string but now we're bringing in the table formats in the catalog to help coordinate both the governance and the life cycle of those files. So you won't
[10:29] have to worry about zombie references or having two duplicate permission models for the same data set.
[10:42] And with that I will pass it back off to Dan to talk a little bit about our first demo. >> Thank you John. So uh with this demo we'll illustrate the real life example which we just discussed. So the challenge is how to basically uh prepare
[10:55] and curate and store the data so that we can allow u AI to run its magic and to extract these insights. Otherwise, if we took some of the existing approaches and John John explained like with binary or
[11:08] string, we might actually risk either running out of memory issues for larger documents, videos or files or basically getting our data out of sync, which over a period of time can be really troublesome. So, um as discussed, we
[11:23] want to identify situations where the self-driving vehicle suddenly stopped for no reason. So our input will be dash cam videos which are already stored in a volume or some other storage location
[11:35] where we are basically um storing them over time and then we need to introduce this new missing piece to uh efficiently store this column uh that references these videos. So basically it's a
[11:47] pointer and then we'll basically feed this column to all the um AI functions uh to effectively extract the unstructured data and put it into more
[11:59] structured data that we can efficiently post-process and get insights from get hazard assessment. So let's go to the demo. So to illustrate what kind of
[12:12] input we have hope this is visible. So we have like a typical onboard camera from a self-driving vehicle which shows like real life traffic uh cars ahead like situations. So we need to basically
[12:26] take this um data and um insert it somehow into our um SQL storage. Right? So we have a notebook here. Um we
[12:38] pre-executed these queries to save some time. So I'll talk you through through this. So first we're creating a regular table that has a file data column that we just introduced uh and we're reading
[12:50] from specified location which is a volume that contains these videos. So once we insert it like one example is that the path will point to this uh volume uh and the MP4 video file uh
[13:03] which has some also metadata information such as size or content type to better help you uh understand what's what's beneath. So then um we will curate this data uh further
[13:18] because we have an AI function which takes images and now we have video. So now we need to create the Python userdefined table function which basically takes an input of this uh reference to video and create a sequence
[13:32] of frames like sampling uh that will be used as an input later on for AI function. So we bu we build this uh um Python UDTF which uh uh takes input
[13:45] video output uh sequence of files and now we want to insert them into a separate table uh which also conveniently has file column but here file refers to images instead of video.
[13:57] So file can be basically any document any uh basically type of multimodel data that that you can imagine. Um so uh with this now we have the output of this
[14:09] table which has images that represent sample frames from each video clip that we took as input. So now it's much easier to take a still image and feed it to AI function which we'll show a bit
[14:21] later on. Uh that will detect any hazards or situations uh why the car stopped or what happened in between. Uh okay. And then we pass these
[14:34] extracted frames from this um table that we inserted in. Uh we create another uh table which is called frame events which uh we will uh basically create by using
[14:49] the AI query function that databris developed. So it takes an input um file uh uh reference which we created right now and then it has like some instructions for the agent what to pay
[15:02] attention to like you're an autonomous driving safety system analyze the image and identify hazards. So as a final output of this sequence we are getting a full table um with structured data. So
[15:16] it can be JSON it can be strings enums which basically say what is the hazard observed what is the primary object like was it a vehicle pedestrian speed limit sign. So this is now much easier so any
[15:29] conventional um database engine can pull out insights from this one. So then finally once we have this data we want to join everything and and then come up with only the rows which answer to our
[15:42] CEO's hypothetical question uh how what are the situations when the car stopped without reason. So we will filter by hazard none and the the car is fully stopped like speed is zero. So we get
[15:55] the output three rows which u efficiently all of these kind of can be run pretty fast within sub minute to to basically answer this question and as a cherry on top like we did this one uh
[16:08] one off as a snapshot of all the files we had in our volume but to unlock the true power of this uh pipeline we would really create a streaming table like SDP pipeline uh which databick supports to
[16:22] subscribe on any changes in the volume in the source location. So as any new file gets uploaded there, we would automatically pick it up and run it through our sequence that I just shown to generate uh automatic insight. So
[16:36] basically as new data gets in from the cars, we would get real time um analytics and answers to this very important question. Thank you.
[16:48] >> Cool. Let's keep going. Okay. Um, so now that uh Dan gave you a little bit of what it looks like in a notebook, let's take a step back and see some highle architecture on what this really looks
[17:01] like. So our goal with the file type starting from the top here is that we've built it into all the interfaces that you're used to working with whether that's Python, SQL, Scola, the AI
[17:13] functions that we'll talk a little bit more about today uh built into data bricks, external engines and external applications we know are also really important. That's why again for the external engines we're having the
[17:25] discussions within the parket community on like how can we make sure that at least this reference is understood by everything that you're using. Um now that all these can communicate with the file type at the end of the day these
[17:38] files are just still in object storage. Um, and that's also a big part of the data bricks platform is that if you're using multiple clouds, it can be sometimes very hard to connect those. But whether it's uh uh Google cloud,
[17:51] ADLS or AWS, uh the file type will be able to reference all types of files. Then one concept that we want to introduce here is the ability for file
[18:05] files life cycle to be tailored to your use case. So um starting at the top right you have all of these places that you're getting documents from. Some of it is like S3 where you have a huge bucket that you'll never move right or
[18:19] you might have um SAS apps like uh SharePoint or G Drive where to operate on that data you'll ingest it into your lakehouse. And that's why there's two types of file variants. Um there is the
[18:31] file external where the file system is really the source of truth for your life cycle and then there's file managed where in the case you're ingesting it into your lakehouse you actually want the table the multimodal table that we
[18:44] were just describing to really dictate the life cycle of that file. And so when it comes to uh where those files are stored, the difference is with files external, it would just stay in the external location. Whereas with object
[18:57] storage and a managed file, it would be in a table managed storage location. So I think one helpful analogy here is with parquet files for just tabular assets for things like garbage collection and
[19:10] vacuum. A lot of times you will rely on things like predictive optimization or a catalog service to manage those operations because those table formats
[19:22] are defined as catalog based commit formats. That's a similar like analogy to what we think of for managed files where it's like hey I'm not going to be in the object storage myself or my users
[19:36] cleaning up those files. it's actually really managed by the table where we're operating on that particular file. And then for both of these cases though, the goal is that you have a multimodal table
[19:48] with lazy materialization. So we talked a little bit in the beginning about binary and how it can be very difficult to be passing this huge binary across every operation. And you'll also have zero copy versioning where if you want
[20:01] to create something like a shallow clone or branch on the table, whether it's a managed file or an external file, you shouldn't be recopying that file just to curate a certain set of rows in your
[20:13] table. So that's the goal for um both of these variants. And one thing to like um zoom out is that Unity catalog is really built for
[20:25] unifying the governance. Um, and so in our keynotes, we talk a lot about things like LakeFlow and the Lakehouse and Lakebase. Just connecting it to this presentation. We want the Unity catalog
[20:38] layer also to be very helpful for your files, your PDF and media. Um, because all these things that you're building in your catalog like lineage, the semantic layer, all your cost controls and maybe
[20:50] you're sharing data out like all these things we want to be in a single surface. And that's why the file type is deeply integrated with everything else in Unity catalog. And then when it comes to choice, of course, Delta Lake or
[21:03] iceberg, we start at the paret level so that both of those ecosystems are able to adopt the file type. Um, and then it's always good to reemphasize, right? Any cloud, any model, any data. That's
[21:16] where we see a lot of teams able to innovate fast, but also keep their costs controlled. Okay, great. Okay, I'm going to pass it back off. >> Thank you, John. So, let's talk a bit
[21:28] more about what's really included under the hood. So, we now introduce the concept of both external file and managed file. So, as John mentioned, both of them have like key value props like uh lazy processing. So, you don't
[21:41] really pass by value. You pass by reference. So, you can lazily load it on demand. So if multiple parts of the stack really need to pass it around, you only ever pass the reference which is low memory overhead and then also zero
[21:55] copy sharing. So there's only one copy of the files that many uh tables can reference to. Um and unified interface that we talked about. So where are the differences? So governance external
[22:07] files reside in their original location. So SharePoint, G Drive. So they have um once we ingest them or in insert pointers to them uh we introduce volume
[22:19] level governance. So which is like a nice convenient thing but for manage file we fully ingest them into our datab bricks internal storage. So then we have fine grained access control which you can um kind of use to your advantage
[22:32] more granularly right and then crucially for garbage collection we discussed this uh so uh for external file you're responsible for your own uh basically management cleanup because it's just a
[22:45] static reference. So if you delete the file the reference remains if you delete the row in the table the file remains. Whereas for managed file these two are synchronized automatically. So you
[22:57] delete either one of them, the row or the uh file itself, the other one automatically gets cleaned up. And this is pretty convenient for you know compliance like GDPR. If you have to clean up data after a certain amount of
[23:10] days, you can just do it conveniently through SQL delete rows older than you know specific amount of days and then the file itself would automatically get deleted. Uh then also for versioning um
[23:22] for external files you're uh take that responsibility whereas for manage this is also built in. Uh and we can do some storage optimizations for managed files namely if there are small files we can efficiently pack them using offsets and
[23:36] their sizes to kind of optimize their read access. So uh having said that why should you use either one of them? So let's talk a bit about that. So external file
[23:48] basically the main advantage is you keep your data in the original location. There's no disruption. So it's already established life cycle in the file system. So basically you just onboard it
[24:01] to to datab bricks add a static reference to it and you can use it across the stack of UDFs a functions as we showed in the demo. Um and you get some added benefit of governance. Whereas for managed file uh every file
[24:15] is basically native as external but then you ingest it one time only to datab bricks for managed file uh and you get this fine grained access control and optimized processing because everything
[24:29] just works as we discussed like uh garbage collection uh governance automated compliance the GDPR example and then also ETL at scale. So each one has its own benefits. So um tailoring to
[24:42] your scenario. Now we'll show another demo which will show the true fire firepower of of the file data type. Thank you. Yeah. So the the next demo
[24:55] we'll show in a few slides is uh global auditing. So like everyone's favorite favorite topic. Um, but uh, in this case, a CEO might be asking a question like, "How do we flag expenses, which
[25:07] got auto approved, but the actual paperwork doesn't provide a valid reason." Um, hopefully everyone's dinners are expensed correctly this week, but uh, I know a lot of data bricks employees are taking a lot of Ubers and
[25:19] uh, maybe our finance team is going to be like, "What was going on?" So, in this case, we're going to be looking at PDFs and SharePoint. We're going to be bringing them in as a fi file column referencing PDFs and then we're going to
[25:31] be parsing the text from those PDFs to help answer our CEO's question. And um before we get into the demo, there's like two things I want to call out. So the first one is file with the
[25:44] AI functions and data bricks for intelligent document processing. We've gotten a lot of feedback um and qualitative feedback from our customers on how can we get agents smarter for asking what can be these natural
[25:56] language questions. And what we see is if you're parsing raw PDFs into the structured text with things like AI parse document and then feeding that structured document data to your agents,
[26:08] it greatly improves and just saves you cost and time when trying to answer these really sticky problems where now you might be giving data to users that like historically haven't been interacting with that data. So they're
[26:21] going to expect to ask things like, hey, where were all the audit claims that were incorrect? And so if you don't do some of this work beforehand, you're not going to get accurate answers. Um, another thing that I want to call out
[26:34] too is using things like the file type and Spark declarative pipelines, you can also save yourself a lot of heartache because when it comes to either the orchestration or the processing, the
[26:47] fault tolerance, or even just the everyday operations around these pipelines, if you're doing it on your own, you will often be, you know, holding a page at 3 a.m. if something breaks. um if one batch uh fails or one
[27:02] even file in that batch fails then you often have to do every time reruns like this is not simple logic for your teams to have to build themselves and so one of the things that we did with file type from the beginning is think about using
[27:14] things like spark declarative pipelines where you can just declare the DAG and it's uh resolved for you um we incrementally ingest these documents so if someone adds a new audit claim in
[27:26] SharePoint we're not running an expensive AI I query on all the documents. We're just going to be processing the new one. And then if something does fail, you at least have built-in retry and recovery. So you
[27:38] don't have to be stressed about an on call rotation. So with that, um, we're going to go back to our notebook. Um, and just to ground the example, this is what we're dealing
[27:50] with like bureaucracy at its finest, you know, like a lot of really dense documents. Um, and for the sake of time, we're gonna we're going to be going through it pre-processed here, but I'll try to highlight the important syntax if
[28:03] you guys want to try this out in preview. So, the first thing we're going to be doing is we're going to be bringing these files into the lakehouse. So, Deer mentioned how or sorry, Deer, Dan mentioned that um there is like
[28:16] multiple use cases for file, right? In some cases, maybe in the autonomous driving example, you're not going to want to be copying these huge dash cam videos that are like gigabytes, probably now the terabytes or pabytes into a
[28:30] different location. But for things like SharePoint, you probably want to bring them into a lakehouse where you can operate on it easily. So in this case, what we're actually doing is we're defining a place in object storage where we want these files to go. So this is
[28:43] the managed file equivalent. And really all that's happening here is that now the table can control the life cycle of those files. You could still access those files. It's in your bucket in S3,
[28:56] but just like parquet files, you don't expect to be operating on them by a path-based access. Like a lot of times it's just it's just not useful to do that. So what we're doing is we're bringing in the files into a managed
[29:09] storage location and then instead of format binary, we have format file, right? It's just that's like all it really is. It's like a simple change into like what format you want to bring it in. It's no longer going to be binary in your table.
[29:22] Um, so like what does that look like if we ingest from our SharePoint site? Uh, now we have like a bunch of references to these these files. Um, so as you can you see they're they're listed out. Um
[29:35] the the first thing and just working on like an individual document here is we can run things like AI parse which um is really cool because now you don't have to worry about like finding the right model for the right task. We do a lot of
[29:48] intelligent things underneath the hood to be using the most cost-effective performant models. Um we won't give that whole marketing pitch in this talk but like that I just wanted to call that out. Um, and so in this case, what we're
[30:00] doing is we're taking what was a very dense document and we're extracting things that are important to us, like the authorized amount, um, like whether someone approved it, at what time they approved it. And then, um, we could
[30:12] scale this up using AI parse along with AI query to create a full table. And then let's start, um, like joining this with structured ERP data. Right? The goal here is that you may actually have
[30:24] a lot of relevant data already on these SharePoint files, but they've never lived in SharePoint. They've lived in notebooks that your analytics team has been working with. And so in this case, now we can start um merging structured
[30:37] data with parsed text to start answering really important questions. Um so this is an example of the table. And then just like um Dan mentioned in his talk,
[30:49] we can do all of this as STP. So like in this case when you have um your CEO asking you, hey, where did we mess up in our auditing? Where did we approve expenses that shouldn't have been
[31:01] approved? It's nice to do it as a manual batch, but that's more of like a retroactive audit. If you want to have this uh be prevented more in real time, then you can create a uh pipeline that's
[31:13] incrementalized. And that's um what this this table would look like, right? We create a series of streaming tables um that all build on top of each other. So um in the last part of this demo, I
[31:26] want to cover two things. So the first thing is what happens if these audit uh documents contain like personal identifying information stuff that's sensitive. Well, if we were using the
[31:38] strings that we talked about earlier in this presentation, now you would have to manage the sensitivity of both the underlying file and maybe some of the stuff you extracted from the file in our
[31:50] table. But when it's all in a unified multimodal table where you have file type, you can just use the things that you're used to. Column masking, rowle access. So in this case, we're just
[32:02] exemplifying how if you applied a masking rule for a certain set of users, now the doc itself is redacted for maybe a user that shouldn't have access to the underlying file. And so that can be um a
[32:15] really important aspect of compliance. But another really important aspect of compliance is GDPR and the right to be forgotten. And this is another thing we see where it's like, hey, I have a team
[32:27] and maybe I have some analysts that have been training a new model and they've been like working with this data and it's very important for them to be able to move fast, but like once they're done with that experiment, I have to clean it up. And if it's an S3, it's not in my
[32:42] warehouse, it can be a huge task to make sure I stay in compliance. And now we're talking about money because if you aren't following these rules, you will get fined. And so we've seen a lot of like our healthcare customers and our
[32:55] financial customers run into this scenario. And um as Dan mentioned like at the end of the day you can just use the stuff you've already used in your warehouse. You delete the row you expect
[33:07] garbage collection to come in and delete that file at the right time based on your time to live policy. So with that um we'll jump back into the presentation and I'll pass it back off to Dan.
[33:21] Thank you, Jack.
[33:34] beginning of the presentation where we set the goal for for this session. So basically file is an ecosystemwide solution that uh we want to support any size like even terabytes of data any
[33:47] location like it doesn't matter which connector which source which cloud and any engine so crucially we want to unify this across all open table formats and open source like paret iceberg delta
[33:59] spark and datab bricks runtime. So all of them will have the exact same schema which will make it much easier to pass it around, persist it and and use it. Um and then like as we mentioned like key
[34:12] value props are passing by reference across the entire stack and zero copy across both operational and analytical workloads.
[34:25] So um we aren't here to tell you we solved everything but file completes the best platform for multimodel data which is data brick. So we are piggybacking on the existing stack to provide the best governance, performance and scale. So
[34:37] you basically govern it only once and then the life cycle is um handled through um through the technology that we added. uh and we're passing by reference zero
[34:49] copy that I mentioned and uh we are doing uh for a managed file at least optimizations to uh bin pack them and bucketize multiple small smaller files uh for better read access and then for
[35:02] scale as John also discussed we have these uh spark declarative pipelines which allow you to incrementalize and to ingest u files as they show up in your storage for maximum efficiency. uh
[35:16] basically allowing to you to process as much as you know pabytes of multimodel data. So this has already been proven in production. So we have a pretty good track record in handling that much
[35:28] amount of scale. This was just the missing piece to allow you to to efficiently process unstructured data of of any size. So with that we would like to make a nice announcement around this. Go ahead
[35:41] John. >> Yeah we should do with the general mic. No, Dan and the engineering team have been working super hard on this. Um, so we're excited to start asking people to join our beta. Uh, it's like it's early in the journey, but you know, try out
[35:54] today ingesting the large files at scale. So, this is built into LakeFlow. Um, start working with your documents using AI functions. We are passing now file type to AI functions that are
[36:06] actually hosted in an external service. So you don't even have to worry now about making sure your warehouse and the cluster size is massive. We've intelligently communicated between the data type itself and the compute that's
[36:19] running things like an AI function or your UDFs. Then um a lot of you are probably already using volumes for governing files in UC. And as we talked about, we're giving you flexibility on like how you want uh the life cycle to
[36:33] be managed. But if you have files that are already in volumes today, you can start using at least the file type as the column in your multimodal data to work with those files. And it's not necessarily a migration. You're just
[36:45] creating the file column on top of it. And then on the road map, we're looking at things like pre-signed URLs because we know for some customers um external applications are very important. So you might be serving this to users outside
[36:58] of the four walls of data bricks. Um Dan talked a little bit about both the optimizing of garbage collection and then the automatic file compaction. So this is where we're working with customers that are maybe deeper in the
[37:10] weeds on training models themselves. If you think about loading, you know, a lot of multimodal data to very expensive GPUs like H100s, you probably want that data to be uh stored in a in a way that
[37:24] the IO is very efficient so that you're saturating your H100s. You don't want to be paying like 50 bucks an hour and then it's the network that's not loading the data onto your GPUs to train as
[37:36] efficiently as it should. So these are all things that are we're also looking at. Um but I think this is the last slide. So you can take a picture of the QR code if you want to tell us you want to try it out and uh we'll we'll stay
[37:49] for questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.