Virtual Delta Tables: Building a Multimodal Lakehouse

Summary

Virtual Delta Tables incorporate original data from multiple sources into a single logical structure using the Delta transaction log as a metadata store, without requiring re-encoding or modification of existing files.
Scribd manages over 400 million user-uploaded documents representing billions of pages and trillions of text and image chunks, which drove the development of scalable multimodal storage solutions at the company.
By exposing a standard Delta Lake interface, Virtual Delta Tables enable the full lakehouse ecosystem — including Databricks and DuckDB — to work with mixed-media data without additional translation layers.

Virtual Delta Tables: Building a Multimodal Lakehouse

Managing multimodal inference is one of the defining challenges facing AI teams. LLMs now consider documents, images, and videos, producing higher quality results and reducing time wasted with data prep. The lakehouse architecture solves part of the problem but many teams are struggling to incorporate an increasing variety of data into their workflow. Rather than ingesting, re-encoding, or modifying massive troves of existing data, multimodal teams are looking to work with original artifacts. This talk introduces Virtual Delta Tables and the associated open source code designed for multimodal inference. Virtual Delta Tables incorporate original data from different sources into a singular logical structure for ease of use by data scientists and ML engineers. By offering a Delta Lake interface the entire ecosystem from Databricks to DuckDB can be used in multimodal work. We'll highlight just how far you can push the lakehouse architecture in the rapidly changing AI data landscape!
Talk By: Tyler Croy, Slop Janitor, Scribd, Inc. ;

Chapters

FAQs

What are Virtual Delta Tables?

Virtual Delta Tables incorporate original data from multiple sources into a single logical structure using the Delta transaction log as a metadata store, without requiring re-encoding or modification. This allows data scientists and ML engineers to work with native file formats across the entire Delta Lake ecosystem.

Why does multimodal data management at scale require a new approach?

Traditional storage formats like Parquet face limitations with binary data at scale, and re-encoding massive document collections is prohibitively costly. Virtual Delta Tables address this by pointing to original artifacts rather than copying or transforming them, using bin packing and offset-based file optimization for efficiency.

How does Scribd use multimodal data for machine learning?

Scribd hosts over 400 million user-uploaded documents representing billions of pages and trillions of text and image chunks, driving the need for scalable multimodal storage. Their work on Virtual Delta Tables emerged from challenges in supporting LLM-based document understanding at this scale over two decades of content accumulation.

What tools can read Virtual Delta Tables?

Because Virtual Delta Tables expose a standard Delta Lake interface, the full ecosystem of Delta-compatible tools — including Databricks and DuckDB — can be used directly for multimodal workloads. No additional translation layers or custom connectors are required to query original artifacts through this unified logical structure.

Full transcript

[00:08] I'm going to go ahead and get started. Uh, this is the presentation. If you want to skip ahead, I will let you scan it for anybody who didn't scan it already, or you can download it, share it, remix, whatever. You know, um, I
[00:21] I've gotten this question before. The tool that I use for presentations that I really like is called Presenterm. It's on GitHub. It lets you just take a markdown file and turn it into a presentation. Uh last year I think one
[00:33] of the first questions I got after a presentation was what did you use to make the presentation? Nothing about what I Oh no, year before last. Last year I did it in a spreadsheet. I forgot. Um but let me take that down and
[00:46] we'll get started.
[00:58] that I've learned in a in the process of going through trying to figure out how we're going to handle multimodal at large scale. Um, so some of this is things that were bad ideas and I learned that they were bad ideas. Other things
[01:10] are more interesting ideas that weren't for the research. Um, so I helped create Delta RS. I'm really interested in the storage layer of the stack. I work at a company called Script. There's other
[01:22] things that you kind of don't need to know about. I ride bikes. Uh I do other things, but kind of who cares. So, Script foundationally is a documents company. We have a tremendous amount of content. We have over 400 million,
[01:35] that's the public number that we share, documents that have been uploaded by users over the last 20 years. The other thing is is a 20-y old company, which is why it's called Script. It's from when every startup had no vowels at the end.
[01:47] Um, but the interesting thing about 400 documents from a data scientist perspective or from an ML engineer perspective is that it's billions of pages or trillions of chunks of text or
[01:59] figures or pictures or things like that. And so over the last three or four years, the interest at script has shifted from looking at documents to looking at billions of pages or trillions of chunks and doing lots of
[02:12] really interesting things with those documents which has presented storage and scale challenges. which has led me here. So, uh, multimodal is a good buzz word. Uh, I didn't really know what
[02:24] multimo like mult multimodal is the name that I learned after I was already doing this that that's what people called it. But in the context of like what we would contemporarily consider multimodal data or multimodal learning, think of if
[02:36] you've used Google's Gemini, I think that's the best example, like a frontier model that people really understand. I've got a I've got a PDF. I've got this thing with other types of mixed media inside of it. I want to ask questions of it and I don't want to do any
[02:48] pre-processing. I don't want to generate text. I don't want to extract data. I just want to shove the PDF into the magic box and get answers out. Um, the nice thing about that is from a data scientist standpoint, you just take a
[03:01] PDF and you shove it into the man magic box. The not so nice thing about that is we have a large stack of data that's built around tabular structured data. All of our data management systems are around tabular structured data and we
[03:14] know how to manage tabular structured data and now we have PDFs and images and videos and audio clips and all sorts of other stuff that data scientists care a lot more about. And one of the other things that I found really interesting
[03:27] as I've been supporting teams that are doing workloads around multimodal data is it also dictates changes in compute that we didn't really think about. Spark historically is running on CPUs and if
[03:40] you wanted to do anything GPU based, you would spin up an ML runtime, run an ML node and that was it. Now we have a lot of mixed workloads where we're running to do local CPU, remote GPU, local GPU, local CPU, mixed, so mixed compute
[03:54] runtime environments as well. Um, which sort of adds to the complexity of moving data around to solve the problems we're trying to solve.
[04:09] So I always try to think about the the user. I I mentioned uh to Adam that Anishh here is my biggest customer. I work a lot with Anishh. Anish can say hello. You can talk to him afterwards. Anishh knows lots of smart really smart things that are different than I'm know
[04:21] how to know how to talk about but he's usually who I go to to try to figure out like what are you actually trying to accomplish up here so that way down here in the storage layer I can figure out what you what you need to do. And one of the big things that data scientists and
[04:33] ML engineers need is kind of not to think about multimodal data. They don't care. They don't want to care. If you make them care, it forces them off of the project that they're trying to to solve for. But for me as a storage
[04:46] person, I care deeply about those objects. In in scripts storage architecture, we have over a 100red billion objects in S3 over probably a 100 to 200 different buckets. And when I
[05:00] have data scientists asking to do things with these hundred billion objects or some subset of 100 billion objects, I'm seeing API request calls. I'm seeing slow runtimes. I'm seeing costs going like this. Um, and things that they
[05:12] don't want to care about and I don't want them to care about either. And so if any of you were in the the talk that was before this, there's some interesting stuff that that Data Bricks is is releasing in beta that is
[05:25] touched on here. But one of the things that the the John who is a product manager at data bricks talked about which is something that I've seen is like the first cut that kind of everybody takes is just shove the data in the table. Just if you need PDFs just
[05:37] shove the data into the table. Park and delta both support this binary type. You just shove it in there and call it a day and do your work. That works pretty well for like I would say small big data or big small data
[05:50] depending on how you think about it. Um, but in practice, what I've seen data scientists end up doing is they'll create one and they'll shove data into it and then they'll create another and then they'll create another and then they'll create another to where you end
[06:02] up having multiple tables that all have the same, and I'm going to use air quotes here, same PDF in it. And as a storage person, I'm fairly confident these aren't the same PDF because
[06:15] somebody encoded that from whatever they read in storage into the PDF binary. And so I no longer trust that this PDF is bite forbite equivalent to this PDF. They might be the same content, but I'm
[06:27] not sure. I don't have any way to attest that they are actually the same the same document, right? Which is a problem. It's also super slow because everybody's re-encoding data. Everybody's reloading
[06:39] data. Parquet allows you to I I mean I don't know how many people are really familiar with how Parket works, but you can do you can exclude columns from the read, but if you're shoving a 10 megabyte binary blob into a row, it sort
[06:51] of throws the dynamics of what Parket can optimize for out the window. So, it's very slow and you start to see storage costs balloon as you might expect when you copy data left, right, and center. And there's the risk that somebody makes a mistake and encodes
[07:05] like a compressed version of the PDF or some other you know encrypted version or whatever. And so what evolved at script and what evolves kind of I think at every company
[07:18] is you start to treat where that object lives as the primary key. So you say this this object lives in this bucket and here's the name of that object. One of the things about about this that
[07:31] I
[07:44] that we tell ourselves is that this is unique. If you give me any key in S3, I can think of at least five different ways to refer to that key that are all semantically identical from the S3 perspective. But from the treat this as
[07:57] a this tpple as a primary key is no longer valid. And so depending on how you store this or or how you encode this, you have to bring these like companywide idea of how we refer to
[08:09] objects into conventional wisdom as opposed to encoding it in the data storage. Uh which makes it difficult. And so when I was thinking about this problem earlier this year, I thought, well, what
[08:23] if I just take that responsibility away from the data scientist and I'm just going to shove additional data into the paret file using binary extensions in paret, which I think is a really novel feature that I haven't found a good reason other than this to use for. And I
[08:37] will implement the functionality to normalize the references to those external resources and put it in hidden metadata sitting in the parket file that just lives there and then each row might
[08:49] have an anchor reference that points to that. The cool thing about these extensions that you can do with parquet is if a reader doesn't know how to interpret it, they ignore it. And so for any reader that doesn't know that this
[09:02] anchor resolves to an external location in some form, they'll ignore it. And this is kind of similar to what data bricks is really advocating for in the Apache paret community right now, which
[09:14] is this new file logical type. Different than a physical type, a logical type, it's just this strct. The cool thing about this strct that finally clicked for me today having read the uh the design document previously is this
[09:26] offset allows them to do bin packing of files behind the scenes which is similar to a system that we've deployed at script that I don't have the time to talk about today where instead of having a bunch of say you know 2 kilobyte files
[09:40] what data bricks can do on the back end is put them into one single key that goes into S3 and then encode the offset and the size in that logical file This they're announcing in beta blah blah blah. But the design is actually
[09:53] public. It's part of the Apache going through the Apache Parket project right now and we'll probably see it in the next couple of months start to get out to the rest of the parket ecosystem. This is basically what I was thinking about with Parket anchors. I talked to
[10:06] Barack about this uh uh one of the more senior engineers at data bricks who helped drive this design uh proposal. when I was thinking about parket anchors he was thinking about this and so we've made really great progress in the ecosystem to push that forward but
[10:20] there's still problems with this approach which is why I didn't pursue the like I think file the parking file type I think is going to be useful it doesn't solve my problem um if you lived in the world where everything can sort
[10:33] of route through unity catalog the first item goes away you know credential vendoring gets to be a little bit easier because you just say I have a magic box in the sky that gives me credentials if I need to access this. But the
[10:45] read amplification problem is probably the biggest concern for script. If I have a parquet file that has 100,000 rows and each of those 100,000 rows refers to a different S3 object, then a data scientist is extremely likely, I
[10:58] won't say they will, but extremely likely to accidentally query for all of those 100,000 files and issue 100,000 requests to S3. And what's probably going to stop them is S3 is going to rate limit somebody and something's
[11:11] going to break. And so it's really difficult for me to reason about this sort of refer to external file over there in a large enough scale um without the appropriate guardrails for data
[11:23] scientists. So that wasn't going to work for me. I worked through this. I looked at all the objects that we had and I thought about data scientists doing select stars and things like that and figured I would move on.
[11:36] One of the things that as I've started to really analyze this problem at script is most of the time we don't have tabular data and binary data or like you know these these multimodal artifacts they're joined but they're not always in
[11:49] the same location. I don't I mean maybe script is unique in this regard but our data model or our organizational data model is um organic. It's not very welldesigned or architected from, you
[12:03] know, things start here and then they go into this medallion architecture. TA data sets pop up left and right, especially when we're doing research on we need to do this with this data, we need to do this with this data, and there's not really a clear path on how
[12:15] the development evolves. But what we do see is a lot of data roots back to that sort of quasi primary key of the document in storage. And a lot of the questions that we want to ask are what
[12:28] do we know about this document? It's not I'm starting at embeddings and I'm trying to do things with the document. It's I have a document. What else can I learn? What else do I know? And so in in Delta there's no foreign
[12:42] keys, but we can sort of hack one together if we we really like. But we're really operating with a quasi relational model at script where everything is a document or everything sort of tees off of this document. But we don't have a really good way at a data architecture
[12:55] layer to enforce those relationships. So I was thinking more about like what do we need to what do we need to do? We need to avoid redundant storage if possible. We want to only store one copy
[13:09] of a document. Um the amount of data we have in S3 is in the pabytes. I really don't want to have duplicate copies of pedibytes of data. Um AWS I'm sure would love for us to do that but I don't want to do that. Um, we also have this huge
[13:23] size size variation to where user generated content can be completely variable. So, we'll have some documents that'll be one page where it just says, you know, a niche on the front and that's it. We'll have other documents that are a thousand pages where it's
[13:35] just a on every page and everything in between. The longest document I found at script I actually found because it broke the system that I'm working on, which was about 130,000 pages. And I just decided 128,000 pages should be enough
[13:47] for anybody. And we kept it at 128,000 pages. But there's this huge variability that can come from a storage standpoint that we have to have to consider. And I want to avoid re-encoding data. And so everything for us comes down to PDFs or
[13:59] docxes or powerpoints or you know whatever. So I started thinking of some some different ideas. And I have a bias here certainly uh because I work on the Delta Lake project, but Delta is actually pretty good. Um you may not know this. Uh I'm a big fan of Delta.
[14:12] It's a transaction log. It's a JSON formatted transaction lawn that sits next to your data and it's pretty good. Um, this is an add action. We're adding some data file and then there's some commit info. And it's pretty easy to understand, which is why I'm able to
[14:25] talk about it. Um, but like what if instead of shoving it into the parket file, we just shove it into delta because I have a hammer and everything looks like a delta lake. Um, and so I I
[14:37] started going down that route to where rather than an anchor in in a reference in the parket file, what if I put it in the transaction log? So, I can create a PDF action and then I can put the necessary metadata into that um into that file. Here, I'm going to do
[14:50] something real quick. If you hit this QR code, you'll get all the slides. Sorry for yall that came a little bit later. Um,
[15:03] so one of the things about parquet that or not parquet, excuse me, that delta is is really important is the transaction log contains metadata about the file. Not for funsies, but so you can skip the file unless you care. And these are file
[15:15] stats and it's pretty important. But when we talk about multimodal data, you don't have a schema and you don't really have file stats in the same way. But there's other metadata that a data scientist might use to determine whether or not they care about that document.
[15:28] you know, does it have images? Is it of the English language or other metadata that we know a priori going into the delta table? The cool thing about the delta protocol is it allows you to do this. Readers
[15:41] like pispark or dbr photon will ignore actions in the delta log that they don't understand which is super cool. And so you can build your own reader that understands those and you can layer
[15:53] additional metadata into your Delta table that you might not have otherwise uh been able to do through a traditional Delta writer. Um but it's a kind of a nutso idea. I wouldn't recommend doing
[16:05] it because if you use anything else in the Delta ecosystem that rewrites the table, an optimizer, a vacuum or something that's going to put a checkpoint on that and they don't know what your action is, they're going to drop it. And so it's if you can
[16:19] completely control your stack top to bottom, you can shove all sorts of things in the in the delta log and get away with it, but if somebody comes along and runs a DBR optimize on that table, you've just blown away all your metadata. And so it's a little crazy and
[16:33] maybe don't do it. Um, so I ruled that one out and I started to go down a different direction. I thought this is my craziest idea. Don't ever do this. In the Delta
[16:45] protocol, the path can be a URL and it doesn't have to be parquet. There's nothing in the Delta protocol that says your data files have to be parquet. So you could put ORC or, you know, Voltron
[16:57] or whatever the hell all these different data files are. And so what if you just added the PDF to the Delta transaction log? That would work. The optimize wouldn't blow it away. But the problem is, while it doesn't specify everybody should read parquet,
[17:10] everybody kind of assumes this is paret and they'll try to read your thing as a paret file. But the statistics that are supposed to go into the ad action are sort of your minmax values for columns on a schema. With a PDF, you would sort of have to invent like a quasi schema of
[17:23] what your PDF or other type of data would be and then generate file stats on top of those, which I mean possible. You can do this. This was even
[17:35] too crazy for me. I've done a lot of crazy things with the delta protocol, but this was the most crazy and it's probably a bad idea. All right, so I struck out with parquet anchors or the file type. File type still probably a
[17:47] useful thing. Couldn't shove it into the transaction log, but I kept coming back to this because it is my hammer. Delta is actually really good and having a transaction log next to your data files. You can do a lot with that. One of the
[17:59] cool things that's in Delta is this notion of shallow clones. Shallow clones is not defined in the protocol. What shallow clones does is uses the fact that the ad action can refer to a URL. And so when you do a shallow clone, what
[18:11] you're doing is you're creating a new transaction log of a bunch of URL references to the other table's data. And then when you write to your shallow clone, it creates an add action with a relative path to your parquet file. So
[18:24] you can shove URLs into there. And at the end of the day, it's just JSON, which is I mean, I can work with JSON. That fits in my brain. And so I started working on this idea called
[18:36] virtual delta tables. Um, which hopefully brings you here today. Um, there's some awful code, and I do mean it. It is awful. This is public open source code. If you just go to rtylervd, you can look at all the the rust code
[18:49] that I wrote for for this. But a lot of it relies on this really great crate called S3S. The really cool thing about S3S is it makes it easy to build an S3 alike service that quacks like an S3 and
[19:02] walks like an S3, but is not actually S3. And so I I knew that I had this crate for another project and I knew that I could make something that looks like an S3 and I knew that I can I can generate JSON like I can build a restful
[19:14] JSON interface. I thought can I fake a delta log? Can I fake data files? Can I fake everything about a delta table? Can I pretend to make a delta table but not actually have a delta table? Turns out
[19:26] you can. Um, one of the nice things about Delta Lake, it's good actually, is it gracefully degrades. So all of the things like CC files, checkpoints, all of these other things don't have to actually be there.
[19:39] You don't have to implement the whole complexity of what delta is if you just have JSON files that are the transaction log. And so you can build a very primitive Delta table. It might not be as optimized, but you can do that just with slinging JSON over HTTP.
[19:53] And if you're going to fake the transaction logs, you might as well fake the data, too. Uh with the parquet crate, it's very easy to generate or you know, PI arrow anything. you can stream parquet over the wire very very easily. And so you can fake the data data as
[20:07] well. And so I I armed with this knowledge uh I decided to go down this route of like what would it look like if I'm going to fake data and present pretend delta tables to data bricks or
[20:20] to you know the data bricks platform. I figured I would need a connector interface and I would need to like distill what are commonalities that we have against different types of data sources that we would have coming in at script. And I came up with a a
[20:34] relatively simple uh trait in in VDT. We always need a schema delta. Even if you don't have structured data, you need to pretend you have structured data and give it a schema. Um we need to compute
[20:47] statistics. We need a row count and we need some shards. And shards are an idea that is not in Delta. But if you're trying to bring any data into Delta, you don't want to put just one big parquet file and say good luck. Have fun. Right?
[21:00] So the connector is in charge of deciding how to break up the data. If you're going to fabricate a bunch of paret files, how many paret files should be fabricated? What should be in those paret files? You need to decide how many
[21:13] of those you're going to do and then present statistics up so that a fabricated delta log can present that to a spark reader or a duct DB or what have you in order to retrieve that data. And
[21:25] so I started implementing this with Postgres because Postgres is my favorite aside from it's my second favorite tool. Delta would be my first favorite. Postgres is my second favorite. And it worked the first time. It turns out it's
[21:38] fairly easy to generate statistics and shards of data from a Postgress table of a reasonable size. And I had this realization that I implemented Lakehouse Federation. Um, and so I felt really proud of myself that I implemented
[21:50] something data bricks announced three years ago. Um, but if you're not familiar with Lakehouse Federation, it is closed source. So if they open source it, then we're then we're in business. But it basically allows you to bring BigQuery, Postgres, MySQL, Snowflake,
[22:02] wire that through to the the catalog level, and it pretends it looks and quacks like a duck, but it's not actually a duck. But I accidentally implemented that on on on my way. So,
[22:14] oops. But anyways, back to the multimodal data thing, right? We have a lot of tabular data and really all we're asking is what do I
[22:26] what do we know in the world? what in the ecosystem of everything what do we know about this key right and so Delta Lake is really good um as I've mentioned and another feature that it has is called column mapping and one of the
[22:38] really useful things about column mapping is it allows you to separate the logical schema of the table from the physical schema of the parquet files this was a a feature that always bugged the out of me with delta in that
[22:50] your parquet files could be a different schema than your delta table turns out that's useful actually Because what we're able to do is if we take a bunch of different delta tables that have
[23:02] parquet files with a physical storage of you know primary key username or primary key feature one feature two blah blah blah blah the paret files have these schemas. With column mapping we can just map those into one big stupid table and
[23:16] what you see as a spark client coming in is this column name but what is actually at the physical storage layer is that the component part. And so aggregated or virtual aggregated tables was sort of
[23:29] another step along the journey of like how do I pull as much information about this document together and present it to an end user. And so column mapping actually does the trick. And it does the trick so well that it also was very simple to implement. Um
[23:43] you could say I in invented views which is also not something that's new or novel but I did it with Rust so it's better or whatever right? Um, the cool thing about generating this as
[23:56] opposed to building a view is if you're if you're generating this, you can bring in other information. And so one of the things I'm curious how the data bricks team is going to solve this with the file type. I need to know all of the
[24:08] data associated with this resource with this file. I don't know that necessarily at any given point in time. I don't want to maintain a YAML file that tells me this data is here, here, here, here, here, here, here. And so if you can walk
[24:22] through Unity catalog or if you can compute this model to find where those references are, you can generate these sorts of mappings with with relativities, you can technically generate views. And there's other ways to to solve this problem. But there's a
[24:35] lot of useful things that you can get out of just column mapping and generating the delta log. And so for me, I'm still trying I'm still pursuing this like white whale of like how do I know everything about this
[24:48] document across the across the infrastructure, right? And so one of the cool things about the when you have the component parts of column mapping the URL references and you stitch that together with this connector interface
[25:00] and you have things that look like delta tables at your disposal is now it's not just delta tables that I can start to stitch together. It's anything that I can present as a delta table. So if I can wire Postgres up to present like a
[25:13] delta table, I can aggregate on top of that. If I can take other data out of Google Drive or or wherever and present that as a delta table, like kind of anything that I can implement a connector interface in front of, I can not only present as a delta table, but
[25:26] then I can wire that into other things that present like delta tables. And it's just like, you know, virtual delta tables all the way down. Um, I don't know if that's a good idea, but it's possible. And uh it was really
[25:38] interesting to figure out how to do that. One of the big challenges though is predicate push down with delta like we've got to find some way to like the connector interface the statistics implementation is probably the hardest part because with something like un a
[25:52] bucket of unstructured PDFs how do you generate statistics for that so you don't go read every anytime a client comes in and reads the delta table how do we avoid reading every PDF file and so there is a little bit of the devils
[26:04] in the details with the connector interface and how you compute statistics and schema Um, one of the things that's really novel about this and and you may have seen the open sharing rebranding of delta
[26:17] sharing, they're extending delta sharing a little bit. Like I I didn't I never quite understood why delta sharing existed, but I didn't have a tool to say why should delta sharing exists or why should it exist? I kind of have that tool now because I can just shove this
[26:30] thing that looks and talks like delta into any other service that I have. And thinking about this at script rather than pulling how do I pull all delta data or all data into delta or all data
[26:42] into the data platform I can now invert that thought to say who all has data in the world and how do I express delta from everybody individually and pull that into the data platform. I haven't
[26:54] followed this idea to its logical conclusion. We'll see how it goes. Um, but if you can talk HTTP, if you can spit JSON and encode parquet, you can present as a delta table into your into your service infrastructure.
[27:08] And so, let me check my time. Um, as I went through this and as I got even further, I thought, well, why does the data have to exist to begin with, which
[27:20] I mean, we're like deep in crazy town here. Um, why does the data have to exist in the first place? What if again script hundreds of millions of documents lots of lots of data if a data scientist or somebody has to go generate a whole
[27:33] host of data before they need to do something with it. It adds that in important first step which probably should exist but then their project is generate a bunch of data then analyze a
[27:46] bunch of data and then maybe success you know question mark question mark question mark profit. So what if I I just could like not do that part? What if we can just do that part? And so these ideas are are really halfbaked. This is something that Anish and I have
[27:58] been working on. Uh so we can talk to the end of the day about it. But with this connector interface, I now have detached data from having to exist a priori to where we started building a
[28:11] system where we can compute large amounts of data really really quickly to where the park file doesn't have to exist. The data doesn't have to exist. we can generate it on demand. And so if you
[28:23] have a if a if you have a query that's coming in that says I need embeddings generated for these you know these rows we can kick I need this data from this thing out to this intermediate layer
[28:36] flash and detach that from spark which for us is really powerful because one of the key problems that we have is we receive uploads there's thousands of uploads I I don't remember the exact number of it's between you know a
[28:49] thousand and 100 thousand a day I can't remember where between it But all these new documents are coming in every single day and we have to process them and generate the same sort of artifacts associated with each of those documents as we might have done in a in a Spark
[29:01] batch run. And so now we have code living in two places. And so we're trying to pull that code together to where we've got a compute layer that's distinct from our data processing layer so that if a document comes in and we
[29:13] upload it, we can generate embeddings there. Or if we backfill from Spark, we can generate it there. And so if I assume that I can do all of these things like at runtime, I can describe here's
[29:25] what here's how you turn this PDF into text, which is a very common task. Everybody has to do it. We have probably between five and 10 different implementations across script for take PDF, give me text. If I pull that into a
[29:38] compute layer and I can just hit this compute layer, give me here's PDF, give me text, then everybody can build around that and we can build VDT around that to where a data scientist doesn't need to
[29:51] have the data to exist at the the beginning of it. They just need it to be generatable. Um, and the really cool thing about this approach of compute data at runtime as opposed to generate
[30:03] data is you can have sort of a I always forget is it a read through cache or a write through cache but you somebody's reading and somebody's writing the data um when it's needed as opposed to generating that data for every single
[30:16] document that's under the sun because one of the other challenges data scientists have is how do you pick the right subset of data to generate? I don't know. Uh, every, you know, all Portuguese documents that were uploaded in 2020, like who knows? Well, like why
[30:29] do you choose that? Um, well, because somebody else chose that. And if you just detach that to where they can go query that data and that data gets generated of course the first time you generate it, second time you you hit
[30:41] storage and we present that virtual delta layer, then everybody can benefit from that, you know, quoteunquote knowledge that's being gleaned from these documents as it's being computed without us needing to think upfront too much about all of the data that we need
[30:54] for every single document or which documents have which subsets of data. we can stitch that all into here's my document key and here's everything that we know at script about it and that's the ultimate goal of the journey for me
[31:06] is not to shove PDFs into the data lake I don't want to do that I want to answer the question how do how does a data scientist how does a product manager learn everything script has ever learned about this document in a way that's
[31:18] queryable and understandable in the datari platform which is why I really focused on delta and and why I wanted to present things as delta them because I don't have the time. I don't have the team to build an
[31:30] interface. I don't have the team to build a query engine. I don't have the team to build all of these things. I have a storage team. And if we can present data as delta up, then everything in the data bricks ecosystem and a lot of things outside of the data
[31:42] bricks ecosystem now have access to that data and now can do interesting things with that data. And so to me, one of the coolest things about Delta being specified the way that it is, but also fairly simple is Delta itself can become
[31:56] an API for the data stack on top of it. And that's the direction that I've been going. Um, and I've done a fair bit of research on this. I've built a lot of really bad ideas, but to
[32:09] me the big thing for that I've learned over the let's say a year or so is that Delta can be the API in a way that I hadn't previously really considered until I went down this path of how far
[32:21] can I push it before things start to break. Um, there's a whole lot more stuff that we do at script that is interesting on text.script.com everything that we can publish. I'm here until 5 today. I can
[32:35] answer any questions that anybody has about anything script related. Some of the Delta stuff as well. Um, but thank you all for listening. That's it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.