Managing Data at Exabyte Scale for AI Model Training

Summary

Enterprise AI model post-training requires a fast data flywheel, but current infrastructure forces teams to copy data across single-purpose tools for each step—exploration, curation, feature engineering, and training—slowing iteration.
LanceDB provides a unified table that manages multimodal training data through the entire path from exploration to GPU loading, with double-digit millisecond analytics and single-digit millisecond vector search at ten-billion-scale.
With 80% of Anthropic's production code now reported to be written by Claude, AI systems are increasingly driving the need for high-quality, rapidly iterable training data pipelines, making data flywheel speed the key bottleneck.

Managing Data at Exabyte Scale for AI Model Training

More and more enterprises are post-training AI models to be custom tailored to their domain and users. Success in model training depends critically on establishing the right data flywheel to improve iteration speed. From exploration to curation, from feature engineering to training, current data infrastructure solutions are insufficient to support the full iteration loop. Model development teams wrestle with a hodgepodge of single-purpose tools, copying data across systems for different workflow steps. This is a key reason why so many enterprise AI model training efforts are delayed or fail. In this talk, we’ll cover how LanceDB solves this by managing all of your multimodal training data in a unified table and streamlining the entire exploration to GPU-loading path. Through customer case studies, we’ll outline this new architecture, practical recipes for success and how it all fits into your existing enterprise data architecture.
Talk By: Chang She, CEO / Co-founder, LanceDB ; Jack Ye, Software Engineer, LanceDB ;

Chapters

FAQs

What problem does LanceDB solve for AI model training?

Enterprise teams building custom AI models must copy data across multiple single-purpose tools for exploration, curation, feature engineering, and training, slowing the iteration cycle. LanceDB replaces this fragmented stack with a single unified table that covers the entire path from raw data to GPU loading.

What performance does LanceDB offer at scale?

LanceDB delivers double-digit millisecond analytics and single-digit millisecond vector search at ten-billion-scale for multimodal training data. This performance profile supports the fast iteration cycles required by enterprise post-training workflows.

Why is data flywheel speed the key bottleneck for AI model training?

As AI agents automate more of the execution layer—such as code writing and task completion—the bottleneck shifts to data quality and iteration speed. The faster a team can move from raw data to a trained model and back, the better the model becomes over successive training cycles.

How does LanceDB fit into an enterprise data architecture?

LanceDB acts as a lakehouse-compatible format that unifies analytics, vector search, and model training on the same underlying data. This eliminates the need to maintain separate systems for each workflow step and keeps all training data accessible from a single governed location.

Full transcript

[00:07] Thanks everyone for coming. Hope you guys all had a great lunch. And uh this talk is about managing data at Exa by scale for AI model training. And uh my name is Jack. I work for Lens DB. Uh this talk is supposed to be me and uh my
[00:20] boss who is Chunure, but he is unfortunately sick today. So I'm going to just give the talk by myself. And uh a little bit about me, I'm a software engineer at Lens DB. And uh I work a lot in open source. I am a PNC
[00:32] member for lens and also a partial iceberg and polaris. Um and previously I was in AWS leading a few products including SageMaker Lakehouse, S3 tables, EML and Athena and I live in Seattle for anyone who is also coming
[00:45] from there. So uh I want to start the talk with this topic about AI is starting to build itself. This is from a recent blog from anthropic about recursive self-improvements that is um trying to
[00:59] say that okay today autonomous coding and research tasks execution are done by agents one more and the next phase of that is really component efficiency and humans setting a direction of what to do and the agent just go to do that and in
[01:12] the future we don't know when the AI system is going to be designed itself and just self-improving itself all the time and uh the evidence is actually already here if you see that uh from the blog they published 80% of the
[01:26] production code is written by cloud and uh six times more code is shipped per engineer per quarter versus like in the past few years and every four months you can see that the agent is going to do more things autonomously without human
[01:40] interactions. So when the execution cost almost nothing the research taste is becoming the bottleneck. We also have this saying that genius is 100% inspiration and 99%
[01:52] the perspiration but now the perspiration is actually being automated fully and the 1% is actually what is important here and uh the same is shifting also hitting the enterprise AI because enterprise post training models
[02:06] to their own domain users and success depends on the data flywheel right how fast you can iterate on the raw data to produce a better model from exploration to curation to feature engineering
[02:18] pre-training to training to post- training and complete the whole loop as fast as possible. The the more you can between train and complete the cycle, the better your model is really going to be become. And the higher quality your data is, the better the cycle is going
[02:30] to produce the output. And within this whole loop, we can see more and more about agents driving the whole entire loop. Right? From the whole process starting from exploration to the post training, the AI agents are really
[02:42] becoming execution at every single step, which is what we actually see when we interacting with different frontier labs. And uh then the researchers are more and more just sitting at the direction to make key deci sitting at the position to make key decisions and
[02:55] then they just tell agents okay you do this you do that and everything is kind of being more and more automated and uh today the key problem we see when we interacting with these kind of AI researchers is that the data layer
[03:07] becomes the one that is fighting against you when you are one more uh devoting into this kind of model because um different there are all these kind of singlepurpose tools with data copied
[03:19] between different systems at every single step and uh I know this year the lake base is a big thing for data and AI summit and this is kind of after that right you you have even you have data coming from the lakebase into your lakehouse and after that for all these
[03:32] trainings there are all these kind of tools that are specially designed for different purposes like the ETL engines to get data ingested and then the feature stores the vector databases the data training loaders in PyTorch all these things still require you to move
[03:45] data all around and then making the system one and more complicated and uh and what we are uh starting to realize is as we are working with these customers is that really the most impress efficient way possible is you
[03:58] have one table for the entire loop. You have this gigantic table that can do everything from starting start to the finish line. And uh here are like these six different things that I will dive deeper into more details that we think
[04:10] are more the most useful when uh based on the customer feedback. And uh at the lens DB we have been starting to develop a lens format for these purposes and uh at this point uh I joined last year and
[04:23] uh we are showing a similar style last year. This is we are going to power the future of multimodal AI but uh I think at this year we are very confident in saying that we are powering the future of multimodal AI uh with all the
[04:35] companies the frontier labs the the enterprise AI teams that are using lens to build all these systems and uh we we have a pretty good coverage here from just gener generic frontier labs like runway majourney Microsoft AI what labs
[04:50] um two enterprise companies like Netflix uh Uber all these and the ones training autonomous vehicles like Aurora or the physical AI companies like from Nvidia to apply intuition. We have a very good
[05:02] coverage of different companies and this is just a part of that. Uh and most recently I think there are two news coming uh last last month. The first one is the media cosmos 3 that is trained on
[05:14] the on top of lens uh for their physical AI model and the other one is the W model from Yaniccom that is also trained on lens. So uh this talk is really going to be uh what we learned uh as a part of
[05:29] building these data infrastructures for these uh very big research labs and uh uh why they kind of choose lens and what is lens powering them for uh the first
[05:41] part uh I kind of put on the top uh it it wasn't the first one but uh given the context of this conference I put it the first one which that is branching um and uh we we kind of can imagine why branching is so helpful for an agentic
[05:54] or AI based research uh for data training because every single agent can really work on its own isolated branch and this not just apply to a single researcher that is going to run multiple agents and also just many researchers
[06:07] that are going to work together. You want to work on isolated work streams and experiment without the risk of production uh impacting productions and uh uh also you want traceability and lineage come and that is built on it and
[06:20] uh you want to keep what is work and you want to discard what doesn't work uh you want to iterate quickly you want to do merge and rebase so this is kind of um for example we know for data bricks there's the leg base that can do branching and uh this uh this is a uh
[06:35] kind of working beyond that because we are able to even doing the merge back and the the rebase is to keep helping the researchers to iterate on different things. uh so before Lance uh as I said I was
[06:47] mainly working on Apache iceberg and uh a few years back I authored the initial implementation for the iceberg branching and uh that has gained a little bit of traction from the old lab side but uh it was uh kind of wasn't super successful
[07:00] when we are trying to push that for ML AI experimentation and there are lots of lessons learned from that uh one of the biggest issues is that uh in the iceberg model all all the all the branch information are kept in the
[07:13] uh at a single point and every time you update something in a single branch that's actually affecting the the production tables root metadata and that is actually not good ideal for the branching use case that especially for
[07:25] agents because you are impacting the other other people along the way. So the other agents along the way and and then um the delta shadow cone is another alternative to isolate branching. Um and
[07:37] the benefit of that is you are having a fully isolated copy. Um but the problem of that when we are trying to push that model to researchers is that it kind of lose the traceability to the source. Of course you can trace across tables but
[07:50] that is not the mental model when you are thinking about for example I'm using git for code right. um you want the same things related tables to be under the same context and also it is just much less intuitive. So, so when I got the
[08:03] chance to redesign that for lens, essentially what we did is branching but using shadow clone so that you kind of get the best of the both worlds that uh you have the branching semantics with your table and at the same time the the
[08:15] branches are fully isolated. they don't touch each other and uh uh that is ideal for parallel working into the same data set and you can kind of see how that matches to whatever you are doing with git today
[08:28] in the code you can map to whatever operation that is in uh in the lens branching semantics and uh typically this is how things work right uh there are agents that starting at different point in time at a at a
[08:42] lens table the last table is always kind of the main branch from that perspective And then you can develop uh different features do different commits on your own branch. Um and then there is a merge part. Uh the merge part essentially I
[08:54] will take uh whatever I have been working on the branch and then I will merge back to the uh source br source main branch. um that is achievable because uh lens uh have every single
[09:06] type of transaction defined so that you know exactly if there are new things coming to the main branch I know how I can kind of merge this transaction into that branch safely and if there's for example no way that that can rebase the
[09:19] conflict then of course you cannot do that or you need some manual interaction just like how if you you have code conflicts then you have to do manual interaction but usually for the AI researchers workflow what we realize is that most of the operations are very
[09:31] mergeable And that also means most operations are very rebasable. Uh for example, here is a example here that you can imagine two different agents working on two different branches that are forked at different points points in time and I
[09:44] can actually uh and and one might be merged back into May and I I can actually rebase the other um the other branch into based on that um that that that updated main branch so that I I
[09:57] kind of have the updated version of the code updated version of the data that I can work with. Um so and and a very typical uh uh example of that is for example I have a branch that is
[10:09] appending a new feature column or appending a new embedding column and at the same time there are new data coming to table and then that is a typical rebase situation you might want to do so that uh I will rebase with the new data and then I can still work on that uh new
[10:22] new new embedding column. uh all these are already built in uh if you use if you do pivot lens DB um that can you can connect to a table and then you can do all the branch operations today and that is many uh what many
[10:35] frontier snaps are leveraging to scale out their agentic uh research. Uh the second part uh is about multimodel blobs and also what uh is really about why we are talking about exabyte level uh AI
[10:49] model training because uh this is why everything today is xabyte um because in the past when you say I have xabytes of data there's a crazy number of rows you need to you need to satisfy that um
[11:01] especially if it's a single table but uh when it's becoming multimodel with all the models that are popularizing that are making the images, videos, documents much more and more useful. Uh people are
[11:13] putting all these as a part of their table, right? And uh I think if you put just one megabyte each row, it will reach one exabyte with one trillion rows. And uh typically if you put images, videos or in the same column
[11:26] then you just need a few billion rows to actually reach xabyte. So uh this is getting increasing and increasing in the scale that we are working with and uh so far what we've realized is that blobs
[11:38] are always treated as this secondass citizens in all the data infrastructures around uh uh around the domain. uh right you have to uh typically what people do is either you store that using things
[11:50] like web data set which is not everyone knows it's not efficient but people still have to use that or you you put that into things like a pocket file with a pointer to a storage and then you have to build a lot of complications into
[12:02] managing that and uh um uh last uh two years ago actually lens developed what we call blob v1 um that is uh so that we realize that for training actually most
[12:15] of the training data you have to resize that into a smaller size anyway. So why not just inline everything but we'll do a very efficient way to inline that. So so so although you are storing the blobs
[12:27] within the same file but you are going to get much better performance when you are doing doing training and you are trying to seek different uh blobs. uh that is the V1 but um the problem is that uh well you not not not all the uh
[12:41] blobs are of the same size right people people want to actually store the raw data also along the way when you are storing just training the the resize training image so the blob v2 is really about decoupling expressions from
[12:54] storage so that you can actually have different types of uh blobs all stored in the same table in the same file uh while we are providing the abstraction that allows s you to do all the reads and writes and we will automatically
[13:07] make the layout optimal. Uh the way we do that is using uh different storage semantics behind the scene. So there are four four semantics we develop. The first one is called uh inline that is
[13:20] essentially the same as blob v1. You just store everything in line. But uh another way we usually say that is out of line because what happens is that every time you store the image or block or video it will store a um it will
[13:34] store a starting position and a size and the actual blobs are stored continuously one by one with each other so that you can actually seek across different blobs very efficiently. And uh once it cross a threshold which we by default uh this is
[13:48] tuned by a few customers we work with uh uh you you get into the packed word that is the the blobs are stored outside of the same file but uh multiple blocks are still stored together so that you can
[14:00] still do efficient seek and uh it doesn't make a single file super big and and then once you reached a much bigger size which is our threshold is 4 megabyte by default uh it goes into the dedicated mode kind of like you have a
[14:13] pointer to a file but the file is fully managed internally within the table format so that uh every all the life cycles are managed by that directly and then you you always have the external case that is I already have a bucket of
[14:26] blobs uh and can I just link to that and the answer is of course yes but uh what we do is we added this concept in lens called uh base uh and every time you add a additional storage you can register
[14:40] that as a base into the into the table and then starting at that point um the the the files in that base will also be managed by the table. So that um it always goes through the life cycle and
[14:52] uh it always goes through one API u no matter you are using for example uh this is this is I think if you are using python lens SDK uh you do this operation called take blob that is I I have this block column and then I I want to take
[15:05] rows 0 one two three right uh this is interface that is exposed to the user and behind the scene you don't care about how the data is stored u but it will always give you the most efficient way to access that um and uh and and and
[15:19] then it is governed just like any other columns. So that you don't need to think about okay this is special blob column and I have to handle that differently because that's a blob comment. It is exactly the same as any other columns. Uh so the GC will clean up the often
[15:33] often files along the way. You don't have to do a mapping between oh I have this these pointers in this table and I have these files in this prefix and I do I have to clean up the often files. You don't have to do that. Everything is
[15:45] managed together uh within the same table. Yeah. So uh the third part is about curation ETL. Uh this is probably not uh something that is surprising to anyone um that lens is a lakeos uh is a
[15:59] part of the lakehouse. So you you are able to create at xabi scale with uh different compute engines. um whatever engine you like we or support just like iceberg like delta you have sparkle
[16:13] flame gray duct TV data fusion all these engines that can connect uh also through different cataloges uh the unity catalog polaris blue hive rest or even a directory you can connect to that and uh
[16:25] what we added is an abstraction called last name space um that kind of extracts out the differences between different catalogs so that the engines can connect to those through the same interface and then behind the scene talk to the lens
[16:37] tables and uh since we are at uh this summit I'm of course going to talk about spark u and the spark is one of the most popular engines we also use and uh many customers like uh it does able to
[16:50] deliver large scale etls into lens um all the operations like insert update delete are supported um and uh there are also people who want to do analytic SQL through that and actually there's an
[17:03] interesting benchmark we recently did with uh versus iceber pocket This is not to say iceberg is slow but this is just to say uh compared to open source spark somehow we are even faster um than iceberg park. I think the the main
[17:16] reason is because uh for for lens the way we integrate with these comput engines is through the rust binding and uh well in the rust binding layer uh we essentially use data fusion to run uh
[17:30] most of the uh the scan operations behind the scene. So, so technically when you use the spark lens connector, you essentially get uh data fusion comet integration out of the box although you
[17:42] are not using comet if you know what I'm talking about. So uh that is one of the cool thing about this integration and you can also see for example if you have a more efficient park uh reader for example in trino then the difference is not that big but overall people are
[17:55] pretty satisfied uh with using even lens for just OLAP operations. Um and then the part the the fourth part uh which is one of the highlight the biggest highlight that data engineers and the AI researchers like is about
[18:08] feature engineering about lens. Uh because AI really needs to derive data. Uh you have to keep iterating on features. Uh in the past when we say feature engineering is like okay I
[18:21] append I I kind of design this score and I will derive a score from this column. And uh today when we talk about feature engineering the context is much broader. It's like I will run through GPU and add some embeddings and then I I will run
[18:34] through 10 10 different models and create 10 different embeddings right and then everything is stored ideally stored in the same same table uh and also you you uh if you have image I will say I will run through this model and then generate the caption and the caption
[18:47] will generate embedding so things keeps going on right so the feature engineering doesn't go away it gets even more and more complicated and at much bigger scale um that is one why we kind of divide is 2D storage uh for lens um
[19:01] that you can add columns uh with back view almost for free. I think uh one of the things that people keep talking about about uh open table format is that you can do schema evolution for free but
[19:13] that is only for you do a schema evolution of for example add a column and then you add data to the column and then the new data will have the new column right all the old data will have null essentially but in lens what we
[19:25] call as data evolution that is you will directly pend data files into the table and everything will just work um so it's essentially a schema evolution with auto backfill and it is still a
[19:38] metadata only commit. Um there are zero copy and it takes seconds not hours to do that. Uh this is uh an example of how you are going to do that uh with less DB. So you
[19:50] will annotate for example a function with UDF and you will say this data type is let's say a float and uh this is yeah I think this is an example of generating embedding. So you are just return running this through a model and the
[20:03] model will encode the image and then what you do is you just do table add column and that's it and it will do you can and you can run table backfield and it will back field for you behind the scene and this can run uh on CPU on GPU
[20:16] um in different use cases and also powered by different engines. You can run that in spark you can run in ra in whatever engine that is that your data data engineering or AI research team is using.
[20:30] uh and after the feature feature engineering then what we enter is the so-called uh actual training phase right this is when you are actually feeding the data into the GPU and uh trying to get a good model out of that you you can
[20:43] be a frontier model that is developing the latest larger language model you can be a enterprise training team that is training a customization on top of a large language model you can do all sorts of things and this also all feels
[20:55] in that domain and this is when GPU is very important And we are a typical sentence we say is if the GPU isn't busy then data loading is your problem because you are not really saturating
[21:07] everything um well enough and uh um what data loading really needs uh is uh very different from use cases such as OLAP uh
[21:19] because what uh it will go through a few phases for example shuffle will randomize you want data to be randomized to avoid a lend order right and uh you want to do split and subset So that you can do that in training and eval and you
[21:31] want to uh every time you want to sample just a very small slice of that and then you want to distribute to many servers massively parallel and each have its own shared and you want to transform tokenize decode and normalize uh so on
[21:45] the right there's a this uh very very simplified example for example I have a row table with radies 0 to five and after shuffle I will have a random model of that and uh then I would take the
[21:57] split and set and distribute. Uh and in that case you can see that why random access is so important for this use case because in the end uh you want to random access a very specific number of rows in
[22:10] a table and you don't want to do that up front, right? That will be crazy to transfer all those data after you read that. uh you you you you all want to do that uh whenever you actually need the data and uh and and then the traditional
[22:24] solutions in the lakehouse like park is really bad at that uh uh because you have to read megabytes to get just one row and uh I think this is already well known at this point that lens is much better at random access um
[22:39] uh so what we can guarantee with lens is that for whatever that is a fixed size field uh either it is a scalar field or it is for example embedding that is a fixed size list. Uh you will only do one
[22:51] IO for for that kind of data and for variable lens field you can do or we'll do two IO's. Um and and it is optimized for object storage. Um but it is not just optimized for
[23:03] object storage because uh the ideal case for uh training such as on PyTorch is that you you want to do memory mapping as much as possible and uh um but on the other side from the storage form
[23:16] perspective you want to do uh encoding such as compressions as much as possible to save space. So uh for example, park is on the extreme end of saving space because it has all the con compressions
[23:28] and encodings to make sure uh things are very small. And uh the uh the thing lens is trying to answer is can we do that while still making it as memory efficient as efficient as it is to do
[23:41] memory mapping. And then I think we have reached that point and that is also one of the key things that every time we talk to a AI researcher they um especially the people who are doing the PyTorch training that is what they are
[23:53] looking at and what where they like lens and a lot of that is because we are what we call arrow native because what we do is that everything is in and out using arrow and arrow is already memory mapped and what we do uh is try to make lens as
[24:07] memory mappable as possible to the arrow inmemory data format while still having as many compression as possible. So the last part is after you train the data you have the post training. Um I I
[24:21] wrote here about rag and search but it's not just about rag and search. Uh I just I think I kind of take the advantage of that to talk about search for lens uh because that is a part of the key feature that is enabled but uh but
[24:33] search is uh and retrieval is a part important part of post training. You do all sorts kind of different things uh like evaluation, hard active mining, ddup all these kind of things need retrieval at scale and uh that is when
[24:46] people starting to load different data into different systems like open search or melas, quadrant, pine cone all these back databases or those search search engines and and then people uh people realize actually can I do that with the
[24:59] data on object storage and the answer is yes you can uh with lens um and uh that is powered by index index and index are just lens files uh under the same lens table. So essentially what we do is that
[25:12] we use the file format to power the index and uh then we use a table format to guarantee the transactionality for updating and using the index so that you get uh the index embedded into the whole
[25:24] system in this way. And as of today we have already supported a very wide range of indexes for different kind of search use cases. And for the vector index we have basically all the major uh index
[25:37] types supported. Um and for FU tax we also have uh inverted the latest one recently Microsoft contributed for the FM index and uh we support JSON tokenizer multilingual tokenizer all
[25:50] these kind of things and uh uh scale index is also a interesting part that more and more people are uh contributing actually I just had a call in the morning with Roblex and they are uh they are using B2 a lot for their random
[26:02] access for just reading key value store using lens u which is super cool um but um we are adding more and we have already added quite a few um that I think almost satisfies most of the use cases. Um and what what people do with
[26:17] that is that you can essentially now do hybrid search in one query with uh lens tables uh that is staying object storage uh and it can still be pretty efficient. You have you can you can do a vector
[26:30] search combined with futexp 25 with a metadata filter that is filtering against a scalar index and combine all those results into a ranked score uh right for improving your retrieval and
[26:42] this cannot just be done in one engine it can actually be done through multiple engines we just added the spark vector search and the full x support uh if you are interested to know more about that uh there's a talk uh another talk I have
[26:55] tomorrow if you search my name you will be able to find a Yeah. Um so this and this is the most important thing is this is the whole data that you are trained on. You are you you started curation with the same data. You start uh you add features of
[27:09] the same data. You train with the same data and then now you are searching and post training with the same data and uh uh from performance perspective typically uh well I guess uh from LDB
[27:21] enterprise perspective uh most of the queries run around double digit milliseconds. Um some of that more complicated ones around three three number millisecond latency. Uh for for
[27:33] vector search we are able to push much better uh by using lot lots of fancy stuff like gravity q. Uh we are able to push uh single digit millisecond latency at this point and it is not just at a
[27:45] small table it is at uh at least 10 billion scale that we have tested. uh yeah so everything so far I have talked about is well on one table for the entire uh flat well for training uh
[27:58] AI model from the starting to the end and uh uh and and with this I think what we are seeing that is we we are evol we have evolved lens in order to satisfy this requirement into a so-called
[28:10] lakehouse format it is not just a table format it's not just a file format it has also this whole new index format thing and also we support all the catalog backs through the length base format. Uh so everything is bundled
[28:22] together so that we can we are able to say that whatever use case should live in the right level and we kind of design everything together to integrate smooth smoothly and uh uh there's no locking
[28:34] there everything is open source and it's cleanly decoupled every single layer. Uh oh by the way um this is we we recently published a not a paper but a workshop paper I guess this is called the lake
[28:46] land cost format. it is in the format uh uh sub subconference of the sigmo conference this year. So there's a paper that people can find and read more. Um and yeah uh from that what we say lens
[29:00] DB uh what we offer to the customer is the so-called multimodal lake layhouse right uh you can of course do whatever lake cost can do that is analytics ETL but also you can do semantics and hybrid search you can do model training
[29:12] everything truly combined end to end with a single format that is staying on top of object storage and uh and and just a recap we we talk about branching and uh unified blobs and curate in place
[29:26] and Then we uh from pre-training which is back to features to training on GPU to post- trainining with index and retrieval everything using a single solution. Yeah that is what we do for Xi
[29:38] training for with lens DB. Thanks everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.