Lance and Apache Spark for semantic vector search at scale

Summary

Exa uses Lance with Apache Spark Structured Streaming to index tens of billions of web pages for semantic search, serving AI agents and enterprise customers including Cursor, Notion, and HubSpot.
The production pipeline orchestrates web crawling, deduplication via Simhashes, GPU-powered embeddings at 10,000 rows per second, and vector indexing using Spark and Delta Lake.
Lance's column fragments enable zero-copy backfills and row-level reads for low-latency retrieval directly from S3, with an emerging memo wall specification supporting streaming upserts via LSM trees.

Lance and Apache Spark for semantic vector search at scale

Lance is an open lakehouse format purpose-built for vectors, multimodal data, and fast random access. Exa uses Lance with Spark Structured Streaming to power semantic web search, processing billions of documents for AI agents while maintaining data integrity and retrieval performance.
Learn how Exa's production pipeline orchestrates web crawling, deduplication via Simhashes, GPU embeddings, and vector indexing using Spark and Delta Lake. Discover Lance's innovations: column fragments enabling zero-copy backfills, dense embedding compression, and row-level reads for low-latency retrieval from S3. Explore how Spark Structured Streaming integrates with Lance table functions for full-text and vector search, plus the emerging memo wall specification for streaming upserts. this video reveals architectural patterns and open source technologies powering semantic retrieval in modern AI applications.
📂 GitHub repository: https://github.com/lancedb/lance
🤝

Chapters

FAQs

What is Lance and why is it used for vector search at scale?

Lance is an open lakehouse format purpose-built for vectors, multimodal data, and fast random access. Exa chose it because column fragments enable zero-copy backfills, dense embedding compression reduces storage costs, and row-level reads support low-latency retrieval directly from S3 without scanning entire files.

How does Exa's semantic web search pipeline work?

Exa's pipeline starts with web crawling, then runs deduplication using Simhashes, generates GPU-powered embeddings at 10,000 rows per second, and builds a vector index using Apache Spark and Delta Lake. The result is a search API that returns semantically relevant web content for AI agents and enterprise customers.

What is the memo wall specification in Lance?

The memo wall is an emerging specification in Lance for handling streaming upserts using an LSM tree approach. It is designed to allow continuous updates to Lance indexes while maintaining data integrity and retrieval performance at scale.

Why do AI agents need a dedicated semantic search API?

AI agents increasingly rely on up-to-date web knowledge that cannot be stored in their internal model weights, so they invoke tool calls to retrieve information in real time. Exa's search API is built specifically for agents, providing semantic retrieval rather than keyword matching so agents can find contextually relevant information across the entire web.

Full transcript

[00:08] Welcome to our talk. My name is Jan. This is Jack. I'll introduce us a little bit more in a bit. But today we're going to be talking about how Exa uses structured streaming together with Lance and some tricks to be able to go from like crawling the crawling the web and indexing that and serving that
[00:25] to our customers. Um So, my name is Jan. I'm My whole career I've worked around the edges of AI and large data systems. I'm currently leading the index management team at Exa. So, we're responsible for making sure what ends up
[00:41] in our index, what do we return to our customers and for what reason? And we're managing tens of billions of pages of the web already and we're scaling up quite a bit. Um before I worked in the aviation industry, I had my own ML startup and
[00:57] before joining Exa, I worked at Databricks on Genie code. Yeah, hi everyone. My name is Jack. I work for the Lance DB and open source project. And I My career has been helping the open source. I'm a PMC
[01:12] member for Lance and Apache Iceberg and Apache Parquet. Um before joining Lance DB, I was mainly working for AWS leading a few products including SageMaker Lake House, S3 tables and the EMR and Athena. Yeah.
[01:27] Cool. Um So, what is Exa? So, a lot of people when they search the web, they use Google. But now more and more people are interfacing with information through the use of agents, whether that's purpose-built agents like coding agents or more generic agents in
[01:45] when you're using Anthropic or or OpenAI. And these agents, they rely on LLMs to kind of like do reasoning, but in more and more it's clear that Uh, they can't rely on their own kind of internal weights for knowledge. So, that's why
[02:01] these agents have gotten very good at invoking tool calls to be able to interact with the world and as well as get information. So, Exa has has made a search API that is meant for agents to be able to fetch whatever or find whatever information is available on the
[02:16] web. And and we provide kind of information layer for uh, for agents. And some notable customers are Cursor, Notion, HubSpot. We have a lot of, yeah, digital native as well as like a wide variety of enterprise customers.
[02:33] And so, we do semantic web search and that's what we'll be talking about a bit how we're using the Spark and GPUs and uh, to maintain that and how we're using Lance to actually represent the data. Uh, and we can fetch you contents. We
[02:49] can give all kinds of citations, etc. Um, so to be able to do this, of of course, there's a lot of technologies involved. So, we have our own custom vector database. We have these ingestion pipelines where we have a whole crawling stack where there's like thousands and thousands of web
[03:04] crawlers continuously crawling the web. But we're going to be talking today about the the step between the crawlers and the serving them to our vector database. Uh, and of course, it's very important that whenever new information comes up
[03:20] in the web, that we know about that and that we make that retrievable for our customers. Um, because that's that's what what they're paying for. So, if you look at uh, let me see if I can make this a bit bigger. All right. If you look at what how the data flows
[03:38] through our system, like we have these crawlers, they dump all the data in like small batches onto S3. Um, then we want to make sure that we're removing any like bad quality things or things we're not interested in our index. We we kind of deduplication to make sure we're not processing the same
[03:55] data multiple times because we re-crawl things and if that data hasn't changed like we shouldn't uh spend any more effort on it. Then we actually need to do these embeddings. So we have a whole research team that's responsible for building like these custom embedding models for the
[04:10] documents, for the queries, for the re-rankers to give uh yeah to that and that allows us to have the best search available. And after we have trained these like custom new embedding models, we need to embed every single document that makes it through our system using this embedding model so that we can
[04:26] match the queries against those and and get the best content. Then after we have these embeddings, which is on around half the crawls that we've done because we filter out a bunch of things before, we can then do a lot more classifications on top of these embeddings, things like moderation, like is this things that we want to do or or
[04:42] like more elaborate deduplication things. And then at the end we insert it into a lens table and that's how our interface with the vector database works. So the vector database team picks up any changes from there so we delete things from there, we add things there, and that's the way that that we
[04:58] maintain our index. So I want to go over a few of the technologies and tricks that we're using. So one important one is Spark structured streaming. So S3 is where our data lands and we use auto loader to automatically keep track of
[05:13] all the files that we're already processed. Uh we can we put that in a in an initial delta table and then with like checkpointing and like structured streaming from delta table to delta table or to lens tables, they all
[05:28] provide us exactly one semantics which makes make sure that we uh we don't lose any data and we're not like processing the the same data multiple times. So one of the tricks is like okay, you have tens or hundreds of billions of of
[05:44] pages, like how do you do deduplication? Like if you want to do it properly you need to do a full kind of like a full table grouping and and that's just infeasible especially if you want to do things kind of real-time. So one trick that you can
[06:00] do is called Simhashes. So most people will be familiar with the normal hash so you have some kind of like byte representation and you put that through a hashing function and that maps to a 64-bit or 128-bit representation and if you but if you make any small tweak on the input side
[06:15] you get a completely different hash. That that property is broken with Simhashes. So there's a way to represent strings that if you make a small change on the input side the hash also only changes a little bit. And and this allows you to have these kind of discrete representations that are very
[06:31] cheap to compute that you can actually use to make decisions without having to to do expensive things. So here you can see maybe the input has changed a little bit and we have 64-bits and now two bits have changed. There is some notion of the distance between how much has the input changed and this allows you to
[06:47] make decisions like oh the previous version of the URL the page was at this Simhash now it's only changed by one bit that's not sufficient for us of a difference to re-embed everything. So this is a nice trick that we're using.
[07:04] And how do we actually do this in in Spark? So we have this like append only history table where all the crawls come in. We have this we calculate the Simhashes here based on the content. And then this table is huge as you can imagine like the amount of data that goes in
[07:19] every row is pretty big already and then we're have a lot of throughput. So what we then do is we keep track of this index table which is actually a merge table is very skinny and we keep track of some metadata about every page there and we normalize this based on the URL. So if we have a recrawl the same URL
[07:35] gets merged into that into the index table and we have the previous version of the Simhash there and the new one and we're using change data feeds to be able to see uh whether things have changed. So, if it's a new page, it's uh we'll just process it. If it's an updated page, we can actually look at the previous
[07:51] version of the sim hash and the new one, and then make decisions on whether we actually want to uh process further. But then we still need this big content, and we can't store that in this index table because it's way too big. So, what we then do is we join this back to the to the big crawl history table to then grab the content only for those rows we
[08:08] want to continue processing. So, after we've decided which are the things we're processing, we need to uh embed everything. So, we have a whole fleet of GPUs with our custom models that take in these like prepared strings that go from
[08:24] the raw content to some representation that on how these models are trained. And they need to be uh batched into GPUs, and uh they produce these like fixed-size embeddings that is that are then used for retrieval.
[08:39] The issue with uh Spark is that it's natively it's very batchy, and with GPUs uh they're as you all know, they're pretty expensive. So, you want to make sure that you maintain a continuous pressure on your GPUs to get the utili- keep the utilization as close to 100% as
[08:55] possible. Which is kind of hard with Spark uh unless you do some some tricks. So, what we're actually doing is we're kind of side-stepping Spark as this inter- for this intermediate step. Uh so, what we do is we have a we're writing these uh the content uh into a Delta table, and
[09:12] we have a process that I pulse the version of the Delta table, and every time a new version comes in, we know which Parquets are being added to the table. We push these onto an SQS uh topic, and then we have uh a whole fleet of GPU pods in Kubernetes that are
[09:28] picking up these Parquets from uh from S3. And we we pipeline them like asynchronously. We load the data from S3. We at the same time the previous batches are still processing on the GPU and when they are ready they pick up the next batch, but then we write new
[09:44] parquet files on the output side. And this allows us to basically maintain close to 100% utilization and because we have a single SQS topic we can use the back pressure to scale very very quickly up and down. So this way we maintain a very low latency. We It makes
[10:00] it through this process with it on the average on 2 minutes. And we end up with this S3 the basically the same parquets with the embeddings and some other derivatives on the output side. And and these are both this delta table and the output side are more like
[10:15] buffers. These are not long-term tables. These are just like a transfer uh communication protocol between the different parts of the system. So um after the embeddings we can still make other filter decisions. As I
[10:30] mentioned we have like moderation filters so there's different types of content we don't really want in there. Um we also This is the only place where we actually have at least once because we it can happen that a pod gets killed. We use a lot of spot instances so it could happen that we write the same parquet twice.
[10:46] So we do like in batch deduplication with like a time to window to make sure we're not writing the same data multiple times. But in the in the steady stream that's the low latency job we just insert all these append all these rows into this
[11:02] lens table. So this lens table is a representation of our index and that's the steady state is just append only. But then we have a lot of reasons to delete older URLs. So let's say we recrawl the URL. We have the previous version still in there. We don't want to have them twice in there. We only care
[11:18] about the latest one. Our vector database only returns the latest one if if we were to retrieve both, but we have an asynchronous process that ends up cleaning up these older URLs. So we have when our crawlers hit a 404 or for
[11:33] whatever reason we no longer want to maintain them in our index, we maintain a buffer of all the URLs that we want to delete. And we also do a global table scan over the whole lens table to find duplicates that are from the same URL with earlier versions. So,
[11:50] we have this we run it every 6 hours. And it ends up deleting like 100 million rows every every 6 hours or so. And we're applying these deletion vectors to the lens table with the Spark connector that Jack is going to tell.
[12:06] Tell a little bit more about. Sorry about that. Um There's a few other components that I didn't really go into yet. Outside of the crawl history table and the lens index
[12:23] that we're using to serve our retrieval product, we also store content that is used both to return the right results to our customers, but also for our re-rankers. So, the the first pass for retrieval just does a
[12:38] first dot product over the whole index, but then afterwards we have a number of candidates and we want to do much more expensive GPU operations on a much smaller set. And for that we actually need the the fresh content that were that was went into the embeddings
[12:54] because we can then compare that to the query and and re-embed everything in a much more high-quality way with more compute, but that's only on the a few hundred pages. So, for that we need to like get the content very quickly. And for this we're actually using S3
[13:10] Express, which is a pretty nice um product from AWS, which is very similar to S3, but it's much much faster and at a cost, of course. And yeah, so we're also on the side maintaining that index with a way in
[13:26] production to basically fetch this at like 10 or 20 milliseconds, we can grab the content from S3 Express by maintaining our own index on where we can find this. And and that's kind of a side product of this this pipeline. And to fill this content cache, we're actually using
[13:41] WarpStream as an intermediate layer. So, Spark streams these the content that needs to be updated into WarpStream, and we have this like set of workers that continuously ingest from there and and feed it in there. Because it's really important that we maintain this content cache because if it's not in there, we
[13:57] cannot actually serve it to our customers. Um yeah, so I've talked a little bit about our ingestion pipeline kind of end-to-end and and our use cases. But why do I actually use Lens as the final sync?
[14:12] And let For those of you who don't know, like you'll learn more in a bit, but Lens is a similar kind of file format as Delta or Iceberg. So, it's it's meant as like a data lake table format. Um and it
[14:28] has a lot of the similar semantics like deletion vectors and like commits, etc. But there's a few pretty key differences that are very important for for us, and it was purpose-built actually for vector search. So, one of the things that's that's really nice for our domain, but uh not
[14:44] just for our is if you have something like Delta or Iceberg and you have a you want to add a new column, you can do that with like a metadata change and your for new rows, you can add that column without any issues. But if you want to actually backfill that column because it's derived, let's say
[15:00] a really expensive operation that we do sometimes is like re-embed the whole index. Then you would actually to do that in Delta, you would have to rewrite the whole table because you cannot append a column to existing rows. You have to rewrite the whole row to include that column. And
[15:16] in a way, you have these row fragments, and Lens supports like a second dimension. So, you can add column fragments. So, you can write the columns uh new columns that are aligned with the old fragments. And then it's a metadata update. And the reader will just know like, "Oh, to get this row, you need
[15:31] both of these files." And this means that there's zero copy like backfills. There's some data types that are very specific to our domain. So, like these embeddings are are are pretty dense. They're typically 16-bits, uh but over like thousands and thousands of
[15:46] dimensions. So, it's a lot of data at at our scale. So, being able to have like a a purpose-built data structure that is uh yeah, the lowest amount of bytes that you actually need, as well as like readers understanding how to use it. That's nice. And then the third one is uh you can do
[16:03] row-level reads. So, there's this concept called a row address. Uh and that row address is uh consists of two pointers basically. So, the first one is representing a specific fragment, which is kind of like a a data file or a set of data files. Uh so, those
[16:19] translate by looking at the version or the commits, they translate to the files on S3. And then the second half is basically a row index within that fragment. So, this this allows you to point to a unique row in your data set. And the way that you can use byte ranges
[16:35] to only download a subset of your of a file on S3, this allows us to basically with very low latency, even on regular S3, to just grab uh a specific row. And that's why it's very convenient for uh storing data, but also for even for serving data if you're using S3 and
[16:52] you're okay with 300 milliseconds of latency. If you know where to find it, you can actually grab it very quickly. So, those are a few things that are uh are I think really elegant about Lens DB. And that's one of the reasons why we're using it more and more.
[17:07] All right. So, now I'd want to give the uh speaker or the microphone to to Jack. Yeah, thank you very much. Um and uh you can clearly see this is stitching two different company slides together because we are changing the theme of the slides.
[17:24] And um and also thanks very much for the introduction for the last format. I actually had a talk yesterday. I also already see some familiar faces here coming from the talk yesterday, but there was a much deeper dive into how the last format works and why it is used
[17:40] by all these frontier labs and for training and serving purposes. I think there will be a recording of that. So for anyone interested please take a look at that. And this is mainly about the last spark integration and what we have achieved as a part of working with
[17:56] customers like Exa and also what we are working towards as some new features coming forward. And there are three things I like to highlight here. The first one is what we call the blob V2 support integration. And the second part is about searching searching using spark. And the the third one is about
[18:13] streaming up search. So let's first go through the first one. Um Um so for people who don't know what last blob the so called last blob V2 is so the feature is essentially that in
[18:29] the last format one of the highlights of that is it provides very good support for storing very large blobs such as images, videos and large documents. And the way we have been supporting that is that instead of storing that just as any
[18:45] normal binary column value in in in formats like parquet what we do is that you are just storing a size and a pointer of that. So the so so the blobs are still in the same file, but it will be stored what we call out of line. And when it's out of lines all the blobs are
[19:01] stored continuously together and then there are pointers internally into the file into those blobs so that it can support very fast random access and and for different training purposes. But what we realized is that uh this works very well for training, but um um but what what if the user wants to
[19:17] store like very large documents like gigabytes or in in some of our use cases there are like movie movie videos that are even larger than that. So, so essentially we want to support a wide range of blobs that are can be stored all in the same file or or in some cases
[19:32] it was stored as what we call a packed file so that it's a separate file, but also the the the the the blobs are stored continuously. So, we we solved that uh as a storage layer at what we call the uh the blob V2. Um and uh now the question is how do we integrate that
[19:48] with Spark? So, here is a very very simple problem that is presented here. For example, you are running a merge into and uh you are merging into this table and uh you are probably matching some ID columns and uh then you are updating some values. And this can totally not touch a blob column if your
[20:04] table has a blob. But, uh behind the scene actually what is happening is that uh the For example, if you're using Iceberg table or Delta table, then it will during the write phase it will read through all the blobs and then write it back. Um and that way is taking a lot of time.
[20:19] And and what we are doing today uh in in Lens is that actually it will integrate end-to-end with the storage layer and uh so that it will minimize the amount of reading you need to incur during the uh engine processing phase so that uh whenever you are doing a join or doing a
[20:34] merge or doing update that might possibly touch a blob and have to read it out, uh it will only read the pointer uh in what we call a descriptor out and then only when it is actually writing at the point of you are writing the file um that is when we are going to read the
[20:50] blobs and doing the minimal thing to to process that. And uh today um if you look into what people are doing, uh it's essentially two ways. The The first way is what I've just describing that you kind of have to take the cost of uh reading out of the blob. Or the other way is that you uh
[21:07] you store a pointer to a different location in some S3 bucket or some whatever stuff you're using. And that is also not ideal because in in that case you are essentially thinking you have to think at the business layer about, okay, this blob is stored in the file and that
[21:23] blob is not stored in the file and I have to handle them differently. And what we are doing in the Lance writing integration of this is that you don't have to think about that anymore. A blob is just stored as any other data type and and the the the internals of how the format works handles all these kind of
[21:39] thing behind the scene. And as I said here, it is essentially just one column and whatever layout is that is at at the storage format layer and it is not a concern of anyone that is using Spark to process data or
[21:54] actually any other engine that is doing the same thing. So here is kind of example of how how things will work comparing to it used to be or if you are working with some other formats. Consider that you are using you are
[22:10] doing a scan and you have to join with some other tables and then you are doing a doing some kind of shuffle and merging it back. In the in the blob V2 integration here, so everything will use the descriptor along all the way up to the point that it is writing and then when it's writing it will choose
[22:26] automatically the best layout. For example, suppose you have some Let's say actually so there's a external so there's a mode in the blob V2 storage called external. So it will say that, okay, I have these blobs already in
[22:43] inside some buckets and now I want to and and I don't want to really rewrite those into my last table yet. So I'll just point into two those files at this time but maybe after some time you want to rewrite that due to some merge operations. And then at that time when
[23:00] it is reading in the last green box here, that is where the actual last writer will see, "Okay, these are external blobs, but maybe those blobs are not really huge and I can write them into a packed format and it will automatically write them into the packed format." And those are
[23:15] handled everything automatically behind the scene. And by doing that, you essentially you no longer have to think about, "Okay, I I have this merge that might incur some very big cost if I have a block column." You you are never needed to think about those things
[23:31] anymore. So, that that's the first feature. And the second feature, which is now actually you can do search in Spark using the last format. And we support three different table functions. The first one is search, which does full text search with being 25. The second
[23:48] one is vector search with all the vector similarities. And then hybrid search is you combine full text and vector to produce some kind of ranking across two. So, the first thing you will do to enable that is you create an index and
[24:03] you can create an FTS index or vector index or even we also support like scale indexes at B3 and ZoomMap. So, there is a syntax for doing that, create an index. And then you can run the full text search and vector search using syntax here.
[24:19] I think when I implement that is kind of mainly following BigQuery syntax. So, for people who are familiar with that, that is what you get. And when it's doing the hybrid search, the currently the default is the RRF.
[24:34] So, that essentially we are doing parallel the vector search and the full text search and then it will use RRF to re-rank based on whatever is in the existing ranking. So, that there's a ranking of how important the document is in full
[24:50] text and there's a ranking of how important a document is in vector search and then you rank that using the algorithm. And and then there's a a question about how it's executed. Uh so this is actually also why although Lance is kind of known for its search capability, but
[25:06] we didn't really add Spark integration at the at on day one because usually people use Spark because they want to run ETLs and uh and to to load data into Lance table or maybe uh they they use for example uh Lance DB to run vector search and then they they
[25:22] also want to run some OLAP queries on top of that. So that that is when they are using Spark. Um but um we we realized that some customers um they they have uh for example they have a contract with Databricks so that they they want to use Spark as much as possible or maybe Spark is a part of
[25:37] their already data um uh analysis or training pipelines. So this uh and and that cases this is becoming very useful because uh then you can kind of embed these function calls into uh into the operation. But the cost is that uh every time you run this is
[25:53] essentially spawning another worker to to run this and it will uh so so you pay the cost of spawning another worker compared to you run some direct search uh search engines like Lance DB. Um and uh and and then uh the way it is uh calling the actual search function is
[26:09] through the so-called Lance Nexus interface which is the common interface that all the engines integrate with Lance DB. Uh and there's uh this API called query table which you just call it to query the table. And when we say query in this context is essentially search. Uh and that actually has a very interesting
[26:26] benefit. Uh not on this slide yet. The um a very interesting benefit is that uh now actually uh instead uh in addition to you can run that uh only in Spark I can also connect to a a remote server to run that. That is how our customer for example
[26:42] uh can I say this customer or I cannot? Uh so some of the some of our customers run that with SDV enterprise which is there's enterprise clusters that is doing the distributed search and also with all the caching benefits, and then you can still cost back, and then that
[26:57] actually routed to the the the cluster to run distributed search behind the scene. And that is when all the search benefits can come for This is also talked about in my last talk yesterday that currently
[27:12] the expectation is that if you run that directly against S3, you kind of get like seconds level performance. And then when you run with caching and organization for distributed search, for vector search we can push the limit into single digit milliseconds. For full
[27:28] text, it's usually double digit milliseconds at this point. Yeah. The last part of the feature is about stream up sir. This is a feature that is still work in progress, and I think lots of people are really looking forward to this. And
[27:44] this is related to how table format is implemented today, including both Iceberg and Delta and and Lance. So, eventually you are bottlenecked by how you commit the table because all these formats are table formats MVCC. So, every time you do a commit, you create a
[28:00] new version, and then for especially Delta and Lance, they they both commit directly to server your storage. So, so so they are kind of essentially bounded by how many transactions you can perform against a single file in object storage, which is pretty low. It's like five transactions per second.
[28:17] And uh and that that's that's actually pretty pretty bad for streaming use cases because you have to it kind of limits how fast you can commit in a streaming applications. And that is how we are thinking about
[28:33] can we do better than that? And the way we are doing right now is what we call the memo wall spec of the Lance table format. The memo wall essentially just means memo table and wall, and it's a part of the whole log structure merge tree uh structure. So, what we What
[28:48] we're doing now is that there is uh there is already there is a wall format in Lens, so that uh you can append data directly into the wall. And uh the the wall uh and and then there's a wall for every single shard. So, you can horizontally
[29:03] scale to different shards uh of the same table. So, a table a Lens table can have uh uh almost infinite number of shards because every shard is stored in object storage. And then you will just append to the every single shard um uh for um for the data of the wall. And of course,
[29:19] that uh if you read that back, then you have to replay the wall, and that is very not performant for the read side. So, that is why you need a memtable. That is the way you are writing, you write to both the memtable and the wall. And and then when you are reading it back, you are mainly reading from the memtable until unless you are kind of bootstrapping in the beginning.
[29:36] Uh so, that is what what will give us the horizontal uh write scale-out performance when you are doing structured streaming in Spark. So, that you can imagine that every single Spark uh partition will essentially write to a a shard of the wall. Sorry,
[29:52] uh the wall in the in the shard. And and and then um and and and then at the end of the of the pipeline, there there can be a reader that is replaying the wall to hydrate the memtable and then be able to read from the memtable.
[30:09] Um and and essentially this is how we are breaking the commit bottleneck uh uh from existing object storage-based table formats. And another benefit uh that is coming out of this is because this is a log-structured merge tree, you can essentially do upserts. You It's not just insert, this is actually upsert
[30:25] that uh if you have a row that has a newer value, uh then uh you are writing just append into the wall, and then when you are reading, it is automatically log-structured merge read that is going to automatically get you the latest row sorry, the latest version of the row
[30:41] uh throughout the throughout all the historical versions. And and and as I said, the read stays fast because it is a log-structured merge read. And what we do when at the read time, actually this this read massive read is not just uh like from the common sense, if you talk
[30:57] about log-structured merge trees, it's like reading the rows. Actually, it's also about vector search and full-text search. So, uh so when we are writing the data, it's not just writing into the wall, but actually when you are accumulating the memtable, we also what we also do is that we
[31:13] uh you you are write the index of the vectors the vector search index, the full-text search index, or if you are doing point lookups, you can you can also write a B-tree uh while you are writing the data. So, when you are flushing the memtable, uh all these indexes will be flushed directly into the last index format. And
[31:30] then we are reading reading it back, every single layer of the log-structured merge tree is going to have index covered in that. So, you are essentially never going to have all index data read uh throughout the throughout the path. And that is how we make things always fast at the read side. Uh and then this comes uh
[31:48] and this becomes a uh CAP theorem kind of thing that you you will be able to trade some kind of performance and durability with consistency uh because this is essentially uh moving to a system very similar to like HBase, Cassandra. And uh
[32:04] and if you want to really fast durability, that is writing directly into uh object storage every time, then you kind of pay a little bit of cost of either consistency or performance um because uh every time you write to a wall, it's you are writing a file. And uh for object storage, they typically
[32:21] like bigger files. So, if you uh so so for higher throughput, what we do is that we kind of batching the uh the writes of batch in into object storage into the wall um uh through through multiple multiple writes instead of a single right.
[32:37] Yeah. And uh from integration perspective, this is actually very similar to how uh search is is integrated today. Uh this is what we are working on right now in Spark, which is uh there is also through the next phase the merge insert API. That is the API that people write into uh
[32:56] uh uh writing to a lens table. And then there there is a local implementation, and then there can be also a serving implementation that is provided by product side enterprise. And and also there are also uh other open source implementations like I think Gravitino, for example. So so you can always get accelerations out of that through the
[33:11] same code base and the same code path. Yeah. And uh those are the three features that are uh very related to the search and uh uh um search and ingestion workflows for lens. And uh uh we are looking very looking forward to the new feature as well. Yeah.
[33:28] That's it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.