Scaling AI observability: handling complex multimodal data

Summary

LanceDB introduces Lance format, a columnar storage system combining fast scan performance with constant-time random access, designed to replace fragmented AI data infrastructure that previously required expensive ETL pipelines to synchronize across parquet files, vector databases, and analytics platforms.
Lance format stores petabytes of multimodal data — including traces, embeddings, video, and sensor streams — in a single queryable layer, enabling interactive analysis without pre-aggregation or background jobs that traditional data lake approaches require.
Companies including Netflix, Uber, and autonomous driving teams use LanceDB as a unified storage foundation for AI observability workloads ranging from semantic search and trace analysis to real-time monitoring across thousands of concurrent agents.

Scaling AI observability: handling complex multimodal data

Observing AI systems at scale requires new infrastructure designed from the ground up. Traditional data lakes fragment AI data across parquet, vector databases, and analytics platforms, requiring expensive ETL to synchronize. LanceDB introduces Lance format, a columnar storage system combining fast scan performance with constant-time random access, enabling single-source-of-truth observability for all workloads.
Learn how Lance format stores petabytes of multimodal data, traces, embeddings, raw video, sensor streams, in a single queryable layer. See efficient root cause analysis through timeline correlations, interactive analysis without pre-aggregation or background jobs, and how Netflix, Uber, and autonomous driving systems leverage this architecture for real-time observability across thousands of concurrent agents.
🤝

Chapters

FAQs

What problem does LanceDB's Lance format solve?

Lance format solves the fragmentation problem where AI applications must store different types of data across separate specialized systems — parquet for lake storage, vector databases for search, BigQuery for analytics. This fragmentation requires expensive ETL to keep systems synchronized; at Cruise Automation, this meant running thousands of dollars in ETL jobs just to move approximately 30 petabytes of sensor data around daily.

What makes Lance format different from traditional columnar storage?

Lance format combines fast scan performance for analytical queries with constant-time random access for point lookups, a combination that traditional columnar formats like Parquet cannot provide in one system. This single format handles all workloads — creation, evaluation, monitoring, and observations — without needing separate specialized systems for each.

What types of data and use cases does LanceDB support?

LanceDB stores multimodal data including traces, embeddings, video, sensor streams, and code, and is used by Netflix and Uber for video analysis, autonomous driving teams for sensor data, and code review tools like CodeRabbit for semantic search. All of these diverse workloads are served from the same unified storage layer.

How does LanceDB enable interactive AI observability analysis?

Because Lance format supports efficient queries without pre-aggregation or background jobs, analysts and agents can slice and filter data interactively in real time. This enables root cause analysis through timeline correlations and allows distributed agents running on object storage to be observed without centralizing data into a separate analytics system first.

Full transcript

[00:03] Maybe you know about us Les TV. We well company actually building a new format new data format and new data engine on top of that format. So we are actually building a very low level of abstractions for for those
[00:19] AI applications out there. We have been working on this technology for about 3 years and our technology is right now have a big adopt by several generations of different AI companies out there. For
[00:34] example, Netflix and Uber are using us for videos and for autonomous autonomous driving cars. And we have a genetic code review tools like Code Rabbit out there is using us for their regular database search. And also we have
[00:50] working with a bunch of other world models, image generations, and autonomous driving robotics company out there that this is for different modality of say video data, sensor data, and a bunch of other data all together. We see different very
[01:06] different kind of users, but all unified within this like unified storage foundations to to offer their different way to to access the data. So the problems we see today is um when I
[01:22] before starting this company, I was working in Cruise Automations. We built a lot of infrastructures try to observe uh like 20 30 petabytes of new cars that sensor data from the car every day, right? At that time our state of
[01:39] art infrastructures dump data into parquet in a lake dump data into TF record into another inference infrastructure, then dump data into bigquery, right? Every every piece we have we are generating 30 uh like 30
[01:55] petabytes data every every single day. And they run thousands of dollars of ETL just to move the data around, right? And then we realized that um even makes those each data silos synchronize correctly with each other is
[02:12] very challenging for them. And to present the data correctly, be usable for different workloads from say creation, evaluation to like monitoring to to observations. That's um they do need very different workload
[02:29] than last generation of a big uh big data infrastructure who most of time just a scan the data, do aggregation, and do a bunch of like statics over there. So, then we pull back a little bit to see what we can actually do to
[02:44] um have this maybe one one system to support different kind of workload and what characteristic those of workload is. And the traditional um data lake out there was the was the state of art like 4
[03:00] years ago, 5 years ago before we started this company. And that's where we dump data into like you know that search, different vector database, and different different uh database where other like just to try to do each individual workload separately, right?
[03:15] And right now we achieved to a goal that we have one single um representations of the data. However, that can be used for for search, for training, for eval, and for a lot of like slice dicing to examine your your data. And by the end
[03:33] of the day, we just realized to achieve this, we just need two characteristics. One is that we have a we still maintain the very fast scan over the data as fast as a parquet and other like columnar format. Thus, in the same time, we 3
[03:49] years ago we are very um this is a controversial idea that we enable very fast random access within a columnar format. And 3 years ago, this is a new concept. And we just realized with that, we can do sub
[04:04] 10 millisecond single digit millisecond search directly from object store. Just a server those all kind of a search problems here. So, once you can store different modalities data within the same infrastructure and be able have to
[04:20] have sub like single digit millisecond read over object store, you can actually lay out a good data foundation that store all your different types of modality within this data and be able to search that in
[04:36] different ways, right? For example, this is a this is a example that Netflix is using. They have a image data, metadata, and captions of different frames and clips. And actually have the vector search for those clips.
[04:52] Within the same thing, you can search on different columns and then you can re-rank and join all all the data together. And some of them are not even text, right? You can have videos and image within it and actually search over over that. So, that's that's the like
[05:10] that's the angle we want to achieve. So, actually 2 years ago, we exempt a few different technology to to try to get here. However, we just realized we have to design something from scratch from the ground up because there's no tooling
[05:26] out there can achieve this goal. So, the solution we came up is a new columnar format is called Lens format, which we try to get one single source of truth for this math model data and different types of the data
[05:41] lake. So, this is a quite different from today's uh common say data lake or or columnar format in a way that we design the file format and the lake format all together within this framework, so we can do like cross
[05:58] uh cross abstraction optimization between these two layers, right? It uh it allow us to iterate very fast within uh basically what whatever we need, we can add that very fast. So, as we mentioned the most controversial uh
[06:15] feature 3 years ago is that we we support uh constant time random access to every single cell within within the data set, no matter how big the data set is, even that's still a columnar format out there. And the other one is the this so-called two-dimensional storage. We
[06:31] support zero zero copy data evolution, which is quite different from say zero copy schema evolution that you know today from other popular uh table format. Uh and everything else every other features are built around this thing. And a good
[06:48] side effect of that um is that we realize once we have this very good uh random access performance, we can build a lot of a different secondary index on top of it. That's why 2 years ago we uh people base know us
[07:03] for say vector search or full text search, but at the end of the day it's just a vector index to point to every single cell within this data set and be able to um serve that directly from a So, our latest performance number is like we can deliver 60 ms
[07:20] latency directly from uh actual block storage for 2 billion vectors out there. So, that's a Yeah, that's kind of uh unthinkable from the parquet world, right? So, this is the where we build everything together. So, to achieve
[07:35] this, we have a few different ideas how to design format from scratch. The first thing we uh get rid of because we want to store if you think that you want to store the boolean boolean values, integer values and the like videos all together,
[07:51] we the first thing we we face is like uh is very hard to group them all together in a efficient way that for both boolean column have a very very good scan performance and video column won't cause out of memory because you need to accumulate a lot of numbers within the
[08:09] memory before you flash to each each group. So, the first thing we get rid is uh the real group concept from popular uh popular format. Right? And basically, we will flash the boolean boolean columns, integer columns
[08:26] way faster than way slower than flash out the the image or video or or large tensor columns, right? They have different pace, so we don't align each of these chunks into the same same number of before we flash them. However, we can
[08:43] still provide uh like a one time to to access each of them. So, this is a some very nerdy concept uh within the storage data format before but however, uh once we count how many
[08:59] how many IOs we need to read any single rows, well, we can amortize that within one or two IOs to no matter how deep your the how deep the nesting structure where where the binary is, we can use like up
[09:15] to two IOs to read any of them. People usually think that you can only choose between very fast random access or very good compression because compression means that you read a lot of you compress a lot of data, then you lose
[09:31] the way that actually can pick each one single one of them. However, uh we are smarter than that. So, we we will be able to actually have a way uh not way better, a little bit better uh compression ratio than than parquet today because uh
[09:48] we have this concept of a mini blocks that is very aligned with the minimal IO size within say S3 where where actual block storage. Anything that's smaller than IO does not change your latency at all. So, we try to take advantage of this buffer as much as
[10:04] possible to do the compression all together. And especially for those like text and uh like JSON field and all bunch of those like long long text uh long text uh fields, we we show significant better
[10:20] uh compression that way. So, those workload is very different from say TV TVCH or other uh OLAP kind of workload. This is uh all all of them are machine learning um data set and we show uh better results in in this way. And also, we
[10:36] think Lance is not only a data data format spec, it's also the one single implementation of how this data is accessed. We have a library for that. And within that library, we re- re- build the query plan that optimized for
[10:53] large blob of reading instead of uh if you familiar with the traditional OLAP like ClickHouse or Spark or or Trino, once you want to read us for example, if you want to read a problem and image column with uh say between 1 and 10, we think Spark and the Trino, usually they will scan a quite large of uh
[11:11] a large chunk of image altogether before filter filter them out within within the memory, right? And we basically re- uh we re- order the query plan to make make the read at the last stage. So, we
[11:27] try to issue the limit uh the minimal amount of IO to read those large blobs that will significantly improve our latency to read for example 10 or 100 like vectors out of S3. And another features is called zero cost data
[11:44] evolution. This is very different from say today's Delta Lake or Iceberg. For example, if you have the first version, we generate one file for this column and another parent operation we are generating another file. If the third version we try to add the new features
[12:00] within this data set instead of rewriting this data set all together, we write a new new column just for for for the for the new files for that column and use metadata to glue them all together. This is particularly interesting when
[12:17] you are working with those like multimodal or self-driving car companies. You already have 50 petabytes here and you want to add one boolean column to say you have maybe a a a children wearing
[12:32] green green green t-shirt walking through the street. You just have this boolean column. You don't want to actually copy 50 petabytes just adding these new features, right? So this is very this is distinctly different from other formats out there. So what what does that bring
[12:48] if we build a new observability stack on top of this format, right? You We know that today many of those format for example DataDog, ClickHouse and many of them are building on either their internal column format or building on parquet based
[13:05] format. And this is showing basically new opportunities what you can achieve if you build on this new stack. So first thing is most obvious things like we can start to do where semantic search within this single framework,
[13:21] right? So you can just use natural language to describe I want to find the example that match to this video rate rates and semantically similar to something else, right? This thing within today's infrastructure, you need to have
[13:36] ETL into multiple different systems then later joined together. But within this, we can discover the problem that within the system itself. Also, we know that today's monitor like a lot of observability
[13:51] uh platform in order to offer your uh real-time analytics capabilities, there's a lot of pre-computing need to happen, right? You have a pre- pre-designed schema and pre-designed like uh basically say how how much aggregation you want to keep in memory
[14:07] before you can serve the uh serve the data. But here, you can because of flexibility of the search capabilities between different types of index, you can uh describe the problem that you did not anticipate one before when you ingest the data into the
[14:23] system. Right? And we are not only can do individual search altogether, we can we can have a multiple column have a multiple tags or multiple vectors. And we can actually join all together then use like a models to re-ranking uh
[14:39] basically have a re-ranker to to have a this combined score to give you more uh relevant results. So, another thing that we try to uh achieve so that we can actually store all those data within single rows. The events will from user aspect logic
[14:57] logically, you will see all the trace and the event and the metrics and generate assets all within this same same single single row within the table. It allows you to basically you don't need to go to S3 to find a URL to to
[15:13] download that file before you can analyze anything, right? And also, we because we enable the search, so you don't need to rebuild different search to actually then different search from Elasticsearch search and a bunch of them to be able to join join things together.
[15:29] You will have the a whole history view of every single data point you generate within this system and it's very easy for for you replay or reproduce everything. If you are paged at 2:00 a.m. every morning, you want to see the problem as soon as possible, right? You
[15:46] want to actually slice dice your data set into the problem and jump between trace to the relevant systems as as fast as possible for you to identify the problem, right? However, we we realize that it's actually just a translate to a
[16:01] very fast random access. Plus, we have a very rich secondary index to support your your search capabilities. And within that, you can easily jump between between the data set between the source of
[16:17] traces where we seeing the trace and we're seeing a slicing of the trace during busy stars start timestamp and end timestamp. Timestamp. It's very easy to actually do slice dicing there. And then, you don't need to every time run a
[16:33] 2 minutes of a re-aggregation everything before you can see the result. So, you can take this as a more interactive as a as a web product instead of waiting some background job to aggregate the data for you. As
[16:49] mentioned, lens deep lens format itself is very scalable. The largest installation out there we have a few hundreds of petabytes of data within uh all this store, right? However, we do we
[17:05] have very lightweight SDKs that you can download open source SDK. You can just download and install within the process. That's a memory process. They have a exactly same capabilities. Just a much smaller much smaller memory footprint.
[17:20] And for some reason that we got lucky, Open Cloud choose that for as a default this memory storage plugin and then a bunch of other cloud is using cloud product is is using that. So, if you
[17:35] think that because our data is on object storage immutable, you can just launch millions maybe too much, but thousands of different agents to do data analytics on top of it because they they see the same immutable rows and
[17:51] they can just do branch by themselves. This is very easy for you to build a fleet of agents to just basically understand that data in different ways. Lastly, as we mentioned zero copy evolution data evolution is basically
[18:09] say in the beginning you don't need to be perfectly design your schema. Especially if you are building this ability product, right? You don't Basically, if you have a 10 years of trace, you don't need to get things right first. You can always
[18:24] incrementally adding new features or aggregating new features and this very cheap. You don't need you don't need to worry that much. And the last thing is actually we are taking advantage of our zero cost data evolution features and we build a
[18:40] compute compute backend to take advantage of that. And the we think that the data set itself, we allowed you to define any arbitrary Python code where you can just call Open AI, Anthropic or just write your own logic and the declare that as
[18:57] compute compute features within the data set. So, every time when there's a new models coming, you will have a better signals within your data set. It will and you can choose open source or or like open eye kind of
[19:13] proprietary models to help you understand better within this. And Lens DB itself will manage manage the computation for you so you don't need to worry. For you, you just declaratively to say this feature need to drive from that two
[19:30] features plus this my my Python function here. Then that's that's all you need. We can we orchestrate the computation for you. The whole concept I want to present here is like once you have more capabilities within the storage with more way to
[19:48] access those data, you will have better what better usage or better way to actually understand this data better which is very useful for observability product out there.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.