Lance vs Parquet for storing embeddings and multimodal data

Summary

• Parquet is an open source, column-oriented data file format. Embeddings fit in a LIST of FLOAT values, and the format documentation describes no vector similarity index.
• Lance describes itself as "The open lakehouse format for multimodal AI", with vector indexes in the format (IVF with a FLAT or HNSW sub-index) and blob columns for images, video and audio.
• On Databricks, Delta Lake extends Parquet data files with a transaction log, and a Databricks AI Search Delta Sync index can index an array[float] column of a Delta table.
• Lance's own Databricks guide, last updated 2026-02-19, documents the Lance Spark connector on classic compute and lists Unity Catalog (as the connector's catalog) and SQL warehouses as not supported.
• We believe the choice comes down to where vector search should live: in the storage format (Lance) or in an index over governed tables (Parquet-backed Delta tables with AI Search).

Lance vs Parquet for storing embeddings and multimodal data

Both formats can store embeddings. Parquet is a general column-oriented file format: an embedding sits in a LIST column of FLOAT values, and similarity search comes from the engine or index layered on top. Lance is a lakehouse format built for multimodal AI, and its format includes vector indexes and blob columns. On Databricks, Parquet files back Delta tables, which Databricks AI Search can index, while Lance's guide documents a Spark connector on classic compute.

What is Parquet?

Apache Parquet describes itself as "an open source, column-oriented data file format designed for efficient data storage and retrieval" with "high performance compression and encoding schemes to handle complex data in bulk." Its primitive types include FLOAT (IEEE 32-bit floating point values) and BYTE_ARRAY (arbitrarily long byte arrays). The LIST logical type annotates types that should be interpreted as lists, so an embedding can be stored as a list of floats, and a FLOAT16 annotation covers half-precision values. The format documentation covers types, nested encoding, data pages, a bloom filter and a page index, but not a vector similarity index.

What is Lance?

Lance calls itself "The open lakehouse format for multimodal AI": "a file format, table format, and catalog spec for building a complete lakehouse on object storage". Its specification describes vector indexes built from IVF clustering, a FLAT or HNSW sub-index and PQ, SQ or RQ quantization, with l2, cosine and dot distance metrics. You build an index with create_index, for example with index_type="IVF_PQ", and search with to_table(nearest={...}). Lance's tutorial says that without an index, a search scans the entire dataset computing distances to the query vector. For multimodal payloads, Lance blob columns can store "images, videos, audio, model artifacts".

Lance vs Parquet: how they compare

Dimension	Parquet	Lance
What it is	Open source, column-oriented data file format	A file format, table format and catalog spec for a lakehouse on object storage
Embedding storage	A LIST of FLOAT values, or FLOAT16 for half precision	Vector columns in a dataset, for example written with lance.write_dataset
Vector index in the format	None documented; the docs cover types, nested encoding, data pages, a bloom filter and a page index	IVF clustering with a FLAT or HNSW sub-index and PQ, SQ or RQ quantization
Building and searching	Through the engine or an index layer, such as a Databricks AI Search Delta Sync index over a Delta table	create_index, then to_table(nearest={...}); without an index the search scans the whole dataset
Multimodal payloads	BYTE_ARRAY holds arbitrarily long byte arrays	Blob columns for images, videos, audio and model artifacts, read with read_blobs or take_blobs
On Databricks	Delta Lake extends Parquet data files with a transaction log, and PARQUET is an external table format	Lance Spark connector on classic compute; its guide lists Unity Catalog (as the connector's catalog) and SQL warehouses as not supported

How does Databricks approach embeddings and multimodal data?

Delta Lake "extends Parquet data files with a file-based transaction log". To search embeddings stored in a Delta table, create a Delta Sync index in Databricks AI Search. It "automatically syncs with a source table, incrementally updating the index as the underlying data in the source table changes." The precomputed embedding column should be array[float], and standard endpoints need a change data feed on the source table:

ALTER TABLE main.ml.doc_embeddings SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

For multimodal sources, Unity Catalog volumes enable governance over non-tabular datasets such as images, audio, video and PDF files. One layout keeps the files in a volume and their metadata and embeddings in a Delta table.
Lance's Databricks guide, last updated 2026-02-19 and tested with Databricks Runtime 16.4 LTS, installs the Lance Spark bundled JAR on classic compute and requires LanceNamespaceSparkCatalog. Its support matrix lists Unity Catalog (as the connector's catalog on classic compute) and SQL warehouses as not supported, so check the current guide before you plan around it. Databricks documents DELTA, CSV, JSON, AVRO, PARQUET, ORC and TEXT as external table formats.

FAQs

Can I store embeddings in Parquet?

Yes. The LIST logical type annotates types that should be interpreted as lists, and FLOAT is an IEEE 32-bit floating point primitive, so an embedding can be a list of floats. Parquet defines storage, and the documentation lists no vector index.

Does Lance create vector indexes automatically?

In Lance's tutorial you build the index explicitly with create_index. Without one, the search scans the entire dataset and computes distances to the query vector.

Can Lance store images and audio?

Lance's Blob API documentation says Lance can store large binary objects, such as images, videos, audio and model artifacts, in blob columns that are treated like any other column payload in the dataset.

Can I query Lance from a Databricks SQL warehouse?

Lance's Databricks guide lists SQL warehouses as not supported and documents classic compute with the Lance Spark connector instead. Delta tables with a Databricks AI Search index are one documented Databricks option for embeddings in tables.