Parquet vs Lance for machine learning

Summary

• Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. Lance is an open source lakehouse format for multimodal AI with vector and full-text search built in.
• Parquet is supported in many programming languages and analytics tools. Lance lists integrations with Apache Spark, Ray, PyTorch, DuckDB, Pandas and Polars.
• On Databricks, Delta Lake tables extend Parquet data files with a transaction log, and any Delta table with a primary key constraint can serve as a Unity Catalog feature table.
• Unity Catalog lists PARQUET among its external table formats and lists no Lance format. Lance's docs say Unity Catalog does not natively recognize Lance.
• A Parquet dataset converts to Lance in one line with PyLance, and the Databricks AI Search docs describe the HNSW algorithm and do not mention Lance.

Parquet vs Lance for machine learning

Parquet and Lance are both open source columnar formats, and their docs describe different design focus areas. Parquet's docs describe a column-oriented file format for efficient storage and retrieval of data in bulk. Lance's docs describe a lakehouse format for multimodal AI with built-in vector and full-text search. Match the format to the data and the queries: tabular features that several engines read fit Parquet-based tables, and embeddings or multimodal data that need vector or full-text search fit Lance.

How do Parquet and Lance compare?

Aspect	Apache Parquet	Lance
What it is	An open source, column-oriented data file format.	An open source lakehouse format for multimodal AI: a file format, a table format and a catalog spec.
Documented focus	Efficient data storage and retrieval, with compression and encoding schemes for complex data in bulk.	Fast random access and scans for AI workloads, with vector and full-text search accelerated by secondary indexes.
Table features	A file format. Delta Lake adds a transaction log on top of Parquet data files for ACID transactions.	The docs list ACID transactions, time travel and schema evolution with backfill.
Data it targets	Complex data in bulk.	Images, video, audio, text and embeddings alongside tabular data.
Ecosystem	Supported in many programming languages and analytics tools.	Integrations with Apache Spark, Ray, PyTorch, DuckDB, Pandas and Polars.
Unity Catalog	Supported for external tables, and Delta Lake tables store their data as Parquet files.	Not a listed table format. Lance's docs say Unity Catalog does not natively recognize it.

This page makes no speed comparisons between the formats.

When does Parquet fit machine learning data?

Parquet fits tabular training data and feature tables that several engines read. Spark reads and writes it with spark.read.parquet and write.parquet, Delta Lake tables on Databricks keep their data in Parquet files with a transaction log, and any Delta table with a primary key constraint can serve as a Unity Catalog feature table.

When does Lance fit machine learning data?

Lance fits embeddings and multimodal datasets that need vector search or random access to rows. Its docs show approximate nearest neighbor indexes (for example IVF_PQ), full-text search, and a PyTorch integration, lance.torch.data.LanceDataset, for training and inference loops. A vector query with a filter looks like this in the Lance docs:

import lance

ds = lance.dataset("s3://my-bucket/docs")
ds.to_table(
    nearest={"column": "embedding", "q": query_vec, "k": 10},
    filter="year > 2020",
)

How do the two formats work together?

The Lance quickstart converts a Parquet dataset to Lance in one line:

import lance
import pyarrow as pa

parquet = pa.dataset.dataset("/tmp/test.parquet")
dataset = lance.write_dataset(parquet, "/tmp/test.lance")

In the other direction, the Lance Spark connector reads a Lance dataset into a Spark DataFrame, which you can write to a Delta Lake table:

df = spark.read.format("lance").load("/path/to/dataset.lance")

Databricks supports custom third-party Spark JARs on dedicated compute, and not on SQL warehouses, serverless or standard access mode compute. A Databricks AI Search index can be built from a Delta Lake table that holds precomputed embeddings, and you query it with index.similarity_search, passing query_vector, columns and num_results.

How do you choose between them?

Ask four questions about each dataset:

1. Which engines and tools must read it?
2. Do you need vector or full-text search over it?
3. Does it hold large binary or multimodal columns next to tabular ones?
4. Which catalog must govern it?

FAQs

Can I convert between Parquet and Lance?

Yes. lance.write_dataset writes a Parquet dataset as Lance. To go back, read the Lance dataset into Spark with the connector and write a Delta Lake table, whose data files are Parquet.

Does Databricks support Lance tables in Unity Catalog?

Databricks lists Delta Lake and Apache Iceberg for managed tables, and Delta Lake, CSV, JSON, AVRO, PARQUET, ORC and TEXT for external tables. It lists no Lance format. Lance's own Databricks guide marks Unity Catalog as not supported for its Spark connector.

Does Databricks AI Search use Lance?

The AI Search documentation describes the HNSW algorithm for approximate nearest neighbor search with the L2 distance metric. It does not mention Lance.

Do I need to choose one format for my ML pipeline?

No. They are separate formats and a pipeline can read both. Choose per dataset, based on the queries that dataset needs.