best format for storing and querying vector embeddings at scale

Summary

• Store embeddings in a Delta table, as an array[float] column beside a primary key, the source text, and metadata, and query them through a Databricks AI Search index built over that table.
• Databricks AI Search is built into the Databricks Data + AI Platform. Its indexes are governed by Unity Catalog, and it uses the HNSW algorithm for approximate nearest neighbor (ANN) search, with hybrid keyword-similarity search available.
• Three index types fit different update models: Delta Sync with Databricks-managed embeddings, Delta Sync with self-managed embeddings, and Direct Vector Access, where you update vectors yourself.
• At scale, the endpoint type matters: standard endpoints sync continuously or on trigger, while storage-optimized endpoints hold larger indexes, partially rebuild on each sync, and support triggered sync only.
• Lance is an open source format with vector search of its own, but Unity Catalog does not list it as a table format.

best format for storing and querying vector embeddings at scale

Store embeddings in a Delta table and query them through an AI Search index. The table is the open, governed system of record: an array[float] embedding column next to a primary key, the source text, and metadata. The index is the query structure that AI Search keeps in sync with the table and serves approximate nearest neighbor search from. The format question therefore has two answers: Delta Lake for storage and an index for search.

What is a vector embedding?

The Databricks documentation defines embeddings as "mathematical representations of the semantic content of data, typically text or image data." Each embedding is a vector of numbers, and similarity search returns the vectors nearest to a query vector. You set the embedding dimension when you create an index over precomputed embeddings.

Which options store and serve vectors on Databricks?

Layer	Option	What the documentation says
Storage	Delta table with an array[float] column	Source for a Delta Sync Index with self-managed embeddings
Index	Delta Sync Index, managed embeddings	Databricks computes embeddings from text in the source table
Index	Delta Sync Index, self-managed embeddings	The source table holds precomputed embeddings
Index	Direct Vector Access Index	Supports direct read and write of vectors; you update it
Endpoint	Standard	Continuous and Triggered sync both update incrementally
Endpoint	Storage-optimized	Larger capacity; every sync partially rebuilds the index; Triggered sync only

How does AI Search query embeddings?

AI Search "uses the Hierarchical Navigable Small World (HNSW) algorithm for its approximate nearest neighbor (ANN) searches and the L2 distance metric to measure embedding vector similarity." Queries default to ANN search, and hybrid search combines vector-based embedding search with traditional keyword-based search. From Python, call similarity_search() on the index. In SQL, the vector_search() function is in Public Preview and is not available on classic SQL warehouses:

SELECT * FROM vector_search(
  index => 'main.support_docs.index',
  query_text => 'Wi-Fi issues LMP-9R2',
  query_type => 'HYBRID',
  num_results => 3)

Indexes "appear in and are governed by Unity Catalog." To create an index you need CREATE TABLE privileges on the catalog schema, and endpoint permissions use access control lists.

How do I load and update embeddings?

For self-managed embeddings, write them to the Delta table. Standard endpoints need a change data feed on the source table, and tables with row tracking enabled use one automatically:

CREATE TABLE doc_chunks (id BIGINT, chunk_text STRING, embedding ARRAY<FLOAT>)
TBLPROPERTIES (delta.enableChangeDataFeed = true);

With managed embeddings, Databricks computes the embeddings from text using a model that you specify. A Triggered sync starts from the UI, SDK, or REST API (index.sync() in the SDK), and Continuous sync on standard endpoints keeps the index synced as the source changes. On storage-optimized endpoints, the embedding dimension must be evenly divisible by 16.

Use cases

The documentation says AI Search "powers retrieval for AI applications such as RAG systems, recommender systems, and image and video recognition."
Retrieval-augmented generation (illustrative): A support team stores ticket resolutions and their embeddings in one Delta table and retrieves the closest matches to ground an LLM response.
Recommendations (illustrative): A retailer stores product description embeddings in a table and retrieves the nearest items to a shopper's query vector.

FAQs

Do I need a separate vector database?

AI Search is "a search solution that is built into the Databricks Data + AI Platform." With a Delta Sync Index, the index is created over your existing Delta table and stays synced as the table changes, and Unity Catalog governs the index.

Which endpoint type should I choose for a large index?

Use standard endpoints when query latency is critical. Use storage-optimized endpoints for larger indexes when cost efficiency per vector matters, accepting slightly increased latency and Triggered sync only.

Should I use ANN or hybrid search?

The performance guide says to use ANN queries whenever possible because they are the most compute-efficient, and notes that hybrid search can reduce throughput even though it can improve recall in some cases.

Is Lance an option for vectors on Databricks?

Lance describes support for vector similarity search with secondary indexes. Unity Catalog documentation does not list Lance as a table format, and Lance's Databricks guide lists Unity Catalog on Classic Compute as not supported.