Best vector search for large-scale semantic search in a consumer app

Summary

• Consumer apps need semantic search that understands meaning ("show me running shoes") rather than just keywords, enabling cross-format retrieval where text queries return relevant images and vice versa.
• Databricks AI Search is purpose-built for billions of vectors with two deployment options: Standard endpoints (320M vectors, tens of milliseconds latency) and Storage Optimized endpoints (over 1B vectors, hundreds of milliseconds).
• Scale requires distributed indexing on Spark, vector compression, and separation of storage from compute. A Rust-based query engine handles both async I/O and CPU-bound vector computation independently.
• For smaller datasets or operational workloads where embeddings arrive continuously, pgvector extends Postgres, eliminating the overhead of managing a separate vector system.

Best vector search for large-scale semantic search in a consumer app

Semantic search is now table stakes in consumer applications. Users expect to search by meaning, not keywords. Search for "red running shoes" should return images of red shoes, even if product listings don't use those exact words. This requires encoding your catalog into a high-dimensional vector space, then finding vectors similar to the user's query vector.
Scale is where vector search gets hard. Billions of vectors consume storage and compute. Standard database indexes don't work on high-dimensional spaces. You need specialized indexing, distributed computation, and careful latency tuning to serve search results in milliseconds while keeping infrastructure costs manageable.

What makes a vector search system production-ready?

Production vector search needs several properties: fast query latency, high recall (the results actually match the query), support for billions of vectors without rebuilding indexes, and the ability to scale compute separately from storage. Many vector systems start fast on small datasets but degrade as scale grows. Building vectors once per day is different from streaming new vectors continuously.

Key capabilities to evaluate

Capability	Why it matters
Latency at scale	Query response time must remain fast as vector count grows. Distributed indexing and hierarchical clustering prevent latency spikes.
Recall vs. speed trade-offs	Approximate nearest neighbor search is fast but imperfect. For high-value queries (product search, recommendations), recall matters more than speed. Evaluate both.
Batch and operational workloads	Some apps build indexes nightly (batch); others stream new vectors continuously (operational). One system may excel at one but struggle with the other.
Cost at scale	Indexing 1 billion vectors should not cost thousands per month. Evaluate compute cost during indexing and serving cost per query.
Cross-format search	"Find images matching this text description" requires shared embedding space across formats. Many systems only handle single-format search.

How Databricks approaches billion-scale vector search

Databricks AI Search is designed from the ground up for billion-scale. It decouples storage from compute: data lives in cloud object storage (S3, ADLS, GCS) while Spark clusters build indexes on ephemeral compute nodes.
The architecture uses distributed clustering and vector compression. Documents are partitioned into clusters during indexing; during query time, only relevant clusters are searched, not the entire corpus. Compression reduces memory footprint and network traffic without destroying quality.
For latency, Databricks uses a Rust-based query engine with a dual-runtime architecture. Async I/O (retrieving clusters and vectors from storage) runs on one thread pool. CPU-bound vector computation (similarity calculations) runs on another. This separation prevents I/O from blocking computation or vice versa.
Two deployment options handle different use cases. Standard endpoints keep full-precision vectors in memory; they support up to 320 million vectors with tens-of-milliseconds latency and are ideal for catalogs that fit in memory. Storage Optimized endpoints separate memory from storage; they support over one billion vectors with hundreds-of-milliseconds latency, enabling massive catalogs (e.g., the entire internet image index) at manageable cost.
Indexing is built on Spark, enabling horizontal scale. Building a billion-vector index is achievable on modest clusters. Indexes are immutable once built, enabling safe concurrent reads without locking.

Use cases

E-commerce product discovery: Index product images, descriptions, and attributes. Users search by image or text and get relevant results across the catalog. Cross-format search (image query against text descriptions) improves relevance.
Content platforms: Index articles, videos, and user-generated content by semantic meaning. Recommendation engines retrieve similar content based on user behavior, improving engagement.
Media and entertainment: Search libraries of photos, music, or videos by semantic tags or descriptions. "Find all scenes with rain" matches semantically relevant clips without manual tagging.
Travel and hospitality: Search properties by amenities and experience ("pet-friendly beach resort with a spa"). Semantic search captures nuance that keyword matching misses.

FAQs

Do I need a vector database to search vectors at scale?

Not necessarily. Specialized vector databases are simple to set up, but Databricks AI Search built on Spark offers more flexibility and lower cost for very large catalogs. For small datasets or operational workloads, pgvector within a Postgres database is simpler and cheaper.

How do I handle continuous updates (streaming vectors)?

Semantic search typically builds indexes in batch. If new vectors arrive continuously (e.g., new products uploaded daily), rebuild the index nightly or on a schedule. For true real-time insertion (embedding one document at a time), pgvector is better suited. Hybrid approaches (batch index for bulk search, pgvector for recent additions) are common.

What embedding model should I use?

Embedding quality directly impacts search relevance. Start with open models (Sentence Transformers, OpenAI's text-embedding-3) or foundation model APIs (Databricks Foundation Model APIs). Evaluate on your data by manually scoring search results. If off-the-shelf models don't capture your domain (e.g., fashion, medical), fine-tune on domain examples.

How do I avoid re-indexing when the embedding model changes?

Store raw documents separately from embeddings. When a new embedding model becomes available, re-embed the corpus and build a new index without touching source data. Both indexes coexist; gradually migrate traffic to the new index as it proves better.

Does vector search work for text-only queries?

Yes. Embed both documents and user queries using the same model. The query embedding sits in the same vector space as document embeddings. Cosine similarity or other distance metrics find the most similar documents. The key is using a consistent embedding model across queries and corpus.