How does Lakebase Search combine vector similarity with BM25 full-text search?

Summary

• Lakebase Search merges vector similarity (semantic search using embeddings) with BM25 (keyword-based full-text search) to handle both concept-driven and exact-match queries.
• Vector similarity excels at finding contextually related content even when keywords don't match; BM25 captures precise terminology and acronyms where semantic search may falter.
• Hybrid search lets applications tune the balance between semantic relevance and lexical precision, combining results from both methods into a unified ranking.
• This dual approach reduces the need to choose between building separate search indices for different query patterns.

How does Lakebase Search combine vector similarity with BM25 full-text search?

Hybrid search in Lakebase combines two complementary ranking methods: vector similarity finds semantically related content by comparing embeddings, while BM25 scores documents based on keyword presence and frequency. The two methods rank results independently, then Lakebase fuses the rankings into a single, calibrated result set. This approach handles queries that neither method alone would answer well. You avoid the problem of choosing between semantic recall (where synonyms and related concepts matter) and lexical precision (where exact terms and acronyms dominate).

What is hybrid search?

Hybrid search pairs semantic search (vector similarity) with lexical search (full-text matching) to cover queries across a spectrum: from purely conceptual ("show me something about data quality challenges") to highly specific ("find reference documentation for the MERGE statement"). Vector embeddings capture semantic meaning, encoding relationships between concepts. BM25, a probabilistic ranking algorithm, emphasizes relevant term frequency while penalizing repetition. By running both and combining their scores, hybrid search adapts to the user's intent without explicit routing.

Key capabilities to evaluate

Capability	Why it matters
Semantic coverage	Handles synonyms, paraphrases, and conceptually related documents without keyword overlap
Keyword precision	Retrieves exact terminology, product names, and acronyms that semantic search might rank lower
Result ranking fusion	Combines scores from both methods into a coherent ranking; miscalibration favors one method unfairly
Query performance	Hybrid search incurs compute overhead from running both methods; evaluate the tradeoff between result quality and latency
Embedding quality	Downstream of the embedding model used; weak embeddings limit semantic signal

How Databricks approaches hybrid search

Lakebase Search runs both vector similarity and BM25 in parallel, allowing applications to weight semantic relevance against keyword matching. You index content once; Lakebase maintains both the vector embeddings and the full-text inverted index automatically. At query time, specify the balance between methods (typically a linear combination of both scores), and Lakebase returns merged results ranked by the combined score. The capability integrates with Lakebase's unified catalog layer, so indexed content inherits the same access controls and metadata as your underlying Delta tables and Neon databases.
This design eliminates the operational burden of maintaining separate search infrastructure or specialized vector databases alongside your primary data layer. Because indexing happens on-demand within the Lakebase ecosystem, schema changes and data updates flow through naturally.

Use cases

Product documentation search: Users search for both exact API reference ("MERGE statement syntax") and broader topics ("how to update multiple rows efficiently"). Hybrid search serves both without separate indices.
Customer support chatbots: Queries arrive in natural language ("why is my query slow?") and also as specific error codes or feature names. Vector similarity catches conceptual matches; BM25 ensures exact codes don't get demoted.
Internal knowledge bases: Teams search by acronym ("RBAC") alongside conceptual questions ("access control patterns"). Hybrid handles the mix.

FAQs

When should I use vector similarity alone instead of hybrid?

Use pure vector similarity when queries are consistently paraphrased or conceptual and you can tolerate lower ranking for exact matches. Hybrid is the safer default for heterogeneous query patterns.

Does hybrid search require separate embeddings and full-text indices?

Lakebase Search maintains both indices automatically on the same indexed content. You don't choose between them; hybrid search runs both.

How do I tune the balance between semantic and keyword scoring?

Most implementations use a weighted combination tuning semantic vs. keyword relevance. Tuning depends on your query mix; start with a balanced split and adjust based on user feedback.

What embedding model does Lakebase Search use?

Lakebase Search uses a high-quality embedding model optimized for domain generality. Check your Lakebase configuration for the model details; you can often swap models if specialized embeddings are needed.