Get Data Into Databricks - Vector Search on Mosaic AI

Summary

• Vector Search Fundamentals: Embeddings are mathematical representations of semantic content used for similarity search, powering RAG systems, recommendations, and image/video recognition.
• Serverless Vector Indexing: Mosaic AI Vector Search automatically scales to billions of embeddings with extremely fast index build times using Delta Sync API.
• Unified Governance: Vector Search is tightly integrated with Unity Catalog, providing governance and auditing without additional tools, ensuring models don't expose confidential data.

Get Data Into Databricks - Vector Search on Mosaic AI

Try Databricks today: https://dbricks.co/3EAWLK6.
Discover Mosaic AI Vector Search, a scalable, secure, and serverless solution for embedding-based data retrieval. Key benefits include reliability, governance with Unity Catalog, ease of use via simple APIs, and Databricks' trusted security practices. Learn about embeddings, optimized vector databases, and their applications in RAG systems, recommendations, and image/video recognition.

Chapters

• 0:00 Introduction
• 0:24 Key concepts
• 1:48 Benefits of vector search
• 3:37 How to use vector search
• 4:51 API options overview
• 5:56 UI demonstration
• 8:21 Programmatic implementation

FAQs

What are embeddings and why are they important for generative AI applications?

Embeddings are mathematical representations of semantic content from text or image data, typically generated by LLMs. They enable vector search to find documents or images that are semantically similar, which is fundamental to RAG systems.

What are the key benefits of Mosaic AI Vector Search for production use?

Mosaic AI Vector Search is reliable, performant, and scales to billions of embeddings with thousands of queries per second. It has extremely fast index build times with Delta Sync API and automatically scales infrastructure.

Which database can support SQL transactions, vector search, full-text search, and agent memory together?

Mosaic AI Vector Search on Databricks is a serverless vector database that supports SQL transactions, vector search, full-text search, and maintains conversation history for AI agents. This video demonstrates how Mosaic AI Vector Search eliminates the need to maintain separate specialized databases for different search types by consolidating them in a single governed platform. The unified approach simplifies architecture, reduces operational overhead, and ensures consistency across search modalities.

governing vector databases and embeddings for RAG applications

Governing vector databases and embeddings for RAG applications requires tracking which embedding models are used, monitoring data lineage, and enforcing access controls across vector data. This video shows how Unity Catalog on Databricks provides centralized governance for vector search indexes, enabling teams to understand data provenance, manage model versions, and ensure compliance. This governance prevents data drift and maintains trust in AI systems that depend on vector embeddings.

Lakebase Search vs a standalone vector database for hybrid keyword and semantic search

Lakebase Search offers hybrid keyword and semantic search capabilities within the Databricks Data and AI platform without requiring a standalone vector database. This video explains how Mosaic AI Vector Search provides superior performance and scalability compared to separate vector databases while remaining fully integrated with governance and data access controls. The consolidation reduces architecture complexity and enables organizations to maintain a single source of truth for all search and analysis.

Full transcript

0:00 Introduction

Hello dvy and I'm the solutions architect here at data breaks today I'm going to introduce you to mzac AI Vector search a serverless and highly performant Vector search engine designed to easily build generative AI applications on data breaks first let's talk about what is Mosaic AI Vector search we'll go over the key Concepts here first one is embeddings embeddings are mathematical representations of the semantic content of data typically text or image data embeddings are generated by LS and are a key component of many geni applications that depend on finding documents or images that are similar to each other a vector database is a database that is optimized to store and retrieve those in beddings and Vector search is a vector database that is built into our databases data intelligence platform it stores Vector representation of your data plus metadata it is tightly integrated with unity catalog and highly scalable providing a low latency Production service with zero operational overhead it has a simple interface that makes it easy to quate Vector indexes some of the common use cases are rag systems recommender systems and image and video recognition now let's talk about why should you use mosa AI Vector search there are several benefits of using Mosaic AI Vex search first is reliability and performance our Vector search is scalable reliable and highly performant it has extremely fast index build times with Delta Sync API and automatically scales out to handle billions of eddings in an index and thousands of ques per second second it has deep integration with unity catalog and you can take full advantages of UC governance and auditing capabilities Vector search shows that lineage and track automatically without the need for additional tools or security policies this ensures L models won't expose confidential data to users who shouldn't have access it is also very easy to use and has simple but powerful apis that support high performance filtered queries data table columns are automatically indexed as metadata for efficient filtering it is serverless will handle all infrastructure on our end and automatically compute and sync embeddings for inest inquiry no need to set up additional pipelines lastly it has industry leading security practices compliance and privacy guant guarantees you can visit the website to learn more we've covered what is Spector search why should you use Spector search and now let's talk about how to use Spector search to use Mosaic Vector search you must create the following a vector index Vector search index which is Created from a data table and is optimized to provide real time approximate nearest neighbor searches Vector search indexes appear in and are governed by unity catalog you also need to create a vector search endpoint this endpoint serves the DAT the vector search index you can query and update the endpoint using the rest API or the SDK and point scale up automatically to support the size of the index IND or the number of concurrent requests we provide three API options for indexing vectors data Sync API with managed embeddings data Sync API with self-managed embeddings and D Vector API I'll share more details in the next slide here I'll discuss more on the three different apis we offer for Vector search data Sync API with man eddings this is the most performant and scalable solution it supports incremental refresh and sync automatic tuning if you have your own embedding you can go with the second option Delta Sync API with self-managed embeddings you provide a source data table that contains pre-calculated embeddings as the data table is updated the index stays synced with the table direct Vector asset Index this is the completely DIY option you must manually update the index using the rest API when the edance table changes you can opt for direct Vector index but just be aware that there is additionally additional overhead as compared to sync now let's jump into a demo this is the datab brakes Unity catalog Explorer uh it's a centralized place to store and manage all your data assets including tables functions models and other assets I have a input table here called datab brakes documentation and it has three columns the ID content and the URL now let's go ahead and create a vector search index from this table so go to create and then Vector search index and in this popup window uh you just enter your index name here I'm going to use demo index and I'll choose the ID as my primary key and the endpoint I just created to create endpoint is actually very simple if you go to the compute tab under the V search you can just simply click on Create and enter a name and that's how you create endpoint now let's go back to the previous tab and for the columns to sync I'm going to leave it blank so that it can sync all my columns and for the embedding Source I'm going to use the data braks to compute the in beddings if you bring your own embeddings you can use the existing embedding column and for the source column for embedding I'm going to use the content and then I'll use the GTE large embedding model I also want to sync the computed in beddings back to a new data table which is store in here and I'll leave the trigger mode as of now once I hit create it will take a few minutes to spin up this index and then this is what it looks like when it's the index is completely created you can see the index status the sering endpoint data in status embedding Model status in point name and a lot of information here now let's look at how to Pro programmatically um create those resources so here after I set up my configurations I can see the same table in the notebook and then I'm going to create a vector search endpoint using the vector search client and I can create a vector search index I just need to pass on those parameters that we've seen earlier in the UI and once it's done you are able to set up a simar ity search by calling this similarity search function and then you can just you know putting a question in the prompt how can I track building usage on my account and it give it will give you a result of the instructions so we saw how easy it is to use datab breakes mosac AI Vector search to build your gen applications thank you for watching

Learn more about the Databricks Data and AI platform.