Building Search for Agents with Lakebase

Summary

This video explains why search has become an OLTP workload in the agent era—agents constantly write chat history and memory that must be immediately searchable—and introduces Lakebase as a Postgres-based solution that combines vector and full-text BM25 search in a single SQL statement.
Lakebase extends pgvector with IVF and HNSW indexing, achieving a 32x storage reduction over standard pgvector while supporting high-recall, low-latency retrieval at billion-row scale with a 2x smaller compute footprint.
The video demonstrates building a complete hybrid search pipeline with Lakeflow, from document ingestion through chunking and embedding to hybrid BM25 plus vector queries, with instant branching for sandbox testing and no index rebuilds.

Building Search for Agents with Lakebase

Traditional search breaks under the bursty, concurrent demands of multi-step agents. In this video, learn how to build agent-native search on Postgres that extends pgvector with advanced indexing and first-class BM25, with vector and full-text indexes living right alongside your operational data. The payoff: no brittle app-level joins, just single-statement SQL that fuses vector rankings with structured filters. Watch it scale to zero when idle, burst to billion-vector scale with no performance hit, and branch instantly for sandboxing and A/B testing, all without rebuilding a single index. Then get a first look at what we are building to make Lakebase the foundation for the next generation of AI agents.
Talk By: Pranav Aurora, Senior Product Manager, Lakebase, Databricks ; Zhou Sun, Senior Manager, Databricks ;

Chapters

FAQs

What is Lakebase and how does it support AI agent search?

Lakebase is Databricks' managed Postgres service designed for OLTP workloads including the bursty, concurrent demands of AI agents. This video shows how Lakebase extends Postgres with first-class vector and BM25 full-text indexing so agents can perform hybrid search directly alongside their operational data in a single SQL query.

What are the limitations of standard pgvector and how does Lakebase improve on them?

Standard pgvector can struggle with storage efficiency and latency at agent scale; this video describes how Lakebase addresses this with IVF indexing that delivers a 32x storage reduction over standard pgvector and HNSW indexing optimized for high-recall, low-latency retrieval. The result is a 2x smaller compute footprint compared to running equivalent workloads on standard Postgres with pgvector.

How does Lakebase handle the economics of storing and searching large vector indexes?

Lakebase is designed on the principle of charging for what you search and query rather than for what you store, which means teams can index billions of rows without provisioning and paying for a large always-on instance. This video demonstrates scale-to-zero behavior when workloads are idle and burst capacity for high-concurrency agent queries.

How do you build a hybrid BM25 and vector search pipeline in Databricks?

This video walks through the full pipeline: ingesting documents using Lakeflow, building silver tables through a chunking and embedding stage, creating both vector and BM25 full-text indexes in Lakebase, and then querying them together with a single SQL statement. The architecture allows iterative improvement—swapping embedding models or adjusting chunking strategies—without rebuilding indexes from scratch.

Full transcript

[00:08] Hey everybody. Um, it's 4:10 so let's get started. Um, I'm proud of um, I'm on the Lakebased team and I'm joined with Joe. This is actually kind of weird. Like Joe and I used to be co-founders this time last year and the last time we were doing this we were like trying to
[00:20] pitch our old company
[00:34] going to be available to you through lakebased search. Um and a core principle for this is retrieval is everywhere and the end user are agents. So how can we combine you know the needs
[00:46] of agents with the new needs of search and and and give you a search engine that you need. And this is it. I'm done with my talk. You now have a Postgress instance that
[01:00] you can run hybrid search on. It's just Postgress built on top of PG vector and TS vector. It can scale up to a billion row scale and it's 2x smaller compute
[01:12] footprint than using PG vector on a regular Postgress. If there's one thing to take away from this slide in our 40 minutes, it's this. Take a picture and this is this is what lakebased search is all about.
[01:27] But one of the like so let's go take let's take a step back like actually what do the next 10 billion apps and agents need especially for retrieval. The first principle is like search is actually an OOLTP workload more than you
[01:40] think. Uh and agents make this really real. They constantly write chat history memory and this needs to be immediately searchable. So the days of search being a siloed engine that sits away from the
[01:52] old TP stack that sits away from writes just doesn't work. Search is an iterable process. What I mean by that is you're not going to get your hybrid search query right with the right parameters on the first go. You're not going to get
[02:04] chunking and embedding right on the first go. You're going to try new models that come out every month. You want to try new chunks and weights and eval. This is an iterative process and we want
[02:16] to help you with that. And it should be economical. The core principle here is we should charge you for what you use, for what you search, for what you query, not what you store. So you can go in and you know actually search and index
[02:29] billions of rows without having to provision a last large instance and keep that always running. We we wrote about the move from a monolith database to a lake base almost
[02:44] a year ago. Databases started in this world of being one box that we sold and gave you, you know, storage and compute tightly bundled together. 10 years ago, we started decoupling these individually.
[02:57] These are systems like AWS Aurora where we gave you a database engine and a storage that were separated. You could kind of scale them independently. And with some of the stuff you saw today with Lakebase, especially with the
[03:09] announcements around LTAP, we've now kind of realized Gen 3 of the database where data is stored for OLTP in cheap data lakes that are open to other
[03:21] engines. And there real benefits for this. You can isolate analytic workloads from the OOLTP workload. You can isolate bulk rights from the OOLTP workload. The reason why I talk about this is we believe search will follow a very
[03:34] similar pattern. So gen one of search engines have been you know again a a a monolith box a separate search island away from your old TPS state. Uh and you know this compute's always running it's
[03:46] it's what you index um and it's it's it's you sync data from your old TP system into the search silo. Gen two is kind of where we've been in the last couple years where we've seen a
[03:58] lot of databases integrate search into it. Postcrist being one of them with PG vector. Um and that's kind of been a one one route people take for for medium or most sized data. And for very very large
[04:11] scale data we see object native search solutions. That's still a that's still sort of the divide that we see ourselves with. And we're excited about Gen 3, which is very similar to what we've done
[04:23] with the lake base. Store data in cheap object stores, throw compute at the problem for search when you need it, uh, and and run things like data processing and chunking and embedding directly on
[04:36] storage um, and and away from the serving path. So, it's inevitable kind of what we saw with the OLTP. We're kind of hoping to see the same thing with search.
[04:48] One thing I found most interesting here is it takes almost a 30 40 years for OTP to go this kind of transformation. But a similar kind of paradigm shift happens on search in about three years.
[05:02] It's not a coincidence. So what's special about search? I think the first thing is like the type vector will just blow up your storage. So uh think about you have like 200 words text. It's like
[05:16] 1 KB. It's like super small for all TP database. But when you start like embedding them, start making them searchable, we'll be turning them into chunks and then each of them go into like a uh more than like 1500
[05:33] dimension vector which in total is like 25 kilobyte. This is like a 25x storage blow up. with this storage amplification any like uh the economics are different
[05:47] from a search engine versus like your old pure OLTP metadata store it's probably fine to keep your like OLTP database to be always on but if your uh
[05:59] storage size is like 25x uh keeping them always on is no longer option the other part is like the pattern we are seeing is like especially with like
[06:11] a lot of agents connecting to data. So by default like there's burst workloads. It will be like it's usually like not normal for a human to start like typing a 100 search queries uh
[06:25] instantly. But for agents this is just normal when you start like doing a research and retrieval will be like part of every single agent call. And another important part we found is like agent
[06:38] behaves much better if you give it it like a single system of record, single source of truth. When there's a multiple source of truth, there's like OTP database, there's a search, there's like other systems, agents will just like
[06:52] confuse itself and try to retrieve the different like the wrong kind of information from the wrong system. So search demands a new kind of database
[07:04] reality. The large storage means like this should be object store native and this the system need to support scale to zero. We have elastic compute unsharded storage and for search specifically we
[07:18] want constant experimentation. So copy and write branching is like no longer a good feature. It's like a must. So what we want is a lake base. It's
[07:30] like a fully uh serverless operational system built on shared object store with nvme cache and object store as the source of truth.
[07:48] Cool. Let's build it. Well, I guess like we have lakebased Postgress. We announced it like G uh earlier this year. is being announced like last
[08:02] We we have a Postgress database. We change the storage layout. So it's like fully integrated into the uh separation storage and compute architecture.
[08:19] So you may ask Postgress already have PG vector. Can we just like put PG vector on lake base and call it a day? Well, it's not quite there. Let's take like a 100 million row uh
[08:33] search workloads and try to implement that on PG vector. So, first it's like let's build the index. It turned out like we take more than 40 hours to build the index
[08:48] and 100 million rows like not that large for big data. and 40 40 hours is just like not acceptable for any agentic workload. Moreover, like this will require 256 CU
[09:01] instance with more than 500 GB RAM. Uh this actually is not a available on base yet. Also, we just take about the storage footpoint footprint. The total data set
[09:14] will be like 250 gigabyte with the in index size will probably be 300 GB like larger than the data set and this whole thing need to fit in memory to make it
[09:26] performant then it's like this is just no go. uh plus like if things are owing memory to get performance we cannot get like most of those uh nice lake based
[09:39] features like branching instant scale up and down let's look into the problem like why would we hate that the problem is because of algorithm like this is the
[09:51] most fundamental one uh most of the small scale like uh golden standard to build a vector index is something called ashsw. It's a pretty nice graph
[10:03] algorithm but the uh process is like you build a graph of all the vectors uh and then upon search you start re traversing
[10:15] them to find the target. But uh the nasty part here is it's all dependent tra traversal. you need to traverse to one stop and then
[10:28] you you'll know like what's the other step you'll be heading into when everything's in memory everything works perfectly on local SSD it's okay
[10:41] but when the storage can be object store this becomes like extremely high latency and unacceptable
[10:54] so in order to build vector search index For agent scale, we want the index that's like memory light and fewer round trips and prefetchable. So it's actually can work with arbitrary kind of storage back end. And this index must be fast to
[11:08] build. So agents can like just uh start gathering knowledge and index them and search them. So SSW is not nongo.
[11:23] Then we need to start innovation here. The first part is kind of easier. We know vectors are large. So the first step is we need to find ways to make vectors smaller or at least like what's represented in local storage or in
[11:36] memory smaller. This is like a
[11:48] is like we will be doing a random rotation of the uh input vectors and after that we have some property so that it's mostly universally distributed and
[12:00] then we can only use like one bit to represent originally like 32-bit vector. uh just a link to this paper and a nice property here is actually guarantees
[12:12] like even when we threw away 90 like 96% of the data like from 32bit to one bit the system actually have like a bounded precision when we compute the vector
[12:24] distances this is very nice we cut the storage cost by 32x at least like what's need to be stored locally
[12:37] and Then things like SNSW is not possible. We need some kind of IVF or cluster based index. But then index build was still very slow. Uh in order to find clusters, we usually need to run like a what's called
[12:51] C mining algorithm to just divide the uh clusters and trying to find like centroids of each of them and then do it like repeatedly until we uh converge on
[13:03] a set of centroids. We need this to be faster and a lot faster. So the first thing we do is like we start building this hierarchal cing algorithm. So instead of like just
[13:16] always working on the full data set we will be start dividing them and then after a few rounds of clustering start clustering like each subsets and then further going on. This is a recursive like uh
[13:29] itself like kind of improve like the speed up by 10 lakx and then what we are start exploring is since we have those like massive uh distributed compute engines we
[13:42] started and then this like here hierarchical came in is actually paralyzable since a lot of the work is independent we start to use like the spark engine to build it instead of postgress
[13:59] so this is what The index would be looking like with like a heracle IVF plus Rabbit. We'll be trying to store like the upper layers of centroids and metadata in local and then only like
[14:12] uh only do fashioning once and touch touch object store to do a final reranker and it turned out like this thing this architecture actually works perfect with
[14:25] leg bases catch layers. Think about the different workloads. So if you are building like a search bar or something that's always on, you need ultra performance. Then the goal is like let's
[14:37] build put everything in local in local memory and keep it always on. Um we can put like all the metadata centroids all the bit vector of the rabbit Q and then or even the four
[14:49] vectors for re reranking all in RAM and then this will be extremely fast extremely uh high concurrency. uh but for many other use cases we don't
[15:01] have like that extreme it's like more bursty workouts then uh what we set up the system would would allow us to
[15:13] only put like the hottest data like metadata or centroids in RAM and then we we have the bit vector on local disk and full vector mostly on our storage system page server in this setup we can
[15:27] actually autoscale our compute. So we can scale to like super small when there's not many query and then scale to pretty large when there's a burst of queries to make the cost like very efficient and then for the other kind of
[15:41] use case when we people building rack apps a lot of them may be idling for like mononges with no one quering them then we will be just storing everything on cheap object store or page server
[15:55] then we can effectively scale this computer to zero uh without worry about it burning any uh additional money here.
[16:10] And another part like that's interesting about search is search fundamentally is a serving workload. Uh but however it also fundamentally have some uh non-serving part of the workload. uh usually start from like you have a
[16:24] row documents we'll start preparing them chunking them embedding the data do a lot of processing and then you need to start ingesting and building the index and probably also even rebuilding the
[16:36] index all of this like extremely uh compute heavy there's what this is where the lap or the actual leg based architecture
[16:50] uh comes to the If you haven't uh heard the time the term lap is something we're announcing earlier this morning lake transactional analytics processing
[17:02] which that it means is like we have lake based compute we have lake house compute which we can have like those two different kind of computes operating on the same storage layer. Uh so depending
[17:14] on different kind of workloads or different kind of jobs within a workload we pick the most appropriate computes to uh to achieve that. How does it look like for search?
[17:27] We will be be uh building all this like prepare the search doing chunking doing embedding here and then in the future we'll even just do the index build with
[17:39] lakehouse compute and then after this is done in instead of like giving them to the lakebased compute to ingest lakehouse can directly ingest to this storage layer and then lakebased compute
[17:52] will just like serve them as is so next question is it's very cool we start building something with lakebased
[18:04] architecture but how does that fit into lakebased postgress cool so the reality is Postgress is loved by agents and Postgress has a great
[18:18] ecosystem for search there's PG vector which is actually available on lakebase and almost every postcrist provider This is kind of your de facto vector search solution in Postgress. It's built on IVF sorry it's built on HNSW indexes
[18:30] and it has a great ecosystem that people have built on top of um and sort of standardized on the vector types operators um lots of semantic search and rag applications are built on PG vector. You
[18:44] also have TS vector for full text search with gin indexes and this gives you foundational search capabilities in in Postgress. So Postgress already has a good ecosystem. Can we just enhance it? And
[18:57] this is a very very key property of lakebase. It is not a replacement of these two ecosystems that exist. They are enhancements. And as a result, we
[19:09] keep 100% compatible with PG vector types and TS vector types. You can take it to any Postgress. There's no uh you know weird lock in we're doing here. We're just building better indexing
[19:21] methods and better optimizations for the lakebased architecture into Postgress. So let's make them better. First extension what we're launching tomorrow in beta across lakebase in in
[19:35] in AWS and Azure is lakebased vector. Lakebase vector is a drop in compatib like a drop-in enhancement on PG vector. Um and the way to think about it is it brings a faster cheaper index with all
[19:50] the optimizations that Joe shared about on the indexing uh on the index type designed for object store as well as building the index um into Postgress and lakebased text brings the first BM25
[20:02] full text with full compatibility with TS vector types and operators and these are built as Postgress extensions. No migrations from your existing setup with PG vector and TS vector and yeah
[20:15] available tomorrow in lakebase in beta. If you now look at like the data life cycle or where a query comes in on the left hand box I have Postgress with these two extensions you know it can
[20:29] either come in through a SQL query or an MCP hunt data the the working set that you're actually using to serve is stored in our local file uh in our LFCs with our shared buffers and this is like the working set of data whereas the
[20:43] lakebased storage built on the LTAP architecture is like the full source of truth and this architecture works really nicely where you're only working on a small subset of data but you have a massive you know data set to index and
[20:55] serve out of the the unified storage. So queries come in are served out of this you know fast NVMe cache where this the full durable storage is the lakebased storage which is now you know part of
[21:08] the LTAP story and we are able to deliver pretty insane performance uh while keeping recall. So like a lot of benchmarks on vector search end up being kind of you know
[21:20] hard to understand. Recall is a very important kind of metric to anchor these uh these these these results on. Um and you can see sort of sub 30 sub 30 millisecond P99 latency at high QPS on a
[21:33] 100 million row benchmark. Um and the cool thing here is we're running this on only a 90 CU instance on Lakebase. So this is almost a third of the size if we were to do the same benchmark with PG Vector. So go back to you know really
[21:47] good performance add economics that you can rely on and the reason why we can get run this on smaller hardware is because the entire index doesn't need to fit in memory to be to get this kind of performance
[21:59] and yeah this is my point so it's a much smaller compute footprint to pg vector um yeah you can you can you can see that and the the last point here is index builds are a lot faster it's still not ideal right now it's still at 90 minutes
[22:13] with a 26x improvement over pg vector, but we have a path to make this minutes with um offloading some of this to spark and distributing index builds on on a on spark and not talking to to lakebase as
[22:26] a part of this sync. A cool thing here is with with lakebased vector you can scale the instance to zero and on first query you can get you know responses back in
[22:38] in just seconds right so a p90 for the first query after a scale to zero is 1.13 seconds on this 100 million uh data set benchmark so the way to think about it is now you can actually go out and
[22:51] index all of your PDFs and your knowledge base with lakebased search right and then and have that ready for rag that can scale scale to zero when you're not using it and on first query we can give you reliably second level
[23:04] retrieval you go this goes you know completely changing the economics of how you thought about search engines before which is a always running large compute based on the index size maybe data is not often queried
[23:19] so you should rethink your search TCO and your footprint the first one I I I've seen you know a lot of our early design partners work towards is index fixing large knowledge bases. Um, and a
[23:31] common pattern I've actually seen is like having a lake base per agent or per knowledge assistant and having that scale to zero when not needed and and and scale up as as as people start
[23:43] querying it. Um, you'll see some announcements here tomorrow with agent bricks and how they're using lakebased search um to power some of the the new capabilities that they're launching. A lot of use cases on lakebase are
[23:56] actually agent memory and chat use cases. the chat history, how are users using my chatbot? I want to, you know, use uh I want to store all of this in Lakebase. And these end up being extremely bursty. Um you know, you're
[24:08] you're sure you're you're storing each agent run agents can go spin up hundreds of sub agents. Um and the way we work the way we work here is designed for this kind of workload because we just pay for ex um active usage and we can
[24:20] scale up really quickly um on these agent turns. And for workloads that are always on like your search bars, a way to think about it is you need a much smaller Postgress or a much smaller
[24:32] compute instance because the entire index doesn't have to live in memory, right? So if you're going from an extremely idle workload where you're just wanting to, you know, just index the codebase and have it ready in case somebody needs to to retrieve it,
[24:46] lakebased search is a good option. For the middle use case where, you know, you expect this to be always running, but there might be very bursty patterns. Lakebased search is a great uh story and for use cases that are always on you
[24:58] need very high performance you know p99 of milliseconds lakebased search is a good solution some things that we've been pushing the team over the last week is like how does
[25:10] this work at billion scale like we always talk about a 100 million benchmark in the vector search community but 100 100 million is quite big billion is like absurdly big and and and in our
[25:22] early testing we're able to deliver millisecond millisecond level latency on p50 um for for billion billion scale vector search on lakebase. And some cool things here is like actually ingesting a
[25:35] billion rows of embeddings into lakebase only took an hour and a half. Um I know that sounds like a lot but it's like I can walk you through some of the optimizations we're doing here. Um and the next steps are can we paralyze the
[25:48] index build on this scale, right? So it's not going through the lakebased compute. It's not impacting your serving workload. This is offloaded to an engine that's designed to do this kind of distributed processing. And as Joe mentioned, the index design is actually
[26:00] very parallelizable. So what does this give you? This gives you one Postgress query, one Postgress backend to kind of do it all. You're already using Postgress to build your your apps, your OTP, and you're already
[26:14] using that to serve it maybe, you know, some of the context uh to agents with agent memory, right? And this kind of completes the entire story. You can run all of this in one agent back end with with search with true OOLTP joins and
[26:26] filtering. A common thing we see is like you want to filter on tenant on users so that agents are only showing data to users that they can see that lives in an OOLTP system compar like combining that
[26:38] with search uh an external search system was always really hard and now you can push these things to the database layer um and give the make these problems of the database to solve. So yeah, it's like you get tenant isolation. You can
[26:50] think of it as like the one backend for agents. Agents love Postgress and the lakebased architecture is actually very designed for these origin ergonomics. And now you have search capabilities
[27:02] that are pretty world class to give you this one backend for for for your agents. I want to go into deina. I've been speaking a lot and I know it's like 5:00 4:00. Um so let's go into some demos.
[27:14] The first thing I want to show is like a we have a very common pattern of people throwing in raw documents in the lakehouse um processing it with the lakehouse. We have some great tools here like AI parse
[27:26] document AI query to create these embeddings and then you create your sort of silver tables which are like your process data with a bunch of embeddings. What we can do today is load this data
[27:38] extremely efficiently into lakebase and lakebase becomes your serving layer. So I would think about it as like in your medallion architecture, your gold table is now lakebased with these search
[27:50] indexes built in. Um and loading data into it is really fast. Um and you can go and connect your agent to to lakebase. That's kind of the rough flow.
[28:05] And so here I start with in the lakehouse side I am in a notebook. I start with, you know, some files in my volumes and I'm going to use some of these great functions um to to
[28:17] go in and parse this document, you know, get rid of any unnecessary stuff with AI parse document. So that's the first step. Um and then from there I go in and create this sort of parse doc table
[28:30] which is like I would consider my my my bronze table in the architecture and I can check how it's working. These are some financial documents. So a bunch of letters from you know different banks and PDFs across uh earning reports and I
[28:45] can then go and uplevel this to create my silver table which are the chunked versions of these documents. So I start with raw documents. I prepare it to be chunked and then here I use the function
[28:59] AI prep search. These are all lakehouse functions that exist today um and you know run on spark. They're extremely distribut um they're extremely paralyzable. And here I go create my silver table.
[29:12] You must be wondering like why am I showing you Spark and these functions? But hold on. Um and then after I create these silver tables, uh I go in and
[29:25] create my embeddings on these tables. And I can I can just do this with a materialized view in the lakehouse. So I take my silver docs table here create a materialized view calling this model
[29:37] that exists in my in my warehouse. All of this is running within my data bricks platform. So the bottom sort of query here is to create these embeddings with uh an embedding model. And from here I
[29:50] create my gold table. So this is like data that's ready to serve. What we have with lakebase today is something called synced tables. Um, so sync tables are a mechanism to get data
[30:04] that lives in open formats in your lakehouse extremely efficiently into lakebase to serve. This is part of the LTAB story. It's how we get coler data, cache it and serve it in a row oriented
[30:18] format and build indexes as a part of it. Um, and now this works really well with vector search. So create my gold table, create a sync table to get this data into
[30:30] um into lakebase and I would just you know name it. Here we we handle all of the things to do with like large uploads into lakebase. So you don't have to think about lock management about ownership. This is all fully governed by
[30:42] UC. Um and cool. Now I get this table in Lakebase and I'm in my Lakebase UI. I get my table with all my embeddings after it's been cleaned up. It's a very
[30:54] large table and now it's available in Lakebase. So that's the first step we see with a lot of customers like how do we take raw documents, process it with a lakehouse and now you can get them into lakebase to serve extremely efficiently.
[31:07] And a a benchmark you should think about is like previously to load 500 GB of you know data especially embeddings into any system especially a search system you
[31:19] write your own custom JDBC sync. This was like the manual reverse ETL process. You know we've seen these kind of workloads take hours to run. We launched sync tables last year which was an mechanism to get data into lakebase. We
[31:32] got that time down to to hours to to you know single-digit hours. um with some cool mechanisms with with um with postress copy and what we're launching today with with sync tables is it's
[31:45] powered by direct write mechanism and we can get this down to minutes. So we can get a 500 GB table ready to serve after being indexed in minutes. So that's the first step of search which is like you need to prepare the documents, you need
[31:57] to get it in in place to search and this process we can make really efficient with lakebase. Another common pattern we see with customers is like you're not going to get these, you know, the the the the
[32:09] performance you need out of your search engine on the first try. And here, lakebased branching leans in really well. So, you can branch data, iterate on it, and iterating here might be trying new indexing methods, trying new
[32:22] embedding models, trying new chunking strategies, and if your eval fails over time or you got users complaining that, hey, the change you rolled out sucks, you can instantly roll back. And these are all built into Lakebase today. It's
[32:35] this mechanism. It's called copy and write. But that makes a lot of teams ship a lot faster with lakebased and feel confident in rolling out changes, prototyping uh with different parameters for search. Um and I want to show you
[32:48] how this kind of looks like like so this was me last night. Um and I was just playing around with some queries on on Postgress. So this is within a Postgress. I'm just writing Postgress
[33:00] SQL. I'm in a UI called table plus and all of this is just agent generated because I don't know how to write SQL very honestly. Uh and if you look at the query I'm running this looks exactly
[33:12] like a PG vector query, right? So this is a vector search query I'm running. It looks exactly like what you would do with PG vector. And so your agent's going to go generate it. It is actually just a PG vector query but it's using our index underneath the hood. And this
[33:25] is the PG vector operator for vector search. Um, another cool thing we have is like a lot of the functions that you love from the lakehouse will be available in lakebase as well. And what I mean by that is you can uh embed user
[33:39] questions on the fly with our AI query which will call an embedding model and then run vector search on it. So otherwise you would go in and you know figure out how to call your embedding model get the get the embedding results back and then run vector search. We can
[33:52] kind of make this all part of the same Postgress query. Um, so cool. That's a vector search query. It is fast. I wish I could show you some timings somewhere. Um, but that's not the point. Sorry, let
[34:06] me go back.
[34:20] Um, I want to show you a full text search query. So like a a agents love not just semantic search they love you know full text search and a common way to know this is like most agent hardnesses have like built-in tools for GP which these require like full text
[34:32] kind of capabilities. So we're taking enh we're making enhancements to the TS vector functionality again keeping syntax very close to TS vector and now you can run a BM25 query using our tokenizer and this will give you exact
[34:45] match on you know when to not just semantic meaning but like exact keyword match and it's the combination of two that lead to very high success for agents in terms of hybrid search and giving you the recall you need.
[34:59] So this is how I'd run a you know a hybrid search sorry a full text search query. One thing I do want to show is and this is bad uh graphics but the reason why search is so finicky is like
[35:11] these are two queries with just the only difference is the tokenizer. I'm using a simple tokenizer in one and the English tokenizer in one and the tokenizer is returning totally different results. So just like there are so many parameters
[35:24] in terms of you know which tokenizer to use, how to chunk these documents and then how to combine hybrid search in terms of the weights. They're just it's it's it's quite finicky and an old way
[35:36] we used to do this and I've seen customers do this is like just go create hundreds of tables. So I have you know one from BM25 I have like okay this is the wrong BM25 this is now a different
[35:48] version tokenizer and it's like okay sorry like this is the wrong embeddings one I'm going to try new embedding model I'm going to try the new open AI model and you just end up with hundreds of these tables it's very hard to figure out which one is actually working
[36:01] production which one is leading you to good results you know you need to constantly test and eval this and a very nice mechanism we've seen is branching so branching is a native capability of lakebased that creates copy and write
[36:14] reads. You're not just duplicating data. You're just creating a new you know metadata pointer to this table. I want to try a new tokenizer like you know I don't like the simple one. I want to try the English tokenizer. I would create a branch of this data
[36:27] you know get this new endpoint my application to it. Maybe log some of these evalu creating new indexes maybe add new data. maybe change the tokenizer, run some of
[36:41] these queries in the back end and if I'm not happy, if I get a lot of, you know, spike of user complaints, I can just restore this instantly. So I can go restore from a particular time and that's it. So it's this mechanism of
[36:55] being able to branch try new indexes try new you know types of data maybe maybe tokenizers and then you have the ability to go back in time and restore it you know if you know that the eval failed or customer started complaining um that
[37:09] give you like this continuous search experiment right and this is very common in in in workflows we see customers roll out um so yeah that is that is lakebased search today you're going to be doing a
[37:21] lot of this customization yourself. There is work for us to be done to do on you know making this more seamless combining this with like the eval stuff that we have on the lakehouse. Um and hopefully you don't have to go in and
[37:33] tune these indexes um and your entire search pipeline to be very precise. So take away from this talk is lakebased Postgress is Postgress built into the
[37:45] data bricks platform. from its mission critical OLTP and now it has a search engine that is extremely wellbuilt and designed for agents. So lakebase should be kind of how you think about you know the the back end for for serving agents
[37:58] the data and context it needs and it has search capabilities to finally power that. So what will you build? Cool.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.