AI Search: High Quality Retrieval Made Easy

Summary

Databricks AI Search simplifies building production-grade retrieval with hybrid keyword-plus-semantic search, automatic reranking, and domain-tuned models—eliminating the need for large teams of experts to tune complex DIY search engines.
This video shows how to create a hybrid index with smart defaults, evaluate retrieval quality using NDCG metrics, and build feedback loops that continuously improve search relevance for a specific corpus and query distribution.
Customer results presented include Eurowings, Rue Group, and BMI Healthcare, demonstrating measurable improvements in search quality across multilingual, high-volume enterprise search and recommendation use cases.

AI Search: High Quality Retrieval Made Easy

Measuring and tuning performance and retrieval quality for Search bars, recommender systems, entity resolution traditionally required quarters of tuning, a team of domain experts, and ongoing maintenance that kept engineers away from building the actual application. Databricks AI Search makes it simple to build a state-of-the-art retrieval system, with quality bespoke to your application. In this talk we’ll show you how easy it is to deploy a production grade Search with the latest advances in hybrid (keyword-plus-semantic) search, reranking, domain-tuned models, evaluate retrieval quality for your corpus, and build feedback loops that continuously improve your retrieval quality. You'll see what's shipped, what's next, and walk away with a production-ready stack and a working playbook for measuring and improving retrieval quality
Talk By: Ankit Vij, Engineering Lead - AI Search, Databricks ; Sanjit Jhala, Software Engineer, Databricks ;

Chapters

FAQs

What problems does Databricks AI Search solve compared to DIY search engines?

DIY search engines require large teams of search experts to tune hundreds of configuration knobs for quality, performance, and latency—while still falling behind on a constant stream of feature requests and traffic changes. Databricks AI Search provides state-of-the-art hybrid retrieval with smart defaults that deliver high quality out of the box, without requiring specialized search expertise.

How does Databricks AI Search evaluate and improve retrieval quality?

AI Search uses NDCG (Normalized Discounted Cumulative Gain) as the primary retrieval quality metric, allowing teams to measure relevance for their specific corpus rather than relying on generic benchmarks. This video demonstrates how to select an evaluation strategy, run load tests, and build feedback loops that continuously improve retrieval quality based on real user signals.

What use cases does Databricks AI Search support beyond traditional keyword search?

Databricks AI Search supports search bars, recommender systems, and entity resolution use cases, all powered by the same hybrid index infrastructure. This video also shows multi-language search with filters and sorting, and describes customer deployments at Eurowings, Rue Group, and BMI Healthcare.

What is a fine-tuned reranker and how does it improve search results?

A fine-tuned reranker is a model trained on domain-specific relevance signals to reorder candidate results from the initial retrieval stage, improving precision for queries that matter most to a given application. This video presents fine-tuned reranking as the third act of the AI Search quality improvement workflow, following hybrid indexing and load testing.

Full transcript

[00:09] Hi. Welcome to this uh session about AI search. Uh I'm Sanjeev. I work on Databricks AI search. Uh I've been working on search, particularly search quality, for about the last 20 years. Um
[00:21] a lot of that in the consumer space and then more recently in enterprise. >> Uh and hi, my name is Ankit. Um I'm the TL for the team. I've been part of the team since forever from the very start.
[00:33] Um and I've been working in the search space for more than 10 years now, as well. >> All right. Before we get started, um everything in this talk is for
[00:45] informational purposes only and our road map is constantly changing as we're learning from customers such as yourselves. Um so today we'll talk about what's AI search,
[00:58] uh how to build a production-grade uh search engine and deploy it on AI search, and then finally we'll give you a sneak peek on some of the features we're currently working on.
[01:13] So, I've been building uh search engines for a while now, and every once in a while, um you know, I'll have a pointy-head manager come up to me and say, "Hey, I'm not happy with the quality of the search results." And usually this is pretty vague
[01:26] feedback. Uh there's no um queries, there's no uh metrics, there's nothing you can measure. Uh it's mostly just a sort of vibe. Um
[01:39] and without some of that, it's really hard to know what to fix um and what you need to do next. And part of the reason I think uh a lot of us that work on search are stuck with
[01:51] this problem is uh we're sort of stuck between uh two not great solutions. Uh the first is sort of DIY search engines. These do provide a lot of
[02:04] flexibility, uh but I'm sure many of you have shot yourselves in the foot uh misconfiguring these. I certainly have. Uh these typically need teams of uh fairly large
[02:16] teams of search experts that know how to tune hundreds of knobs to make sure that, you know, you have the right performance, quality, uh and latency, and so on. Um
[02:28] and then, you know, sort of you need to constantly be keeping up with your uh changes in your traffic stream. Uh you're going to be doing all this while you get a stream of uh feature
[02:41] requests from your product team, and so on. Uh the second solution is sort of the canned solutions. Um these can often be black boxes, uh so there's no real way to tune relevance
[02:54] uh specific to your queries and your corpus. Um and it's often, you know, these are fairly expensive solutions. They're hard to scale. It's kind of hard to keep up with your traffic uh both in
[03:06] terms of quality and performance. So, we believe at Databricks that uh high-quality retrieval should be default and not a craft. Uh and that your search
[03:20] engine's quality and performance should be simple and constantly improving with your traffic. So, with that, uh we're introducing uh Databricks AI
[03:32] Search, formally known as Vector Search. Um and we're going to be shipping best-in-class quality whether your application needs full-text search, um vector retrieval, or a hybrid of both.
[03:45] Uh we also offer uh a fairly high-quality reranker, which you can further fine-tune on your queries and your data uh to get even better quality.
[03:58] Secondly, we're in the business of making sure you can go to production as quickly as possible and deliver great experiences to your users with sort of minimal amount of fiddling
[04:12] with all kinds of knobs. So, you don't have to worry about those 100 fiddly knobs. We will get you into production and you know, we we scale pretty significantly so you can get up to 10K
[04:25] QPS, billions of documents in your index, all with four nines of availability and no babysitting whatsoever. And finally, we're incorporating
[04:38] feedback loops. And so, what that means is that your search engine is just going to get better over time.
[04:50] Now, many of our customers are using AI search for a variety of different real-time search use cases. The most obvious of these is, of course, search bars. So, this is things like product search,
[05:04] you know, maybe finding needle in the haystack across, let's say, for communication audits, things like that. The second use case that's fairly common
[05:17] is building often personalized recommendation engines. So, this is things like, "Hey, find products similar to one that a customer is browsing, suggest related content, images, and so
[05:30] on." And the third is sort of matching. So, this is find entities similar to a certain entity. So, this could be matching job applicants to job listings,
[05:44] finding duplicate or near duplicate records, things like that. And now I'm going to hand it over to Ankit to make some of these use cases more concrete.
[05:57] >> Um thanks. Thanks, Ankit. Uh so, before I I deep dive, I wanted to just do a brief overview of what the architecture is for AI search. So, we built it we
[06:09] rebuilt it from scratch uh and it has complete decoupling of storage and compute and not only decoupling of storage and compute, it also has decoupling on the uh serving as well as the ingestion layer that provides you with the maximum
[06:22] possible uh flexibility in terms of workloads um and also can scale to billions of records. Uh so, this is this is the architecture that is powering our
[06:35] uh high QPS as well as high scale indices which Sanjit just talked about. And and customers have been adopting it very very widely. So, I wanted to give a
[06:47] few examples here. Like for example, Eurowings uh power their website search through AI search and and very recently they they saw a great lift in relevance all while
[07:00] meeting their low latency requirements that is needed for a real-time search bar. And Rue Group uses it for recommendations. So, they have uh two websites where AI search powers their
[07:13] personalized style recommendations. And and they've seen a meaningful lift in in conversion rates uh all while keeping the latency uh in in in checks. And finally, we have BMI who's using AI
[07:27] search for matching. Uh and uh they recently saw a 40% improvement on the hardest queries of data set. Uh and it basically uh they they gave us
[07:39] a very good quote saying that yeah, it was pretty pretty significant improvement for what they were seeing before. Now, I want to go and show you how you can build a high-quality high high QPS
[07:53] search bar just like Eurowings and Rue La La Group with Databricks AI search in in three acts. So, the first act is is search and quality. Uh all needs to be working
[08:06] quickly. We we start with POC and I'm going to show you how you can do all of this with Genie code uh so that you can start building applications and start playing around with them very quickly.
[08:18] So, let's change modes. So, the first thing I wanted to show what we are building. So, we already have uh the UX and the UI folks gave us a website that needs to power our search
[08:31] bar. But, as you can see, there's various components on it. So, there is the search bar that power that needs to return a bunch of results. There's also sorting happening on relevance, price,
[08:43] rating, name, etc. And there's a bunch of aggregations also getting powered like category, brand, price, consumer rating, so that with every search query, you start seeing uh
[08:56] these categories and brands so that customers know what they're searching for and they can also potentially filter for it for ease of use. So, let's get started. So, the first thing that we need is uh is an endpoint.
[09:11] The The way it would work is we we would ask Genie uh to create an AI search uh endpoint uh that can uh called, let's say, days 2026 demo
[09:26] life. And that's pretty much it. Like, Genie would start looking at the uh at the query that we have. And that's pretty much it that you need to give Genie. It would automatically figure out what
[09:38] endpoint it needs. It already saw that it needs a standard endpoint because it gives you with the maximum possible flexibility. And it it leaves advanced options like budget policies and target QPS that can
[09:51] be configured later. And once it's done that, it automatically navigates you to the page so that you can start building things there. So, once we are here, we need indices. And I'm going to ask Genie to to create
[10:05] an index for us. So, let's create a hybrid search index called uh on the source table uh let's say days 2026.data.
[10:22] shopping uh data. Uh and all while using smart defaults. That's pretty much the query that you need to give. Going forward, you don't need to worry about any of the private
[10:34] key or the public uh the primary key selection, the embedding model selection, everything would be taken care by Genie. So, Genie, the way it would work, it would start reading uh the source a table to figure out what defaults it
[10:47] needs to use for the index. So, here it picked up that it would it needs, let's say, the primary key needs to be doc ID, the index type needs to be delta sync. It also picked up the embedding source
[10:59] and the embedding model. And once it does that, it starts creating the AI search index for you. So, once it creates the index, it shows you on the on uh on the endpoints page that yes, the index is up and running. And then
[11:12] finally, it opens up the index for you to start uh tracking the progress. Now, this is an example of hybrid index, but we can also create uh other types of
[11:24] indices uh on this. So, you can create a pure full text index, which is in beta now, uh and that would and AI search would automatically handle all the schema
[11:36] as well as the configuration needed to power some of these faceting or aggregations that is needed as well as sorting. So, we also recently added support for language detection, which would allow you to like if you
[11:50] have a table that has multiple languages, we'll start looking at automatically what analyzers or tokenizers are needed so you can pretty much focus on what is needed for your search use case.
[12:03] Um, and all of this is done pretty much seamlessly. So, you don't have to worry about any of these configurations that are happening. Now, it takes a few minutes to build the
[12:15] index. So, once it is built, it pretty much looks like this. So, it has all the data that you need. You have the brand, the category, description. It also has a Spanish description that we need. And it
[12:28] try and the and the vector. Um, so once the index has been created, the next challenge is to figure out what type of queries I want to use on this index. Like do I need a full text query or do I
[12:41] need a hybrid query or or what should I use? And in order to in order to do that, we we recently launched the search quality evaluation tool, uh, which is in beta
[12:54] and which is in beta now. And we that helps us use industry metrics uh, to figure out what is the best query type for your index. But how do we how do we start
[13:06] this? So, I'm going to use Genie for this as well. Uh, so all you need to do is click on this button called evaluate search quality. And that's it. And once you hit on this
[13:18] button, it would it would kick off the the search quality evaluation on this endpoint. Um, and it it it basically would show you the progress uh, once it is once it has started.
[13:32] Now, we did all of this via Genie, but you can also schedule this via jobs. But eventually, the goal for AI search is that we want to do this automatically. So, you don't have to
[13:44] like worry about when to run this evaluation, what query types you need to pick. All of this would be done automatically for you, and all you can need to focus on is what index I need to
[13:56] build or what is the data that I need to be be in in order to power my search. All of the quality and everything would be taken care by by AI search automatically.
[14:09] So, once so it it basically takes a few minutes for this to run. But I I already had something pre-prepared for this. And once it is done, it would show you what is the
[14:21] highest quality on the index, what of what query type, what is the status, what is the experiment, when was it last run. And as you can see right now, it it gave us that hybrid with re-ranker has the
[14:33] highest possible quality. But how did it come up with this? Like, what did it go what did it do to figure out what it needs to do? So, we going to understand the search quality evaluation also using Genie
[14:46] codes using this button. So, once we click this button, like Genie automatically knows what experiments was run and what was happening and it starts analyzing those experiments to give you give you a summary of what why it chose hybrid plus
[15:00] re-ranker. So, as you can see in this, like it it did multiple strategies and there was a clear difference between what the hybrid gives us, what does the hybrid plus
[15:12] re-ranker give us. And it it basically chose gave you why it chose the hybrid plus re-ranker using the the industry metrics that we use as part of the evaluations. And then it gives you uh steps like
[15:25] switch to hybrid re-ranker, find inner re-ranker if you want even more quality, and other other suggestions as well that you can improve chunking and data prep uh to in order to improve your quality furthermore.
[15:38] So, once we Now we have the index built, now we know what the quality is, we need to come back to our app to like make sure everything is working fine. So, I already hooked this all up uh in
[15:50] the back end. So, once everything is hooked up, now you can see that there is categories, brands, everything is getting powered uh here. Um and now we can run a few queries here to to make
[16:03] sure everything is is up and running. So, I'm going to make a simple query here. Uh it's it's for headphones. Uh and once as you can see once the query hits, the category counts changes,
[16:16] everything changes, the the uh even the results change. And now I can filter on let's say Beats. Like Beats is something that I personally like. So, once I filter down to it, now it it has both
[16:28] aggregations plus filtering running together. And then I can also sort on some of those things uh to like make sure that I'm working end-to-end as a full website. So, once we have this, there's another
[16:40] thing. I wanted to test what the language support was looking like. That is something that I I also mentioned. So, we created the index with this support. And now we're making a query uh for noise-canceling headphones. So, once
[16:53] you make the query, we see that it's still giving you the right results, what we were looking for. And you can also filter it down to like make sure it's working it's working perfectly. So, that bri- uh that brings us to the
[17:07] uh to the end of our first POC. Uh and I wanted to briefly recap of what we learned in this POC. Uh so, the first thing we learned is we can create endpoints and delta sync indices all by
[17:21] natural language prompts in Genie code. We have full text support. We have multi-language schema detection automatically being done by AI search, so you don't have to worry about any of this. And we ran search quality
[17:33] evaluations uh, all and saw the quality lift all in Genie. And the And the prompt here is like all of this can be done in a few minutes. But on the old stack, this was easily a
[17:45] quarter. So, once we this Once we're done with uh, the POC-ing stuff, the next part comes is we need to productionize it. And productionizing means that it's not
[17:58] just the quality that we're looking, we're also looking at the scale and the performance of it. And we need other things. We need reliability and observability as well. So, I'm going to show you next how you can do all of this via Genie code.
[18:20] Okay. So, the the first thing that we need is to simulate uh, is to simulate the the load on on our endpoint. So, it's it's very easy to simulate a load. Uh, we have this uh, notebook uh,
[18:32] that you can easily use on any of your endpoint or indices to like load test this. Uh, and all you need to provide is a bunch of configurations on the um, on the endpoint or what indices it's being
[18:45] using um, and and that's pretty much it. And once you do that, you can go back to the endpoint page and it has now metrics on it. So, it shows our endpoint health, it will show you request per counts, um,
[18:58] also the which routes that you're using, and also looks at uh, the request latency. Now, you can get for all you can get for ANN, you can use for hybrid, you can use for full text,
[19:11] and also the error rate of what type of errors are happening on the send point. But, what should I do with this information? Like, I got this all my information, but how should I debug this for uh for me? And we're going to use
[19:23] Genie for it as well. So, I'm going to tell Genie to to basically debug uh this endpoint. Uh Uh and use all the metrics that it has
[19:35] uh and uh and help me understand what is happening uh with this endpoint. Uh and that's pretty much it. That's the prom that we give to Genie. Genie has all the metrics. Can look up uh not only the
[19:48] last hour, but also 7 days worth of data. And once it has all of these metrics that it is receiving now it has received, it starts to make recommendations for you. So, it it
[20:00] thinks about what were the findings, what was missing in the endpoint, and what steps we could take to finally make it working. Uh and the And once it has done it, it
[20:12] it would show you what it came up with. It it tells you that oh, you were not using the the non-optic You were not using the network optimized route, so that's why latency was bad. The QPS was also not up to the
[20:25] mark, and it gives you recommendations on it. Like, move to network optimized route, set a higher QPS, uh and we can do all of these things with Genie automatically, so that you can like
[20:37] focus on building the app all while Genie, and you don't have to like go and figure out metrics on your own. All of this would be available to you via Genie on this endpoint page.
[20:49] So, let's see once we have hooked all this up, we want to make sure it's working it's working fine. So, the first thing that we we want to just make a couple of queries here, um and let's say we make the same query. Everything seems
[21:02] to be working fine. Um and And that brings us to the closure of act two. Uh
[21:15] And with act two, we what did we learn? So, we learned that you can load test uh and do observability uh on the endpoint uh going forward. You can use Genie to analyze all of those endpoint metrics and what is happening
[21:28] on the end endpoint to debug it so that it would give you what is missing and would give you options on how to improve the uh the performance of your endpoint. And finally, we want to we we would give
[21:41] you a human-readable knob to configure QPS. You don't have to like worry about auto-tuning ANN metrics, sharding, replicas, everything under the hood. Uh everything would be taken care by the
[21:55] the platform. So, the the beat here is it it's minimal knobs about ANN, sharding, replication, and the platform will take care of everything. And you can use Genie to uh analyze any of the
[22:08] uh any of the performance regressions that are happening on the endpoint, and it would start giving you options on how to improve it.
[22:20] And this brings us to act three. Act three is 1 month later. We want to close the quality improvements uh and but how can we do that? Uh we are Genie code. Uh so, I'm going to show you with act three as
[22:34] well. So, what would happen is in a month, we start collecting query logs uh and it
[22:47] would show you the query text, the user, what would the relevant docs, and all of these payload logs would be automatically coming to AI search. Like this is a feature that is coming soon, but all of your indices would start
[22:59] showing these logs so you can start looking into what's what's happening uh under the hood once you have all of this data. But, once we have all of these query logs, what should we do with it?
[23:12] Now, we all of these logs would be in- flowing into all of our search quality evaluation tools. And that's going to help you improve the
[23:24] quality of the system on an ongoing basis. So, one of the things that we had with search quality that we did before was it recommended us to do a fine-tuned re-ranker. And how do we do that is it's basically
[23:38] we use Genie and we click this button and that would help you start a fine-tuned re-ranker on this index. And that would help you not only improve the quality of your index, but also answer some of the tough questions from from
[23:51] customers of why my precision is not improving. So, once it starts it it looks like this. It automatically picks up that it's using the logs that were present for this index and it starts the evaluation
[24:04] on on the re-ranker fine-tuning. Now, this whole evaluation would take a few minutes. So, in the interest of time, I already did this for you. And once it is done,
[24:18] it pretty much would look like this. It would give you the status and it would also log the model so that you can view later of what it was doing. And once we do the the search quality evaluation on it again,
[24:30] it it now shows that hybrid plus fine-tuned re-ranker is the better option and it has the highest possible quality. But, we want to still know why what was the quality gain, what happened under the
[24:43] hood, and we're going to use Genie for it using the same button that we used before. So, once we hit this button, Genie automatically picks up all the experiments that were run
[24:55] and we it starts to analyze everything under the hood. So it shows you why it picked up this, what was the quality differences between the other strategies that it looked, and starts to recommend
[25:08] next steps. So one of the recommendations is just to switch the fine-tuned re-ranker. And also other things that it would also start recommending you to like look, like basically validate the latency, improve
[25:21] chunking, and everything. But since this all of this is is happening in in an MLflow experiment, we have more options here. So we can pretty much go
[25:35] and look at at MLflow experiment and start comparing options. So here I'm I'm picking the the hybrid the two re-rankers that we have. One is for the
[25:47] fine-tuned one and one is for the generic one, and we hit compare. And we can look at our bar charts for different types of metrics that we want. So we have a variety of metrics to choose for and here I'm going to just pick like the
[26:00] highly relevant percentage. So as you can see the high the re-ranker that was built before had a 48% and what the fine-tuned re-ranker had about 64% of highly relevant docs. So it basically
[26:13] boosts all of your precision while keeping everything under your control. Then we go back to the app. Now we hook all of this up. We still want to make
[26:25] everything is working fine and we do a bunch of queries here. So we make sure everything is working exactly as before, nothing changed in terms of regressions. And one thing I wanted to also call out is you can do
[26:38] stuff like more like this as well from AI search using the same indices. So it's it's basically recommendations on your on your search bar. And you can So if you can power not only
[26:51] search bars and recommendations, but as well as any types of real-time search applications, all while using the same platform. You That will give you the same quality and the scalability that uh that was happening before.
[27:04] So, this this basically comes back to uh when we close uh the the act three. So, in act three, we learned that we can do payload logging and quality
[27:16] monitoring would automatically pick those payload logs. We have a fine-tune reranker to improve your quality even further, and it will show us and we can do the evaluation again with uh the
[27:28] search quality tool that we have. And one thing I wanted to just re-emphasize is everything is still governed in the same way that we did before. So, all your indices, all your data, even the payload logs, are still
[27:40] governed with UC, so you control what what users can see, cannot see, and can own. And eventually, the quality will get better over time. Your traffic or the payload logs that we're seeing becomes
[27:54] your training data, eval set, tuning signals to improve it every week without you touching it. And the idea is that at Algolia Search, we believe like developers should focus
[28:06] on killer apps, and the platform should handle the infra, the scale, the performance, and quality automatically. With this, I'm going to hand it back to Sanjeev, who's going to talk about roadmap and exciting things that we're
[28:19] building. >> All right. Thanks for the awesome uh Wi-Fi proof demo. Uh all right. Let's uh round it up with some of the uh exciting features that
[28:31] we're working on currently. Uh so, firstly, as Ankit mentioned, on the quality side, we're uh working on uh continuous quality monitoring and feedback loops. So, many of you already
[28:43] processing your logs and so on on Databricks, and so uh you know that it makes total sense to harness the full power of the Databricks platform to build these continuous feedback loops
[28:57] into your search engine using AI search. Secondly on the full text search side we're working on a number of exciting features like auto complete, query expansion,
[29:10] spell correction and so on. And then on the performance and scaling side we'll continue to push the envelope in terms of maximizing sort of performance while keeping costs
[29:24] under control. Um a couple of call outs these are not things that we covered in the demo today but many of you do have applications
[29:36] where you're building rack chatbots. Please check out the Databricks knowledge assistant product for that use case. And then if you have batch workloads please do check out the vector search
[29:49] SQL function. And finally with that I hope you agree that at Databricks AI search we're making high quality retrieval the default. It's not a dark art that
[30:03] people need to reinvent with every search engine. And then secondly we're going to make sure that quality and performance are dead simple and constantly improving
[30:15] with your traffic. Um if you would like to try some of the preview features that we're working on please scan the QR code and let us know which which features you're interested
[30:28] in or if you have any other feedback you can use the link from the QR code to get in touch with us. Right.
[30:40] Okay, thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.