Sponsored by: You.com | Building AI Search Infrastructure for the Agentic Era

Summary

This video explores how AI search infrastructure must be redesigned for the agentic era, where Cloudflare data already shows that bot-generated requests outnumber human requests when fetching HTML — a trend expected to accelerate as AI agents increasingly depend on web search.
You.com, which started as a human-facing alternative search engine in 2020, has transitioned to building agent-first web search APIs after recognizing that AI agents are the true power users of search, requiring structured and parameterized queries rather than keyword-based interfaces designed for humans.
The video warns about a concerning consolidation trend — including major platforms retiring their search APIs — that risks making every business a data provider to dominant platforms, and argues that open, accessible search infrastructure must be preserved for developers building AI applications without platform lock-in.

Chapters

FAQs

Why do AI agents need different search infrastructure than humans?

AI agents make structured, parameterized search requests optimized for information retrieval accuracy, whereas human search interfaces are designed for keyword queries and visual browsing. Agents also generate orders of magnitude more requests — Cloudflare data cited in the video shows bots already exceed human request volume for HTML fetching — meaning infrastructure must be designed for high-volume, programmatic access at a much lower cost per query.

What is the consolidation risk in AI search infrastructure?

Major platforms are closing their search APIs — the video cites Bing retiring its search APIs as an example — which reduces the options available to developers building AI applications. This consolidation risks a future where AI agents can only access information through a small number of dominant platforms that control both the search index and the AI model layer, increasing lock-in for every business building on top of them.

What does You.com's research agent architecture look like?

You.com has built a research agent that combines web search retrieval with LLM reasoning, and the video reports state-of-the-art benchmark results on FinSearch Comp and CompFinBench evaluations. The architecture is designed specifically for agent use cases where reliable, factual, source-cited answers matter more than the interactive features valued in consumer search experiences.

Why does evaluation methodology matter when choosing AI search infrastructure?

Different AI search systems use different evaluation methodologies, which can make performance comparisons misleading when benchmarks are not aligned with a real-world use case. The video argues that selecting the right AI search provider requires understanding how benchmarks are constructed and whether the evaluation setup actually reflects the type of queries your agent will make in production.

Full transcript

[00:08] Well, thank you so much for for visiting and attending. Really excited to talk to you about how we're building the AI search infrastructure for the agentic era. So, my name is Sahil. I'm a CTO at this at u.com.
[00:22] At a very high level, AI web search was originally built for humans. Um, this has been the case, you know, since pre2022, but it's rapidly changing. We're now seeing that AI agents need search more and more. Um, for a little bit of context on ourselves, we actually
[00:35] started off as a company building web search for humans back in 2020. We were building an alternative search engine to Google, which was really exciting. But little did we know that the actual user that we were building for didn't exist yet, which were AI agents. Um, in 2022,
[00:49] 2023, we were one of the first folks to merge web search with LLMs. And this was really exciting for a couple reasons. And then, you know, in the last couple years, as AI agents have become more and more powerful, we've realized that they
[01:01] need tools like web search and we've transitioned to focusing on web search for agents. Um, right now we're fully focused on building the best web search APIs for AI agents. And I'll talk a little bit about what that means, how we're keeping and building an open, you
[01:15] know, API for for uh various companies. Um, and that's a little bit about our trajectory, our history. We've been thinking about search information retrieval since 2020. Um, and that's been the fundamental through line throughout our history as a company. Um,
[01:27] and I'm really excited to share more about that. And one other piece of context is as we think about search for agents, it's really different than search for humans and we've optimized along a series of axes. Um, AI agents are already the new
[01:41] power user. This is a stat published by Cloudflare where it shows that the number of requests made by bots already exceeds the number of requests made by humans. And this is specifically when fetching HTML. So this is no longer theoretical or abstract. It's already
[01:53] happening and we expect this trend to continue over the next couple years. Um, very soon you'll see the number of requests for web search being made by agents will vastly outnumber the requests uh from web search made by
[02:05] humans. And if we kind of take if we take a step back before we dive into some of the details uh we should zoom out and see the big picture foundation models have improved rapidly. The way I like to
[02:17] think about this is by analogy to self-driving cars. Foundation models are like self-driving cars, but unlike self-driving cars, foundation models have become very capable and intelligent before the roads have been built. So essentially, we have self-driving cars
[02:29] without any roads, which is why we haven't seen foundation models fully penetrate the economy yet. Once we build these roads out, we'll see it develop more and more. We think about it in terms of two types of companies. Uh we have the core intelligence layer. These
[02:42] are your foundation model companies that are really focused on building the best intelligence, the reasoning layer. And then we have the augmented intelligence layer, the folks building the roads that these self-driving cars or these foundation models drive on. Well, the
[02:55] road that we focus on is AI search. And we believe that by building the best roads over here, we can allow foundation models to go further. When you take AI models with and core intelligence and you add and augment intelligence, you
[03:07] can have AI agents that work and are able to actually automate and improve very aspects of the economy. And that's what I'm really excited to talk about. Um, and the road that we've chosen to drive on is the search road, the search
[03:19] infrastructure layer. We believe that this layer is really interesting for a couple different reasons. If we look at, you know, the trajectory over the last couple years, we've seen token costs continue to go down. Um, however, we've
[03:31] seen search costs relatively stay stable. And the reason for token cost decreasing over time is twofold. one on on the one hand you have these foundation model companies and these proprietary models getting more efficient with a lot of the innovations from the big labs and on the other side
[03:43] you see a lot of open source pressure there's a ton of new open source models that are much cheaper are more efficient and we see this routing mechanism now emerging and becoming a big topic where companies are now routing between different models to save cost um however
[03:56] on the other hand search infrastructure um the costs over there have generally remain stable and the reason for this is that while you can host an open source model on your laptop you can't really host a search index on your laptop up. This is one of the reasons why we'll see more and more economic spend go towards
[04:09] the search layer um instead of just the token layer as a percentage. Uh what this essentially means is that it's really more it's very important for everybody who's building AI agents and AI companies to think very deeply about
[04:21] the search layer that they're using. And we'll talk a little bit about what it takes to kind of evaluate and um vet an AI search API. We're very focused on building the best AS search APIs and being a, you know, providing very useful
[04:35] economic spend on this search infrastructure layer for companies. If we step back a bit, search is not new. Search has existed for a while for humans. Um, however, the econ the
[04:47] incentive for a lot of these big search indices of the past um to align with startups of the present are not there right now. Um, and the reason for this is that search indices are the crown jewels of many companies like Microsoft and Google. We saw this with Bing. Um,
[05:00] Bing actually retired their search API a couple years ago when they saw the value of AI search emerging. And the reason for this is that it's a really important uh crown jewel for their business and as
[05:12] such it's they're not incentivized to open it up to other developers and other companies as it you know it makes sense for them at their scale to you know keep it for themselves and we see this with Google as well. Um Google search index
[05:25] is also their crown jewel which is why there's a whole industry that's emerged called the SER API industry. What this essentially means is there's APIs that crawl and parse Google results. Um but the issue with this is twofold. one is you know they're currently you know
[05:37] Google is taking lawsuit and action against them reliability can be up and down given the fact that um Google can change this policy and the third piece is when you think about zero data retention and those types of pieces if
[05:49] you're passing your data to Google um they will still own some of the data um and kind of you know as we as we think about this and then other other kind of factors are the SER API schema changing
[06:01] so for example we see this happening recently what this means means is that we need an open search infrastructure layer um that companies can use. We need an independent search layer that can power the next generation of AI agents and
[06:13] startups. And this is what we're really we're really focused on building. We want to democratize search um in a way that the gated indices and of the past can't because it's not in their economic interest. Our model essentially is to provide an API and we directly are
[06:27] incentivized to provide the best API for companies because because as we get more API calls, we grow our business and that's it. you know, we don't have advertising and we purely monetize based on consumption from our users. So, the
[06:40] more we can have AI agents consuming our APIs and the more we can open up our APIs and demystify our index as well as our technology, the more you know, we want to build trust with our customers.
[06:55] This is our AI search ecosystem. Um, on the one hand, we have what we call low orchestration APIs and on the other hand, we have higher orchestration APIs. Our primitives are our content and search APIs. The search API allows you to do a web search and we'll talk a little bit about how we benchmark that.
[07:07] Our contents API allows you to access content from a URL. Together, these form very powerful primitive tools for agents and we are investing in improving both of these. And then on the other hand, we also offer higher orchestration tools
[07:19] which can also be consumed by agents like our answer API which stitches together and uses our content and search APIs to provide very good answers as well as our research API which essentially performs a lot of orchestration using our search and
[07:31] contents API to provide very in-depth responses at scale. We essentially see this graph over here where we believe that agents will be in charge of their own budgets which is why it's important to provide a menu of options. So we have
[07:43] various search skews in line with our philosophy that AI agents will be handling their own budget budgets in the future. On one hand we have our search API which is the atomic skew which is lower latency and then we have higher latency offerings that are higher cost
[07:56] like our research API and even within our research API we scale with our research effort. This is something that we've called scalable research compute where the more compute you can give these research agents the better they can do. And the idea behind this is we
[08:08] want to give as much flexibility to developers who can choose how to use some of these APIs. The other topic that we think is important is parameterization. In the old world of Google, consumer
[08:20] search um has very few configurations that folks often use. And the reason for this is that humans are lazy. Um you know, humans are not as willing to customize their search because at the end of the day, you know, we're always in a rush. You know, our our time has
[08:32] very different constraints than the time of agents. However, agents on the other hand are not lazy. they're very willing to parameterize their search APIs and do a lot of this work. So, our philosophy is to open up the box and allow agents
[08:44] to heavily parameterize their search in order to get the best results for their applications. Um, and this is something that we're constantly supporting and working with our customers to improve. And I think it's also a broader trend we'll see in the industry. We'll see more and more parameterization of APIs
[08:57] that are used by agents and foundation models because these models are uniquely equipped to do this type of work in ways that humans aren't. When we think about web search APIs, we also think about freshness. A lot of the
[09:09] benchmarks out there right now are static benchmarks. This means they're developed once and they stay the same throughout time. While this is really great for testing complex um capabilities, one of the issues and downsides with having static benchmarks is that they can be gamed and they often
[09:23] don't test one of the fundamental uh you know needs of foundation models which is the ability to access information that's not encompassed in the weights of the foundation models. When we think about this, we're actually investing a lot in
[09:35] benchmarks like live QA, which is a benchmark that is constantly changing every week. And you know, this is something that will open up soon. And we're really excited to continue to pioneer and focus on how can we provide the freshest content in the industry to
[09:47] AI agents and our customers because we believe this is one of the core deficiencies of foundation models as is, which are always going to be lagging behind world events. Our research API, as I mentioned, is
[09:59] built on top of our search and contents APIs. And the idea behind the research API is to allow agents to offload more of the work and more of the orchestration to other agents. What we see over here is a paradigm known as
[10:11] agent agent where agents are actually calling and using other agents to perform tasks. And similarly, we have developed various parameters so that agents can actually use these sub agents as well. Um and this can range from
[10:24] things like our research effort which allows you to scale compute to the very simple pieces like country and schema. Um the idea here is that you can also request structured schema from your research API. And this is you know I'll
[10:37] go into the details a little bit to give you a taste of what's happening behind the scenes and also some of our learnings. Our research agent is is quite simple at the end of the day. We've wor it's simple in the sense that we have this harness which uses tools and the two main two tools as I
[10:49] mentioned are web search and contents API. A tool to search the web and a tool to extract content from URLs as well as other tools like a calculator tool to do calculations. And the amazing thing is that by having a simple harness um with
[11:01] you know and when I say simple a simple but powerful harness with good tools we can get state-of-the-art performance on various benchmarks. This learning is really important because what it shows us is that these underlying models are getting a lot better over time. And we
[11:14] need to position our products to continue to benefit as the models get better. Um and it's really important then to focus on the tools that you're using and the model that you're using. If we go back to the self-driving car analogy, we want to make sure that you
[11:26] pick the right car and that you pick the right roads that those cars are going on. And those are kind of the two major decisions that we make when we're actually building AI agents. Um, if you're not working at a foundation model company necessarily
[11:39] over here, we show that we constantly want to operate at the pto frontier between cost, latency, and accuracy. This is kind of the outbound curve. And if I go into a little bit more details on how we've built this research API and
[11:51] some learnings here, we've built a harness optimized for agentic search. And we found that single agent inference scaling up to 10 million tokens with 10,00 tool returns works really well. Um, in certain cases, you want to use multi-agent orchestration, but in many
[12:04] cases actually having a single agent that can store the context and keep iterating can actually have very good results. And this is something that we've discovered through a lot of experimentation with single agent orchestration as well as multi- agent orchestration. The reason for this is
[12:16] that the underlying models are getting better and better and a lot of the work around orchestration is actually getting less important and I think you know this is an example where less is more sometimes. We actually back in 2023
[12:29] launched one of the first deep research agents called ARI or advanced research and intelligence agents and we were actually able to improve it by taking out more and more orchestration over time um as these models got better because earlier on you needed a lot of
[12:42] scaffolding because the models weren't as good but then later on as the models got a lot better that scaffolding that became an advantage actually became a disadvantage. Um so it was really interesting insights for us that sometimes less is more and simplifying
[12:56] your orchestration as the models get better will likely lead to improvements. Um and what that means is that tools are going to be the big differentiator and how you use tools with your agent is also a big differentiator. One optimization technique that we pursued
[13:08] that we've actually published technical blogs on and are willing to share is this idea of budget awareness as an optimization strategy. The idea here is that when you're building agents um giving agents access to a budget and allowing them to decide how to allocate
[13:21] that budget allows them to do a lot of work and you know operate at the predto frontier in terms of performance um with certain cost in mind. So this is a very interesting observation that we that we
[13:33] you know observation we have that's actually the backbone behind what we call scalable search compute in our research API. Our agents are essentially extremely knowledgeable about the budget that they have. And our tools are designed to provide clear budgets. So
[13:45] the agent can actually make very smart decisions and as the models get better, we have orchestrated our research API in a way that the research API will continue to get better and better. And this is something that we see as being a trend not just for us but for many companies that are building and using
[13:58] tools like web search and contents. The other topic that is important is web search in some ways is very general. We've also started thinking a lot about vertical intelligence and we want to
[14:10] help companies in particular verticals continue to improve and that's where we have taken a dive into particular verticals and I'll go through a case study example of finance. Within finance what we've done is we've
[14:22] accessed and you know we expose licensed data that we've procured uh to our customers so they can actually access information that is not available on the public web. This is information such as information from the S&P 500 as well as global macroeconomic data from
[14:35] governments and funds. And what this allows us to do is answer really complicated questions by using really strong orchestration with really strong web search tools with unique proprietary licensed vertical data. And when you put
[14:47] all these together, we're able to essentially get state-of-the-art on various benchmarks like fins search comp. Finer comp is a benchmark of complicated finance questions that require access to license data essentially as well as complex
[14:59] reasoning. And what we see over here is by doing all of this work, we're able to get more than 10 percentage points higher than the next best alternative. And we continue to realize that by marrying key search APIs with licensed
[15:11] data, we're able to do really interesting things. And we want to partner with companies in verticals to help them further differentiate their offerings, whether it's finance, legal, um, you know, automotive, etc. We believe that the future is marrying
[15:23] vertical capabilities with general web search capabilities. Our bread and butter is our web search capabilities, but we're also willing to help work with various companies that are trying to uh, you know, really have success in particular verticals.
[15:37] Really excited to kind of announce our MCP integration. You know, as we think about this, right now we're able to serve large volume and data bricks has been an amazing partner in that respect. Uh right now you can actually use the UC.com MCP server from the database marketplace and you can kind of test out
[15:50] the API. Um and you know I'll talk a little bit about evaluation. I talked about how independent APIs are really important to enable the next generation of agents. Evaluating them is also important. For this reason we've open
[16:03] sourced our eval framework. What we've done is we've taken an open source framework that runs industry standard benchmarks like simple QA frames browse comp and deep search QA. But we also know that public benchmarks only tell half the story. We work really closely with companies to design custom email
[16:16] benchmarks because we believe that the query distribution of public benchmarks and actual customer use cases a bit different which is why we've open sourced our eval framework on the public ones and we work really closely with companies to design eval sets for their
[16:28] particular use cases. Um you can actually access the GitHub link here. It's our oss eval repository and I might actually talk a little bit about some research we've done. Uh the state of eels is in a really interesting place right now. Um you know as we're all
[16:41] aware companies are in a rush to continue to release their new technologies and benchmarks and often the standard has been to release a single number on a particular benchmark. Um and the reason for this you know I I get it we're always in a rush the from
[16:54] the foundation models to search companies to continue to ship updates. However we believe it's really important to do research into evaluation and benchmarking in order to make sure that we're providing visibility especially for high stakes use cases in healthcare
[17:06] and other verticals. And kind of an example of this is some of our research which we published in the triple AI conference and received the best point award. Shout out to Zyra and Abel and Vega on our team for a lot of this work. What we've realized is that when we
[17:18] think about variance which is often reported it's even more important to understand the source of variance. When you think about a benchmark there variance can come from two different sources. It can be agent inconsistency which is the agent itself having variance as well as task difficulty on
[17:31] the benchmark. And right now we often don't realize what the source of variance is. Whether it's the actual different types of questions of the benchmark or whether it's the agent itself. What you generally want is agent inconsistency to be low if you're using
[17:44] the uh the tool or the agent in a you know production environment. Um and task difficulty will generally have higher variance. And what we find is that we have recommendations on how to use techniques from measurement theory which
[17:56] has often been used in healthcare to better quantify and decide how to modulate this threshold. And this is something like interclass correlation coefficients. So this is the type of research that we're doing and we're really interested in partnering and talking to other folks who also have
[18:08] ideas on how do we properly evaluate some of these AI agents? How do we properly evaluate this tools? This is critical to the AI industry right now and something that we take really seriously and we're excited to invest further in.
[18:20] Just want to end on a note. Thank you so much for listening. Um, you know, I took you, you know, today I kind of went through a tour of a couple different concepts. Starting with how we see the industry shaping up, uh, with, you know, core intelligence companies, augmented intelligence companies, and us operating
[18:33] as an infrastructure company building one of the roads that will allow your agents to go further. Uh, we then kind of went through some of our search APIs and research APIs, shared learnings on how we think about longer horizon agents and what is really important to pay
[18:46] attention to, aka the engine as well as the roads. And last but not least, we touched a little bit on evaluation and some of the pitfalls there and a lot of the interesting research that's happening in the space. If you're interested in partnering with us, talking about evaluation, talking about
[18:58] search APIs, or even joining us, let us know. We're really interested in talking to some folks, and we think that together, you know, there's a whole generation of startups as well as bigger companies that can provide a lot of value in this this uh AI economy. Thank
[19:11] you so much and have a have a good rest of your day.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.