Thinking Fast & Slow: How Databricks Built High-Speed and Deep Research Agents

Summary

Databricks AI Research built a unified agentic search architecture supporting two modes: an instant mode that delivers single-digit second latency at scale for real-time applications, and a thinking mode that performs multi-step plan-search-verify-synthesize reasoning for expert-level deep research tasks.
The system uses an Instructed Retriever architecture and an Aroll framework to sit at the Pareto frontier of cost, speed, and quality, with benchmarks on program-manager and financial document corpora showing state-of-the-art performance in thinking mode.
A key training lesson is that models trained only on long reasoning outputs forget important context, requiring a balanced fine-tuning pipeline that combines supervised fine-tuning with reinforcement learning reward shaping to maintain retrieval quality across both modes.

Thinking Fast & Slow: How Databricks Built High-Speed and Deep Research Agents

In today's competitive agentic search landscape, two operating modes are mission-critical. The first is a low-latency, low-cost mode for consumer-facing scale. It must meet strict tail-latency budgets for real-time, high-throughput use cases at minimal per-query cost, without sacrificing quality. The second is a compute-intensive deep research mode enabling expert-level analysis—from financial due diligence and technology mapping to clinical review and manufacturing diagnostics. The system must plan multi-step retrieval, triangulate sources, and synthesize coherent answers. Using our Instructed Retriever and Aroll frameworks, we built a unified agentic harness supporting both modes in a single architecture. Our system sits at the Pareto frontier of cost, speed, and quality. Instant mode delivers single-digit second latency maintaining accuracy, while Thinking mode achieves SoTA-level performance across enterprise domains.
Talk By: Michael Bendersky, Director (Research), Databricks ;

Chapters

FAQs

What is grounded reasoning in Databricks AI research?

Grounded reasoning, as defined by Databricks AI Research, is the ability of an AI model to reason over heterogeneous real-world enterprise data rather than relying solely on its training knowledge. The goal is to deploy attention only to data that is relevant to a given task—a concept the speaker draws from Daniel Kahneman's book "Thinking, Fast and Slow."

What is the difference between instant mode and deep research mode for agents?

Instant mode is optimized for low latency, low cost, and high throughput—suited to applications like voice agents, embedded copilots, and customer support—and delivers results in single-digit seconds. Deep research mode is compute-intensive and performs multi-step planning, multi-source retrieval, and synthesis to produce expert-level analysis for tasks like financial due diligence or clinical review.

How does Databricks benchmark its agentic search systems?

Databricks AI Research evaluates its systems against program-manager and financial document corpora, with the thinking mode achieving state-of-the-art benchmark performance according to this video. The research team also developed a benchmark called OfficeQA for measuring grounded reasoning in enterprise settings.

What is the Instructed Retriever architecture?

The Instructed Retriever is an architecture developed at Databricks that enables the agentic search system to match the performance of larger set-level models while operating within a single unified harness. It is one of two core frameworks—alongside the Aroll framework—that power both the instant and deep research operating modes from the same architecture.

Full transcript

[00:08] Good afternoon or good morning everyone. Um thank you all for coming to this talk. Um I'm Michael Benderki and today I'm going to be talking about thinking fast and slow and this is a talk about how uh here at data bricks we built u
[00:20] high speeded and deep research agents. Uh so hopefully you'll find this interesting. Um I'll start with a brief introduction of myself. So just so you know uh who am I and like what am I
[00:32] working on here at data bicks. So I'm uh here director of data bricks AI research. Um the goal of data bricks AI research is slightly different from other research uh labs in AI. Our goal
[00:45] is to teach agents what we call grounded reasoning. So basically making sure that these AI models are grounded in heterogeneous realw world data. And you know our lab does both foundational research. We publish papers and
[00:58] benchmarks um to hopefully advance the field of AI. Uh for instance we have this benchmark called office QA that some of you may have uh encountered in the wild. Uh and we also build applications. So we bring the things we
[01:12] learn uh about these foundational uh findings to our products like Genie agents. Uh and today I'm going to talk to you a little bit about both both the research aspect of things and how we bring these things into our products.
[01:26] Um so I define grounded reasoning and I think I like to give this quote from the thinking fast and slow book by Daniel Kman which was also the inspiration for this talk. Um intelligence is not only
[01:38] the ability to reason, it's also the ability to find relevant material in memory and to deploy attention when needed. So we don't only want to reason. We want to reason over some shared memory or data in the enterprise setting
[01:52] and only deploy attention to the data that's relevant to our task. I think that's kind of the inspiration for this grounded reasoning um metaphor. Another name for uh grounded reasoning that you
[02:05] can see maybe coming up in blogs and literature is something we call agentic search. And this agentic search idea has kind of bifurcated I would say in the last maybe year or two. Um it's kind of
[02:18] became this problem that has two operating modes and increasingly they are being solved by very different stacks. So one mode is this instant mode where we have very fast, very cheap, very
[02:30] scalable search and it has a lot of applications you know so you you all use Google use web search uh voice agents uh customer support you know embedded copilots those applications where you
[02:42] really want your agent to give your answer really really quickly so you can go on with your task and then there's this the other mode uh which we call thinking mode this is kind of maybe what you're familiar with with like deep research agents
[02:54] or agents that do things like financial due diligence or clinical evidence review or diagnostics. Um these are very thorough multi-step expensive agents and increasingly people
[03:08] who work on these two tasks are kind of working on separate stacks. So just to explain you know how these things differ. So in the case of instant
[03:20] system one kind of uh mode we have again this kind of real-time scalable retrieval applications and usually people expect these things to retrieve things with very very low latency. So
[03:33] you know expecting probably less than two seconds for the first token. Um they also expect relatively low costs right something like in the single sense or even less. uh they also expect kind of
[03:47] deterministic quality evolves. Does the agent retrieve a right document things like that and then even if you look at the research patterns that kind of like evolved or like the lineage of the research that led to these systems they
[03:59] look more like coming from the information retrieval literature you know learning to rank dense retrieval ranking and then feeding into these kind of like very very fast agentic systems.
[04:11] uh you may recognize this system one description is something that people you know call rag or retrieve augmented generation as well and then there's a sec second part of this which is this kind of thinking mode of system two uh
[04:23] where this kind of like multi-step agent reason plan search read verify synthesize and repeat they may take again tens of seconds maybe even like minutes
[04:37] their execution costs quite a bit so you know people expect these agents to run for longer. So obviously the costs will pile up and again when we evaluate their performance sometimes the evaluation is kind of non-deterministic. It's more
[04:49] like very expert high quality greater words. So you know the evaluation may be somewhat subjective as well. And again if you look at the lineage of the research that led to these agents it
[05:02] comes from more like a gentic AI perspective. So coming from papers like react uh tool former and then you know later on to tools like deep research on uh Google and openai. So I describe how these things are
[05:15] different but I also think these things are somewhat related to each other and I believe and I hope will convince you today that we can kind of build both of them on the same unified approach
[05:29] and the approach consists of three things. uh share specification. So we want to propagate all system instructions, index schemas, the various retrieval constraints
[05:41] through every retrieval stage. You want to make sure that the agent always propagates the uh specifications to the tools. Two shared tools. So one realization we have is that both modes use the same
[05:54] search primitives. And so we want to make sure that both the fast mode and the thinking mode both use the same retrieval primitives. And third shared evaluation. So you want
[06:06] benchmark and judges that kind of generalize that they can work in instant mode and also they can work in this research mode. And so once you build sort of one system, you can scale it up to do more steps and become more
[06:20] reasoning or you can scale it to do maybe higher query throughput or lower latency and you know why does it matter to like our data bricks customers. So a you do
[06:32] want to have like a one platform to serve you know your customers or your internal users and your internal users or your customers they have all the different types of information needs. they have both instance search queries
[06:44] and deep research agents. So I think it's important for enterprises to be able to sort of access potentially in the same um and you know if you actually want to
[06:56] build something like this by yourself uh it allows you sort of faster iteration on the platform you have consistently quality improvement on both modes by just iterating on one single platform. Okay. So what I'm going to hoping to do
[07:10] today is basically a the share foundation very briefly and then talk about these two uh uh things we built with this uh uh framework. One is uh instance search with instructor
[07:23] retriever one which actually uh recently uh deployed in knowledge assistant. It's actually been GA uh in in this week. and then two this research work that we developed on deep research with this
[07:36] agent called Carl and then I conclude with some lessons that may be relevant to you right so again the share foundation so I think the share foundation as I mentioned before there's like three big
[07:48] things that we need to define for the share foundation to be possible one is tools uh two is data synthesis I'm going to talk about a little bit how we synthesize data to train these things and three is unified evolves
[08:01] All right. So um you know roughly six months ago we pro we published this paper called instructor retriever. was a blog uh on our uh data bricks research blog where we
[08:13] sort of proposed architecture and we actually implement this in our knowledge assistant product um where specifications are explicitly propagated to all the stages of both retrieval and generation where specifications include
[08:26] things like user instructions, labeled examples, schema of the indices being uh um retrieved from and this specification flowing to these different parts of uh
[08:38] kind of like retrieval workflow were very very important for uh quality. Uh this is just an example of like how this instructor retriever differs from you know traditional rag and in terms of
[08:50] quality we see that here so the red dots are the instructor retriever the instructor retriever outperforms significantly the sort of simple rack solution doesn't propagate specification by a large margin in the sort of
[09:05] singlestep approach. So what we call instant approach and it continues to outperform rag when we add multiple step to the agent. So it's just a better framework for doing uh retrieval in both kind of instant and thinking modes.
[09:19] Um so that's like one thing we define the tool. So we have this instructor retriever tool. Then we have to think about like how we actually train these models to be better right. So we can use offtheshelf model but you know it may be
[09:33] too slow or too expensive. So we actually want to train model to do better at this task. So this led us to the fact that like there's not a lot of data out there that can be used to train those models. So we kind of have to
[09:47] resort to doing agentic data synthesis. And for this what we found is that there's like two big problems that you need to solve. One is your training data must be diverse, grounded and difficult
[09:59] and sort of cover different use cases that you want to evaluate. The other problem which is kind of counteracts this previous problem is that it also needs to be distinct from the benchmarks that you evaluate on right because if
[10:12] you train on the same data that you benchmark on you will overfeit. So it has to be both difficult and representative but also very different from your benchmarks. So that's like one problem. Uh it's kind of the two problems here. Uh how we solved it. So the approach uh
[10:26] I'm going to describe in the next few slides but basically we first take a approach where we build a synthetic corpus by sampling a broad set of documents where we predicated on a set of you know defined domains that we care
[10:38] about. We then sample a few short realistic information seeking prompts. So sample a few seed queries and then we have this QA synthesizer agent exploring the corpus and proposing questions and then we generate the answers grounded in
[10:51] retrieved evidence. So all of the questions and answer are grounded in the corpus. This is what it looks like. We have more details in the paper I'm going to show you later. So basically we
[11:03] provide both these few short examples of seed queries and the corpus to our prompt synthesis. Uh we generate these question answer pairs ground in the corpus and then we have this
[11:15] dduplication agent that makes sure that we ddup some uh questions that are repeated. Then in the second stage we go further. So we take these questions we generated so far and we run them through multiple
[11:27] solvers and then we filter things that are either too hard or too easy. So meaning if your synthetic data generation provides questions that are solvable by every single agent that's probably not a very useful data. Also if no agent can
[11:41] solve it it's also probably not a very useful data. So we kind of have this like pass rate where we remove things that are either too easy and then we have another filter that sort of make sure that our both prompts and answers are high quality and then this data
[11:54] actually becomes our training data. Okay. So I discussed this uh synthesis pro uh uh data synthesis pipeline. So we now
[12:08] have the tool, we have the training data and now we need the evaluation. Right? For evaluation we developed this benchmark which both covers uh data sets you can uh kind of openly
[12:21] access and also some internal data sets we have uh and we call it Carl bench. This benchmark tries to cover several agentic and very very hard search tasks and again we
[12:33] judge that sort of unifies scoring across tasks. So we can have like a unified scoring across all the benchmark. And if you look at the list of uh benchmarks here, we kind of cover everything from kind of like long
[12:45] document report synthesis to procedural reasoning over technical documents to exhaustive entity search and to some uh use cases on enterprise search like this PM bench which is a program manager
[12:59] bench based on exhaustive fact search on the a set of pro program manager documents. So we defined the tools, we defined the training data and
[13:12] we defined the evals. Now it's time to actually start training models, right? So I'll start with the thinking fast, our instant model. Uh we call instructor retriever one given the fact this is kind of instruction retriever tool and
[13:24] it's the first model we built on top of instructor retriever. Again, I'm not going to go too much detail what instant means. We talked about this, but just to kind of like refresh your memory, what we are targeting here is low latency, something
[13:36] that they can give you an answer with less than 10 seconds to final token and then produce the first token of the answer within two seconds. So the search should be really really fast and it's really important because when people start sort of seeing the result is when
[13:49] it matters, right? People don't want to wait too long for seeing the first talk and can appear and then and that's really important like nobody wants very very high speed you know product that has like really really bad you know
[14:01] search quality. So you want to both deliver a high speed experience but you also want to deliver you know at least comparable experience to what you had with your incumbent that was slower. So you know you want to match or exceed the
[14:14] quality of your incumbent uh with the DII solution and again the cost should be like sense cost like you don't want to build something that costs dollars here. So that was kind of like our dy
[14:26] data I would say when we start building this thing. What we end up building is this parallel architecture we call IR1 which basically does what we call parallel test parallel
[14:38] test time compute scaling in both retrieval and ranking stage. So what we do is we take the user prompt and we generate many many many queries or multiple queries we run at the same time
[14:52] that helps us to increase recall. Once these uh queries are done and because we run in parallel that can be done pretty fast. We then run reanking on the results we retrieve from all of these queries. And when we do the
[15:05] ranking we also do it in parallel. I'll explain a little bit how we do this in parallel. And then we sort of now have all the results you need that have high precision and high recall provided to the response generation model that that can then generate the
[15:18] response. And note again that we propagate specifications. So whatever your user constraints are, your metadata expressions or whatever you want to both the query generation and the rerank stage to sure that we respect all of your constraints.
[15:32] Okay. So I explained this model where we do query generation ranking. I also want to clarify that this is actually a single model. So it's not that we have a single model for query generation another model for query rank for
[15:45] ranking. We have a single specialized handles both. So you know this is a choice and then why did we make this choice? I mean a we have this you know benefit of sharing representation across different capabilities. So we don't have
[15:57] to learn different representation you know for different tasks. We also have a much friendlier serving footprint. So we need to only serve one model instead of two. And uh three now we can do sort of
[16:10] end toend performance using a combination of SFT supervised fine-tuning and RL reinforcement learning. Um you may have been in the previous talks uh I guess on Monday when people talked about scaling RL. This is
[16:22] our sister team in research. Uh they talked more about like how we use RL for training these models and in fact we are we are one of their kind of like favorite clients. So
[16:34] we train this in in two stages. First we train the query generation stage that again scales with a number of queries as I mentioned we talk do this parallel test time compute uh scaling and then we do SFT training when we clone the
[16:47] behavior on like very high quality synthetic uh data generation that I explained before and then sort of we then RL or reinforcement learn by sampling multiple candidates that we get that have very very high recall
[17:00] orientation. So we basically try to optimize for recall on this stage. Once we have this model done, we then want to train our reranker. I explain a bit what is ranker looks like, but basically what we do is we
[17:14] take this uh reranker. We again do the same thing. We supervised fine tune it on like some high quality synthetic traces. But then what we do is we take the documents we found from the query
[17:26] generation stage and we take the permutations from the query generation stage that have the highest precision oriented rewards. So we kind of like teach the reanker to optimize precision in this stage by using the results that
[17:39] were produced by the C generation stage. So in some sense it's a joint training that allows you to both optimize recall and then the second stage optimize precision.
[17:52] And this is the basically the outcome of this. This is a knowledge assistant that now running for uh data bricks users with this instant mode. Uh I'm going to maybe wait for this to finish and then just show you when is starts typing a
[18:05] query. Uh and then sort of the query starts running. You can see it takes one two roughly two seconds to basically finish and then have the first token appear and then the model generates things very very quickly. And the result
[18:18] is done in less than eight seconds. This is a market improvement compared to the knowledge system we had before which was not using this uh custom model. And in fact you know we both got
[18:31] customer feedback. We had this private preview with a few customers who said yes this is much much faster and actually comparable or better quality and see it in our benchmark. So we do we do see consistently
[18:44] uh up to 3x in our you know. So time to first token and around 2x in overall latency of the knowledge assistant. So if you're doing you know a lot of queries in knowledge assistant that's really really important. So we cut down
[18:57] the search time by 3x but the important thing is that we actually did not degrade and even slightly improve the retrieval quality and we see it on on several benchmarks consistently that we
[19:09] were able to outperform the existing system that was slower. So that's that's amazing. Um this is kind of some of our benchmark results. We actually able to achieve
[19:21] performance comparable to set model with this instructor retriever architecture. Um and I want to talk a little bit about how we achieve this performance. So in the last stage I mentioned this we have
[19:34] this reanker that runs and how the ranker actually works at inference time. uh it basically has this notion of pivots where we have reference documents that allow us to do multiple comparison.
[19:48] Each candidate is is scored relative to the pivot and then what we do is we do group wise scoring. We score a group of documents against the pivot which captures discriminative signal
[20:00] compared to this pivot. Now what we do is we then have multiple pivots per group and so the more comparison we do compared to different pivots, the more discriminative signal we get. And then the fun part is that we can
[20:13] actually do scaling because we have these groups and pivots and we have different comparisons. So we can parallelize them and parization is a parameter. But if you scale the comparisons uh to sort of run more in parallel you
[20:25] can achieve higher paralization with more comparisons and like basically actually better results. So in some sense both the query generation and the ranking allows us to buy time by doing things in parallel.
[20:40] And again we see this multi-pivot effect where if we increase number of pivots we actually get better better performance. In this case we basically starting to see set level performance and even slightly better. And this pivot
[20:52] technique works no matter what your model is. So like in this case we are comparing oursel to set and then also within the set family you can run reranking with one or two pivots. So it's kind of nice to see that like this pivot approach generalize across
[21:04] different models not just our model. All right. So, I've covered the thinking fast section. Um, so we built an agent that's able to run things really, really fast. Now, I want to cover the thinking
[21:17] slow section. Um, this is based on a research we published uh called CARL, which is uh stands for knowledge agents via reinforcement learning. Okay,
[21:30] to go back to the one of the first slides, what does the thinking here means? So thinking is kind of the opposite end of the para frontier from the instant mode. It requires multi-step planning. So the
[21:43] system needs to be able to reason about search results and plan the next steps and can do and do it iteratively. Right? So it just should keep going until it found the right answer. Due to that it has to be able to do
[21:55] context management well because it will accumulate a lot of context while it runs. So we should gracefully handle growing context without degradation of performance. Um and it should do this long horizon
[22:08] synthesis where our outputs are these expert grade reports and you know the runtime can be measured potentially in minutes and cost much more than the instant mode right so these are the constraints here and obviously the
[22:21] performance should be much better than the previous mode the instant mode right so that's how we sort of developed carl that's our kind of framework for this kind of custom for grounded
[22:34] multi-step reasoning. Uh yeah, you can take a picture of the slide. There's a link there to a paper that we published on this. So again, it will have much more details than I can give at this talk.
[22:46] Um the nice thing about Carol is that we can kind of use very similar tricks to what we used in the instant mode. Uh and we call it parallel thinking where we take our agent
[22:59] uh we solve a task a little bit and then run it multiple times, right? So then we now have multiple responses but they generate at the same time. So we kind of save some time there and then we have the last step aggregator agent that runs
[23:13] and this aggregator agent its job is to basically aggregate the responses and provide the new response. It can be just choosing one of the responses that the other agent provided or maybe a different response based on these responses. So in some sense we are now
[23:26] running this end times but you're only paying kind of like in terms of time twice the cost right and the important thing is that a aggregator agent is actually the same as the solver
[23:38] agent. So we don't train two agents, we just train one agent that can do both tasks. And two, the aggregate agent has to have access to tools and corpus to figure out its answer, right? So it's not just blindly picking the results
[23:50] from the results that the previous like agents provide. It actually looks at the corpus to like verify that them and make sure they're grounded. With this scaling that I just described, we actually able to get to the what we
[24:02] call a prito frontier. So predier means your model is better than the other models at all different cost levels, right? So the red bars here, the red sort of line here, dash line is where our Carl system sits. And it's basically
[24:15] able to pretty much outperform all of our, you know, existing models, you know, and it's like at the time when OPU 4.6 was released and achieve parity with Oppus with much much lower latency and much
[24:29] lower cost and it kind of dominates all the models that are cheaper than OPU in terms of quality by by a lot. Uh yeah and here this line shows sort of carl and then carl with different sort of parallelism.
[24:41] So if you as you see as you scale the parallelism you do pay some cost but like your your quality actually improves quite a bit.
[24:55] Okay so in terms of how we got here so it wasn't easy. It wasn't you know just like we we ran RL and everything worked. We did surface many many different failure modes that we had to solve over time. And I think it's important for people to realize that it's not trivial to build
[25:08] the URL system. Um so like I'll just give you like some sort of uh failure modes that you can encounter. Um like one failure mode is this output doom loops. So you know if you train your
[25:21] agent with like very very long outputs the model will collapse to just generate a lot of what we call repetitive dil right it will just keep talking because that's what it trained to do. So you
[25:33] really have to diversify your training data to include different output length. You should not just be able to produce very long reports but you should be able to produce you know shorter factual answers or some lists and things like
[25:45] that. So you shouldn't just like train on long outputs because the model will just learn that the long outputs are good. Um verbose reasoning is another failure mode. So you know if you over rely on reasoning
[25:57] uh with no answer quality improvements will keep reason for very very long time without actually getting anywhere. So again you want to make sure you penalize longer reasoning traces. So if you're already able to get to your goal
[26:09] in short amount of time you should reward that. Um the other thing is giving up. So we found that you know models are kind of like humans they want to do least amount
[26:21] of work possible. And so on some of these like very hard agentic tasks such as browsecom plus they would just stop and not continue working. So they would find some plausible answer and then just stop exploring further. So we did
[26:33] incorporate these complex tasks like browse comp plus that require many rounds of re of search which allowed the model sort of to learn the behavior of not giving up too easily and like not just finding a plausible answer but finding the right answer.
[26:46] And then I think number one issue is the context limits. So forgetfulness you know is something that we needed to solve because otherwise you would just exceed content limits on on long uh
[26:59] traces and I would say the number one solution for this is basically context compression. So context compression or context management is just a mechanism to compress context as you do longer
[27:12] rollouts. Right? So basically, you want as your agent runs to reduce the previous context it's seen and compress it to something that's like sizable while still keeping most of the information that the agent needs to finish its task. The way we implemented
[27:26] this and this is something that again came up after multiple runs of iteration is we trigger compression automatically once the conversation triggers the sort of passes the token threshold.
[27:40] And very importantly, we don't actually have a separate model for summarization or training or you know for summarization or some other training on summarization task. We have the same model solving this the summarization
[27:52] task as the models solving the actual problem. So what we do is as we train our agent end to end, we also tell it to once in a while compress its content and
[28:05] the compression model is kind of learned automatically while you train your agent to do search and other tasks. So what happens is that you try to maximize your reward. So the compression model will just learn to maximize the reward by
[28:17] doing compression in the best possible way to maximize that reward. And you found it to work really really well. So but then basically your model is doing its own compression. You don't have to maintain two models and also your model compression target is aligned with what
[28:30] you're actually trying to do which is achieve a search task and get it done. Okay, I'll just going to go briefly to some capabilities that the Carl model can do. I think it's kind of exciting to see that we can do multiple of these
[28:42] tasks. So here's an example of a very long synthesis of a document that basically a structure report is in the medical domain covering several thematic
[28:54] categories each ground with citations. You know it's a question about a certain disease and and like how different medicines like affect this. So this is like a example of a you know medical domain. You want to produce like a very
[29:06] long synthesis of several papers you've read. Here's another capability. This is a multihop query. So we want to query to match four independent constraints.
[29:18] TV show decade, episode pilot, actors, medical history, and childhood biography. And we want to identify an exact episode where this uh person appeared. So the query is kind of a multihop structured query where the
[29:30] agent needs to do a lot of searches to find the right result. It's not example that again the same model that did the previous capability does this one. Uh here's a third one. Uh here the goal
[29:42] is to find an ex like to locate an exact expenditure an exact number which is in this case uh uh $1.577 million
[29:54] uh in the cash flow statement. And so what we do here is we traverse this like very long K K10s and cross validates the figure against a different table that's in there to like validate the numbers correct. And again the model is able to do that and do this very very
[30:07] precise pinpointing of the number in the document. So again, all of these are done using the same model. All right, so I'm kind of like at the end of my talk. I'm going to briefly talk about three lessons we learned
[30:19] during this and maybe those are kind of helpful to you and the things you should take away from this talk. And I'll start with this lesson, which is a plan for convergence, right? Do you want
[30:31] instant mode or thinking mode? You probably want to do both eventually. Like that's what we found. Eventually people start with one and then they kind of realize they also want the other. So plan for this to happen as you build your systems and you probably want to do
[30:43] both. Uh to do this you have to make sure your tool definition is correct right your system specification propagate uh through all the tool calls and make sure that whatever model you're
[30:55] using for the instant mode either trained or you know off the shelf is used as a tool for your thinking mode inside a multi-step trajectory and then prepare for your evaluation suite to
[31:07] cover diverse use cases and evaluate both modes. That's why you build Carl Bench because Carl Bench allows us to evaluate the instant mode and the thinking mode together. Otherwise, you'll spend time building these uh you
[31:20] know benchmarks and then you won't be able to reuse them for the other mode. So that's lesson one, plan for convergence. Lesson two and I think that probably holds in all the talks about AI, you should invest in data and metrics
[31:32] on data side. Again, if you plan to train your own model, data synthesis is key. it's very hard to you know find you know data in the wild for training this system. So investing data synthesis is very very important and you have to both
[31:46] invest in sort of how we can make the data more general and more specialized at the same time while crafting carefully few short query examples and and so on. Uh you want to
[31:58] make sure you're u understand what your reward shapes are look like. So you know for instant again you want to say maybe optimize for equal k for think you want to optimize for final uh answer quality
[32:10] and then also again understand your operating points you know how much you're willing to pay per query in both instant and thinking mode so lesson two invest in data and metrics I think again this is something that you should always think about and again lesson three is
[32:25] you know you always should think about how do you invest in paralyzing your systems because then they can run faster And we saw it in three cases that I showed in this talk. One is parallel
[32:37] query generation. We gen multiple complimentary formulations to increase recall while not incurring any latency gains. Uh par ranking we do this multi-pivot comparison comparing multiple anchors at
[32:49] once. And in the thinking mode we do parallel thinking where multiple agents explore different trajectories and then we just unify their answer with a final aggregation step. Again, we can do n steps in just two uh steps in terms of
[33:03] parallelism. All right, so this is the third lesson. Parallelism all you need. Okay, so hopefully you like now learn all of these lessons. But again, remember that if it's seems like very very daunting, you don't have to build
[33:15] it yourself. We actually build this for you. So learn from my experience like everything we share today is now available in data bricks products in Genie. U we have knowledge assistant that provides you fast ground answers powered
[33:28] by instructor retriever. Now the model I described earlier and you have supervisor agent that can be used on top of knowledge assistant to provide this kind of deep research powered by a carl model.
[33:41] Right? This is all for my talk. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.