Dialogues with Data: Accelerating Pre-Sales Discovery through Agentic AI

Summary

This video describes an Agentic Research Assistant built at Databricks that helps solution architects navigate fragmented internal GTM documentation by orchestrating Knowledge Assistants as parameterized tools within a DSPy ReAct framework.
Lakebase serves as the stateful session store for conversation memory and tool-call trajectories, enabling freeze-and-resume functionality so agents can persist complex multi-step research sessions across interactions.
Performance benchmarks showed strong tool call accuracy and measurable time savings for solution architects, with future work including GEPA-based prompt optimization and email drafting automation using DSPy to continuously improve the agent from historical trajectories.

Dialogues with Data: Accelerating Pre-Sales Discovery through Agentic AI

Pre-sales Solution Architects (SAs) lose momentum navigating enterprise information silos. At Databricks, hyper-growth GTM and a fast-changing product stack amplify the problem. We built an Agentic Research Assistant that turns internal GTM knowledge into a conversational solution-design engine aligned to real pre-sales workflows.
this video is a technical deep dive into the pipeline: a streaming event-driven architecture integrated with Databricks Model Serving, MLflow, and Lakehouse-native state management for real-time adaptability and observability.
Highlights
Workflow-to-tool modeling for field-aligned agent actions
Lakehouse state store for conversation memory + tool-call trajectories
Intent-aware retrieval with Instructed Retrievers
Conversational reasoning using a DSPy ReAct extension
Continuous optimization with DSPy + GEPA from historical trajectories
A blueprint to move beyond basic RAG to self-improving agents that accelerate technical discovery and solutioning.
Talk By: Colton Peltier, Senior Staff AI FDE, Databricks ; Luis Moros, Senior Manager, AI FDE, Databricks ;

Chapters

FAQs

What problem does the Agentic Research Assistant solve for solution architects?

Solution architects spend many hours daily researching and writing highly tailored answers for customers, a process complicated by fragmented documentation spread across multiple repositories and tools with no single end-to-end workflow. The agent consolidates access to knowledge bases and CRM context so architects can get relevant, context-aware answers faster without manually searching each source.

Why was a standalone Knowledge Assistant not sufficient for this use case?

A standalone Knowledge Assistant delivered good out-of-the-box retrieval, but solution architect questions often span multiple document repositories and require inferred parameters or clarifying questions before the right knowledge base can even be selected. The DSPy ReAct orchestration layer was needed to handle this multi-step reasoning and dynamic tool selection.

How does Lakebase support stateful agent sessions in this architecture?

Lakebase functions as the session state store, persisting conversation history and tool-call trajectories so the agent can resume a complex research session from where it left off. This freeze-and-resume capability is essential for long-running pre-sales discovery workflows that may span multiple interactions over time.

What improvements are planned for the Agentic Research Assistant?

The team plans to add GEPA-based prompt optimization that uses historical agent trajectories — the record of which tools were called and how accurate the final answers were — to automatically improve the agent's routing decisions without manual engineering. Email drafting automation using DSPy is also planned, moving the system toward a fully self-improving agent for pre-sales workflows.

Full transcript

[00:08] Welcome everybody. Thanks for joining us on this presentation about dialoguing with your data. Which is uh actually an internal project that Otto and I work on for Databricks for our sales
[00:20] engineers. The problem that we had is that solution architects or the sales engineers spend a quite a a long time uh
[00:32] many hours during the day researching and tailoring answers for our customers. And the the whole workflow was was not very effective. So, we based on this
[00:45] we look at the at the main sources of problems for this space and we realized that the documentation was fragmented. The answers they have to to
[00:57] write were highly tailored for each of the customers. You need to have a lot of context about the customer in order to write effective answers back. They They have several tools, but no tool will
[01:09] help them end-to-end for all this this problem. And the whole workflow was was very ineffective. So, we saw this is a this is kind of a a classical
[01:21] problem for an agentic approach where we have the agent decide which like document repository to use and also to interrogate the CRM system that that we
[01:34] have in order to obtain context. So, it seemed like a really really basic and this was our naive attempt to to use it.
[01:47] When we were kind of idealizing the architecture the a new innovation for knowledge assistance was deployed. This is the instructed retrievers that work
[01:59] inside or is is used by the knowledge assistant agent bricks, which is the knowledge assistant is basically a managed rack solution. These instructed retrievers take care of re-rendering the question,
[02:12] the prompt that you're sending to the knowledge assistant in a way that is more effective for the retrieval, and also rendering any filtering that is appropriate to obtain the the correct information. They can handle
[02:25] >> many many indexes and then do the the ranking of the chunks obtained from all the index and then render the final question. So, it was a a really really good solution for the problem that we
[02:37] had, and when we tested out of the box, we had really really good results. So, we thought, well, this is a trivial problem. Let's just use a knowledge assistant and we're all set. And then, very rapidly, when we started
[02:50] looking at the questions the solutions architect were asking, we realized we needed a little bit of more complexity. Uh this was a typical question that they would ask, and uh it's we know it will be covered by the
[03:05] industry documentation, but if you look closely, the question doesn't tell you everything you need to know in order to be effective answering it.
[03:17] So, the natural approach for this would be to have the agent asking back clarifying questions where the information is not enough. So, the ways we frame
[03:29] this issue, or the solution for this issue, was the following. We have a supervisor agent that then uses knowledge assistants as tools. Every tool is basically a a function
[03:42] call, and every function has parameters. Some of the parameters are required. For example, if you're calling the industry knowledge assistant
[03:54] passing the name of the industry is a required parameter. The agent then will infer that parameter from the question and if it's not able to infer it will answer with ask back a clarifying
[04:08] question to obtain information and then call the appropriate knowledge assistant with uh all the information required.
[04:21] So with with this idea, with this framework, with this frame uh in in mind we selected then the our text stack. We use for the agentic framework the Aspire that gave us really rapid
[04:33] prototyping and modularity extensibility. It has already a react strategy, which is what we wanted to use and we get a prompt optimization as a plus. If we want to use prompt optimization,
[04:45] we didn't have to do anything uh besides. Then as I mentioned before, we'll we'll use in then the knowledge assistants as the information retrieval mechanism. And then finally, in order to glue
[04:57] everything together, we use LakeBase for the session management. So when you answer the question when you ask the question, the clarifying question to the user, we would be storing the current
[05:09] state, the current trajectory of the react loop into LakeBase and when the answer comes back, we'll be retrieving it from LakeBase and use.
[05:25] At the DSPy level the integration that we did was based on extending react. You don't have to have that behavior that we mentioned. So react will see the knowledge assistants as tools.
[05:37] When uh the parameters of the tools could not be extracted from the from the question or the history of the dialogue it would then ask clarifying questions
[05:49] and at that point it would take all the trajectory of all the tools that have been called and all the outputs and all the reasoning and put that in lake base and then answered send the the
[06:01] clarifying question to the user. The user will answer it and when the answer is is back then react would take that answer and retrieve the trajectory from lake base
[06:14] and what the agency is is the full trajectory with the last call being the call to the ask clarify question tool and the answer is the output of that
[06:26] tool. The answer of the user is the output of that tool. So that allows you to have a to ask clarify questions at any level of the react tool. So it may be that it
[06:38] decides to call a knowledge assistant, get some information, based on that information decides to call another knowledge assistant and at that point it realizes, oh, I need more information and then ask the clarifying questions. So at any level of the loop it will be
[06:51] able to to do that. The tools, the knowledge assistants were implemented as DS 5 modules. So everything was a DS 5 module and as as you probably know a DS 5
[07:05] module uses a DS 5 language model. It's a it's a class that kind of encapsulates how DS 5 calls the language model and what it's called in in DS 5 the adapter
[07:17] which basically tells you how to transform a signature, DS 5 signature into the prompt. So what we did is that we inherit from those classes and extended those classes
[07:29] to do a wrapper to for DS 5 to know how to call the knowledge assistant and also an adapter that would take the information, the parameters and all these and transform it into the the prompt that would be then sent to the
[07:42] knowledge assistant. So, for DSPy module, it was calling a language model, but in reality it was calling a knowledge assistant.
[07:55] >> Everyone, my name is Colton Peltier, and I also worked on this project, so let me talk to you a bit about some of the the details on session management and some of the larger architecture. So, >>
[08:07] Um the reason is for a few things. One, people need to ask follow-up questions. Um when people ask follow-up questions, we need to track the state of the agent when they last asked the question, right? So, if someone asks, um you know, compare Databricks versus Microsoft
[08:20] Fabric for this use case, we want to freeze our analysis once we respond, and then we want to wait till the user answers some follow-up question, we need to retrieve that session and resume it. So, we needed somewhere to store that
[08:32] with unique IDs. Um we needed low latency and easy Databricks integration, so Databricks Lakehouse was a great choice. >>
[08:44] a second, walk you through it a little bit. So, the request was coming in with a session ID from the front end, and all as we need is that session ID to coordinate with Lakehouse to retrieve any previous session information. So, we would retrieve the the JSON state
[08:57] from Lakehouse. Our DSPy module would reinitialize at that same state. It would do acronym detection and expansion, so if someone asks about HLS, we would expand that to healthcare and life sciences. And then our freeze mode
[09:10] react would pick up where it left off talking to the knowledge assistant. And finally, before sending back our answer, we want to save our JSON state back to Lakehouse. >>
[09:24] >> So, here's some examples. So, here's an example where you asked it, you know, give us Databricks versus Fabric for a real-time electric use case electric grid use case. Um and we had instructed our agent that if someone asks about integrating with a the
[09:36] um we should always clarify whether they're trying to win the whole account or whether they're trying to do a better together story like integrating the technologies together. So, you can see it replies and asks, are you know, are you looking to choose between the two or you looking to integrate them together?
[09:53] So, at the top you can see the user responded takeout. So, now it's going to go through with its full analysis and it's going to come at it from that angle and saying, this is why you should choose our platform over the other platform for this specific use case. So, it'll find specific metrics in our
[10:05] competition tables, it will look for specific metrics that are relevant to Databricks versus Fabric in this case, and it'll pull those out and it'll put those front and center to say, this is why you should choose Databricks in this instance.
[10:22] So, how did it perform in the end? So, we built a lot of relevance metrics and tool calling metrics. So, you can see one that's like tool call accuracy and then tool call coverage. So, sometimes we would call multiple tools. Tool call coverage means eventually it called the right tool. It
[10:35] might have taken two tries, but it got there. Tool call accuracy is like first try it hit the right tool. So, overall we got really good responses from essays who tested this. They saw really good results and it really reduced their time
[10:48] in terms of being able to draft responses back to their customers with really tailored specific information that was cited and gave them grounded answers. So, what comes next?
[11:00] So, the first thing you might have heard us talk about supervisor agent, right? Well, the Genie supervisor agent did not exist at the time of this. So, we were doing all this from scratch. Now, I would probably try the Genie supervisor agent first rather than building my own
[11:13] supervisor agent. But, the next thing we wanted to do is optimize the prompts with Jeppa. So, this would let us sort of use a reflection model to improve our prompts. And as Luis talked about, you know, we
[11:25] hooked together with DSPy by extending it. So, we can even optimize our prompts going to the knowledge assistant as well. We also built this on Databricks model serving, which was the standard at the time. Um and now we would probably migrate to
[11:38] apps. Apps are a bit faster for iterative development, and at this scale, uh apps would be more than fine. Additionally, email drafting. So, we want to take that answer and look at the actual email that the essay got from the
[11:51] customer and draft a full response back. So, that would save them a lot of time, and then they could go through and double-check sources before sending, but at least the full draft would already be there.
[12:04] So, some lessons learned. Um one thing that we really learned is that second bullet point there is like being able to optimize the full compound system at once um is ideal, um and you need a lot of diverse data to build out your eval set for that and to hit all
[12:16] the different paths of tool calling. All right. Thank you so much, everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.