Skip to main content

Databricks on Databricks: Transforming the Sales Experience using GenAI Agents at Scale

Summary

  • Moe AI agent: Databricks built an internal GenAI agent that automates tasks, retrieves data, summarizes content, and enables conversational AI for 4,000+ sellers.
  • Secure integration: The agent securely integrates Salesforce, Google Drive, and Glean through OAuth, processing both structured and unstructured data on the Databricks Data and AI platform.
  • Evaluated deployment: Using AgentEval and Model Serving, Databricks demonstrates how to build, test, and deploy production-grade agents at enterprise scale.

Databricks on Databricks: Transforming the Sales Experience using GenAI Agents at Scale

Watch: Databricks on Databricks: Transforming the Sales Experience using GenAI Agents at Scale
Talk by Akhil Aggrawal, Enterprise Architect, Databricks ; Manjeet Singh Chhabra, Sr. Director Product Management, Databricks
Databricks is transforming its sales experience with a GenAI agent — built and deployed entirely on Databricks — to automate tasks, streamline data retrieval, summarize content, and enable conversational AI for over 4,000 sellers. This agent leverages the AgentEval framework, AI Bricks, and Model Serving to process both structured and unstructured data within Databricks, unlocking deep sales insights. The agent seamlessly integrates across multiple data sources including Salesforce, Google Drive, and Glean securely via OAuth. this video includes a live demonstration and explores the business impact, architecture as well as agent development and evaluation strategies, providing a blueprint for deploying secure, scalable GenAI agents in large enterprises.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What problems does the Moe agent solve for sales?

Moe automates routine tasks, retrieves sales data from Salesforce and other systems, summarizes customer information, and enables conversational search. The agent reduces seller workload and accelerates pipeline decisions.

How does the agent securely access multiple data sources?

Moe integrates Salesforce, Google Drive, and Glean through OAuth authentication. The Databricks Data and AI platform provides governance and audit logs to track all data access securely.

How do I evaluate whether my agent is working well?

Databricks uses the AgentEval framework to measure agent performance across multiple dimensions. Continuous feedback and tuning ensure the agent remains accurate and helpful over time.

Can I build a similar agent for my organization?

Yes. This video shares the architecture, development patterns, and deployment strategy used to build Moe. The same blueprint applies to sales, customer service, and other knowledge-intensive functions.

Full transcript

0:00 Introduction and AI playbook

Yeah, before we start some housekeeping items. Uh this is our forward-looking statement. Uh at the end of the presentation, you'll have an opportunity to provide feedback. Please make sure to do that. Okay. So, hi everyone. And I am senior director product management at datab bricks working on internal geni applications. I agraal. I'm the enterprise architect within databicks IT. Yeah. So I'll start with our AI playbook.

0:41 Business functions and AI

Um basically our AI playbook has essentially three major parts. First is we want to make sure that every AI play we do has some kind of a productivity gain and and we want to make sure that it's applied to all the applications we build. Second is we want to improve the user experience for our employees. We want to make sure that with the new modalities coming out of Genai uh for interactions like chat, voice and video, we want to make sure that we apply these into our our uh user experiences across across the IT uh in all our systems.
And third is we want to make sure that uh we we have unified data AI governance and security. We want to make sure that data is coming from uh authoritative sources. It's clean and whatever data we use is is is used in a secure manner. Okay. As all as all of you know AI has huge potential and uh every single function is up for disruption. Um um in our case we are applying AI in all the business functions that we we we serve for um in IT support I I'll give some examples of that.
In IT support we already reduce the number of tickets to 60% by self-s serving through AI. AI acts on the tickets and responds back to the to the user automatically and that's one of the use case we did. Second is we are using AI for uh all of our reporting loads to um we have moved all of our reporting loads to either Genie or AIBI and uh on the revop side finance side we are actually working on u comparing the contract PDF with the actual order in the ERP.
This is a huge problem we have where revops team has to manually compare the terms and conditions on the order as well as all the line items order with the with the ERP system. So we are automating that. And lastly we have created several chat bots u for different functions like revops. We have a chatbot for revops which can answer questions regarding revops. We have commissions chatbot that can answer commission related questions and we have u equity chatbot which can respond to questions about equity.
So we are trying to put put AI into all our functions across it across datab bricks. Okay. So before uh before we discuss how we built our agentic experience I want to discuss uh uh our data architecture within datab bricks. Um u as you know data is the is the foundation of uh AI and we use datab bricks as our our warehouse for all of our AI related loads and the pipelines. Uh our datab bricks workspace is one of the largest uh across all of our customers.

3:21 Data architecture foundations

Uh we we currently have about 3,000 monthly active users as well as 100,000 uh jobs running every day. Um and we we follow the standard medallion architecture. We collect data from more than 200 SAS applications and we ingest the data into into our bronze layer. From there we convert the data into silver and gold layers and that data is available for our business users to use. We have several we have created several uh gold layers for commissions product all of that.
Um and this gold data is then fed into into the AIBI uh reports as well as uh sometimes it's reversed ETL back into into the system of records. For example, we we put our data commissions data back into Salesforce for the sellers to look at. Um we we follow the similar architecture on our AI use cases. Okay. So for sales uh specifically for GTM we wanted to try we wanted to solve problems that uh that our sellers face.

4:55 Seller productivity challenges

So the so what problem do they face? What the most important most prominent problem they face is data is sitting in silos and they have several applications to to deal with. For example, for opportunities, they go into Salesforce. For forecasting, they have to go to another tool. And for um for out for customer communication, they they go to outreach, which is yet another tool. So there are many data silos, many different tools to deal with and the user experience is very confusing, right?
So to solve that specific problem u um we wanted to make sure that we provide a unified experience to the seller and that's the problem we we are talking about today. So this is how a day in the life of a seller looks like. A seller comes in, they log into uh various systems, find information, prepare for meetings, prepare documentation as well as they end of the day they log uh their their um uh their discussions with the customer into into the system which this consumes most of their time and they cannot focus on the work they actually need to outcome they actually need to get out of uh their their whole day.
And with AI, AI provides them uh information at their fingertips. It with ease prepares them for the meeting, generates the EBC doc, logs the notes automatically um uh by summarizing the transcripts and ingesting into into the systems. Um and think about this like this is just for a single seller. If you if you multiply this across 4,000 sellers, it's a huge productivity gain. Okay. So now I'll talk about our u our uh genai journey.
We started with traditional chat bots. The chatbot that could provide answers to the question user question which was which didn't have any knowledge of history of the questions or didn't have much contact but it was just plain question and answer. Then we the in the second stage we started building rag applications where we collected data from various sources and started summarizing and started extracting information from that.

6:53 The GenAI journey evolution

That was our second step. Now we had more than one data source connected in this stage. Now currently we we have a um we are at a stage when agent and tool became more powerful. We started building a generalpurpose chatbot which could answer all kinds of questions or perform actions on various uh data sets. So it not only provided queries but it started generating content like EBC document generation as well as or chart generation.
So in this sta stage we had our agent running and supporting the seller uh across various systems. Uh in this stage we also uh also uh created a evaluation framework where we could figure out if the data is relevant, accurate uh or not. And this was super important for us to is to continue to try keep track of to provide a better uh user experience and improve upon our um our our agent. Okay. So um uh and now what we are doing is we are moving to a stage four where we are building more than one agent not just one agent the today's agent is general purpose and we are building agents to perform specific task like for use cases we are going to build a single use a single agent for commissions or or consumption we are going to create a second agent these agents will be able to talk to each other to provide a better user experience and very excited about announcement today about agent break you know this is the direction we are going to move our implementation as well.
Yeah. Okay. So with that u this is let me introduce Mo. This is our uh agent that we created on top of Salesforce. This is an in theflow uh agent which provides answers where the seller is. So the original problem as I mentioned was that there are too many silos. They have to switch the tools. We didn't want to create a separate applications where user application where user goes which which will become another application.

9:03 Introducing Moe AI agent

So we wanted to provide our experience right there where the seller is and in our case it's the Salesforce as a CRM. So we build MO on top of Salesforce which which started providing the features that I mentioned previously in the MO and we'll have a demo as well. will be providing a demo. Um so Mo has now access to not just Salesforce data it has various other other uh data sets that will will show a demo of and this is the beauty of the agent you seller doesn't have to switch the context the information is provided the documents are generated right there uh in terms of gen capabilities of formo uh we we use standard database stack and on top of that we can do various tasks like summarizations uh summarization of uh uh various things like notes or support cases and things like that as well as mo can generate content as I as I described previously.
Um it can do full conversation with a chat history and it can figure out what was the what was the previous question and based on that it can uh it can answer the next question without providing the full context. uh it can also uh perform action and update data as needed. Okay. So we have about uh 60 to 70 tools in our library from various for various areas for the for our agent. Uh on the account intelligence side we can the seller can ask for 10k summary or or support cases in an account.
Um our business is is a a consumption based business. So we the sellers care a lot about the consumption. So MO can provide quarter overquarter um growth trends or when the customer is going into burst. Uh that kind of information it can provide uh it can also provide information for customer references. uh you can find similar customers for similar use cases for the customer and uh one of the important thing that sellers care about is the use case hygiene right are the use cases updated properly in the salesforce or not to and mo can answer the questions like okay what cases are missing next steps or or uh or certain other information about the use cases and lastly you can ask you do uh content retrieval through sales content retrieval for example give me a pitch deck for certain use case so you can get that information from mo as well uh with that we'll jump into the demo akil will be providing the demo thanks mjit uh so what you're going to see is a live demo in front of you is essentially a standard Salesforce screen I'm logged into Salesforce and I'm on the homepage for Salesforce and Mo is available to you to all our sellers on the right hand corner of the screen.

12:25 Live demonstration of Moe

All I need to do is click on more and it's available to you to ask any question. The first important thing is Mo is contextual. That means it knows where you are in Salesforce and based on that context it can answer the right question. For example, if you happen to be on account page, it knows which account you are on and all the answers will be relevant to that account. If you're on a happen to be on specific opportunity, it will be able to answer the question about that opportunity.
Now, at this point, I am in the seller homepage, right? I'm not in a specific context. I'm in overall Salesforce context. So, I can ask any question about any account. For example, let me ask about the summary of the recent 10K from Nike. Now in this case the agent will go and summarize the 10k information and but what happens here is we have multiple Nike account in our Salesforce instance. So agent is smart enough to detect that and give me the choice to pick the right Nike account in this case.
So I'm going to pick the customer account for Nike and now the 10k statements are always already there in our data warehouse in our internal data lake. So what mo agent is doing is actually going to our is running in our data intelligence platform and getting the 10k and displaying the result back to me. The result itself is displayed in both graphical format as well as in textual format. So the summary information is something that we have prompted ourself that means we are able to deduct the um the financial information the balance sheet highlights as well as the digital transformation the informations that are relevant to our sellers.
Now, one of the important another important feature that our sellers wanted to get was getting the org information for a for an account. So, I can I can actually just type in here. Now, I don't have to specify Nike here. All I need to say give me the OC chart. Since the since it since our agent is conversational, it already knows the context that you are you are referencing the nike account from a previous chat. So it can continue with that context and it can give me the ag chart information.
Now this a chart information is coming from third party sources. What we have done is we have put a much better uh interface to allow the user to drill down into the right. So what you saw here was essentially information that is available from third third party sources you know public domain sources. Now what if a seller wants to get the information which is a first party which is available in our CRM itself you know to demo that let me go into the specific demo customer account.
So we created this account just for the demo purposes. So here I since I since I'm in the specific account page now Mo is now contextual to that account already right you can see that uh uh the MO is now displaying the frequ frequently used preceded prompts that we have uh based on our analytics we have figured out that these are the prompts that our sellers are asking. So we have already contextualized it for you. You can ask to generate a EBC document or you can uh get the completed text track of this account or you can get customer references.
You can actually also ask free form questions right here uh like any any chat experience. Right. So let me start by getting the quarter overquarter growth for this account. What Mo is doing is uh is going back to our agent uh and getting the information from Salesforce and summarizing the information and presenting that. The important aspect was when we actually implemented more, we wanted it to not to be just a chatbased chatbot.
We wanted it to be much more richer experience. So we implemented specific implementation to show both graph based information as well as textbased information. So what you see here is a quarter overquarter summary as well as the summarized information about uh the growth itself. Now I can ask more information. Let's assume that I'm preparing for a meeting with this customer and I want to get uh the top use cases now.
So I can just type in the top use cases. So you know use case is a is a custom object in our system. So we are essentially querying the custom object information and then uh the agent is smart enough to determine what is the top use cases in our in our in our context the top use case is determined by dollar dbu. So that's the definition of top top use cases and it's listing all the top use cases in a tabular format again the markdown we have uh and I can click on any of the use case and go deeper into it in case I need to.
Now I got the top use cases. You know I'm I had a meeting last year with this customer. Let me see what was the summary of that meeting. So I can ask uh the agent itself to summarize me uh give me the summary of the recent meeting. So in this case we are going against our customer nodes and we are pulling all the customer nodes or recent customer node and summarizing it. So in this case it gave me the most recent one.
I can ask it to give me the most recent three meetings summary. So it should ideally put in this case the most recent those three recent meetings actually. So it's smart enough to detect what the context of the user is and get the information. Uh again I want to see um what kind of credits are available so that I can have a meaningful conversation with the with my with my account. So in this case this query is special.
So far what you saw was information that is coming from the CRM itself. Now in this case I'm going to our knowledgebased system which happens to be glean internally. We we essentially are going against glean retrieval API to get the knowledgebased content around what are the different credits available for this account. So I can see you can see that it got uh various different credit offers that I can talk to my customer about uh and see what what makes sense for them.
Now I want to I want to create a pitch deck also. So I can ask for pitch tech again the same context that means um it will go and fetch the knowledge article I don't have to go to different system I can just stay here and ask for the pitch deck document and it can actually retrieve the information for me so in this case again it's making a call back to the backend system um and getting the information from glean uh the agent is running in our environment but making the call to all the downstream systems.
So you saw all the standard retrieval and summarization use cases. Now one of the important feature is since I'm preparing for the meeting, execs are attending that meeting. So I need to prep the my execs for the meeting and I need to create a EBC document. Now traditionally this EBC document for a seller takes a long time. They may take anywhere from 30 minutes to an hour to create a EBC document for a given account.
And here in this EBC document, it contains both the first party information as well as the third party information that generally gets added to the to the document briefing document. And now I can generate that EBC document, right? With a single click, right? Um the document itself is generated created in Google Drive and share with the seller. So the seller can actually share it back. So now this document contains the details of the account and all the red color information is something that the agent was not able to find.
So the seller is supposed to fill that information on their own but all the other information for example account specific information is available right here. Um it contains the public domain information about the accounts itself as well as it contains um the use cases information and in case since ours is a consumption business if it has consumption this account has consumption then it will show the consumption trends and all the other relevant information right here.
Yeah, as you can see, this is a b hugely beneficial tool for improving the seller productivity within our ecosystem. So that was a demo. So now let's go back and talk about how we did it. Thanks. Okay. So before I talk about our implementation, let's talk about what is agent tool actually happens. What is it really? So agent is essentially a brain that does the reasoning and decisioning and invoke a set of tools as determined by the LLM.

20:48 Agent tool mechanics

So in this case the user is asking for give me opportunities more than a million dollar. So agent will reach out to LLM and find out what are the set of tools that will be able to fulfill the goal of the user. In this case, get opportunity is a tool that will be selected. U and the way the tools are selected is each of the tool has a description and that description is what is used by the LLM to figure out what are the right set of tools.
It can be one or more tools and it can be chained together. So, and then the tools are invoked and to invoke the tool we need to bind the input parameters parameter with the tool. In this case, the input parameter happens to be a million dollar which is coming from the user as well as the account that the user is in. And that's a binding that automatically happens uh uh by the agent. You know the important part is we have not coded for anything.
All we have done is we have created tool provided the description and the agent and lm are doing the job. Right? So this way we can actually keep adding the tools as long as the description is clear agent will be able to figure out what is the right set of tool to invoke uh in the flow. The tools itself are external functions. They can be Python or SQL function. In this case, we have a Python function which is making a call to Salesforce and getting the real-time information from Salesforce and returning it back to the agent.
Agent all it needs to do is essentially take the next action whatever that might be. It may actually call another tool set of tool or it may in this case summarize a result and return it back to the user return back for example in this example it's set of two opportunities which are figure out from the system. So with that background let's talk about what we did. We are for our implementation we are using mosaic AI platform essentially um in the in the in the previous uh uh thing you may have heard about AI bricks that's the next evolution but we had last year we announced the mosaic platform which is the agent platform we are using for our implementation um which is essentially serving the agent through our model serving endpoint we have deployed the agent in the uh in the platform which is essentially giving us a uh agent serving endpoint which is what we have used but we have before we actually deployed the agent we had to work a lot on the data so uh it's super important to make sure that your data quality is good we wanted to make sure our sellers can trust the output of the agent and thereby we created a gold data set for our agent for specific data sets for some of the tools the agent itself is a custom implementation we started with lang but we pivoted over to our own implementation for certain design considerations And um when we first launched our agent um there were a lot of question from our business.

22:48 Building and deploying agents

What is the efficacy of this agent? Is it working? Is it returning the right result? And we had no data to prove it. So we had to write our our evaluation criteria and the mosaic AI platform actually provided a rich way to set the evaluation judges which is what we have implemented in our system. We started with a pilot and then we launched it to all the 4,000 sellers. So right right now it's available to all our sellers and they're using across the board.
Let's talk about the components of the system. There are four components. First is the UI layer. Uh you may have observed it or it may be not very obvious. It's a Chrome plug-in. The way we overlaid our agent on top of Salesforce, we essentially enable a co Chrome plug-in which is deployed systematically across all our agents, all all our sellers, right? So, it's available in the browsers uh by default. And the Chrome plug-in itself is a React application.

24:23 System components and security

So, uh it was important for us to make sure that as the new tools are introduced, the contract between the back end and the front end is kept whole. So we we came up with our own uh markdown to provide a good experience good richer experience. So along with the standard markdown we have come up with a construct for graph-based displays. Okay as you as you see that it's in the flow that means no matter where you go in the salesforce uh chrome plug-in is available to you in the context.
Um it can be extended to other application in the future. You know we can look at other surfaces that the seller goes to and we can extend the chrome plugin to those surfaces as well. The front end talks to our back end. Uh it's essentially a oath proxy. Uh in our experience, you saw that I was logged into Salesforce. But all the calls that agent is making a call back to Salesforce or Glean, they're happening in the seller context.
You only see the information in the agent that is that you have access to in Salesforce and in glean, right? And the way it happens is when you when you log into Salesforce, we essentially use our back end uh OOTH to do a token exchange with Salesforce in the back end. And that allows us to and we pass a token back to all the way to our tools to actually make the call back to Salesforce. So that was hugely important for us to provide a seamless experience.
We don't want the user to grant permission to Salesforce separately to green separately and you know multiple grants experience is something that we wanted to avoid. The agent is itself is running on the basic platform. Like I said, it's a custom implementation. All all our tools are UC functions in Python and SQL and they are governed by UC catalog. The data layer is essentially primarily we created our warehouse in our in our platform to make sure that all the data all the gold data set that I talked about earlier is available to the agent as well as for some real-time information some of the tool goes back to Salesforce or to glean uh and in case we need to write to Google drive it writes back to Google Drive as well from architecture perspective um when the request comes from the end user it goes through our proxy on AWS and hits it hits a model serving endpoint.

26:53 Architecture and evaluation

The agent is deployed in our model serving endpoint and the agent has access to all the available LL models. In our case, we are using at this point we are using GPD40. Uh but we continue to experiment with different LLM models all the time. Right? The agent itself called the tool in the serverless compute right and the tools have access to any any resource. It can be it can make a call to vector database um or now to AI brick that was announced today or it can be it can go to gen room or to a external service like uh Google drive or G Suite or or Glean or Salesforce.
All of this interaction is captured through ML flow traces, right? And that is super beneficial for us, you know, because the Mosaic AI platform provides a very rich uh evaluation framework that allow us to create their judges available out of box, but it allow us to extend it and create custom judges as well, evaluation judges, right? Which is what we have done in our implementation. We essentially extended the standard judges evaluation judges and we created our own relevancy judge to extend the implementation that we that came up uh in the in the platform.

28:05 Continuous tuning and feedback

We also did the ground grounded judge which essentially means whether the reagent response is grounded in the data that was provided to it. And we also implemented uh the tool selection judge. And what I mean by this is u when we started with our journey and we started adding more and more tools now we have more than 60 tools we started to realize that our tools are conflicting with each other. Remember I said that the tools are selected based on the description.
So if the two tools are have the similar description agent will get confused which tool to book and that was happening a lot right. So we wanted to make sure that the description is accurate and we introduce a judge to make sure that uh the judge is helping us to figure out whether when we introduce a new tool uh is it clashing with the existing set of tools. So it helps in the regression testing a lot. We also use the um uh this tracing data for operational metrics.
For example, how many requests are coming in you know week over week uh uh trend on the agent. All of that business metrics and operational metrics are available through the trace tracing. We have created our own custom AIBA dashboard on top of the traces to help us with analytics information. Um I forgot to show one piece you know I can go back and show that there is a feedback mechanism as well. you know I can give thumbs up, thumbs down and I can provide this feedback that feedback is also available uh through the ML traces for us for our tracing mechanism right so we can provide we can give this information back to the product management to analyze as well as we can uh we can make it useful for our future improvement of the to of the agent.
So that's how we implemented our implementation. So let's talk about the lessons learned. Yeah, along the way in our journey we learned lot of lessons. First one is which is the most important is data is messy and with data is scaling to 10x on structured as well and unstructured data it's important to find the right sources of truth and have a clean data uh structure and make sure that it's governed by UC. Um we I'll give an example.

30:05 Lessons learned and conclusion

We have ARR in multiple systems. So we want to make sure that we use the authority source of truth and clean data everywhere. So this is super important and we we had to do a lot of work on this area to make sure data is clean. Uh second is managing user expect expectations right if you are building a chatbot you need to make sure that user expectations are set around what the chatbot can do. The expectation becomes that you can the chatbot can answer any question even if it is not salesreated.
In this case, user can user asking users are asking a lot of uh different questions. So, how do we solve that problem? One is educating the users about telling them the boundary conditions and what the agent can do or cannot do. That's one. Second is increasing the footprint of the agent itself uh by adding more and more tools and filling the gap where the gaps are. So, these are the two important lessons we we saw in this this area about managing the expectations.
And third is uh measuring the ROI is very difficult like once the user adoption happens then only you know what the RAI is you have to start with some assumptions and start building your agentic experiences and once the user adoption happens you need like said like measuring the groundedness or or accuracy relevancy is is very important and capturing the user feedback as well to make sure that what we are building is really useful for the user or not and tracking those.
So what we are doing is we are gathering all these feedback and we are categorizing the questions into answerable or non-answerable. If the questions are not answerable is there a reason is it in the scope and what are the topic uh topics uh the question belongs to. So we are doing a topic classification as well to figure out what are the most demanding tools that we should build. So that is one of the important aspect of this and the last is change is the only constant uh user the technology is changing every every week if you look at in the in the geni space and keeping up with is uh uh is difficult as well as user demands are increasing as well they expect a lot so you need to continuously evolve your model if there are thousand tool you can't build thousand but you need to find out the perfect technology to deliver your aentic experiences.
Yeah, that's all we had. Uh we can take a few question. If you want to ask a question, maybe you can come up to.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.