Building Explainable AI Research Agents with Databricks Genie

Summary

CVS Health built a Genie Research Agent on Databricks to help retail leaders understand forecast drivers across 7,000 stores, reducing analysis time from 1–3 days to minutes for demand forecasting questions.
The agent uses Databricks Genie Spaces for natural language Q&A on governed data and agent mode for multi-step root cause analysis, grounded in Unity Catalog metadata, business rules, and SQL expressions.
By automating ad hoc forecast explanations, the solution freed data scientists from manual reporting to focus on improving model accuracy, while making retail intelligence accessible to both technical and non-technical users.

Building Explainable AI Research Agents with Databricks Genie

Answering complex forecasting questions across 7,000 retail stores traditionally required days of SQL analysis and manual report writing. Data scientists spent more time answering ad hoc business questions than building models, while stakeholders had no clear explanation for forecast drivers. CVS Health built an AI Research Agent powered by Databricks Genie to deliver fast, plain-language forecast explanations grounded in governed data.
Learn how Genie Spaces transform dashboards with natural language Q&A, how agent mode enables multi-step root cause analysis across unified data, and how to configure Genie with Unity Catalog metadata, business rules, and SQL expressions. Discover how one agent reduced forecast analysis time from 1-3 days to minutes, freed data scientists to improve model accuracy, and made retail intelligence accessible to both technical and non-technical users.
🤝

Chapters

FAQs

What is Databricks Genie and how does it enable natural language data analysis?

Databricks Genie is a domain-specific AI chatbot that transforms dashboards with natural language question-and-answer capabilities. It allows users to ask questions about their data in plain language and receive grounded, governed answers without writing SQL.

How did CVS Health configure the Genie Research Agent for retail forecasting?

CVS configured Genie using Unity Catalog metadata, business rules, semantic definitions, and SQL expressions specific to demand forecasting. They also defined joins, relationships, and business logic so Genie understands domain-specific context when answering store-level questions.

What business problems does the CVS Health Genie agent solve for store leaders?

Retail leaders previously had no clear, fast way to understand why a specific store's forecast differed from expectations. The Genie Research Agent delivers plain-language forecast explanations grounded in governed data, replacing a process that previously required 1–3 days of SQL analysis.

How does Unity Catalog support the Genie Research Agent at CVS Health?

Unity Catalog provides the governed data foundation that Genie queries, ensuring all answers are grounded in authoritative, access-controlled data. CVS used Unity Catalog metadata and AI-powered metadata builder tools to define table descriptions and semantic context that improve Genie's answer quality.

Full transcript

[00:07] Good morning everyone. Can you hear me well? Nice. So, we have a speaker today, um, Adrian Sun and the Chad Novick. We are from CVS Health and Rohan Parikh is our Databricks partners. Today we are going
[00:23] to introduce what is Genie Research Agent, why and how we build this Genie Research Agent to support thousands of retail leaders to understand not just what happened, what's going on for a store, but why. Before we get started, I want to spend
[00:38] some time to introduce our speakers. So, I will start with myself. I'm a senior data scientist at CVS Health. My My primary focus is on the demand forecasting and support the business leaders to do planning decision every
[00:53] week. So, over the past 2 years, I work on everything from machine learning model to auto reporting, process optimization, to generalize solution. And that journey actually lead us to this stage to build a Genie Research Agent and we are going to share today.
[01:10] So, let me pass it over to Chad to introduce, uh, yourself. Thank you. Go. Thank you, Adrian. Yeah, of course. Um, hey everyone. My name is Chad Novick. I'm a lead director of data science at CVS and, uh, I lead basically our labor demand modeling team. So, we
[01:26] answer how much labor to put in the stores as well as, um, forecasting for where we're not going to meet those labor needs in the future. So. And Rohan. Thank you, Chad. Hey everyone. My name is Rohan Parikh. I'm a solutions architect at Databricks. Uh, I've been
[01:41] in the, uh, healthcare space for the past 10 years, uh, but the last 2 years have been dedicated to supporting CVS and excited to talk about our success story together. Great. Thank you, Chad. Thank you, Rohan. So, now you have already met the team behind this project. You can see it's a
[01:57] highly collaborative effort between data science, business, and also Databricks. So, I will simply go over what we are going to cover today. First, we will talk about the retail forecasting challenges from data science perspective, from leadership perspective, and then how we come up
[02:12] with the Genie solution, and then we'll jump to a live demo. I think that's the most exciting part of this talk, and then we'll come back to share what benefits Genie brought to us, and finally we'll do a short recap uh to share what we uh you can take away.
[02:31] All right, let's dive in. So, I will start the retail challenge from data science perspective. So, when we are doing the forecasting, every week your business partner will ask a lot of questions. For example, why are we under forecast? Why are we over forecast? Why did sales drop? Is this due to weather?
[02:48] Due to promotion? Due to an event? Store closure? So many kind of questions posed from um the leadership and also from the field. So, the real challenge was not a forecasting problem itself. It was a explanation problem, right? The
[03:04] traditional way we are handling those kind of ad ad hoc requests is like the business user, they will generate tickets and send to data scientists, and then we need to review the problem store by store, and then do a scenario testing from hypothesis, and write a final write
[03:20] table, and do query, and also frame the story, finally write a report, send it back to the stakeholder. And it sometimes takes 1 to 3 days later. Um so, how can we uh respond to those questions at scale and then also deliver the answer they actually needed?
[03:37] So, at this moment, you probably have a solution in your mind. We have dashboards, right? Dashboard can solve some of the questions, but the problem is like everyone, like everyone of you here, look at the same dashboard differently. So, some people might want
[03:53] to see the regional view, and some people they probably want to see the store level sub-specific details, and some people want to see the like product category deep dive. So, different requests. So, the data scientists we keep adding charts, adding adding
[04:08] matrices, adding different diagram, eventually the dashboard becomes a monster. So, only the one who truly understand it is the developer. So, they still need your help to navigate through the dashboards. So, the dashboard itself like cannot
[04:24] self-serve. So, that's the problem. So, that's the challenge from our data scientists perspective. Now, I want to invite Chad to share from the leadership perspective how this challenge will impact the organization. And I also want to invite Rohan to talk about why do you
[04:41] think Genie can be a good fit for this use case. Yeah. Cool. Thank you, Adrian. And I think the the biggest part of this is really our data scientists are operating more like analysts. Unfortunately, you know, we have brilliant people like Adrian who are spending their time writing SQL queries,
[04:57] answering the questions, and formatting emails. When really the end problem is that we just don't have good enough reporting on our forecast accuracy, on what's driving the forecast. Our stakeholders are not going to look at shaft plots, they're not going to look at
[05:12] a lot of the visuals that we use to understand what's driving the forecast. And they need it basically in a nice text explanation that often times takes a lot of joining in SQL to basically get to a point of easy explanation. So, from just a time management perspective, want to basically enable
[05:28] people like Adrian to spend more time on development than on writing queries and answering emails. So, that's kind of the biggest thing for us. And then also building trust with our stakeholders is really important. And being able to answer immediately to
[05:43] the requests as as to maybe like hours or days if it's something really complex is a huge help and has actually really helped build trust in our models and also in our conversations with various partners like finance and merchandising. So,
[05:59] that's been the clearly the the biggest help for us and I think also in the time that you've spent on the models now that we've deployed Genie, we've actually improved accuracy quite a bit. It's not the only thing that's helped us. So, it's it's definitely enabled us to to move much faster. I've been one of the
[06:15] biggest tools there. So, and everyone. So, I want to give you all more context about how I approached this. I want to take you back to middle of last year when I first had this conversation with the CVS team. We It when we first
[06:30] started speaking about this, it started feeling like a reporting problem. Maybe we need more charts. Maybe we need to show the right visualization type. But after speaking to them more and more, it seemed that wasn't the problem. We also made sure that we checked that here like did you have the right data
[06:46] granularity? Like do you all are you all forecasting pipelines running correctly? And it made me realize and and everyone in fact that hey, the problem is starting after the prediction is made and even after the dashboards are published. And the reason is because every time a
[07:02] stakeholder would ask why, the charts wouldn't show the answer. It wasn't enough. So, what we started wondering is how can we solve this and that is where the Data Bricks Genie Spaces came into play. And for people who are not familiar yet
[07:19] with what Genie Spaces are, these are going to be domain specific natural languages natural language answering chatbots that end users can use to ask questions in natural language and get answers. Note that I specified domain specific because you can stand up specific domain
[07:35] rooms Genie Spaces that is and then basically configure it to make sure you get the right answers pertaining to this specific business problem. So when we started playing around with this, there is obviously an art to it. There are ways you can configure it, but
[07:51] note that this is not a simple chatbot itself. Towards end of last year, CVS team was one of the first ones to try out our agent mode as well in the Genie spaces and that essentially enabled them to perform multi-step root cause analysis across various data
[08:07] sources hosted in Unity catalog so it would give you a more accurate answer grounded in truth rather than just a simple Q&A. So that really started helping all of the end users and rather the data scientists get answers quickly rather
[08:23] than spending days and days from their traditional work into getting answers. So this really enabled them to get answers fast, trustworthy, and made sure that this can be given to end users very quickly and help their business problems. But rather than me talk more about it,
[08:40] Adrian, why don't you talk? Show them how it works and you've been working on it day by day for the last year. Yeah, thank you Rohan and also Databricks team bring such a great solution to us. So now with Genie research agent, we'll be able to transform our operating model from the
[08:56] traditional push model to a pull model. The business user now can directly talk to Genie and the Genie will perform the research and return the answer automatically. So we no longer need 100 dashboards for 100 user. We just provide
[09:12] 100 different perspective from the same government data foundation. Our ultimate goal is try to democratize our analytics works, bridge the gap between the data aspects and the decision makers. We are trying to make this retail intelligence more accessible to the end user, both
[09:29] technical and non-technical users. Right, I think it's the time to the demo. So here, let's start with the world without Genie. It's a dashboard. So here I want to emphasize this is the
[09:44] synthetic data. We use the PySpark data chain to simulate a retail world. There were several sessions covering that topic, so I won't go into detail here. So let's back to the dashboard. This is a very typical sales performance dashboard you can see anywhere in your
[10:01] organization. Like we have sales overview, promotion effectiveness, or a store performance and efficiency, regional and market analysis, so on and so forth. You can see the dashboard has already have a lot of information.
[10:16] However, it still cannot satisfy all users' need. So what can we do about it? The first thing you can do is to add values to it. For example, you can connect your Genie research agent to your data. For example,
[10:32] uh if you want to check this this line chart for monthly sales change, and you see there's a dip here, we can click the small icon on top right, open up the chat window. I already asked this question for you because this question takes longer time,
[10:47] so I loaded it for you. So what I basically asked is just like why did sales drop in September 2025? Then you can see how Genie thinks about it, and investigate all the relevant data source, reading them through it, and finally will generate a summary
[11:03] report for you. So the conclusion for this question is that because the data in September is not complete, and that is true because when I doing the simulation, I intentionally have a sale the sales data just set to September 16,
[11:19] so we only have 2 weeks of data. And then we just ask a one one sentence, and then Genie can identify the root cause based on all the our fundamental data they learned. So you can do the same for different, like, visualization. Like, you can click
[11:34] on different parts, and then you can see the uh the Genie room will, like, select the visualization. You can ask questions to any of the visualization in this dashboard. But, we still have questions outside of this dashboard. So, we can go to the
[11:51] dedicated Genie space. Right. So, this is the interface of Genie space. So, you can have a clear description of what this Genie space is working, the title, and what the data models, and uh what's the business
[12:08] definition. Because for end user, when they just open up a blank page, they probably have no idea how to do it. So, it's sort of like a manual. So, you can follow the steps. So, why not let's just try our first questions.
[12:29] Let's like, tell me the top performing stores. Right. And hit enter. Now, we submit the questions, and you can think uh you can see that Genie is actually thinking right now. For the sake of time, uh let me switch back to the slides.
[12:45] I'll we can go over how we make this happen. So, the first principle is actually not AI. The first first principle is data governance is really critical. So, one thing we learned very quickly is that if you throw
[13:00] LLM with the massive data, it will throw you the massive answer. So, the this is very critical thing. So, we register our data on Unity Catalog. And on Unity Catalog, there is a AI-powered metadata builder. So, you can simply set a
[13:17] description, and we will collaborate with domain experts to validate those descriptions to teach Genie the business meanings. So, let me show you the interface of in Unity catalog so you can have a sense on what is that about.
[13:33] Right, this is the example like catalog explorer. We are using the GD demo sales table. Here you can see, let me zoom in a little bit. Okay. So, you you can see the column and also the data type and also the
[13:48] comments. So, the comment here is a very important area that you can teach Genie and leverage the AI-powered metadata generator. Let me just click the edit icon. Sorry, let me zoom out.
[14:08] All right, let me do it this way. Okay. So, you can see there are some Genie icon here. You can uh generate those description automatically and then you can collaborate with your domain expert to validate those description. For example, the date here means the sales record and
[14:24] it it is also a foreign key to join with other tables. So, you have all your domain knowledge, you deal with all kind of ad hoc questions every day, so you know what tables to join together. You can define it here. So, they know the meanings. And you can also preview the sample
[14:40] data. And then ask some what questions. What is the sales trend over time for the open store? For example, and then Genie will think through it and then write a sample query here and then show the results here. All right, so you can leverage the uh
[14:58] Genie here to explore your data table. Now, let's go back to the our first questions. Right? So, Genie has already provided answer. So, let's first check the what Genie is thinking.
[15:13] So, Genie is identifying the top performing store by total sales. Perfect. So, uh Genie provide a horizontal bar chart for your review, but if you want to change the layout, you can basically click edit button here.
[15:28] And then you can see you can change different visualization type. And also if you do want to um do some cross analysis, for example, I want to understand which store and the cross analysis with region, you can simply like change it
[15:44] here. So it's really simple. If you don't want it, you can just remove it. And then you can download this visualization to save into your slides or you can directly add to your dashboards. So a lot of flexibility you can play with here. You can also change the
[16:00] color. And then the table is the same. So for example, we we are just talking about the um total sales and the top performing, maybe some features are irrelevant. So let's remove some example like the days
[16:15] open probably not relevant. We just remove those information. And then you can also twist the languages by interacting with this Genie space. Just like adding this to the chat Genie. So whenever you finalize your report, you can just download the PDF. Or in the
[16:32] future, now we we know we have Genie 1, so we can automatically send it out. Right? We just see how Genie performed on the what questions. Next, let's try the why question. So we can see um
[16:47] yeah, this is a live demo, so I I also my first time seeing the answer. So the top performing store is actually store A37. So let's just check why store A37 is performing so well.
[17:04] Yeah. Right? Let's give Genie some time. Now Genie is thinking. So let's go back to the slides. So we just talked about metadata. So metadata is not enough. Business context matters. So, if we we if we
[17:22] don't have context, Genie just see data itself, but with context, Genie can see the business. So, we didn't say uh we didn't tell Genie what does top performing means, but Genie knows it. Why is that? Because we have a dedicated Genie space, we can
[17:39] do all kinds of configuration. So, this is a retail specific Genie space. So, we only feed the retail specific like knowledge. Like some concept like promotion or holiday season or some key metrics for time series forecasting or what kind of like
[17:55] business flow, for example, labor planning. We usually compare with the budget. Those kind of information can be embedded in this Genie space. So, let me jump to the uh Genie room, so you can have a clear view. I will walk through how we do a setup.
[18:11] So, in a Genie space, there is a configuration button. It's very simple. Let me enlarge this a little bit. So, the first step is the about. This is the general information, like a manual. Like uh remember when we first open this up, you
[18:27] see those information, you can edit here. You can also provide some frequently asked questions here. So, whenever the user just don't join uh that open up the conversation, then they can direct ask some like uh repeat questions here.
[18:43] The next is the data tab. And we also walk through the Unity catalog. So, here you simply import the relevant data tables here. Right? This is simple. You you can also do a edit here. So, like for the data columns
[18:58] descriptions, you can do a edit here, same as the uh Unity catalog. And the third one is the most important one. Here is the Genie spring. So, the first in general instructions, I won't repeat here. So, basically the
[19:13] information you can see at the about tab. The second one is the joints. Yeah, as I said, like uh you deal with a a lot of questions. You know what tables to to join together, you can define here. And then just define the Make sure the key
[19:29] are matched. And the third one is the SQL expression. I think this is the most important one because this is place which teach Genie business context. For example, in this uh example dashboards, we we are learning like sales performance. So, you
[19:45] may want to know like what's the sales per square feet, and then you can click edit. You can see it's the equation. It's sales divided by the uh square footage. And let me show you one more What's the same thing for sales per staff member?
[20:02] It's more like operational efficiency, by the way. So, it's sales divided by the uh yeah, staff count. Right? So, you can also do measures. For example, the total sales, it will be the sum of the sales.
[20:17] And the same for the customers. And you can also do filters. For example, you just want to understand the sales performance on the promotion day, you can also like set it here. So, all kind of like business meanings and the concept or restrictions, etc., you can
[20:33] set it here. I just did a few of example for the demo purpose. And our real like uh retail uh agent, we have a lot more. So, the final tab is the SQL queries. So, instead of just giving some knowledge or context about the some
[20:50] business terms, you sometimes you already have a answer to answer a specific questions, then you can directly write it here. So, in our real use case, we have already trained our uh Genie space for around 9 months. So, now we have a lot
[21:06] of training questions. That's why the Genie can be smart. Right, that's the uh very simple setup as you can see, very user-friendly. So, now let's go back to see how Genie answered the why questions. All right, so let me make it bigger.
[21:24] Okay. So, that's the first thing we want to see like how Genie think through this problem. We are trying to compare that I guess store A37 and the other top performers based on the key characteristics and the key metrics. So, you can see why
[21:41] store A37 performing so well. Yeah, it provides like several like advantage. For example, this is the second largest store in that area. And also uh a lot of time they are on promotions. That's why they can bring a lot of sales.
[21:57] So, on and so forth. You can see like uh Genie link to all the relevant tables, find all kinds of possibility and reason and provide the answer. Um I will not like repeat the the answers here because we're not here to learn why store A37 is
[22:13] performing so well. So, but you can see the capability. They provide a lot of scenario. So, traditionally if we check this problem, we need to think what kind of possibility it is and then we'll go to find a right a right table and do the joints. Now, Genie can directly
[22:30] implement all kind of scenario and provide that to you. It's really convenient. Right, let's try one more questions. So, um
[22:45] compare the performance with the chain average. This is also a very frequently asked questions. Right? So, this questions, what I'm trying to test is if we can get this conversation going. Um this time I do not say like store
[23:02] A37. Like in the previous questions, I do say like uh why store A37 is performing so well. So, I want to see if Genie can actually remember the context. So, let's see how Genie's thinking. We probably um we don't need to wait for
[23:19] the answer complete, but you can see in the thinking process. Genie's actually working on it, like I'll compare the store A37 performance against the chain-wide average. So, Genie can remember the context. So, that's the power of the conversational AI. And the
[23:36] most important thing is you can see the entire investigation it just take a few minutes. So, just imagine we open this up to all the retail leaders, they can like uh self-serve and answer their own questions. Right? Um, for the sake of time, I
[23:53] probably don't try more questions, but if you are interested, feel free to come and play after this session. Right? So, we just covered the conversational intelligence and we also demoed the uh some exact example questions in the
[24:09] demo. So, I probably won't repeat here. I want to go back to the very beginning about our challenges and what happened after we adopted Genie. So, compared to 1 2 3 days response time, now we significantly reduce the
[24:26] turnaround time for the ad hoc request. So, it frees up space for our data scientists. We can spend more time improving the time series model, do more advanced model to improve the models. And so, you can see the forecast accuracy is also improving. So, Genie is
[24:42] not only helping non-technical user, it is also helping the data scientists to facilitate the entire demand cycle. And because of that, the stakeholder now trusts us more because they can understand why's much faster.
[24:58] So, they have more time to react and plan. So, the biggest takeaway from Genie is that they are not going to replace analysts. They are just like generating high-quality analysis and making more available to everyone. That's the point.
[25:14] Right? So, at the end of this talk, I wanted to some recap to show what we learned and then what's something you can really copy and uh paste to your domain. So, we covered three important elements. One is
[25:30] domain knowledge and data governance and also conversation AI. Remember, our problem was not a forecasting problem or any kind of problem in your domain. It was a explanation problem. And the stakeholders, they don't want more dashboards. They just want answers.
[25:47] Right? So, with these three elements, we covered the Unity Catalog. We used the AI-powered metadata builder to give the definition, to teach the business meanings. We also go over the Genie room. Like we showcased all the
[26:04] configuration steps like to teach the business flow, business logics, and some caveats, etc. So, just like onboarding a new hire, you need to prepare a lot of documentation and do knowledge transfer, right? And we also demo the live conversation.
[26:20] We actually see how this new high new high behaves. So, it can respond in just few minutes and do a really great job. So, with all these three together, we have successfully conquered our challenge not by generating more dashboards, but by helping the
[26:35] stakeholder to get their answers much faster. And it's all because of Genie research agent. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.