Anthropic + Adidas + Databricks: Unlocking 400 Hours of Productivity Weekly with Conversational AI

Summary

Adidas deployed Claude via a multi-agent orchestrator called D.Va to give 200+ marketing and CRM users conversational access to data, compressing insight-to-action cycles from weeks to days and unlocking up to 400 hours of productivity weekly.
Anthropic's applied AI team explains that agent harnesses must be co-designed with the specific model underneath, as model-harness fit has a measurable impact on accuracy — Cornell PromptBridge research found that optimized harnesses achieved 98% versus 92% accuracy with a default setup.
A production semantic layer was a non-negotiable prerequisite for D.Va, serving as the context engineering foundation that allowed Claude to navigate Adidas's data landscape reliably on the Databricks Data and AI platform.

Anthropic + Adidas + Databricks: Unlocking 400 Hours of Productivity Weekly with Conversational AI

LLMs are transforming how enterprises explore data and make decisions. But how do you integrate conversational AI with existing analytics and data ecosystems? In this joint session, we'll explore the common use cases and design patterns that leading enterprises are adopting. Anthropic will share insights from across their customer use cases on model selection, serving infrastructure, governance and cost optimization, and integration with downstream BI & applications. Then, Adidas will walk through their production implementation: how they deployed Claude via a multi-agent orchestrator to enable 200+ marketing and CRM users to explore data conversationally, compress insight-to-action cycles, and unlock productivity gains. Adidas unlocked up to 400 hours of productivity weekly and accelerated insight-to-action cycles from weeks to days Learn how Claude, Agent Bricks, AI Gateway and Databricks Genie come together to transform analytics from a bottleneck into a competitive advantage.
Talk By: Hosea Kidane, Member of Technical Staff, Applied AI, Anthropic ; Vikalp Yadav, Sr Director Of Digital, adidas AG ;

Chapters

FAQs

What is D.Va and what did Adidas build it for?

D.Va is a conversational analytics agent built by Adidas that allows marketing and CRM users to explore data through natural language instead of writing SQL or waiting on analyst teams. It is deployed on the Databricks Data and AI platform, powered by Claude, and has accelerated insight-to-action cycles from weeks to days for over 200 users.

How did Adidas measure the productivity impact of D.Va?

According to this video, Adidas unlocked up to 400 hours of productivity weekly and compressed insight-to-action cycles from weeks to days after deploying D.Va. The system enables business users to ask questions in plain language and receive governed, accurate answers without relying on data analyst intermediaries.

Why does Anthropic say model-harness fit matters for agent accuracy?

Anthropic's applied AI team explains that everything built around a model — instructions, context, tools, guardrails, and error checks — must be tuned to the specific model underneath. Research cited in this video from Cornell University found that an optimized harness achieved 98% accuracy compared to 92% for a default setup, demonstrating the measurable impact of proper model-harness alignment.

Why was a semantic layer a prerequisite for putting D.Va into production?

Without a semantic layer to resolve ambiguous data terms and business metrics, a conversational AI agent cannot reliably answer questions about real business data. This video describes context engineering as the currency of production AI, and the D.Va architecture treats the semantic layer as a non-negotiable foundation — without it, agents return untrustworthy or hallucinated results.

Full transcript

[00:08] Good afternoon. Uh, thank you for joining this session. My name is Hosea. I am a member of the technical staff on the applied AI team at Anthropic. Our team takes the learnings from our product and research teams to help
[00:20] customers build impactful AI agents. Before Anthropic, I spent four years at data bricks, specifically working with our retail and consumer goods companies. With me is Vik, senior director digital
[00:34] data and AI at Adidas. His team built D.Va, a conversational analytics agent that accelerates time to insights for analysts and it is in production on data bricks powered by cloud.
[00:48] So as for today's agenda, we'll start with how we at Anthropic think about evaluating models for agents. Then VOP will take you inside D.VA covering why
[01:00] Adidas built it, what they learned along the way, and why they chose Claw to power it. And then we'll close out with some Q&A. So want to kind of start with the statement here. you know, day-to-day we
[01:14] see customers putting in a lot of their effort on building like the perfect agent harness, you know, getting it working with their preferred model, uh, but then never really touching it again, right? And when a new model comes out,
[01:27] they swap it in underneath, you know, under that same harness and just expect things to get better. I actually remember a time not too long ago when I had to convince a customer to, you know, totally replace their
[01:40] complex rag system that took them months to implement um and replace it with just one MCP tool. So things do change quite fast. Uh but one thing that's kind of stayed consistent is that models are
[01:53] getting better. They're getting better faster and the harnesses around them must adapt very quickly as well. Um, and when I say harness, you know, we mean everything built around the model
[02:05] to turn it, you know, from just having a raw capability into a result that you can actually use. So that would be like the instructions you give it, the context, the tools you hand it, uh, the guardrails around it, and the checks
[02:18] that catch its mistakes. So every one of these pieces has to be tuned to the specific model underneath. And how much instruction, you know, does it need? How much context can it really hold? How
[02:30] literally does it follow your rules? And where does it, you know, tend to slip up? So, if you change the model, you kind of get a different answer for for all of these. And we know this to be true anecdotally
[02:43] at anthropic, but there's also measurable research that's been done by Cornell University. There's a study uh framework called prompt bridge where they took a model tuned for one model um
[02:56] sorry one prompt tuned for one model and moved it unchanged to a much stronger model and with that old prompt the stronger model scored around 92% on their tests but with a prompt actually
[03:08] tuned for that stronger model it reached 98%. And in our experience, these are, you know, a few percentage points, but they really do matter. And it could be the difference between something you demo versus something you actually put into
[03:20] production. Um, and that gap is exactly why we care so much about getting the harness and model fit just right. And every time we release a new model, we publish a prompting guide alongside it so that you
[03:32] have exactly what you need to tune your harness to that specific model. And because that's the purpose of a harness, right? to make sure the model reaches its full potential. Um, so how do we actually measure that
[03:45] fit and what should you do to your existing harness when a new model comes out? The answer is eval right first you'll define, you know, your tasks. This is where you want to spend the most amount
[03:57] of time, but it doesn't really have to be that complicated. You can define 20 or so tasks or more that are critical to your application and have a clear pass fail grading criteria. And you'll know a
[04:09] task is good enough if two domain experts can independently look at the output and reach the same pass fail verdict. And if they don't agree, that means your task is probably too vague to
[04:21] be measured by an eval. The next step is actually to get a baseline of the models. We call that finding the floor. So you
[04:33] run your tasks through each model, record some baseline outputs, and you'll want to analyze these by hand at first to understand where the models are failing and what kind of output each one is producing. From there, you determine
[04:45] what task can be checked with code, which ones need maybe an LLM as a judge. Uh but you want to do that manual pass first before you scale up with with LM judges. And that's to ensure that you understand the failure points uh
[04:58] yourself. Then of course you score the models on quality um latency cost. Every model is going to have some trade-offs between those three and you have to decide what's most most important for your use case. So this is what we call
[05:12] finding the floor. Floor is kind of for filtering that first stage between all the different models and you'll probably screen out a lot of different models based on just latency and cost alone. Then you want to do finding the ceiling.
[05:25] So this is where where you take what you learned from, you know, the first set of traces, the specific ways each model has failed, shape a separate harness around each one. Uh that could mean adjusting the prompts, adjusting the guardrails,
[05:38] maybe loosening or tightening them, uh or handing it different tools, and then you rerun those eval, you know, make sure that the harness is tuned and you're getting the best results possible. Um so the floor is for finding
[05:50] out what the model does on its own. The ceiling is basically trying to figure out what's the the potential and what would the end result look like for for users. Then of course you select the model right and you attach your production
[06:02] harness around it. Um so really end to end we're talking about selecting your task for your evals getting baseline scores tuning the harness to get maximum output and then selecting that model
[06:15] harness combination that's going to be performing the best. Um, but we always want to keep in mind that every piece of the harness is encoding some assumptions about something the current model can't do.
[06:28] And so you don't want to forget to re-evaluate when a new model comes out. Uh, your assumptions may grow stale and you may have built something that compensates for an existing capability
[06:40] but may be holding the new model back. There's an example of this uh from our experience back when sonnet was really at the frontier. We noticed it had something called context anxiety. So
[06:53] this is where it would wrap up tasks kind of quickly as it was nearing its context window limit. Um and we built a harness to compensate for this where it would reset the context kind of write off some handoff notes and then clear it
[07:07] to the next session. But new models don't really have this anxiety. So we had to rewrite this harness uh including something that you guys are probably well aware of called compaction uh that allows claw to continuously run for
[07:19] hours. So a new model release is a signal to go back to this step two and kind of repeat this process as you uh as you tune your harness. Um and we do this ourselves of course
[07:31] every time we release a new model. Um and thousands of data bricks customers are doing it as well running it across their own eval. And a lot of them tend to land on claude as the final model. So what are they seeing that that gets them
[07:44] there? So Adidas is a great example. They built their own evals uh ran it through a bunch of different models. Uh and Claude came out top for these reasons. First is
[07:56] instruction following. So D.VA gets valid schema output from Claude 99% of the time. and Claude had the lowest rate of schema violations of any model they evaluated.
[08:08] Valid schema is what keeps the routine like the routing chain from breaking, right? So if you have a malformed response, that's a retry or added latency that is going to affect your users. And on tasks that demand a very
[08:22] strict format, you know, a routing decision, a specific tool call, Claude returns just that. On long context coherence, D.Va Eva classified user intent correctly 97% of
[08:34] the time. And to hit that metric, Claude has to hold Adidas's metric definitions, YAML documents alongside
[08:54] Reliability comes next. When paired with citations, Claude has the lowest hallucination rate of any Frontier model. Uh, and for an analytics agent where users are going to be making real decisions on these answers, they need to be able to trust what comes back.
[09:11] And most importantly, at Anthropic, we care a lot about safety and safety research and model uh development uh are actually combined into one process, which means safety is trained into the model itself. This gives you a baseline layer already
[09:25] built in working alongside any additional guardrails that you may add into the harness so that the layers you do build are going to be lighter and cost you less in latency. So these are some pretty amazing
[09:37] numbers. Vicap uh please tell us more about div.a. >> Thank you for for a moment when you showed the numbers there. Um I thought I'm as an
[09:51] analyst I'm going to uh lose the job. The benchmarks are like really high. But then I said hold on I'm the one who is presenting DV on stage today. Hi I'm Vikalp and before we jump into this I
[10:04] would also like to credit what's what's the mystery behind this DVA. So this DVA is the name like the decision intelligence virtual analyst. I I I credit this name to my partner in crime church who's not able to join me on the
[10:16] stage today but we was together we uh we we are supposed to present this um DVA. Now this is uh how uh we leverage these principles which uh you
[10:28] know we just talked about and uh in one of the biggest sports brand in the world and this is what I want to walk you to today.
[10:42] In 2005 uh we we kind of built a data warehouse. Yeah. We we collected all the data sets that we have together all the distributed data sets. We brought that together and we went to our business stakeholder and said, "You know what?
[10:55] Here you go. Uh, let's start using this." And I said, "Okay, great. Uh, can I ask it a question?" So my response to him was or the analytics team responded, "Hold on, let's check with the analyst."
[11:12] 2015, we built beautiful dashboards. We kind of really uh, how you call uh, self-service the heck out of those dashboards. And then we went back to the uh analyst community or the business
[11:24] user community and then we asked them here you go and then they said okay can you tell me why did the net sales for Germany drop in the last week said hold
[11:36] on uh let's get the analyst involved we used all the best and the greatest of the uh features the functionalities that
[11:49] the the semant semantic layer the the the govern data ecosystem that we created. We we brought all of that together and then we again went back to the stakeholder and said guys here we
[12:03] are all of it is governed. It is ready for you to use. Okay. Uh can you tell me why was the conversion rate still not picking up? And guess what was the answer?
[12:20] Exactly. Why does the analyst needs to get involved every time? Why does each business question goes to the analyst? And that's exactly the challenge that we are trying to solve. So we solved the
[12:32] biggest problem that we had which was like the fragmented data. We solved like the self-service enablement. We solved like bringing the best tools which are available. We brought them to together. But yet the question was lying in that
[12:45] one more click which uh takes us to the analyst and and that's exactly uh what's keeping away our analysts from focusing their energy on actually more
[12:57] challenging complex problem statements that they could solve. But where they're busy they're busy in this additional one-click challenge. And that's what we call as the one more cut problem. And
[13:09] that's exactly what we are trying to solve with um with with DVA. Show of hands, how many of you have gone through this in your in your the one cut problem? Oh, I see almost 60 70% of you have gone
[13:23] through that. So uh what I did was went back and looked at the backlogs that we had over the last one year and what we saw was more than 50%
[13:36] of what uh topics the analysts are working were were exactly can be categorized as this oneclick problem. Uh and and these are not necessarily complex uh challenge statements. These
[13:48] are more contextual ones the methodical ones which there needs to be addressing. Thank god. Thank god we have uh AI and that's what we uh relied on.
[14:01] So before we move ahead uh here I start with a uh quick demo of DA. This is not the latest version of it. This is what I could convince my uh communication team
[14:13] to get through. Let's see it starts. Yeah. So this this is the front end that we have for Diwa. It has like you know three key uh uh buckets. you have a clear help uh section in which you can reach out and see you know uh what can
[14:26] you do with that tool. You have an agent setup. This is just one of those agents. Um I see my business partners and my platform partners frowning on me saying guys why do you have just one of those?
[14:38] But this is a demo version. Uh and then what we have is a chat space. This is where we we asked we went ahead and asked the tool uh DVA can you show me for all campaigns that are currently
[14:51] there in North America tell me about their conversion rates open rates and the uh and the CVS basically so it it took its time the the first step and then it comes back with these set of
[15:03] data sets disclaimer these are uh demo numbers uh don't rely on them on your uh analysis so it gives you that uh detailed information across across the board and then comes the challenge
[15:16] statements of one the next level of clicks that we are requiring. Okay, the question is perfect. Can you give me a daily trend between a given uh time period, a given range of time period and that's where it comes back I think takes
[15:28] a couple of seconds to kind of uh structure that and gives you that feedback at uh a day level. So that's the next grain that we are t uh tackling. Post that we're looking at okay and you're of course you're able to
[15:41] you know tune tune with the graphs the features the functionalities that are uh provided here then you go to the next level of grain we're looking at okay looks like the conversion rate seems to be slightly higher should I look at like
[15:53] it's a football world cup going can I remove that impact of the football world cup on the campaigns and then see you know what's happening behind the screen there and that's where we say okay let's do more exclusions give it more more the
[16:06] the one more cuts in the system and then let's see how that's performing as well. This is where you see uh that brings you that information and kind of then you know you circle back saying okay but what all what all campaigns did you
[16:19] exclude. So you know so these are like some of the campaigns that that it looks at and then comes back saying okay these are the campaigns that I have that that that has been excluded in the ecosystem. One more important thing that we have in
[16:31] the mix is um we are extremely extremely uh conscious about the uh reliability of the data. We have index called data resilience and reliability index which we channel across the entire foundation
[16:43] data set and the application data sets that we have and uh here we kind of also try to address that. uh what you see at the back is uh us also equipping the uh user with the logic behind the SQL code
[16:57] behind the scene and also the explanability behind the scene. You the the tool has to build trust and that's what we are heavily focused on as well. Uh that's the freshness level that we
[17:09] are looking at from the tool and and uh it kind kind of calculates that for us and circles back with information on that. Um, freshness is just one of the dimensions of the data resilience and reliability index. There are five or six
[17:21] more that we address. And I don't know why the person who took this uh added the last part of it to uh analysts as we are we love to export things into CSV. So it has that feature as well. Yeah.
[17:36] So that's what we have uh from a uh Diwa demo standpoint. But let's open the hood. That's what the discussion is about. Let's open the hood and let's look at it. What's behind the scene? So when when normally somebody sees diva uh
[17:48] you look at like okay that's a bot or is that like a uh you know to the agent journey and then you're looking at what's the reasoning behind that. So the AI actually is the easiest part that we
[18:02] handle the the exper the the experience side of is the easiest part that we kind of handle. Uh let's start let's start from the bottom. What's under the hood is more critical. So what we have is the
[18:14] data foundations uh which is which is the ones which are like the CRM data the sales data that we have the consumer data and the segmentation data. So these are the basic foundations that we have kept into play and all of this information already
[18:29] exists within the organization. So if the data isn't trusted there's no AI model that can that can help that can save you. Basically that's what we believe in. Then the next is it's not
[18:41] only the um uh it's not only enough that you you have the foundation layer. It's also important that you have the semantics in place and that's the next layer. It needs to understand the context connect the data sets together
[18:54] and that's where the semantic lay layers come into play. Uh the layer defines how we measure things. uh it defines the dimensions the relationships the business definition
[19:07] across the board and it answers questions like okay uh what exactly a member is or like how do we calculate NPS so that's where that's where we are uh storing and saving these information
[19:20] then you have the layer which turns the raw data uh and by the way this is the layer that turns the raw data into the business context and then we are moving to the KPI governance layer and that's that's a layer which many a times is
[19:32] kind of overlooked Yeah. And perhaps the most uh most underrated uh when it comes to governance, but every KPI has an owner. We have a canonical uh setup and every every KPI has an owner. Every
[19:45] definition is governed. Every metric is certified before it gets exposed out and AI will simply scale uh if if that is not the case, AI will just simply scale
[19:57] the disagreement across and which is what we don't want. Yeah. So governance for us is not the bureaucracy part of it. It's something that makes the layer that we create more trustworthy. Uh then comes the knowledge layer that
[20:11] we have and now uh we are more moving into the the DVA core part of it which is the top part that you see there and that's knowledge layer that we have created. Uh the the uh this is where we capture the Adidas uh how Adidas thinks.
[20:24] Yeah, that's that's where the the brain sits and what we're looking at is the business rules, the playbooks behind the scene and we're looking at the domain expertise which are the ones which sit in that layer as well.
[20:36] This layer uh also allows DVA to answer questions beyond what we have stored in the database as well. It provides that interpretation basically that knowledge layer. Then that's the topmost layer, the most
[20:48] interesting one which we are looking at and that's uh you know this is the reasoning engine. This plans that analytical tasks for us and this is where you know Diwa evolves from simply answering the questions of what which
[21:01] means what happened at a point in time towards why it happened. Yeah. And the last part which is exposed to the end user that's the experience layer uh which you just saw uh this this this is the part everyone sees that's the
[21:13] conversational layer and I'm I'm glad that you know we are as we progress ahead with the adoption of it this getting opened more and more to the end stakeholders end users that we have. Ironically the part uh user see is only
[21:25] a small fraction of what's under the hood what's under the architecture and the most of the work happens underneath.
[21:38] Uh what I would also like to walk you through is how these layers that we have they have an interplay and how we have been evolving the versioning of the road map uh across. So one of the most common questions we get is if AI
[21:50] powerful why don't we just jump directly into prediction recommendation and you know autonomous decisions. Our answer is simple because trust must scale um uh before the autonomy scales and and
[22:03] that's the reason uh this staircase approach is what we are we are adopting. The version one is where we are trying to answer the what part of it. Where we're looking at is this is where the
[22:15] most organizations begin and and this is where you know user ask questions like what were the sales, what were the open rates, what were the conversion rates and so on and the and the objective here is that we improve um accuracy and that
[22:30] helps adoption. Before AI can explain the business, it must first prove that it understands the business thoroughly. Then we come to the second part which is
[22:42] the why and and that's where once the user trusts the number that naturally ask they they start to ask naturally more and more interesting questions and these are the why part of it. Why did
[22:54] performance change? Yeah. Why did a KPI move in a certain direction? Why did one segment act differently than a different one than another one? So this is where
[23:06] it becomes more interesting and that is where we also start building we started to build up the guide rails in the in the in the ecosystem and these guide rails are now the ones uh the system must reason across multiple data sets.
[23:20] They must correlate signals and build explanation. That's why we introduced a multi-step reasoning and an anti-h hallucination engine as well in the mix.
[23:32] Then comes the third version or the version three release that we had and this is where we are as of today. That is where we stand as of now. Uh um this what we call as the enhanced Y. Okay.
[23:44] Instead of analyzing just a single domain now we are looking at multiple domain sets of data. for example, a a drop in the email engagement and combined with a change in the consumer behavior combined with a campaign
[23:56] performance combine combined with a u um uh campaign trend or a membership trend which is there. So this is where we start that the AI starts to behave uh like a less like a reporting tool and
[24:08] more like a virtual analyst and that's where the value starts to get uh slowly realized. Now this foundation is very important and what is important is that every step
[24:21] is powered by a foundational layer. We constantly keep evolving step by step. The version four that we have is the next frontier. That's the whatif part of it. That's what we are envisioning towards in the coming year. We're
[24:34] looking at what if we increase the campaign spend like by at least 10%. What happens then? Yeah. What if I move inventory from one location to another? What happens at that point in time? So
[24:46] this is what we are trying to equip the um uh DIY engagement with. Let's let's now deep dive into how that works, how that operates. So let me show
[24:58] you actually what happens when a business user asks DVA a Y question. Not just a simple metric uh uh lookup which is there but actually a why question
[25:12] which is like why did EU open rates drop last month and which segment drove it? Yeah. So the first thing DVA does is to establish that trust and I'll move from
[25:26] left to right. So it would establish trust the user authentication through an enterprisewide identity provider is what happens and before any reasoning starts
[25:38] we do a rowle and a column level checks uh for and this means that diva in no way is going to show a data which the user is not entitled to do. This governance step is not an afterthought.
[25:52] This is a well thought through step and this is the number one step that we were extremely particular about. Then comes now where we have the where we have many AI projects failing and
[26:04] that's the stage which is called a context context layer. Most systems immediately send a question to an LLM. Yeah. So from a question to an LLM to a answer. But what we did was we did
[26:16] something opposite. We we kind of built a context layer. And this is where you know Diva tries to first put across all uh to tries to learn everything that it could learn about the question that is
[26:28] being asked. Yeah. The historical analytics pattern or uh for that matter matter of fact the session previous session summaries that we had or any past semantic cache that was currently
[26:40] registered across for that. So it tries to capture all that information that we have and then consolidates it before it even starts to think forward. In other words, before the AI think, it learns everything relevant about that question.
[26:53] And that is why we believe that the context engineering is more critical, more important than prompt engineering in this case. Then after that goes through what we are
[27:05] looking at is now only now does the planner comes into play. So that's the planner phase. That's the the third step that you see here. Now this is where we start decomposing this question into smaller analytical problems and what so
[27:19] for example what happened with the uh open rates which segments got changed or like uh which which consumer behavior was differing from a different uh from from each other and so on and each of
[27:32] these become an independent analytic task which then starts working in parallel. Now at this point what we are having is a parallel investigation happening and DVA behaves like an less
[27:45] like a chatbot and more like a team of analysts which have come together and that's the step four. Yeah. Uh where you see an independent CRM analyst and an independent uh campaign segment analyst
[27:58] working together in tandem collecting their information and bringing that together. Now both this execute and this is a critical because um analytic depth can only come with uh cannot be done with one single SQL query but it requires um
[28:12] contextual uh depth. Then comes uh the retrieval part then this answer when as in when it gets generated it needs to get certified as well before even it it reaches the the
[28:25] the end end user and the unity catalog control assets. So every join, every relationship, every KPI definition that we have comes from our governed semantic
[28:37] layer. The model never invents number like what you saw yesterday. There was a number 24 which was floating around. I think that's the next answer to all the problems that we have. 24 the the model never invents the number. It retrieves
[28:51] them from a trusted database and and shares it back with the user. Now comes a now comes my favorite part. I also don't trust this AI by the way.
[29:04] Okay. So uh before any answer reaches to the end user what we do is we pass it through an hallucin anti-h hallucination engine and this is where uh we validate
[29:17] the schema consistency. We validate the business grain. We validate the temporal logics and so on. And then this is where the AI has to prove that the answer that is giving to the user is actually it's
[29:32] trustable. Is it allowed to speak back to the user? Last point is the synthesis part of it where once all these analytics data sets are complete then diva assembles this
[29:44] narrative all of that together and then equips the end user not only just with the answer but also the logic the rational rational behind it the query behind it and that's where the underlying SQL the source system all
[29:57] those information are also floated around so that's the trust that we try to build uh and that comes only with transparency
[30:11] So uh closing on this slide, what I want to look at is the important takeaway isn't the technology behind it. The important takeaway from the slide is the sequence of how you approach that. So most AI mos AI system follows question
[30:23] to model to answer. What we did slightly differently in this case was question to context to reasoning to governance to validation and then to answer. And that's the difference between a chatbot
[30:35] and a production grade uh decision intelligence platform. Uh another question that we get often asked and uh this is where you know uh thanks to the folks from enthropic. So
[30:49] one of the question that we asked was why did Edidas chose claude as the intelligence layer behind behind this diva engine. The answer is actually very simple. We didn't start with a model. We
[31:01] started with a problem. Okay. And our goal wasn't to create a chatbot. Our goal was to create a trusted virtual analyst. So we treated this entire setup like an engineering problem statement.
[31:14] We we created benchmarks with close to around 131 uh questions which typically a business user would ask. Why did the sales change? What segment drove the result? What campaign was the one which was trending? uh what was the uh what
[31:28] why and this is what explains the movement in this set of KPIs basically. So these were the 131 set of questions that we pre-aligned and made sure that these are the ones which we pass through
[31:40] and test the model. Uh what we needed from the model was that uh and what we learned very quickly was that the intelligence alone wasn't enough. We needed four things. First was
[31:53] reliability. The model had to be consistently uh returning back structured outputs which then would feed into the further process uh further steps in the process.
[32:05] And this is very important because if the structure of the response fails at at any given stage the entire workflow crashes. So that reliability was of the highest importance. The second thing
[32:17] that we focused was the context. Now, DVA works on metrics and KPI descriptions and business rules and the conversational history. The model needs to understand this simultaneously and
[32:29] that's what we heavily focused on. Uh we also looked at we also looked at uh as we as we went ahead we looked at um um understanding of the context. Uh
[32:44] yeah. And we looked at the third which was the instruction that it followed. Yeah. Did it follow what we needed the model to be disciplined about and following the governance rule that we had put across for it and avoided any
[32:56] assumptions. And last but not the least which you also highlighted the safety part of it that was one of the most most critical part where you know was uh the at an enterprise AI level was the model
[33:08] safe. So these were uh parameters which we kind of looked at and and the results that you saw are clearly reflected here. the the correctness of the classification was about 97% and and the and the routing was quite reliable up uh
[33:22] close to 60% plus and all of this by the way all of this was coming at under 3 seconds of of latency. So uh that was uh that was a uh governed and convers
[33:36] conversational grade interaction that we had and that's what kind of led to how we interact with DA. So we didn't choose clot because it was the smartest model. We chose it because it was the most
[33:48] reliable teammate that we had in the mix. Uh last slide the key takeaways that we had uh uh from this setup uh and this is what I want to emphasize on the first
[34:01] thing is context engineering. So build a context pipeline that context is the context is the currency of the future. We focus on context. Semantic layer is a prerequisite. Do not
[34:14] do not compromise on the quality of the semantic layer. That effort that you put in there is worth investing. Start with that. Uh explanability builds trust. The adoption the pace at which we were
[34:27] adopting uh our markets uh are trusting the components the semantic layers that we are bringing in. And that only comes with if you make the data explainable. focus on that uh benchmark before we
[34:41] ship and the benchmarking exercise that we did was extremely extensive. There's no compromise before you move the PRs. The benchmarking has to be extremely strict. Uh we also learned that uh the
[34:53] uh the authorization part if you look at it retrospectively it's going to be huge pain. uh with all the norms especially in Europe uh it is important to start with O and make sure that uh uh the
[35:08] models that you build are able to access the data uh which they are allowed to uh last part is the observability part of it the the token maxing challenge that uh that the entire industry is facing I
[35:21] I believe all of us are on the same board that observability is what we built in uh right from the beginning uh creating a tech depth out of it becomes a huge pain later on. So these are some
[35:34] of the learnings but what excites me the most is after listening to what uh datab bricks has to announce on the ontology and the uh genie once uh the solutions that are coming up. I I think this is
[35:46] going to help us accelerate the journey for the version 4 that I was talking about in the earlier slide and I'm really looking forward to uh uh really looking forward to that part. Um
[35:59] when we started the journey uh we thought we are building an AI tool but at the end what we learned was we are going to change the way in which the folks in Adidas are going to interact with data. Now techn technology would
[36:13] keep evolving. Uh but one thing that doesn't change is the foundation that we have created at the back and that foundation is what's going to define or distinguish between organizations. an organization which distingu between
[36:27] organization which has tons of agents and an organization which has a governed data ecosystem together. Thanks.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.