Data Intelligence for Marketing Breakout: Agentic Systems for Bayesian MMM and Consumer Testing

Summary

• GenAI copilot for marketing science: An AI interface democratizes complex Bayesian probabilistic models, allowing marketers to run sophisticated analyses without statistical expertise.
• Bayesian MMM automation: Agentic systems automatically scale Media Mix Modeling analyses, accelerating insight generation for accurate marketing spend decisions.
• Consumer testing at scale: GenAI enables rapid experimental design and consumer testing execution, supporting data-driven marketing decisions across campaigns.

Data Intelligence for Marketing Breakout: Agentic Systems for Bayesian MMM and Consumer Testing

Talk by Dan Morris, Marketing Solutions GTM, Databricks ; Luca Fiaschi, Partner, PyMC Labs
This talk dives into leveraging GenAI to scale sophisticated decision intelligence. Learn how an AI copilot interface simplifies running complex Bayesian probabilistic models, accelerating insight generation, and accurate decision making at the enterprise level. We talk through techniques for deploying AI agents at scale to simulate market dynamics or product feature impacts, providing robust, data-driven foresight for high-stakes innovation and strategy directly within your Databricks environment. For marketing teams, this approach will help you leverage autonomous AI agents to dynamically manage media channel allocation while simulating real-world consumer behavior through synthetic testing environments.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does GenAI simplify Bayesian probabilistic models?

AI copilots provide natural language interfaces to complex statistical models, translating business questions into model configurations—making sophisticated Bayesian analysis accessible to non-statisticians.

What is Media Mix Modeling and why does it matter?

MMM analyzes how different marketing channels (TV, digital, social) contribute to business outcomes, enabling data-driven budget allocation and ROI optimization—critical for enterprise marketing.

How do agentic systems accelerate marketing decisions?

Agentic systems automate experiment design, data preparation, model execution, and result interpretation—compressing months of analysis into hours while maintaining statistical rigor.

Can AI agents design and run consumer tests autonomously?

Yes, agentic systems can autonomously design experiments, formulate hypotheses, configure test parameters, analyze results, and generate insights—enabling rapid consumer feedback loops.

Full transcript

Good afternoon everyone. We are at the beginning of a revolution on how marketing science operates. We're moving from manual processes to intelligent automation. Today we will explore our Aentic AI and particularly for baian media mix modeling and consumer testing. It's a reality built on top of the datab bricks platform. We are all familiar with all the advances that AI has made in marketing in automating processes.
But in the next 40 minutes, we will unpack how AI is moving from automation of processes to be a cognitive partner for strategic decision making. Whether you are a technologist that's uh looking for ideas on implementation or you are a business leader wanting to make the next decisions with impact, you will find actionable ideas. Today we will start by looking at the current state of AI marketing. Then we will dive into two solutions that we have developed which is an agents to build by Asian MM and a workflow of agents to build consumer testing and support consumer testing and then we will weave them together at the end.
But before we get into these topics, let me uh introduce myself briefly and why I'm so excited today about talking about this topic. All right. I'm a partner at PMC Labs. We are a probabilistic AI consultancy. Um, we are the author of PMC, which is an open-source library used um by many different businesses, Fortune 500 and smaller companies to optimize their marketing analytics processes. My passion in the field started many years ago over a decade ago as a tech executive at consumer brands like Stitchfix or Lofresh or companies like Rocket Internet SC where I was responsible to build data teams and as part of my responsibility I had to support marketing teams in allocating hundreds of millions of dollars in marketing budget.
So from that perspective firsthand I really lived the problems that you have when you have to support decision making with such a high stakes. So let's see what's actually is preventing us from taking full advantage of data or actually before we get into that let's review the state of uh AI marketing. So today, according to the American Marketing Society, 71% of marketeers already uses AI tools weekly to generate new content.
There are reports from major personalization platform that claims that AI can boost AOV 40% and 98% of AI using sales team report strong efficiency improvement. So AI is already automating a lot of the workflows that are related to marketing and you've seen throughout the conference this week so many example of that. However, when it comes to what are we actually marketing and how are we marketing, we are still lagging behind.
In fact, according to a Gartner survey from April, 61% of CMOS report that they um lack insight from consumer testing on their plans and 70% of CMOS feels like they face challenges improving the ROI of their campaigns. So the solutions that we're going to discuss today is try to address exactly those problems. And the key idea is that the role of AI that's already automating a lot of marketing processes also needs to change to start revolutionizing how we think and support decision- making and AI is the perfect tool for that and we're going to look into that in a moment before we review after we review the challenges.
So these are problems that I faced firsthand by leading data team at uh large organizations and what you typically confront yourself with is an expertise gap. Building sophisticated models really take um advanced skilled data scientists and there is a talent shortage in the market and these people is really hard to retain. So uh nearly 76% of data and AI professionals report a talent shortage today. On the other hand, there is an iteration lack.
Building sophisticated models takes time. You need to collect source the data, build the appropriate model by understanding what's going on in the underlying business process. You need to um build a report afterwards that delivers value to the stakeholders. that can take weeks and even months sometimes. In fact, stuff like MM LTV models traditionally in enterprises are really only run once or twice a year. And most importantly and the most difficult thing to address, it's really a translation barrier between business stakeholders that speak a specific language and data scientists that speak the language of math and statistics.
And often this translation barrier makes so you cannot really understand each others and you cannot action upon the insight that you get from your data. So this really hits you as a lead of data in some of this company and you spend most of your time solving actually these problems and what it gets to is that most of the decision or like some important decisions gets made on gut feeling rather than on actually proper modeling.
And as a head of data, you know that you can do much better than that. Now, arguably, and this is the key thesis of the talk, AI is really the solution for that. Why? Well, AI really democratize access to expertise. These LLMs have read virtually any paper of open access on the internet. So, it's like having your PhD scientist in your pocket. You can always talk to. In addition to that, they're always on. So they can reduce the iteration lag just by brute forcing it by the fact that they're always available to answer any questions and they can answer sophisticated what if questions and most importantly here they are extremely good at communications. they can really tailor the message or the insight in a way that's um understandable by the stakeholders or even by a five years old.
So we're going to look at two key examples that we are going to very soon open source. The first one answer the question of what should we market? It's uh we call it the innovation lab and it's a solutions that rapidly validate new concepts for new products and assess how um perceived they are from a synthetic panels of consumers that representative of the US population. The second solution is a collaborative workflow to build and develop MMSM that makes building media mix models an exercise of conversation.
So it's important to underline before we get into this that these the idea of these agents is really to act as cognitive partners. They support decision making for uh data strategists and data scientist accelerating the time to insights. So let's get into how we simulate human behavior. And what I'm going to do is going to first give us a few examples just to ground ourself of what um in product innovation means for consumer products.
Then we're going to look into the different components of our workflow and I'm going to wrap up with some key benefit of doing things like this. So these are you maybe recognize them. These are infamous examples of product flops. um Dero, New Coke, Segway, Fire, iPhone. Um they were launched into the market putting millions of dollars into their development, but didn't really work out. Take for example the new Coke. It had actually a decent taste, arguably unfortunately didn't really meet a consumer segment that would resonate with.
In fact, product development is full of these pitfalls and arguably um companies spend or companies spend millions and millions of dollars every year in before developing this product in testing the concept with human panels. Now this process is slow. It goes from ideations to concept development to prototyping testing the feedback with the human panels which often is relatively small actually to market validations that is when I actually launch the product our ideas is to automate the entire process and accelerate the entire process using an end-to-end platform powered by AI agents.
So how do we do that? So there are two key components to this workflow. First, it's a an agentic workflow, a system of agents that generates new product ideas based on a brief. Um here you have uh four of these agents. You have a supervisor, you have a marketing expert, you have a designer, you have a product managers which are entities which are empowered with tools like web search and image creations to come up with refined product ideas and what you come up is something like this.
You get uh some nice images of the product which are created by the designer. You get a description of the key uh characteristics of the products. what what is the value proposition of this product? You get even uh a price point for some of these products. Over at the bottom, you get also a summary of the reasoning process from these multi- aent systems that came up with the final solutions. Why would you use a multi- aent system?
So, some of you may think, can I do this in CHBD by myself? Well, you've seen already many different talks around this during this conference, but the key idea here is that um you by providing different agents that provide feedback to each others, you can actually refine and improve the quality of your output by specializing each agent in one single task. In fact, if you had to do this yourself by using chbd, you would probably spend a very large amount of hours just by trying to refine the output of the system.
And this allows you to come up with candidates products that are actually relatively high quality already in an automated fashion. Now, the second key component is a synthetic consumer panel that's representative of the US populations. And here you see um a virtual which is 35 years old lives in the US um is a woman who's a manager at the company. This is wrapped into a multivision model or a multimodel model that can view the images of the products.
We put them in front of them and can understand the key characteristics of these products. Now because we sample these characteristics of synthetic consumers in a way that's representative of the real population of the users in the US you can start getting something that's looks something like this which is um asking questions about product liability statistic of whether they would like the product and you can get even uh elasticity curves that tells you um whether they would purchase is the product and a specific price point and you can even study what the market size of that potential product would be.
If you think about that, that's extremely powerful for CPG companies because you have a data point that tells you when you're getting into the market, you know what you're going to get at the certain price point in terms of like overall market potential. Now, the question you should ask yourself is how reliable are these machines? how how well they can really understand and replicate human behavior on this specific problem.
And to actually define that, we need to start defining what do we mean with replicating human behavior? So let's make an example that you're asking this uh machines how likely would you be to purchase a product? The typical answer is in a scale from one to five which called the liyker scale. Now what I want is that if I put in front of a synthetic panel a 100 different products the ranking that comes out of it which products they think are good and which products they think they are bad comes out as a human panel would have uh scored them and you can measure that by uh correlation Pearson correlations and on the x-axis you see um the um ratings the average ratings for hundreds of products from real users and on the y-axis you see the ratings from synthetic users and the high correlations between each one of them tells you that this machine scores the products as human would score them.
Now notice that humans are inconsistent with each other. So if I take two populations of people exactly like each other with the same characteristics in term of geographics and demographics, they actually would correlate only up to 0. 8. So a correlation of 0. 7 is actually a pretty strong reproduction of what the human panel would have done. Another way which is a stricter way to understand this is if I look at the distribution of rating for a specific product.
What I want is that um the distribution of real humans responses which is in white here um is actually replicated by the yellow one which is the distributions of synthetic responses and we can use a metric which is called the cologor of smeov distance which is uh over there to assess how close they are to each other where one is perfect similarity and zero is no similarity and 0. 8 is typically considered good threshold.
So how do this actually do on a real u actually assessment. So we made the major study with Colgate Palmolive which is one of our clients on 57 products. Some of them are AI generated some of them are actually real. These included 9,300 9,300 unique users. And what we found is that there was a correlation of 0. 7 which is 90% of what the human panel would have achieved them with a mean similarity on the distribution which is 0.

8. But even more interesting than that I think it's uh important to take a look at the qualitative answers that this panel are actually giving you back. So let's dive into this. Here you have um answers of synthetic consumers and human consumers on the questions um what did you like about the product typical answer could be as simple as it's good um or how likely would you be to purchase the product and here you see some um very long answer which is given on the right which is I might consider purchasing it because blah blah blah blah blah which Which one do you think were AI generated and which one were human generated?

Well, we find that on average AI responses are twice as long as human responses. Why? Because during preference fine-tuning, these LLM are actually trained to give very clarificatory answers, very clear answers that people can understand. And that's the reason why they tend to be a little bit more lengthy. Now, while this is an actual difference between human responses and uh AI responses, it can be actually advantageous in some cases.
Why? Because you get context out of the rating that you wouldn't get normally without teasing it out. And uh this context can be helpful to help you refine the product. In fact, we did a funny experiment. We gave um absurd products to these LLMs like here you see an example a dental micro microbiome fortifying elixir and we asked humans and LLMs to rate this product and what we saw is that um humans tend to rate this product ier than the corresponding LLMs and it's it's a nonbias when you do surveys that this can happen you see that the distribution of human responses are actually skewed to towards four and five while the distribution of synthetic responses is skewed towards one and two and even more surprisingly than that lamps they would tell you exactly why so this product look gimmicky looks fake I don't trust the claim that it can do the magic we claim it can do another interesting thing for application is price sensitivity in fact these LLMs can reproduce prices elasticity characteristics of real responses.
Here for example you see uh a segmentation by income level on humans and um synthetic responses. Notice that the scale is different for the two plot. So they correlate but they're not exactly the same. And what you can see is that the pattern is however the same. People who has a higher income, they tend to have a higher purchase probability for the products because they care less about the price. At the same time, on the right side, you can see that um um like a premium product, they have typically a lower purchase probability than value product.
They're more mass market. So you can see the same on the subsegment, the same characteristics of human and AI responses. So what are the benefit of doing things like this in the entire innovation products when you are launching consumers um consumer items? Well, there are four key uh advantages. One is a significant reduction in cost. Uh product innovation cost tens of millions of dollars to these big CPG companies. You can do that at an unprecedented speed because organizing humor panels take you a long time.
And what you can actually envision to do with a system like this is to go from ideations to validations in the synthetic market to refinement of the product multiple times until you get converged better results. You get the reliable validated feedback according to the study we have done. Now what I should say is that there are specific prompting technique we have developed under the hoods to make this possible. And the last biggest advantage is of course that you risk innovations and you don't have the issues of a product like the new coke by using a a system like this.
So now we have looked at about we looked at um what we market let's look at how we market and let's switch gears and let's talk about how agents can also help accelerating media mix modeling. And I'm going to um make a comparison with a product that everybody probably knows because you heard about this multiple time at the conference which is Genie. Genie is an agent that uh allows you to talk to your data. So you can have a conversations and it's a translation between human language and SQL querium.
Now the evolution through this concept and what you actually want to do to move from descriptive analytics to predictive analytics is to start talking to your models. You want to have a deep conversations that allows you to ask the questions of not what has happened but what would happen if I had done XYZ which is a causal type of questions. Now a big question in marketing which is that type of questions is media mix modeling and it's basically the question of answer and it's a question of where should we invest our marketing dollars.
It's a big problem. Uh Garner reports again that 64% of all marketeteers um in the US were working on uh media mix models in some form or shape during 2024. The idea here is that you have a mathematical model that takes into account how much money you put on your media channels. Could be TV, search, takes into account exogenous factor, external factors like sales, trends, seasonality to give you an uh predictions of how much of many sales or how much revenue or how many conversions you're going to generate.
Why would you want to do that is to get insights like this. These are called um saturation curves. On the x-axis you get your u media allocations. On the yaxis you get the response the incremental response of the channels. This allows you to understand whether for example a channel is saturated. you can't spend more because you're going to get diminishing returns or whether channels are still room to push more media to find an optimized budget allocations and this can really boost up your ROI.
So the typical situation that happens today when we look at about these uh models is that um your CMO comes to your team and it's the end of the quarter and wants to know uh how to allocate the money that you're spending in marketing across all your media channels to hit a specific ITA target. Now the answer from your data science teams tends to be often great questions. we're going to get back to you. Give us a few weeks.
Why? Because maybe they didn't they hadn't collected the data exactly in the format that the is used for this analysis. Maybe a new media channel was introduced so that they have to rebuild the model. Maybe you are in a situation in which um the questions was formulated in a slightly different way which requires a model uh reformulations. For example, here the question is formulated on what is the AIA that what is the media location that actually gets to a certain AIA rather than what media location maximize your revenue which is a slightly different way of asking the question.
So that leads to an iteration lag that often can take quite a bit of time. However, our CMO has only until Tuesday to answer this question because she has to talk to the board. So what's the solution? The solution is to have a co-pilot or to empower your team with a co-pilot that allows them to build these models very effectively. And the idea here is that elim eliminates um all the cycles and iterations for a quicker and more iterative analysis.
And this essentially an interface that looks like this where you can get the code and you can get text you can write to an LLM to develop the model together. And it's composed of three steps. Configure, explore, and understand. We're going to go through that uh in a minute. But before we go into that, I want to briefly mention how we built these systems on top of the data bricks platform. So there are two buckets of key solutions here as we used them.
One is the expertise in AI from our team at PMC and the open-source libraries that we have developed in the past 10 15 years. This allowed to build media mix models uh in a very effective way using uh things like accelerated GPUs processing um different specifications for the media mix models and so on and so forth. We have uh used these libraries on top of the datab bricks platform. The datab bricks platform provides essentially two things or three things.
One is um a unifi unified marketing lakehouse where the data is already consolidated almost ready for modeling. It gives you access to flexible compute and this flexible compute is um extremely important because some of these models we have clients that have models that they have hundreds thousands of parameters takes days to train on a GPU cluster and uh you have um ML flow and mosaic both providing capabilities for managing the complexity of these models both in terms of creating the agents and in terms of versioning what the agent does and uh how you build that model and the result of that model so that it's then production ready.
So but we will see you will see a conversational interface which is built on data bricks apps when the user interact with actually a multi- aent system. There are three types of agents. one that interprets code and interprets the user intent and langu and uh language and other agents that can understand plots and then you have an agents that's good at building models and kicking off runs for the models. The data resides in the Unity catalog.
We use data bricks workflows to do the training and of course the key component of that ties it everything together is MLflow. ML flow is where we save the model di diagnostics that are essential to understand whether we had the correct fit of the model and whether the model is reliable and everything can be investigated and traced back to MLflow so that you can really make your model production ready afterwards and you can validate the results.
So let's uh have a look at that. So here you see an example. The user decides to upload the CSV in the Unity catalog. As soon as you kick off that you get some like high level validation which is automated like how many columns um um whether there is missing values in the data and so on and so forth. You can configure the catalog to know where the your data is going to flow. You can you can configure your MLflow instance whether you want to use data bricks or you want to use an external one.
Then it's time to do what this typical data science would do which is exploratory data analysis. And similar to Genie here you can ask the agents to build plots. Um and what it's going to do is going to like write code uh to actually execute these plots. And on top of that, it's going to suggest you analysis that are relevant for media mix modeling because he has a context of what media mix modeling a good media mix modeling means and how you do it.
And the other things it's going to do is going to give you an interpretations of what you see in the plots. For example, here you see that the um the agents um has identified that um there is a trend beginning around March in the smoothed average of sales and there is a drop in sales around the period of March and he has alerted the users that something like this um can happen and can influence the results of the model.
Now you could do that by yourself, you could do that with Genie. So the model here uh as a context of what good media mix modeling is and really act as a cognitive partner. The idea is that maybe you even you know you did know how to do this type of analysis. You hadn't thought about that while you were doing it and so you have a sparring partner that gives you feedback. Now it's time to kick off the model run and um you can see that you can track the status of the model fit in the back end.
Uh this creates um data bricks workflow that uh it's executed where you can see all the code uh of the model runs automatically. We premp compute several quantities that are meant to um like bodel diagnostics to tell us whether the fit of the model actually was correct and accurate. And everything is then saved into MLflow and you return you get returned the link where you can actually explore all these model results and u really understand these plots if you want to drill deep dripper or you can keep exploring your data while the model fit runs in the background.
So you can decide what you want to do. Finally, when the model is fit, it's time to ask questions to your models. And because the agent has an understanding of what the parameters of the model mean, you can ask questions like the saturation curves that we have seen at the beginning. You can ask the agents to plot them and to interpret the results of your saturation curves or other reshaping of the parameters like the adstock curves which is another uh byproduct of these type of models that tells you um how much is the lag in your media effectiveness over time.
And most importantly you can ask causal type of questions. For example you can test you can ask to test scenarios. What if my competition would increase of X percent? What if I allocate the medias in a different ways? What is my optimized allocation according to a certain budget limit for example? So these are all questions that the agent knows how to answer and reruns the models and provides you the results. So what are we have seen?
We've tested this uh to a group of our clients. It's around 10 15 clients that we have um provided this prototype uh to since then we have seen a dramatic acceleration in uh time to insights. Um we some clients report 10 to 50% reduction in effort. Um they reported that this has really democratized the way they built these models because now u more junior data scientists are empowered to actually carry over the entire process by themsel with minimal supervisions from a more senior data scientist.
Um this often gives richer and nuanced understanding to our clients because they can um get insights that according to anecdotal reports they tell us they wouldn't have gotten otherwise because this has agents has the knowledge that we have infused into it uh of how to build effective mediumix models. It has certainly freed up expert time so that their more senior part of the team can actually dedicate themselves to more strategic topics.
And finally, of course, because of these accelerations, you get a higher ROI. In fact, our clients that implement media mix model report 10 to 20% increased in u ROI effectiveness. That's significant considering that they have uh sometimes hundred of millions of dollars in budget invested in marketing. Now we talk about these two applications um the innovation lab and the agents for media mix modeling. What's next and how they are connected to each other?
Well the first application the innovation lab answer the questions of what should we market with? what should we bring to market and how should we present it to our consumers and it gives you a workflow for doing something like that. The second one the media mix model tells you what how should we allocate the media and how should we market behind uh when we have launched already the product. The missing link here is that you could actually imagine a situation.
And we are working actively on these in which um you want to enter new media channels and you want to have an understanding of what consumers you find in that channel and whether they like specific ads and specific messages resonate with them. So after you launch the product, you can use the synthetic consumer panels to get that type of feedback and close the loop between the two systems. In fact, if you have a certain average feedback from the distribution of the population of synthetic consumers, you can think about translating that in a basian prior that then you can attach uh to your media mix model.
So to wrap up, so what what is the um takeaways that I would like you uh to have today? So first of all that um this agentic cognitive augmentation is really now we are moving beyond simple task automation in marketing and this was just you know a few months ago that really the whole thing started to really augmenting how humans reason and think with these AI systems. Um second that everything is built very conveniently on top of the data bricks platforms and we're going to open source accelerators that you can use to actually use these technologies on top of data bricks.
Um and datab bricks provide access to consolidated data again tools for versioning models and also tools for flexible compute. And then the last one is that really the future is collaborative and strategic. We are u moving towards a reality uh where teams of AI agents and humans really work together to tackle biggest challenges in the enterprise both at the level of processes but also at the level of uh strategic decision making.
So you may ask yourself how do I start taking advantage of some of these technologies? Well um my recommendation to you is to really start thinking whether you are um an executive or you are a data scientist in want some of enterprise and some teams. What big decisions does the business needs to make and whether there is the space for using agents and AI to get to an a solution faster and to make a more reliable decisions with informed data and often CMOS comes to us and they don't know where to start.
Um they think may think that their team doesn't have the skill set to actually build some of the solutions that we have seen today. So that's where we can help and you know we work with many enterprises in providing an acceleration of the AI programs. So if you are interested in talking more about this topic uh please feel free to reach to me or to uh any of my colleagues which are around the conference and I love to understand what your challenges are.
Thank you very much.

Learn more about the Databricks Data and AI platform.