Sponsored by: Pogo | You're Overthinking It

Summary

Pogo's AI research platform analyzes purchase data from 3 million Americans and lets brands conduct AI-moderated video interviews with verified buyers, delivering consumer insights in hours instead of weeks.
After building a complex multi-agent system that proved fragile and hard to iterate on, Pogo replaced it with a single coding agent, which demonstrated dramatically better reliability and performance in production.
The key lesson is to exhaust the capacity of a single agent before adding sub-agents, using skills packets to load context on demand and avoid system-prompt bloat.

Chapters

FAQs

What does Pogo's AI research platform do?

Pogo operates a mobile app through which 3 million Americans share their purchase data from linked financial accounts, Amazon, Walmart, and physical receipts. The AI research platform built on top of that data lets brands analyze consumer spending patterns and conduct AI-moderated video interviews with verified buyers, with results available in hours instead of weeks.

Why did Pogo abandon their multi-agent architecture?

Pogo's initial system used a supervisor multi-agent architecture with specialized sub-agents, but it proved fragile in production because errors from one agent cascaded through the system and the whole setup was difficult to debug and iterate on. When they simplified to a single coding agent with full context, reliability improved significantly.

What are skills packets and why does Pogo use them?

Skills packets are structured context bundles that give the agent specific capabilities on demand, without loading everything into the system prompt from the start. This approach avoids system-prompt bloat while still providing the agent with the expertise it needs for a given task, keeping the single-agent design lean and maintainable.

What is the main lesson Pogo learned about agent architecture design?

The core takeaway from this video is to squeeze the single agent as hard as possible before introducing sub-agents, because complexity compounds failure modes and makes systems harder to maintain. Frontier AI labs' own designs also point toward simpler architectures, reinforcing that the instinct to add more agents often creates problems rather than solving them.

Full transcript

[00:07] Good afternoon everyone. How are we doing? Good. Awesome. Well, thanks for coming out to our talk. Uh my name is Dom uh co-founder CEO at Pogo and this is Luke, our head of data.
[00:20] >> Hey there. And um our talk is titled you're overthinking it. What analyzing consumer sentiment and billions of dollars in consumer spend taught us about building AI agents. So the way
[00:32] we're going to structure the next 30 40 minutes uh first is we're going to tell you a little bit more about Pogo, the data that we collect and the different agent workflows we've built on top of it to show you guys uh a little more about
[00:45] you know um some of some of the agents that we built ourselves. and then uh Luke will talk about some of the lessons learned that we had. So, hope to keep this fun, informative, and um we'll do a little Q&A at the end. Sound good?
[00:59] All right. So, little background about Pogo and what we do. Uh we founded the company 6 and a half years ago. The big goal at Pogo is to help everyday people in America earn money from their own data. And we help three million people
[01:12] in the country do this. And how it works is we operate a mobile app on the consumer side uh where millions of people download it, link data about what they buy mainly. Um and then we have a
[01:26] AI research platform which is where we're going to talk about some of our agents. Uh where brands are able to connect with people that bought their products, talk to them, understand them, analyze those different purchasing patterns.
[01:40] And on our AI research platform itself, we allow brands to run video moderate uh AI moderated video conversations with these consumers, quant surveys, different insights, um all powered by
[01:52] our AI research agent. Um little context about the data assets and the data that we're working with. Uh when people download the Pogo app, uh
[02:04] they have the ability to choose to link multiple types of data. And it really starts with what you buy. So kind of like how you would use a personal finance app. You could link your financial institution. So you link your Chase account. We can pull in multiple
[02:17] years of transactions historically as well as the live transactions going on on a go forward basis. Um people can also link things like their Amazon, Walmart, Target accounts where we can pull in the items that were purchased.
[02:29] So the actual products and SKs. Um email receipts where we get you know inapp purchases etc. And people even take photos of their their physical receipts to upload them. So you know after they
[02:41] go to the grocery store you upload that we can understand what was purchased in that basket itself. Um folks can also link information about the mobile apps they use. So you can understand hey why did people switch from chat GPT to
[02:53] claude for example right and understand you know those sort of behaviors location etc. All really important is all this data is 100% user opted in uh through active consent. And uh you know
[03:05] what that means is we actually capture you know close to half a trillion dollars in transaction volume that we see uh which is kind of the fundamental data asset that we built agents on top of.
[03:17] And a little more context you know not only do we pay people transparently for this data but we also based on whatever data you link help you find ways to save a ton of money. So it could be you know hey maybe based on the banking data we see that your bank is charging you a ton
[03:30] of fees. um uh we'll actually draft the note and help you get a refund. Or maybe you bought a product that's on class action, Cliff Bars, for example, if you bought one of those, you know, we'll help you join the settlement, get dollars back. Um or maybe uh you know, a
[03:44] brand wants to have your opinion on on the new chocolate bar they launched and you can get paid for for doing a 15-minute interview. And all in all, this has allowed us to scale to yeah, millions of users in the United States. And you know, we've been ranked the
[03:56] number one loyalty app in the US the last two years in a row, which has allowed us to grow this data asset over the half decade. Um, to one that captures one in 150 shopping trips in the United States in this sort of like
[04:09] full path to purchase view where we can understand, hey, someone walked in to Costco based on location data, what they bought, and then, you know, what they did afterwards as well. Um,
[04:21] and we're venture back. We've raised $32 million to just scale this thing further and further. Um, so hopefully that's a little bit of helpful context on, you know, the the data asset. Now, we're going to show you some of the applications we built on top of it. So,
[04:35] uh, at the beginning of the year, we launched the world's first AI video interviewer powered by purchase verified respondents. Um so the first you know agent that we decided to create was a uh research agent that you know actually
[04:47] has thousands of conversations with you know thousands of consumers at the same time. Um and uh you know we saw AI interviewing you know customer interviews as a real
[04:59] 10x solution to our customers which are these brands and research teams that want to understand more about you know what their customers think about the new products that they launched. So uh we think applications of AI into you know
[05:12] customer interviews is real 10x solution because what would have taken you know three months to talk to you know 100 people in a focus group and schedule hundreds of conversations you can now do simultaneously uh you know with a AI interviewer. Um,
[05:27] but the next challenge is making sure you're talking to the right people. And you know, I don't know how many folks of you do uh consumer research out there, but you know, in the in the research world, the market research world, fraud has always been a pretty big problem.
[05:39] Uh, you have, you know, bots taking surveys or people lying to qualify to uh get into interviews. So, um, and you know, even if it's not a bot and it's actual real person, let's say you're one
[05:52] of our customers like Nestle or Hershey's and you're trying to, uh, you know, find someone that bought the new product that you just launched. Um, in the traditional approach, you'd have to screen through thousands of people to
[06:04] find one buyer, which makes it pretty untenable to actually find uh, people to talk about uh, your product. So that's where our approach with Pogo is we work backwards from all this data um to find
[06:16] the exact people that bought that specific product and um deploy our agents to you know help you understand uh more about that consumer. So we're going to illustrate you know the actual workflows with an example here. Um who
[06:31] uh remembers this over here the CEO Tik Tok. Okay. Um, so for context, uh, if if you haven't seen the viral meme here, uh, you know, McDonald's launched the
[06:43] Big Arch Burger in March, and the CEO did a little taste test on TikTok, but got flamed for taking the smallest bite ever out of the burger and calling it the product.
[07:01] >> That is so good. M, that will be enough of the product for me. Thank you. No, seriously, I'm stuffed. I couldn't eat another slight touch of the lips. >> So, it went pretty viral. Uh, and the internet had a field day. So, I'm going
[07:13] to put you in, you know, one of our customer shoes. If, for example, you're McDonald's and you're the marketing team or the insights team or the data science team, you might want to understand, you know, one, uh, who's buying this new burger that we bought. Uh, two, do
[07:25] people like it? And three, is this viral campaign helping or hurting us, right? Um, and how you do that in Pogo's AI research platform is you first use the cohort builder agent that we built to uh
[07:40] actually go find that audience. So in plain English, you just type out people who have bought the big arch burger. Uh we actually go search through those, you know, hundreds of billions of dollars in transactions to find the exact uh users
[07:53] here. And you'll see here, you know, we found 11,000 people that have actually bought this exact big arch meal. You'll see in the sample data it shows you and and what's cool is you can actually have a conversation with it. So if you want to say hey I want to find people that
[08:05] bought twice but never bought again or people that bought the big arch burger in Chicago use natural language and the agent itself will actually build the SQL query uh and you know build that cohort for
[08:17] you. Now after that what's cool is out of the box um because Pogo has all this other data about what folks are buying um in that group uh we already you know know a bunch of things without asking a single
[08:29] question. So we have a module that gives you all sorts of insights about who these people are compared to the general population these big arch buyers etc etc.
[08:46] And you know what's cool over here is um you see the viral video launch date in that red bar and then the actual launch of the product. You know by the time the burger actually launched um Pogo already had thousands of people that you could talk to that same day um to to help you
[08:59] get rapid feedback. Now once you're done that uh all you have to do as a researcher is you know uh upload an interview guide or just tell our agent you know your key
[09:11] business questions and we'll actually go and create a survey uh guide and a you know interview questionnaire and once you're happy with that you hit send and what happens is uh yeah the AI
[09:24] interviewer will actually go and have thousands of these conversations uh with actual buyers of the big arch burger in the app itself and then users, consumers in America will get paid uh for participating in these interviews. So
[09:38] over here, you know, our research agent or our interviewer will actually uh go back and forth and dig deeper uh depending on what the consumer says. So uh yeah, we're just going to show off
[09:50] a couple couple, you know, uh reactions, too. So here's one reaction from a consumer what they thought about the Tik Tok. >> Well, I mean, initially, I mean, I thought it was funny. Um, I thought
[10:02] mainly because I figured people in that tax bracket most likely don't eat McDonald's often. But um, it also seemed like a CEO was
[10:17] far removed from his company and his product. >> So, you know, this gentleman and many others felt that um, it was negative because the CEO is far removed from his product. We got a
[10:30] contrasting view here. Uh, I have seen the viral Tik Tok video from the McDonald's CEO about the big arch. Um, I think it was, you know, the
[10:44] guy is a CEO at a big company, you know. I mean, he's not obviously uh one of these foodie guys that go out and say, you know,
[10:56] guys. And then uh what's cool is yeah, we found folks that didn't see the Tik Tok but were influenced by other channels as well. >> I watched this person on Tik Tok named
[11:08] Tama. She was eating it and I was like, you know what? I'mma try it. Had I had seen that Tik Tok video before Tama's, I
[11:21] don't think I would have been sold cuz it just seemed like he didn't really enjoy it. And that bite was not as big. It was like, okay, I'm going to take a
[11:33] big bite and it wasn't. It was like the small bite. So um we have all these transcripts and what we also built was a study chat agent that basically takes all these um
[11:46] raw transcripts and allows you to query that and ask any questions there. So you know maybe one question is what percentage of the respondents actually even saw the Tik Tok itself and you'll be able to actually get those results um
[12:00] through the you know study chat agent that we built over here too. And uh I guess >> of the fresh onions and I just I love an onion on my burger in general. So the
[12:13] onions made it 100 times better than the Big Back in my opinion. >> Yeah. And this was an interesting insight too where um you know McDonald's marketed the uh sauce as a key ingredient, but actually the onions were
[12:25] really the most loved piece uh based on the actual consumer research that our agent was able to conduct.
[12:37] Cool. And then what's also interesting is because we have all this quantitative data already, you can also use the research agent to um do different sort of queries. So maybe one question you might have is, you know, what percentage of McDonald's buyers actually had the
[12:49] big arch burger in their basket. And you'll see our research agent will actually go through um the data. and we saw that it kind of peaked at roughly 10% of orders having the big arch burger before it kind of um went back downwards
[13:03] that way. You're able to also run uh quant surveys as well. Um so not just you know video interviews but also quantitative pieces which also feeds into the agent itself.
[13:17] And then you know uh to wrap up one of the workflows here you'll see that um I don't know if anyone saw the BK response. So, uh, you know, BK took this opportunity to throw some shade at
[13:29] McDonald's. >> The Burger King president threw some not so subtle shade at the McDonald's CEO.
[13:43] >> Not bad, right? >> Only one thing missing, >> a napkin. So um yeah after that you know another a agent deploy that I built is we actually have a always on sort of competit competitive intelligence agent for these
[13:56] companies that scans you know all the news out there um and uh you know we see that BK launched new whopper to kind of combat this and already uh the agent
[14:09] automatically fielded a study out to those buyers you know it's becomes a transcript of the pogo library and Um, the last thing the agent can do is actually synthesize all these different findings across multiple studies. So,
[14:24] you know, Pogo is running tens of thousands of interviews every week. And with this agent, you can kind of look across all these different studies to answer any questions that you might have. Um, so hopefully that's a uh yeah, some
[14:37] context on the different agents that we built and I'll hand it over to Luke to talk about some of the learnings. >> Awesome. So, as Dom chatted about, we have a lot of different agents that we've launched.
[14:50] Um, and with that, everything went pretty smoothly and we didn't have any issues. As you all have guessed, there are tons of issues when dealing with agents and trying to build these systems. They're incredibly complex. We
[15:02] all know they're non-deterministic and that makes things complicated. It makes it complicated when you're dealing with a single agent and we have ambitious goals. We want to build lots and so trying to figure out how we can take the best learnings along the way has been a
[15:15] major focus for us and so hopefully we can share some of those with you all and can pick up a couple of things along the way. So our question here is actually trying to tackle let's think about how we tackled the hardest of all the agents
[15:27] that we have. So Dom went through a lot but I'd say the one that has the largest scope is actually the AI research agent. This is an agent that we hope can actually be capable of honestly doing anything across our platform. It should
[15:39] be able to find audiences. It should be able to do analysis both on the qualitative transcripts as well as the quantitative data that we have along the way. And we wanted to do it well. Answering any question from a user
[15:51] across the entire platform across all of our data sets. Kind of scary. Along the way, we have a challenge here, which is that we have two massive data sets and neither are super easy to
[16:03] analyze. Dom chatted about the fact that one of our data sets is the billions in purchase data that we have. It's structured, it's tabular, it's rich with all these behavioral insights like what did a consumer actually purchase. But honestly, it's also kind of nuanced. We
[16:16] have receipt data, credit debit data, etc. And that is a lot of information to have to figure out exactly how you analyze it. And on the other side, we have millions of interview transcripts. This is very unstructured text. There's
[16:29] a it gives us the why behind every purchase which is incredibly valuable but it's also quite messy and it's something that realistically historically was almost impossible to go through in any sort of efficient manner
[16:41] and so because it was a complex problem we addressed it with a complex architecture and so the very first thing that we tried was this multi- aent system it was a mess it was something
[16:54] that when we looked at the old architecture we decided that we were going to have a supervisor at the top and this would be the brains of the whole system. And underneath of them we would have a team and this team would be a SQL agent and a Python agent and then
[17:06] an analyst who could run some calculations. And then afterwards we might have a reporter and this report generating agent would be somebody who would take all the insights the team pulled together and make it really nice in a markdown with some like charts etc.
[17:20] And the supervisor would just make sure that everything hummed along the way. Now the thing that we ended up figuring out along this process is that the complexity we introduced to the architecture made it incredibly hard to
[17:32] build and incredibly hard to maintain. When there was an error that occurred in one agent, it cascaded to every other agent. It made it so that one of the things we focus on more than solving the user's problem was about how do we
[17:45] figure out the right handoffs and interaction and coordination between the different agents and how they talk to each other. Even something like, okay, which agent should have what context was a conversation that we would spend hours on instead of just trying to figure out
[17:58] what was the best way to answer the user's questions. And as a result, anytime that we would see that one agent broke, we would go fix it and we'd fix it really well. The issue is fixing that actually caused a problem in another
[18:10] agent and how it received things and it was just too much. So, we simplified it. What if we did something crazy and threw it all away? What if we collapsed everything down to a single agent? The new architecture is one that if anyone
[18:23] has used a coding agent probably feels pretty familiar. I have an agent that can run code. It can see the output and it can self-correct. And then when it sees that output and itself corrects, then it runs it again and it tries to
[18:35] see how things go from there. The single agent has full context. The single agent is allowed to see its output and selfcorrect. It doesn't have to deal with all the handoffs. It has all the context, all the things that went wrong
[18:47] in the process, all the things it's learned along the way, and that really helps us a ton. What we also have done in along the process, now that I've run out of battery on my monitor,
[19:05] is made everything a coding problem. So one of the things that we have really found is that okay we have these AI labs that are putting a ton of investment into making their agents the best at coding that they could possibly be. We should just ride that wave. Agents are
[19:17] good at specific tasks and one of them is now coding and so you should be in distribution i.e. ride the wave that the frontier labs are pushing on to make sure that you get the best results. If you give your agent an execution
[19:29] environment where it can safely run code on your behalf, you've now empowered it to essentially answer any type of problem that a human could do on their own computer. So our agent can run SQL, it can run Python, and we also give it
[19:42] tools where it can essentially do anything that a user could do in the platform itself. This makes us that validation is much more simple. It's much more easy. You can run things the agent ran again and see how it works. You can even run tests and validation
[19:55] systems automatically. So an agent can't even show a user an output before it's passed certain test thresholds etc. And then lastly is skills. So again folks have used coding agents this is going to
[20:08] feel a little familiar. One of the things that we wanted to do with our multi- aent system was there's just a lot of context on all the business problems that we work on on a day-to-day basis. And so we want to give it as much information as possible. But sometimes
[20:20] we kind of overwhelm the agent and it gets a little confused. we poison the context or we make it so that it has like way too much it has to deal with or handle. And so we tried to deal with that by having a whole bunch of different agents. But instead we figured
[20:33] out that skills is actually a reasonable way to do this. So skills are these packets of information essentially has a title of like what the skill does. It has a short description and that's what the agent gets when it starts its
[20:45] session. Now if the agent says I want more information around how do I think about panalization in a longitudinal data set which you don't really need for every question I can go access that information I can get every single piece
[20:57] of detail around what's the best way to go tackle that when I need it so that it's not always bloating my context this allows you to have deep expertise on demand it allows you to keep your context clean and then we didn't have
[21:09] the issue where when we added paragraphs and paragraphs of a new capability we wanted the agent to do the problem where you ended up making all the other capabilities worse because you were overwhelming the agent.
[21:21] So lastly, why does this all matter? Because that thing I was saying before around the fact that now we get to actually focus on user questions. When you stop fighting complexity by
[21:33] admittedly trying to be really really excited by the intellectual problems and the engineering that you're doing, you actually solve real user uh questions and real user value and you can focus on what the agents are actually good at. So
[21:46] before we would say how did the supervisor misoute this? How do we get reliable handoffs? And now we started to say wait we have this quantitative data, this qualitative data. How do we combine these in a way that's never been done before? How do we make insights faster?
[21:58] How do we make them proactive? And what could we do that a team of human agents honestly couldn't do and do it much faster? So the unlock for us was that AI is a language technology piece. The
[22:10] quantitative analysis is actually something that we already had data scientists who were able to do on our team, but it honestly took a long time. It may take them hours. It may take them days and the agent can do it much
[22:22] faster. But the qualitative analysis, if you think about the consumer insights industry, it takes weeks and weeks to do the interviews, a ton of no-shows along the way, and then you try to take all this very messy qualitative analysis,
[22:34] and somebody reads it and rereads it and rereads it, looking for patterns and trying to figure out exactly how to put it all together. An agent's natural at this. It's able to do theme extraction. It's able to see the key insights. It's able to cross compare in a way that is
[22:46] like very, very hard to scale. And what that allows us to do is essentially combine these into a single workflow. We can have an agent that is able to do both quall and quant in a way that allows you to say what did this
[22:58] person buy, when did they do it, how often, and then why did they do it? And what drove them to make that decision and what would actually bring them back to you as a person who just wants the
[23:11] brand that you're working for to succeed. So that's it. It's very simple. What we learned along the way is that we should go to a single agent and keep the feedback loops as the advantage. Now
[23:25] there may be questions later around is there ever a time for sub aents? Some coding agents have sub agents. The answer is yeah, there are some times for sub agents, but I would urge every single person in the group to think about how they can get as much as they
[23:38] can out of a single agent before they ever think about a multi- aent system. Really juice it for everything you have before you add complexity to your system. Two is if you make every analysis question a coding question, you
[23:50] give the an like agent the ability to essentially do tons and tons of things that are honestly incredibly incredibly powerful and things that honestly surprise us every single week when we turn up for work. And then the last piece is once you start to simplify and
[24:03] once you give your agent the power, you actually allow yourself to unlock the real prize of answering questions that your users actually care about. for us that's putting qual and quant together and it's something that we've seen a lot of happy customers utilize.
[24:18] >> Cool. Well, thanks Luke and uh yeah all those findings have kind of powered the pogo platform itself. Uh we're going to take some questions but before going into questions you know as a thank you for uh coming out to our talk as well.
[24:31] Um if you are a business that uh is interested in talking to consumers um we are offering you know five free trial studies this week first come first serve. So yeah, if you want to use a
[24:43] pogo platform to talk to, you know, actual buyers of products, um, you know, whether you're a consumer brand that sells chocolates to a consumer app or whatever it is, um, yeah, just scan this QR code or or just shoot me an email
[24:57] domjoinpog.com with a subject line data bricks to to claim one of these free studies, uh, where we'll help you, you know, talk to, yeah, dozens of consumers about different things.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.