Self-Healing Product Catalogs: Drift-Aware Computer Vision for Retail & CPG on Databricks

Summary

Neurolabs runs image recognition across over 100,000 SKUs in 28 markets, with all inference results landing in Unity Catalog on the Databricks Data and AI platform where every shelf crop is scored for its training value.
A Delta Lake medallion architecture (bronze, silver, gold) governs which image crops feed the next model version, enabling active learning that selects high-value crops—such as new pack variants or misclassified SKUs—for labeling rather than processing all millions of monthly images.
Uncertainty quantification and model health monitoring detect catalog drift before accuracy falls, creating a self-healing pipeline that keeps the computer vision model sharp as new products, seasonal SKUs, and regional variants continuously enter the market.

Self-Healing Product Catalogs: Drift-Aware Computer Vision for Retail & CPG on Databricks

Product catalogs are living things — new packaging, regional variants, seasonal SKUs are always cycling through — and every change impacts the model's accuracy.
Neurolabs runs image recognition across 20,000+ SKUs in 28 markets, generating millions of shelf crops every month. Most are routine; only a handful contain the signal that makes the next model better — a new pack variant, a misclassified flavour, a confused competitor. We've moved that selection loop onto Databricks. Inference results land in Delta, every crop is scored for training value, and Unity Catalog governs which crops feed the next model and surfaces catalog-health metrics back to the CPG brands that own the SKUs. The result: a product catalog that heals itself, and a computer vision model that gets sharper with every shelf photo.
Talk By: patric fulop, CTO, Neurolabs ;

Chapters

FAQs

What is a self-healing product catalog and why is it needed in CPG retail?

A self-healing product catalog automatically identifies when new packaging, seasonal SKUs, or regional variants cause the image recognition model to drift, then selects the right shelf crops to retrain on—keeping accuracy high without manual intervention. CPG brands need this because product catalogs change constantly with events like FIFA World Cup and NBA promotions, making it impossible to maintain model accuracy manually at scale.

How does Neurolabs use the Databricks Data and AI platform and Unity Catalog for visual AI?

All image recognition results from millions of shelf photos per month land in Unity Catalog on the Databricks Data and AI platform, where a Delta Lake medallion architecture structures the data and governs which crops feed the next model version. Unity Catalog also surfaces catalog-health metrics back to the CPG brands that own the SKUs.

What is uncertainty quantification and how does it help with SKU classification?

Uncertainty quantification scores each image crop for ambiguity, identifying cases where the model is unsure about a classification—such as when a new pack variant looks similar to an existing SKU. These high-uncertainty crops are prioritized for human labeling through active learning, ensuring that only the most valuable training examples are sent to annotators.

How does Neurolabs use agentic analytics on Databricks?

This video demonstrates an AI query demo where agentic analytics on the Databricks Data and AI platform answers business questions about share of shelf and promotional compliance directly from the gold tables that store inference results. This gives CPG brands real-time insight into how their products are placed and priced on shelves across markets.

Full transcript

[00:08] All right. Hi. Hi everyone. Sorry, didn't see that. Uh Nice to Nice to see you all today. Uh I'm Patrick, co-founder and CTO of Neuralab, and I'm excited to present today what we've been building on Databricks. And uh I want to highlight the
[00:20] complexities of building visual AI technology for the consumer packaged goods industry, with its unique challenges, but also how we're using Databricks to enable both an analytical product for our customers, and also path
[00:32] for uh improving the model with the right data, and choosing that data carefully. And finally, the product catalog is the foundational aspect of our work, and I'm going to cover quite a bit on that.
[00:45] So, first off, uh a bit about Neuralab and and what we do. So, we provide supply chain visibility for the consumer goods brands, CPG brands. And from images captured in the field,
[00:57] brands care about how their products are placed on the shelves, whether the promotions they've set up for the new World Cup are actually running, and whether the prices are actually the prices that they should have. And they want this data in a matter of
[01:09] days, not weeks. So, we use image recognition to turn all of this raw data and image data into actionable intelligence that then connects the front of store execution to the back end
[01:22] uh operations of the CPGs. Now, um there's a unique challenges in the industry, and the catalog is the first one. We have over 100,000 SKUs in our
[01:34] catalog, and every pack variant, regional label, and seasonal launch, we carry that, and we also need to make sure that we keep up with the constant changes. There is a lot of uh changes in the space, and it's an extremely dynamic
[01:46] catalog. You have FIFA World Cup, you have Star Wars, you have NBA, you have NFL. There's a lot of changes, and we constantly need to make sure that the catalog is up to date. We operate across 28 markets with these customer catalogs enabling multiple use
[02:00] cases across different and diverse environments. For example, we do display execution, price compliance, promotional executions across different places such as the display, the shelf, or even the
[02:12] back room. Now, all of these image recognition results land in Unity catalog and they help us answer both business questions, but also reinforce the model in terms of what is
[02:24] actually useful. And we need to make sure this is a streamlined and efficient effective or efficient process because we have millions of images per month that results in billions of actual rows in the catalog.
[02:41] Now, explain our labs image recognition and what we do in 30 seconds and this is exactly how long it actually takes for the image recognition to flow through and actually get back the results that you care about. So, this is a sample of an image coming from the field and we're going to get
[02:54] full shelf intelligence in 30 seconds. What actually mean What What is that What does this actually mean? We're going to recognize each SKU, we're going to recognize where it is on the shelf, and we're going to recognize the prices it has, the
[03:07] promotions, but also the gaps on the shelf and this allows us to create rich insights and execution scores. And as you can tell here, >>
[03:20] shelf one, row one, and position eight, and we also assign different prices for this product. Now, the way we do this is we have a gallery of references that come from
[03:33] synthetic data, from real data, and we match these with the crops that are incoming from the real world. And this is the process by which we actually issue the image recognition works. So, a healthy catalog and healthy
[03:46] references directly equate to high accuracy, which is what our customers want to see. And we've built a system that allows us to uh
[03:58] create these robust models uh that generalize across use cases. And the way we've done it is we've trained with synthetic digital twins um going from real images to synthetic digital twins.
[04:10] We're able to take a product, place it in different environments. I mentioned all of these use cases, fridges, coolers, shelves, POS displays, uh back rooms. They were able to train robust vision
[04:22] models on this data such that it works across different use cases. Running a new campaign, you have Star Wars, you can just update the artwork, you can generate new synthetic data, train the
[04:34] models to be uh to learn before the product even hits the shelf. For competitors, we've built an app to digitize any SKU in under 1 minute. And this allows us to stay ahead of the competition and keep up with the
[04:47] constant pace of catalog changes. Now, all of this data that I've shown you and all of this image recognition data lands in Unity catalog. We will
[04:59] build analytics and insights on top of it. So, let's look at some of the architectural choices we've made to ensure that this works at scale. So, we have uh image recognition core platform
[05:12] that lands in Unity catalog. We have the bronze layer, just raw data. Then we have silver. We have some cleaning, we have some cross data set comparison between SKUs, promotions, and prices.
[05:24] And then all of these land in the gold layer, which is where we have six different tables that I'll speak to in about a few minutes. And uh there's a semantic view on top. This is really what makes it
[05:37] makes a difference. The the semantic view allows us to create custom KPIs for brands like promotional execution. What what does that actually mean? This is a coded uh metric view on top of the gold
[05:50] table that the customer can actually change and impact. Then all of this data is passed through Delta Share to our customers and they can build and build their own workflows
[06:02] and their own pipelines. So, I mentioned the six gold tables. We have a facings table which is individual product facings.
[06:15] This is the most granular table and it contains all of the information that is required to merge with the other ones. The shelf contains uh data at the shelf level, so row level. Uh then image
[06:28] which which which store visits were captured and promotional. Promotional pricing only runs for a certain period, so we have the dates, we have the exact promotion that are running. And then we have dimensions table. So,
[06:41] we have the catalog which has a rich metadata in terms of brand, flavor, categories, and so on. And then we have where are those actually where are those images were taken. So, subsection type could be a display, it could be a
[06:54] cooler, it could be a different locations. Now, the idea is that these dimensions stay small. The catalog can go from 500 to 5,000 to 10,000, whereas facings can go from
[07:06] 1 million to another million the next month. So, let's look at a specific example. We've gathered a data set from New York
[07:19] area of soft drinks and energy products over a couple of weeks and we can answer questions like what promotional prices do you observe for Alani and which stores have the best prices. Now, what Genie does is
[07:33] takes the facings of all the SKUs and all the facings of these products that are recognized and it connects it to the to the catalog. First, it filters by brand, then it filters by region. Here, example New York. Then, it goes and
[07:45] says, "What promos do I observe for this timing and for these regions?" And then, all of these are aggregated and returned. And you can see there that the uh 7-Eleven in Clifton has the best
[07:57] deals on Alani products. This is what uh category managers use to to to understand and to act and improve the execution in store. Now, you can see that we haven't touched
[08:09] the shelf table or the image table. And this is how we're able to serve fast res- fast results through Genie, to AIBI, and uh through different workflows.
[08:23] So, the takeaway is there's one gold layer and each brand can shape up how they interact with that gold layer. The same govern metric views serve all of these uh different workflows. And I have one example here that I want
[08:36] to show, which is second, we'll see. Um we have some competitor intelligence in terms of Red Bull versus Celsius priced
[08:48] prices in New York area. And we can see in here that the Genie is able to go in, query the data, and find out the prices. But then, it can also do next best action. So, you can ask it, "What should I do to improve my compliance? What
[09:01] should I do to improve my share of shelf for Red Bull?" And it goes and finds the lowest performing stores, for example, but it goes all the way to the level of the shelf row. So, you can see exactly Maybe you want you know, you've you've
[09:13] negotiated with a uh a retailer uh positions on the shelf, and you want to make sure they're actually eye level. So, it goes all the way to that level of granularity.
[09:26] Great. So, now that we talked a bit about the catalog, we've looked at all of these data flows and how uh they uh they serve our customers and um streamline the process across their organizations for insights and next best
[09:39] actions for commercial execution. Now, those same data points that land in the gold layer are used to improve how the model behaves over time and how it generalizes across new use
[09:51] cases. So, this is a an end state of a catalog for a kind of Coke. And you can see in here that there's quite a few products. We've got uh some gen original Coke, and
[10:04] then you've got Star Wars Coke, you've got uh Christmas Coke, and then you've got the FIFA World Cup Coke. These are all variants that are present in the store, and it's imperative that the image recognition recognizes each one of them as they hit the shelves.
[10:21] Now, the truth is health degrades. These products, when they hit the shelf, sometimes, especially for competition, you cannot be proactive. You need to be reactive. So, when you're when you're processing all of this data from the shelf, the
[10:33] idea is to understand do I do I how do I select the right crops, the right uh images from my from my incoming data such that I can improve the health of the the catalog and not not have a
[10:47] degrading health. This is some examples of reference images from uh the original one. We have some real data. We have some synthetic data on the on the right. And the idea is to have enough data that's
[10:59] representative and captures how the shelf looks like, but also diverse. So, you have different light conditions, different angles, different use cases require different types of references.
[11:13] So, our model is using embeddings, and if you're familiar with embeddings, you know that that's that's how the the space operates these days. And the idea is to have uh the representative um samples that uh anchor the recognition
[11:27] in a product's real-world appearance. So, if we start with just a thumbnail, we're going to have a prototype for that thumbnail and we're going to have a prototype number two for a Coke Zero, for example. The red dots on the screen
[11:41] denote the references that would belong to the Coke original product, right? So, you've got the Coke original product, you've got three references, they look very similar, and then you've got one that's the dark the dark one there that
[11:55] is between the Coke Zero and between the Coke original. So, the question is, how do you actually label it? How do you use it? What what do you do with that? And for that, we use a measure of uncertainty with we use entropy and we that measures
[12:08] the uncertainty for that crop and says, I don't know which where I belong. Do I belong in Coke Zero? Do I belong in Coke original? And that's how we're able to add both representative samples, but it also informative samples.
[12:22] So, let's look at some examples here. We've got here the original one and then we've got some representative samples. You can see that they look very similar, right? So, this is a typical pack shot that helps bootstrap and anchor the recognition to
[12:35] to get going. Then, you observe new products, you observe new light conditions, you observe new use cases, and the idea is to add more richness to the data, right? So, you
[12:47] have to add more diverse samples. So, you pick the ones that have high entropy. For example, in new angle or new light. And this ensures that we have coverage across our different use cases, but also
[12:59] generalize ability across these use cases. And of course, we'll have the odd one out near duplicates, which we cut out.
[13:11] So, let's look at the full loop now and how we improve the process with the same pipelines that run the image recognition for our customers, they run this self-healing loop. So, first we recognize products on the shelf. This data is passed into a Unity catalog
[13:25] where we have pipelines to monitor, for example, the lowest confidence one. So, here I you can see the lowest confidence predictions uh over that data set I was mentioning earlier on in energy and soft drinks, and we can directly target only
[13:38] these ones for uh labeling or relabeling. How does the catalog update actually look? So, you're you you're going to score each of the crops. You're going to have millions of crops. You need to make sure you take out of, you know, 1,000
[13:52] crops recognized for Coke, you only pick 10. So, which ones do you pick? We're going to score them, then we're going to select them, and then we're going to promote them into our catalog gatekeeper in our platform where a human is
[14:04] actually going to review that the data is useful. But, the core is we cannot review thousands and thousands of images. We need to review the right images.
[14:19] So, to bring it all together, we've got the gold tables that we're going to share with our customers so that they can build insights, analytics, and answer queries uh with Genie. And then we have the same pipelines
[14:31] that are monitored for improvement of catalogs. And um this process is a automatic process that keeps updating every every few weeks and
[14:43] as data comes through. These are some of the use cases we cover. So, you can see that we've got quite a lot of variety. We've got back
[14:56] room inventory, we've got pricing compliance, we've got display execution, we've got field sales execution, we even have uh menus on on uh and bar bar tabs um and planogram compliance. So, all of these
[15:08] use cases, the common denominator is the catalog. You need a healthy catalog, you need a process by which the catalog constantly updates, and uh is maintained throughout its its
[15:21] lifespan, let's say. This is what enables us to go and use our image recognition in visual AI across the supply chain.
[15:36] So, three things to take away. Um model the grain. So, we've got one gold table that is speaking the language of the business. We've got facings, we've got promotions, we've got shelves. We make sure that this data is structured in such a way
[15:48] that the business understands what it's working with. Then the mod once we have a foundational model or frontier model that's been trained let's say every year, every 6 months, the idea is to make sure that the
[16:01] references that this model has are very good. And that ensures that the high accuracy is consistent. So, accuracy is really a data problem. Uh and Databricks helps us makes this
[16:13] make this measurable and actionable. And then the idea is the last point is you need to select the right crops. You need to score every every every one of them and you need to ensure that these
[16:26] crops have model value. Otherwise, your health will degrade and your accuracy will degrade and customers will not be happy. Uh thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.