Generative AI Merchant Matching

Summary

• Cost-effective AI engineering: This video demonstrates how to build sophisticated AI systems with modest budgets by combining fine-tuned Llama models, hybrid search, and careful optimization—achieving 400% latency improvement while keeping costs low.
• Enterprise merchant enrichment: Learn how Mastercard uses fine-tuned LLMs and AI judges to automatically parse transaction descriptors, match merchants to known businesses, and enrich real-time transaction data for 100M+ daily transactions.
• Production AI patterns: Discover practical techniques for prompt engineering, fine-tuning cost management, latency optimization, and hallucination detection—showing how small teams with limited budgets can deploy effective AI systems using Databricks and fine-tuned models.

Generative AI Merchant Matching

Talk by Tomáš Drietomský, Senior Data Scientist, Mastercard
Our project demonstrates building enterprise AI systems cost-effectively, focusing on matching merchant descriptors to known businesses. Using fine-tuned LLMs and advanced search, we created a solution rivaling alternatives at minimal cost. The system works in three steps: A fine-tuned Llama 3 8B model parses merchant descriptors into standardized components. A hybrid search system uses these components to find candidate matches in our database. A Llama 3 70B model then evaluates top candidates, with an AI judge reviewing results for hallucination. We achieved a 400% latency improvement while maintaining accuracy and keeping costs low and each fine-tuning round cost hundreds of dollars. Through careful optimization and simple architecture for a balance between cost, speed and accuracy, we show that small teams with modest budgets can tackle complex problems effectively using this technology. We share key insights on prompt engineering, fine-tuning and cost and latency management.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does the merchant matching system work?

This video outlines a three-step process: a fine-tuned Llama 3 8B model parses unstructured transaction descriptors into structured JSON components (merchant name, payment facilitator, address). A hybrid search system finds candidate matches in the merchant database. A Llama 3 70B model then evaluates and selects the best match, with an AI judge reviewing results to catch hallucinations.

Why use fine-tuned models instead of just larger models?

The video demonstrates that fine-tuned smaller models (Llama 3 8B) are faster and cheaper for entity recognition than using only large models (70B). By combining fine-tuned specialized models with a larger model for final validation, Mastercard achieved 400% latency improvement while managing costs. Fine-tuning each round cost hundreds of dollars versus thousands for larger model inference.

How does this system balance cost, speed, and accuracy?

This video reveals the optimization trade-offs: using smaller fine-tuned models for initial parsing (cheaper, faster), hybrid search for candidate retrieval (combining semantic + keyword matching), and a larger model for validation (accuracy). An AI judge adds another layer of validation to catch hallucinations without excessive cost.

What insights does this approach offer for building AI systems?

The video emphasizes that expensive, brute-force approaches (using only the largest models) aren't necessary. By combining smaller fine-tuned models, clever architecture, and careful optimization, small teams can build enterprise-grade AI systems cost-effectively on the Databricks Data and AI platform.

Full transcript

All right. Uh let's get started. Hello everyone. I am from Slovakia and uh I am a data scientist at Mastercard for almost three years now. And today I'll be presenting generative AI merchant enrichment or the game how we call it. So we've built end to end uh geni powered merchant matching engine for the price of a ticket to the conference and I would like to show you how we did that and what was driving our decisions when we were building up the methodology.
So let's set up the context first. I believe that all of you can relate to the situation when you open your banking app and you you see a transaction that you don't fully recognize. Uh this particular transaction is from my trip to New York last year. And you can see that in the transaction overview, we have the merchant name which is the SQ asterisk uh sunshine coffee row and the address is large. Well, it doesn't tell you much, does it?
Um Mastercard has a transaction enrichment product that is based on Mastercard's realtime merchant directory that can help you with that and enrich the transaction. So what happens is that you send us over these kind of transactions and we'll ret retrieve the complete information about it. So in this case we identify that sq asterisk is a square which is a payment facilitator. Um uh sunshine coffee row is actually a merchant uh with comp its complete name is sunshine coffee roasters.
Then we retrieve the complete address as well uh and contact information, latitude, longitude, merchant category, uh and many more. Now let's discuss what's actually happening under the hood. So what you see over there is what we call a transaction descriptor. A transaction descriptor is a bank transaction message that can contain various entities such as payment facilitator, merchant name, street, city, zip, state, country, phone number, website, MCC codes, and some other stuff.
Um, the game can handle variations of the unstructured transaction descriptors. So, it doesn't really matter which entities do you provide in which order. It's a bit doesn't matter anymore. Um so when it comes to the enrichment there are two main parts. The first part is the entity recognition. So the LLM extracts key data elements from the transaction text and uh structures it in a JSON. So in this particular case we are going to label SQsterisk as a intermediary or or a payment facilitator.
We are going to label as a merchant and Larchment as a city. Now the legacy solution was based on fuzzy matching logic, right? Comparing strings, calculating level distance and the distance between uh the strings. It works but there are serious drawbacks. Now when it comes to LLMs, one of the main strengths of the LMS is its generalization. So it recognizes the patterns beyond the literal text understanding variations.
When it comes to fuzzy matching, if you would have a truncated merchant like this, it'll be a huge problem. But LMS don't really care about that because it understands the underlying context behind the text. Semantic understanding comes hand inhand with that. So the LLM understands that Larchment is a US city and it also understands the location hierarchy. So it understands that there are certain zip codes related to this city.
It also understands that Larchman is part of certain US state and US as a country etc. In addition to that we have cities which are present in multiple US states such as Springfield, right? And DMs also knows that and that's all utilized while parsing these transactions. Now the second module of the enrichment is the matching process. For that we use another LLM that is uh that identifies the correct merchant based on the parsed information.
So in this case we put the intermediary aside because it's not really useful for the matching. So we just take the merchant name, we take the city and we match it in Mascard's merchant database. All right, let's dive even deeper and let's talk about the architecture. So the query contains the transaction descriptor right sunshine coffee roasters. The first part is the entity recognition. So for that we are using fine-tuned uh LLM.
So for training we have created a train set which consists of these transaction descriptors and the expected output is the JSON with labeled entities. We are using llama 31 with 8 billion parameters and we are fine-tuning it via mosaic API. All right, the second step candidates retrieval. So the candidates retrieval part is based on the vector search. So we took Mastercard merchant database merchant names the address information, contact information etc. and we have embedded it into the vector space using bg small en. and then constructed a vector search index.
So the input for this part is the parsed query from the entity recognition and the output is top 20 candidates for a match. Right? And we we select these top 20 candidates based on cosign similarity. Now we are in the in the third step we perform entity linkage. So we take the parsed query from the step number one from the entity recognition. We take the top 20 candidates for a match and we are using llama tree with 70 billion parameters to determine whether there is a match between those 20 candidates.
If yes, we select that merchant record record and retrieve the enriched transaction information. If there is no match, we simply say no match. This whole architecture is uh in our data bricks workspace and is completely offline. We are not using any third party services uh emphasizing the privacy. All right, let's talk about benchmarks. So what you see here is the benchmark for the named entity recognition part. You can see various llama models uh starting with llama 31 with 8 billion parameters, 70 billion parameters etc.
We are measuring the success on precision here. So this is the foundation version. So no examples, just the best prompt that we could write. Now when it comes to the prompt engineering, uh there is several strategies to be considered when you're building your prompt. Uh one of them is including examples, relevant examples. Now you can see there is only marginal difference in the precision when including more examples.
And in the case of llama 32 with three billion parameters, there is even worse performance because the model got confused with examples. So based on our experience, there is a couple of concepts that we follow when we are building the prompts. So concise prompt versus detailed instructions. Based on our experiments, the longer prompts with detailed instructions yield better accuracy and more consistent results. Uh when it comes to reasoning pathways, um there are basically two main types that we considered.
Chain of thought in which you basically teach the model how it should think about your inputs and how it should how you should lead the model to the expected outputs. While in the multi-stage approach, you break down your prompt into small several smaller tasks and then putting it all together. So how do we actually choose the best prompt? The obvious choice is you take all the prompts, you take all the models, you run your benchmarks and you select the best one, right?
That's what anyone would do. But something we consider uh while building the prompt is the instruction teachback. So we actually make the LLM to explain the instructions back to us. So we can see that it fully grasps all the detailed instructions. And we've seen that especially for the longer prompts, the LMS tends to prefer some of them and ignore the other instructions, which is not really good because you then you're just making your prompt bloated for no reason.
Now fine-tuning it. We were actually impressed to see how much does the finetuning affect the accuracy and you can see that even in llama 32 with three billion parameters there was significant increase in the precision where we went from 52% to 96%. Um some basic concepts that we consider when we are fining the models. So having perfect ground tr ground ground truth labels is an absolute must. And I cannot really emphasize this enough because incorrect and inaccurate labels propagate and amplify the errors.
Um when it comes to robustness, you have to make sure that your train set is diverse enough because and you have to make sure that you're only including the relevant examples uh in your train set because otherwise you're going to promote something that you didn't really intend to do. when it comes to the predictions and structured output design. So what we've seen especially with the smaller llama models is that let's assume we take a transaction that contains like a Walmart and we've seen that the model will just keep repeating Walmart until it runs out of the context window and that's why we are limiting the output uh uh maximum output tokens by uh I think 1,000.
But uh this is a problem with llama. We've seen this in other use cases and you simply have to manually intervene and just stop it, retrieve uh sorry, retry and try to parse the transaction again. If it doesn't work out, then you just have to use the original input and proceed. All right. Now, let's talk about the embeddings and the vector search. So, embeddings unlock incredible potential for the entity linkage problem, right? uh in natural language processing I would say that the string similarity metrics are the standard but when it comes to stuff like this when you have Kentucky Fried Chicken and KFC it's hard to find a metric that will tell you that these two entities are these two uh strings are related to the same entity same goes with some locations New York City and NYC but there are it's not just about acronyms you have Walmart and WM super centers which stands for Walmart super centers Right?
So there's no string similarity but they yet they refer to the same entity. Um when it comes to embedding model selection the optimal choice depends on the data domain and task requirements. Um on the x-axis we see the encoding time per query in milliseconds and on y axis we have the recall. Now the marker size uh represents the model size in millions. Now based on our experiments and for the for our use case BG small EN and Ember V1 yield the same recall but the thing is BG is much smaller model therefore it's cheaper to run it and it's also much faster so BG small was obvious choice for our use case back search.
So the key factors to consider when building a vector search is the data scale, accuracy, and latency. And you always have to kind of play with these three levers and find the right spot for your use case. Data quality, duplicates, and ambigu ambigu ambiguities hinder search performance a lot. So in our use case, we are retrieving the top 20 candidates right in the candidate retrieval part. Now if you have duplicates there you might unfortunately retrieve the duplicate records 20 times and therefore you will never get to the right match.
So data quality is super important input structure. Um so usually especially when I was also attending other sessions people are always having longer documents and they are chunking them into smaller parts and then embedding them. We are dealing with quite short merchant records so we don't have to chunk them. So we just have to figure out the right way how to structure them before we convert them into the vector space.
So that's something that is significantly affecting the accuracy as well. And this not only related to the building the vector search but the K is the number of candidates that you are retrieving. And what we see in this figure is on x-axxis we have the relative inference time to the baseline where we selected five candidates and on y-axis we have the relative match rate to the baseline of five candidates. Now as I said the sweet spot for us for our use case is k20. show yes we increase the inference time by approximately uh I think 250% or or 200% and we gain around 50% uh bigger match rate we could go with K80 why not but it's just dependent on the use case and if you can really afford that uh huge inference time all right now let's put everything together so as I was saying there are the strategically levers that you have that you have to play around with to actually uh create a working solution that is satisfying your needs.
Accuracy, speed, cost and complexity. Now when it comes to pulling these levers, as I was saying, the retrieval scope lifts the recall and the match rate, but it adds latency and throughput cost. model size. Models with bigger models, models with more parameters might increase the answer quality, yet they raise uh latency and the cost. Advanced reasoning logic. This is something we have tested long time ago where you basically take your input query, you take multiple LLMs and you chain them.
So you take the first query, you run it to the first LLM, you get the output, then you take both that, submit it to another LLM and then do it again. Uh based on our experiments, um if the first LM is not figuring out your problem, the second one is unlikely to do it. So that's why we are just keeping it simple and using one LM uh for the final outputs prompt length. So as I said earlier, longer prompts with detailed instructions yield more accurate answers, but it adds latency and again the instruction teachback.
You have to make sure that the model is fully utilizing your prompt. So uh you're not adding the latency without g gaining anything out of it. All right. And now some results. So this is a bench this is the same benchmark uh uh as I showed earlier but here we are actually categorizing the transactions in our benchmark set to three classes class A easy. So how do these look like? So these are usually the complete merchant records.
So when you have merchant name, street, zip, city, state, country, it's quite easy to match those because you have to match it on all these entities, right? So that's quite easy. Uh on the other hand, it's also very easy to match certain merchants that are unique. So if I do if I would have a my own bakery, let's call it awesome bakery uh in Prague Czech Republic where I live, it would be quite easy to match that because there's just one.
It's not a merchant chain. It's it's not complicated. So these kind of transactions are what we call consider easy to match. Now when it comes to class B moderate uh these are the transactions that don't contain that many entities. So an example would be Walmart 1 2 3 4 which stands for the store number Houston Texas right? So we just get three entities but when it comes to Walmart there is a lot of them in in Austin.
So there's multiple stores and the LM has to choose the right one. So it has to fully utilize the store number information and pick the right one. So the class B for us is when there is multiple there is multiple relevant candidates and the LM has to choose the right one. Um class C so the class C are the most difficult transactions where the legacy fuzzy matching system usually fails. These are transactions that are usually uh processed via payment facilitators or they contain what we call dynamic descriptors.
So some uh e-commerce merchants are actually submitting the order numbers and the transaction ids into the merchant names and each transaction is therefore unique. So fuzzy matching cannot handle that right like you you can have for example Amazon 1 2 3 4 Amazon 1 235 Amazon ABCD whatever and fuzzy matching cannot really solve that but LMS can because it's simply detecting the patterns in the in the transactions and then uh pares it out.
So the class C is hard and now to wrap it all up we can see that for class A there is only marginal increase in the in the match rate but for class B and class C there is significant amount and uh we are very happy with the performance so far we keep working on it we keep re reiterating with our success team make it even better and yeah the last topic I would like to discuss is how much does it cost to run game and we'll break it for each module.
So when it comes to the entity recognition part where the input is the transaction and the output is the structure J JSON with labeled entities, it cost approximately $130 for one fine-tuning run which takes around 27 minutes. So it's quite fast. We really love Mosaic ML. It's very convenient and very easy to very easy to use. Uh after the model is fine-tuned, we use data bricks mosaic serving. Uh and that cost approximately $7.
4 per hour because we are using the provision throughput. Now the embeddings and the vector search. So it cost approximately 9 $9. 4 per hour to host our vector search and approximately $2 to host the embedding model which is BG small en. And then finally uh the hosting of the judge or the last model that is actually determining the match cost approximately $48 per hour which uh in total gives us around $70 per hour of running this solution.
Uh that will be it. Thank you very much for attending. Me and my team will be over there to answer any questions. So that's it. Thank you.

Learn more about the Databricks Data and AI platform.