From Rules to Reasoning: AI Agents for Data Enrichment at Scale

Summary

YipitData built a three-step AI agent architecture combining entity extraction, hybrid search (lexical and semantic), and constrained reasoning that achieved 92% accuracy for mapping messy company names and transaction descriptions to correct entities, up from 70% with regex and NLP.
The platform grew entity coverage 33x from approximately 3,000 to over 100,000 companies by replacing brittle rules with adaptive AI agents orchestrated through Databricks Workflows and batch inference via Spark UDTFs.
The video covers architecture patterns and GPU trade-offs between serverless and self-hosted options, with practical guidance for data engineering teams building AI-powered enrichment pipelines at petabyte scale.

From Rules to Reasoning: AI Agents for Data Enrichment at Scale

YipitData processes billions of unstructured data points at petabyte scale to serve institutional investors and Fortune 500 companies. Regex and NLP techniques plateaued at 70% accuracy, leaving critical information unstructured. The core challenge: mapping messy company names and transaction descriptions to correct entities required reasoning, not pattern matching. This became a bottleneck for their enrichment pipeline.
Learn how YipitData built a three-step AI agent architecture combining entity extraction, hybrid search (lexical and semantic), and constrained reasoning to achieve 92% accuracy. Discover orchestration with Databricks Workflows, Spark UDTFs for batch inference, and GPU trade-offs. See how this adaptive platform grew entity coverage 33x from 3,000 to 100,000+ companies, enabling continuous enrichment without brittle rules.
🤝

Chapters

FAQs

Why did regex and NLP fail for YipitData's data enrichment at scale?

Regex and NLP techniques plateaued at 70% accuracy because mapping messy real-world company names and transaction descriptions to correct entities requires reasoning over ambiguous cases rather than pattern matching. Edge cases caused regex rule sets to explode in complexity without producing meaningful accuracy gains.

What is YipitData's three-step AI agent architecture for entity resolution?

YipitData's architecture combines three sequential steps: entity extraction to identify candidate mentions from raw text, hybrid search using both lexical and semantic techniques to retrieve potential matching entities, and constrained reasoning to select the correct entity from candidates. Separating these concerns allows each step to be optimized independently.

How does YipitData run batch AI inference at petabyte scale on Databricks?

YipitData orchestrates their enrichment pipeline through Databricks Workflows and uses Spark UDTFs for batch inference, enabling parallel application of AI models across billions of records. They evaluated GPU trade-offs between serverless and self-hosted configurations to balance cost and throughput for their scale.

What business impact did YipitData see from switching to an AI agent enrichment pipeline?

YipitData grew entity coverage 33x from approximately 3,000 to over 100,000 companies and improved accuracy from 70% to 92%. This increase in structured, reliable data directly improves the quality of intelligence delivered to their institutional investor and Fortune 500 customers.

Full transcript

[00:07] All right, good afternoon everyone. Thank you all for coming. Really appreciate it. I know it's late in the day. Well, so we'll get right into it. Today we're going to be talking about a problem that probably all of the data teams in this room are experiencing. Which is how to turn you know, messy real-world data into
[00:23] actionable information that your data teams, your customers, and your agents can trust. And so to do this, my name is Anoop Segu. I'm the chief architect at Yep data. I'm here with Ed Gu, who's our head of data engineering. And we've spent the last 12 months or so
[00:39] really trying to hone in on a process and turn our data in transformation pipelines on their head and instead of using traditional text-based approaches to doing transformation and data enrichment using AI agents at scale. And so we are going to be talking about architectures,
[00:55] patterns, and lessons learned about how to do this type of data enrichment at scale and hopefully you'll have examples that you can take with you after this presentation and try yourself. So why don't we get into it? As a quick disclaimer, all of the statements here are of our own, not a
[01:11] formal statement of our company. And I want to start by just kind of taking stock of like where we are right now in the data and AI landscape. I think throughout the summit we've kind of recognized that AI is only really a good as good as the data that goes into
[01:26] it. You know, we've talked a lot about context engineering, ontologies. All of that is really going to show that what we put into these agents is really what controls their output quality. Now, that's great and we've been talking a lot about that, but I don't think we've been talking enough about what pressure that is putting on those source
[01:42] systems of records to have clean, structured, organized information. That today is still managed probably by humans, data engineers, data analysts. And what we're recognizing is now our agents can do incredible analytics at scale. They can operate with incredible
[01:58] productivity, but we have a supply problem. We don't have enough structured data to give to it. And as for us as a data organization, we're really feeling that pressure because our company and our products really depend on having well-structured information, and it's
[02:13] now becoming a bottleneck. So, what we've been trying to figure out is how can we use AI agents to scale this process? And the question really becomes is can we actually do this? And that's been our journey over the last couple of months,
[02:29] and we're really excited to get into it. So, a little bit about us as a company before we get into it and help frame why this is such an existential prob- problem for us. You have the data, you know, we provide insights to Fortune 500 companies and institutional investors. Our data products are essentially
[02:46] multiple channels. Today, our most popular product is actually agentic experiences, so our customers log in and ask questions to agents and get answers based on our data, but we also provide data through data feeds, dashboards, and research reports. And so for our customers, the thing that they care
[03:02] about most is do they have information that they can trust, and do they have convenient answers uh to the questions that they're asking? And that all gets back into really having really well-structured and granular data sets. We have a petabyte-scale lake house um
[03:19] and with that, billions of data transactional data points. And we spend a lot of time transforming that data, restructuring it, organizing it, and that's really important because that's ultimately our product that our customers experience. And so, we can't really depend on operating at this scale
[03:35] if we need to keep processing billions and billions of rows if we have to have data teams manually tagging and enriching these data sets. That's really why we've kind of gone down this whole uh experiment to see if AI can help us. So, to kind of illustrate this with a
[03:51] real example, you know, our current mission as a company is to become a best-in-class provider of information on any business on the planet. Uh so, we want to span and provide information granular metrics on thousands of companies. But, to answer questions like, you know, what is this company's
[04:06] revenue last month, we have to have well-structured data that organizes the company that's uh producing that uh per- that transaction, the the time frame of it, the amount. And, really the challenge is is our source data set. The good news is we have data that spans
[04:23] thousands of companies. The problem is it's very unstructured. So, this is an example of the type of data that we have in our source data sets. You see, it's a bunch of abbreviations, um a lot of different terms mixed together. And, you can just think about this is like one example of maybe hundreds of billions of
[04:38] rows in our data set with different permutations. How are we going to like enrich that? And, just to like put it to the audience, like a show of hands, what company do you think this belongs to? You know, anyone think it's Microsoft? Okay. Electronic Arts?
[04:54] Yeah. Is it a company called True Up? Hm. Probably not. Or, is it none of the above? Or, hard to know? It's really tough. For humans, we can look at this string and kind of have an idea of what it could be. But, when we're trying to use traditional approaches like regex or substring matching, uh fuzzy matching, there's a
[05:11] lot of challenges to scaling this. And again, this is just one row of billions of rows that we need to process. So, we can't just depend on kind of brittle rules. And, ultimately, that's the crux of the challenge here. Cleaning data is really hard. It was hard before. The only difference is we have more technology at
[05:27] our disposal to really look at the different uh types of permutations of strings that we have in our data set. So, um we've dealt with all kinds of problems. Right now, we have complex data pipelines that have thousands of regex and if-else kind of conditional
[05:43] logic-based rules. And, that's only gotten us so far. When we look at strings like that, it's really hard to think about how to write good, clean transformations that work for the variety of companies that we deal with. Um we've also tried, you know, as we started to integrate AI, you know,
[05:58] zero-shot LM prompting and seeing that that plateaus around 70% accuracy no matter what model we use. We've also tried to use reg in this process and seeing that that has its own quality bottlenecks. Um and ultimately what we're kind of realizing through this whole journey is
[06:13] you can't look at trying to solve this problem with just one technique. You need a concert of solutions together and that's where I really want to um take this conversation next.
[06:29] So, to really understand why we are so focused on AI as a solution, you have to kind of understand why like the regex and classical solutions that we've had today don't really work. And the problem is with our data set, it's very ambiguous. You have strings like what you're seeing on the screen here where you don't know which company belongs to or you have to make assumptions about
[06:46] some of the strings in concept like in relation to other parts of that string. So, you see here that, you know, we're missing a lot of metadata in these strings about like what these companies are or these entities are. Um there's subsidiaries being referenced, so you have to know that when you're kind of doing this mapping.
[07:01] And uh this is all human generated. So, this source data set is kind of coming from like ERP style systems where everyone has their own annotations. They have their abbreviations, they have typos and, you know, when we have regex based rules trying to do this, you have to account
[07:17] for those typos and that gets really hard to do when you have, you know, infinite permutations. You also have to have like domain-specific knowledge. Um so, sometimes you might have to know like, okay, iTunes is a product of Apple, but if you haven't encoded that in your, you know, classification transformation
[07:32] rules, you have a problem. Now, that's a simple one, but if you think about tens of thousands of companies and all the products that they have, there isn't a good way of kind of handling this at scale. And that kind of led us down a fundamental realization, which is we were trying to solve a reasoning problem
[07:47] by mapping substrings. Like when we Ultimately, the goal here is trying to map or associate a given transaction to a company. Once we do that, the rest of the analytics becomes really simple. But we have a categorical mismatch with the tools that we had available to us, you know, using regex or fuzzy matching,
[08:05] using NLP, or any of these like classical ML models, they only get us so far with for all the different permutations of the data that we have. And so, what we really recognize is, well, we have LLMs now. They can do reasoning. They kind of understand the human world around us, and they can make
[08:21] uh helpful decisions in trying to do this mapping exercise. So, uh we started to experiment with it, and we went through a lot of trial and errors, and that's why I want to turn it over to Ed to help you kind of understand our journey. Thank you, Anoop.
[08:40] So, when you're staring at a broken pipe line, the first instinct for data engineer is to start coding immediately. And that is exactly what we did. We We spent weeks testing different uh methods, obvious methods, and each one made sense in isolation, but it also
[08:56] failed us uh when we run it in production at scale. Our first attempt to solving this problem was using an AI-generated uh rules for regex. We tried to use LLM to write all the different various regex uh regexes so
[09:13] that we can try to solve the problem uh at scale. And at first, it might sound like it's a really clever idea, but that was a trap, because what ended up happening was it became regex explosion, which means that we were
[09:29] spending more time maintaining the regexes versus uh improving and tagging the actual data that really matters. And this is a very critical part to our process because our analyst, our business require all these data sets to
[09:45] be tagged as accurately as possible because it fuels and power all the products that we have at Yodlee data. So, if you look at some of the examples where regex worked really well, Salesforce example there,
[10:01] that description is very very clean. It's It's something that will work perfectly every single time. Now, if you look at the other example with Microsoft where the O is clearly missing, that will break your regex
[10:16] implementation because it never considered that, hey, you can misspell Microsoft by missing an O. So, that These are some of the pain points that we have run into that we're trying to solve for. And that led us to, you know, a different path, which is
[10:33] what about we just um do, you know, zero-shot prompting? What do What don't we just give the model the information and just let it figure it out? The problem with this, uh which is the method two, where we call direct
[10:48] inference, this is where we ran into a lot of hallucination problems. A good example here, if you if you see the first example with Chase CC processing fee, the model assume uh that Chase here represents Chase Bank.
[11:05] And that's not what it represents. When you When you and I see it, we can see that Chase is most likely refer to a person's name and the CC is a credit card processing fee that's being reimbursed for. So, the model saw the keywords and it
[11:22] made an assumption and the assumption is wrong. Another example is the uh Go GitHub Copilot at 200 seats. That for for this one, the model is tagging it as GitHub, which you can make
[11:37] the argument that that is correct. But GitHub is actually a product, and what we wanted to do is tag it to the entity level. So, the right answer here is uh should be Microsoft. So, these are like examples of where model can perform really well. It it understands um
[11:54] you know, some of the problems that we're trying to solve, but it doesn't have enough understanding of our corporate hierarchy in order to tag it to the way that we want it to be. So, we pivoted to uh experimentation number three, which is using a search
[12:11] database. And we started with lexical search uh because it's the simplest uh search that we can implement. And this is our attempt to strip away the AI's reasoning and went back to the simple uh literal string matching.
[12:28] And by doing so, you know, to be fair, lexical search isn't useless. It It does help. It handles um a lot of the simple typos. It handles the mis- slightly variation of the names. It might handle some of the misspelling.
[12:43] And that is something that the regex was very hard uh to get it right because you have to custom uh write these regex to handle those edge cases.
[13:00] Lexical search also ran into a lot of other issues, and these issues are related to um not being able to understand that the context of of um the words and how it uh how it maps back to a specific entity. So, if you look here for BigQuery, BigQuery here,
[13:15] um the model is not able to tag it to anything. It It doesn't know that it should be tagged to Google because there's no information out there. S3, it sees S3, but it doesn't know that it has a relationship to AWS. And this is a problem that um that we've seen a
[13:33] lot when we're using lexical search. It was able to solve the hallucination problem, but it created a massive recall problem for us, and that's a challenge for um for us as a uh data provider. And that led us to a semantic search
[13:48] capability. We stopped trying to match strings and started trying to match the concept using vector embeddings. The goal here was to see if the uh the meaning of different words grouped together, if that can be good enough to
[14:04] map back to a natural entity that we care about for that transaction. And this is where we actually had a lot of success. The transaction AWS EDP uh infra prod,
[14:21] it was able to tag that and retrieve it as Amazon because there were enough um semantic uh meaning where Amazon was the right candidate to choose from. And then we have the product mapping example where chat GPT subscription,
[14:38] chat GPT is a product we wanted to tag it to openAI because our database has openAI as an entity with the product of chat GPT associated with it is was able to retrieve that and highly correlate that as the right candidate to tag to.
[14:59] And lastly, if you look at customer 360 platform, um each word is generic, but together it represent a specific meaning. So, the embedding model actually knows the combination of of those words and is able to associate it with Salesforce. So, we thought we finally figured it
[15:16] out. Everything works. We have no problems. And that's wrong because semantic search might be able to handle a lot of the uh previous edge cases. Where it did fail was not understanding
[15:31] the um the business context or the business rules that we want to apply to our data set. An example here is for PayPal GPT 1 month subscription. Um this data will be tagged to uh PayPal, which is wrong because PayPal
[15:48] is actually differential uh payment. We actually want to tag this to the actual company that's receiving the biggest spend. And in that case, it it should have been OpenAI. The second problem that we ran into it was name collision. Uh and Unity is a good example here
[16:06] because how do you know if they should be tagged to Unity software or should it be tagged to Unity Bank? Both have the word Unity in there and there's no way that semantic search has the reasoning or the knowledge to understand then what you want to do.
[16:28] So once we experimented with all these uh different techniques, we had a an aha moment where we realized that the search alone uh was good, but it wasn't good enough. And we need some reasoning in order to help improve the accuracy for
[16:43] what we care about and what we wanted to do. But the question is how do you force the LLM to uh understand and reason correctly? And how do you stop the LLM from hallucinating? And the way that we approach that is to feed the model with a lot of hints, a
[17:00] lot of evidence so that it has the right knowledge to choose from. And that's where we have a three-step process uh that we're running today. So step one brings us to uh what we call entity extraction. As you can see from all these different
[17:16] descriptions, there are a lot of words that are potentially um what we consider noisy. So, the concept of entity extraction is to extract the keywords that it knows where it might be associated to a product or a company. And that's going
[17:33] to be very important for our next steps because that is what we're going to feed in into the retrieval process. Now that we have the potential entities that uh was extracted from the raw description, we now have to decide
[17:51] do we use the lexical search or do we use semantic search? Well, why not both? And that's where we we started to use experiment with the high hybrid retrieval process. And this is like the best of both worlds. Lexical search is able to handle, you
[18:06] know, the exact word. It's able to handle some misspelling. It's really good at at a lot of those examples. And then semantic search is really helpful where it's able to catch the intent of like the meaning behind the words and it associated with an actual
[18:22] uh an actual entity that we actually care about. So, when you combine both of them together, you actually you actually build a retrieval system that is significantly more robust than uh either one of them alone. Once we have this, we
[18:39] put all the uh keywords into a search engine uh against a taxonomies of entities that we already indexed in our database, and we pull a list of five to 10 candidates that we're going to feed into the LM uh in like the next steps.
[19:01] So, we've done extraction where we are removing a lot of the noise and we're potentially trying to figure out are there products, are there companies, and what are like the keywords that we actually care about that can help us uh get the results that we want. In step two, we decided that once we had those entities, we're going to retrieve
[19:18] it against the list of databases of known companies that we care about so that we can map it to a transaction and so we can map the transaction to an entities um that uh that we like and that we care. And that led us to the final step, which
[19:34] is step three, where we call structural reasoning. This is the reasoning aspect where we leveraging the LLM the model to figure out how do we actually tag it to the business rules that we care about and how do we give it enough context so it
[19:49] doesn't hallucinate. And I think this is where most team gets it wrong because most most team teams that I have worked with will naturally just throw everything into an LLM, what we did what we call zero-shot prompting, and ask it to guess or, you know, identify what
[20:06] company it should tag to. For us, we stopped doing that. And what we did was we fed the list of candidates that were retrieved in step two and then we made it a multiple choice problem for the LLM. So, what we have done there is we
[20:21] eliminated the possibility of hallucination because in order for the LLM to choose and tag an entity, it has to be a list that was already defined and provided to it. So, when you constrain the universe of potential answers that the LLM has, it
[20:38] actually helps reduce hallucination and you increase the possibility that the tags that it's able to choose is highly correlated to what the transaction is referencing. So, I know I know, you know, uh the crowds are probably thinking about how
[20:54] do we what is the architecture? How's it work? How do we ensure that uh this is running reliably? So, why don't you just walk me through that whole process? Yeah. Awesome. Thank you, Ed. And so, you know, Ed really did a good job of laying out the playbook, right? It's a three-step process. We need to be able to do entity
[21:10] extraction. We need to be able to do a search-based architecture on all the rows of data that we have, and then we need to pass it into a reasoning model to basically make that final decision. So, how do we do this at scale given the volume and variety of data that we have?
[21:25] Ultimately comes down to doing orchestration the right way. And so, I think something that's kind of like less common in the AI ecosystem today is how to do batch inference, and that was at the core of front that we had to solve. A lot of a lot of the AI stack right now is very optimized around chat-based experiences. So, you have APIs, you have
[21:42] MCPs, uh you have like chat-based applications, and everything is about real-time inference. What we wanted to do is spin up a workflow, have it process, you know, thousands of rows at a time and pass an LLM on a row-by-row basis and get an answer. And that really required us to build in a very different
[21:58] way than uh what you would typically see uh across LLM vendors. So, the nice thing is we were because we're kind of already a big uh Databricks customer, we have a lot of this uh a lot of our data already in Unity Catalog and in the Lakehouse, there's plenty of building blocks for us to get started. So, as we
[22:15] architected this data flow, Databricks workflows were a key part of this. Uh really solved the orchestration challenge of being able to spin up some compute, have it run for a long period of time. You have retry semantics, uh you you're able to get good visibility into those workflow runs, and so it was a great like building block to start on.
[22:31] We had to use some couple of other tools as well from Databricks. So, one of the things as far as this data flow that we had to do and get right at the beginning is make sure we're incrementally processing our source data. So, we used the Cloud Files connector to kind of stream data into the core workflow where we do this enrichment. The reason why we
[22:48] want to do incremental processing is LLMs are expensive, you have to be careful with it. We don't want to reprocess tagging a transaction to an entity once we've already discovered it. So, really had to adopt a streaming-first approach to all of this to make sure we were uh you know
[23:04] only processing all the data you know exactly once. And so through that you know we had a structured streaming process and the other key kind of unlock for us was using Spark UDTFs. If you haven't used UDTFs they're kind of like a class-based transformation paradigm that's in Spark.
[23:19] I think it's gotten a lot of upgrades in Spark 4.0. And so you can define transformation just as like a Python function inside that class and that's something that operates on a row-by-row basis. Why is that important for us? Well like I said the AI ecosystem is still pretty new. Not all of the LLM
[23:35] operations that we need are built-in Spark functions as well as the search capabilities. We need to be able to run vector search on each row of our database or sorry each row of our source data set. And so that required us to kind of jump into like Python functions to express those transformations.
[23:51] UDTs are not the most performant so this is where UDTFs kind of shine. It allows you to operate that UDF on a batch of rows which very much tied to this architecture and very reliable. And the other thing UDTFs gave us is just the ability to kind of do some interesting things with
[24:07] preparing some computation before the batch starts and before the and after the batch ends. And we use that to basically apply threading. With all these LLM and search operations it's very IO heavy so when you're just if you're don't do something like that you're effectively doing a for loop and
[24:24] that will take forever to process. But we were able to really get a lot more throughput this way by being able to use concurrency queues threading in this kind of approach. And so with UDTFs being our little escape hatch to be able to connect to the AI ecosystem we're able to connect
[24:39] to our search index we're able to connect to different LLM models. The other thing with this data flow that was very important is for each one of those steps we checkpoint that data as a delta table. So you kind of think of it as we stream data in in that structured stream step one we do entity extraction.
[24:55] We were using Agent Bricks for that so their information extraction service really gave us a lot of easy ways to stream that operation. We store those extracted entities as a Delta table. Then, we stream from that table and start doing the search index to find all the candidates for each row of that data
[25:11] set. Store that as a Delta table and then stream from there. And that's where we do the final reasoning loop. Why is this important? It may seem like why don't you do it all at once? Well, if the process breaks down, you have to start kind of from scratch. And you lose a lot of traceability of what's going on here. This is a pretty complex
[25:26] operation, but if you can store each step of that operation in separate tables or separate features in a table, you get uh you get an easy way like if the process fails or stops for some reason to restart from that point onwards. And also be able to trace and debug what went wrong if something does go wrong. So,
[25:42] all in all, what was awesome about this is the big takeaway is we weren't, you know, investing in complex infrastructure to get all of this working. These are all building blocks that, you know, on the Databricks ecosystem you have today. So, for us being already on Unity and the Lake House, it made this uh very actionable.
[26:00] The next big piece of this architecture that I wanted to zoom into is really like the batch inference piece. This is probably the hardest part to like actually execute on because, again, LLMs have been very optimized for the chat real-time based workflow. Like, how do you do batch inference and get it right? Um the good news is there's a menu of options. The bad news is there's a
[26:16] trade-off between cost and complexity uh and how you want to approach that. So, for us, you know, what we kind of learned is using Agent Bricks and information extraction was a great starting point. It's a no knobs, no you know, just turnkey operation. It's built-in SQL function, so you can just
[26:32] call select AI extract, pass in the prompt and extraction schema that you want, and then you get the output on your data frame. Super easy to get started. And what we like is it helped us prove the value early on of like can agents really help answer and clean up
[26:48] our data set without having to invest a lot in our infrastructure. For the information extraction step, like this was perfect. The step one, like I think this checked the box and we continue to use it to today. For the following steps where we had to do, you know, the search and the agent reasoning
[27:03] uh capability uh or you know, the agent reasoning and selection, we kind of had to scale beyond it um and so what we started to use is AI runtime or serverless GPUs. Um this was nice because what it unlocked for us is model flexibility. So, for the agent reasoning
[27:19] step, for example, we've seen a lot of success with GPT-OSS. We're using 120B uh and we find that that's a good balance of price and performance. Uh but we needed to fit that model onto GPU. Kind of need a large GPU for it, so it would be tough to kind of reserve and get that just directly from a cloud
[27:35] provider, but with serverless GPUs, we can spin that up in a few minutes, you have access to it. You have a pre-built environment, you can install additional Python packages if you want. Um and then you have the model itself and then you kind of we implemented a logic to just kind of do the batch for that reasoning step inside the
[27:52] serverless GPU and then put it back in a Delta table. And that worked really well for the incremental workflows. Uh the one exception is we have a lot of historical data. We had to backfill everything once we rolled out this new process. And to kind of handle that scale, which was, you know, millions of rows, uh
[28:09] serverless GPUs weren't necessarily the best selection because we have a long-running process. It may have take several hours to several days to do that backfill. And with serverless GPUs, you're not necessarily reserving that GPU yourself, you know, it's being contested, it's being shared by other customers or other jobs even within your
[28:26] Databricks workspace. And so that led to some performance constraints. So, we did finally adopt kind of self-hosting GPUs. Um and so for that, you know, we kind of reserve those instances through our cloud provider or spin them up in Databricks. Benefits is you get full access to the
[28:42] hardware. You're able to do anything you want on on that GPU instance, so you get a lot of control to be able to utilize the CPU and the memory, um package flexibility, environment flexibility. The trade-off is it's really complicated to reserve GPUs. They're very contentious.
[28:57] Everyone's trying to get access to them. You have to reserve them in certain blocks of time, which was a certain a learning for us. And so you need to be really planned about when you're doing those batch workloads because you're kind of blocking out that GPU for a certain amount of time, and it definitely costs money. Uh for us, you know, the cost and the trade-off and the ROI was worth it, but
[29:14] we had to be very intentional about when we want to spin up those self-hosted GPUs. And so all all in all, I'll say like there are options to do this, but you should kind of adopt the complexity as you need it. Start simple. Um and then, you know, in fact, in terms of I'd say like, you know, in most cases
[29:30] and most data volumes, yeah, probably the information extraction using AI query is going to be enough. Um only jump into the GPUs as you need that kind of scale and throughput. So, the last big piece of the architecture I really want to call out is just our search infrastructure and
[29:46] what we did there. Uh we talked about a little bit, but we actually had to develop an entire ontology to figure out what we wanted to put in the search index. It's not a simple list of just the company names, but it's also a bunch of attributes about the companies. So, we built kind of like a relational data model. We put
[30:01] in a Postgres database of entities and attributes about those entities. So, the entities might be companies like Microsoft or Electronic Arts, but then we have a bunch of attributes about them. So, we have like their name, their website, uh potentially their ticker if they're publicly traded, different
[30:18] product names. All of those are attributes about the entities, and they get put into the search index as well. So, we are taking subsets of this ontology and creating vectors on top of it to power the semantic search. Um we use Databricks Lakehouse for this. Uh it's great solution. It It's managed Postgres made this really a turnkey
[30:34] operation for us, and also made it really easy to query the Postgres instance in our Databricks workflows. The other knock-on benefit of Lakehouse is we were able to actually put our vector embeddings directly in Postgres. So, PG vectors a great plugin so you can just automatically turn any other data
[30:50] in your Postgres instance into vectors and we didn't have to worry about orchestrating it, putting that data somewhere else outside of that Postgres instance. And we also had a lot of control over how we want to do the search logic. So, with PG vector you can write semantic search kind of operations in SQL, but you can also combine it with
[31:06] just traditional SQL operations. So, we can get a lot of flexibility over how we want to do our hybrid search on that database. Uh so, the other knock on benefit I'll say about this ontology is well, we have all the vectors, we can do that search, build the candidates of entities to
[31:22] basically pass on to that model to do the attribution. Uh once we do that attribution, we can then join this ontology back into our output data set and we get all the other attributes for free. So, again, our goal was to get properly enriched data sets uh to know
[31:37] what company is attributed to what transaction, but then there's a lot of follow on attributes about maybe the categories they operate in. What are their competitors? What's the description? What's the website? All of that stuff because we've mapped it in our ontology kind of comes for free after we do that kind of classification. So, there's like a dual purpose to this
[31:53] ontology which has been a really kind of like uh important piece of the system. So, lessons learned to kind of like wrap up here. Um start small. Don't invest in the complexity in the infrastructure until you need it. If you're going down this route, you really need to prove out
[32:09] that this agentic approach makes sense for the data problems that you have. Uh in many cases it might be fine like regex and NLP is fine. We still have that, but there are certain data sets that are really ambiguous and hard to classify and that's where I think this approach has merit. Uh but you need to prove that out on a smaller set of data.
[32:26] That's kind of the second learning that we had is like you really need to build a feedback loop. So, invest in setting up an eval data set, have people review those results. We did a lot of iterations on our evals. We still do a lot. Um that's the only way you know if these agents are going to be giving you better results or not.
[32:42] And finally, you kind of really need to build you need to pick technologies that you can build on. This is a really, you know, emerging ecosystem and space and technology. Choice is everything, flexibility is everything. For us, we needed to try different search indices, we needed to try different models, we
[32:58] needed to try like different orchestration patterns. So, using components like Lake Base gives a lot of flexibility, using Databricks workflows, UDTFs kind of gave us just like the best of all worlds where you don't have to worry too much about the low-level infrastructure and we could spend a lot of time on the business logic without
[33:14] feeling trapped in one ecosystem or one provider. So, that really gave us a lot of ability to experiment and see what makes sense. Um the last thing I'll call out is Databricks apps. This was something we kind of learned as well on the technologies that we really liked. Um
[33:29] it's a complex process like I mentioned. Visualization is everything because you need to give your teams trust that they that the system is working. And so, I think the data engineers at our company have built a a really great visualization tool to help people understand, you know, what what is the
[33:45] data set, what where did it start, what were the search candidates, what did the model pick, and like have a really good way of visualizing that data. And that just helped really kind of pull everyone in and kind of get confidence in the system cuz, you know, LLMs they are unpredictable. You need to really build a lot of structure around them.
[34:00] So, where did this kind of get us and like what was the final impact? Where we started before at the beginning of this journey is a really kind of set of brittle brittle pipelines. There were tons of regex and conditional based rules to process our data. It's really slow to onboard companies.
[34:16] It would take us weeks or months to properly tag a new company in our data set because of how complex these rules were. And there's a long tail of degradation. There's a lot of edge cases that we could never figure out just with these techniques alone. Um and as our data set grew and new data came in, the same rules that we thought were working
[34:33] wouldn't work and it would turn into kind of a treadmill of us trying to fix these rules over time and patch it. Other teams would start creating silos in their own versions of this data set and they'd make their own corrections and it was just it was just not a good place to be in. Fast forward to now, I think what we've started to emerge on is really an
[34:49] adaptive enrichment kind of platform where we have some fundamental building blocks. We have a initial pipeline that does the entity extraction. We have an anthology and a search index that basically powers the candidates of rows that an agent can then make a decision
[35:05] on of like what vendor does this transaction belong to. And the nice thing is these pieces don't really change depending like we can slot this in to a different data set and kind of take the same strategy. It's very flexible and reusable. Uh and so what our data engineers are doing is much less time intervening and
[35:21] kind of, you know, playing firefighting on a pipeline and instead they're just growing our anthology, adding more search terms to it. So as we onboard more data sets or source data, uh those companies are automatically tagged. And so we've gone, you know, from uh weeks to months to kind of adding companies to hours because it's just
[35:37] about getting the right metadata into the anthology and then reprocessing any of the data that we haven't tagged yet. To put some numbers into it, uh we have seen about like a 33-time increase in the coverage entities in our database uh
[35:53] or our source data set. So before we had maybe about 3,000 companies tagged, now we have about over 100,000 and it's constantly growing. Uh we have a repeatable playbook so we can take this strategy and apply to different data sets and we're already doing that. In terms of accuracy, uh through, you know, using this multiple-choice type
[36:09] approach, we're really seeing about 92% uh today, which is pretty solid. Uh I think we're going to get better on this as the models improve and as our anthology grows and we have more attributes that we can link source data onto it. And then from an operational perspective, our data engineers, they're
[36:26] spending like less of their time firefighting pipelines. Uh Uh they're going shifting from maintenance to more feature development, working on new data sets, new like areas of our stack where they can be enriching data sets. And that's just like a better business value proposition. And so like ultimately like
[36:41] what we have kind of trans transformed into is going from a system of rules that just decay over time to a system that adaptively learns, that we can take the best of these models and have them, you know, apply the reasoning on our data sets at scale,
[36:56] uh but ground them in a lot of source data sets that we have or ontology that we've kind of built a lot of trust on. And that's really provided a lot of value for us as an organization. Um really excited for what this is enabling for us because now we can speak to so many more companies. It's just still early days for us. I think we're going
[37:12] to be expanding this across many more data sets that we have. And that just means, you know, more products, better answers for our customers, and really just turning into a data provider, operating at a scale that I think was unimaginable before the world of LLMs. So, really excited about this. Hope you took away some interesting findings and,
[37:29] you know, reach out if you have any questions. Thank you all.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.