Stop Guessing, Start Scoring: LLM Evaluation for Production Extraction

Summary

CliftonLarsonAllen built a Databricks-native LLM evaluation system using MLflow, custom scorers, and Databricks Apps for SOC audit document extraction, replacing a manual process that took up to 10 hours per report.
The evaluation architecture uses section-level scoring with confidence-based monitoring guided by LLM judges rather than a single aggregate metric, enabling auditors to trust extraction results and identify quality degradation over time.
A feedback-driven loop captures auditor corrections through a Databricks Apps review interface and feeds them back into evaluation datasets, creating a continuously improving extraction system that balances cost, accuracy, and speed at scale.

Stop Guessing, Start Scoring: LLM Evaluation for Production Extraction

Robust evaluation is critical for LLM-based systems in production. At CliftonLarsonAllen, a leading accounting firm, millions of SOC audit documents require precise data extraction. The challenge: how do you know if your LLM extraction pipeline works well, scales reliably, and maintains quality over time?
this video deconstructs a Databricks-native evaluation solution using MLflow, custom scorers, and Databricks Apps for structured feedback capture. Learn how to design evaluation from the beginning, measure quality with section-level scoring instead of single metrics, use confidence-based monitoring to guide human review, and establish feedback-driven evaluation loops that directly improve your agent. You'll see how to handle complex structured outputs, align predicted and reference data, and balance cost, accuracy, and speed at scale.
🤝

Chapters

FAQs

What problem does CliftonLarsonAllen's SOC document extraction system solve?

SOC audit reports are 50–100 pages of unstructured text that auditors previously reviewed manually, taking up to 10 hours per report with resulting inconsistencies, missed fields, and variability between reviewers. The LLM-based extraction system on Databricks automates this process and provides confidence scores to guide the human review that remains part of the workflow.

How does CliftonLarsonAllen evaluate the quality of their LLM extraction pipeline?

They use custom scorers in MLflow that evaluate extraction quality at the section level rather than as a single aggregate metric, revealing exactly which document sections the model handles reliably versus where it struggles. Confidence-based monitoring with LLM judges flags low-confidence extractions for prioritized human review.

What is a feedback-driven evaluation loop and how does CliftonLarsonAllen implement it?

A feedback-driven evaluation loop captures corrections that auditors make through the Databricks Apps review interface and feeds those corrections back into the evaluation dataset used to measure and improve the extraction model. This creates a virtuous cycle where production use continuously improves model quality without requiring separate labeling efforts.

Why is section-level scoring better than a single accuracy metric for LLM evaluation?

A single aggregate accuracy metric can mask the fact that a model performs well on straightforward sections while failing on complex ones such as subservice organizations or user entity controls in SOC reports. Section-level scoring reveals exactly where the model struggles, enabling targeted improvement and giving auditors confidence about which extracted fields to scrutinize.

Full transcript

[00:09] All right, let's jump into it. So, if you've ever worked with AI, you've probably had a moment like this, where you've built and deployed an AI agent or an AI solution, and then you find yourself asking the question, "How do I know if this actually works?" Hello, everyone. I'm Darshana from CLOUDE and this is Michelle from
[00:25] Databricks and we're excited to share how we solve this problem. At CLOUDE, every audit engagement produces tons of complex reports. Now, multiply that by every client, every engagement, every audit cycle. You've spent millions of hours collectively
[00:41] extracting data from these complex unstructured reports by hand. LLMs can obviously help with the data extraction or document AI. But, how do you trust that process? How do you verify and measure quality when an LLM is extracting tons of data at
[00:57] scale? Today, we're going to share how we stopped guessing whether our extraction pipeline is doing a good job, and instead, we started scoring, measuring, and continuously improving.
[01:15] Okay, quick note on the agenda. First, I'm going to talk about the business context. I'm going to share what our problem statement is and how we tackled it. Then, I'll jump into a quick application demo. Post that, I'll hand it off to Michelle for technical architecture and our evaluation
[01:30] strategy. And finally, we'll close out with takeaways and any questions. Okay, business context. For the focus of this discussion, we're going to be focusing on SOC reports or
[01:45] system and organization control reports. These are part of the audit process and they're created by service organizations to document the controls they have in place in their operating environment. Our audit teams use it to ensure their clients' vendors have the right
[02:00] safeguards in place. These reports are typically 50 to 100 pages long, and they consist of several unstructured sections like control objectives, subservice organizations, user entity controls. And the format varies a lot
[02:16] based on the auditing firm creating these reports. So, in the older process, auditors went through all these 100 pages one by one pulling out different metrics and fields, which took them up to 10 hours per report.
[02:32] When you do something manual like this at that scale, you end up with missed fields, missed extractions, typos, um extraction varying person to person, and that eventually leads to an incomplete work paper.
[02:50] So, to solve that, we built the SOC data extraction workflow. It essentially takes in an unstructured SOC document, uses an LLM-based solution, and then converts that into a structured format. There are three things here that I want
[03:06] to call out. The first one is speed. A process that used to take over 10 hours per report now completes in less than 10 minutes. The second thing is auditor-centric review. We don't want to replace the professional judgment, we want to
[03:23] accelerate it. So, your input SOC document is displayed side by side to your extracted data, which helps the auditor kind of look at both alongside each other and complete their verification process faster. The third piece is the confidence
[03:39] scoring. Each extracted value has a confidence score associated with it. Michelle's going to talk to us in a bit about the actual evaluation strategy behind that. The guy The goal here is we want to guide the auditor to the fields that
[03:54] require their focus. So, a field that came back with high confidence is something that the auditor could quickly review and move on. If something came back with low confidence, the auditor should ideally spend more time reviewing and editing that.
[04:10] We also have a golden truth set that we validate our results against that helps us measure F1 and accuracy across time. Another design decision that I really want to talk about is we did not build a
[04:25] black box workflow. This is a human in the loop workflow. So, step one there, the agent extracts the data elements from an unstructured PDF. Step two is critical. The auditor is in control.
[04:40] The auditor reviews the extracted content from the agent and then performs corrections if there are any. Those corrections are captured as feedback and that feedback is then fed back into the agent. So, this builds a
[04:56] continuous improvement loop. Every review matters and capturing these reviews over time only help improve that agent. Okay, with that said, let's jump on. Words are great, demos are better. Just give me
[05:12] one sec while I set up.
[05:42] Okay, so this is the landing page to our SOC data extraction workflow. This is where auditors land and they can either start a new extraction or resume an older extraction. You'll see a bunch of historical extractions there with their status, the last modified timestamp, and the user that ran through
[05:57] that extraction. Let's go ahead and start a new extraction. You have to put in an extraction name and then you can upload a single file or a batch of files. In this case, we're just going to upload
[06:13] a single report, which is the SOC report that we talked about. When you click on extract files, what's happening here is Sorry. Apologies.
[06:33] Okay, I don't know how to bring that back. All right. This is unfortunately the way the play is, it's at the bottom, so I'm going to have to tackle that. We'll get through it together.
[06:52] So, jumping back to that, we were uploading a file. This is a single SOC report that we're uploading and we're going to have the agent trigger to extract that file. What's happening behind the scenes is the agent is essentially going through the classification process, extraction process, and the scoring process. You'll
[07:09] see the job that I submitted is now in queued and it's processing. Michelle's going to talk about what's happening behind the scenes in the technical architecture. Once that job completes, we should be able to expand that up and look at the status,
[07:26] which files were processed, as well as what the data type was, or how it was classified. In this case, we're focusing on a SOC file, so it was a SOC file. You can also drill down into details. So, in this case, I drill down and I reach the the review page. This is where
[07:42] the auditors spend most of their time. You'll see the elements up top. That's what the agent has extracted and presented in this format. So, on the left there, you have the name of the service organization. In the middle is the auditing time period, and to the right is the auditing firm.
[07:58] You'll also notice the little green and red icons with the confidence scores. If something is green, for example, the one to the right, the auditing firm, the auditor looks at the 100% that built some confidence. The auditor can quickly review and move on.
[08:15] If something doesn't look right, like that 50% to the left, the auditor has the capability to go in and edit that value. So, in this case, the auditor is just adding an additional word there to ensure that's corrected. We want to capture each of those
[08:30] corrections to build improvements into the agent. You'll notice now that value says the field's been edited, which is a nice way to see that someone has reviewed this process.
[08:47] We also store our edit history as an audit trail in Lake Base. This is important for compliance teams to in explainability and traceability, specifically in an audit context.
[09:03] Okay, let's scroll down to the bottom half of the application. These are all the sections I talked about, so sections two to eight, and these are essentially the different parts of a SOC report. You'll notice how each section is really different. It looks at different fields,
[09:18] different number of columns, and it's not consistent. On the right, you have your actual SOC report that was uploaded. So, this view presents the auditor with an easy comparison to ensure the extracted data and the input data go side by side.
[09:50] Okay. Now to display the confidence scores more, I'm going to jump into section six, which is the control objective section. Control objectives are essentially noted here along with exceptions noted. We're going to focus on that first one, the management oversight and governance.
[10:06] You'll see it has a confidence score of 50%. Now watch what happens when I click onto that record. The PDF automatically jumps to page 39, which is where that particular control objective is detailed. This makes it
[10:24] super helpful for the auditor to view where the extracted data is coming from along with citations instead of having to go through each of those 100 pages, and that really speeds up their review process. They can go in, compare the data that's
[10:39] on the PDF, pull out the facts that are required, and then edit the field if needed.
[10:57] Okay. Another feature I want to talk about is the export feature. So, we essentially have a feature that enables us to export the content that the agent spit out into an Excel file. I guess no audit process is complete without somebody opening up a
[11:12] spreadsheet. Okay. Take a look at this Excel file. It's really complicated. You'll look at all the sections. Everything has different number of columns. It's color-coded. It's got check boxes. It's
[11:27] got collapsible sections. I guess the point we're trying to drive here we need to ensure we have the right value in the right place in the right format. And just eyeballing an LLM's responses is not good enough. We need a
[11:43] scoring system and we need a scoring system that works over time. The last view that I wanted to show was the system monitor. This is the ops view behind everything
[11:59] else that you saw in the demo. I want to take this back to the title of our presentation. Stop guessing, start scoring. This allows our AI and ML teams to stop guessing. They don't have to guess what the
[12:15] latency is, what the confidence looks like, what the number of tokens or costs look like. It's all here in one dashboard to view. You can also drill down into the jobs to view trace IDs to investigate if something's going on.
[12:33] I guess to close the loop on this, I'd like to say confidence scoring and human edits build confidence for your users and your practitioners. An ops dashboard like this and ground truth validation build confidence for your AI and ML teams. We need a combination of both of these
[12:50] to ensure that something that's a prototype can become into something in production with trust. With that, I will hand it off to Michelle to go over the technical architecture, which is the more exciting
[13:07] part. Not not at all. Not at all. Thank you, Darshana. Really appreciate the intro. So, now that we've seen what the user experience is like, I'd like to walk you through what our architecture is doing behind the scenes. The system we've designed is built
[13:23] entirely on Databricks. The UI, the orchestrator, the SOC processing workflow, and analytics layers all leverage Databricks tooling. The UI is hosted as a Databricks application, which you can see here on
[13:39] the far left. The UI submits extractions to our orchestrator, which is another Databricks application. The orchestrator manages extraction requests and intelligently de-queues tasks based on available tokens.
[13:55] That triggers the SOC workflow. The SOC workflow runs as a LakeFlow job that performs OCR, classification, extraction, and confidence scoring. The UI, the orchestrator, and SOC
[14:11] LakeFlow job all use LakeBase as a persistence layer. It's used to store the task queue, extracted fields, and reviewer edits. I'd also like to note that the SOC LakeFlow job is linked to a Databricks
[14:27] managed MLflow experiment. That manages traces and evaluation metrics for us, granting us the great observability metrics that we've been able to create, and that Darshana shared at the end of her demo, that operations dashboard we have.
[14:47] Finally, we've also set up a LakeFlow Spark declarative pipeline, which syncs data to our analytics layer from LakeBase nightly. The entire system uses Databricks tooling end-to-end, and that makes integration and support so much simpler.
[15:04] Now that we've walked through the overall architecture, I'd like to zoom in on our evaluation setup. Often the hardest part isn't getting the agent or system to generate an answer, we all know that. It's knowing whether that answer can be trusted. So, let's say our system extracts 1,000 SOC documents a day.
[15:20] Do we really want auditors to go and review 10,000 plus fields with the same amount of attention? And even if we want that, are they going to be able to? Do they have that much time in the day? And do they have that much attention to pay? Or, let's say it's 6 months from now.
[15:37] How do we prove that our agent and our system is performing as well as it was when it was deployed? Our evaluation design focuses on answering two broad questions. First, what should the reviewer pay close attention to right now?
[15:53] We answer this question with confidence-based monitoring, that box at the top of the diagram. That is powered by an LLM judge, which is used to identify fields or sections that need a closer review. So, those confidence you saw you saw in the beginning of Darshana's demo.
[16:10] The second question we needed to answer is, how do we know the system is performing well over time? We answer that question with feedback-driven evaluation, that box at the bottom of the diagram. Once auditors correct outputs in the review UI, those corrections become
[16:27] evaluation data. We then compare that with the original model output to calculate and track metrics like precision, recall, and F1. So, let's take a deeper dive into our confidence-based monitoring setup.
[16:44] This system allows us to satisfy the first mandate from our product organization of giving the audit team a little bit of additional help in knowing what to pay attention to when they're reviewing the extracted output. It allows us to provide the same signal as well to our AI teams responsible for
[17:02] monitoring and maintaining the SOC workflow. For auditors, again, the confidence scores provide review guidance. So, here you can see the service organization name received a very low confidence score of 0.14. That indicates to the
[17:17] auditor that they really need to pay special attention to that field, verify it in the document, and go into the exact location, and make sure that name is indeed correct. For the AI team, for Darshana's team, the same signal is aggregated across documents. Confidence scores become an operational
[17:34] monitoring signal available in the ops dashboard that gives the AI team a real-time signal as to how the system is performing without any additional effort. The key point here is that confidence scores don't require ground truth at
[17:49] all, and therefore can immediately provide value, not just to the auditors, but also to the AI team. However, confidence scores need to be treated as a triage signal. They don't prove that something is 100% correct or incorrect. We still need SME
[18:05] verification, and we still need ground truth. That's where feedback-driven evaluation comes in. And that starts, again, with the human review workflow. On the left, auditors are reviewing extracted fields in the UI and making corrections. In this example, the model
[18:22] extracted variations like CCC or ABC hyphenated management services. And reviews are corrected to the complete, fully spelled-out organization names by our audit team. In the middle, those corrections become a metric signal.
[18:38] If auditors keep correcting the same field, or if section 1F1 drops over time, that tells us where the workflow is struggling. And on the right, that gives the AI team something specific to take action on. Maybe we need to clarify entity
[18:54] boundaries in the prompt, maybe we need to normalize logic for abbreviations, or maybe it's an auditor education problem we need to take to take back to the team and understand why they're making those corrections. We can We can make those corrections and make the improvements, deploy to production, and then, on the next run,
[19:12] we can measure whether the change was actually helpful. So, that's the loop. Reviewer corrections become evaluation data, and then that is passed to our AI teams, so they can make corrections and push those into production, and hopefully on the next loop, we're able to see the benefits of of those of those
[19:29] extractions. Of those corrections. So, before we get into our scoring solution, I want to break down why offline evaluation is harder for this workflow than it might seem at first. So, for a simple classification task, I
[19:45] think probably many people in this room have done, evaluation is fairly straightforward. You compare your prediction to your ground truth. If it matches, awesome. If it doesn't match, you know you have some work to do. For us, SOC extraction is not just a
[20:02] single label, though. It's structured output with sections, nested fields, and repeated objects. We faced three main challenges when designing our evaluation framework. First, a single score for the whole JSON
[20:18] is just not sufficient. It does not tell us which sections are strong, and it doesn't tell us which sections are failing, and where we need to focus our improvement efforts. Second, some sections return lists of objects that don't have stable IDs. Before we can score the fields, we first
[20:34] have to figure out a way to align the predicted items to the matched review items. Third, we must have a way to handle equivalent answers. Two answers can look different as strings, but still be correct from an auditor's perspective, and we need to be able to adjust our
[20:50] system to amend for that. Taking a closer look at our first challenge, we can see an example here. Here you can see two example sections, one for subservice organizations, and the other for complementary user entity
[21:06] controls. We need to be able to score separately to understand which sections or sections which section or sections need more attention. If we were to score this entire JSON output as one whole object and generate one score, we have no idea
[21:22] where we need to focus our improvements. Second, list items don't come with a reliable key to match on. For sections like subservice organizations on the left or user entity controls on the right, the model extracts a list of objects and the
[21:38] reviewer output also has a list of objects. So, before we can score whether the fields are correct, we first need to align the objects themselves. That means deciding which prediction should be compared to which reference item, usually based on the content itself. Like organization name or the control
[21:55] description. This matters because bad matching creates bad metrics. If we compare the right answer to the wrong reference item, the score is going to look worse than extraction actually was. We're going to lose faith in our scores and then we're going to lose faith in our
[22:10] system as a whole. The third challenge is that correct answers do not always look exactly the same. In the example, the two control descriptions are effectively saying the same thing, but the text is not
[22:25] identical. There may be small differences in punctuation, abbreviations, formatting, or wording. If we use exact string matching, this gets marked wrong even though our auditor might consider it correct. So, the scoring logic has to be flexible
[22:41] enough to adapt to the task at hand. The goal is not rewarding random paraphrasing. The goal is to avoid penalizing differences that do not change meaning. This is especially important when you're collecting a ground truth from SMEs that might change
[22:57] the phrasing themselves or punctuation where your LM would extract the exact verbatim text you want, but maybe it doesn't match your ground truth, and you still need to adjust for that, and you need to be able to count that as correct.
[23:13] This is where custom scores become important. Instead of treating the full JSON as one output to compare and to score, we break the extraction down into smaller facts, section by section and object by object. Then each field can be scored in the way that actually makes sense for that
[23:28] field. Some Some values should use an exact matching strategy, like booleans or fixed labels. Some fields need fuzzy matching. Think of names or short titles. And longer descriptive fields may need semantic comparison by an LLM.
[23:45] For list-based sections, we were also able to align the predicted objects and the reviewed objects, and then score the fields inside those matched pairs. Now we can measure precision, recall, and F1 section by section. And because those scores are logged to a
[24:00] mouth flow, we can compare runs over time, whether a prompt change, a model change, a workflow change improved the extraction, and measure it um run over run with the same data set.
[24:16] So, stepping back from the implementation details, I want to close with the main lessons from this work. These are the points we think apply beyond soccer boards to any team building LLM-based extraction workflows.
[24:33] First, you should design for evaluation from the beginning. Evaluation is so much harder to add after the workflow or system is already built. The schema, the review UI, the model output, and the way you capture corrections all affect whether you can measure quality and whether you can
[24:49] measure quality correctly and well. Second, you should capture quality signals from any existing user workflows you have. In this workflow, auditors already needed to review and correct extracted fields. The key for us was capturing those
[25:05] reviews in a structured way, so we could use it as useful feedback for the AI team without any additional effort from the auditing teams themselves. Third, structured extraction needs structured scoring. When the output is nested JSON with
[25:22] sections, lists, objects, fields, exact match is usually not enough. Oh, sorry. You need scoring logic that understands the shape of the output and can tell you where the system is strong and where the system is weak. Finally, confidence scores should guide
[25:39] review and not replace human judgment. Our confidence scores are useful for both our auditing team and our AI teams, but that doesn't that doesn't replace creating a ground truth set that you can trust. So,
[25:54] to take away from this presentation, um you're not able to if you're not able to measure your output, you're really just guessing and focusing on evaluation throughout your process and from the beginning to the end of your design is
[26:09] very important and will help you have a system that you can trust by the end of by once you reach once you reach production. Awesome. Thank you so much and we are happy to take questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.