Human-in-the-Loop Compliance at Global Scale: Databricks Apps + Lakebase Casework + SME Labeling

Summary

Mastercard's 'PolicyOps' pattern on the Databricks Data and AI platform converts long policy documents into clause-level records with page-aware citations, turning policy interpretation into a versioned, queryable governance layer.
A human-in-the-loop SME review queue built with Databricks Apps—backed by Lakebase for case workflow state management including assignment, status, and approvals—replaces an unscalable Excel-based review process.
The result reduced a months-long review backlog to days while systematically converting implicit expert knowledge into reusable structured labels that improve agentic solution accuracy over time.

Human-in-the-Loop Compliance at Global Scale: Databricks Apps + Lakebase Casework + SME Labeling

In this video, we share a practical pattern we call “PolicyOps” on Databricks: converting long policy documents into cited, versioned, queryable policy units, then operationalizing a human‑in‑the‑loop workflow that captures domain SME judgment as structured labels. We’ll show how to parse and structure large policy PDFs into clause-level records (including page-aware citations), how to extract decision criteria into queryable tables, and how to connect policy interpretation to real operational review. We’ll then demonstrate an SME review queue built with Databricks Apps — designed for high-signal feedback (not just thumbs up/down) — with Lakebase used to manage case workflow state (assignment, status, reviewer actions, approvals). The result is a scalable workflow that keeps humans in control of edge cases while systematically converting implicit SME knowledge into reusable labels and corrections that improve quality over time.
Talk By: Salman Chaudhry, VP AI Engineering , Mastercard ;

Chapters

FAQs

What is PolicyOps and why does it matter for regulated industries?

PolicyOps is a pattern for operationalizing policy documents in agentic AI systems by parsing them into clause-level records, extracting decision criteria into queryable tables, and connecting policy interpretation to real operational review workflows. It is designed to help regulated organizations achieve the determinism, safety, and auditability that compliance officers require.

How does Lakebase support the case workflow in this architecture?

Lakebase serves as the transactional backend managing case workflow state including assignment, status tracking, reviewer actions, and approvals for each policy interpretation decision. An append-only case ledger preserves the full history of reviewer actions, supporting the audit trail requirements of regulated industries.

Why did the Excel-based SME review process fail and what replaced it?

The Excel-based approach could not scale to the volume of policy interpretation cases generated by agentic systems, and it failed to capture high-signal structured feedback from domain experts. Databricks Apps replaced it with a purpose-built review queue that captures nuanced SME judgment rather than simple thumbs-up or thumbs-down ratings.

What business results did the human-in-the-loop PolicyOps approach deliver?

According to the speaker, the system reduced a months-long review backlog to days by automating high-confidence cases and routing only genuine edge cases to human reviewers. Structured labels captured by SMEs are reused to improve agentic solution accuracy over time, creating a systematic feedback loop.

Full transcript

[00:07] Welcome everyone. Thank you so much for for joining the session. Um I never thought policy ops would be such an interesting topic to have this turnout at 5:20 p.m. So, thank you.
[00:28] Okay. Before we dive in, um a quick poll of the room. How many of you are building uh agentic solutions at work? And how many of you are getting positive ROI on those use cases?
[00:45] Okay. And and how many of you think that your solutions are auditor ready? So, if an auditor walks in tomorrow, you can walk them through the deterministic solution outputs coming out of it.
[00:57] Uh the focus of my talk is going to be somewhere between question one and three. Um that how do you go from building POCs uh and
[01:09] governance-based agentic systems to solutions that you can you can have positive ROI and can also have uh can also walk your auditors through that.
[01:21] Before we go into that, um any agentic solution, first thing comes to mind is ROI. And the literature on ROI is is varied. So, you start from MIT's
[01:34] research saying that 95% of the use cases fail to to to reach production. Um then you get the varied kind of responses. McKinsey saying uh 39%, PwC
[01:46] saying 20 out of 74, BCG 5%. Um there is difference in methodology, there's difference in numbers, but one thing is consistent that directionally
[01:59] not many people are getting good ROI from their agentic solutions. Um I was recently at a talk by Tom Davenport, who is a professor at Babson College. Uh he's kind of authority in
[02:12] ROI measurement of AI systems. Uh and he shared his personal research number. He said he pulled about a thousand different executives and his number was around 2% of those people are
[02:26] actually getting positive ROI on their agentic use cases. Um but the good thing that there is that of those 2% the value attributed to the to agentic
[02:38] outcomes is around 22%. So of all the the value being generated 22% value is coming from those 2% early adopters. And that's where I would like
[02:50] to focus that how we think we can get into that 2% group um of of successful uh AI agentic solution implementations.
[03:06] Before I go into this um when we focus on building agentic solutions for regulated industries, there are basically three things that needs to be true.
[03:18] Number one determinism that when you run your solution although the models at core are probabilistic at a at an outcome level,
[03:30] it should provide you deterministic outcome all the time. The other thing is about safety. So whether the solution is in the defined bounds put out by regulators. So your
[03:44] audit trails they are there, your data piping into the solution that is approved. Um data handling is in line with what your uh regulators have put out. And then then the most important part is
[03:58] uh trust in the system. So, obviously you build the solution, you'll pass it on to some users. For those users to have trust in the system, they would require visibility. Visibility as to how those those
[04:11] decisions are being made. Uh what is causing the model to switch from one output to another or the agent to move from one decision to the other side of the decision. Um, so extending from that, there are
[04:25] these four kind of modes of failure. The way gen AI systems are designed, they are plausible but wrong there times. So, you all of you must have seen
[04:38] those examples of you ask your chat GPT or Claude a Claude a question and it will be like, "Yes, this is how it is." But when you push back, they're like, "Ah, it was a fair push back. Let me reconsider that." So, they're plausible
[04:51] when they provide you that answer, but that answer is is wrong. And in terms of measurement, most of the evals are not able to capture that because they are looking at plausibility rather than the
[05:03] accuracy of the solution. The next part is about traces. So, in compound multi-agent systems, when you do uh a model call, you're using a tool, you're making storing
[05:15] something, analyzing, even chain of thought happening within the model, there are traces of that, but usually those traces are locked in the development side of things. So,
[05:27] engineers have access to that. For SMEs, it's a different story. And then when it goes to the SMEs or end users as well, they they have this very
[05:39] limited window of thumbs up and thumbs down. You look at an output, you're like, "I like it or I don't like it." Um, that's good in its own and at a at a at a at a meta level it helps
[05:51] tune your system in a certain direction, but it doesn't provide the output to the system that this is why it's wrong or this thing needs to be fixed. And then the last but the not least and
[06:03] and definitely most painful is the integration burden. When you're building all of these solutions yourself, you are stitching through multiple services for it to become reality and and go into production. And
[06:17] this is where we'll focus on on what we do to fix these solutions. So, the approach we take is that AI prepares, humans decide,
[06:29] traces expose, and then feedback improves. So, this kind of becomes a continuous cycle. So, the initial output is prepared by AI. So, the agentic solution is looking
[06:42] at the out is is looking at the data you have, how you have structured your policy, um, all the related things, but then it passes that information on to the human. And human being is the one
[06:55] who is making that decision. What and and what they need to make that decision accurately is to understand what went in the system when that
[07:07] decision was being made. So, what kind of information the system was relying on to make the that decision. And then the last but not the least part is is feedback improves. So, that SME input is
[07:21] going into improving the system further for for the next iteration. And this is where we introduce policy ops.
[07:33] So, ML ops ship models, policy ops ships defensible decisions.
[07:45] In terms of So, if you want to understand what's at the core of it, the unit of the work in this instance is the case. So, whatever case that is being decided by the case worker.
[07:59] And then the unit of quality is the traces. Whatever is happening at the core of the multi-agent system. And then the unit of improvement is the SME labels. So, the SME looks at the
[08:11] solution, they come up with the decision, and then they move forward with that with the process of optimization. Um This is where I I now want to take a bit
[08:25] of a step back of how we were doing things before we landed on this final version of solution. Uh we had stitched a solution together um like most of you in the room.
[08:37] Um our engineers had access to every interaction of the solution. Every model call, every tool call, every chain of thought, all of that was
[08:49] stored. The only thing is I'm I'm talking to the engineers in the room, you can tell for yourself like how good a legal professional you are. I definitely am not.
[09:01] Um so, then that becomes a challenge. It lives in a system where your S- legal SMEs cannot get access to. And when they ask you for that, you have to pull it from your data dog or ML flow
[09:13] or other place of traces and then pass it on to them in a file, they decide, it goes back into the system, and the system you were designed to reduce the case work time from hours to minutes
[09:27] now taking 5 hours to validate a decision. So, it defies the whole purpose of of why the system was put in place. Um for one of our use cases, the way we used to do was we would run the solution, put all the
[09:41] outputs out in an Excel file, pass it on to our case workers who spent two to three weeks analyzing those case files. Then they pass it back to us, and then the the engineers would translate
[09:55] it. So, comes with a massive translation burden. This shows at a high level the chaos we were dealing with.
[10:07] So, when you are stitching together a solution yourself, you're building your own UI, you're you're managing your own database instance at the back, you need to manage auth for all of those
[10:21] things together, then you need to have your vector DB on the side, your agent orchestration framework uh on one side. So, it requires a massive team to first
[10:35] stitch it together, and then manage it. And at some point you start asking the question of why we're even doing it because it becomes so complex to manage. Uh it's like juggling multiple balls in
[10:48] in your hand. Um but around last year, something changed. So, Databricks ships Databricks apps. Before that, we had uh an app running on on AWS
[11:03] where we were managing everything from auth to uh to to kind of access controls, everything by ourselves. Um with the launch of that apps, or then we we we had a new capability, a
[11:18] new surface that we could share with our end users. So, we took the output of the system, we put it on that apps, and pass it on to the users. Now, all of a sudden our users had this capability of
[11:31] thumbs not just thumbs up, thumbs down, but also provide qualitative input. It was good, but wasn't great. There was still step missing of how do you take that from
[11:44] from from that output into a repeatable framework where you are training the models again and again, and you have that cycle going on for you. That is the thing that was missing at
[11:57] that point. Then the next thing that happened was when Data Bricks Agents, that was initially Agent Bricks rebranding, came out. That helped us with some of the other
[12:09] aspect where you take policy, you take the policy structure document, you pass it on to process to the agents, so the agents can extract both schema and the
[12:22] information out of it and turn it into a kind of a table, and then you can apply that in the whole multi-agent system for you to to process.
[12:37] Here we're extending it a little bit further of of what is happening in these three three pillars. The beauty of it lies in the Unity Catalog one identity model across all three. So,
[12:50] it reduces the number of services that we had to manage ourselves, we had to structure ourselves. Starting all the way from SME interfaces, so in app you have your own app, and that shows you
[13:04] what kind of cases are in the queue for you, what case details are there, and then it also enable you to put your structured feedback back into the solution. With intelligence layer agents, you are
[13:17] getting the information from policies, you're extracting that information, you're retrieving relevant pieces of of policy, and then you are you are able to apply the rationale onto
[13:31] the cases. Um in lake base, lake base provides the OLTP back end where all of this is being synced in in in near real time where cases, events, and review actions are
[13:44] being stored. And all of this happens in um in in a seamless manner. Here is an example of how we started doing it. So, most of the policies
[13:56] um start with some kind of a document. If you are lucky, that would be a PDF document. If you are unlucky, it would be a written document that was written 40 years ago, and then you might need to
[14:08] scan it. Uh and translate into something that can work for the solution. Um in our cases, most of it we started with the PDF documents. So, what PDF does is So, here with this
[14:20] uh this this aspect of the solution, we're doing two things at once. So, we are using the managed service to a extract the parts of the document. So, it looks at the document, it looks at
[14:32] different elements in it which is relevant and which is not. Um and then the information extraction agent So, you define a JSON schema beforehand that here is the policy,
[14:45] these are these are the policy fields that are relevant. Um and then textual information related to your own use case. Um the key design choice here was
[14:58] chunk for accountability, not for embedding. Uh because the traditional traditional chunking where you create a chunk on the basis of length, and then you have 50
[15:12] tokens overlap on either side, is chunked for at times can cut the policy in half. It it end up losing the lineage of what you're trying to do. So, that's was the
[15:24] main goal for us to preserve. And the next slide we have two examples of that. And before I
[15:37] go into uh I go deep into this slide, I just want to uh call something out that this is not an actual policy of MasterCard. Uh the policies that run in production um
[15:49] we cannot show in public, so we came up with a synthetic example. So, the synthetic example is that you are an organization who deal with customers and your customers file for returns and
[16:01] you have different kinds of policies for those returns. So, one policy is that if a return is filed within 30 days you go ahead and process the refund of that. The other policy is if it's filed
[16:15] outside of that standard window, but you are able to provide documentation around force majeure, then that policy gets overridden. And now you can go and make that
[16:27] decision uh in line with that policy. And the important points to see here is that every clause is which is extracted
[16:40] from the policy documentation comes with its own unique clause ID. It also refers to the original policy documentation that it comes from a certain page. And then that again it's
[16:52] got the policy version as well. Because you can have changes of policies in between different uh iterations and times. So, it's it's beneficial to uh kind of link it
[17:05] back to the source of that. So, we we have run the policy through this. We turned it into a queryable table. And this is the next step of where that
[17:18] agent is actually looking at it. So, the agent goes, it looks at the evidence and on on the on the far right, you have the case information. So, there is that case
[17:30] ID, um action type, how many days has elapsed from the actual initial uh order. Um exception code, what's the potential
[17:43] expe- exception here, and documentation, do we have the documentation available to us or not? And then, the next step happened. This is the thing there where most
[17:55] interesting stuff happened that it the solution kind of pulled the REF 5.21, which is the policy for standard returns.
[18:07] But, it missed the retrieval of the clause that is force majeure. That's basically an exception clause. Um Now, a question of of of general ability
[18:20] of the solution here, you can think of it as any exception that can potentially come out of your legal compliance work. Which we know are more often than not is the case that you have a lot of edge
[18:32] cases. So, you can have multiple policies being relevant to a question at hand. And the agent system can make decision on your behalf, but for you to have
[18:45] kind of view into that, enable your you and your case workers to make those fine-tuning decisions that if this happens, then in this scenario, here is the step action we will take.
[18:57] Um Why the agent missed RFE 7.42? Like, for the for the purpose of the demo, we made it miss that. But, in in reality, it can happen for a number of
[19:09] reasons. Um there can be similarity in terms of um um um the retrieval. There can be uh challenges with the prompt that you're using or there could be a number of
[19:23] reasons for why the system does it. The system's job the agent's job is not to be right, it's to flag uncertainty. So, what's happening here is that
[19:35] uh the solution pulled the one policy and then missed the other policy. But, it was able to look at the narrow in on the exception side of things. So, it realized that it is not a
[19:48] clean-cut case of refund submitted after time. It's there is exception potentially present and then there is also documentation that is present. So, that pushed down the
[20:00] confidence level from say 80% that would have been the threshold for auto processing to 71% percent and then it suggested that it's out of bound and pushed to an SME for review.
[20:14] And it also provided the uncertainty reason that the exception present below threshold. So, it needs to pass on to the to the user. And this is the inner workings of the
[20:27] system. So, each step that the system takes on in multi-agent orchestration is recorded as a trace and that trace is
[20:41] pulled to the user. So, it shows that it initially loaded the policy version. So, all relevant 420 policy units that were relevant to returns were loaded.
[20:54] Um and in from there in terms of application, the retriever candidate clauses were three that should have been selected. It selected two of them, but then it missed third one, which was
[21:07] related to force majeure. Um and then in terms of application, it it does it went through the right steps that were needed there to be done. So, it extracted the evidence field, it root the case in line with the the kind of uh
[21:22] setup that was given to it, the routing rationale that was provided. But, draft rationale uh is where where another issue is that it wasn't able to reach the right decision, so it passed it on to the
[21:35] stakeholders, to the SMEs. So, the it we have shifted the solution from the AI was wrong to retrieval step was missed, and that caused the the rationale to
[21:47] change, and uh Mr. case worker, you need to decide on that. Um this is a mock-up of the kind of UI that we have created. Um
[22:00] one thing I want to stress here is that the design choice we made here. Um we could have given them access to our current observability tools, but
[22:13] obviously being non-technical users, they weren't familiar or comfortable using them. So, we went through a design iteration where we actually used UI UX designers
[22:25] in the room who went and talked to those uh end case worker and asked them what kind of information you would like to see here. So, which level of altitude we need to go down? Do we really need to show you every interaction happening, or
[22:39] there are summarization steps that we can do? And as a result of trial and error, um we landed on something that looks like uh the case flow on the right. So, they see all the cases that are in their uh
[22:53] pipeline, they see the the priority of those cases, they also see the reasons or the causes of why a certain thing is happening. Um one thing aligned to this app is so
[23:07] because of Databricks apps, we're able to link it to uh same authentication model. So, if a user is not allowed to see certain information, they won't see it in their app. So, that helps us
[23:21] rather than stitching uh a separate identity process together, we're able to uh use the lineage from the system to get to a solution. Um and on the right side is kind of
[23:36] it the information there is um synthetic, but it's kind of in line with how how our case workers use this uh this model in in production.
[23:53] In terms of um linking it back to what happens at the on the Databricks side of things, so we have three tables. Um one is cases, so cases is where everything related to cases live, so it's you can think of it as the operational
[24:07] state or working memory of the solution. Um case events is more like append-only ledger that is used for audit purposes and replay purposes.
[24:20] And review actions is the SME structured SME level. So, whatever actions a reviewer is taking, they get stored in this table. And then obviously we have the
[24:34] the ability to run queries on it and do analysis, um but the best part that we can do here is is this compounding circle of of what we are able to do.
[24:46] So, uh the the main aim here is that the solution makes the mistake only once. And when the you when the SME goes into the system, they go through the case
[24:59] processing and they make a decision one way or other, then we have different then that information gets synced back to Delta. And then we have different levers we can pull there. So, maybe in
[25:13] some instances we'll fine-tune a a llama or an open-source model that will that will make sure that that kind of similar kind of error doesn't happen again. In other cases, we'll just add it
[25:25] to the evaluation framework. So, then it gets the system runs the evaluation on similar kind of cases before it pushes anything into production. Um and
[25:37] through this kind of six steps, so SME labels it, from there to it gets written into lake base, from there it's synced into Delta, from there it added into
[25:49] evaluation upset evaluation data set, and then ALHF tunes the next agent for that. This is the cycle that we have seen multiple
[26:01] percentage of improvement adding multiple percentage of improvement to to to our solution. And it's taken the threshold from where our legal stakeholders were wary of the
[26:13] solution that we don't think it actually works, it's random at best, to them saying that this is actually having real positive impact on the work that we are doing and they turned into the biggest
[26:28] sponsors of the solution. And as a result of this, what changed for us, we went from months of stitching to
[26:41] weeks. Now the solution is is a data data data bricks asset bundle and it can be with with three lines of code, it can be replicated anywhere we want it to be.
[26:55] Um and the other side other changes on the human side of things. So, our SMEs from third-party reviewers of the output of the solution, they became the owners of the solution. So, engineering doesn't
[27:09] own that solution anymore. We are helping them run it, but it's their solution. They provide us the requirements, they tell us how it should work, and they have visibility into all of that. Um
[27:21] and it kind of freed up all of our engineering time that was going into plumbing the solution to building one better operating model. So, now we are more focused on how do we add layers
[27:35] onto the solution that makes it better for our user. For example, we added a layer into policy flow that we didn't have before. So, initially, we were just pulling from policy via agentic solution. Now, we have a
[27:48] loop there where an SME can actually go and review the information pulled from the policy PDF documents is in line with what they think it should be.
[28:00] Um and the feedback loop and and survey rounds have actually increased that helped us increase the accuracy of the system um much more than it was before.
[28:13] Um One important point I want to mention here is that with AI, obviously, there is that kind of feeling that potentially it can take our jobs. That that feeling is under the
[28:26] surface everywhere. Um What this use case shows is that there are gray areas, but there are some very clear ways where AI can actually
[28:39] help people do heavy lifting in their job. So, all that mundane work of of doing reviews yourself, of of going through the policy and implementing the policy
[28:51] that gets translated into a system that is actually helping you to be 10x more impactful than you would have been otherwise. And for an organization like MasterCard where we have
[29:04] hundreds of millions of merchants, thousands of partners, billions of transactions happening, we want to empower our teams to be able to deal with the the scale of the operation we
[29:16] run. And this had been a great solution for that. Thank you so much.
[29:29] >> Thank you so much for listening to such a dry topic. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.