Sponsored by: Weil | Goldmines and Minefields: Building AI Systems for High-Stakes Work

Summary

Weil Gotshal treated AI deployment as a systems architecture problem, building modular micro-systems that solve specific high-value legal tasks rather than a monolithic platform.
A multi-run process using sequential document packetization and randomized context windows generates confidence scores from model consensus, reducing correlated errors from any single frontier model.
The SCS (Scope, Context, Sample) framework structures complex legal documents such as contracts and judicial opinions for LLMs while meeting the firm's privacy, security, and auditability requirements.

Chapters

FAQs

How does Weil approach AI deployment differently from a typical tooling upgrade?

Weil treated AI as a systems architecture problem, focusing on modular micro-systems that solve specific point-in-moment problems rather than building monolithic applications. This approach lets the firm identify which use cases are durable and then build a scalable system underneath them.

What is the SCS framework used in Weil's AI system?

The SCS (Scope, Context, Sample) framework is Weil's method for structuring complex legal documents—such as contracts and judicial opinions—for use with LLMs. It addresses the challenge of imprecise correctness in legal summaries and ensures AI outputs meet accuracy and auditability standards.

How does Weil reduce hallucination risk in AI-generated legal analysis?

Weil uses a multi-run process that packages documents into sequential bins and randomizes packets across varied context windows, then runs multiple frontier models to surface consensus-based confidence scores. Outputs with low model agreement are flagged for human review, reducing reliance on any single probabilistic model.

How does Weil handle ethical walls and client confidentiality in its AI system?

The system is designed around ethical walls and client confidentiality requirements, ensuring that matter-specific data is siloed and never crosses client boundaries. The architecture includes privacy controls and auditability features that meet the legal profession's obligations around sensitive data.

Full transcript

[00:08] Well, word on the street is that we're supposed to start now. So, thank you for the large audience here today, but I appreciate you all coming. So, maybe forward-looking statement. Oh, I'm supposed to tell you, please do
[00:20] your surveys. We would data bricks would like to know how things were. So, the title of this talk is gold mines and minefields. Why is that? We are a law firm and when we are using various
[00:34] probability models to do interesting things, sometimes having probabilistic outcomes is not great. So we can produce tremendous value, but there is risk in doing so. The point of the talk today is
[00:47] how we've begun architecting systems to minimize the risks inherent in using probability models in the practice of law. I think that's extensible to a variety of other fields and I encourage
[00:59] folks to reach out and talk about it because a lot of what we're doing here we are seeing early success with but there is no 10-year track record. >> All right. So who are we? So Wild Gotel
[01:12] is a full-ervice law firm large law firm. We have we operate in um 15 offices around the world. I am the chief AI and innovation officer. a class of job that didn't exist even a couple of
[01:25] years ago. With me today is Dalton Cochril and not here today but a part of the team leading the AI charge is Crystal Shu. So Daltton you want to introduce yourself? >> Absolutely. So I'm the leader of Laser
[01:37] at Wild which is our lawyer augmented solutions engineering and research. Uh basically for that group we spend our time focused on how we bring AI solutions to the attorneys and bridge the gap with our developers to get them effective use cases and solutions
[01:50] quickly. >> Exactly. The idea is to not build a monolithic system no one wants. We want to find the point in moment solutions that are useful. Figure out which are durable and then build the system
[02:02] underneath that can serve hundreds of this mosaic. All right. So what are we talking about? Let's use an example. So, a very simple question and this is something a client came to us with. And so, we built this
[02:15] because a client asked, how is a specific judge likely to approach a particular question or questions of law in an upcoming motion?
[02:27] Well, in this instance, they said, can't you use AI to look at everything the judge has ever ruled on and to put every transcript of every hearing that he's ever had? And in this particular judge,
[02:39] it was an Obama era appointee. So he had been on the bench for quite some time and there was quite a record to go through. This presented from, you know, our world three problems. First, we have
[02:51] to have confidence in what we're extracting >> because models will hallucinate and we are looking for needles that we know what they should look like, but we don't know if they're actually there and we don't know exactly where they're
[03:04] going to show up or how. So that's problem one. Problem two is what I'm calling imprecise correctness. So in the practice of law, having the right semantic meaning is good, but words
[03:17] really, really, really, really matter. So if you summarize it incorrectly, you can lead, you know, the lawyers astray. So that's a problem. The third piece is confidentiality bounded generation and
[03:30] learning. And what do I mean by that? For many of our clients, well, for all of our clients, we have an ethical wall, right? So lawyers can't be saying, you know, if we're serving client A and somehow they're adverse to client B in
[03:43] some distant part of the legal universe. Those two can't mix. But also confidentiality. Most of our clients say we don't want you using our data or even if you improve a prompt or a way of
[03:55] approaching a problem on our matter, we don't want you using that to the benefit of other clients. So we have to design systems that can respect those obligations.
[04:10] All right. So what we're going to talk about today is what I'm calling a micro system. It's the SCS platform that underpins a bunch of cool stuff we're doing including the example I just mentioned prior about the judge.
[04:22] Then we're going to talk about what I think this means for a larger system for the practice of law or other what I'll call high stakes situations. Let me briefly explain. So there's three
[04:35] kind of buckets I think and the way we think about it. Human input, machineto-achine loops, and then a m machine to human interface. So on the human input side, we start by pulling
[04:47] all of the documents that need to be analyzed. In a typical run, we're dealing with anywhere from 5,000 pages to 100,000 pages of information. So it's a lot of text, right? It's certainly not stuff you would ask a human to read
[05:00] through every word of. That's not a realistic task. Uh but it's also not, you know, terabytes and terabytes of data. We then have to say, well, what is the question we're actually trying to get at? So for every question you want to
[05:12] answer thoroughly, you need to run the loops we're about to talk about. Third, for purposes of testing and not prompt creation, what are the hypotheses around the expected results? So this is
[05:25] where testing becomes important. And then finally, can you give us some examples of what the output might look like? You think lawyer? And again, we're not going to put that into the prompt. We're not going to use that in any of the runs. We're going to use that later
[05:37] to verify in the middle stage. Dalton's going to talk through the details in a moment, but what we've designed is a fourrun process where we are running
[05:49] sequentially different frontier models. And the reason we need to run different frontier models is to avoid a coariance problem, right? Because models with similar architectures will hallucinate in similar ways. And we need to avoid
[06:02] that. We'll talk about that in a little bit more. By the way, not everything needs to be a large language model. That's just happens to be how we're running the system at the moment. Finally, in the machine to human part of it, we are producing a variety of
[06:16] outputs. Not just the one output. There is a single output here. We're producing a variety of outputs where we are asking the lawyers in this case to say this is what the machine's insights are. Here's the sources for that. Which of these are
[06:29] the important ones that we might want to triple check and which ones can we just move on with? So that's the basic process. Dalton, I'm going to hand it over to you now to talk about the specifics. >> Perfect. So, as Andrew said, that first
[06:41] component, we bring in this giant set of legal corpus to serve as our evidence base that we're actually going to walk through. I'm going to talk through four different runs that we do as a part of this process. So that way, we can build up that confidence in the actual answers
[06:54] that we're extracting. The first one we go through and do a normal sequential bin packing packetization process to go and take that large set of documents and get them into analytical groups based on the type of information we're going
[07:06] after. So in the example we talked about before where we have opinions and then we have oral argument transcripts, we group like with like. We're then going to go through and do multiple runs at high temperature to basically try to induce hallucinations in the given
[07:18] models to understand where those kind of challenge points are going to come up. As we do this and we have cross document findings, we're going to make sure that we track those over time so that we have a sub agent basically managing what are
[07:30] the things that don't just happen in the analysis of one document but across the set of documents that we're looking at. As he mentioned on covariance that next piece is to basically say how can we pre-wire ourselves on these prevetted
[07:42] sources and understand what those questions need to be so that we can go and do that validation in the end. And then we're also going to limit how much of the context window we use for input to make sure that we leave enough headroom for reasoning, keeping that around 30%. But that's one of our
[07:55] tunable variables. So that first run goes through and does a normal process. We're going to get a set of outputs from that. But we're going to approach this three different ways after that to basically help triangulate the outcome. So the second
[08:08] thing we're going to do is we're going to actually randomize those packets. So initially, say we had documents A, B, and C in the first one and D, E, and F in the second. I might swap documents B and E. So that way I can say, okay, now if I have a different set of documents
[08:21] grouped together, do my insights change at all? And basically when we see those changes, those are the things we then kind of raises flags of, okay, we might be less confident in those. They might require more validation. The third run is then to do the same
[08:35] thing we did in run one, but we're going to use that tunability of how much context window goes or how much context window goes to input and start to overlap some of those documents. basically say if we give you a little bit more information from another packet
[08:48] does that again change the actual takeaways that you have. The last one we're going to do is do basically random walk sampling of individual documents. So basically this lets us take out the what are we grouping together in a
[09:01] packet factor and say let's look at one document at a time on a on the right sampling basis to basically understand what are the insights we look at them solo and how does that line up against the other three runs. What we're then able to do with all of this is basically
[09:14] say where do we have confidence from all four runs giving us the same outputs. Those things are things we can be really confident in upfront without as much validation. Where we start to see deviation from those where we might only
[09:26] have three of the runs that line up or two of the runs. They just require more follow-up for us to go through and say what's the actual insight behind these. >> That's right. And the interesting thing about this is as we begin working with clients in real matters more and more
[09:39] they're slow as they understand how it works they're slowly saying well actually we're going to wave a human looking at this piece over here because we understand it's really the likelihood is so high that that's true we're good
[09:52] we take the machine's word for it right whereas over here we want someone to check and I think in the in the professional services space we will see how that plays out but it's very interesting to introduce the concept of
[10:04] statistics to folks that weren't traditionally working with it. So, all right. Normally, this is where we would say, let's debate some of this, but this isn't that kind of forum, but I would love to talk to people about this
[10:16] because I think we've got something very interesting here. But I'm willing to bet lots of folks in the crowd might have 34 other really good ideas. I promised I would say let's what would happen if we extend this concept beyond
[10:28] that micro system for a bigger system to design and build for the practice of law or delivering other professional services on top. So we currently think this system has these six components.
[10:42] Obviously this is a generalization. So it starts with retrieval. I think retrieval happens from a variety of databases and so re-ranking and ranking become very very very important
[10:55] in the type of work we do in the end points in the end products we have to deliver we'll talk about that a little more I think this is for us at least one of the most promising learnable surfaces
[11:07] especially where I have to live with confidentiality bounded retrieval and generation that I mentioned at the beginning right I need to make sure I am not retrieving things that I'm
[11:19] technically not allowed to retrieve due to bespoke contract arrangements I have with the thousands of clients of the firm next piece is generation right so obviously we need a generation layer
[11:33] then provider reconciliation so when I talk about generation this is where we are talking about calls to models some of them could be open weight models some of them could be the closed frontier models but each of them are generating
[11:45] in parallel Well, and the point of this methodology is to then reconcile where those calls agree and disagree which requires you to figure out what are the discrete insights so you can ultimately
[11:57] generate a histogram of where did the models agree versus not then evalgate. So there's a variety of companies I've talked with here uh actually this week brain trust being one
[12:10] who I think are going to be important in helping us design full evaluation where are the agents not you know silently failing but importantly where was the output not so good and we can tell that
[12:23] in an automated way and where was the output not so good based on how the humans interacted with it in that machine to human part of the loop and finally taking the results of the eval and feeding it back to the learnable surfaces there. Each of these could be a
[12:37] learnable surface. My first priority right now though is ranking and reranking because for the most part at the generation stage for the type of work we're doing, it's working right now. I'm sure we will
[12:50] learn more pushing further. How do we set all this up? Right? To do retrieval well, I think we're going to need parsing and extraction at scale so we have better machine readable formats.
[13:02] I met some great folks, Llama Index for instance, that I think are going to be able to help with these types of things. >> So, I'll pause here. Dalton, what would you add to the system? >> Yeah, I think the biggest things in this
[13:14] are not just dealing with the retrieval side as well on how we actually manage those inputs, but what you mentioned as well on the feedback loop and evals. Like so there's not just the segmentation of information for law firms on just how are we using their
[13:26] actual data they give us but also that learning component. Being able to segment out how you learn and where you apply those learnings becomes really important because that client is going to want you to improve the prompt for them. So you can't just be reinventing
[13:39] from scratch every time you do the same work for them. But you have to be able to split that out to your applications everybody else. >> That's right. All right. That was the spiel. The question is who
[13:51] has questions? I'm happy to repeat them. I would love to debate this stuff because this architecture on the screen has not been built. It is our very strong hypothesis of the direction we want to walk in. To
[14:03] repeat for everyone's benefit, Shane was talking about as a former lawyer himself said, "Hey, I know legal documents tend to live in a document management system. There are two dominant ones in the
[14:16] industry. I manage and net documents. Um, and he was talking about it being very important to make sure the doc IDs match up so that way as you evolve some of the documents, they are able to write
[14:29] and pull from the right documents in the DMS which already has the client confidentiality boundaries built in. Right? That's where the ethical walls live. 100% correct. So yes and if we go back a slide that's why packet ids and
[14:43] doc ids are so crucial to us in that bottom right corner I have a open theory that I only want to read and write to that system in batches or it is going to
[14:55] really really slow down and so the question we are grappling with and I'm curious people's thoughts is should I be abstracting parts of the DMS sanitizing it using that for the retrieval and and
[15:08] then only writing back to the DMS as needed because that is a it's not built for rapid readr. Let's put it that way.
[15:23] Great idea. By show of hands, is anyone building anything for a legal department or working with a law firm in any way? These guys. Does anyone want to? >> I don't know. Who wants to build
[15:35] something in this stack? I think it's going to be fun. Has anyone had to explain machine learning to a lawyer? I would love to trade notes with you on
[15:48] how you did that. I'm finding I'm finding ways to do it, but uh that's been an adventure. Awesome. All right. Well, look, I want to use everyone's time well. I
[16:00] appreciate you all taking the time. We're going to hang around. Happy to answer questions or like I said, debate.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.