Building Trustworthy Agentic AI: Contract Extraction with Verified Evidence and Guardrails

Summary

DealSumm and JLL replaced one-shot machine learning models with a multi-agent agentic AI system for commercial lease abstraction, raising accuracy from 65% to 92–98% while reducing review time from 130 to 85 minutes per lease.
Specialized agents extract, judge, correct, and verify every claim with source citations, providing evidence-based verification rather than confidence scores to meet the standards required for high-stakes legal documents.
The approach delivered 30% lower costs versus the previous ML pipeline and 70% savings versus manual review, built on the Databricks Data and AI platform using Unity Catalog, AI Search, and an Agent Supervisor.

Building Trustworthy Agentic AI: Contract Extraction with Verified Evidence and Guardrails

Commercial lease abstraction requires extracting 2,500 data points from complex, multi-jurisdictional documents with 175+ legal frameworks and 45 languages. A single 1% error, a missed amendment or wrong date, can cost millions over decades. Agentic AI, not traditional machine learning, is uniquely suited to this problem because it reasons, verifies, and provides evidence rather than predicting and hoping.
Discover how DealSumm and JLL achieved 92-98% accuracy, cut review time by 35%, and reduced costs by 30% by moving from one-shot ML models to orchestrated multi-agent systems. Learn how specialized agents extract, judge, correct, and verify every claim with citations, why evidence matters more than confidence scores, how Databricks Unity Catalog and AI Search enable trustworthy data sharing, and how this approach extends far beyond leases to any high-stakes document understanding use case.
🤝

Chapters

FAQs

Why is agentic AI more accurate than machine learning for lease abstraction?

Traditional ML models predict answers based on patterns and cannot verify their outputs, leading to 65% accuracy and costly errors — including a client who paid the wrong rent for 8 years. Agentic AI reasons over documents, verifies every extracted claim with citations, and corrects its own errors, reaching 92–98% accuracy in production.

What results did JLL and DealSumm achieve with their agentic AI lease abstraction system?

The system achieved 92–98% accuracy compared to 65% with the previous ML model, reduced review time from 130 to 85 minutes per lease, cut AI processing time from 30 to 5 minutes, and delivered 30% lower costs than the old pipeline. The 92% figure represents the average across the entire production estate.

How does the multi-agent architecture for lease abstraction work?

The system uses a salami-slicing approach where specialized agents each handle a distinct task: one extracts data, a judge agent verifies the extraction, a correction agent resolves discrepancies, and a final step provides source citations from the original document. An Agent Supervisor on the Databricks Data and AI platform orchestrates the full pipeline.

How does DealSumm handle data security and compliance for sensitive lease data?

DealSumm designed their pipeline so that model inference is isolated per session and per tenant, with no training on client data and no logging of sensitive content across tenants. Unity Catalog and AI Search on the Databricks Data and AI platform enforce data-sharing governance boundaries for all clients.

Full transcript

[00:10] Thank you everyone for coming. Um this session will be about machine learning versus agentic AI prediction and want to say thank you fora and guy the Israeli team from datab bricks that making that a reality. So thank you
[00:37] a client at JLL found that they had been paying the wrong rent for 8 years. Not wrong by a small amount, wrong by enough to matter. Over that 8-year period, the data had been checked manually. It had been signed off. It had been signed off
[00:54] every year. And still the lease data was wrong. Everyone did their job. It was still wrong. The client was out of pocket. We were in the doghouse. And
[01:11] so that story is the one we want to tell today. My name is Andrew Ray. I support transformation across JL's businesses. And this is Jonathan Bowman who is the head of
[01:27] technology and R&D at Dealsum. We're going to spend the next 40 minutes showing you how we fix that problem and we're going to show you why it is not fixed in the way that you might expect.
[01:49] So what changed here are the numbers. The smaller numbers are our old machine learning numbers. The new ones are are with our new Aentic AI supervisor solution.
[02:07] Accuracy is up between 92 and 98%. 98% is for some of our leases. 92% is the average across the entire production estate. Across all production volume, that average is 92. That compares to 65% in the old machine
[02:25] learning world. Review time per is down from 130 minutes to 85, 70% versus manual. AI processing time is down from 30 minutes to five. We will spend the rest
[02:40] of the session explaining how and why our approach is not what the rest of the industry is doing. cost. I want to address a commonly held assumption
[02:56] that machine learning is commonly perceived to be cheaper than using agentic AI. So that's true per operation, but we're not measuring operation. We're measuring
[03:13] abstracted accurate leases. On that measure, cost is down 30%. versus our old pipeline and 70% versus manual. Compute goes up, sure, but review time drops, net cost drops,
[03:31] and that's the measure that we really care about. Okay. So, let's talk quickly about safety and compliance. So
[03:47] the way we build that pipeline is basically the same thing as going to the doctor's office and having a needle in your arm. That needle will not go back into your arm or anyone else for that sake. It will be just throwing into the
[04:05] bin. And this is essentially what we did. It's like one model that uses once per session per tenant, not even twice on the same tenant itself.
[04:21] So no training data, no logs, no nothing. The second we finish that prediction layer, we just delete it and that's all for now. Um, and also this is something we did
[04:39] many times so far because it allowed us to change the model. If we find the new one or better one, cheaper one, we just can replace it with zero cost and zero complexity. So from JL's perspective, that point
[04:57] about data security is hugely important. Our clients lease data is commercially sensitive. It's not a nice to have. We cannot have AI sorry our data being trained on AI
[05:12] models. In many cases, it's a contractual restriction. So Jonathan's architecture delivers that by design, not just by policy. And that is what gets us through the reviews with
[05:27] our clients legal and compliance and security teams. So, quick context on JLL. Not not a history lesson, but the scale is important to the problem we're trying to solve. We work in over 80 countries,
[05:42] 113,000 people, and we have 5 billion square feet of managed real estate. Every square foot sits inside a lease. Every lease has to be read, abstracted, and managed
[06:01] accurately. every day at scale. That is our reality. So if we think about complexity, that's just the starting point. JL adds another layer. We work across 175 legal
[06:16] jurisdictions, each with their own rules, formats, names. We have 45 languages. Every client has their own schema, their own field names, their own requirements. A lease for a technology company in
[06:34] Singapore looks completely different from a retailer in Germany or a logistics company in Brazil. There is no universal template. There is no shortcut. The margin for error is zero.
[06:58] So, commercial leases, 2,500 data points is what we typically abstract from a lease. That's not unusual. It's fairly typical. In commercial real estate, the 1% that is wrong is usually the field that
[07:15] matters most. a wrong break date, a missed rent escalation, a calculation error buried in an amendment. I mentioned earlier a client who paid the wrong rent for eight years. Everyone had done their job. The lease was still
[07:30] wrong. Getting accuracy right at this level of complexity is genuinely difficult. Almost correct is still incorrect. This is not a quality standard. It's a legal and financial reality.
[07:47] So let's talk quickly about Dilam. We are in the business for about 12 years now and we started with machine learning when it was um the modern way to do things and at the end we provide 100%
[08:03] accurate lease data meaning that we read lots of document and long legal documents. we provide um an abstraction for those documents and for that particular tenant um we read all of the
[08:21] documents that are existed uh under his um uh folder and a human verifies the data because we can provide 100% without a human in the loop. So we will always have a human in the loop that verifies
[08:37] the data um correctly. So essentially dam is a lease data library that at the end holds golden data set of real estate um commercial real estate leases.
[08:57] So leases have been around for a long time. We started off doing them before software with parallegals that would take a week. We then did spreadsheets. We then did applications and then finally we did machine learning as Jonathan alluded to.
[09:12] Each time we've hit the ceiling. We've sure we've improved the timing and reduced that time, but we've always hit the ceiling. The hard parts, reasoning through amendments, cross referencing definitions, resolving conflicts,
[09:30] all of those still need an expert human. Jonathan will explain why this time it's genuinely different. Yeah. So let's for a minute compare two things.
[09:46] We know that in demos AI look very impressive. So in demo leases when you build an AI pipeline usually it looks like this uh demo lease with the straightforward
[10:02] sections that are showing exactly the clauses that you need to extract with everything in one place. Um the rent is explained very well. The renewal options are very straightforward. The permitted
[10:17] use is genuinely written explicitly. But in production, you may have documents that are with 30 plus amendments and you have amendments that
[10:32] overrides each other. And sometimes you will have dates that derived from another date or a number that has been derived from another number. And also you will have a very large amount of variations of leases. So this is the
[10:50] example of one PDF that is super straightforward compared to 30 PDFs that is messy and basically all over the place. So commercial lease is a contract. A tenant gets the right to use space for a
[11:06] fixed period in exchange for rent with obligations on both sides. That's a super simple way to define it. Very difficult to interpret when that space is over multiple jurisdictions, multiple
[11:22] languages, and the amendments go back years and years and years. So, a lease will basically break down into three categories. Money, not just rent, but escalations, abatements,
[11:38] common area maintenance charges, and operating costs. Time, not just the term of the lease, but also renewal options, break rights, notice windows, miss a notice window, and you can lose a right
[11:54] worth millions. rights they are what the tenant can and cannot do with the space. So the challenge here is that these three categories interconnect a common area maintenance calculation depends on the rent base
[12:11] which depends on the escalation history which runs across amendments spanning potentially decades. Each link is a place where simple pattern matching fails.
[12:29] What's needed is real reasoning. Let's give that into a real life example. We earlier saw the left side of the slideshow which is the demo lease that it's kind of straightforward. And then on production you will have a 1993 lease with 29 amendments that was
[12:48] scanned in 2008 and you will have multiple long documents of that lease. amendments that overrides another amendment and clauses that are referencing another clauses for that
[13:05] particular lease. And this is the example of a lease that we actually deal with in production. Um, so let's give it like a bit of a timeline for that specific list. So we
[13:23] are seeing like an office list that was signed in 1993 and is valid until 20 uh 29. We have uh 29 amendments over 32 years and a lot has been changed during during that time. So essentially a floor
[13:41] was added as a temporary floor was removed and each of those amendments change the square footage and the rent. So if for example if we want to ask the AI what is the current square feet or
[14:00] how much rent we need to pay we can have dozens of answers. Each one of those is correct because it has citations to the original document and should be valid answer. But in reality, we want to get
[14:15] the latest document and the latest version and it's not so easy and straightforward.
[14:31] So the cost of 1% a retail lease 10 amendments. The construction allowance starts at 453K. The first amendment raises it to 667. Second raises it again to 683K. If the system misses the amendment chain
[14:47] and delivers the original number, the tenant is short $230,000. That is one field on one lease and JL processes thousands of leases with thousands of data points. The exposure
[15:02] compounds really quickly and it's silent. You don't see it until you're eight years in. This is why we say that almost correct is still incorrect. It's not a quality goal. It's legal and financial
[15:19] statement. So let's talk about accuracy real quick. So accuracy is a well-known issue. Imagine um doctor or lawyer or financial analyst
[15:37] getting recommendation by AI but just ignore that because they don't trust the answer. It simply gives the answer but doesn't give enough evidence to support that answer. So what we built here is basically a
[15:53] change instead of understanding which citation we want to do or the human in the loop need to understand um which answer is right with machine learning with a Gentai. He just needs to
[16:09] pick the right answer and to verify that the citation and the evidence that was provided is good enough for in terms of confidence in order to select that answer
[16:27] and this is exactly where machine learning fails. So sometimes in lease documents you will have implicit answers. For example, commencement date will be 30 days after something that happened. And machine learning understand the formula but doesn't
[16:42] understand which date you need to put in that contract. Same goes for rent escalates. If the rent escalates per year by greater of 3% but you don't know the CPI of last year essentially you
[16:58] know the formula but you know you don't know the number and the same goes for conflicting answers as for the example we showed before if you have two amendments saying two different numbers at the end you need to pick one of them should be right
[17:14] and the other should be not and if you will give that document or problem to general model AI as of today, you will just um can hallucinate one answer which will
[17:30] be right because you have citations and you have reference to the original document and it seems true but in reality it's not because you need to account for all of the amendments date
[17:46] by date by date until the latest one and by then find the right answer. Another issue that we have is partial data. Sometimes data is not explicitly written in one paragraph. So you can use
[18:02] traditional rag and embedding to find the right answer but you need to combine it from multiple sources even with the same lease agreement. And the last is ambiguous clauses. Sometimes machine learning can think or can decide that he
[18:21] has specific answer which is right wrong we don't know but agentic AI and the capability of the reasoning in the AI can provide us an ability to really understand if the clause is supposed to
[18:37] be true or not true if we got the right data or not. So let's dive quickly into datab bricks tools that we use in order to solve that issue. So first of all we have the AI
[18:54] search which is the retrieval system that gets the data from the documents and then we have the unity catalog plus delta share which makes the integration with JLL and downstream system so smooth
[19:10] and agent supervisor agent will control and orchestrate a bunch of agents that will do that task that we will explain later on. Genie will give the end user the ability to ask any question that he wants on thousands of leases and get the
[19:30] right answers back because after the human reviewers reviewer confirmed that we have 100% accurate lease data then Genie becomes much more powerful. LLM judge is the tool that makes the
[19:46] agents um appear reasonable and provide real evidence and model serving is the same thing that we explained before about safety is the basic mechanism that allowed us to use AI one time use and
[20:03] then dispose that AI model. we are essentially deleting the AI endpoint when we are finishing using it. So from JL's perspective that Unity catalog ma matters most. We are a global
[20:18] operation with the data strict data requirements one governed data layer shared by both JLL and DSUM is what makes this an enterprise ready platform.
[20:39] So let's dive into the high level steps that we are doing inside that pipeline. So first of all it's preprocessing preparing the documents um for the AI to engage and act and then the latest one is the post-processing and delivery which will be streamed into Genie and
[20:57] downstream system BI and so on. So we will start with pre-processing which is the document index. We gather all of the documents of that tenant and structure them in one specific index and
[21:12] search index build will allow us later on to find any specific data that we want to in in inside that tenant document list. And then on top of that we built um
[21:28] sophisticated rag on top of the on top of on top of datab bricks AI search capability. So we know that leases are being built with a structure that is um contains table of content. So each section and
[21:44] each clause in that lease has a specific reference a specific section that he's been assigned to. And we took all of those data points and we gathered them in a document graph that holds all of
[22:00] the documents combined. Because if we want to ask something about rent, for example, we want to make sure that we have the rent information from all of the document that has been uploaded into this specific tenant. If we will miss
[22:16] one of them, what will happen is that we will get an answer but not the correct one essentially and it might be leading to silent failure because if you haven't read all of the amendments of that lease essentially you can be sure that the
[22:33] data that you are providing to the human reviewer is actually true. So let's understand real quick the LLM structure that we built which is I know a bit of
[22:50] non not con non let's say usually what we do is we take the biggest AI model that is available we are getting we are taking um all of the
[23:06] documents putting them inside, getting all of the list of questions that we want to get an answer of and putting that inside as well and essentially praying really hard that the model will not elucinate and we
[23:22] will not get answers that looks very much right but essentially they are not. So what we did is we took that complicated problem of lease abstraction and long document
[23:39] um of lease data and we slice it into a salami version. So essentially instead of solving like a big problem we solved very a lot of very very very small problems.
[23:55] So cutting it into very thin layers. So the AI will be able to do those assumptions almost without getting it wrong. And
[24:12] let's give an example for that. Let's say you have a equation, a simple equation that any 10 years old kid can solve and you want to solve it quickly, not accurately, just quickly.
[24:29] Who will trust that will make it faster? Einstein or 18 years old kid? Essentially that the 18 years old kid will solve it faster. Why? Because he doesn't have the
[24:46] capacity to hallucinate. So Einstein will take that idea and start theorizing many things and dive into many theories. But at the end the kid did that in two seconds and Einstein just keep thinking.
[25:03] So this is essentially what we did. We slice everything into a very thing problems, equations that we can solve very quickly and efficiently and we can reduce the hallucinations to almost nothing
[25:19] and the supervisor agent is only the orchestrator of the of the pipeline. So we have four different agent. Each one of those is specialized with one task
[25:34] and one task only. One of them he's doing the first pass extraction. He doesn't have any reasoning any thinking. He doesn't need to site anything from the contract. He doesn't need to understand nothing. He just need to take that question and get the answer he
[25:51] think is right. And then comes the quality judge. quality judge look at the answer and understand whether this answer is right and have sufficient evidence or not. If he thinks that this
[26:06] is maybe not the right answer to this question, he will just pass it into the second pass correction that will take that and work with a bit of reasoning to understand whatever this one is the right answer or not. And the last one is
[26:25] the evidence augmentation which enriches the answer. If we think that this answer is the correct one, it will find out the best possible citation in the contract to make sure that the human in the loop receives as
[26:42] much context as he can in order to understand that this is the right answer. So the agent has four things to think of or more the pipeline have four things to think of. One is did we find the right
[27:00] answer for that question. Did we interpret it the lease correctly and we understood that this question is actually answering the problem that we the question that we want to answer. And then
[27:16] did that answer is actually fit into the client schema and and data. And last of all, did we prove sufficiently enough that the answer is genuinely
[27:32] um okay and can be can answer the question that we want to answer. And that prove it step, that's the one that changes what the reviewer does. Before the reviewer is investigating, having to dig into all of that detail.
[27:49] Now they're just confirming that it's the right answer. That's a very different job. So yeah, what we wanted to do and what we managed to do is to make sure that the human in
[28:05] the loop has enough evidence that supports his decision whether the data that we obstructed and extracted is actually true or not. So as I explained the AI reasoning has
[28:21] these four steps extraction, judge, correction and evidence at the end. Each part of those is independent to each other. So the extraction comes first come first and then the judge flags whatever we need to change or rethink
[28:40] about the answer and then the evidence comes last and make sure that we have sufficient evidence and last is evaluation because in our use case evaluation is a bit different. Usually AI you ask something you get an
[28:55] answer back and the there is like basically two answers whatever he got it right or wrong here it's a bit different we need to evaluate whatever the answer that he provided back is sufficient
[29:10] enough to support the claim that we wanted to. So we need to also check whether the answer is good and also whether the um the proof that he provided is sufficient enough. So we
[29:26] started building the evaluation layer second and if I would do that today the entire pipeline I will I think I will do the evaluation first and the last thing is post-processing
[29:41] and delivery. So final quality check when when the pipeline is finishes and then data standardization which means we format the field into dates, dropdowns, addresses so on and so on and then we
[29:58] compile all of the answers and provide them to the human reviewer uh to make sure that he can do um his decision efficiently and quickly enough. And the last thing is result storage. So when we
[30:13] finish and we have 100% uh accurate data. So we push it backwards into datab bricks and downstream system and Genie as we will show in a second.
[30:29] So this is our implementation of Genie chatbot and we think it's the most interesting thing we built so far because it has the ability to query dozen of leases in one go.
[30:49] Andy, so we're asking it which tenants will expire in the next 90 days. It's going through the entire data set, all of the abstracted data and the documents and it's coming up with the answers. We're then going to then ask it what the
[31:06] new base rent is for those so that we can figure out what the cost impact is of those lease changes. And you'll see that it's coming up with the answers. And what that does is allows a portfolio manager, for example, to rock up in the
[31:23] office at 9:00 a.m., ask those questions, and get instant responses back. Now, that was impossible up until today. And the answers are verified, validated. We know they're correct.
[31:45] So, integration is a big piece of that. And as it comes to moving the data between SAS platforms as we are and enterprise data storage, we automated that part as well. We work closely with JLL to close the gap and make sure that the data is
[32:02] being added automatically into all of the ERPs. This is not just like one downstream system. They have more than 20 downstream system and every system need to hold the data. um the real data with confidence. So no manual entry
[32:18] whatsoever. Yeah. So let's go over very quickly of what we had so far. we had a specific architecture that is not so
[32:36] triv trivial let's say that build maybe a little bit different that uses AI a little bit different um with a separate and agent endpoint and document graph and so on and so on.
[32:53] So this came from a specific requirements or a specific road map that road map that we had with JLL for a long time and it came from trial and error. So we didn't knew that we are going that way
[33:09] all along. We just follow the common sense and ended up with the pipeline that we just shared with you. So we started this session with three sets of numbers. 92 to 98% accuracy,
[33:26] reviewer time cut by 35% and cost down by 30%. Versus machine learning. Those n numbers came from one decision and that decision was to really understand the problem before we delved into the
[33:41] technology. the pattern behind them, the specialist agents rather than one powerful model, the evaluation built in from the start, the governments as a design principle, not a bolt-on extra. None of that's
[33:57] unique to lease data. It works wherever the risk is high of an overconfident AI bot giving a well-sighted incorrect answer. So if you're facing that problem, do please
[34:13] come and see us afterwards. What the partnership has taught me is that most importantly the investment was not in the technology but in understanding the problem. Einstein said it best.
[34:28] If you have an hour to solve a problem, he'd rather spend 55 minutes understanding that problem and five minutes finding the solution. We didn't move fast. We moved deliberately and carefully. And the results now speak for
[34:43] themselves. Thank you. Yeah.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.