Sponsored by: Reducto | Better Agents with Better Data

Summary

Agent performance is increasingly limited by data quality rather than model capability, particularly when agents must process complex PDFs, scanned documents, and mixed-format files at enterprise scale.
Reducto, which has raised $108 million and processes pages equivalent to more than twenty times the height of Mount Everest, focuses on building production-ready document ingestion pipelines that format data appropriately for its specific downstream consumer—LLM, embedding model, or human reviewer.
This video covers six best practices for production document agent pipelines, including orchestrating classification and splitting before the LLM, evaluating at each pipeline stage, and designing for use cases that go beyond simple retrieval.

Chapters

FAQs

Why is document data quality such a significant bottleneck for AI agents?

Enterprise data is predominantly unstructured by default, and complex PDFs, scanned documents, and mixed-format files introduce parsing challenges that degrade agent reliability. Reducto argues that agent failures are frequently attributed to the model when the actual root cause is poor data ingestion upstream in the pipeline.

What does it mean to format data for its consumer in an agent pipeline?

Different downstream consumers require fundamentally different data formats: an LLM consuming retrieved text needs different formatting than an embedding model or a human reviewer. This video presents the principle that effective ingestion pipelines must be designed with the specific downstream consumer in mind rather than treating all outputs the same.

What are the key failure points in a document agent pipeline?

According to this video, the three primary failure points are parsing (extracting content accurately from complex formats), retrieval (surfacing the right content at query time), and context (ensuring the agent receives information in a usable form). Addressing all three stages is necessary for production-grade agent reliability.

How does Reducto's approach differ from traditional RAG?

Traditional RAG focuses primarily on parsing and retrieving documents to answer questions. Reducto's approach extends beyond retrieval to making agents capable of answering questions, making decisions, and generating artifacts that resemble real human work—requirements driven by enterprise contexts that need agentic verification equivalent to human-in-the-loop review.

Full transcript

[00:07] Thank you for joining. Just by way of quick intro, I'm Adeet. I'm the co-founder and CEO of Reducto. Just quick gut check, have any of you used Reducto before? If not, great. I wanted to spend today primarily walking you through what I see
[00:20] as one of the maybe less fun, but really, really important problems for making agents production ready in the real world. Um we have had the fortune of working with two very distinct ends of the market. Um
[00:34] a lot of the vertical AI apps and AI native companies that you've seen use Reducto to power their ingestion and processes downstream. Um we also work with the largest enterprises in the world. Um the largest financial institutions, Fortune 10 tech
[00:46] enterprises, so on and so forth. And across those companies, what we've really set out to do is build the agentic document platform for world-class teams. Um for us, that means many different things, and it's also
[00:58] evolved over time with the sort of sliding lens of what AI applications look like. Just quick snapshot. Uh we've been fortunate to raise about 108 million in funding, and I think the more interesting data point is at this point,
[01:10] if you stacked every page that Reducto's processed, it's something like 20 times the height of Mount Everest. Um that number is an underestimate. Uh I've gotten kind of lazy with the plus at the end, but we process many, many
[01:23] millions of pages every single day for our customers. So, the thing that I wanted to focus on today is one to sort of frame this conversation. A year ago or maybe 2 years ago, I would assume that every single talk
[01:35] that was given here about AI was about rag. Um and that's very much not the context in which people are thinking today. I think that sort of frames what good ingestion looks like as well. Um it's not just about parsing and reading from your documents, but making agents able
[01:48] to answer questions, make decisions, and actually generate artifacts that are closer to real human work, right? A question that we get asked often is how this matters in the enterprise
[02:01] context, and I'll walk through the context that we see, but also why PDFs are still hard. Um for us, this is very much a long-tailed problem. There's a lot of nuance to what it means to solve this problem well.
[02:13] And we put a lot of time into thinking through where we see benefits with things like traditional CV and VLMs, and we'll walk through all of that. And then from there, I think the more interesting part of the session is going a bit deeper on where we find the next
[02:25] frontier for this category of work. Uh we've seen a lot of really interesting approaches with things like agents in the loop. Um there's a lot that we've learned in terms of formatting for your individual use case, and also where we
[02:37] see the industry overall heading as agent harnesses get better and better. So, putting aside the agenda, the first thing that I wanted us to talk about is what I mentioned a second ago. And that's the sliding lens with which
[02:49] people are approaching these products today. For a while, most AI applications were really just information synthesis tools, right? Like they were a better version of search. You could chat with your PDFs, and you could ask questions and
[03:01] answer from there. But today, most of you have probably used tools like Claude Code. Um code generation is I think the frontier of where AI has been really, really helpful. And we're starting to see that apply to traditional enterprise use
[03:14] cases, too. Um we're getting closer and closer to a point where you can do end-to-end loops. Uh things like taking information from some context and creating a form, filling out a document, generating the
[03:26] artifacts that you as a human would have. And I think a lot of what that looks like is you don't just need to parse the contents in your documents. You need the agent to be able to perform any sort of action that you as a human would have,
[03:38] right? All of the little interaction modalities of the way that you split information, the way that you get structured insights out of your documents, and edit and modify that data really starts to matter. And across the board, we find that even
[03:52] as models get more and more intelligence, an agent that is acting on bad data is something that you just can't rely on in production, right? This is a real case of an insurance loss run documents, and we find tiny source of context like
[04:07] agents pulling from the wrong row, um hallucinating data in the table. They compound over the course of each step in your pipeline. And in enterprises, that ends up being even more importance.
[04:19] Um if I can go back here, because one, enterprise data is unstructured by default. Uh you don't have cleanly organized and tagged documents. Things sit across different
[04:31] folders, different databases. Teams are not organized around this idea of a centralized source. Um that information is going to be scattered across contexts. And a lot of the real world looks much uglier than
[04:43] sort of what a demo would look like. Um the real world PDFs that are a challenge are not just the like pretty sort of Word document generated PDFs. It's going to be a lot of stuff that is messy and handwritten. Um it's going to include
[04:56] things like checkboxes, really complicated tabular regions, and that's where we see the bottleneck become a challenge. PDFs as a data format are very much not new. Um
[05:08] I have spent about 3 years living and breathing PDFs, but honestly, there are people that have worked on this longer than I've been alive. Originally, the goal for PDF as a file format was simply to represent the content faithfully when
[05:21] you were printing it. Um and so in the 1990s, the context or the frontier of PDF processing was making sure that printer drivers could read the PDF properly. And now we've ended up in this form where so much of the world's context is
[05:34] locked inside these documents. This is how you naturally think to email an attachment to somebody. And so given that, um if you're thinking about the world's brokerage research reports, the invoices, POs, pay stubs,
[05:47] all of that is locked in a file format that ultimately wasn't designed around this sort of use case of agents autonomously acting on it. And VLMs, I think, were the biggest step change. It's part of what motivated our
[05:59] decision to found Reducto. Because for a while, you would hit this sort of wall when you were trying to process PDFs, right? Traditional OCR always struggled with things like handwritten text. Um, tabular structures are not uniformly
[06:12] defined, and so you have things like merged table cells that become really hard. You have things like poor skewed scans, and those are the sorts of brittle pipelines uh that IDP platforms used to struggle with. But the flip side is we don't think VLMs
[06:25] are a one-size-fits-all solution. Because if you're a financial institution dealing with billions of documents, it's not just about having, you know, cool demo with handwritten contents. You care a lot about things like
[06:37] traceability for your decisions. You care a lot about determinism. That structure matters a lot. And there are a lot of cases where sub-100 million parameter models can actually present a better solution for the task. A lot of our team comes
[06:50] from self-driving car research, where we find that techniques like object detection and segmentation, especially with the newer category of models, are a really, really good complement complements for VLMs, which also
[07:02] augments the other side of the equation with a lens of reasoning and understanding semantics of the information. From a practical lens, what this looks like is we find that VLMs end up being
[07:14] the right way to address the long tail, while traditional CV is still great for addressing the vast, vast majority of cases. Starting in 2024, we started looking into an approach very closely inspired
[07:26] by what's known as speculative decoding. And if you've ever used a tool like Cursor, you've seen how it applies a single forward pass, like a really fast edit on your code. That's the same approach that we took for correcting the last mile mistakes
[07:38] that we saw in document processing. So we are going through and we're catching all the little details. Like maybe OCR messed up a zero versus an O, a period versus a comma, the tiny details that actually do matter a lot in the real world. And we'll go through and address
[07:52] those on a token level. And that's not just a question of latency and costs. Um it changes things like the determinism profile, right? You could imagine a naive solution being you OCR the document, you send a really
[08:04] pretty prompt to Gemini saying, "Please don't change things up too much." Um but in the real world, we find that as models get more intelligent, they do things like when they see a total at the end of a table, they'll actually start adding up the values instead of only
[08:16] being faithful to what was in the document. So, if the document had an error, like humans make mistakes when they're generating these, models will start correcting those errors erroneously. And this approach ended up being a more reliable way to be faithful to what was actually in the image
[08:28] itself. So, what that looks like is we can get really, really close to what is the equivalent of historically a human in the loop review, right? Um instead of having a human spend hours going through
[08:40] and fixing errors manually, we can now have the combined parse, verify the results, and then have a high confidence output across those documents. The other thing is there's a lot that is under appreciated in terms of things
[08:53] like data formatting. And data accuracy is not just a question of what will make LLMs perform better, but it's about getting everything in your pipeline to perform as effectively as possible. And before I jump into the specifics here, I think there's a really
[09:07] interesting paradigm today with language models as they stand. Um if any of you know Surge, uh it's a data provider that works with uh pretty much every foundation model company. Um similar to like a Scale AI.
[09:19] And Surge released this benchmark called GDP.pdf. Uh it's a set of tasks for agents to reason through. Where Fable, which set a new frontier, was still only at about 30%.
[09:31] Unfortunately, we couldn't test this with Fable because we don't have access. But we went through and we tried testing the same sort of pipeline where initially the model was directly
[09:43] given the PDF, and we didn't change anything. The prompts are identical. All we did was also add the structured text representation in addition to the PDF context for the model. And we find that a worse model,
[09:56] something like a GPT-5.5 or an Opus-4.8, same amounts of reasoning, performs better on the same task than even a larger model like Fable. And it's not just top-line accuracy, this ends up
[10:09] changing the profile overall, where given better inputs, all models improve uniformly. And the models also start to consume fewer tokens.
[10:21] They need to reason less, and as a result, also have a lower latency uh in their answering problems. And this might not matter matter if you're just building a simple demo, but if you're building an enterprise application
[10:33] that's going to be used at large volumes, those differences really really start to compound. So, going back to that idea, um there's clearly a lot that you can do to improve LLM performance, but the ideal output
[10:46] structure is not just about language models. You know, there are a lot of considerations that you should weigh here. And one example, this is a snippets of a paystub. Understanding this fundamentally requires understanding the merge cells
[10:58] that are in the documents, right? Like you can see the little structures of hours worked each day, etc. And so if all you were doing was converting your tables into markdown tables, you would lose some of that information, and models would reason less effectively.
[11:10] And so clearly, something like HTML is a better data structure there. But on the flip side, HTML is going to waste a lot of tokens, and you don't need that for all of your tables. What we landed on is we look at the complexity of the table, depending on the number of merge cells.
[11:23] If it's a simple table, we'll use markdown. If it's a complex table, we'll use HTML. And that way, we're sort of getting the best of both worlds. The step removed from this is it's not just about the language model, right? If
[11:35] you're building something like a retrieval pipeline, models are great at reasoning on that HTML table structure. They don't have an issue with understanding the table given that. But, embedding models have different limitations from language models.
[11:48] If you look at the HTML chunk representation, this is a large blob of TD tags and numbers. And in the real world, the question that a user is asking is not enumerating the values in that table.
[12:01] They're asking something like how did revenue change over time, right? And so if you're doing something like cosine similarity, you find that it's almost random in terms of the chunks that you're retrieving. Um and so any sort of search application really starts to suffer not
[12:13] from a model intelligence perspective, but because you're not feeding the model the right context. A really simple way to address this is embedding models perform a lot better if you give a small summary of that same data. So,
[12:26] you can kind of imagine forking your pipelines, right? You have one version that is optimized for embedding and retrieval. You have one version that's optimized for language models. That's exactly what we do with our parse results. We'll give you one chunk, which is the ground source truth. It's exactly
[12:39] what was in the documents. And we give you one that is only optimized for retrieval, any sort of use case like search. Given that, um the next thing that I wanted to talk through is I think a lot of the focus in this domain has gone
[12:53] towards things like parsing and extraction because I think it's one of the more visible tasks. But, we think that the right set of tools is also about things like orchestration. It's how you classify your data. It's how you split documents. We have a
[13:06] use case that we've seen in financial services where some of the large financial institutions have, you know, massive massive warehouse operations for mailroom operations. So, hundreds or thousands of humans go
[13:18] through. They receive an envelope. You don't really know what's going to be in a paper mail envelope, right? Humans make all sorts of mistakes where they don't clearly cleanly interleave their contents. Um, and so you'll have pages 1 through 3 are
[13:31] the new application form, page 4 accidentally was a different form, and then page 5 is again the original form. And doing things like understanding what is where becomes really really important for VOMs to then do their best work. You
[13:43] can kind of remove the reasoning effort by decomposing the task. And so what we'll do is one, you have your raw input as the uploaded files. The first thing we recommend is having a lightweight classification layer with
[13:56] smaller models to understand exactly what you're passing in. Um, and this can be based on complexity, it can be based on document type. From there, choose your parsing and extraction pipeline of choice, but we think it makes a lot of
[14:08] sense to actually go through and split those documents up. Um, and you can think of it as reducing the amount of work that the model needs to do downstream. Instead of giving a schema that's, you know, maps to a generic set of documents, you can have a really
[14:21] personalized set of like a schema that maps directly to the subset of pages that you're focused on. The last thing uh in terms of where we would really recommend paying attention is I think agent harnesses are some of
[14:35] the more interesting work that we've seen in the space recently. And one good example of this is sort of along lines of the work that we said of having evaluations and refinements. But that's not just a matter of getting the text
[14:47] correctly. There are modalities that we find are really interesting like line charts. Um, where this is a real chart, we have the World Cup game playing in the background. Um, this is a chart of goals
[15:00] per game, right? Obviously, this is not the data that enterprises are worried about, but if you imagine any trading firm or hedge funds, a lot of the work that analysts do is looking at time series data. Um, and that's really really hard. Uh, you can try any
[15:13] frontier model at this sort of task with max reasoning, and getting the underlying data points is really difficult. So, I don't mean asking a visual QA question like did revenue go up? I mean being able to manipulate that data the way that you would have if it
[15:26] was a Pandas data frame. And the only way we've been able to get that to work is with the harness approach where we take frontier VLMs, but we also give it a bunch of tools like the ability to re-render the chart that I extracted, to modify individual
[15:38] data points. We give it the ability to mask the image, understand the axes, etc. And today's frontier VLMs equipped with those tools are able to get to near pixel-perfect accuracy for all of the
[15:50] little granular data points, not just the axes points, but the data points in between. And that's really, really important across a number of domains. That also applies to things like deep extract. Like we see use cases in customs and border patrol where you
[16:03] might have 80,000 fields in a single document that you want to extract. And it's not that models aren't intelligent enough, but recall on those tasks is really, really low. Like they'll start dropping fields in the process and you don't know exactly what is going to be
[16:15] dropped. And so having this sort of iterative loop where a parent agent is defining success criteria and sub agents are dispatched to go through and do the individual tasks, makes it possible for you to catch those errors before they
[16:27] are entered into your system. The last thing, um, and this is less about capabilities and more so just like best practices. I really think evals should underpin all of your decisions and I don't think evals should be just a
[16:41] top-level eval. Um, you can look at benchmarks like GDP.pdfs to understand how overall LLM accuracy changes. But to us, what we see all of the best AI teams do is really understanding each step of
[16:53] their pipeline. If you're doing something like rag, you could have mistakes at the parsing layer and maybe you weren't faithful to what was in the documents. You could have mistakes at the retrieval layer. Maybe you're not even retrieving the right context and whatever you do on parsing, whatever you
[17:06] do on your prompt downstream will not matter if that is the bottleneck in your pipeline. And so understanding each step is the best way to actually address the overall accuracy. And of course, you should have macro-level eval so I understand how
[17:18] it's changing the output decisions that you're seeing there. Um I'll close out with sort of the initial framing of this conversation of we really do think that in the next year
[17:30] you're going to see something analogous to Claude Code for real human work. And I think Claude Code work is a great example of that where people will start to generate their PowerPoints autonomously. This was generated by Claude Design. Um took me about 15 minutes which would have been impossible
[17:43] to imagine a few years ago, right? And if you're going to do that sort of work, if you're going to imagine a world where you're pinging an agent in your Slack and it's going to do work that is going to have your name as the authorship,
[17:55] it's really important that you're feeding it the right context, that you're giving it the right tools, that it's not making mistakes along the way um in terms of how it applies the changes. Um and that applies in terms of both the tools that you give it, you know, skills
[18:07] like giving it the ability to edit effectively, um but also in terms of how you structure things like your data. For a while in rag context, everything was chunks stored in a vector DB. We
[18:19] find that there's a lot of interesting work today where we have a CLI, for example. Um a lot of our customers will have a file system where they have both the markdown representations, they also have additional metadata that's available for
[18:31] the agents. And the agent is now making decisions. Um it's not a deterministic pipeline. Um the agent is able to use the tools when it thinks it's relevant. So, I'll close out here. Uh key recommendations to take away. One, I
[18:44] really recommend that you decompose the parsing problem. Um there is a right tool for the right task and that applies on an accuracy level but also a cost and latency level. Two, I think agentic verification is a really big step change that you should
[18:57] have in your pipelines to make sure that you're catching things that you would have other otherwise missed in a first pass. Three, it's really important that you're formatting your data for its consumer, um whether that's a language model, an embedded model, or even a human
[19:10] downstream. Four, you can create a lot better um get a lot better results if you do some of the orchestration work before it hits the LLM. Um so, classification splitting
[19:22] are really important tools in your pipeline. Five, evaluate at each stage. And six, think about more than just retrieval. Um think about what your agent is going to do along the way. Cool. If you have any questions, I
[19:34] unfortunately need to run out, but please feel free to email me. My email is at D at reducto.ai, but thank you all for coming today.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.