Automating Document Work with Long-Horizon AI Agents

Summary

Approximately 90% of enterprise data is unstructured and lives in document containers such as PDFs, PowerPoint files, and Word documents, making document AI a critical frontier for enterprise knowledge work automation.
The evolution from naive RAG chatbots to agentic pipelines has shifted the key bottleneck from model capability to context engineering—specifically, the ability to accurately extract, structure, and retrieve information from complex document layouts.
LlamaIndex provides an agent-native document infrastructure platform that addresses remaining gaps in long-horizon document automation, including advanced vision-based document parsing, hierarchical retrieval beyond naive top-k, and cost optimization.

Automating Document Work with Long-Horizon AI Agents

~90% of enterprise data is unstructured, and a lot of that lives in the form of documents - PDFs, pptx, docx, etc. - powering a majority of enterprise knowledge work.
AI agents have the potential to simultaneously better make sense of docs and also automate human workflows over them. Existing document AI use cases have either been long-running but constrained (e.g. a batch invoice processing through RPA), or non-deterministic but short-horizon (e.g. ChatGPT deep research over a collection of existing files). Going forward there's a huge opportunity to build long-running agents that can understand, reason over, and edit documents at scale.
In this video, we'll walk through core advances in document OCR and agent orchestration that have given rise to initial use cases around doc extraction workflows and general-purpose chatbots (and share real-world customer use cases). We will then articulate some of the remaining gaps needed to achieve "long-running" document work automation.
Talk By: Jerry Liu, Co-founder and CEO, LlamaIndex ;

Chapters

FAQs

Why is document AI important for enterprise knowledge work?

Approximately 90% of enterprise data is unstructured, and a large portion exists in document containers such as PDFs, presentations, and Word files that power the majority of day-to-day knowledge work. AI agents that can understand, reason over, and edit these documents have the potential to automate significant portions of operational work that humans currently perform manually.

How has the approach to enterprise document AI evolved since 2023?

The field began with naive RAG chatbots in 2023, which used simple top-k retrieval over document chunks. It then evolved to deterministic workflow graphs with structured agentic pipelines, and has now reached a stage where generalized long-horizon agents can delegate multi-step document tasks. The bottleneck has shifted from model capability to the quality and completeness of the context layer.

What is context engineering and why does it matter for document agents?

Context engineering is the practice of equipping AI agents with the right sources and structures of information before they attempt to perform tasks. For document agents, this means accurately parsing complex layouts with vision-based models, building hierarchical indexes beyond naive text chunks, and retrieving the most relevant document segments for each query.

What are the remaining open problems for long-horizon document AI agents?

This video identifies several remaining challenges: handling complex document formats that require vision-based OCR and fine-tuned parsing models, scaling retrieval beyond naive top-k approaches for large document collections, managing cost as agents process large volumes of documents, and maintaining determinism over long-running multi-step workflows.

Full transcript

[00:08] Hey everyone. Uh, I'm Jerry, co-founder, CEO from Llama Index. Um, we'll tell you a little bit more about our company and especially if you heard it heard of it before. Um, you know, I think we've kind of evolved our mission a little bit uh as a company. But, you know, I think,
[00:20] uh, today what I'm really excited to talk to you about is automating document work with long horizon AI agents. uh which is especially topical nowadays as you know both the models and the harnesses themselves are getting really really good and folks are increasingly
[00:33] delegating more and more tasks to these generalized agents to solve more and more end pieces of knowledge work and and not just software engineering but basically all forms of knowledge work and beyond. Um huge shout out to data
[00:45] bricks uh for being a fantastic partner as you see you know I think we use them for for inference under the hood for a lot of our capabilities u but really excited to be at the data AI summit. All right, let's get started. Um, so I think
[00:58] uh today uh we'll talk about kind of like three things, right? One is just like this evolution of kind of how knowledge work has been done with AI over the course of the past three years ever since LMS came came to life. Um, you know, I think uh we'll kind of trace
[01:11] through an evolution from like the basics of naive rag um to kind of what we call agentic engineering or kind of how do you actually use generalized modern agents today. We'll also talk about what you actually need at the
[01:23] context layer, right? Before uh actually being able to do stuff with AI, you actually need to equip it with the right sources of context um to actually perform the day-to-day tasks that you want. Um and a lot of our focus is basically providing this like document
[01:36] context layer for agents to unlock the massive swats of unstructured data that are present within the enterprise are locked up within document containers. Um and then of course, you know, there's kind of some remaining challenges, open
[01:48] questions and next steps. um especially as the space evolves and the context layer itself evolves as well. So let's get started. Um sorry, give me one sec. AI agents are the new knowledge worker.
[02:01] Um they're starting to automate good chunks and pieces of day-to-day tasks that you would typically do as a human. Um but you're able to now start delegating a lot of these things to an agent. This obviously includes coding u
[02:13] but also includes a lot of like operational work that you typically do across various functions within an enterprise or you know even within a startup. But 90% of the world's unstructured data is locked up within human native document containers like uh
[02:27] PDFs um word documents, powerpoints, excel sheets and more. And unlike humans, agents don't actually have the eyeballs to actually you know read these documents natively. they actually need to use the right tools to kind of uh
[02:40] both read but also structure and understand these document containers. The fundamental issue is that even as agents get really good if they have access to bad data and bad tools they will fail. So before going into the rest
[02:52] of the talk, just a quick snippet of the company. You might have like seen us as a rag framework from three years ago. You know, I think this is back when the orchestration layer was, you know, pretty early. Um, and people were still just trying to figure out the best patterns and applications to basically
[03:05] connect their LLMs with. Um, and so over time, uh, we kind of evolved from a rag framework into really building the best-in-class document infrastructure for AI agents. We have the best AI native uh document processing to help
[03:18] unlock even the most complex PDFs, PowerPoints, Word docs that you have, extract information from it, reason over it, and give you back kind of like uh core context that you can use for any downstream agent workflow. We're
[03:30] fortunate to kind of serve both uh you know kind of fast growing AI native companies. So some of the largest enterprise organizations out there. Sorry, I got to get the click right just all right. Uh let's go through um an
[03:43] evolution of rag uh to a giant techic engineering. First off, maybe like back in 2023, how many of you have built like a naive rag chatbot? Just curious. All right. I I think that's like over
[03:56] half of you, right? And and so I'm sure all of you are probably familiar with the basics. So this is going to be a little bit of a nostalgia trip um to the past. So the first AI use case was like the rag trap bot. Um I think basically back when LLM first came out no one
[04:09] really knew what to do with it and so people started building really simple applications. Um this is basically okay how do I you know I have chat GBT right that thing just came out how do I connect it with my own private sources of enterprise context um and that's what
[04:22] I call like simple rag chat bots you know you ask a question it gives you back an answer um it's basically search like how much does search help a human probably like anywhere from 2 to 10% of like helping to automate day-to-day tasks as agents got a lot better um at
[04:36] being able to actually reason and loop over like you know tools and other types of things um they started being able to you know start doing like deeper research, being able to synthesize like deeper pieces of context and piecing it together. Um, and especially as coding
[04:48] agents started really taking off as a generalized proxy to computer use, now you're actually able to dedicate uh, you know, entire tasks to like a swarm of agents to not not only write code but actually start doing kind of end to end
[05:00] bits and pieces of knowledge work end to end. So back in 2023 um in March uh you know I think basically everyone was kind of building this like simple chatbot and back then you know we had this uh pretty
[05:12] popular open source framework that helped you basically get up and running with a naive rag chatbot and five lines of code um you have like some input documents or kind of some pieces of text you know you chunk it uh somehow you put it through an embedding model put it to
[05:25] a vector database do top k retrieval um and then feed that uh like kind of fixed context into an L1 to basically give back can answer. I think even back then, and I'm sure all of you probably realize the limitations. Um, and of course we
[05:37] did too. Um, like it was obviously not a great uh abstraction. Um, you know, I think there were a ton of issues. Uh, almost like too many to name. But basically, you know, retrieval was not good. Um, generation was not great
[05:50] either because the LLMs themselves were not great. And then also, you know, it was only really useful for asking like simple questions. if you have like kind of uh simple tasks that you wanted to search over your knowledge base uh it could give you back an answer but if you
[06:02] wanted to even ask like a multi-part question um it wasn't super good so I think what happened was you know especially late 2023 2024 um and you know we came out with the stuff too right so we were kind of in the thick of
[06:14] it we came out with a bunch of these like what I call hacks to basically just like make rag a little bit better um we tried to you know like tune chunking we try to tune uh you know like the way we do search uh do hybrid search ranking
[06:26] instead of just like naive embedding based lookup. Um we try to come up with like a bunch of these kind of like advanced techniques like query rewriting hide um like you know recursive retrieval and a few others to basically just like decompose more complex
[06:39] questions and try to actually search your knowledge base in a bit more detail. I would say this still kind of created a bit of a fixed pipeline as in it wasn't really offloading any intelligence to the overall agent so to speak. Well agents weren't really a term
[06:51] back then or to the LLM. is really just like kind of designing really going deep and try to actually handle it at the retrieval layer. What it didn't account for was the generality of agentic reasoning. You know, I think even back
[07:04] then it was maybe a little bit harder to predict that just like agents especially when you loop them um in some sort of like uh just like have them call themselves recursively um we're going to get really really good at just like kind
[07:16] of solving more and more tasks end to end. And so I think some of the initial agentic loops um it was basically like react right um how many of you have heard of react I think back then um just
[07:29] it was basically really simple you could try to write it yourself with you know your favorite coding agent these days if you want um but basically you just have like a prompt uh to an LLM and you just tell it okay when task is not done um
[07:42] please continue to call tool um otherwise it's done right and that's basically kind of what you prompt it and you call it in a while loop until it's done um basically This kind of started creating some kind of uh higher level
[07:54] abstractions where it wasn't just due to the capabilities of the model API through a oneshot call. You could actually start to kind of use uh agent loops to start both like making multi-port decisions, right? Because
[08:06] like it could decompose a higher level task into multiple steps in the loop. Um and also use it for like self-verification and correction. Um and so this is where kind of this idea of tool use um as well as like an outer
[08:18] agent uh kind of started to take shape and what we call agent harnesses today um is basically what like a react loop was back in you know like mid to late 2023. Um so I think I basically like all
[08:30] modern agent advances are kind of just like some really fancy react loop that's like drastically oversimplifying what the you know actual implementation is for like cloud code for instance. But like I think this started to kind of shift the world towards okay what if we
[08:43] just make the orchestration layer itself intelligent um instead of just like the retrieval piece um but still kind of like give these agents that are getting better and better over time uh kind of like basic tools to be able to access their data.
[08:57] And then you know as people start playing around with these like generalized loops um the models probably weren't very good to actually loop uh for like more than five steps at a time. Um, so you can actually do too much with it and trust the results end to end. So
[09:10] what people started building was kind of more deterministic graph-based workflows. You know, I think you can see a few of these frameworks. We have this thing called like workflows which is like you know event- driven workflows. There's of course like langraph, there's like temporal even. Um, honestly like
[09:22] everything is basically just like you just write some code in a kind of like imperative set of steps and you have pockets of LM reasoning in the middle. Um, and what this does is it introduces some inherent guard rails and determinism, right? instead of like just
[09:36] trusting some like agent which back then was not very good to do the entire thing end to end um you actually define through code the precise series of steps that the agent should take um and then it would be able to kind of like follow step one follow step two maybe have lens
[09:50] in the loop but achieve the task there's still a good class of kind of I would say operational work these days um which is basically kind of a little bit more repetitive that you can and probably should codify in a workflow whether it's like you know for us invoice processing
[10:03] uh kind claims processing or being able to kind of like review and redline a contract. There's like usually some sort of defined series of steps where if you compile it into code, you get a certain level of determinism um versus kind of
[10:16] trusting a pure agent to do it. Basically what we call uh kind of like standardized interfaces for agents to actually access their data. So you know I think u all of you at this point are probably familiar with like MCP right
[10:28] which is basically a standardized protocol for AI agents as they get better to be able to call out to tools. Um and of course like there was also some emerging abstraction around skills which is kind of like moving the programming abstraction layer to English
[10:42] as like the models and harnesses got better. So they could actually start doing more and more tasks. I would say the tool layer also started to standardize. Instead of some like kind of fixed algorithm that you would have to hack together to actually fetch context, you could actually just start
[10:55] connecting to various types of MCP servers available, you know, via uh via various like systems of record, whether it's your CRM, ERP, uh or you know, your structured database, vector database, uh
[11:07] or kind of sharepoint. So the benefit here is, you know, as all the frontier labs started building all these AI agents, um, you could actually start having kind of like standardized interfaces to connect to a lot of this context.
[11:23] And so I think basically this jumps ahead a few steps but we kind of arrive at this idea of like the modern generalized agent which is you know first off uh like agents um have gotten incredibly good at uh coding which means like um you know besides just like being
[11:37] able to actually reason and call tools. They started getting really really good at being able to of course like help you write code. Um what people also realize is you know like writing code um was a good proxy for kind of generalized
[11:50] computer use to start operating your computer. And so even when like cloud code came onto the scene um you know people started using it for a bunch of other stuff besides just coding. uh even today, you know, if you were to kind of
[12:02] uh you know, try to uh basically like reconcile your financials or something, um you can basically use cloud code and it'll write code to actually analyze your Excel spreadsheets, you know, your income statements, your cash flows, um
[12:14] and be able to try to like make sense of everything. Um and so I think one the act of like coding actually really boosted uh these agent capabilities uh or the model like kind of increase in capability in coding. The second is um
[12:27] you know humans started actually started you know kind of writing less and less code and more and more things in English. And what we see is agent building and kind of uh like uh basically like being able to actually solve various tasks sort of collapsing
[12:41] into this abstraction layer where you basically write English and English compiles into both kind of a repeatable task which is a program um or you know you can use English to actually solve something ad hoc. We see this across
[12:54] software engineering too because basically if you write kind of uh something in English right now and you use like cloud code or codeex um you're able to actually compile that into you know production code that you're trying to ship um but of course if you're just
[13:06] a user of these agents you also prompt in English to try to solve a task and so I think the interesting thing here is basically the abstractions for both like prompting and actually compiling an agent have moved up the stack towards uh
[13:18] use you know to towards English and also of course the agents have gone extremely good and generalized with all the core capabilities like you know sandbox layer memory tools uh and a bunch of other fun
[13:30] tricks to start doing more and more things end to end. And so that's roughly kind of where the save the world is today. Um you know all the way from you know how do you actually just like fetch some really simple context and ask a
[13:42] question to this like idea of having a generalized agent that can start doing more and more things. And for us, what we think about is kind of how these abstractions like kind of carry forward as the models get even better and what do you actually need to build around it
[13:54] at the context layer in terms of like the current way that agents interact with your data um in order to actually do knowledge work. the retrieval complexity has moved from kind of being able to fine-tune a lot of the core uh
[14:06] you know like basically create these like hacks around rag for instance and actually make it a little bit manual like you have to specify the specific tricks to you know retrieve a certain piece of data and it's moved a little
[14:18] bit more into this like generalized agent harness um if you have a good enough agent and I'm sure you all see this today with a lot of the coding agents that you use um the agent harness can basically interle even like relatively simple tools to try to
[14:31] compose really good results. For instance, you know, you can get pretty good results if you just run um you know, cloud code over your file system and it'll just use bash commands like ls, gp, and cat and whatever um to
[14:43] actually try to search and iterate over the files. This is obviously not perfect. Um there's still uh we'll talk about in a bit, a bunch of stuff you have to build at the context layer to actually be able to unlock this information precisely, but in general,
[14:56] retrieval complexity has moved from like human hacks into the generalization harness. The idea of like context engineering has also moved up the stack a little bit. Um, if you've heard of the word context engineering, I think in the beginning it was more around like how do
[15:09] you, you know, more specifically find like manage the context window within these agents so that you know they don't have like uh context rot that like they're able to kind of handle the like salient pieces of context within the
[15:21] limited attention that they have. I do kind of see this as also moving up the stack a little bit in terms of concerns. Um, especially as more and more people use these agents. And you see this with the UIUX of like Codex and Cloud Co-Work too. Um, you're going to probably stop
[15:34] worrying a little bit more about how do you actually manage the context and more just how do you actually define the context and use it to help like automate pieces of work and just let the agent itself kind of handle the compaction and and all other stuff like for you. And of
[15:48] course like the last piece which I kind of mentioned is you know um being able to actually define tasks um has kind of moved up to this level of just like writing English. Um English is a great uh and other languages of course uh
[16:01] communication interface uh between humans. Um and it's higher level than something like a programming language. And we kind of see this where you're basically just like defining wikis, runbooks, skills across software
[16:13] engineering and beyond to again both like solve something ad hoc as well as build something repeatable.
[16:25] So I mean just in terms of evolving knowledge work patterns um 2023 to 2025 it's like you ask some simple questions through like a chat GBT wrapper. Um, if you wanted to actually build uh, you know, like an agent, uh, you would typically write a like some pieces of
[16:38] code. You would use our framework. You could use other frameworks, too. Um, you could also use like one of those noode drag and drop builders where like, you know, you kind of like drag and drop a module one, you hook it up to module two, and then have it like basically uh,
[16:50] you know, define an automation for you. Um today I think uh like the product interfaces are probably kind of like uh simplifying and also advancing a little bit where you're basically able to solve like both ad hoc tasks and and kind of
[17:03] like build repeatable programs and workflows through English, right? And we kind of see that happening both um especially with the full version of of like AI tools that use today. I think the one um like big remaining
[17:15] gap, especially as we look forward to the future, is right now you still have to define a decent amount of stuff through a somewhat blank chat interface um to actually be able to do things. Um and of course you could hook it up with
[17:27] MCP skills to do things and you know have access to your data. But I think the interesting thing going forward is how do you actually start solving tasks in a more end-to-end fashion as task horizons get longer from you know like
[17:39] 30 minutes to hours to like LLMs can just like run infinitely non-stop for 247 assuming like cost is not an issue. How do you actually like make use of these agents and what's the best interface? like what's the minimum bits
[17:52] of information you need to prompt these agents for it to do stuff and for it to actually prompt you proactively instead of you having to do you know kind of like figure out how to prompt at every step of the loop.
[18:06] So that's kind of at the you know like general agent layer. Um and I think what we see especially since uh our mission is very tied towards helping to you know unlock context for ever increasing forms of knowledge work. We see kind of like
[18:18] two main patterns emerging. There's of course like the chat interfaces with all what we call like assist of agents. These are agents that start off with like a simple chat interface. Um you know I'm sure you all use um how many of you use cloud co-work by the way? Just
[18:31] curious. How many of you use codecs? Okay, great. Yeah, so it's probably some mix of those two plus like other other AI tools that you purchase. Um but basically they would assist with like open-ended research generation and work.
[18:44] Like you still have to generally prompt it. It's a little bit more open-ended. Uh maybe it's hooked up to your context. um it'll do things for you. Um and you know I think basically it would uh give you back generally some sort of like synthesized uh report or actually be
[18:58] able to complete a task for you and it would be an interactive chat session. we still see kind of like some increased opportunity for UIUX like tuning especially in the world of automation where there is still a massive volume of
[19:10] kind of like repetitive high volume document work especially in the worlds that we operate in like whether it's loan origination KYC invoice processing claims and others where you know I think being able to actually design a proper
[19:22] interface where you can operate a set of agents at scale um to do batch tasks and actually you know be able to review all the outputs in a batch setting and correct them if needed. Um, it extends a
[19:34] little bit beyond a chat interface and a little bit more in terms of like a batch job interface. Um, and so we kind of see that across a variety of different companies that we work with and we see that complementing a lot of the assisted agents that people use.
[19:47] So we talked about how the agent harnesses have kind of evolved over time to kind of, you know, be able to uh do more and more tasks uh for you. And I think one of the missing pieces uh for us really is you know how do you
[20:00] actually equip it with the right context to do things. Um, especially for us, we care a lot about unlocking context within the massive pool of like complex PDFs, PowerPoints, Word docs spread
[20:12] across various organizations across different silos and making it accessible um so that you actually reduce kind of the amount of setup time and learning for any individual to make use of these agent capabilities within any uh kind of
[20:25] enterprise organization. So for us, context is everything. Even as models and agent harnesses get better and better, you could have, you know, like Fable 5, Fable 6 come out, you could have like GPD6 come out or
[20:37] whatever the next model is. Um, but it will get really good, but there's still like blank slates. Um, you know, in the end, you have to do something with it uh to actually uh bring value to whatever you're doing. Um, and the biggest
[20:50] barrier I think is going to become our own ability to actually provide context and make use of these models. Um, I think the more complex the task, uh, and the more complex the specific process you're trying to automate, the more
[21:02] context you're actually just going to have to feed the agent in order to kind of achieve the outcome that you want. This includes, of course, like the input prompts, task specifications, runbooks, whatever. Um, you know, just being able to feed it with kind of like the
[21:14] description of what you want. But this, of course, includes kind of um, you know, the various connectors, tools, skills, MCPs to your existing like organizational information. This includes web search so that you know
[21:26] it's able to keep updated with the latest like kind of state-of-the-art what's going on. This includes of course like structured data within your lakehouse if it needs to do kind of like deeper analytics. And for us you know this includes the massive amount of just
[21:39] like terri like you know just um trillions of pages per year produced of like unstructured document context whether it is in the form of like HTML markdown or in the form like PDFs, powerpoints, excel sheets or documents.
[21:53] So for us we think about how do you actually build um an agentnative document platform you know instead of using like humans using a tool like Microsoft SharePoint Box Dropbox that type of thing how do you actually build
[22:06] kind of some piece of software that's able to provide context to agents right um and also in addition to humans and provide both the right tools infrastructure and of course like the user interfaces and the kind of like uh
[22:18] other surrounding features um so that you know both humans and agents can collaborate on document context and make it available to any front-end agent that you would want to use whether again it is like openi claude gemini or anything
[22:31] else. So you know today um I would say we're focused a lot on kind of the first part which is you know if you look at the different categories there's like kind of the parsing layer the semantic and storage layer agentic layer there's
[22:44] capabilities around like uh document OCR that we're working on um being able to actually make use of kind of best-in-class document extraction document search and starting to get a little bit into the document workflows
[22:56] as you um I think the remaining territories which u might not actually have time to talk about but basically uh kind of include some agent native like document formats. Um as more and more uh agent generated content pr proliferates
[23:09] kind of core management tools that you need like versioning editing especially if agents are kind of doing a bunch of the changes. How do you actually make sure that you have like the right git traces almost um so you can do roll backs and as a human review kind of the
[23:22] changes are made and correct if needed. Um and of course like kind of move from workflows to longunning orchestration tasks
[23:35] for us. Um especially even just solving the first layer is pretty complicated. You know there's a massive volume of kind of unstructured documents out there. Um there's uh everything from like financial research reports to kind of like manufacturing like spec sheets and just like giant purchase orders and
[23:48] uh you know like uh rate sheets for insurance. But what if machines could just like automatically read, understand and synthesize all the information present um even some in some of the world's like most complicated document container formats.
[24:01] Um for us you know we focus a lot of applied research efforts on this um you know there's been uh kind of this like OCR industry been around for like 15 to 20 years uh or actually even longer. A lot of that's based on kind of heristics
[24:14] and traditional machine learning techniques. With the rise of generalized vision models, um both the frontier ones as well as like like you know I think GBD4V was one of the first to some of the openweight models with uh vision capabilities. Now you can start like
[24:27] taking a screenshot of a page and feeding it in um to one of these VLMs to get back uh some sort of like parse representation. There's still a giant long tale of edge cases though. If you actually try to operate over documents at scale and try
[24:40] to digitize, extract all this information, you need to kind of be able to not just like get to 80% accuracy, but oftentimes 99% plus with the right like annotations, grounding, um,
[24:52] confidence scores, and others. Um and that's to what like to some extent basically what we're building you know um to some extent we're basically building uh the best-in-class uh document parsing capabilities at one of the first layers to really help unlock
[25:05] even the most complicated documents out there. Um and you know like tune it for uh uh like basically various use cases and make it accessible for your agents. Um whether again it's for kind of like a chatbot you're trying to upload a bunch
[25:17] of files to a knowledge base and chat over it or for any sort of like automation. So this includes, you know, a fine-tuned agentic harness, being able to auto route between both like frontier models as well as like specialized
[25:29] fine-tuned models from our own research team. Um, and of course like being able to uh actually like tailor these specialized models for like table understanding, chart understanding, and others. This also includes kind of being
[25:41] able to get really deep into the internals of how PDFs work. Um, and I could talk for like another hour on that, but basically just being able to really make sense of the underlying text representations within the document. Um we've also super invested in just
[25:55] like document understanding benchmarks. Uh we think every frontier model um and and like every model development like should be measured uh on some like like better benchmark that actually represents real world complex
[26:07] distributions. Um the reality is most of these like frontier labs out of the box if you use uh GPD 5.5 even like Fable 5 um we benchmarked it and it generally does not do very well. Um, and so, you
[26:20] know, I think in general if you look at like Gemini, Haiku, Opus, um, you know, and and some of these others, um, they're both quite expensive at scale. Um, and they probably get you to like 60 70% accuracy on our benchmark at most.
[26:34] And so for us, we really care about just making sure that as models get better, not only in terms of reasoning, they also get better in terms of just like generalized document and visual understanding so that they can make sense of the unstructured data within the enterprise.
[26:50] Besides kind of obviously some of the stuff we're working on on the commercial side where we actually have like these fine-tuned uh models plus agentic harnesses that are very vision- based. Um we also have kind of some fun open source tools. Um this is something you
[27:02] can basically give it a shot today if you want. It's completely free and open source. It's called light parse. It is the fastest PDF parser on the market. Um and right now as of today it also gives you back markdown. Um I think we just
[27:14] released this like 30 minutes ago. Um and so compared to like PI PDF, PIME PDF, uh PDF to text, market mark market down, PIM mu PDF for LM and open data
[27:26] loader. Um it is like orders of magnitude faster and also kind of tops the benchmarks in terms of accuracy. Um this will not parse the most complicated documents that you have, but it's actually a great tool if you want to plug it in as a cloud skill into
[27:39] co-work. Um because you know the thing is when you like let's say drag and drop a bunch of files in a folder you feed it to claude it needs to be able to understand the files really quickly um and also in an accurate manner. Right now it writes a bunch of code and uses
[27:51] one of these like other open source libraries. Um and you kind of lose a bit of representations under the hood. So chug it out. It's fully open source uh you know kind of no no strings attached. And basically you know if you just want
[28:03] to look at the speed uh this is me or one of our engineers uploading a document. And I mean you didn't even see a process. A process only 003 seconds. So it basically just gave you back the markdown representation.
[28:21] Great. Besides this, uh there's also kind of like being able to actually add annotations and semantics on top of the data. You know, besides kind of actually digitalizing like the documents and making it available as context where you can, you know, then chunk it, embed it,
[28:34] put it to a vector database. Um there's also kind of this need to be able to structure and make sense of documents and and for us we have like these document extraction capabilities where no matter how complicated the document is, you can give in some sort of like
[28:47] input schema like basically what you want to get out of the documents. And let's say you just have a million of these like market research reports or kind of like invoices, purchase orders or kind of like user like tax forms or
[28:59] something at scale. They're all maybe a little bit varied in formats. Uh they might have handwriting, other annotations on it. Um we are able to kind of like give you back structured outputs so that you know you basically extract from all these documents in a
[29:11] highly accurate manner. Um and you know uh regardless of complexity and be able to get back grounding um attribution and semantics. So if you uh kind of extract out for instance like the line items
[29:23] from like an invoice, we'll be able to actually give you back the exact bounding boxes where that line item came from. um as well as highlight the confidence scores. Um you can use this in ETL into you know for instance data
[29:35] bricks um or you know you can actually add it as kind of just additional metadata in whatever like database that you or or just like whatever document management platform that you have to just be able to add additional information.
[29:47] There's also kind of like this idea of document search where um you know again 2023 was like naive topk retrieval in 2026 a lot of the kind of core focus now is basically on this idea of like
[30:00] agentic retrieval where you give an agent a wide collection of tools whether it is semantic search BM25 grap file system access plus like you know kind of read and scroll operations. So it can basically given like a giant box of
[30:12] information it can uh make use of all the tools that both both rapidly search for stuff and also navigate around to kind of uh be able to complete the task at hand. I'm going to skip this really quick but
[30:24] basically what's next um in terms of just like things that are still I would say relatively unsolved and I would say these are kind of open challenges for the rest of this year and beyond. There's both kind of like um you know
[30:36] document OCR I would say is like unlocking like human native formats like there's a lot of PDFs out there but especially as agents start generating more and more markdown HTML what is like the right representation for agents to actually start generating stuff um it
[30:49] has to both be easily readable and editable by a human and also by an agent um there's also kind of this like question around document versioning editing permissions collaboration if you have an agent start like writing a bunch
[31:01] of stuff or redlining a contract how do you like actually log that somewhere where um and I don't think the best answer is just use like Microsoft Word like how do you actually provide the right interface so that agents can uh humans can kind of like roll back the
[31:13] versions if needed actually audit what the agent is doing in a very intuitive easy to use manner and then of course there's also kind of uh just longunning agents on these documents where it can start getting uh acting on just like
[31:25] event triggers instead of chat prompts um and being able to kind of do things in a 24/7 manner. Um, you know, there's a lot of pros and cons to markdown and HTML. Um, I think Markdown is great
[31:37] because it's really easil uh easily readable. Um, you know, it's actually pretty easily editable, too, if you use something like Obsidian. Um, the issue is it just doesn't really uh it's not very good at representing like complex like layouts, right? Um, it's kind of
[31:50] just like uh you can embed some images and stuff. I guess you can embed some like HTML too, but at that point it just gets hard to edit, too. it doesn't kind of have a great interface like you know like Microsoft Word or Google Docs for just like being able to uh collaborate
[32:02] and edit the document directly for visual stuff. HTML um a lot of people are bullish on HTML. Um it's an emerging popular format. Obviously it's like you know web format and so you can store a lot of visual stuff on there. I don't
[32:14] think humans can really edit HTML though. Um you know you you basically can't really go in and just like edit a web page on your own with at least with like the current editors out there. And so I I do think there's still some interesting work to be done. I know like
[32:27] there's kind of some various open source efforts here, but yeah, I mean in general we' be interested in kind of a better agent native document format. I'll kind of skip this really quick. Talked a little bit about versioning
[32:39] permissions and that type of stuff and I'll skip the background agents. I think maybe one general discussion topic that I think is interesting especially for today is kind of cost optimization for any sort of like knowledge work task that you do. You know, there are things
[32:52] I would say require like a high amount of intelligence like solving like doing a math theorem. Um or you know things that honestly like don't require that much uh kind of like thinking um you just kind of need to do it uh cheaply at
[33:04] scale. Um I would say kind of like being able to like review invoices for instance um is not something that requires a high amount of thinking but you just have a lot of it especially if you're a large company and you need to be able to do it at scale. And so
[33:17] especially as Frontier Labs tune models for higher and higher intelligence, I think every organization including you know us and also yourselves uh we'll start thinking about how do I actually get um you know the best value per token
[33:30] or per dollar that I'm spending on the tokens in order to actually optimize my task at hand. Um for us we care a lot about it from a document OCR perspective. Uh we're the most like price performance document extraction tool on the market. But you know maybe
[33:43] for you it's some like endtoend workflow where you do have to think about am I going to use like GPD 5.5 or you know use kind of like GPD 5.4 for nano um or some openw weight model like Quen or you
[33:55] know GLM or something to actually be able to to do the task and um especially as um like long horizon knowledge work from some of these like frontier models gets more and more expensive um you know
[34:07] I I think it starts making sense to and and at that point you're basically trading off between agent spend versus like human labor um it starts making sense if really think about like cost so I mean just a quick note you know we kind of talked about a variety of things
[34:19] but again big shout out to data bricks Um we do care a lot about making sure we unlock the right context from unstructured data. Um so that you know everybody within the organization can even as you use kind of ever better
[34:32] agents can make sense of the massive pools of unstructured document based data within the enterprise. Um, and we're we're fortunate to basically be able to use a lot of like data bricks model serving capabilities to do some of that like auto routing cost optimization
[34:45] so that you basically are able to unblock context at scale from both like frontier models um and also kind of open weight models. Um, and so just again um you know they have a great inference platform available uh kind of capacity
[34:58] and access to a bunch of different like uh models and techniques under the hood. Um and so yeah, I think that's basically it. Um, if you're interested in kind of like, uh, coming to our booth, uh, come
[35:10] check it out. Um, and then we're also doing a giveaway for, uh, a few Meta Rayban, uh, glasses, um, if you want to come check it out. But otherwise, I'll share the slides online in case you miss any details. But thanks for thanks for taking the time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.