Building Agent-Ready Data: Best Practices for Intelligent Document Processing at Scale

Summary

This video presents a framework for turning unstructured enterprise documents — invoices, contracts, call transcripts, and support tickets — into structured, agent-ready data using AI Functions on the Databricks Data and AI platform.
Accuracy, cost, and throughput form a three-way tradeoff when building production document processing pipelines; the presenters share a decision framework and playbook for navigating this tradeoff, including a real evaluation loop that improves extraction accuracy from 92% to 99%.
A live demo builds a contract intelligence pipeline using AI Parse Document, AI Classify, and AI Chunk functions composed into Spark declarative pipelines, with LakeFlow Connect for ingestion and Genie Code for auto-generating extraction schemas.

Building Agent-Ready Data: Best Practices for Intelligent Document Processing at Scale

Roughly 80% of the context that powers your business—invoices, contracts, call transcripts, support tickets—lives in messy, unstructured data and documents. Turning that into a structured data layer is one of the biggest opportunities in your organization: it's what lets you build accurate and reliable agents and analytics on your enterprise's full context. However, when building these massive scale pipelines, going from prototype to production is harder than it looks. Accuracy, cost, and throughput stop being independent dials and become a three-way tradeoff you have to design around. In this video, we'll share the framework and the playbook for winning that tradeoff. We'll show you how to compose AI Functions—Databricks' research-backed, task-specific building blocks for intelligent document processing—into production-ready agentic workflows. You'll learn how to choose the right function for each task, iterate on quality effectively, and tune for cost and throughput at scale. Along the way, we'll demo new features across AI Functions that push what's possible. You'll walk away with a concrete decision framework for building large-scale agentic workflows over unstructured data, plus the best practices to get them into production.
Talk By: Archika Dogra, Product Manager, Databricks ; Nihit Desai, Software Engineer, Databricks ;

Chapters

FAQs

What are AI Functions in Databricks and what do they do?

AI Functions are research-backed, task-specific building blocks in the Databricks Data and AI platform for intelligent document processing at scale. They include capabilities such as AI Parse Document, AI Classify, and AI Chunk, which can be composed into production-ready agentic pipelines for document intelligence workflows.

Why do AI agents struggle with enterprise documents?

Research benchmarks cited in this video found that frontier AI agents — including Claude Sonnet 4.6, GPT-5.5, and Gemini Pro — answered questions incorrectly more than half the time when given complex enterprise documents directly. The issue is not reasoning ability but rather the need to first convert unstructured documents into a structured, agent-ready format before passing them to agents.

How do you improve document extraction accuracy iteratively?

This video shows an iterative evaluation loop where extraction results are reviewed, errors are analyzed, and the extraction schema is refined in successive rounds. Using Genie Code to auto-generate and refine extraction schemas alongside AI Functions, teams can hill-climb quality systematically — the demo shows going from 92% to 99% accuracy.

What is the tradeoff between accuracy, cost, and throughput in document processing?

In large-scale intelligent document processing pipelines, accuracy, cost, and throughput cannot all be maximized simultaneously — improving one typically requires a tradeoff with another. This video provides a concrete decision framework for choosing the right AI Functions and configuration to balance these three dimensions for specific enterprise workloads.

Full transcript

[00:09] I think we're ready to get started. Hi everyone. I know it's been a long day for all of us. So, we appreciate you guys coming out to our session. Um, even though it's in the evening. My name is Archa. I'm a product manager here at Data Bricks. >> And hi folks, I'm Nah. I'm on the
[00:22] engineering team here at Data Bricks. And so for the next 40 minutes, we're going to show you how to turn your enterprise documents into agent ready data at scale and in production. So
[00:34] let's get into things. Quick forward-looking statement. Remember to do your surveys after every session. The feedback is super helpful to us. Um, and let's go into today's agenda. First, we're going to dive a bit into why
[00:47] document processing matters in the agentic era. Then we're going to walk you through a few demos and best practices on how to build production grade IDP pipelines. And lastly, we're going to round things off with a road
[01:00] map of some really exciting features that we're going to be building next. Cool. So, we're all probably here because we have some shared understanding that document data unlocks massive value, especially at scale.
[01:13] Enterprises have all kinds of documents and a lot of it holds some of the richest context about your business. And a variety of different workloads depend on the context that lives in those documents. For example, we see a
[01:25] lot of customers trying to automate business processes and workflows through document intelligence, build search and analytic systems on top of those documents, and then also power intelligent agents for decision-m based
[01:37] off of the context that lives within those enterprise documents. So what actually happens when you give your agents documents today? Our
[01:50] research team a few months ago actually conducted a few different benchmarks on some of the frontier agent harnesses out there. It was Claude uh Claude, I believe Claude Sonnet 4.6, GPT 5.5 um
[02:03] and Gemini Pro as well. We gave it some pretty difficult complex documents like this and asked it to perform different degrees of difficulties of tasks on those documents that require analytics
[02:15] and reasoning like this. And what we found was that when we gave those frontier agentic harnesses these enterprise documents, they were actually wrong over half of the time. And so the
[02:27] first thing that you might think about is like, oh, maybe these agents weren't really good at reasoning over the documents. But we actually took a step back and had a different hypothesis that these agents weren't actually good at reading the documents. Enterprise
[02:39] documents comes in all kinds of ways, shapes, and forms. They have messy tables, charts, images, handwriting. Some of them are scanned. Some of them are digital. And because they come in all these different kinds of formats, we
[02:51] hypothesized that agents couldn't do a very good job at actually reading the document itself. So, we took this hypothesis a step further and we decided to test it and figure out what happens when you give these agents glasses or in
[03:05] other words, you try to help them read that document data better. Here are the steps that we took. First, we parsed those raw PDFs that we used in those benchmarks into structured text using one of our AI functions, AI parse
[03:18] document, which I'll get to in a little bit. Then, we fed that structured document data to those agents. And after that, we asked them to reason over the exact same tasks and we changed
[03:31] nothing else about the agentic uh harness. All we did was kind of change the data layer that those agents read from. We saw some pretty incredible results here across all three agentic harnesses. We saw that our agents
[03:44] actually somehow got smarter. They were able to perform a lot more accurately on those same reasoning and analytics tasks. But not only did they get smarter, they also got faster. We saw that those agent
[03:57] harnesses were using less tokens and making less tool calls because they had a much better and more grounded understanding of the data that lived in those documents when it came out as clean, structured text for them to read.
[04:15] So essentially what this really means is that intelligent document processing is the bridge to turn your messy unstructured files like these ones into structured data. And what that ultimately enables you to
[04:28] do is to build a ton of things on top of that document context from Genie to custom agents to AIBI analytics and dashboards to automating those manual processes that you historically have had in your enterprises to even using them
[04:41] as features to train models for certain uh predictive tasks that you might want to do based off of the document data.
[04:54] So let's make this more concrete. Let's pretend that we're Spark Live, which is this venue conglomeration that's going to be managing all of the venues that the FIFA World Cup is happening at right now. They're in charge of supporting venues across the continent. And this
[05:07] means there's a lot of contracts that detail a lot of the important contexts of these venues and events with food vendors, media companies, staffing agencies, you name it. And it's not just a handful of contacts uh contracts, it's
[05:21] actually millions of them. So, Spark Live wants to turn those ever growing set of contracts into structured data. And there's ultimately two outcomes that they want to deliver here.
[05:35] Once they take those contracts, turn those into extracted terms, they want to automat automate contract extraction, which has historically been a manual, brittle data entry job, and then they want to use those uh extracted values to
[05:49] actually build a contract intelligence agent on top of it. So they can monitor their terms with different vendors over time and detect anomalies and make sure that they're getting the best deals on the contracts and agreements that they're signing. What this ultimately
[06:01] results in is better, faster insights and coverage over their documents. And so where do we start here? Let's say I am a data engineer and this is my task. I want to do document extraction and the
[06:13] first thing that I'm thrown into is this wild west of APIs. There's hundreds of LLMs out there. I don't know which one is going to be the best at my task. I hear good things about some of them, bad things about other ones, and then the
[06:25] moment that I want to try something out, turns out it's going to get deprecated in six months. So, this is over already overwhelming for me to figure out what are the APIs and tools to best parse and extract the key information that I care about and power the pipelines that turn
[06:39] these documents into agent ready data. And so then I decide to choose, you know, one of these LLMs and embark on the PC. So I test this LLM out on, you know, 10
[06:53] documents and I'm like, "Oh, this looks pretty good. Uh, quality is looking good. The extracted values are coming out correctly. Everything seems to be going down the happy path." Then I come back to this PC in a day or two after doing some other work. And I give it 500
[07:06] documents this time and continue vibe checking it. Well, now I'm seeing some weird quality behavior over some new diverse documents that maybe this model didn't see beforehand. And also, I've
[07:18] noticed to get through these 500 documents, it's taken the LM 10 minutes. So, I'm a bit worried about performance, especially because what I'm really trying to do is productionize this on a million documents. And as I look through
[07:31] some of the cost, quality, and performance um profile of the PC that I ran, I'm realizing that this is actually untenable at scale. I don't know if it's doing a very good job. If I were to actually run this at the scale, it would
[07:43] cost me hundreds of thousands of dollars. And I don't even think this job will get done in months. Forget about uh in years. Forget about months. And so I'm really worried here. I have a deadline to hit. I can't go to my boss with this. So I need to go back to the
[07:56] drawing board. So because I'm desperate and I have a deadline to meet, I end up stitching together this desperation pipeline where I'm using all kinds of tools and fallbacks to try to optimize for cost,
[08:09] quality, and throughput all by myself. And I end up with something super complicated like this. And what sucks about this is that I've done all the work. I've made a really fragile, brittle pipeline. I'm holding the pager on it for weeks on weeks. And I can't do
[08:22] this for every single use case. This is a really difficult thing to make a repeatable playbook. So we decided that we would take a stab at solving all this for you.
[08:35] And that's why what really inspired us to build data bricks document intelligence with three of these core functions generally available as of this summit. So we're super excited to share that. Um these uh AI functions are
[08:48] essentially composable building blocks that are optimized for different tasks within your document intelligence pipeline. We have AI parse document to parse the contents of those full documents into structured text just like we saw in those research benchmarks that
[09:00] we ran. We have AI classify that can go and class uh categorize and route documents based off of type extract to actually get the key structured insights that you care about from those documents and prep search to prepare them and
[09:13] chunk them for downstream retrieval and search pipelines and workflows. And we built these composable building blocks with three principles in mind. The first one is that we wanted them to be task optimized and research specialized so
[09:26] that we could deliver best-in-class accuracy with cost efficiency at the exact same time. Then we wanted to be really intentable intentional about ensuring that those could scale in production. We wanted to make sure that
[09:38] these building blocks could scale not just for our customers that were using them on ten to hundreds of documents but also millions. And lastly, we were really particular about ensuring that these were composable. All of these different building blocks are chainable
[09:51] end to end for a simple uh pipeline from the moment you get your document to the key insight that you want to extract from it. And they're also integrated with LakeFlow which is super important. You can use these AI functions today
[10:04] with LakeFlow connect which helps you ingest your unstructured documents and data from a variety of sources like SharePoint, uh S3, One Drive, etc. And then also you can orchestrate these workflows with Spark declarative
[10:16] pipelines to make sure that they're robust and observable at scale. And ultimately all of this is governed end to end in Unity catalog with V with lineage across your medallion architecture.
[10:31] And so let's go a little bit into some of the really hard work that we've been doing to push the frontier of both cost and quality at the same time. So here are some of our freshest uh benchmarks hot off the press. And this is our data bricks document parsing benchmark. So
[10:44] remember AI parse document. What you can see here is that we're super excited to share that on a cost quality frontier, we are the most cost-effective solution out there on the market with also state-of-the-art accuracy versus other
[10:57] vendors and LLMs out there uh in this landscape. And we see very similar results for our document extraction uh building block as well where we have two modes. one of
[11:09] them that we're going to talk about in a bit. It's it's one it's a new one. Um our standard mode and then our precision mode. And with both profiles, we see that we're cheaper and more uh and higher quality against um from some of
[11:21] the frontier models uh out of the box. So this is super powerful. And again, remember the way that we're able to achieve this is that our research team, engineering team, and product team is basically investing the energy to specialize the systems behind these AI
[11:35] functions to be highly effective on the tasks that they're set out to be good at at scale.
[11:47] Cool. I'll give it to Nit now. >> All right. Uh, awesome. Uh, thanks Archa. So, we've shared how AI functions, right? Our research tuned high-quality building blocks residing on top of the core platform
[11:59] components of Spark and LakeFlow can help you build production grade IDP pipelines. Let's see these in action. So in the demo, we'll cover the end to end journey. We'll start with our data
[12:12] set of raw contract documents and build a Spark declarative pipeline that processes these documents with high quality and is ready to scale. And along the way we will touch upon these five
[12:24] best practices for building and scaling IDP on data bricks. In part one of the demo, we will focus on building the pipeline, evaluating and
[12:37] iterating on output quality. And we really think of this as a three-step process. Start with a small sample of documents, build the pipeline, see it working end to end. Right during this step, it's super important to have a
[12:49] fast iteration loop. something that lets us make and validate changes quickly. And then the next step is now to rigorously evaluate quality by comparing generated outputs against a ground
[13:01] truth. And then the third step is to iteratively hill climb on quality. Right? Once we have this representative curated evaluation set with clean labels, we can measure how well we're doing against it and then systematically
[13:15] help line and address issues until we're happy with the desired quality and ready to go go to production. All right, so let's get started with the demo.
[13:27] Before we jump into uh the pipeline itself, I just want to take a moment to show you the kinds of documents that we're trying to process. Right? You could see this has all the
[13:39] messiness of real world data. It's these are scanned PDFs. They have pages and pages of line items. The scans aren't perfect all the time. And not only do they contain text, but a set of rich
[13:52] layout elements, things like signatures, stamps, and such. The first step is to get this data into data bricks. And in order to do this,
[14:04] you can use LakeFlow Connect. LakeFlow Connect is our managed product for injection. We provide a bunch of connectors out of the box, things like SharePoint, S3,
[14:16] and also a bunch of connectors to SAS tools. No matter where your data resides, you can use Lakeflow Connect to first bring that unstructured data into data bricks. For our demo, I've done
[14:28] this part already. And so, we're ready to get started. So, I'm going to go to the agents page and start building my IDP pipeline. Right. The first thing I'm going to do
[14:41] is to point it to the data that we just ingested.
[14:54] And in the background, what we've started doing is processing and analyzing the documents that we've just ingested and we've started parsing them. While that happens, I can start working
[15:07] on my extraction schema. Thanks to Genie code, we don't have to do this manually. We can start with just a highle description of the problem,
[15:23] right? Yeah, we can just start with a high level description of the problem and then Genie code is running in the background to analyze what my problem statement is. take a look at a sample of documents that I've ingested and then recommend a
[15:37] schema for me. I can of course change anything I want in this extraction schema. I can add or update fields. I can update my extraction guidelines, change the types
[15:50] of certain fields. And once I've done that, I'm ready to save and run the extraction on my small sample of data set just so that we can start this quality check and validation
[16:02] loop. Before I do that, I'm going to turn on citations and confidence scores. Uh these are features that we've rolled out very recently
[16:14] and we've seen customers be quite successful with it when it comes to getting a quick quick sense of how is your IDP pipeline doing are the extractions valid are they correct what
[16:27] is the source material in which they're grounded and what subset of low confidence fields might you want to focus on when it comes to providing feedback and impro and improving the
[16:39] extra ractions further. So as you can see uh we've already completed the first round of extractions on this data set of documents and I can now start examining it in the
[16:53] UI. One of the things that uh that where where citations are super helpful is that I can click on any output field and just click on the citations and it takes me to where in the document was this
[17:07] value picked up from. right? What is the source material uh that this extraction response is grounded in? Further along uh you can also see that every field now has a confidence score
[17:20] associated with it. And so I can do this with a few documents essentially validate that the responses you know generally look okay and then I'm ready to proceed to the next step.
[17:34] Now yes like the responses look okay but of course I need a lot more assurance. I want to be a lot more rigorous before I'm willing to say that this is okay and this is good to go to production. In order to do this uh you can use our
[17:48] evaluation and manage iteration loop with Genie code. Uh this is something that uh we haven't yet released. It's something that is new and hot off the press and I'm super excited to show it
[18:00] to you. Uh this is going to be part of our next release. All I need to do is point it to my evaluation data set and just tell it which columns in this data set contain the inputs and the expected
[18:13] or the ground truth responses. And what we're doing behind the scenes is kicking off an evaluation run on top of ML flow. For every single document in
[18:27] our evaluation set, we're running the current pipeline, looking at the extracted response, and then comparing it to your ground truth using an LLM judge.
[18:40] So, we'll wait for a few seconds uh for this to to kick in. Okay. And you can see that the responses are already starting to stream in. While that happens, I can already start examining
[18:53] the responses. uh I can take a look at the generated response and compare it side by side with the ground truth and then further we have some visual cues in the UI to
[19:06] tell you which subset of the fields might you want to focus on because that's where the disagreement it's in this document only there's a subset of fields that disagree with the ground truth and really that's where I want to focus on so all of this is good I can
[19:21] start doing this for you know every single document one by one but That gets cumbersome pretty quickly, right? And so we've gone a step further and once the evaluation run completes,
[19:33] we now ask Genie Code to analyze the run for us and tell us what is it observing? What are some of the systemic issues that it's noticing
[19:45] across multiple documents that I might want to look in and fix? And so for example here it's highlighted uh something about the vendor category uh that might be missing a few values u
[19:58] there's probably something in the total contract value that um we might want to update in terms of the iteration in terms of the extraction guidelines and so on. If I want I can always go in and
[20:10] see exactly which documents um is it sort of surfacing these insights from and then I can always go look at what is the per field extraction performance.
[20:24] I'm going to go a step further and essentially just ask Genie code to see if it can fix these issues for me. Right. So here you can see it's uh here you can see it started to look
[20:38] at analyze the results of this evaluation run and it's now trying to figure out how to fix some of these issues. As part of that it might update the system prompt.
[20:50] It might update some fields in my extraction schema or it might play around with any of the other knobs that are available to it in order to finally arrive at some candidate uh some some
[21:02] some candidate improvements. And so here it's summarized what it thinks is needed to fix, right? Like here it said, okay, there's five specific fields that we need to fix and
[21:14] here's specifically the things that we're going to change for each of them. Um, so I'm going to accept uh the the suggestion, update the configuration, and then rerun the evaluation to see if
[21:27] it actually improved anything. One thing I should note is that every single update to your agent is really just stored as a version. And so you can
[21:39] always go back to the version history to see exactly what changed. If you want to understand exactly what was changed by Genieode in going from version one to two, you can always go take a look here. And also you can
[21:52] restore previous versions in case you make or push any accidental changes. Okay. Oh wow. Okay. Look at that. Right. Uh in one iteration in less than 5
[22:05] minutes, right? We've gone from about 92% in overall evaluation score to north of 99%. And the number of documents that completely pass at 100% uh all of our
[22:19] evaluation criteria is up from about three documents to 14. And you can see how this kind of an iterative evaluation and improvement loop is really how we can systematically
[22:31] help climb quality and keep working on our extraction guidelines agents until we're ready to put this into production.
[22:47] So I'm going to I'm going to this is the first part of our demo. Uh we're going to go back um into the slides. Uh but just to quickly recap, right, like what we learned here in terms of climbing to
[22:59] production quality is that we really think of this as a three-step process. You iterate off of initial signals. At this point, really what's important is to have this fast iteration loop so
[23:11] that you can build, make changes, validate them quickly. Then you want to validate and run a more systematic evaluation using some ground truth data and then really use this
[23:24] ground truth as a way to systematically help climb on quality until you're happy with it and ready to go to production. >> Awesome. That was really great. Um we saw how Genie Code kind of helped you
[23:37] systematically evaluate and then actually hill climb um on your task. But let's go from 50 documents to five million because ultimately where we see a lot of customers kind of get stuck is the scale story.
[23:51] So this is going to be the second part that we're going to focus on which is basically how do you actually orchestrate these bash processing workflows at scale with some best practices that we've seen as we've worked with you know hundreds of customers that have been deploying IDP
[24:03] at scale on data bricks and then also once you get that done how do you then go ahead and monitor cost and performance via spark declarative pipelines and the query insights that you might have access to after running these pipelines.
[24:17] So the first thing that I really want to touch on is what is the platform that all of these building blocks are built on top of. We call it serverless batch inference which is a platform that's optimized for largecale batch workloads
[24:29] and pipelines. So our customers come to us with a bunch of data. This could be hundreds of thousands to millions of rows. And we basically give them three key benefits of using serverless batch inference. The first one is that we are
[24:42] provisionless. They don't need to buy capacity or spin up capacity to get these data pipelines going. Um, we'll do that for them whether it's a big backfill or recurring uh workload. And we're basically going to try to give
[24:54] them predictable completion times without having the overhead of that capacity management story. And so another note of this is autoscaling. So our customers come to us with a variety of different sizes of workloads from again 10,000 documents to a million. And
[25:08] what we're going to do here is we're going to autoscale for our customers with this infrastructure. We're also going to handle parallelization for them. So, we're going to be parsing multiple many many pages at once or many different documents at the same time.
[25:20] And we have retries built in as well. So, if anything fails, we'll automatically retry it a sensible amount of times before we send that failure back to you. This means that you don't have to write any of this custom logic that we see a lot of our customers
[25:33] writing to run large scale batch processing workloads anymore because we handle all of that for you on this infrastructure. And lastly, of course, we're integrated with Spark, so you can use this within existing pipelines, notebooks, and queries, however you do
[25:46] your large scale data transformation workloads today. And so, a really key benefit of this is this 2x throughput improvement that we saw recently from March to April 2026. Because we manage
[25:58] both the underlying infrastructure that these AI functions work on alongside the building blocks itself, we can do optimizations at any stack uh or any layer of the stack. So, we were really excited some major improvements
[26:11] throughput wise just a couple months ago or I guess six weeks ago that saw our customers using the same AI functions get two times faster batch processing speed without them having to do anything or touch any lines of code in their
[26:24] pipeline. So this basically comes to our customers for free. And so then the next question is like okay should I just go and orchestrate this myself or should I use Spark declarative pipelines? And what we do is
[26:36] we really do recommend our customers to use STP for four key reasons. The first one is orchestration. So STP all you have to do is kind of declare what transformations and tables you want at each step of the pipeline and we'll
[26:48] resolve that for you in a DAG. Then we also handle um the you know the risk of reprocessing everything for you because STP is incremental. So we'll make sure that we only process or reprocess new
[27:00] documents or change documents. Also STP has builtin fault tolerance. So any bad rows are quarantined versus you know breaking your entire pipeline if those come through. And lastly is the
[27:12] operations gets a lot easier. So instead of you having to hold a pager at 3:00 am, SCP has built-in retries and recovery along with observability. So it's a lot easier to manage these production pipelines at scale.
[27:27] And so we just talked to you about some really powerful tools, the infrastructure that's backing these AI functions alongside the orchestration layer that Spark Declarative Pipelines gives us. But I want to move into some tips that where we see you have the
[27:40] highest levers to make sure that your batch pipelines run at scale in the most optimal way possible. And so the first one is that we always encourage our customers to reach for these task specific AI functions first. And again,
[27:52] why is that? It's because we've put in the effort to make sure that these functions are really good at these specific tasks, but also performance and costefficient at scale by building specialized systems under the hood. And so this is a quick benchmark of AI query
[28:05] which is one of our AI functions that uh is more general purpose and lower level. So you kind of give it an LLM and a prompt versus AI extract. And we compared uh this LLM with AI extract that were meeting meeting like
[28:18] relatively similar quality profiles. And what we actually saw was that AI extract got the same if not better quality as that LLM on AI query but it was actually three times faster. So again by choosing a task specific AI function you get
[28:32] really good quality out of the box and then you get all these performance gains for free via our specialized optimizations. The second tip is remember a lot of these AI functions today are optimized for batch workloads. So we really
[28:44] encourage that our customers give us sizable batches so that our systems can work as well as they can. Um and so what this graph kind of shows us is that at a certain level um batching amortizes the
[28:56] spin-up overhead. So if you give us like one or two documents, it might look a little bit long because we're spinning up all this compute in an expectation of like a large scale batch workload, but then when you get to like 10,000 documents, you're going to actually
[29:08] reach a pretty optimal uh rows per minute performance profile. And so we do really recommend that if you're going to send documents through our AI functions that you batch them as much as you can um with at least, you know, 10,000
[29:20] documents that you give in any given um AI extract or AI parse document call. And so another thing that we see um a lot of our customers maybe not realize initially is that cutting out excess
[29:34] input output tokens is really valuable in making these pipelines go a lot faster to the human eye. We kind of look at this and we're like oh this is pretty much the same thing like I can reason about this in the same amount of time. But, you know, LLM treat tokens um in a
[29:48] and tokenize these sort of charact weird characters that you might have or excess, you know, words in a way that makes them a lot less efficient at scale. So, if you can even, you know, clean this up from almost 2,000 tokens to 10,000 tokens and then you apply that
[30:01] to millions of rows, you can see that those systems go a lot faster because the functions are only looking at the data that they really need to look at. So, cleaning these um your inputs really makes a huge difference if you want
[30:13] things to go faster. and also be cheaper. And lastly, we have some really amazing features that Nit just showed like confidence scores and citations, but we
[30:25] do have some unbarnished truth here where these reasoning features, if you need them for your workflows because of regulatory or auditing reasons, definitely keep them on. But if you don't need them in production, because this is an extra reasoning step that
[30:37] we're taking um on every single extraction that you're doing, um it'll be a lot faster. So with citations off versus on, we see a 2x difference. Um, still with citations because we've optimized our AI functions quite a bit,
[30:49] it should be pretty fast, but you can get even faster if you turn them off in case you don't need them post quality assurance or pre-production evaluation.
[31:03] And so ultimately what you want to do once you've implemented some of these tips, made sure that you're using Spark declarative pipelines and are reaping the benefits of our serverless batch inference architecture, uh, you want to predict your throughput and cost. so that you know you don't just go blindly
[31:15] run a pipeline on five million documents and then see some crazy surprises there. So there's you know two journeys that we really recommend our customers take. The first one is for cost. We have public pricing calculators on our website that
[31:27] can help you estimate your workloads. um if you kind of provide you know how many tokens you're going to you think you're going to be using per input and what like how how large your schema might be or for AI parse document how many pages
[31:40] you want to actually parse and then we'll give you like a rough estimation of how much we think that workload is going to cost you but you shouldn't go off of that alone we really recommend that our customers actually benchmark their pipeline on either a production
[31:52] workload so if this is something you're running every day just like run one day's worth of data and see how much that costs you or if it's a backfill Again, try to do this on at least 10k documents as a subset of that backfill.
[32:04] And once you do that, you can go in and check the system billing table, which will actually have the job ID associated with it along with, you know, the user that ran that pipeline or that AI function. Um, and that will give you a really good signal of like, you know, on
[32:16] a given day of productionizing this pipeline, how much is this going to cost me? So, those all exist within those system billing usage tables. And then for performance, you can kind of do this in parallel. So now you've run this benchmark on you know 10,000 documents
[32:30] and what you can go is you can what you can do is you can go in and check the query profile uh from that benchmark to see what each step of that pipeline took or how long it took and then you can use that to extrapolate at scale. But
[32:43] remember again we recommend at least 10k documents uh because these are batch optimized functions and so you're going to get that highest performance profile as long as you have a substantial batch to kind of give us.
[32:55] Awesome. I've kind of walked you through some of these things, but we think it'd be a lot more helpful if you could actually see some of these um cost or performance optimizations and best practices uh through a demo. Then it's
[33:07] going to take over.
[33:21] >> Is your mic on? >> Hey, I think we might need the mic on for him. >> Mic on. All right, cool. Thanks. Uh, cool. So, let's pick up where we left off. Um, we concluded just a few minutes back that
[33:35] we're good on quality and we're ready to put this into production. Let me just show you how easy it is to get started with Spark declarative pipelines. All I need to do is go to use agent and with a
[33:47] single click, I can create a pipeline that already encompasses everything that I've done until now, right? all of the iteration steps that I took, all of the different steps that I have in my pipeline, parsing, extraction, storing
[34:01] that output to a table in UC, all of that is already pre-authored for me here. Now, of course, I can make further changes to this if I really want. Um, but this becomes a very good starting
[34:13] point, a good template, uh, for you to do this much much faster so that you don't have to start from scratch. Let me just show you a pipeline that we've already run
[34:30] and we can take a quick look at what does the interface for STP look like. Uh so here I can see there's I've set up a multi-step pipeline that parses my documents, categorizes them, extracts a bunch of structured output from them,
[34:44] and then stores these into a few different locations based on the category or like the tag classification of the document. If I go to the performance tab, I can quickly check all of the different steps
[34:57] that are running in this pipeline. How long did each of them take and how much data did they process? If I want to look into this further, I can step into this and see the query profile and the query
[35:11] query profile will actually show me the full query plan that was executed as part of that step. And so really like with right like just to summarize what we've done as part of
[35:24] the first and second parts of the demo, right? With a combination of Genie code powered evaluation and improvement tool, serverless badge inference and Spark declarative pipelines, you have all of
[35:37] the building blocks that you need all in one place so that you can go from raw documents to a production pipeline all in one session and all in one platform.
[35:51] I'll hand it over to Archa uh to show some exciting new things that uh we've been building out. >> Awesome. So, what we just took you through was this before and after where
[36:03] you started off with wild west of unoptimized disparate APIs. Instead, chose some of our task specific functions that were optimized for the things that you need to do in your pipeline. took us from a PC that initially couldn't scale to production
[36:16] to using building blocks that were optimized for scale from day zero to going from a desperate Frankenstein pipeline that you have to build on your own to this very simple composable and integrated story thanks to AI functions and lakeflow. But now I'm super excited
[36:29] to show some things on our road map coming soon and give you guys some of the first sneak peeks into this. The first one is precision mode and AI extract. So this is really tailored for your most complex difficult documents.
[36:42] So, we have customers coming to us with documents that are 500 pages long. They need to extract hundreds of fields from them. And within those fields, there might even be a hundred more line items. Um, and so we're going to show you a quick preview of what this actually looks like and what's coming soon. So,
[36:55] this is like one of these kinds of contracts. Super super long. There's this table. I don't even really know when it ends. It almost makes me dizzy to look at this. Um, and so you a lot of the times like LLM will give up on a
[37:07] document like this. You know, they'll extract the first three line items and they'll be like, I'm tired. I don't want to do this. So, we're enabling precision mode. And what we're going to show you here is we're going to show you how well this mode does on those documents. And
[37:20] so, same document over here, but in precision mode, you can kind of see that we're doing a pretty good job of actually extracting all of the line items in that document. And so, this is like the scrolling is almost to the effect of the fact that there's hundreds
[37:32] and hundreds of entities that we're extracting here. Um, and everything looks to be like pretty good. There's not a lot of nulls or anything. it seems like we're getting that information out. So, that's what we're really excited about bringing is this is actually like
[37:44] state-of-the-art document extraction that, you know, an LLM out of the box couldn't really do very well from what we've seen across our customers. Um, but the next thing that we're excited to share is governance versus via Unity catalog. So, very very soon all of your
[37:57] AI functions will be securable objects in system.ai, which is our catalog for basically governing a lot of our AI native features, models, and products. Um, and so very easily your admins will
[38:09] be able to grant, execute permissions, and revoke those permissions from users across your enterprise. We're also, we talked a lot about batch processing for a lot of large scale document intelligence use cases, but we see a lot of customers coming to us with
[38:22] realtime workflows that they want to automate. So, we're very excited to announce some REST APIs very soon for interactive apps, streaming pipelines, and even to be used as tools for agents. And then of course there's tons of
[38:34] things that our customers are trying to do beyond parse, extract, classify, etc. Um, and so we're excited to add even more capabilities to our AI function suite coming very soon as well.
[38:46] And so what did we eventually ultimately, you know, cover today? If there's anything that we want you to take away, it's to follow these five best practices for bringing your document intelligence from pipeline from an idea to something that's actually
[38:58] production grade at scale. And we're super excited that you guys all came out to, you know, see some of these new features that we're releasing and learn more about how to use this product. And so if you want early access to any of
[39:10] these upcoming features that we've talked about, um, definitely scan this QR code. It should take you to a form where you can enter your email, your name, and some of the features that you might be interested in learning more about, and we'll be sure to reach out to
[39:22] get you into some of those previews extra early and tailor the product to your needs. So I definitely encourage everyone to, you know, scan this QR code. we won't spam you or bother you that much. It's just me and Nihit out here. Um, and you know, get to know our
[39:35] product better and what's coming up very, very soon. So, thank you guys so much for, you know, taking the time out of your evening to come join us for today's session. Um, and yeah, hopefully you guys are going to be able to go and productionize some of your intelligent
[39:47] document processing workflows very soon and at large scales. For questions, we'll be around here um at the front for a little bit, so feel free to come up as well. But thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.