PDF Document Ingestion Accelerator for GenAI Applications

Summary

• Unstructured Document Challenge: Financial services customers encounter PDF and image documents with varying quality and complexity, requiring scalable ingestion solutions for GenAI tasks.
• Optimized Streaming Workflow: Databricks presents an optimized structured streaming workflow for complex PDF ingestion using Apache Spark optimization, multi-threading, PDF object extraction, and skew handling.
• Solution Accelerator: This video presents the unstructured document ingestion accelerator designed to scale PDF processing for entity extraction, RAG-based Q&A, and downstream GenAI applications.

PDF Document Ingestion Accelerator for GenAI Applications

Talk by Qian Yu, Specialist Solution Architect, Databricks
Databricks Financial Service customers in the GenAI space have a common use case of ingestion and processing of unstructured documents — PDF/images — then performing downstream GenAI tasks such as entity extraction and RAG based knowledge Q&A. The pain points for the customers for these types of use cases are: The quality of the PDF/image documents varies since many older physical documents were scanned into electronic form The complexity of the PDF/image documents varies and many contain tables — images with embedding information — which require slower Tesseract OCR They would like to streamline postprocess for downstream workloads In this talk we will present an optimized structured streaming workflow for complex PDF ingestion. The key techniques include Apache Spark™ optimization, multi-threading, PDF object extraction, skew handling and auto retry logics
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and goals
• 2:34 Importance of unstructured data
• 6:21 Architecture and building blocks
• 9:31 Parsing and inference strategies
• 19:23 Decision-making rubrics
• 25:07 Solution accelerator overview
• 26:50 System design and performance
• 29:12 Demo of the accelerator
• 33:12 Future roadmap and conclusion

FAQs

What challenges do financial services face with PDF ingestion?

PDF documents vary significantly in quality (many are scanned historical documents) and complexity (many contain tables and embedded images requiring OCR). Financial services customers need streamlined approaches to process these documents at scale for GenAI tasks like entity extraction and RAG-based Q&A.

What techniques does the accelerator use for PDF processing?

The solution uses Apache Spark optimization, multi-threading for parallel processing, PDF object extraction to handle document structure, skew handling for load balancing, and auto retry logic for reliability. These techniques optimize the ingestion and parsing pipeline for complex documents.

What is RAG-based knowledge Q&A in this context?

RAG (Retrieval-Augmented Generation) based knowledge Q&A enables systems to answer questions about document content by retrieving relevant information from ingested PDFs and generating responses using AI models. This application requires clean, structured data extracted from source documents.

How does this accelerator support GenAI applications?

The solution accelerator streamlines the ingestion and processing phase of the unstructured data lifecycle for GenAI. By scaling PDF processing efficiently, it enables organizations to build GenAI applications that leverage large document collections for entity extraction, knowledge bases, and Q&A systems.

Full transcript

0:00 Introduction and goals

Slight change in the title of the talk. It was says PDF. Um but we added we changed unstructured because it's there's actually other type of document we want to support for our solution. uh for example image can be a version of PDF or it can be a DOCX another so a late change but but you know the main format is still the PDF uh document because that's really the most pervasive uh file type so the topic of my talk is unstructured document ingestion accelerator for gen application a little bit about myself um I am a specialist a solution architect focus on geni and data science MML.
Um I joined data bricks a year and a half ago. Um my background is in data science and machine learning engineer. Um apart from AI and big data I'm a soccer fan. I support Arsenal. Any Arsenal fans? Any uh Tottenham fan or Man United fans? Okay. fans. So, we're gonna stand up and sing North London Forever. Just Just kidding. Um Um So, I'd like to pay tribute to my team, uh Jazz and Yan. Jazz is a lead SSA in my team and Yan is the leader of our group.
So, without them, I won't be here. Um there are two main goals. Um in the first half I would like to uh share my perspective on the importance and challenges of unstructured data um PDF in particular and also the life cycle of unstructured data in the world of geni. um also how that on data bricks then I talk about the solution accelerator we're building um the solution is aimed to solve the problem of scaling in particular for the ingestion of parsing part of that life cycle and for a batch workflow.
Um it is available now. Um um why unstructured data? um because it's enormous. Um this picture is a little old I think. Um but analogy of unstructured data as a hidden iceberg is is very resonant. I'm pretty sure it's to a lot of you. Um if you think about it, the knowledge of LM is essentially trained and learned from open source uh unstructured data, right? Um um in enterprise however uh this huge potential hasn't been fully discovered I think um so there's a lot of potential and the discoverability for unstructured data in the enterprise world I worked with um many financial service customer because um that's actually where my group is belong to Um but I think the type of unstructured data actually is pervasive across multiple industry.

2:34 Importance of unstructured data

So I'm listing so many unstructured data under the financial service uh with those subverticals. Um in banking you think of financial statement, balance sheet, receipts, um insurance claims, policy documents, transcript and real estate with loan document, loan application, appraisal. How many of you did a refinance lately? Or I guess interest rate is too high, but last time I did a refinance, it was it was very painful.
Um so PDF format is still pretty dominant format here but there's other format um I do want to mention those sometimes it's very important as well um how can we extract intelligence um from them so let me show a more concrete um example um so imagine you have a stack of receipts and invoices in PDF some may be a scan PDF um to get intelligent of it. Maybe the first step you want to ingest them and also get the metadata and content parsed um in a table form.
Um here the content is in markdown tax. The reason markdown tax is is very useful is because LM really understand markdown as a language. So if your downstream is performing some kind of AI task, uh markdown format is is going to help. Um then you could further maybe um extract some semistructure data like JSON to get some key information out. Uh for uh for this invoices you can get itemized um uh quantity or or price and those out.
One way for downstream is you can actually transform this into actually a structure data, right? This can be used for aibbi dashboard or for machine learning model. Um but but ultimately I think the idea is you want to ask a question like this. What is my cash flow outlook? Um assistant or application can intell your cash flow situation. the trend is positive or there's no warning and it can also give you a visualization, right?
That would be great. Um how do we harness uh unstructured data? Um I'm a visual person so I'm going to show some visuals. Um the first diagram is a vertical stack. Um I think um there's those are the key building blocks you will need to harness the power of unstructured data. Um at the foundation the data lakehouse architecture is where you can store right unstructured and the structured data with this unified uh governance layer with security and more importantly you can discover them.

6:21 Architecture and building blocks

You can find the lineage. Say if you transform unstructured data into structured data, you want to know where those data coming from. Then on top um you want a very robust data engineering um pipelines capabilities um with serverless compute that support batch um real time or or streaming jobs. Um it is no doubt um I believe data bricks platform is is the idea platform for these two foundation um blocks then it is the parsing and the processing technology and the middle blocks these two orange blocks um we see that typically there can be a a two-step process or or a singlestep process um I think two-step is more mature um sort within this two-step process.
First is you're doing a parsing. Um depends on the complexity of your document. You can use an text parser or it can be a OCR object uh object detection parser or can be a VRM as well. Then on top you perform inference task such as key value key information extraction. That's what KIE stands for. You can do Q knowledge QA translation and ETA. Um then with the latest sort of stateofthe-art that's acronym below um BLM um prompt engineering oneshot can work as well but that will require um figure out the right prompt and maybe transform or split the document into uh convert the PDF into images.
Um so there a little bit pre-processing and some engineering work as well. um after model inference evaluation is critical. I think um in his keynote he mentioned about why we think evaluation is important. It's really that's you really want to know whether what you build below that is really ready for production right so so evaluation layer um with MLflow evaluation LM judge on data bricks all are those capability will be helped and lastly you have the application on top so I'm g I'm not going to spend time talk about the bottom two layer I feel u most of the audience are familiar with um um lakehouse architecture and our our platform.
So I'm going to double click on the u the middle to the orange blocks. Um we're not short of parsing technologies. Um as you can see actually there's many right. So on the right hand side I have a circle of open source ones like tessorax OCR for Google was very popular and we have pi mu pdf and dockling is something quite new recently from IBM. Um they all has their own specialties. Um on the left hand side we have the SAS version. um um list a few like unstructures actually our data brick partner uh Azure doc intelligence and llama index I think yesterday uh llama index had a keynote that or had a talk that talked about their parser so their bricks also introduced uh we're introducing a parser called AI parse there was another talk on that I'm gonna uh mention that towards the end so you guys can check them out um yeah So in my perspective, you know, determine which part is the best is is really not the right question to ask.

9:31 Parsing and inference strategies

Um because for your situation, for type of use cases, all these tradeoff are very complex. It's it's not always good and bad. So I thinking we should consider in in terms of consideration of those trade-off, I list a few um but there are actually more. Um so there's process time. What's the time it takes to process each page that determine like can you actually finish processing within certain time frame right the accuracy the cost the compute um and also the flexibility in in downstream AI task.
So there are a lot of considerations after parsing the next step is inference with LM or or the VLM. Um with data brick model serving um we have a lot of options. Um with our model serving we have foundation models. We also have AI gateway. You can bring external models. Many of these models I listed here are available to use for data bricks users. cloud model for sonic 4 3. 7 they're very good LM as well as they also have modality for images uh llama 4 it's also support images 3.
370B is a very general purpose LM GPT41 and 40 are both multimodal models and Gemini 2. 5 pro are also multimodal models um all these option are there but with unlady model access doesn't really give your use case complete because there are other things need to be built around it, right? Like if if you want to do information extraction, you want to make sure your output are conformed to the schema you define. So it's not always one step.
That's why um we have built data brick build a set of AI tools. Um I'm highlighting couple of really important ones. The first is AF functions actually is set of UDF functions in SQL that you can talk to LM directly and allow you to perform a very high efficient batch inference. Then the agent breaks um maybe if you heard from the keynote are task specific agent for specific business use cases whether it's information extraction or knowledge assist which is essentially a rack agent. um and model specification for fine-tuning.
The trade-off is again there are trade-off of time, accuracy and cost. And as mentioned, we're going to help you to have find that trade-off. Um these sort of a highlight of double click on those middle two blocks. Um you had seen the vertical vertical view. Now I'm going to turn 90°. Um let me me show a horizontal view. Um so this is a end to end view of unstructured data life cycle on data bricks. Um starting from the left um you can have multiple type of unstructured data being ingested in raw format. um um had a model little model here in the in the in that box is potentially you want to have some intelligent ahead of time for you to triage the type of document if you do have metadata available you can potentially classify it and determine which task or which string you want to go right that's that's the beginning you in the bronze layer you have those raw data ingested in the bronze table Next block in the silver layer you have option of using parsers or models to extract the content out of documents.
You can use OCR models. I also added AR parse here. There also a VLM or text parser here. That's where your civil yeah that's where you get the information or the content from the document out. um branch out in the gold layer. Um the top half is sort of a batch workload. Um on data brick we have AI functions as I mentioned previously that allow you to perform those tasks of information extraction or translation in the batch format.
On the very top this this robot sign is the agent bricks. those agents are also designed to perform those tasks. Um, typically after the batch workload, the downstream application can be a dashboard or a genie or it can be a table used by other system or by other internal stakeholders. Right? The bottom half is a real time type of workflow. Typically you want to start to building an knowledge bot or an agent. So this code layer typically a chunking strategy going to apply based on what content you extract.
You want to see whether what kind of chunking approach I'm going to be using to ensure that my rag bot can get in the right information. For example, what's your overlapping or what's your chunk size? You chunk by page or chunk by chapter or chunk by markdown headers. Right? Those are the strategy you want to iterate. Um typically the output that agent will go to a app and on data breaker will also log what agent produced.
So you have a way to monitoring what's happened with agent. Then around that you do want to build the evaluation malflow evaluation block that give you that dev loop. So you can continue iterate within the goal and evaluation until you reach the quality your ML flow evaluation give you the metric to the level that you're satisfied for your application. Then you can deploy uh to end users. Um again at the bottom these two layers I kind of mentioned the foundation like the server serverless compute with those tools for the data engineering as well as unit catalog where we govern all kinds of asset other than just tables datas.
We also um govern indexes models agents and the tools for the agents. I hope this um overall architecture diagram give you guys a good picture of what unstructured data life cycle is from the ingestion all the way to the application. Um despite you know all these technology um here are the steps um simplify it into those block diagrams. Um there are still challenges um because you know devils are in the details the implementation itself is is not straightforward.
Um we see three main challenges. First actually it's the trade-off. Um I think trade-off itself is very hard because you have so many different factors. Um so it's hard to make that decision. Um the second challenge is the ingestion. Um we have a lot of great AI engineers, data scientists while analyzing uh unstructured data but they had a hard time getting the those data in or if they have a huge volume it's difficult to establish that pattern to have those data in.
That's where we hope our accelerator can can help and add value to your workflow. And third is is really the evaluation and I think this is one of the strengths of data bricks. um because we have his we have a very I would say a leading technology in ML MLOps traditional classic MLOps we want to bring that rigors of MLOps with LM ops to enable this evaluation uh dev cycle so that you know what you build whether it's agent or the application for unstructured data is reliable and is production Ready?

19:23 Decision-making rubrics

Before I go to um before I go to the uh u the accelerator, um want to spend a few minutes maybe talk a little bit about the trade-offs. Um I think trade-off um I find it's really hard. Um probably because I'm a person of indecisions. Um, so I like to use a rubric or a decision tree to make that decision. So, um, actually when I start to prepare for the presentation, I create a decision tree. Um, but it's too big. It doesn't fit into the slides.
Um, I I I kind of iterate like three times. Um finally um was feedback from my boss and uh say you you cannot do this for presentation. Nobody gonna read your small font. So uh so I decided to simplify it. So I would say boil down to two key decisions. Um the first decision is um what is in my document or what do I want for my document? Um starting from the left hand side um the simp simpler case if you only have text or I only want a tax actually it's it's this is more straightforward you don't necessarily need any fancy parsers you can just use naive parse you can or using OCR parser you can you can fairly quickly parse the document for many rack use case uh these can be applicable um or if your PDF document is quite simple, right?
So this is this is this is this is can scale really well. So you don't you actually can build a simple pipeline can scale. Um but if your document has text as well as a table structure and you do need to maintain that table structure and for example if your question in your knowledge bot require you to identify a row a column in that table. you wanted that table structure to be maintained or your key value extraction agent be able to get those relative relationship from that table out.
You do need to have that. Then you will need at the minimum an OCR plus object detection where the bounding box of the table in the PDF file can be identified and the table structure is maintained when you parse out. Then the third branch if you need more than that you want everything from the document you want a table you want a text you also want an image you then you want then you need something more powerful so there are multiple solutions data brick AI parse is one of them um one of the great thing about data brick AI parse is it not only parsing the components from the document out but it's also parsing the contextual information meaning that you can get information from where each component belong to what page.
So a lot of time these are very useful um for this middle branch and the third branch that's where our solution accelerator is currently uh built for um we hope to help scale help you scale that pipeline. And the lastly of course there's always an option on table that you can you can um using prompt engineering to on getting the top models to do prompt engineering VLM on that but there's a little bit more cost um getting the document be ready for the VLM is not always straightforward.
The second decision is to me is what do you want for your application? So there's three common types what we see from the gen world. The left hand side is is probably mostly happening last year is the rack, right? The most popular thing in the rack is this knowledge bot Q&A. But I think starting from the late last year to early this year, we we see a lot of people are doing automation. Essentially aging can orchestrate and automate certain workflows that um bring to like you want to you build a build a workflow that automate to extract some information or you want to convert some text or you want to do some translation.
Right? Then the third one is to me is to getting the structured data into sorry unstructured data into structured format. Then you can perform um dashboard analysis and so on. Um I I still want to highlight some of the outof-box tools here. Um for each of those work stream knowledge assist is our uh rag agent. Agent framework is for that evaluation. Then the information extraction is our key information extraction agent. model specialization is for um fine-tuning models for maybe for converting certain format into another format or um generating certain information from um unstructured data.
Then lastly, the AI functions for batch inference and the AIBI uh dashboards. Okay. Um, with that I'm going to jump into the accelerator next. Um, so the main goals of our solution is to really to enable a scalable, reliable and high performing batch workflow. Again, it's it's a batch workflow. Um we want to use that workflow to help customer ingest and structure data and parsing it. Essentially you can think of is the first two steps of that life cycle.

25:07 Solution accelerator overview

Um while we focus on that we we do we think um last year a lot of customer discovered how powerful LM is how interesting you can build a quickly build a rack get information out but what they having trouble is how to bring the key data they want to get information out into the into the system so they can easily build the agent on top of it. So and sometimes a lot of those data are in high volume. That's that's what we think we want to solve that problem or help customers solve that problem.
Um the architecture is straightforward. Um again mapping to the first two step two two layer ingesting unstructured document as a raw binary in bronze layer. Um also capture metadata. Um the down um then in the server layer we parse the content out save as a markdown tax. Um at the downstream you could choose choose your path or choose your application whether it's it's it's a rack for you do you perform chunking strategy or you want to do information extraction then you do build agent for that.
Um, we built a pluggable parser framework. Currently, it support the open source OCR and object detection model as well as the data brick AI parse model. Uh, the framework is flexible. Um, we will add more parser um in our future road map. Um, how do we manage large files? Um, the key design principle is really bucket the documents. Um so we give user option to set threshold. One option is set threshold manually based on the file size or based on the page count.

26:50 System design and performance

Uh as you're set up the bundle you had a choice of choosing um your threshold if you want to set that. If you don't set that um by default you would just do a statistical three standard deviation to get outliers as the large files. Then we also have the um optional microbatch being automatically optimized to avoid skew using salted um spark approach um performance um it's linearly scale with the cluster resources uh concurrency with microbatch uh with thread pool um we didn't put any hard number here um the reason one is some of the data are testing running our uh confidential data from the customer we don't want to disclose that then second is the variety of PDFs is so much I I think even though we tested various different kind we feel like we didn't test all of it um there are so many different kinds I think the number we going to say it's going to representing generalize across PDF is probably not not the right thing to do um but we're pretty confident we are building a solution that can really help add a lot of value to the customer um in just fairly wide range of uh PDF format.
The error handling is pretty standard. We have virtual logic with exponential backoff and false error logging to another table. Potentially you can use that for audit trail. Going to do a short demo on that. Um, learn from the past. I should not do live demo. So, I recorded but I couldn't find my mouse. Okay. So, where's my mouse? Do you guys see my mouse? Oh, this angle is difficult to see. Oh yeah, thank you team. boy. you can give a project name after that you choose cloud provider currently we just support AWS and Azure you can define your threshold I define 10 page more than 10 page then it's a large file for this particular demo uh it takes a little while to uh to have this python wheelbased bundle file being generated after generated you have certain configuration file you can config the workflow but here I'm showing an example where I define the workflow exam uh workflow configuration as uh variables catalog schema um volume the number of workers you want to use um what kind of parcel you want to use after that you can deploy it on as a workflow on data bricks take a little bit to uh upload it.

29:12 Demo of the accelerator

Now I deploy it you can launch um the workflow on the command line. Um that's one way to do it. I think next going to show the workflow. Um so this is the toy example but we tested hundreds thousand of files. Um you get for dev or product product you have a async workflow for async large file then you have a regular workflow for the general uh batch files. This is based on structure streaming. So you have uh bronze job layer.
So in this case majority of time is actually um bring up the node uh the parsing because I have very little file. So the parsing is really quick. Then this is showing the large file has its own separate node and has its own separate string. For example, if you have like 5,000 jobs, 10 are really large file. They're going to run on their own, but they're not going to block the rest. This is showing the raw file being ingested.
You get the raw binary as well as the size and the time stamp and the page count. Page count is blocked. I think it's going to show to the end. There's a page count. There's one large file has 16 p 17 pages. So that would be a because I set the threshold to 10. So that's the large async job. Then after the server layer the markdown content is being extracted. Um the table format is being maintained. uh for example in this case where you have a invoices you want to itemize each itemize to be to be listed within a table so they can be inferred by the LM thank you all for guiding my mouse probably need needed help again I think I can do it this Oh, I'm gonna jump to next one.

33:12 Future roadmap and conclusion

So, it's coming. So, we are continue working on this. Um, we're just getting started. So, we want to add memory management. Um, we also want to further enhance the processing performance especially for those large file. Currently the large file just submitted into uh async work work uh async jobs and have its own dedicated uh compute. Uh you can set that computer to be larger than than the other ones but ideally we want to optimize that large shop maybe using map reduce approach to maybe split the document and bring them together.
So this is something we're currently working on. We want to integrate new parser and the VLM options. And lastly, um it's our innovative project. We're trying to see whether we we want to introduce a declarative document parsing which is an idea is very interesting. Um is that instead of say I want to parse everything from your document. If I already know something, let's say you have 10K from a company report, I only want the revenue or I only want a certain part of that document.
Can you using declarative language to identify uh where I want to parse So, a lot of things coming, but we're working on it. And call to action. Um, try our, um, repo link I'm sharing here and, uh, reach out to us. This is my email. I want to shout out to AI function which I think is a great feature for batch inference. Also, this talk, this is a talk I mentioned about the document intelligent document parsing the AI parse team are doing.
Um, so I think it's maybe ahead of this one, so you might have to check out the recording. Um, perfect. I have five minutes for questions.

Learn more about the Databricks Data and AI platform.