Intelligent Document Processing: Building AI, BI, and Analytics Systems on Unstructured Data

Summary

Unstructured data transformation: This video demonstrates how to convert documents, PDFs and scanned images into structured data using AI Parse and Lakeflow Connect on the Databricks Data and AI platform.
Scalable AI extraction workflows: Learn techniques for building automated document processing pipelines at scale, extracting data from complex tables and diagrams, and maintaining quality governance with Unity Catalog.
End-to-end demo implementation: See a live example of ingesting unstructured data, parsing with AI functions and creating analytics dashboards on newly structured data on the Databricks Data and AI platform.

Intelligent Document Processing: Building AI, BI, and Analytics Systems on Unstructured Data

Talk by Adam Gurary, Product Manager, Databricks ; Jason Ping (Product), Product Manager, Databricks
Most enterprise data is trapped in unstructured formats — documents, PDFs, scanned images and tables — making it difficult to access, analyze and use. this video shows how to unlock that hidden value by building intelligent document processing workflows on the Databricks Data and AI platform. You’ll learn how to ingest unstructured content using Lakeflow Connect, extract structured data with AI Parse — even from complex tables and scanned documents — and apply analytics or AI to this newly structured data. What you’ll learn: How to build scalable pipelines that transform unstructured documents into structured tables Techniques for automating document workflows with Databricks tools Strategies for maintaining quality and governance with Unity Catalog Real-world examples of AI applications built with intelligent document processing
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

How do I extract data from PDF documents and complex tables?

This video shows how to use AI Parse to extract structured data from PDFs, scanned images and complex tables. Combined with Lakeflow Connect for ingestion, you can build scalable pipelines that handle both tabular and document-based data on the Databricks Data and AI platform.

What are best practices for document processing at scale?

Learn how to build declarative pipelines on the Databricks Data and AI platform that automate document workflows, maintain data quality through Unity Catalog, and create real-time feedback loops for continuous improvement and error handling.

Can I use AI agents for document extraction?

Yes, this video demonstrates custom AI agent-based extraction techniques that can handle complex document structures and evolving requirements without risking production, making it easier to build and maintain document processing systems.

How do I ensure data quality in extracted documents?

The Databricks Data and AI platform provides Unity Catalog governance to maintain quality standards, create immediate quarantine views for invalid data, and capture complete audit trails for all extracted information.

Full transcript

`0:00` Introduction

All right. Hello everybody. It's great to see you all here. Thanks for uh coming to the session about document processing. I'm a product manager on the AI team here at Data Bricks. I'm product manager here on the data injection team at data bicks. And today we're going to be discussing building AI, BI, and analytics systems on unstructured data. We've talked to hundreds of customers just over the last year. And so we'll start off by sharing kind of the typical journey we see when customers try to implement document processing applications.
Then we'll talk about how we're making that journey a whole lot more simple on data brick. Starting with how we manage the ingestion of unstructured data. Then we'll get into how we parse and process the documents at scale. how we make it easy to apply AI onto those documents. And then we'll dive into a demo of ingesting unstructured data and applying AI to it. And finally, we'll finish up with uh with sharing what's next for document processing on data bricks.
Before we get into it, just have to say this session is forformational purposes only. Our road map is constantly shifting as we continue to learn from customers like you guys. And please hold questions till the end. We'll be hanging around uh to chat with you guys after. So, we're going to start. I'm gonna tell you the story of a typical customer journey. And you guys will tell me if any of this sounds familiar. Meet Alex. is an engineer at Brick Air, an aviation company.

`1:19` Customer journey challenges

And she was tasked with enabling her maintenance technicians and their managers with a dashboard to quickly understand the aircraft that they're responsible for and keep the airplanes flying on schedule and in compliance. And so starting with the documents about just one fleet of planes, built a system that ingested and parsed all of those service records. And on top of that parsed data, she used an LLM in a pipeline to extract the key compliance fields like, you know, when the aircraft was serviced and who serviced it and how long it took.
And so with all the Genai tooling you can find out there on the internet, this was actually very easy and actually pretty fun at first. But then when the technicians tried it out, it seemed like any information that was coming from tables or diagrams was either missing or just totally wrong. And so she tried using vision language models like GPT40 or other document parsing services. But she found that she now needed to use multiple tools in one stack just to get the quality and data type coverage that she needed.
And so this took her a few weeks, but she was finally able to get this done. The quality was good and all she needed to do was expand to all of the fleets that Brick Air was responsible for. But when it came time to expand to the data for all of the fleets, she spent hours and hours trying to create a pipeline that could actually execute on such big data. This was hundreds of thousands of PDFs. But eventually, after writing a ton of custom Spark jobs, this was working.
It was finally time to go to production. and she was almost there. But then she's told that there's even more documents she needs to use and they're scattered across multiple sources. Some are in Google Drive, some are in S3, some are in SharePoint. And of course, all of these sources have new documents arriving every day. So she spends weeks writing custom pipelines for each source and built out all this fragile incremental code to ingest and parse it to try to ensure that her pipelines are efficient.
But now she's constantly on call when things break, which they always do, to figure out what changed and where. Schema evolution changes, corrupt files, random changes to thirdparty APIs and libraries she used. And of course, her team now wants an internal chatbot, an external chatbot, AIdriven search, AI powered dashboards, and 17 agents because of course they do. Excuse me. She never expected to deal with any of this. and she simply cannot move this fast.
So, problems are really the symptoms of three main issues. There's a lot of operational overhead associated with building and maintaining efficient ETL processes. There's a very high operational cost of building, managing, and scaling pipelines that ingest and transform data that is coming nearly daily and from multiple sources while also being incremental and efficient. Complex documents are also really hard to parse.
A lot of tools simply don't do well on all documents, especially when the document has tables, charts, graphs, figures, and handwriting. And ultimately, this is just slowing down and making it hard for her to ship document AI applications into production. She just can't keep up with what's needed. So, show of hands. Did any of that sound familiar to anybody here? Okay. Okay. Uh, which part of it was was familiar for you guys?

`4:44` Customer pain points

Different sources, complex documents, different sources, complex documents. Yeah. Any Anybody else? Yeah. Scaling efficiently. Lots of documents. Making sure it it works incrementally and efficiently. Okay. Anybody else? Yeah. Cost. The cost. Yes. Yeah. We've definitely been hearing that parsing, you know, tons of documents and processing tons of documents with those jobs can be very expensive. Well, cool. That's great.
Uh well, over the last year, we've been helping tons of customers like and uh like uh solve these problems in hours, not weeks on data bricks. And today, we're going to show you how. So, I'll hand it over to now to talk about what we're seeing our customers building and how we are helping. Awesome. Thank you, for passing it off. So what we've been hearing from our customers more and more now than ever is that 60 70 upwards to 80% of the data that their company technically owns lives within unstructured messy documents that are really difficult to extract PDFs, PowerPoints, videos, images, screenshots, dog X, anything that's you know obviously not the traditional CSV file.

`5:36` Unstructured data use cases

But what we've also been being told is a lot of these customers want to use and convert these messy documents into business intelligent outcomes. Our customers aren't just. You know, many of you in this room probably are working on similar projects now or have it planned for the near future. I imagine clearly so if you saw our description and decided to sign up for this wonderful session. Generally though, what we've been finding is the most common use cases fall between one of three categories.
The first one is rag. Rag models, building chat applications using LLMs, chat bots, etc. Automating manual processes is the other category. Extracting key clauses from contracts, classifying documents. And last one is driving business intelligence, creating dashboards, for sentiment analysis, or for quality assurance. And over the last year, we've actually been able to help dozens of customers just like save weeks of headaches to accomplish their document processing needs.
Awesome. I'll give you guys a few seconds. Great. So, today we're going to walk through how driving business outcomes from unstructured documents is simpler on data bricks. I want to first introduce you to data bricks lakeflow a unified data engineering solution that could have used to simplify her ingestion her transformation and her orchestration of her unstructured data workflows data bricks lakeflow is essentially our vision for what data engineering should be in the evolving world of data and AI lakeflow consists of three components lakeflow connect for data ingestion lakeflow declarative pipelines for data transformation and lakeflow jobs for workflow orchestration and monitoring.

`7:17` Data engineering with Lakeflow

Let's dive into each one. Starting with LakeFlow Connect. Lakeflow Connect makes it really easy to bring data in with native data injection connectors. These connectors are simple. With just a few steps, you can easily set them up and now you have fully managed injection pipelines. And in so far as whenever we have the ability to actually solve the problem, data bricks will be the one holding the pager. That includes automated schema evolution, data lineage, automatic retries, error handling, automatic connector upgrades when thirdparty APIs change, std type 2 change data feed, so many other more features all centered around designed to lowering your managed uh managed and maintenance costs.
Lake connect is also unified with the rest of the lakehouse. So you can do ingestion in the same place where you do orchestration, analytics, ML without needing to learn another platform. And finally, it's efficient end to end incremental reads, incremental writes, incremental transforms. So that when you process the data right from the beginning, we're doing it as efficiently and as cost- effectively as possible. Now, with LakeFlow Connect, we already support a variety of native data connectors to SAS applications, databases, and most relevant to this talk, file sources that typically handle unstructured data.
So, no longer are the days where an engineer like has to author and maintain custom ingestion API to every single data source that her team gives her, nor does she have to be indefinitely on call for her customcoded pipelines. With our managed connectors, ingesting data into data bricks is now easier than ever. The next pillar of LakeFlow is LakeFlow declarative pipelines. Our solution to transforming data and helping your team build reliable data pipelines with minimal toil.

`9:37` Declarative pipelines

We're offering a simplified authoring experience with minimal coding. We're doing intelligent operations with automated scaling and incremental processing to lower your operational overhead and also a unified batch and streaming experience in a flexible yet cost-effective manner. The last pillar is LakeFlow jobs where you can orchestrate workflows across the entire data intelligence platform ensuring seamless automation from ingestion to transformation to downstream consumption.
LakeFlow jobs provides simple authoring for all data practitioners via either UI clicks or in your IDE. You're getting actionable insights from real-time monitoring as well as a proven track record of reliability. Today, we already have thousands of customers who are trusting and running millions of production workloads to run all their workflows today. And with LakeFlow, data engineering is no longer a patchwork of disconnected tools.
It's a streamlined experience that makes delivering data simple, scalable, and reliable while also easily integrating into the rest of our product offerings. One offering I'd like to highlight in particular is our new AI functions. Our AI functions are data bricks unique DBSQL functions that are fully unified with Mosaic AI to easily apply AI transformations on your data at scale and efficiently. With our AI functions, you can leverage Gen AI directly into your ETL.

`10:57` AI functions for parsing

Your ETL workflows are applying this Gen AI efficiently and only incrementally as new data comes in from your sources. And we have a variety of AI functions available already, but I'd like to highlight one in particular. Today, we'd like to introduce you all to our newest AI function that's coming soon, AI parse document. With AI parse document, users like can easily convert unstructured files into structured formats to quickly extract value from.
And so earlier I explained how LakeFlow connect makes it really simple to be able to ingest multimodal unstructured data from various file sources into data bricks. Well, with a new AI parse document function, you can then transform this ingested unstructured data into those structured formats that are required for your ML and analytics use cases at the end. Using multimodal foundation models, AI parse documents allows you to parse text, extract tables, reason over figures, and by integrating in with LegFlow declarative pipelines, you can do this in a simple and fully managed experience that's scalable, performant, as well as it's a broad format support so that it's one unified API regardless of the source, regardless of the file type.
Now, once your data is parsed, the world really is your oyster. But at data bricks, we've gone one step further to also provide outofthe-box AI tools tailored to the most common unstructured data use cases. Whether you're building rag applications with knowledge assistant, utilizing our agent framework, maybe you're using information extraction or model specialization to automate processes, AIBI dashboards or our AIB genie or our suite of AI functions that cover use cases across the board.

`12:38` Lakehouse unification

Our tools are designed to help you unlock that value fast. And we're streamlining the path from raw documents to productionready AI so you can accelerate that time to impact. What really makes these products special is our unification with the rest of Lakehouse. Your unstructured data workflows are going to be governed by UC, orchestrated via LakeFlow, monitored with Lakehouse monitoring. You can combine them with your structured data to unify insights across both structured and unstructured data sources.
And you can seamlessly integrate into our new suite of AI products and so many more. So you don't need to learn another platform or use a myriad of thirdparty tools to solve a chain of individual point solutions. Instead, you can use an entire you can do your entire unstructured data workloads from ingestion to preparation to the deployment of AI tools in the exact same place where today you're doing your traditional orchestration, analytics, and ML.
And altogether, our goal is to really help you take advantage of the all the reasons you chose to use the lakehouse in the first place and to seamlessly plug into your existing workflows. But don't just take it from us. The two of us obviously work here. We've already started to get great feedback from some of our early customers managing unstructured documents and creating powerful AI tools. We're able to help G3 enterprises create powerful legal chat bots with the high performance of our parsing tools.

`14:13` End-to-end demo overview

And we're confident you all will see similar experiences. Now, instead of just talking about it, I'd love to show you just how simple this can be done today. Great. As an overview, we'll be walking you through an endto-end flow of leveraging unstructured data. We'll show you how to ingest unstructured PDFs. then how to parse the PDFs into a structured format and then extract out the key information we actually care about.
Then we're going to use that key information extracted to populate a dashboard automatically. And finally, we're going to create a custom AI extraction agent. So for step one, we're first going to show you just how simple it is to ingest PDFs from SharePoint using our LakeFlow Connect SharePoint connector. Bear with me. Live demos. You You guys all know the adage. Great. Awesome. Let's imagine I'm that engineer that works at Brick Air and my manager gave me a link to a SharePoint site.

`15:29` Ingesting data with SharePoint

In the site is a document repository full of a bunch of PDFs. Let's open it up and take a look at what we have. Awesome. So, what we see here is we have a PDF of an aircraft maintenance log book that an aircraft technician would fill out. Clearly, you know, we see some handwriting. It's supposed to be a table, but also the person wasn't too great about being able to write within the lines if you look at the top left.
And so, we have a myriad of these different PDFs. Now, let's talk about what it means to actually pull that into data bricks and how we can leverage LakeFlow Connect. Once I'm on data bricks, I'm first going to go to the external data page within our catalog explorer. I'm going to hit create connection. Let's give it a name. Uh SharePoint data AI summit demo inest. Why not? I set the connection type to SharePoint and I want to use OOTH to authenticate in.
Step one is done. For step two here, all I need to do is provide in the credentials. These credentials can easily be found from your SharePoint source itself. Once I fill those out, I simply click sign in and then I can create the connection. So, for the purposes of this demo, I've actually gone ahead and already created a connection here. Next, what I would do is I open up a notebook. And now what or any other user using our layflow connect connectors are first going to do is copying and pasting in boilerplate code that comes along with the connector.
So we can ignore everything in cell one. Again, boiler plate, no adjustments, no modifications needed. I want to highlight and focus on the actual pipeline spec itself. And so here's where or any other engineer would actually have to fill out information specific to their use case. I can give it a name for the pipeline. I need to pass in the name of the connection we just made so that the lakeflow connect connector knows what credentials to use.
I provide the source location on SharePoint so identify what documents to actually ingest in as well as the destination catalog and schema of where I want that data landed once it's in data bricks. then I would simply run the cell. And so I've actually gone ahead and also done that here as well where I've given it a name, a connection type, some information, and I want it landed in a destination table. Once I run that, we're given a pipeline.
So let's take a look at what that pipeline looks like. Here we see a diagramraph and effectively what we've now created just from setting up the connection and running that one cell is we now have a streaming table called documents in the location that I specified that's pulling in data from SharePoint. Now I can schedule this pipeline to run on a regular cadence. Let's say once a day, our connector is now going to go into lickflow or sorry, go into SharePoint, check if there's any new PDFs or if any of the previous PDFs have been modified, and if so, only incrementally ingest in and process those new files and let it automatically into the documents destination table.
So, let's take a look at what this document table itself actually looks like. Here, I just have a notebook to pull it up. And this is the format of it. Effectively, every single file gets landed into a delta table already as its own row. For each file, we're given a unique file ID, file metadata, as well as source metadata related to the location where it was within SharePoint. But I want to focus on this content column.
This is where the file itself actually lives. It's stored as a binary within the delta table cell. And so we have this stored across every single document across these tables. And there we have it. Just from those two steps of giving providing the credentials, giving the pipeline a name and giving a few locations, we now have a streaming table pulling in these files, landing them as binaries within a delta table. But now that we actually have these unstructured binary files, what can we actually do to convert them and parse them into the structure formats we're looking for and then extract out key functions or key entities I really care about using AI functions.

`19:42` Parsing documents with AI

And so we'll show that as well. We have the exact same notebook here. You guys all just saw this. You have a file binary content. Next, all I'm going to do is run a SQL query. And it really is as simple as this. I run our AI function AI parse document. I toss in the location of where the binary content itself actually lives and I save that as a new column called parsed. Super simple works right out of the bat natively within data brick SQL.
And so now let's look at the output table. You know I selected the file metadata as well but now I have this new column called parsed. So what we're provided is first a higher level document summary generated by AI to identify what the document generally is about as well as a title for it. But the content of the document itself lives within this elements field. Here we identified one element element zero and notably it noticed that it was a table.
We also provide a summary of what the table is representing as well as an inferred schema just from the PDF alone of what it believes the table schema should be. Then the content of the table is represented both in markdown as well as in text. So you can imagine how easy it would be utilizing the markdown as well as the inferred schema to then be able to explode this out and you can even land this table as a delta table on its own.
But for the purposes of today's demo, I only really care about this markdown representation over here. So after that, I'm just going to run a very simple SQL function. Nothing here is new. I'm basically just going into that column, going into the sub field, pulling out the markdown representation. You guys probably all know how to do this. It looks a little complex. Don't swe it. Nothing data brick specific. Once I do that, again, I have each file, but now I just have that markdown representation of each document itself.

`21:29` Information extraction

Following this, I want to chain that into another AI function. In particular, I want to use AI extract. AI extract is one of our other AI functions that basically runs a model and we provide it a preset defined set of key information entities that we want to extract out from every single document. In this case, I told it I want the date of maintenance. I want the technician's name. I want to know how long the technician spent.
And I want to know how long the entire maintenance job took. I want those defined as columns. And that's it. I pass in the column PDF text, which was the markdowns that I just pulled out earlier. And let's take a look at that. Now, what we get is another table with all four of the elements that I wanted from every single document for every document in which the model was able to confidently pull out this information.
And if it wasn't, it'll provide it as a null. But regardless of the input, you're going to get a very consistent output. So that way it's really easy to scale this across a variety of documents, even if one stray document that doesn't fit your traditional format somehow ends up in SharePoint, which of course it always does. Now, I just showed you how easy it was to ingest PDFs from SharePoint using Life Connect in a two-step process.

`22:54` Pipeline and dashboard creation

And then by just simply running three notebook cells, I was able to parse the content and extract out key information I actually cared about using our new AI functions. But what does it mean to actually integrate this in with our Lakeflow declarative pipelines and LakeFlow jobs? I'm going to show what it means to actually utilize this to where we can even populate a dashboard automatically and have that dashboard refreshed every time new documents are being pulled in from SharePoint.
Going to go to our workflow workflows page right here. I'm going to click create ETL pipeline. Now we're brought to a brand new UI for our LakeFlow declarative pipelines ETL pipeline editors. So here let's give it a unique name. Uh we can do parse and extract. Great. Let me put this into our location. Awesome. And I can even start off with sample code in SQL. Now, this page is very overwhelming. I know it's probably new to most of you.
It was new to me for sure. Um, don't worry, I'll walk us through it. But effectively, what we're presented now is a sample pipeline connected with a bunch of different sample transformation files as well as a really easy, intuitive, beginner-friendly UI to explain how to actually define those pipeline transformations. And so, I've actually gone ahead and already populated in this pipeline. And I've closed out all the other cool fancy visualizations just so we can focus on what really matters here.
In this case, I literally just copied and pasted over the exact same SQL function or SQL query that we saw earlier. There's no difference other than now I'm simply creating or refreshing a materialized view. I'm saving the parse out contents as documents parsed. Same thing, same markdown extraction you saw earlier saved as a materialized view. document markdowns. And finally, AI extract. Same thing as earlier, but now documents extracted info.
Super simple, but now within the context of a pipeline. We can actually see what this graph looks like. And so this is our pipeline graph. We have the three materialized views that this pipeline is creating. It first creates document parsed, then from that creates document markdowns, and finally the extracted info that we actually care about. And so to little describe about what's happening under the hood, LakeFlow declarative pipelines is automatically figuring out the dependency map between your transformations to ensure that each transformation only runs after the previous one accordingly.
Additionally, there's automatic scaling, incremental processing, meaning that every new run of this pipeline, only the new data that's being streamed in will actually be the ones that are being processed. This ensures that your pipelines are computationally efficient and cost effective. So that way you're not rerunning your AI models over documents that you've already been parsing and you've already been extracting from.
And now if we focus in on this final materialized view that you guys saw earlier, which is that key information that I extracted out, let's see what it means to populate a table from there. So I've gone ahead and already created an AIBI dashboard here, some visualizations. This is all just powered just from that technician time entries information that I've extracted out. And so great, now I have a AIBI dashboard that's powered by PDFs.

`26:09` Workflows and orchestration

But what does it mean to be able to ensure that this dashboard is being kept up to date without me manually having to go in and refresh it or setting some arbitrary schedule to run it? How can I unify the entire process? And so that's when I go back to my workflows page as earlier, but now I'm going to create a job. Let's call the first one ingest PDFs from SharePoint. I'm going to select the ingestion pipeline that we created earlier.
Create desk. Let me add a new one. Now I'm going to attach the parsing and extracting pipeline. Great. create task. And finally, I wanted to update my dashboard. And awesome. So, just like that, now we have the entire workflow chained together. Every single time this job runs, it's first going to ingest the PDFs from SharePoint and check, are there any new documents? If so, it's then going to trigger a parse and extracting job to actually parse and extract just those new rows that get filed in.
And finally, it's going to update that dashboard automatically for us. But let's take a step back. Oh, and see what else we can do here on jobs. Now that I have this job, I can actually set a trigger. Let's say I want this entire pipeline to check for new files once a day. We also have a bunch of different types of schedules and triggers that you can set. And let's say I also want, I don't know, a job notification. And I don't really like being on call.
So, I'm going to have be the one to receive a lot of this. And just cuz I'm a little annoying, I'll make sure that he gets notifications for absolutely everything. Okay, maybe not. I'll leave it just a failures. And I'll be nice. I'll even mute notifications for skipped and canceled runs. So, once I were to save this, it's all set. Ideally, once I create the pipeline, I can set it and forget it. I don't need to constantly be monitoring it.
I have the assurance that the pipeline and the orchestration is going to work properly. And if it doesn't, it's going to bring in the right person into the loop so that they can fix it. No longer are the days in which you have to wake up in a cold sweat and suddenly check to make sure, oh gosh, did the thing I spent months working on suddenly break? It's going to notify you whether it be through email, through your company communication media, regardless of however you set up your notifications.
And all of this is also governed again like I mentioned by Unity catalog. Every step of this way, these are all objects within UC that you can ensure certain people have access to, certain people don't. No need for governance outside of that. Awesome. Let's go back to this dashboard earlier. Some of you may have noticed that there's actually a few null values here, right? And so what we did earlier was we used AI extract right out of the box.

`29:16` Custom AI agents

I didn't even give it any examples of what a good answer would be. I just defined the column names and the model did a pretty good job in terms of pulling out most information. Granted, there are a few documents it wasn't too confident in and doesn't believe that information actually exists in. Now I know that information does exist because I'm the one who made the data set. And so now I'm going to pass off to show us what it might look like to be able to create a custom AI extracting agent tailored to our use case.
Thank you. That was awesome. So I want to take this information extraction to the next level. And I want to customize it on my data to get better quality so that our dashboard is as accurate as possible. So what I'm going to do here is I'm going to switch over to the agents tab. You can see we've added a new tab here in the sidebar. And so the agents tab is our new way to easily build AI agents for common business scenarios.
And I'm going to start with the information extraction agent, which is going to help me customize this information extraction step to be more accurate for our use case. And I'm going to select our document markdowns table. Now this is the table that created earlier that contains the parsed logbook entries with the markdown extracted from them in this column PDF text. Now we have to provide a JSON schema of the content we want to extract.
And this actually generated one for us in advance and looks like this is actually pretty reasonable. Technician hour, time, date, total time. Uh, but I'm actually going to replace it with the exact schema that used earlier, which is the date of maintenance, the name of the technician who performed the maintenance. I put um as an example. The technician time spent, and then the total time spent. And I'm going to give this extraction agent a name.
Let's do aircraft extractor. And I'm going to hit create agent. Now, this usually takes about a minute. Behind the scenes, we're creating an agent that can look at each of these logbook entries and extract these key pieces of information from them in a standardized way. So, right now, this actually is very similar to the AI extract function that showed us earlier. But the idea is that once I've created this agent, we'll have the opportunity to test it on a few inputs and see if we can actually improve the quality.
So, looks like it ran. Let's check out what it did. So, on this example, it looks like we're getting the date just fine. We're getting the technician name just fine, but the technician time spent and the total time spent are showing up as null. Okay, this is this is a sign that maybe it's not able to find it or something's wrong. So, you know, we we see that the numbers are clearly here. So, let's see how if we can improve it.
So, for technician time spent and total time spent, I'm seeing that it kind of decided that it might be an integer and we know that these are decimals. So, one thing I can do right off the bat is change this to a number so that we can hopefully capture those decimals. Now, let's try a few more things to try to kind of hint this model to be a little better at extracting this information. So let's say this is usually the first number you see after the date and it's under the tac hour time field usually.
Let's try that. Um and I'm going to do something very similar for total time spent. I'm going to say this is usually, you know, the second number that it sees after the date and it's usually under the total time field. Okay. So, let's hit update agent. Oh, you're right. Okay, let's fix it. No, that's a great uh that's a great call. Let's fix that. So, this is usually the second number you see after that. Oh, okay. Well, it still worked.

`33:08` Agent-based extraction

Let's hit update agent and see if it gets better. But it seems like it worked anyway. But thanks. That was a great catch. Um, so basically behind the scenes, uh, our instructions are actually propagating to the agent to make it better at performing this specific information extraction task. So, it looks like it's, you know, it's it's doing its job now. It's getting 322 and 329. 3 just fine. So we're going to be able to iterate on this quality and look at examples with this prompt engineering quality loop until we're satisfied with the quality of this agent and then we can replace the AI extraction AI extract function with it.
So what I'm going to do here is I'm going to click use my agent and I'm going to click start extraction. So this is going to be sam giving me sample code of running this information extraction on my actual data. So I'll hit run. This should apply the information extraction on my real data. So let's see what happens. And so looks like it ran and we're just going to get this response column that has this structure of the extracted information.
So looks like it's going great. We have the technician name, the technician uh time spent, the total time spent, the date of maintenance. Now all we need to do is use this customized information extraction step instead of AI extract in our pipeline. So what I'm going to do is I'm just going to copy this ID. This is the ID of our information extraction agent and we're going to need it. So I'll switch over to the pipeline.
This is where showed us calling AI extract to extract those pieces of information with this generic AI extract function. And we're just going to replace AI extract with AI query. We're going to provide the ID of our agent here. And we're actually just going to get rid of this array because this agent already knows to extract those pieces of information. And there we go. It's as simple as that. I can hit run pipeline now. and it'll use this information extraction agent to get even better quality than we were getting with the generic AI extract function.
So, I'll switch over to the dashboard here. This is our dashboard that is using the new information extraction that is more accurate on our data. And as you see, we actually have no nulls here hopefully. Yeah, we have no nulls here now. So, uh now we have a super high quality dashboard that our maintenance technicians and their managers can use to keep their aircraft flying in compliance and on time. So that's it for the demo folks.

`35:40` Demo recap

Oh, thanks. Thanks. Appreciate it. So quick recap. We started by ingesting our messy unstructured data. Then we automated a pipeline that could parse our documents and extract the key information from them like when the airplane was serviced and who serviced it. Then we set it up so that our automated pipeline would populate our dashboard. But it just wasn't enough. We wanted to make sure that the quality of our dashboard or the quality of the information extraction was super accurate so that our dashboard could be super accurate.
So we customized the system on our data so that it would be highly accurate for our particular use case. Now because we've all we've done this all on the lakehouse connecting to the data, orchestrating the data, applying AI to the data, this was all out of the box. Everything along the way, the UC tables, the dashboards, the agents, the pipelines, this was all governable with the ability to grant permissions to those who actually need it with unified security, governance, and cataloging.
Now, for, our engineer at Brick Air, the having her unstructured data on data bricks meant that her whole use case could be solved in a matter of hours, not weeks, on data bricks. So, let's revisit problems when she tried to do this herself. So she faced a ton of operational overhead building and maintaining efficient incremental pipelines to bring in and process her data. She found that the complex documents were super hard to parse and required her to have multiple tools in her stack.

`36:52` Overcoming challenges

And ultimately getting the PC's together, getting PC's was easy, but actually getting to production for any use case was just a total nightmare. But on data bricks, she didn't have any of these problems. With LakeFlow's simplified ETL pipeline authoring and maintenance, with our highly performant and unified AI parse document API and our collection of readily deployable AI agents, she was able to confidently and quickly ship AI products into production.
So you might be wondering what's next for us in document processing on data bricks. We have a lot in the works. So coming soon we're we're doing more formats for our AI parse document function. We've been hearing that customers have a lot of unstructured data tied up in DOCX, PPTX, Excel files. We're working on even easier flows to work with the images and diagrams and charts that might exist in your files so that you can easily process them or retrieve them however you see fit.

`37:38` Roadmap and future features

Now, the resounding feedback that we've gotten from customers about parsing documents is that they only want a human in the loop when absolutely necessary. So we'd like to provide confidence scores attached to every document so that the ones that are likely to have poor parsing quality can actually be routed to a human for manual verification. Now we're also working on more more multimodal AI functions. While we do have this AI query function out of the box that can do multimodal, we're working on even more multimodal out of the box functionality.
And finally, we're working on more connectors to make sure that you can use your data wherever it might be. So whether it's Google Drive, Jira, Confluence and more. So we covered a lot today. Uh you know quick recap, we went through the typical journey of building document parsing applications. We went through how we're making that journey way easier on data bricks from ingestion to document processing and parsing to applying AI to the data.

`38:45` Conclusion

We showed you the demo of how it's done. And finally, we shared the road map. So please scan this code if you want some super helpful resources or sign up for private previews. will be in short. We'll be in contact with you shortly. Uh and thanks so much for coming everyone. It's uh it's great to have you guys here.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.