Skip to main content

Developing and Deploying Databricks ai_parse_document for Intelligent Document Processing

Summary

  • Intelligent document processing: ai_parse_document evolved from a hackathon project to production-grade tooling that extracts text and image data from complex documents like receipts.
  • Visual language models technology: Backbone of ai_parse_document uses multimodal visual language models trained for complex document understanding at enterprise scale.
  • Production-ready AI functions: Part of Databricks' expanding AI functions suite, enabling scalable document processing with improved accuracy, robustness, and cost efficiency for enterprise workflows.

Developing and Deploying Databricks ai_parse_document for Intelligent Document Processing

Watch: Developing and Deploying Databricks ai_parse_document for Intelligent Document Processing
Databricks researchers and developers walk through the new ai_parse_document functionality and its applications in intelligent document processing. Hear how it developed from a hackathon project to a production-grade tool and the research behind the technology. A live demo showcases how ai_parse_document can extract data from receipts.
Learn more in Conner Schiisler's blog post, OCR Made Simple with Databricks AI Functions: https://schiiss.github.io/blog/genai/ocr-made-simple-with-databricks-ai-functions/
Speakers
Adyasha Maharana, Research Scientist at Databricks
Archika Dogra, Product Manager at Databricks
Conner Schiissler, Data Architect at Plains Midstream, Databricks MVP
Fanghan Yang, Staff Software Engineer at Databricks
Ziyi Yang, Staff Research Scientist at Databricks
Host
Denny Lee, Director, Developer Relations at Databricks
YouTube Chapters

Chapters

FAQs

What problem does ai_parse_document solve?

It enables intelligent extraction of text and image data from complex documents, moving beyond traditional OCR to understand document structure and extract structured insights from unstructured content.

How does ai_parse_document work with ai_prep_search?

ai_parse_document extracts data from documents; ai_prep_search then chunks the results and prepares them for embedding and vector database ingestion for retrieval workflows.

What is the technology behind ai_parse_document?

The tool uses multimodal visual language models that understand document layout and content, trained on diverse document types to accurately extract data from complex enterprise documents.

How can organizations use ai_parse_document in production?

This video demonstrates receipt parsing as a practical example; the tool scales to handle large document volumes for use cases like invoice processing, form extraction, and document classification.

Full transcript

0:00 Introduction to ai_parse_document and the Team

Welcome everybody to the Adriix community webinar. I'm the host for today's wonderful webinar. We're going to be talking to about AI parse some a really cool functionality from data bricks. But before we get into it, I did want to have introduce the team here. We've got a wonderful group of the from RD and research to talk to a little bit today about the science the engineering behind what we're doing as well in terms of when it in terms of AI pars document.
So saying that I'm going to start off with the PM Archa. Why don't you introduce yourself first? >> Yeah, of course. Nice to meet everyone. Um I'm a product manager here at data bicks working on our AI functions. Um and our team is working super hard to make intelligent document processing on data bricks endtoend a really delightful high quality and scalable experience for our customers production workloads. So really excited to chat about AI parse document and some of the other functions that we're building out over the next few months um with everyone here. >> Perfect.
Thanks very much Archa. All right then let's go next with. Why don't you introduce yourself? Love the fact that we have some great folks from research to talk about intelligent document processing. love to hear a little bit more about your background. >> Hey everyone, uh I'm a research scientist uh on the data bricks team and I work on the AI price document product. Uh during my PhD I worked on uh visual language models and I have continued doing that.
Uh the backbone of the AI pro document are visual language models. Uh so I've continued my work here on how to make them uh work for uh complex documents. That is so cool. Thank you very much. Z, you're next. And again, another awesome researcher. Love to have the fact that we've got you all joining us. So, please Z, introduce yourself. >> Yeah, thank you. Uh, hi everyone. I'm Z. Uh, I'm a research scientist at data bricks.
Uh, where I focus on multimodel, large model, uh, like development on document intelligence. uh have been working on uh this stockman uh model development uh work behind AI pars uh including model training, data quality efforts and uh evaluation. Uh so over the past few months our team has been pushing toward uh more accurate, robust and cost efficient document understanding for uh enterprise workflows. Uh so excited to be here and share more about this and answer questions today.
Z thank you very much I love the fact that you're calling out it's not just about the accuracy and the vision models that Aisha called out but also the fact that we make it cheaper that's really cool that's a very important concept now we this couldn't all happen without actually engineering support so without further ado can you please introduce yourself >> yeah sure hi everyone um I'm a software engineer working in data bricks uh focusing on building document parsing also known as AI parse document uh Our mission is make it fast, cheap and easy to process the query unstructured data using AI.
So we build this AI pass document from zero to one. It's ex it's actually started from a hexon project in data bricks and then uh it we got a lot of interest from customers and from leadership then we build this product of today but in the future it's going to be further be improved. I love the fact that you're f I love the fact that you're calling out that we're basically doing hackathon projects and that's actually how we build a lot of the cool features.

3:16 Understanding ai_parse_document: The Technology Behind Document Processing

At some point we're going to have to this is a side note for anybody listening to this webinar. I'm going to have to go ahead and include our interns too because the running joke we have at data bricks is all of our interns are actually the ones that do the actual engineering. But I digress completely. Uh Connor uh without further ado you introduce you because in fact you were the one that actually did this wonderful post about AI parse document super popular uh lots of folks definitely followed it we want to definitely introduce have you introduce yourself and then let's go ahead and go right into the demo so yeah without further ado >> yeah beautiful thank thanks for having me guys and I actually didn't know that AI parse was a hackathon project I think the other one I know of is data bicks apps was a hackathon project internally at data bicks so that's cool that a lot of like products that customers are using are actually hackathon projects internally or that's how they start.
So yeah, it's it's cool to be a part of this like you know I'm a customer using the thing that you guys are building. So obviously very happy to be here and chat with you guys. Um I work at Planes Midstream. Um I work up in Canada. Um we're a mid-stream oil and gas company and um I kind of got started down uh this path of using AI functions where you know data bricks publishes the idea of like the AI value pyramid. So right at the top there you have like UI and chat bots and agents which are really cool.
Um but we kind of struggled to get value from those immediately. So we kind of turned to uh batch inference uh where we're bringing in um unstructured and structured data from a variety of different sources internally. Um and we were just sort of applying you know those AI functions whether it be um extract or summarize and now AI parse kind of at scale right you know some of our databases have hundreds of millions of rows and uh we've gotten a lot of good results from them and I think that's kind of helped us figure out you know and show to leadership what the value prop is behind generative AI you know show some returns on some of the investment that we're making and uh that's how we got down this path of using AI parse >> oh that's really interesting so one things I'm curious as you described, right?
Is that were because you're typically working with oil and gas, right? That's that's your wheelhouse. Did you find that um lots of folks were like hesitant to try out using things like a parse document in the first or just LMS in the first place? I'm just curious from your perspective as somebody that actually needs to go ahead and convince people to use this stuff. >> Yeah, it's always it's always it's always tough.
Uh thankfully both of my leaders are um exx ex Microsoft so they're very forward thinking from a technology standpoints like that definitely helps um you know being enabled with the right tools and um you know showing back to the business some of the either cost savings or quality of life improvements with some of the dashboards or apps that we're building um is is is helpful as well but we have uh really good support with our leadership internally which which is great. >> Oh that's super good to hear.

5:49 Real-World Applications: Connor's Experience with ai_parse_document

Okay perfect well without further ado let's dive into the demo. I think you said you had some slides to showcase what our scenario was because basically it was I think we're doing a bit of a setup where we're because we we have to do like public data or sorry data that we can show publicly I guess because it's not technically public data. Is that correct? >> Yeah, that's right. And uh were very brave uh to suggest this use case and I hope uh I hope we don't uncover anything crazy in these in these receipts but you know >> I seriously hope not.
Yeah. travel down to Houston for work, I don't like having to take pictures of receipts and then input them into our expensing software. So, um, this is kind of a cool one um where, you've been very kind to provide us uh all of your receipts and uh we're going to see if we can kind of derive some insights uh from them using a variety of different technologies and data bricks. So, all the way up. >> So, by the way, yeah.
Oh, yeah. So to point, so the whole reason I thought of this, by the way, so just a there's actually a little bit of my own past inside this one. Uh so I used to work for uh SAP Concur and so one of our big things was basically expense reporting and so because everybody loves expense reporting. I'm more than and by the way, please let's hope we're not doing a social clip where that's why I stayed. Okay. Uh everybody loves expense reporting in which we all have to go do it.
Whether you're a startup, whether you're an enterprise, you have to go ahead and do expenses. And I was thinking to ourselves, what's a good way for us to showcase the coolness of AI pars document on a scenario that everybody can relate to even if they despise. So that's the background. >>, take it away for sure. So just to add some additional context to that. So, uh, sent me, um, a bunch of his receipts. And to kind of visualize it in the standard medallion, um, architecture and what, uh, we'll go through in the demo is we land, uh, the receipts, um, basically into a volume in Unity catalog and what, you know, we're calling the bronze layer.
And then moving from bronze to silver, uh, we're going to leverage AI parse document, which is going to OCR those receipts and then get them into a delta table. And then we're going to do one more transformation where we use AI extract to um pull out some important information and kind of help guide things like Genie and our dashboarding to answer some questions about what is expensing. So that's how the data is going to flow I guess end to end.
So >> perfect. All right, let's do this. This is going to be painful. >> So So let's jump over to data bricks. Um, and I have preloaded >> zoom in just a little bit so that way it's a little bit easier for folks to see. >> No, that's okay. How's that look? >> Uh, maybe one more, please. >> One more. >> Okay. >> Yeah, perfect. I think I think folks can be able to see that easily. >> Yeah. >> Okay. Beautiful. >> You might have to zoom in and zoom out.

9:01 Live Demo: Parsing Receipts with ai_parse_document

By the way, you know how the UI can sometimes flip on you. So, >> yes. So, yeah. Yeah. So, heads up on that one. >> I I I should have uh I always forget my keyboard shortcuts there, but yes, I'll take care of that. Um, so got uh some receipts loaded into a Unity catalog volume. So, these are just the JPEGs um as is. And this was kind of cool because um like my experience with AI parse thus far has been on like PDF contracts.
So, I didn't know how well or not it was going to perform on top of receipts. And the results are pretty cool, which I'll show you uh you guys in a second. Um so, there's about 240 receipts. Um takes, you know, 10 12 minutes to run. Um, and we'll run an actual example here in a second, but uh, basically the way that the table looks is we have the path, you know, showing which receipt was OCRD. We have the raw JSON information that gets returned from AI parse document with the bounding boxes, uh, the coordinates, uh, a description, um, of any images that were, uh, well, I guess the the receipts are images, so a description and then the extracted OCR text.
And then basically the way that this looks is um it extracts exactly what is on the receipt and we have that in a text column. And then we generate another table um which again I'll show all of the SQL scripts here in a second where we extract some important bits of information to help kind of guide the kinds of questions we're going to ask Genie. So we have the raw OCR text again. We extract things like merchant name, total amount, location, um, as well as the currency, and, uh, tip amount to see if a good tipper or not.
And we'll ask some of those fun questions as well as >> Oh, man. You're going to call me out like this. Damn, dude. >> You you g you gave me full rain, man. So, I take advantage. >> So, that Yeah. So, so basically, just as an ad, while we made the he gave him full reign, he he we are going to share the actual code scripts. We're not sharing my receipts, though. I do want to call that out. We are not doing that. >> I appreciate you clarifying that.
I won't I'll make sure I don't check that into git. Uh Danny, >> thank you. Thank you. Thank you. >> So, um this was the script um that I had written in in SQL to run the AI parse uh documents. And you can see like I the table that we just looked at um is there. So, all of this is parameterized. And just to show how this kind of looks end to end, I'm going to create a new uh temp table. And I'm going to limit to 10 receipts.
And you'll see we're still pointing to that same volume where all of those receipts are stored. So if I just scroll down and just quickly walk through the code here, um AI parse document is right here. And then um the source volume path uh basically what that's going to do is it's going to read through um all of the files that exist inside of that uh storage volume in Unity catalog. So um the cool part here is this all integrates very nicely into the languages that data engineers are very familiar with, right? being, you know, SQL in this case and it tightly um integrates and couples into Unity catalog.
So, no managing API keys or or anything like that. It works um like any of the other AI functions that you've used in the past. >> No, really. By the way, I just love the fact that you're doing a call out to the parameters as well to make this really simple for data engineers so that way they can just go ahead and swap out, toss in a different table. Love the fact that you get the limit there. So, that way we can demo really quickly as well.

12:07 Transforming Data: From OCR to Structured Insights

That's pretty cool. >> Yeah. And this and this should uh be pretty quick. So to run on 10, um I imagine this will take uh just under a minute um to execute. And then while we wait for this to run, uh I'll walk through the uh next script um that I have. And this is actually using another one of the AI functions called AI classify. And this is how we're actually like structuring the data out of the OCR content. So the OCR content's uh great because now we've gone from like a digital you know PDF or receipt and we've gotten that into a delta table.
Um but now to have like Genie interact um with that data or build a dashboard on top of we need to start extracting um some information and uh really that's the transformation moving from silver to gold. So once we have that OCR table generated um we can then um uh extract some structured information into the gold layer. Love it. Um, and then I'm just going to go check on this quick and see how we're doing. And I'm going to uh copy this table name while this is running to make sure that we read from the right table here.
And I'll make sure to give the same limits. And I'll also um update the table name where the structured information is actually going to be written. And uh and then I guess I'll I'll step through the table here. So exactly what we saw earlier on the full data set. Um sim similar columns are exactly the same. We have reference to the receipt um that was read um the raw output from um AI parse uh as well as the OCR text that got returned.

13:51 Interactive Insights: Using Genie with Parsed Data

And you can see there must be a table or something inside of one of those receipts. and it extracted the um I guess HTML or maybe markdown um uh labels here. So um an LLM can kind of recognize that as a table. And then now that we have that table created, so I'll just refresh my uh Unity catalog here. And the other thing I forgot to highlight, this is actually running in the free edition as well. So uh if folks want to try this in their own environment, they absolutely can. >> Oh, I love that.
So basically, you just showcased running this live on the free edition with the 10 receipts. That's brilliant. Obviously, in the in when you want to run for the enterprise, you you'll run the full thing, but at least if you want to play with it right from the get- go, you can use the free edition right from the get- go. I love that. >> Yeah. Yeah. Exactly. So, our new table's created. Um, and then I'm going to jump back and I'm going to run that AI classified script that's going to extract things like um if it's like what's what's the merchant type?
Um, what's the location of the transaction, the currency, the merchant name, and we're going to run it against that OCR fulltemp uh table that we just created. So, the cool part here is now we're like chaining different AI functions together uh to output a table that we can start uh deriving insights from. So, this one's usually pretty quick. I can see that the temp table's been created and we should have uh 10 rows in here. um which we do.
So you can see there's the OCR text that um AI parse extracted and then we have each of our fields >> um that AI classify extracted in a table now that we can start uh interacting with. So what we've done so far, we've taken some receipts out of the bronze volume. We've applied um AI pars to those >> uh receipts and then now we have one final table where we can start interacting uh with Genie as well as AIBI dashboards which I'll show next.
So I'll go into my Genie space and uh I'm going to ask some questions about your expenses here. So >> Oh no. So this is where you're going to call me out. So le before you call me out, let's make sure we just call summarize for everybody listening, right? So you just showcased how you used free edition to go ahead and parse through 10 of my receipts super quick >> um and built that medallion architecture to OCR to basically parse the document in this case images versus the PDFs that you normally do, right?
And then also be able to do the extraction so you can extract out all the data. So now you're going to put Genie in front of this thing specifically to call me out. Specifically to go and ask questions on what the heck I'm doing. Is that is that is do is is that accurate? >> That's right. And if I'm asking too many personal questions on your receipts, you can stop me. So please please feel free. >> Oh, no, no, no. We're live.
It's I don't think there's anything I can do now. So hopefully I didn't send you anything that that receipt metadata full table where we've extracted uh location uh currency tip amount those kinds of things. So Genie should be able to very easily start um answering questions on that table. So the first one I'm going to ask is uh well what's your what's your favorite restaurant? So most common restaurant visited. Cool.
Actually this will be good. I don't I'm not even sure which one I want. Which >> Yeah, I think that's very accurate. Uh, um, yeah. Yeah. So, there's just for I'm not I'm not being I'm not doing a paid endorsement for this, but basically there's a wonderful Peruvian restaurant called. I go to the one in San Francisco often. There's actually one in Belleview as well. So, yes, that is probably my favorite restaurant right now. >> I'll have to I'll have to give that a try when I'm at the Data and AI Summit next year. >> Yes.
Yes. And by the way, uh Oh, yeah. Yeah. We're probably going to run an MVP dinner there, too, by the way. So, or lunch there. So, yes. So, don't worry. I'll make sure you're included in that one. >> Excellent. Excellent. >> Um, now I'll ask a follow-up question. So, I'll say, "What restaurants uh do we have receipts for in Montreal? I'll say mercy location column just so we can kind of do a validation to make sure that uh Genie's returned the right data.
So I'll send that off. >> Cool. Too bad my receipts don't I can't make it a Montreal versus New York bagel discussion, but I I don't worry. I I' I've got a bunch of researchers from New York. I don't need to cause problems right now. >> Yeah, that'll that'll require another session, I think. Uh >> yeah, I think so. I think so. I think so. So, return five rows here. Um, and these based on the location, it does look like that these restaurants are indeed in Montreal.
So, looks right. >> Yep. Those look right. Uh, Dandy is a wonderful I'm surprised Lfinka is not there, but maybe I didn't send you a receipt with Lfinka. Uh, Mapumier is a wonderful Portuguese chicken uh uh poutine place, by the way. Um, and Dandy has got some damn good pancakes, just in case you care. and Bro, um I did want to call them out too. Not I'm not a praise sponsor. Amazingly good Vietnamese food. Like seriously, like I used to go to Montreal once uh once a monthish and I haven't actually heard of any of these restaurants.
So obviously I was I'm missing out if I haven't been here before or to those. >> Okay. No, dude. , don't worry. We're again I'm not trying to uh offsite. I'm going to send you my personal blog that actually has all of this. Okay. >> Okay. Beautiful. See, I know I I knew there were more reasons than just to show AI parse for this webinar. >> I mean, I am secretly going also trying to shout out to my uh my second city of Montreal, which you know, basically you all should go there, but it's a great city. >> Yes, absolutely. >> And the last question I think I'll ask is uh based on this data, can we determine if the uh tip amount was good or not? >> Oh, dang.
Okay, I'm really called out. May may maybe this is uh this might be a hot take, but for me I think I having worked in the restaurant biz, you know, I always appreciate a 20% or above tip. So, let's see. Let's see what happens here. Let's see what Genie is able to uncover about your tipping habits. >> All right. All right. Now, now now flames here. it actually calculated a percentage, which we um didn't extract that in our gold table to begin with. >> Oh, cool.
Um, so let's see here. Oh, and it actually generated a uh little chart for us. So >> nice few uh let's see what do we got here. So it actually looks like it did it right on top of the Montreal restaurants that we asked previously. >> Um and uh instead of the full data set. So we could probably kind of prompt the agent or the genie to do it on the fold data set. But this is pretty cool cuz now you can see exactly like the tip amount that was uh that was given on a particular.
So the tip amount is is the one I'm wondering is it the one that I wrote down or is it the one I clicked? It's okay. Not that big deal. But that's >> cool. A lot of them are the ones that you actually hand wrote. And uh when I checked this to begin with, AI pars did a pretty good job of extracting the handwritten note as well, which is which is impressive. >> Those are always tough. Yeah. >> All right. >> And then just to validate um all this information um I built a built a bit of a dashboard as well on top of this data.
So you sent me uh 240 receipts in total. Um you can see like the most visited restaurants. So um Genie did come back with. So we can see that here um on the dashboard. >> Um your most frequent transaction types. So uh looks like restaurants is number one. I think you speculated parking was going to be number one, but that's not personally what I'm seeing in the data. So some interesting findings. >> That's fair. Apparently I go to a lot restaurants a lot more than I realized.
So that's fair. yes, priorities. Um, and then same thing uh for the restaurants visited in Montreal. Um, we have the five rows again and then this is me just displaying the raw I guess more I guess it's our gold table in this case where we have the location, the merchant name, total amount, basically everything we extracted as a part of that AI extract function. And that basically concludes the demoable. We got all the way from some um unstructured images that were receipts and we were able to put a genie space and a dashboard on top to extract some insights. >> Perfect.

22:33 AI-Powered Document Processing Overview

I really love this and so thanks for calling me out on the tips. So I I I I was there but not quite. So my bad. My bad. Apparently I need to work a little bit better. So I I'll definitely do that. Um by the same token uh love the fact that I got to do a shout out to Montreal. That's always a good thing. And of course a shout out to. All right, saying that uh love to bring back um the researchers and the R&D folks if you can jump back in because uh we've got some questions from the audience and uh they're they're definitely interested.
Um so let me let me just dive into this right away. um if that's okay and then for anybody else that's uh involved not just uh not just or you know for example for the next question Connor Fan Zachcha if you want to chime in by all means go ahead and chime in too but uh let me start off right away so the first question I uh come uh that's just come through the wire um which is the most obvious one Arctica I have to call you out on this one uh so that I'm not the only person called out for this webinar okay um the question is like can you provide any roadmap plans on the next versions of AI functions, you know, like the query, the extract, classify, we showcases the parse obviously uh right from the get-go.

23:42 Audience Engagement and Q&A Session

We showcased uh uh uh extract right away, but I'm just curious, can you provide a little bit of context here? >> Yeah. Yeah, super happy to. Um maybe I can start off by like even just giving some historical context of like how we even decided that we were going to really create a best-in-class experience in this space. Uh so basically over the past few years like or a year or so our product team has been talking to hundreds of customers who are all trying to do very similar things across like every single kind of industry.
They're trying to pull in structured insights from their unstructured data to drive some sort of like business intelligence or automation of like existing manual uh unproductive processes. So one of the first problems we heard was, hey, I need a really simple way to like pull in all of these unstructured documents, PDFs, word documents, etc. into something structured and usable that my agentic workflows can actually like use downstream.
And so Fong Khan's hackathon project was like obviously a huge blessing to us in this respect. um where we got like AI parse document into a beta super quickly and those conversations we were initially having turned into like real momentum like it became one of the fastest growing products in our company. Uh and many of them were actually adopting downstream offerings in our agent brick suite via like our remaining AI functions.
So AI parse document was really just step one to enabling the rest of like an agentic workflow that they were trying to build. So now we're investing super heavily in making this end to end like intelligent document processing or IDP experience uh really amazing on data bricks something that's super simple for customers to build from start to finish high quality out of the box and then also scalable via performance and cost on millions of documents that like single customers are basically trying to run and so that work really spans our like research product and engineering teams that are all like coming together uh to build this experience.
So in the near future what I can talk about in terms of roadmap are some major upgrades to two of the most like popular workflows that we're seeing on data bricks uh most notably through our AI extract and AI classify functions that kind of represent this like pretty like canonical trinity of what people are trying to do um what customer workloads are trying to achieve by like parsing then classifying what kind of documents they're getting and then extracting the relevant information or structured like intelligence from those documents.
So for AI extract, what you can kind of expect are expanded support for richer mo more complex schemas that better represent our customers workloads. Um really imp huge improvements in quality and cost performance at scale with some new researchback techniques that are going to be going into that function. And then also allowing you as a customer to inject your domain knowledge into the schema that you're trying to build through instructions, descriptions, etc. so that your results are faithful to your enterprise workloads because we know that different customers are going to be coming with, you know, different things that they're trying to achieve and we want to make sure that our system does a really good job at that.
Uh for AI classify, pretty similar story there. Um a lot of enhancements in capabilities, broadening our category supports, adding ways to uh enrich those categories with domain specific knowledge and then also delivering the stronger researchbacked classification system with high accuracy, performance and cost out of the box. Uh and then at a more meta level, like I think one of the most exciting pieces of this is that all of these functions uh they're being worked on by like the same team.
So they're all going to be fully composable with each other. So AI extract is going to understand what AI parse document outputs and vice versa. So they're all going to be able to talk to each other really easily. Uh and all of them kind of live in a context that's in line where our customers data already lives. You just need to point it to a UC volume as showed or like a delta table. And so you'll be able to parse, classify, and extract from a document super easily.
No custom like glue code you need to manage. No brittle like pre- or post-processing. Um, which is like one of the main pain points that we see customers facing when they're trying to use a lot of like the fragmented solutions out there. Um, and then obviously because it's all on like our Unifi data bricks platform, uh, everything's going to be governed and there's going to be lineage through Unity catalog for the pipelines that you're trying to build.
Um, and we're also integrating super intentionally with like the jobs and pipelines part of data bricks to make sure that like AIdriven ETL uh becomes like a really great native experience. So I think that was like a pretty long answer but the TLDDR is that we're working on a lot of stuff. Um, and the team's moving super fast. We're iterating really fast. Um, you know even AI parse document has like a bunch of new features and capabilities coming coming up that we'd be happy to talk about.

28:37 Roadmap for AI Functions

So, um, yeah, that's kind of the high level of where we're going. >> Oh, this is amazing. But one thing I definitely did want to call, you mentioned that basically AI parse, uh, AI extract, AI classify, all these different AI functions. Uh, they basically are aware of each other. Um, so I did want to call this out. So, basically, you're able to basically build these chains like uh where they all actually interoperate with each other.
So, they're aware of what the other one's doing. >> Yeah, exactly. because uh that is basically the I mean that was like in a sense workload as well like you're never going to really use one of them in isolation. Most of our customers are like chaining two to three to more functions together to build like an endto-end IDP workflow. >> Awesome. This is Thank you. That's anything. Anybody else want to chime in before I go to the next question that's coming down the wire? >> All right.

29:36 Research and Development Behind AI Parsing

No worries. All right. I know this is going to be definitely targeted more toward our research folks. So Z Josha uh I think the next question here that's come down the wire is like hey what can you tell us about the models and the research behind AI parse right I mean this is really cool but uh you know this demo that showcased but I'm also curious in terms of like hey like what are these like what what's going on like how how can you tell us like how the research the science behind this stuff because you know I think you had you actually hinted at like there there's also there's these vision models that are coming into play Cool.
What does it mean? I'm guess you know from the uh to the rest of us who actually are not research scientists. >> Oh um yeah. So um yeah I can share that a parse document is definitely designed to tackle the complex and real world challenge of enterprise document parsing. uh not just you know OCR tax extraction but also like deep layout for example like co just showed um uh for table extraction it's not just extracting uh the table cell content but also extracting like how exactly these sales are aligned with each other uh and also uh we also have a new functionality that is uh diagram understanding like providing the figure description if there's any figure in the in the input documents.
Um yeah, some of our uh research highlights uh include like uh for example ed training of the underlying document parsing models. Um and also rather rather than you know training like many many subraes uh you know different like um you know subsources uh to extract uh the context uh and also um like we'll definitely focus on you know cost and performance improvement. For instance, um we achieve uh the matching ex exceeding uh leading uh systems like uh you know other sources providers but with uh three five times lower cost.
Um and uh um our research also focus on you know uh research engineering work in model architecture inference or optimization and also dataentric training. um to focus on aboveness. For instance, handling, you know, makes the digital native plus scanned content of for instance a handwritten uh you know, content extraction to run charts uh you know, photos of uh physical documents, irregular layouts to desk. Yeah. >> Yeah.
Actually, look, as a follow-up to that though, because so I I guess the real call out is like the complexity like obviously expenses that we did for the demo is a relatively simplistic demo. Uh not not a shout out to you, but but by but by the same token, I'm just curious like like when you're talking about the comp complicated PDFs, are you talking about like for example PDFs that for example have charts inside them, tables, table headers, table columns?
Like this is the stuff that's still relatively complicated to do? Like is is that the context you're coming from here? Yeah. Yeah. Yeah. So for example, uh the the you know the scenario this mission is a is a really good case of compact documents. Um uh I think there's a you know u multiple dimensions here. uh you know recognizing the special relationship between different elements like tables, figures uh regular parts, the logo logo on you know in a in the beginning uh and also um like uh very faithful highly accurate uh extraction parsing of each element.
So um I'll say uh OCR itself is not a soft problem. this still a teeth uh you know research problem uh with a lot of challenges to address. >> Gotcha. Gotcha. Um any anybody else want to chime anything in before my my next because I actually have my own personal question related to this now. So but anybody else want to chime in? >> Um I can uh add a little bit about uh I want to specifically call out the amount of work we did into uh gathering data for our models. the document understanding use case uh as we talked to customers as we heard from our product managers the kind of diversity in the uh documents that our customers wanted to try out on these models uh the spectrum is like huge uh it can go from like receipts and invoices to uh full page tables uh and very densely uh dense uh documents with a lot document.
Uh and I'm really proud of the work that we have done to ensure that and that we continue to do that. Uh to just ensure that these ples are going to be able to work on any kind of document. Uh it's been an ongoing effort but uh it's been uh yeah it's been really exciting to see the models improve and uh do so well. That's so cool because I still remember when I was at uh uh SAP Concur, right? We end up needing to create rough number because I don't think this is remotely the correct number any longer, but uh like when we had to do um uh process emails from each different airline so we could actually extract out the airline information, we had to I think we wrote something like 250 different templates.
So basically each airline had its own template and so that way we could extract the information and we had to update it like almost monthly because at least like 10% of them in fact I'm sure the number was higher we basically had to modify based on the email that was sent. It's like okay now we have to extract it here we have versus have to extract it there. Now this is a while ago so obviously it was painful and not to mention it was a lot worse with our expenses like dear god that was insane right?
So, so I'm glad that basically the research you're doing right now basically is like maybe I don't have to write 250 different templates anymore. Cool. Um, oh I behoove me I also have to ask another question. That's my own personal question for but again to the researchers here. Are we using RL because we everybody's talking about RL. Everybody's talking about refor. I'm just curious if that's actually what how we actually do a lot of the finetuning of the training or is that involved at all?
I'm just curious the purely personal question here. >> Yeah. Yeah. Um yeah, we use a mix of you know the frontier uh training techniques like SFT uh to you know strengthen uh improve that model's capability to uh to you know towards better parsing and also better understanding uh the underlying structure of the documents. Um, yeah. So, yeah, we are definitely using Frontier training techniques here. >> Sweet. Okay, cool.
All right, I got to switch gears because fun, this started with you doing a hackathon project, which is freaking sweet. Okay, I love the fact that you started a hackathon project and now it's like one of the most popular uh features within the data bricks ecosystem. So, really cool. So, I got to ask you this question. Mind you, it's also coming from the wire too. Like, yeah. Can you describe the engineering behind AR parse because there's obviously you're the one who built the hackathon.

36:38 Engineering Insights on AI Functions

Is it the same by the way? Like what you hacked together and what you actually have fun in production? Is it really different? Did you vibe code? Okay, these are me adding my questions in, but I'm just curious like how like love to understand that a little bit better, please. >> Sure, happy to. You can imagine it has evolved so much. It's very different from what we originally hack on but uh it's currently production grade and um so but first of all let's give me some uh let me give you some context data bricks engineering product and research don't operate like three different teams we really function as one team and we draft and review PRDS and design docs all together and discuss project planning and the priorities together basically we are pretty much always in the same meeting room or group discussion and we are all aligned with one mission as I said earlier that it is uh to make it a fast cheap and easy to process and the query unstructured data using AI.
Our PMs help the team deeply understand and what customers are actually trying to do and what features would generally solve their problems. Our researchers build our proprietary high quality models, benchmark them and maintain a feedback loop. So the models keep getting smarter and smarter and then engineers take all that and turn into something that people actually like to use whether you are data scientist, a researcher, a data engineer or a business analyst.
So now if you look at if you look specifically at the engineering behind the AI pars document there are really two dimensions the big data and AI. The first the big data dimension um we are a big data company. So AI stock has to scale and I mean seriously scale millions or billions of documents, high throughput, high reliability, fall tolerance, you know all the core principles of the leos architecture. It integrates with UC and serverless works across DBC, SDP, Genie, AI, BI tools and uh most importantly it powers agent brakes.
Uh secondly the AI dimension because we are also an AI company and this is an AI product. Applying AI at this scale it's not trivial. It can get slow and extremely expensive and very quickly if you cannot if sorry applying AI at this scale is not trivial. Uh it can get slow and extremely expensive very quickly if you are not careful. Um, so we put a lot of engineering into increasing the model efficiency, improving GPU performance and utilization, uh, lowering costs and making sure things run faster without metering your budget or our GPUs.
So in summary, every piece product design, research, architecture, engineering, all them are critical to make AI pars document and AI functions broadly not just work but uh work really well. Oh, that's super insightful. I really appreciate that. Okay, I just realized we only have about 10 minutes left for this webinar. So uh I want to I there are four questions remaining from the from the audience. Let me just go through them real fast.

40:22 Future Enhancements and User Feedback

Do me a small favor, y'all. like just do the the quick version of the answers just because I didn't realize we we used up so much time already. So I think this is more toward the research folks but again everybody can chime in. Um hey the question is how deterministic is AI par document? Uh will the same document always return the same identical OCR content across cells? Yeah, just curious here. Um give me the quick version because again we got three other questions I want to make sure we chime into real fast. >> Okay.
Yeah. So airport documents is highly consistent but like any other based system there can be thermal variations across runs. Um so we have been uh systematically measuring all the sampling temperature and decoding strategies affect the person quality and also uh you know deterministic LM behavior is active uh research area. Uh you know you can refer to you know blog post uh from thinking machine recently on this topic.
Yeah. >> Oh yeah that's a good blog post. Yeah that's a good call. We'll we'll make sure to link it into this webinar. Um oh uh related to this, how did we improve AI parse quality improvement? Uh can we call some of our plans uh for the future? We we covered some of the stuff already and but I did think it's worthwhile just to do the quick summary version since uh uh since it got chimed in. Yeah. >> Um okay. Uh I can answer this.
Uh so we are continuously getting a lot of feedback uh from our customers uh and uh we get to look at the hard documents so we have an idea of the uh cases where we have to do better. Uh so we're keeping a tab on it. Uh and um of course we're like uh trying to make sure that uh in the next iterations of the models those uh modes are fixed. Uh also in the future uh we plan on supporting more features. Uh one of the features uh that uh Z was uh earlier talking about is that in complex documents there are table headers, captions, section headers and there's a lot of hierarchy between all of those elements which can be important in downstream tasks and so we plan on supporting uh generating the hierarchy between all of the elements uh and all of the bounding boxes at some point in the future.
Um uh yeah so uh we have uh a plan uh for the next few months on uh how we can keep improving the product. >> Oh that's really cool. We'll definitely have to have another webinar just on that alone just because the idea of finally putting hierarchies on this type of stuff is actually really freaking cool. All right sorry going to jump to yet another question. Uh this might be more toward you but again anybody can chime in.
Are there plans to have like confidence scores exposed to the oh sorry the end users? Yeah. Sorry. Apparently I can't read anymore. So I'm just curious like how we're going to uh reveal that information at some point. >> Yeah. Yeah. No, that's a great question. Um and the short answer is yes. Uh confidence scores are on our road map. We've been hearing it from a lot of customers, especially ones that kind of have more like high risk or they need high fidelity processing of like their work streams like financial, healthcare, those sorts of industries.
Um, and so we plan to basically expose confidence scores for the elements that we're parsing um to our customers um to give them more insight into how we think the system performed on a certain parsing exercise within their document. Uh and then they can make decisions downstream on how they route those documents or pages. Perfect. Love this. I guess we got another webinar coming. Uh all right, final question that we have time for.
Uh going right back to you, star of the show. like, you know, outside the fact that you want to call me out, right? Why did you want to use GD to create this demo? Man, >> I I mean, um, if I think back to like trying to do this in Langchain like a year and a half or two years ago, like trying to convert like text to SQL, like there was a lot that you had to think about to get that to work really well. So, like I kind of view Genie as like a text to SQL engine, like a really good one um at that.
And um with some of the um like if you had some like complex joining logic or if you had modeled your data like you can make Genie aware of some of those joins and what your primary keys are and I find Genie just does a really good job of writing uh SQL on top of your data that exists in Unity catalog. So um yeah before we were building it in in Langchain and Langraph and uh once you guys came out with Genie we just loved that offering and it was really good at writing SQL. >> Brilliant.
Okay well this thank you. I love the call out to Genie. I love the call out to, you know, the Montreal restaurants obviously. Uh but we will then also, you and I can definitely off the side, we can go ahead and drop a link to uh to your GitHub repo that actually has all the notebooks without my receipts. Without my receipts. Okay. Just >> I'll make sure I I'll make sure I take those out. >> Yeah. Thank you. Appreciate it, man.
Uh but that's it. Uh we're definitely right here uh near the breaking point for our webinar. So I really appreciate everybody's time for chiming in. I really appreciate for you all uh joining. I definitely want to appreciate Fam Z and Archa for joining today's uh session. Uh you may or may not know but in the background helping us run this webinar. So I want to say a thank you to her. And then final but most last but definitely not least, thank you very much for joining today's webinar.
Really appreciate your time man. >> Thank you for having me. It was cool to hear all the cool research going on in the background of these services. So thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.