Effective document management and retrieval for generative AI

Summary

Frontier AI agents currently score below 50% accuracy on enterprise document QA tasks — the root cause is not model reasoning but poor document retrieval quality, making pre-processing, chunking strategy, and embedding design the critical foundation for any working RAG system.
Production RAG pipelines require choosing among semantic, fixed-size, and hierarchy-preserved chunking strategies, combined with multimodal embeddings, hybrid vector search, reranking, and contextual filtering to push retrieval accuracy high enough for real business use cases.
The Databricks Data and AI platform supports an end-to-end pipeline from large-scale document ingestion and embedding through vector search and groundedness-scored evaluation, and Agent Bricks extends the same infrastructure into autonomous multi-step agentic applications.

Effective document management and retrieval for generative AI

Building production RAG systems requires far more than embedding documents. This 90-minute deep dive covers the complete technical pipeline: document structuring principles, semantic vs. fixed-size chunking tradeoffs, contextual and multi-modal embeddings, and advanced retrieval optimization, accompanied by practical implementation on Databricks. We'll demonstrate a real-world Databricks RAG pipeline covering pre-processing at scale, embedding optimization and advanced retrieval techniques—including re-ranking, contextual filtering and real-time quality evaluation. Beyond retrieval, we showcase how Agent Bricks transforms RAG systems into autonomous, agentic applications capable of multi-step reasoning and complex tasks. The session also explores emerging trends—vision-space retrieval, contextual clustering and real-time quality assurance—critical for production systems where accuracy directly impacts outcomes.
Talk By: Xintia Gyenge, Solution Architect, Databricks ; Yevgeniy Ilyin, Senior Solutions Architect, Databricks ;

Chapters

FAQs

Why do frontier AI agents fail on enterprise document tasks?

According to this video, the best available frontier models give wrong answers more than half the time when asked questions about a 40-page enterprise contract — not because of a reasoning deficiency but because the relevant passage was never correctly retrieved and surfaced. The retrieval pipeline is the bottleneck: without high-quality document pre-processing, chunking, and embedding, the model never receives the information it needs. This makes document structuring and retrieval design the most important investment for any enterprise RAG project.

What chunking strategies should I use for RAG pipelines?

This video covers three main approaches: fixed-size chunking, which is simple but can split semantically related content across chunks; semantic chunking, which groups text by meaning; and hierarchy-preserved chunking, which retains document structure such as sections and tables. Fixed-size chunking trades accuracy for implementation simplicity, while semantic and hierarchy-preserving methods improve retrieval precision at higher processing cost. The right choice depends on document types, query patterns, and the latency and cost constraints of your production system.

How does Databricks support production RAG pipelines?

The Databricks Data and AI platform supports the full RAG lifecycle: Spark handles pre-processing at scale, Delta Lake stores PDFs and multimodal content, vector search with high-QPS indexes and hybrid retrieval serves queries, and reranking plus contextual embeddings improve accuracy further. An evaluation framework using synthetic queries and groundedness scoring measures retrieval quality so teams can iterate toward higher accuracy over time. Agent Bricks can then extend these RAG pipelines into autonomous agents capable of multi-step reasoning.

What is contextual embedding and how does it improve retrieval accuracy?

Contextual embeddings encode each chunk with awareness of its surrounding document context rather than treating it as an isolated passage, improving the model's ability to match queries to relevant content even when individual chunks lack sufficient context on their own. Combined with reranking — which rescores retrieved candidates using a more powerful cross-encoder model — this technique significantly raises the share of correct answers on complex enterprise documents. This video demonstrates both techniques as part of an advanced retrieval architecture on the Databricks Data and AI platform.

Full transcript

[00:08] Great. 50%age, less than 50%age. This is what Frontier agents are scoring right now when you're speaking about office QA. What does this mean? Is
[00:21] literally you're having the best models that you could buy, right? They could write poetry. They could uh pass a bar exam. They could write software. And then I'm going to hand them over um uh a
[00:35] document, a contract of 40 pages and ask what is the termination clause? And it's going to give the wrong answer more than half of the time. And this is not about reasoning. It's actually it's about the
[00:48] winning part. And when it comes to this number, it is exactly why we said okay let's have this session uh at the dice. Let's look why do we have this number and ideally how we can overcome uh this
[01:02] number to make it better and better. My my name is Cynthia Djanga and I'm a solution architect together with Yavani my colleague. We are part of the uh central EMIA team um based out in Turik
[01:16] in Switzerland. And of course we are very excited to be here today with you bringing what we are seeing from the customers. So you will see it's not about just showing what research is doing. It's coming with things that we
[01:29] are discussing with our with our customers because as solution architects we are there on the field. Uh we are having workshops with different industries. We are having architecture review all uh we are doing PC's we are
[01:43] actually in the room when for instance my customer who's a bank u from the banking industry is coming and saying h look uh it's not working like the pipeline is giving me the wrong loan
[01:55] term and based on what we are hearing from that it's exactly how the next 40 minutes is going to be on the first part I'm going to show you what is currently the state the status what we are seeing
[02:08] as well that could work better in the future and as well do a bit of retrospective to see okay maybe there are things that we do not need anymore in this context and then on the later part we are going to have Yav who's
[02:21] going to show how this looks like actually on data bricks like he's going to demo how a specific architecture would look when it comes to uh uh best practices for for rug
[02:33] which means we can just start from the very beginning where do rag break but before jumping into this one I just remembered a bit of logistics to do uh
[02:45] we are going to have 40 minutes right and because there's so much context that we want to share with you we won't have time for Q&A during the session but nevertheless please remember your questions and then afterwards uh if you
[02:58] still can remain and we can have a discussions uh upon that offline okay but during the sessions we are going to park the questions then let's Go back to this part. Five places where production uh rag breaks.
[03:12] Probably this is not something new for you, right? Is it someone from here who believes that there are things that should not be here on this slide? Yeah, that perfect that slide is good.
[03:24] Yeah,
[03:36] you name it and then making sure that you're not going to lose any uh relevant information multimodel content it's not about only text when it comes to document processing we are having visuals charts diagrams then we have
[03:50] retrieval quality which is the most the thing that everyone is obsessed about and I get it also why and nevertheless we have operations. So on the operations the entire flow to make sure that it's
[04:02] in the right place that you are able to evalate you are able to uh time to time to see okay maybe I can do uh better um better quality on what I have in my pipeline.
[04:14] Honestly if you would look on on these 80%age of the customers would focus on the retrieval quality. So as mentioned this is like the part which is super exciting. They're doing uh model
[04:27] swapping. They're doing embedding chunks and so on so forth. Yes, this is important. It's very important but the main pain comes from number one two and three. It's come it's
[04:39] coming from the upstream part. And this is the thing that people do not like so much to uh to to discuss and look into it. But it's actually crucial to have that in the right place just as an
[04:51] analogy right even the best chef in the world is not able to make good cook in case they do not have the right ingredients and the proper ingredients for that. So what you're going to do in the next
[05:03] couple of minutes is I'm going to zoom in into this part and see what do we have today as a baseline for making it better. What you're seeing that could be potentially a good strategy moving
[05:15] forward for you as well with other frontier agents and as well a retrospective seeing okay what are the models that perhaps uh you should rewise in case you have it in place today.
[05:27] Before jumping into that one, I really would like to plant this statement uh for the session and ideally for everyone also after the uh after the session. You cannot ship retrieval, you cannot
[05:40] measure. You will see also uh uh today, right? We have different options. We are going to have even more options. But it's very important not just to take it as granted that say yeah for sure this is going to work for my use case uh and
[05:54] I'm going just to implement it. Measure always it's about measuring. So please bear this in your mind and uh even even after the session when you are with your colleagues back at work and you're
[06:07] thinking about this think about this statement. With that being said let's see what we are what is working today. We have chosen here three parts that we are seeing that they are mature enough and we are seeing that to get today
[06:20] they're in a very stable position. So we are having high bit retrieval. So BM25 with uh with RF this is a seven-year-old algorithm. It's not exciting you know it's not
[06:32] novel however it works and it works really really good when it comes to document processing. Then we do have cross encoder reranking when latency permits. So here actually if you would put rerank on what you initially have uh
[06:47] uh for your um for your scoring it would give like a a lift to your recall like 10 15 points above the benchmark the enterprise benchmark which is truly significant. It's not something marginal
[07:00] and then we have hierarchy preserved through ingestion. So here in case you have a document it's very important when you're doing the parsing making sure that the headings the titles everything is going to be in the right uh in the
[07:12] right uh structure. So these are the things which in case you're doing it today it's actually good you can continue but of course nevertheless always measure like always be uh on uh
[07:24] with your mind on that one. Nowadays everything is moving forward and and evolutionizing the things and this is why we also have frontiers. So we have things which are still not on
[07:37] the productionized phase. However, it's worth already considering it today in case uh it's worth for your use case. Here I want to touch point on five of those. We are having a gentery
[07:51] understanding. So here it's about taking your user query not as it is but having an agent who is able to reformulate to have a look into it and decide what to search for. This is ideal in case you
[08:04] will have like a very complex query uh like with multiple things in a in a specific question. You can think about for instance uh give me the comparison of my top three customers across the two
[08:18] years for Q2. So usually you will split these questions into multiple ones. So for that particular one a genty understanding would be the way. Then we do have vision space retrieval. So
[08:30] vision space retrieval is mainly when you have documents where the visual layout layout is very very important. I myself I did poly techniques. So it was uh engineering and trust me I would not
[08:44] be here with my exams done in case I would have just forgot and do not care about the electricity diagrams that I had on my paper. So when you have this kind of um uh use cases the visual space
[08:57] retrieval was really something that could help you uh having a better result. Then we do have a gentic retrieval. So here the retrieval would act like an agent. the system would identify how to do the
[09:11] indexing, how to do the filtering and how to do the query based on the parameters that you uh that it considers that would make sense. Then we have the last two reasoning time retrieval and reasoning reankers. So the reasoning
[09:24] time retrieval is having the retrieval not only on the pre uh processing step but already on the thought of the on the chain of the thought of the model. And the last one reasoning rerankers here
[09:38] it's about also having uh in your reanker thinking okay why did the passage actually answered the question what I uh uh what it answered so it's about having the reasoning behind it
[09:51] these frontier are great however and there is a very big however it also comes with the cost it also comes with complexity it also comes with um with more things that you really need
[10:04] to make sure that you're not just going to use them because of course by the end of the day you really need to again measure whether it makes sense and when you're doing this measurement that I was saying here we have seen okay what do we
[10:18] have today what we are seeing that it could help in the future but it's also a topic of reviewing what you already have as mentioned so there is a thing which is not so comfortable right because you
[10:30] have to review and you maybe you say okay by that time it was good that we have chosen that model however today doesn't make sense and for that we are having the following slide which is
[10:42] about what to retire we are seeing here semantic chunking by default we are seeing hypothetical document embeddings and we are seeing LLM as reanker by default so we have taken these three
[10:55] also looking on what different researchers uh have been showing throughout the years so for semantic chunking by default this This was really a a very hot technique two years ago. However, research shows now that in case
[11:09] you will have a recursive splitter at around 200 uh uh tokens, it's actually making a very good thing when it comes to cost. So if you're using that today, it's worth also double checking. Then
[11:22] for the hypothetical document embedding, this for the ones who are not aware, it was mainly designed to have zero shot domain transfer. So when you don't have training data and here as well when you
[11:35] looking on okay what do we have today re-ranking actually seems that from a cost perspective is making actually better than uh highly and then the last one lm as reranker by default. Um the
[11:49] quality that you will have for that one seems very compar uh very comparable even better with the uh by having distilled cross encoders. For instance, you you can think and I would think gave
[12:01] you now some numbers. You would be able to speed up like around 170 180 while still maintaining the quality and decreasing the memory by 24.
[12:14] I mean for a production rag it's really really big big and significant numbers. So now we have seen what we have what we could have what we should retire and
[12:26] probably you're thinking okay Cynthia we are at the dice right like what what we are actually going to do with data bricks when it comes to to this so here comes the questions how does it fit with data bricks
[12:39] this is the architecture which ideally is not for the first time that we are seeing in the in this three three uh days this is our platform form
[12:52] discussed so far in in our context there are three parts which are truly fun uh fundamental when it comes from the architecture part we are having the open format section so the volumes making
[13:04] sure that okay I'm having my PDF I'm having all the different types that is needed for my use case then I need to have a proper governance and we have seen the latest right on AI gateway uh gateway as well It is making
[13:18] sure that our pipeline is really having a unified governance that you will be able to track uh your models that you are able to also have uh all the use cases when it comes to uh auditing cost monitoring and so on so forth. And the
[13:32] third one from this architecture it comes to lakeflow. So it's about ingesting the documents that you that you are having. By having this tree, you are actually able to define a proper
[13:44] architecture when it comes to document processing from a datab bricks perspective. When you're saying document management for rags, right? What we are actually thinking many people would say, yeah,
[13:56] it's about only storing the the PDF files and that's it. I'm going to have from uh from nonstructured structured uh documents. Well, actually it's a it's a life cycle, right? We will have the
[14:08] preparation at the source. Then we will have the parsing. Then we will have extraction classifications. Multimodel handling. I just mentioned as well at the very beginning how important as well uh to have the for the use cases
[14:20] that are not only text and then preparation for index ingestion. If one of these would fail, it would really have an impact on your entire document management uh uh pipeline.
[14:35] Now what we are going to do is that we I'm going to go through each of these steps and see what would be the best things that we could do to today uh when it comes to this and for that for the very
[14:47] first one is the preparation at the source and here it's better to give a a direct example we are having this here without hierarchy this statement says I'm not sure whether the ones who are
[14:59] very back can can read this so I just going to say the VTD pin operates at 3.3 volts of you can see that from my engineering background I'm still with my with my physics but I use this example to show
[15:13] that by having this you don't have the context you don't have the terms um you don't like the embedding model is trying to guess what would be this one as a comparison with hierarchy you would
[15:27] have the following uh output you will have Texas instruments you have a data sheet electrical ical characteristic power supply the WDDD pin operates at 3.3V so you have way more information
[15:40] the context is the same what makes a difference is the retrieval signal it's the hierarchy part now the question it is okay how we are able to have this hier hierarchy
[15:52] be present from a datab bricks perspective and this can be achieved on datab bricks only by four functions and lakefro who is going to uh enable the overall pipeline design definition when
[16:04] we think about the AI functions that we would use here there are four as mentioned we are having AI parse document which is GA so it's mainly taking the data from different uh type of uh sources right it can be PDF it can
[16:17] be word you name it then we have AI extract and AI classify which are public preview as the name says right AI extract the user defines which fields they would like to to have and then they're going to provide uh the outcome
[16:30] uh extract the uh what they need AI classify it also comes to the fact for instance you might have different type of uh documents you you can have in a bank for instance you can have loan
[16:43] documents you can have tax uh uh income forms you can have identity documents so it's everything about classifying and then we have the fourth one which is AI prep search this one is in beta and what
[16:58] is this is mainly a chunking function so if you Going back to my example of with hierarchy actually if you would use this this function AI prep search it would directly be able to give us an output uh
[17:12] sorry it will give the output of what we are having here today this is still in beta and everything as mentioned would be defined by declarative pipelines everything that we discussed would work
[17:24] for text however we have to see what is data brick's position when it comes as well on multimodel here if If you look at this slide as a spectrum from most managed to most customized, we will see
[17:37] three approaches. We are having the built-in part which is uh using AI pars document. This is ideal in case you will have documents with text and maybe some
[17:49] parts of charts but it's mainly on uh u on this part of text with a little of charts. Then we have managed which is powered by knowledge assistant. This is an end to end agentic uh flow if you
[18:03] like surface let's put it like that and then we do have the do it yourself where you are going to bring your own embedding models this is when for instance you would have a visual
[18:17] layout that a text would not been able to describe and for that particular one from these three this is when that part when you would use the uh do-it-yourself uh part of
[18:30] So this is how it look like for multimodel. There is one remaining part from our life cycle if you remember which is how we are going to do the chunking and how we is going to work with AI search for this particular one
[18:44] we have two approaches one is the happy path on datab bricks and the other one the full control on the left side the happy path on data bricks is means that you are doing chuck into the delta you
[18:57] create your AI search index and then you sync and everything is uh done by datab bricks that the other one you have the full control which means that you are actually going to do it on your on your own. So you are going to bring your own
[19:10] embeddings you're going to as well uh do the configuration of the compute and as well to have the self-managed uh uh vector search. Honestly when we are speaking with
[19:22] customers some would say okay we would like with full control because it's most production production ready right because it goes over uh our security requirements that we are controlling the things
[19:35] but what we're seeing that is working well is start with with this one and then once that you're having the numbers you will be able as well to think about having more customized and having the full control on on top of that and
[20:03] moving from 0674 to actually 0.957 and this is by doing the things that I was just mentioning until now by going with this pipeline with the four
[20:16] functions by Leflu and and so on just because we are also in the numbers we are still having other economics I'm not going to discuss on that but I think that the slides speak on itself right
[20:28] it's really important when uh you actually have to show the return of investment that you're doing today uh in your use cases these are the economics how you're
[20:40] reaching to those numbers this is literally the if I would take a picture of the slide from today that would definitely be this one until now this is the architecture on the datab brick side
[20:52] uh Yvan is going as well to show in his his demo But mainly what we are having here is of course you're having your document uh your documents then you're going to have autoloader for ingesting and putting it on on bronze. By the way
[21:05] as you can see here everything would be done in a medallion architecture. So it's going to to follow the medallion architecture. Then you will use the AI pars document uh which is going to provide information
[21:19] as well for AI extract and AI classify. And then on this tab on the last two you see as well you have AI prep search and you have AI search index. Now with the AI prep search this would be optional at
[21:32] this case. It depends very much on the length of the text that you're having but overall this could be skipped and I think as well in the demo you'll see with like in the uh demo that particular
[21:44] part most uh is going to be skipped. So this is how it would look like overall end to end and the injection pipeline on data bricks. By that being said
[21:56] what we have seen right we have seen the first part the second the third the fourth and five but we have just stopped on the multimodel content and now what's actually needed is the retrieval quality
[22:09] part and for that one I will hand over to Yv. Yav is going to continue with uh the retrieval quality and then he's going to demo you the architecture that I just presented. Yeah, you have the
[22:21] mic. You already have it. >> Thank you very much, Cynthia, for setting the foundation.
[22:34] documents into the CH how you parse documents and we're going to see today also in code if demo gods are good for us. I have a video as well but I'm trying to do the live demo. Uh so now
[22:46] we're going to talk about retrieval quality. Retrieval is not like your embedding model only. Uh I've seen some implementation where people chasing the most like a frontier model the best
[22:59] model. It's not model, it's a system. So, uh how you really put it into the semantic index, how you treat it, do you use inn, do you use hybrid, do you use ranker and so on. On the data bricks,
[23:13] you have this choice. If you remember keynote from Tuesday from Ali, he said choice is one of our like foundational pillars and you have four options how you assemble the whole end to end
[23:25] pipeline. You use knowledge assistant which is managed agent brick. So basically you point knowledge assistant to documents and then knowledge assistant does the work inest documents
[23:37] chunks documents put it in the index and then does the retrieval. So you basically only need to put documents and then you have an endpoint and you can already uh ask questions. So it's not like only vector search endpoint. It's a
[23:49] gentic endpoint about asking question about documents and so on. So it implements a whole agentic back end. Uh sometimes you need only vector search then you can use datab bricks banish services. So basically you put your
[24:02] chunks into the delta table and any delta table with a text field can be a search index. So you create a search index on top of the delta table with a with a field and then you need to host
[24:15] this index somewhere and this is vector search compute is needed to host this index. Then obviously can do always always hybrid some use some pieces from data bricks for example you don't want
[24:28] to host your own vector search compute doing something like pg vector or whatever you just want that we host it and deliver a queries to you but you want sometimes manage your own index you
[24:40] can do it as well or you can do to completely do it yourself you can use custom models custom retrievers custom pipelines everything there so it go to the absolute lower level But it's all will cost on the engineering maintenance
[24:53] and so on. But again your use case each use case unique and depending on use case um I can do only recommendation start at the most manage level. So sometimes knowledge assistance all what
[25:06] you want. If you want really like vector search go like with the manage vector search delta index and so on. what embedding
[25:21] are two models uh one is the default model datab brick gt large n it has 8k token context,24 dimensional embedding and it's production default for in for English
[25:34] it's good it's used by agent bricks uh sometimes you need to test sometimes this model doesn't work well with different languages and then you can go to the datab brick quen three. So in my
[25:46] demo I use datab bricks quen 3 because I have all the non English languages and this model available. You just select it. You don't need to host it. It's available in datab bricks. It has a big context 30 32k. So you can put the whole
[25:58] document there if your use case makes sense for your use case and it's more than 100 languages and obviously then you can bring your own model for example sometimes you need a VM or uh
[26:12] like multilingual your model that is fine- tuned for your specific language then you can bring any open-source model or anything through the uh custom model hosting and you can use this model
[26:24] endpoint as a embedded model for the managing index as well but I recommend start with a GTE if you have a English context if you has non-English context start with a quen but always you you can you can validate uh evaluation of your
[26:38] retrieval so it was like index embedding model now you need to host index somewhere you need some compute a harness that basically gets your query calls the model calculates embeddings go to the
[26:52] index gets the similar vectors put it uh uh back as a response and potentially enrich metadata and so on. Uh this is vector search or AI search. It's new name of the vector search. You have
[27:04] generally two modes. Uh you can go with the standard mode or storage optimized. Standard mode work very well up to let's say 300 million vectors per endpoint. If
[27:17] you have huge indexes with a billions of vectors, I would go to the storage optimize because storage optimize is very cheap. If you have more than three 3 million 500 millions will be much cheaper than standard. Uh storage
[27:30] optimize is has more latency. It's about 500 millisecond. If you need a very very fast indexes 20 50 millisecond for standard and um it also come with a
[27:43] quite high QPS. I think was like released this week uh additional like uh high QPS and you can go to thousand queries per second with a standard vector search index and this is
[27:56] something that given you just as an endpoint you don't need to manage compute you don't need to manage settings you don't need to manage uh other things and it supports three query modes you can do the pure IN just similarity
[28:09] search you can do hybrid and you can do full text search so if you don't need a semantic search. I just want to search keywords or snippets directly in the document. Corpus you go with full search
[28:22] and full search you need to storage optimize because in this case uh search goes directly to the documents. Okay. Now talking about ranker uh I did test for my demo for like for my
[28:35] document corpus. I I I don't use ranker because I don't see uplift for my document corpus. It's always makes sense to check ranker adds depending on the ranker. If you use llm reanker it adds
[28:48] about one 1.5 seconds and data bricks vector search index you can enable reranker and it adds 1.5 seconds latency because there is llm that just checks
[29:00] the results. Uh another option you can use uh a distilled cross uh encoder and it's 107 times faster cheaper obviously because it's very small model as as good as as llamanca and you you can see
[29:14] recall uh moving from 74% to 89% of data bricks enterprise benchmark but again always check because reranker adds latency even
[29:26] faster ranker cost and uh if you have a significant uplift It's up to you to decide if you want to use as we've seen already you cannot improve what you cannot measure right so you
[29:38] always need to measure how you measure performance of the your retrieval there are two options or like two parallel tracks and you need to implement two tracks in your uh application it's first
[29:51] is runtime like a uh retrieval time and here you uh get production traces automatically written into the delta tables uh you can use built-in scorers in ML
[30:03] flow. So each time the trace goes into the ML flow into delta scorers will run at some uh interval about in 15 minutes and you can have your scorer that validates for example
[30:16] accuracy retrieval groundness other things. So you can use your custom scores with LLM or just Python scripts whatever and there are built-in scorers. So you have again can things like uh retrieval accuracy groundlessness and so
[30:28] on out of the box. So you don't need to run and u this everything will be like a checked and saved again result check saved in the data you can build dashboards a very nice observability end
[30:40] to end uh for your retrieval. Uh you also need to understand when you create an index you need to understand so what should I use should hybrid should I use run and so on and this is index time. uh
[30:52] when you just create your index vector search has built-in functionality called out reval. So you don't need to do anything. It's just a job that runs on your on your on your index generate some
[31:04] queries synthetic queries and uh test everything test uh inn hybrid rerankar and all the things and you you see uh the dashboard with with all metrics and then you can decide what mode of the
[31:18] index you want to use. We're going to see it in the demo. And so end to end pipeline. I know the the last rows is probably not visible. Slides will be shared obviously as Cincy already said.
[31:30] Um you have your document injection pipeline that normally goes through document document uh parsing extraction and then optionally AI prep search that do do the does the chunking and prepend
[31:43] with the hierarchy. Uh then you put it into the index. As I say, your chunks goes to the delta table and delta table is not index. It's just a normal delta table that you can use. But then on top
[31:55] of this delta table, you create an index and then you host your index and vector search. It's like a three-step process of three components. Delta, index, hosting. And one hosting endpoint can host multiple indexes. So you save cost
[32:10] and then obviously downstream consumption. How you consume this? and multiple options you consume directly from your application through the because vector search index gives you endpoint and I show this also in the
[32:22] demo like options or you can consume it through the edged brick you can consume directly from the supervisor uh vector search can be a tool in a supervisor agent out of the box so and normally
[32:35] what you have you have supervisor agent you have a set of indexes for example document corpus memory in my demo I use different type of agentic memory and agentic memory memory in this case is semantic index.
[32:47] The main building blocks is AI functions, AI parse document, AI classify, AI extract, I prep uh search, the knowledge assistant which is completely out of the box managed uh
[33:00] agent for knowledge assistant potentially you can use with function but you can use directly on the document corpus. So you don't need pre-process your document just point to the PDFs images doc files MDF files and so on and
[33:12] supervisor agent normally you need to connect everything and like provide a gentic back end or surface across your indexes corpus document corpus and so on
[33:24] okay let's let's let's try to go do the demo very fast I need to zoom I believe So we start
[33:36] with the documents and I save my document just on the uh volume and you see the documents can be clean PDF just really clean PDF and no worries there's no PI it's all synthetic documents you
[33:49] have some names but it's all full synthetic or I have the uh artificial degradation of the document you see this is all rotated and um have inks
[34:01] everywhere some kind of uh images and so on. So this is all on the uh volume. Then I have a pipeline. Pipeline is a is a medalon is a SDP
[34:14] pipeline spark declarative pipeline. You have uh first all documents loaded into the bronze documents. It's just a blobs and then they all move to the silver parse
[34:26] documents through AI uh AI parse documents. So our spark document takes a blob and extract the raw data with a bounding boxes with captions with with the values with everything with the text. But this is like raw data. We're
[34:39] going to see how it looks like and then you need to do something with this row data and normally you want to extract something you want to extract information. Uh first you you normally do the classification of the document. Is it box? Is it bank statement? Is this
[34:52] ID or this something else? And after you classify document, you can relate a document type to the schema. And schema is is is a JSON schema that says I want to extract the following fields from the
[35:05] document. If you looking at the ID, you want to extract ID, number, name, country, whatever, birth date and uh this is handled by uh extraction function. So AI extract and all
[35:17] extraction goes to the silver uh um table and then I assemble everything into the gold layer. I don't do post-processing here. I just collect
[35:29] everything and put it as a as a as a as a packet. So everything extracted and so on. Uh you you might you might want to enrich it uh uh with a reference data with a customer data whatever and this is all happens in the gold level.
[35:43] So in the application it's delivered all to the application just application just sits on top of the uh extracted documents and you see here we have the fully uh structured data I have like 11
[35:57] documents here and prepare document I have these things like a um structured form field value and so on. So the point here just moving from the volume with a
[36:09] documents and documents can be any quality to the application it's very easy just implementing four function or three function my case I parse document I classify extract and basically you're
[36:21] done because showing this and bringing it to the uh UI it's very mechanical so you're moving from the unstructured data to structured very fast
[36:35] when I do it obviously since it's all done by injection pipeline. Uh you have full observability and uh you can see here for example I calculate this throughput per document. It's lesser than one second and I also calculate here cost. Let's refresh it and we go
[36:49] into C host cost. Okay you can see cost for the pipeline cost per document and this is cost per document. It's 0013
[37:01] cent. Okay. Okay, the documents not too big about one two pages but still uh AI pass documents one of the cheapest options around there. Okay, then uh as I said everything look
[37:13] uh in in the database let's look like uh how pars documents look like and this is silver.
[37:27] So you see like a lot of metadata and then we have this parsed blob and this is all uh variant type you have confidence you have bounding boxes you have text you have table structure you have page structure he footer and so on
[37:39] uh so you can uh already use an AI extract to get information from uh from this information or from document and this is extracted fields already extracted fields I just pay document
[37:52] type I have structured JSON with exactly information that I want for the particular document type. Okay. Um in this case I don't put
[38:06] document in semantic index because I I just need them like a row. I I built like a package of the document on the semantic index. Uh I create index on the on the semantic memory of the of the of
[38:18] the agent. Just to illustrate how it works. Uh this is my chunks and chunks uh contain some MD files that are relevant for the LLM to understand what
[38:30] what have I done in the past what the lessons learned and so on. Uh two important things that index delta index allows you to specify two columns. One column is chunk to embed is basically
[38:43] what is going to be embedded into the vector and the another column is a chunk to retrieve and this is what is given back to the to the retriever and as you see I chunk to retrieve I not only have
[38:56] the chunk but I I have all information when to apply this chunk. This is very important because you can combine the information like a chunk that retrieved by uh by endpoint with some information
[39:08] that you need um as a use case here or here something like this and this is goes to to the to the caller. Then we move into the index and as I say this is just a construct called delta
[39:22] index which in this case continuous synchronization. And as soon as I update data table the changes all go to the index automatically and if I delete something it will be deleted from the index. If I change something will be
[39:34] changing the index because it use changing data fit from delta to that index. And uh you have lineage 100% I just wanted to illustrate it. This is my memory index
[39:52] and this is my chunk table. So you basically see the full lineage and you see also the downstream uh this index used by semantic retriever and semantic retriever sits on top of the vector search. So I have vector search and this is uh
[40:05] let's close this. It's it's built in evaluation and evaluation looks like this. It provides an an a dashboard with information like what is my uh
[40:19] discounted cumulative gain at 10. What is the recommended query type based on the various tests and you see here that like different um information inn plus
[40:31] ranker full text full text plus ranker hybrid hybrid ranker. So you have all options uh together and here you can also see what is my basically uplift. Okay, this is relevance and uplift will
[40:44] be uh here exactly. So you see that it's like ranka doesn't add anything. So it's beta basically worse results for ranka. So we
[40:57] don't use ranka and it also gives you average relevance by rank. This is for free. You just run this dashboard and and you have this and the last you use this agents uh you use this um uh tools
[41:09] uh as a part of the agent. If I have a super agent here that use directly uh knowledge assistant and knowledge assistant sits on top of the documents and this is completely managed
[41:22] pipeline of the moving data from documents to the to the index or I can use it for example here directly semantic index as a add point. So you you uh very flexible how you use uh your
[41:37] uh semantic index in in the downstream. Okay, this is like we will look at the whole end to end uh pipeline from the like raw documents PDF to uh to the to
[41:51] the application. Yeah, I can take down the Oh, yeah. At the very end, sum up because it time
[42:03] flies, right? So, just the takeaway, >> four things here. The most production rack failures leave upstream. As mentioned, be sure that you're having
[42:15] the right start uh uh of your documents. The second one, you cannot cheat with people. You cannot measure. Measure measure every time. Go back and measure when you're having anything new on your
[42:27] use case. match the architecture to the corpus not the other way around. So here you remember what I was saying about uh the multimodel for instance right based on whether you want to have something
[42:39] managed by you or by the database you have to uh match that one to uh to your needs and the fourth one you just saw as well TF mentioned right all of these can run on datab bricks uh on top of of your
[42:51] data. So by all these things I really hope that uh after this session you had uh you have now a better picture a better idea on how actually things working on data bricks in this in this
[43:03] area and nevertheless all any time you feel free you can also write to us on LinkedIn and because we are already running out of time as mentioned the still has time so after the session you can still uh have a Q&A with him
[43:16] offline. So, thank you all and enjoy the rest of the day and uh yeah, enjoy the

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.