Multi-Tenant AI with Databricks Vector Search and RAPTOR

Summary

Atruvia, the IT platform for more than 800 German cooperative banks, built a multi-tenant AI system on Databricks that serves all tenants from a single infrastructure without per-tenant deployments or complex routing services.
Databricks Vector Search with index-level isolation replaces custom routing logic, while Delta tables provide metadata-driven configuration management for auto-scaling, enabling one agent to serve hundreds of bank tenants while maintaining strict data privacy.
RAPTOR, a hierarchical clustering algorithm that builds knowledge trees through recursive document summarization, improves answer correctness by 2–3x over standard retrieval for complex banking document queries.

Multi-Tenant AI with Databricks Vector Search and RAPTOR

Building generative AI applications for multi-tenant environments requires solving strict data isolation while scaling to handle hundreds of queries per second. this video presents two production patterns from Atruvia, a banking platform serving 800+ German banks on a single Databricks infrastructure without traditional routing services or per-tenant deployments.
Learn how Databricks Vector Search with index-level isolation replaces complex routing logic, and how Delta tables provide metadata-driven configuration management for seamless auto-scaling. Explore RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval), a hierarchical clustering algorithm that builds knowledge trees for superior answer quality on complex documents. Discover how these patterns enable one agent to serve hundreds of tenants while maintaining data privacy, improving answer correctness by 2-3x, and reducing operational complexity.
🤝

Chapters

FAQs

What is RAPTOR and how does it improve AI answer quality for complex documents?

RAPTOR stands for Recursive Abstractive Processing for Tree-Organized Retrieval. It builds a hierarchical knowledge tree by clustering and summarizing documents at multiple levels of abstraction, then searches the full corpus at any level, improving answer correctness by 2–3x compared to standard flat retrieval for holistic or multi-document questions.

How does Atruvia maintain data isolation across 800+ bank tenants on one platform?

Atruvia uses Databricks Vector Search with index-level isolation, assigning each tenant its own search index rather than routing queries through complex middleware. Delta tables provide metadata-driven configuration management so the single agent knows which indexes to query for each tenant.

What challenges does multi-tenant AI introduce for a regulated banking environment?

Multi-tenant AI must guarantee that one tenant's data is never exposed to another, which typically requires complex routing services or per-tenant deployments. Atruvia's architecture eliminates that complexity by using index-level isolation at the vector search layer while running all tenants on shared infrastructure.

What knowledge sources does Atruvia's central AI platform make available to bankers?

The platform combines general large language model capabilities for tasks like working with text and documents, bank-provided data products such as product documents loaded as vector indexes, and additional knowledge sources from external providers. The goal is to give 155,000 bankers governed access to the answers they need for their daily jobs.

Full transcript

[00:07] Check, check. Hi, welcome everybody. Oops. Uh just a reminder, complete your survey after the session. So, who here has built or is building generative AI
[00:23] applications? Please, show of hands. Okay. 2/3, I would say. Who has built such applications in a multi-tenant environment? Okay. A bit fewer. Today, we're going to present to you two
[00:39] production patterns for multi-tenant AI that we at Atruvia used and implemented over the last year with Databricks together. Here with me today is Natacha Uberschlag, manager of AI forward deployed engineering at Databricks, and my name is Simon Schmitz, senior data scientist
[00:56] at Atruvia. Has anybody here ever heard about Atruvia? Who knows Atruvia? Yeah. The The people working with us, okay. That's quite expected. Don't feel bad for me. If you don't know Atruvia, we are not a B2C company. We are pure B2B company in
[01:15] the German market, and we are the digitalization partner for the financial for the cooperative financial network in Germany. Which means we make banking software for about 800 banks or more than 800 banks and financial institutions in the German market, and we make everything for them
[01:32] from the mainframes, which are still running in our geo-redundant data centers, to the app on our phone, the banking apps, to the latest generative AI innovations and solutions. So, these banks can be
[01:48] smaller than 10 people, smaller than 10 employees, or bigger than 2,000. That's why they group together to form a company like Atruvia to handle all their IT infrastructure for them. And at my team, we are building the central AI and knowledge platform for
[02:04] all these bankers, and our aim and our goal is to give all these 155,000 bankers that we serve the answers they need when they need it for their daily jobs. So, this platform combines a lot of knowledge sources. First of
[02:21] all, we have plain good old bots like your ChatGPT, your Gemini. We provide the same capabilities working with texts, working with documents, generating images, searching
[02:37] the web, everything in a banking in a regular in a regulated environment where it's safe and sound for the banks to use. Another use case is banking documents. So, the banks provide their own data products, they bring their documents,
[02:52] for example, their product documents. They load it off to us into data products, and we make this knowledge available. And there are several other use cases, several other knowledge sources that we make available on our platform, also from external providers
[03:09] within this financial network who already bought built their own assistance, we integrate them into our platform. So, today we're going to focus a little bit on the second use case on these banking documents.
[03:25] Last year, we set out to build the Datenrobot. Datenrobot, a very beautiful German word. It's an assistant optimized for financial documents. And it used to look like this. The The banks manage their data products.
[03:40] We pull everything from the data product into our standard ingestion pipeline. Our parsing, our chunking, our metadata improvements, metadata extraction, and put everything into a an on-prem Postgres PG vector store.
[03:56] From there, the Datenraum bot then retrieves this information. And this worked fine. We went live with it. Shortly after go live, we already had a 50 50 customers, but at the time we knew that it would be hard to scale this to all 800 banks. It
[04:13] worked fine for 50, but it was not built for 800 because I told you that we're running our own data centers, but we're not Azure or AWS yet. So, when I need a new database, me as a developer, I cannot simply automate everything from
[04:30] scratch. I have to go to another team, provision a new database. So, it's not auto scaling enabled. We needed a new solution for this. And we were already working together with Databricks. We already had structured data
[04:46] structured data products from our banks on Databricks. We would provide them the Databricks workspaces. They could bring their data there, the structured data. And yeah, so Databricks was a good fit for us. We wanted to bring the AI where
[05:02] the data already was. Then we wanted to scale to 800 banks by January of next year. That was our goal. All of this while replacing our current infrastructure that was not managed that
[05:19] we had to take care of ourselves by managed infrastructure. And the most important part is the last point here. All of all of this while maintaining strict tenant isolation. We are working in a banking context. We serve all these banks. They are
[05:35] cooperative in a financial network, but they are all different banks, all single legal entities. So, there must not be any data sharing between them. And yeah, with this we ask ourselves,
[05:51] how can we serve these 800 tenants without building an overly complicated infrastructure for 800 banks? Thank you, Simon. Hello, nice to meet you. My name is Natasha. Thank you for making it today. Perhaps let me get to
[06:08] know you a bit as well. A show of hands, who are developers here in the audience? All right, that's good. I have a few technical hacks prepared for you how to solve those challenges. So, it was not just about thinking about the
[06:24] infrastructure, but it's really how do we build an AI platform that can serve 800 banks in a single tenant. And here's the standard playbook about multi-tenancy. You might think about building a routing service.
[06:40] You might think about a tenant registry to map which bank is sitting in which tenants, which endpoints are where. Configuration management, uh a developer's favorite topic. How do you think about LLM endpoints, consuming them? Are they shared? Is there a single
[06:56] endpoint per bank? How do you track the API keys across 800 of those? And then a developer's favorite topic. I think there are two cases where developers are sweating nowadays. One is when their tokens run out.
[07:11] And the second one is when deployment time comes. So, we didn't want our developers to go through 800 single deployments. We want them to focus on one single deployment, and that deployment should go well. And to make sure that the releases are properly managed. And finally,
[07:29] onboarding all those 800 banks. There is um a time of onboarding at Atruvia. Not all 800 banks come in a single batch. They are coming one bank is maybe onboarding in May, another one in September. How do we do that in a continuous and automated fashion?
[07:47] So, those were the concerns when we came in and thought about how do we actually build the solution out. But the good news was, thankfully, yeah, thanks also to to the Databricks platform, but also a few architectural hacks, we actually needed none of it and
[08:02] it was rather easy with a few technical hacks. So, thinking back about those building blocks, what did we build instead of the routing service? We thought about using a unified string
[08:18] nomenclature where we use simple string concatenation and assign a bank identifier to each bank to route them through Databricks vector search to the right index. So, let's take a step back.
[08:33] Databricks vector search has two entities. We have the endpoint level and we have the index level. And you want to identify what we did was to provision a single index per bank and then route them on the index level. It doesn't
[08:49] matter that the banks were sitting under the same endpoint. That way we were able to ensure the data separation. As for the tenant registry, this is when a new data product comes into Databricks. Perhaps a bank has just onboarded onto the Atruvia Datenraum Bot
[09:06] platform. So, here we had again a unified nomenclature where we assigned yeah, a fixed yeah, schema naming convention where we could then parse over Unity Catalog and auto discover the banks
[09:21] and make sure that they go through the pipeline that that would enable the ingest of the data and yeah, reflect that back into our agent. In terms of config management, this was probably the easiest hack. This what were able to abstract this into three
[09:38] very simple Delta tables and I will show you in a bit how they're actually looking like. And for the biggest headache, per tenant deployments. So, I spoke about about the ingestion part, thinking about the agent part. We did not have to build
[09:55] an agent per bank. What we were able to consolidate was we had one single agent that is the Datenraum bot itself that was consuming the multi tenant setup in the retrieval part in the backend. So, with every release, we had always one
[10:12] single agent. It doesn't matter if that agent in the backend serves two banks or 800 banks. And for the onboarding workflow, it doesn't really exist. We automated it. We're auto-discovering the banks. We're tracking the banks and we're only releasing when there is a change in the
[10:29] actual data processing logic or in the agent logic. So, you might be thinking how 800 indices get distributed. Again, we have the vector search endpoints that can host up to 50 indices.
[10:47] And we also had the challenge that Atruria came to Databricks and said, "Hey, 800 banks, that generates quite a lot of traffic on the endpoints. We need to make sure that those endpoints are available." Simon mentioned Atruria is not AWS or
[11:05] Azure. Thankfully, at Databricks, we have a platform that can scale and we have auto-scaling mechanism, but we need to make sure that those endpoints are well distributed. So, there are two mechanisms at work here. We have the scheduled ingestion pipeline as well as
[11:21] the control plane. When it comes to the ingestion pipeline, first step is to auto-discover the banks and see which are the new banks, uh read the status. We were also using quarantines tables to track if there was anything wrong in the data processing.
[11:38] Then once that is all good, find an endpoint with capacity. And this can be a simple heuristic logic. We used a very simple logic where we would just check how many um how many indices does my current endpoint support at the moment. If it's full, pick the one with
[11:55] the lowest count that is now available. It could also be a different logic. If you have, for example, tenants that have different traffic requirements, high traffic requirements on one tenant, very low traffic requirements on other tenants, you could use a distribution
[12:10] logic that um mixes them together to ensure that the load is evenly balanced. And then finally, um once we went through the capacity searching, we would go and provision those indices in parallel. You can use the beautiful
[12:27] power of Spark for that. So, let's assume we have five banks onboarding on the same day, which is hitting the vector search SDK and provisioning them at the same time. And finally, failures can happen. There is a lot of steps
[12:44] yeah involved when provisioning your knowledge database with documents into vector store. So, if something went wrong, we want to make sure that we are aware of what went wrong. So, the third control plane table that we used was to
[12:59] track the retries if something went wrong and to see when it got resolved. So, our developers can intervene and see troubleshoot and resolve the issue.
[13:15] So, now we solved the retrieval part and it was less complicated than what we thought it would be. This gave us quite some time to actually build the fun and exciting part around yeah, building the actual agent. So, we
[13:31] went back to Atruvia, listened to the requirements. There were a few custom requirements around that. And in a nutshell, this is what we built currently in the agent and Simon has also a challenge around multi-tenancy later to address. But basically, we have
[13:48] a singular retrieval step with this beautiful nomenclature that allows us then in the single agent to route the request to the right endpoint, which gives us that multi-tenancy component. Then a custom requirement and a very popular feature by the users of the
[14:05] Datenraum bot was they did not only want to have the the chunk that was retrieved as the source. They wanted a highlight on the key information. It was too verbose for them, even if the chunk size was small.
[14:20] So, what we provided was an overlay. Here is the key sentence of what is relevant for you and we solved that in quite an economical fashion. So, instead of having an LLM go twice, once generating the response and once reading
[14:35] the generating response, comparing it back to the retrieved chunks and then telling us this is the key information, we use an embedding model for that. Much cheaper, much faster. We achieved very low latency on that operation, parallelized it in 8 seconds, the the
[14:52] job is done. If you have high scaling requirements, provision throughput will be your friend to do that. And finally, we were able to present a response with matched citations. So, in a nutshell, we have the same agent,
[15:07] same code, same endpoint for 800 banks. And this is really what you get to build when you're not maintaining a routing service, especially if you're using already the power of Databricks. It is not necessary.
[15:24] To conclude, the less plumbing you build, the more intelligence you ship. When it comes to the actual multi-tenancy routing, we had to write zero line of code, just a few naming convention hacks, and utilizing the Databricks vector search client
[15:39] effectively. In terms of configuration management, we used free Delta tables as a control plane. And finally, this was probably the biggest win when one single agent serving 800 banks.
[15:55] That solved the multi-tenancy, and all our issues should be resolved now, right, Simon? Yeah. Let's see about this. So, we have this product, we ship it to the customer, they might get an onboarding onto the
[16:11] product in their banks, and then use the product and ask, "What do you know?" And if you're familiar with standard retrieval architectures, you know that hm, a normal retrieval assistant might not
[16:27] be well suited to answer this question. We also saw this in the keynote on Tuesday, right? We find partial information, we find details all across our knowledge base, but we don't find this holistic view of the knowledge base. That's what
[16:43] Databricks is now solving with the ontology. We tried to solve it in a different way. So, yeah, you retrieve your K chunks, maybe, but K can never be big enough. Your context window cannot be big enough to answer such a question about the entire knowledge base. How would you
[16:59] solve this for one assistant? What usually comes up is, "Well, we put this into the system prompt. This is who you are, this is what you
[17:14] can do, this is your knowledge, and when the user asks about it, please answer like this. If you're building one assistant, that's fine. But we don't control the data that the banks bring to the Datennaum bot. We cannot do this. So, we have to derive
[17:30] this information from the data itself. Another use case would be summarize everything you know about a specific document. Here, for example, the EU AI Act. Here again, the same. We retrieve up to K chunks. If our K is not
[17:46] big enough, then we don't get a holistic view of this area of knowledge of this document. Or or third use case, I showed you this platform in the beginning with all the little assistants
[18:03] coming from Atrubia or even third-party vendors, and we all integrate them into one AI knowledge platform. This, of course, nowadays is not state of the art anymore because the user has to go to the right assistant. First of all, they have to know where to post the question.
[18:20] And yeah, that's We don't want that. We We want one question to one assistant. And for this, we are building an agent-to-agent coordinator using the agent-to-agent protocol. In the agent-to-agent protocol, the
[18:37] coordinator agent asks all its sub agents the same question. What can you do? What can I use you for? And the sub agent hands out a so-called agent card. The agent card is like a business card. There it's written, "This is who I am. This is what I can do." For this, again,
[18:53] we need this content-derived information. We cannot simply hardcode it. With all of this, an algorithm to the rescue. The RAPTOR algorithm. RAPTOR stands for Recursive Abstractive Processing for
[19:10] Tree-Organized Retrieval, and it was proposed by Sathi et al. at the group of Christopher D. Manning just around the corner here in Stanford. So, does anybody know one of the authors? If you do, let me know, please. I would like to connect with them.
[19:26] So, as the name already says, we are building a tree here, a knowledge tree. And as good software engineers, of course, we have no problem whatsoever to grow our tree from the leaves to the roots. On the leaves here are our documents,
[19:43] our entire documents. And our first step is to summarize them. We send the entire document to an LLM and get about a one-page summary for each document. Context window sizes are not an issue for this anymore nowadays, unless you
[19:59] really want to want to put Lord of the Rings or whatever in your knowledge base, right? Um but banks usually don't do that. Then, as a second step, we embed these summaries
[20:14] using text-three embedding large and then do a dimensionality reduction. The dimensionality reduction is necessary to make the upcoming steps more computationally feasible because text embedding three large has over 3,000 dimensions, and our next step
[20:30] scales cubic in the number of dimensions, which would leave us in the ballpark of 27 billion. So, a dimensionality reduction is necessary here. Then, we cluster these summaries. We cluster it using Gaussian mixture models, and
[20:47] Gaussian mixture models have one advantage. They are probabilistic clustering method, not a hard clustering. So, one document might belong to several clusters. This, of course, makes sense in the in the real world. One document can be about
[21:03] different areas of information. So, you find the right tuning factor for you and cluster the documents like this. And then again, we summarize these clusters and we have the next layer of our knowledge tree. And this we
[21:21] repeat over and over until we find our root or until we find we reach our maximum iteration depth. And all of this here is happening offline. So,
[21:36] this doesn't have any effect on the latency that the user experiences when asking questions. All of this is done, for example, in a batch run overnight and we have this better knowledge tree the next day.
[21:51] How do we retrieve the knowledge now from there? There are two main approaches to this, also proposed in the paper. And the first one would be a tree traversal. We already saw, also
[22:06] in the keynote on Tuesday, this takes time. You have to take one step after the other and as it turned out, both in theory in the paper and also in our practical results, this doesn't even improve answer quality. So, we have no trade-off to do whatsoever.
[22:24] We can just go with the much simpler approach. We just query the entire knowledge base where everything of the tree is inside. We just do one vector search on the entire corpus and we find the right information at the right level
[22:39] of abstraction. This saves time, saves cost and the user is happier. With all of this, this code is available on GitHub. There is a reference implementation, it's MIT
[22:55] licensed. However, we had to do a few adaptations for our for our use case. First of all, in the original algorithm, what is done is we're not starting from documents, we're starting from chunks. So, we are
[23:11] already have these little pieces of the documents split, and we're starting by clustering these. And this might give you an advantage on answer quality if your documents are a very very diverse with the topics. If you
[23:27] have multiple topics per document and they are spread across many many documents and so on, then this might add answer quality. However, we have rather well-curated data sets by the administrators in the bank, and usually one document is more or less about one
[23:44] topic. So, we get these good summaries in the first layer that we can also use to answer questions like, "What do you know about the EU AI Act?" And the second factor why we changed
[23:59] this the second reason is simply cost. Here you see, if you add a new document, you add a new bunch of chunks, you have to recompute the entire lower layer of the tree. If you add a new document, if you delete a new document, all these clusters
[24:15] change. In our case, when we just summarize the documents from the start, if you add a new document or you delete the document, all the other document summaries stay the same. They're not affected at all. So, we have less recomputation and cost savings.
[24:31] Another adaptation is quite simple, but but important in our case. The algorithm won't run forever. It might reach it its maximum iteration depth. In this case, we end up with a not with a knowledge tree, but with a forest, and we enforce
[24:49] a root because we want to answer this onboarding question from the beginning, "What do you know?" We want to be able to fill our agent card. So, we simply enforce this route by summarizing all the nodes that are still left in the graph.
[25:08] With all of this, to our result. These results are from our evaluation data set that we curated with all these generalizing questions. Questions about documents, questions about areas of knowledge, docu- questions about the
[25:24] entire knowledge base. If you're familiar with GenAI metrics, you see here standard RAGAS metrics on the lower end on the on the x-axis. If you're not familiar with these, on the right-hand side, everything
[25:40] that's context recall, context precision, this basically measures how well are the information, the sources that we retrieve from our knowledge base, suited to answer the question that was posed. In the middle, we have faithfulness.
[25:57] That to make it easy, just means is there hallucination or not. So, we always want high faithfulness, especially in such an important context as banking. We don't want numbers to be hallucinated. We don't want any hallucination at all.
[26:14] And we even improved a bit on faithfulness, but the most important is on the left, the answer quality metrics, answer correctness. We improved by twofold to threefold on the answer correctness metrics for these types of questions, which was quite the success.
[26:31] But, yeah, metrics are always nice. How does it look like for the customer? This first customer who was onboarded onto our platform now asked the question, "What do you know?" again after we published this algorithm. And she got a proper answer, "As Datenraum
[26:48] Bot, I have comprehensive information about two main topic areas, AI regulations and bank and banking processes and guidelines. You can ask me anything about this. So, our customer was happy. To recap, we had this to solve this
[27:04] issue of deriving high-level knowledge summaries from our content. We couldn't hardcode it. It had to be content derived. And we managed this by implementing this Raptor algorithm and improved our answer correctness by twofold.
[27:23] So, what is next for Atruvia now? We are already migrating more of our assistance on Databricks. The second project has started. Then, I already spoke to you about the agent-to-agent orchestrator. This will be made available to our customers soon.
[27:40] So, we're bringing new capabilities to our platform. And we also spoke about these structured data products that our banks, their reporting, their dashboarding, and so on. They already have a Databricks. We will make these available through Databricks Genie.
[27:56] And not only through Genie, but also onto our platform because the standard standard employee in a bank, they don't know about Databricks. They don't go into the Databricks workspace. They come to our our platform. So, we integrate
[28:11] these Genies into our platform. And fourth, but most importantly, we are building an Atruvia Argentic platform, which is about more than knowledge work. Everything that we spoke about today is standard knowledge work. In the Atruvia
[28:27] Argentic platform, we really hook into the banking processes because, as I told you, we're building the entire banking stack. We are building all the banking services. So, we are in a unique position to really hook into these processes, make automation available there. And this is
[28:44] coming up next. So, if you like this implementation, if you like the Raptor algorithm, you can see here the the reference to the paper and to the GitHub if you want to check out the code. I'll leave I'll leave it on for a
[28:59] few seconds. And on the lower part, some of my colleagues who were the main drivers and the main contributors to our implementation of this algorithm.
[29:16] All the pictures taken? So, to conclude, I hope if you're facing or we hope if you're facing multi-tenancy challenges, if you're building on Databricks or building knowledge knowledge about in general, you are inspired you you can take
[29:31] something away from this from this talk. You check out this algorithm if you are to build on Databricks. Natasha and her team are probably pretty pretty happy to help you with this, right? Yes. And yeah, if you have any questions,
[29:47] let's have a conversation. Yeah, open for your questions. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.