Building and Scaling a Multi-Agent App Platform

Summary

Skuld, a 127-year-old marine insurer, built a multi-agent application entirely on the Databricks Data and AI platform to help claim handlers manage twenty to one hundred and fifty simultaneous complex claims, each backed by five hundred to over ten thousand documents spanning multiple jurisdictions.
The evidence-first retrieval architecture transforms documents into source-linked fact cards with exact citations, keeps conflicting facts visible for human review, and uses deterministic controls rather than opaque model outputs to meet strict audit requirements.
Key architectural decisions covered include using Lakeflow Jobs for parsing and enrichment, Model Serving for synchronous agent calls, Databricks Apps for secure user workflows, Lakebase for write-back of user overrides, and Unity Catalog for end-to-end governance.

Building and Scaling a Multi-Agent App Platform

How we designed, scaled and shipped a multi‑agent AI app platform for Skuld — built 100% natively on Databricks — and the unified App, AI and data patterns that made it possible. Maritime insurance claims depend on shifting legal facts, mixed‑format case files and strict audit requirements, and standard chunking plus top‑k filtering lost provenance and broke under load. At Skuld, we built a multi‑agent app on Databricks: Lakeflow Jobs runs parsing and enrichment pipelines, Model Serving powers synchronous agent calls, Databricks Apps handles secure user workflows and Unity Catalog governs access end-to-end. The talk dives into our evidence‑first retrieval pattern, where documents become source‑linked fact cards, conflicting facts stay visible for review and agents return exact citations instead of opaque answers. Attendees leave with a hosting decision framework and benchmark data on parsing throughput, queues, token cost, rate limits and latency.
Talk By: Domonkos Pal, Director of Digital Product Development, Hiflylabs ; Miran Hadziomerovic, AI Lead, Skuld ;

Chapters

FAQs

What is the evidence-first retrieval pattern and why did Skuld need it?

Standard RAG and chatbot approaches break down when claims involve five hundred to ten thousand documents spanning multiple jurisdictions, because top-k filtering loses document provenance and opaque answers cannot satisfy audit requirements. Skuld's evidence-first pattern converts documents into source-linked fact cards, keeps conflicting facts visible for human review, and requires agents to return exact citations rather than synthesized conclusions.

Why did Skuld choose to build entirely on the Databricks Data and AI platform?

Skuld needed a unified platform that could handle parsing and enrichment pipelines, synchronous agent calls, secure user workflows, and end-to-end governance in a single architecture. The combination of Lakeflow Jobs, Model Serving, Databricks Apps, and Unity Catalog provided these capabilities natively without requiring separate infrastructure for each layer.

How did Skuld build a gold-standard evaluation dataset for its agents?

Skuld built its evaluation dataset with expert claim handlers who provided ground-truth answers for representative claims. This domain-expert involvement from day one allowed the team to measure agent accuracy against real professional judgment rather than relying on synthetic benchmarks.

What hosting decision framework does this video provide for multi-agent apps?

This video presents a framework for choosing between Model Serving, Lakeflow Jobs, and Databricks Apps based on latency requirements, synchronicity, and user interaction patterns. It also includes benchmark data on parsing throughput, queue behavior, token cost, rate limits, and latency to support informed architectural decisions.

Full transcript

[00:08] Can you imagine yourself working on 10 small projects at a time? What about 20? How about a 100? How about a 100 large projects? Well, I can't manage myself, but some professions actually have to.
[00:20] So, my name is Miran, and together with Domo, we'll share how we're trying uh to solve these problems uh at Scoo. Firstly, this is my forward-looking statement. After that we can uh move on.
[00:32] So let's start with a brief introduction into the industry. Uh did you know that marine insurance is actually the oldest type of insurance in the world? Way before health, travel, anything of the sorts. Uh marine um shipping merchants
[00:47] uh wanted to make sure that if an accident occurs that their loss is not not impossible to recover. And 500 years later we're still doing this job. So, Skoo, the company I work with, uh, is one of the leaders in marine insurance
[00:59] across the world. Uh, it's been founded in Oslo in Norway in 1897, making it one of the oldest companies in Norway that's still, uh, thriving to this day. Now, of course, from those humble beginnings, we grew a lot, as you can see in the slide
[01:11] behind me. Now, what makes marine insurance different from every any other type of insurance that you might have that you might know is the complexity of both the products and the actual claims. I'm not going to talk about the products
[01:23] as we don't have enough time, but let's just go through an example claim that happens. So, imagine a car carrier uh that's registered in the UK crashes with a tanker that's registered in Italy. The accident occurs on the China waters, but
[01:37] the pollution from the incident spills over to Japan. There's four different jurisdictions in stake and now imagine that schooled uh insurers both sides in the incident. So, there's a lot of complexity here in just uh in just a
[01:50] single claim. That's why claims in this industry tend to last over uh years, sometimes even actually decades. So this is the complexity that uh that we're trying to help and solve.
[02:05] So um for each of the complex parts of a claim, there's a pile of paperwork. There's contracts, survey reports, witness statements, there's correspondence between the different parties. >> It's a lot. I've seen some of the folders. It's insane how much there is.
[02:17] Um, and there's from on on a single folder, so on a single event, there's anywhere from 500 to more than 10,000 documents. Um, and that's just for one case. Now, imagine multiplying that by
[02:29] 20 to 150. That's the actual volume of cases a single claim handler does. So during the the during one year and most of these are running in parallel. So they need to understand one case fully
[02:41] with all the files and now they need to understand almost 100 uh across with a full understanding that they need to have. It's this requires superhuman effort in in my mind. Uh thankfully uh
[02:54] the technology that you might have heard of generative AI is supposed to to help with this. Uh it should be great and analyzing documents great at bridging context between uh various documents and uh making sure that uh that the patterns
[03:06] are also caught. So this is why we at school uh tried to take the administrative and cognitive load of our claims handlers and uh and move their or remove that from their day day job so they can just focus on the hard problems
[03:19] and the problems that only they can actually solve. So, uh, if you're on LinkedIn, I'm sure you heard this so many times. Some company claiming their state-of-the-art AI, how they bought co-pilot licenses
[03:32] for all their employees, or how they created a rag pipeline that uh is excellent and works on so much data. These approaches might work. Uh, for example, if you need to ask a specific question in a large data set, this would
[03:45] work. Or if you need to analyze uh 20 30 documents as a whole, you can you can work like this. But this approach doesn't scale. Uh if you need to understand tens of thousands of documents to be able to answer one
[03:57] question fully, this just doesn't just this just don't work. Uh additionally, if you if you add people to the equation, there's so much inconsistencies. How well do they prompt? Are the right documents in
[04:09] context? Um and on top of that, LLMs being non-deterministic by nature, if you increase the complexity and increase the the amount of documents, the inconsistencies uh occur as well. So this uh this is why I think rag
[04:23] definitely has a place in the industry but it's not a solution for a problem like this and you heard I'm sure the the phrases so many times bad inputs provide bad outputs bad data leads to bad AI but I really want to emphasize it here is I
[04:36] I I think you need to think hard and long about your data is it actually ready to be uh uh to be uh added to to an AI uh pipeline. So that's why we had to find a different way.
[04:49] So uh now covered what we're not building. But what are we are actually building? Well, as the title spoils it a bit, we're building a multi- aent platform, but it's a bit different to or I believe at least it's a bit different to most agentic approaches you might
[05:02] have heard about. So each agent has one specific purpose. It's instructed to be good at a single task with a clear guard rails in place. But everything arounding around the agents, the inputs, the outputs, they're all deterministic. We
[05:15] save all of them through medanial architecture. So the bronze layer is the landing zone. This is where all the raw data is of various data types. After the data has landed there, we perform various pre-processing including data
[05:28] bricks OCR on that. And then the next step is to uh to give it to a subset of agents. So subset of AI agents takes the data from the bronze layer to the silver layer performing some some uh various insights. And then finally another set
[05:41] of uh agents takes the takes the data from the silver layer into the gold air which is where we actually serve serve the users throughout these the flow. The the key pillar that we want to fall is
[05:53] control the controllables. What this means is essentially it helps us using AI where it's actually good at but hiding its disadvantages and its downsides. And this is really important in an enterprise setting. This is what actually provides the real real quality
[06:06] data that you can that the users can trust and um all of this has been done actually on data bricks. We evaluated different platforms but we saw that data bricks can offer everything in one place. So we do the ingestion the
[06:18] pre-processing the AI and also apps with data bricks apps everything in one place control and governance is really important in our world and it's really simple to do with data bricks as well.
[06:31] so many parts of this uh but I would need two to three hours or maybe my own conference. So I'll skip it for today. But I'll focus on one thing that I believe that uh most people overlook in AI in AI projects which is the
[06:43] evaluation. So uh as I mentioned claim sanders are our main users and if you think about uh how would they trust the solution? This is some fancy AI. These are generally
[06:55] nontechnical people. we need to when we need to show them that it's actually good and good here is not something that's generic. It has to be measured. It has to be proven. That's the line here as well. Us uh engineers, we do not
[07:08] know what uh good looks in this world but claim handlers actually do. So our job is to uh take the judgment from the experts, make sure we encode it and then we measure it against it over and over
[07:20] and over again. One one approach to doing this is by creating a gold standard data. So essentially what the gold standard data is, it's creating a key value pair. In in our case for example, we take one closed case that's
[07:33] been closed for a while and that's been handled by an experienced claims handler. After that, we give him a set of questions and and get the answers based on the data that it has. And then we do the same with our AI system. Give it the same data, the same questions,
[07:46] and we get the outputs. And finally, we compare the outputs. And this is comparing not does this look good, but it's comparing how close it is to what an actual expert would do in this case. And this allows us to to prove prove
[08:00] that it's worth it. And it allows us to actually uh continuously test. So as soon as we change something in the pipeline, whether that's a prompt, we add a new step and so on, we're able to at 2 a.m. behind the laptop pretend that
[08:12] an expert is next to us and actually check is this improving our score or is it decreasing it or does it have no effect at all? Um, of course, creating this gold standard data is slow. It's expensive,
[08:25] but it's absolutely worth it. Uh, my my advice here is to start early and as soon as possible. Flag this to your experts and make sure that uh that you start with this uh ahead of time because
[08:37] later on uh it's difficult to do. And finally, just one note uh that we do not just uh evaluate the final outputs. So the golden data but we the golden uh layer but we evaluate throughout the
[08:49] steps. So this is to ensure that we see issues early that we notice where they actually occur not just see that the uh that the result is bad in the end.
[09:03] So since I'm running out of time and I want to hand it over to Domo uh here are my key takeaways from this. So data is way more important than the model. uh try to keep things deterministic wherever you can and use AI where it's actually good at and I would say think
[09:17] about how you do the data first before thinking about getting the latest and greatest model from entropic or open AI. The second point I would like to make is that uh building connected agents might sign might sound scary to some of you or
[09:29] even unreliable in an enterprise setting. But if done right with the proper controls in place and with with uh with the right data flows it will outperform any single uh any single bot
[09:41] app you can think of even from the biggest um biggest producers. And then finally evaluation uh as I said evaluation is really the key. This is what helps you test the solution throughout. This is what helps you
[09:53] improve it without needing time from the experts down the line. So, make this a priority from day one in your projects. Work with the experts to get the the evaluation data set and your enterprise is going to thank you. That's it for me.
[10:06] I'll take it over to Domo. Thank you. Uh I will take it. Yeah. Uh I will take it from here. Uh hi everyone. I'm Domo. And um
[10:20] um let's assume for a moment uh that uh we have the perfect setup for what Miran just mentioned like we have the outstanding data quality, we have the
[10:32] perfect platform and we have the greatest uh with the greatest agents. But one thing might be missing though. Um agents can generate a huge value but
[10:46] sometimes without a proper consumption layer there's no really a point of having them. So we need to make it accessible build trust in other words
[10:59] make something people can actually in interact with. And um to show that or before that uh let's go back in time a little for about 20 years
[11:13] we had this um warehouse mindset where uh the data sits on tidy t tables and on your dashboard like every new question requires a new ticket and probably 3
[11:27] weeks later you get the answer. But and um you have the data context probably you build an app as well so you have the app context as well but with data bricks apps uh you don't just get the same like
[11:41] dashboard view but you can talk to your data you can interact with it and you can be the human in that governed loop you can ask you can decide act and write back and um the benefit of doing this on
[11:56] datab bricks apps is um that strong integration between the app layer, the data and the agents. In other words, the system of record, the agent context and
[12:09] the application state. Um let's take a look at a fairly common uh use case to illustrate this integration. So um here's a scenario.
[12:24] We have thousands of pages like Mira mentioned and um this input has like different do it consists different documents different
[12:36] in different structures constantly growing. Um let's use a preparation for a personal final exam as an analogy. Yeah, for that um like semester slides,
[12:50] textbooks, um class notes, quizzes, recordings, all the different types and structures and thousands of pages of documents.
[13:02] The next step is that we want to extract the key insights. Uh in this case, let's say uh budgeting, credit card interest or crypto. But the third step is ours. We have to
[13:18] review and refine. And let's say you know that budgeting and credit card interest is in the exam but you know that crypto is mentioned but it won't be in the final exam. So you tell the agent that we need
[13:33] to remove it. And then the summary agent creates you the summary you need to learn for the T exam but without the crypto. Uh now we have to tie these
[13:46] together and make this loop real. And uh to understand that and to get a bit more technical I will use a um certain reference architecture uh which
[13:59] is very similar to what we build uh and I will use this as an example. Um but first let me just go over it very quickly. So the entry point is the Genie
[14:11] one. That's where people reach your apps, your dashboards, your genie spaces. And uh but an app can integrate and orchestrate all of it. And in this case, it this is a full stack app with
[14:24] all the built-in capabilities that data bricks provides. Um as a first step, let's say we talk to our out ofthe-box agents, but we don't want to stop there.
[14:36] So we uh build our own custom agents the transactional store in lake base and we refine and scale the product. The next step is orchestrating the agents and uh
[14:48] as a and for the full stack uh end to end buildout. We ship the medallion architecture that Miran mentioned. Um the whole architecture um grows and the sync loop between the
[15:04] um application the leg base and the uh delta tables through sync tables and we deploy the whole thing with with dabs. And on the surface uh this reference architecture contains three distinct
[15:18] layers like the application layer, the intelligence layer and the data plus governance layer. Uh but those layers are not as separate as it uh as the
[15:31] these boxes uh may suggest. over the last year especially the app and the agent intelligence layer fused together and um to illustrate that that let's get
[15:45] uh let's go back to our sample use case and illustrate how uh we start with the classic way. So the insights agent you can find it reads uh from the document
[15:57] and extracts the key insights from the study materials. In this case we write um them down to the silver to gold pipeline. The next step the summary agent
[16:10] generates the summary on top of that. Um yeah and as uh now the re uh results get synced uh back to the application. So
[16:24] from the go tables through the sync tables uh to the application then the application just renders it. So the data in the AI out and you just read it from the app.
[16:40] Uh and now now the interesting part happens. uh you review the key insights and one of them is wrong. So you don't uh have to learn crypto in this example and you delete it uh through the
[16:52] application. Now we could say that we just write it back to lake base then to the uh lakehouse and uh the job is done. But actually um it won't work because
[17:05] these insights depend on each other and the summary is built on the old versions and still uh still contains crypto. So we need to make our first design
[17:18] decision here. Uh your addin shouldn't overwrite what the agent generated. uh the key inside uh the original one and the user approved one s should live uh
[17:31] side by side and we record that as a user of white right in this this case let's say it's a deletion override so now um the summary um uh as an next step yeah
[17:46] the p pipeline reruns and the key part is that uh the the two agents are using this data differently. The the summary agent always wants the latest version of
[17:58] the truth. So it doesn't care uh like uh which fact came from which direction. It just reads the uh v the it it just reads the view where um uh the user override
[18:12] wins. But um yeah but the insights agent and let's stay here for a moment. the insight agent is uh trickier because on its next run it has to see uh both uh the
[18:26] original output and the one the user has changed. So otherwise it will just overwrite and but this approach um but with this approach the um edi stays and
[18:40] u it doesn't touch it and the corrected summary syncs back to the application and uh you uh can see it and read it. So that illustrates the
[18:52] whole point. The system of record uh the context of the uh agentic layer and the state of the application um are working together. And one thing we haven't discussed
[19:05] though uh is how we actually host these agents on data bricks. And for that we have three options. Uh the first one is model serving which is like uh really
[19:18] good in terms of scaling but it has its like five minutes limitation. So um and Devax is uh probably not the best for that. So probably if you have plenty of tools that's not the best option. The
[19:33] second one is lakeflow jobs uh which is uh for use cases where latency isn't the point. uh we just want strong integration with uh within our data
[19:45] pipeline with built-in scheduling and queuing out of the box. I think he has like 30 seconds >> and uh datab bricks apps is a real first class citizen uh for production uh
[19:57] agents with built-in access control, low latency, easy dev and deployment. Uh actually in practice it's rarely one or the other. It's you can combine all three. So uh that's what our time
[20:11] allowed. Uh we had so much more to cover like MLFlow's responses agent how we use that to build the same architecture and how we used agents to build apps like that. But actually we are having a brain
[20:24] date at 1 p.m. So we are happy to continue the conversation there or here for some questions. So thanks everyone for joining. This was Miran and Domo. Thank you. Keep in touch.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.