Real-Time Insurance Pricing with Lakebase and Serving Endpoints

Summary

Ergo Hestia, one of Poland's top-three insurance companies, replaced a standalone Azure Postgres database and adapter layer with Databricks Lakebase and Serving Endpoints, achieving 20-millisecond latency and 400 requests per minute for real-time pricing.
The new architecture supports over 100 pricing models using more than 1,000 variables across 60 source systems, with data scientists able to deploy new models immediately rather than waiting for overnight batch processing windows.
A baby-steps rollout with progressive traffic migration, MLflow model management, Unity Catalog integration, and Azure DevOps CI/CD pipelines enables safe production deployment and validation of real-time pricing at scale.

Real-Time Insurance Pricing with Lakebase and Serving Endpoints

Ergo Hestia, one of Poland's largest insurance companies, built a real-time pricing system on Databricks Lakebase and Serving Endpoints to eliminate external databases and reduce latency. Previously, pricing models relied on an Azure Postgres database and adapter application layer that created bottlenecks, forcing large data loads only at night and limiting real-time model updates.
Learn how Ergo Hestia implemented a unified Databricks platform for real-time data and model serving, achieving 20-millisecond latency and 400 requests per minute with a baby-steps rollout. See how synchronized tables, MLflow model management, Unity Catalog integration, and Azure DevOps CICD pipelines enable data scientists to deploy new models instantly without overnight windows. Discover the architecture supporting 100+ pricing models across 60 source systems.
📂 Databricks Blog: https://databricks.com/blog
🤝

Chapters

FAQs

What is Databricks Lakebase and why did Ergo Hestia use it for insurance pricing?

Databricks Lakebase is a managed PostgreSQL service on the Databricks Data and AI platform that provides low-latency transactional data serving. Ergo Hestia used it to replace a standalone Azure Postgres database and eliminate the adapter application layer that created bottlenecks and forced large data loads to run only during overnight windows.

What performance results did Ergo Hestia achieve with their new real-time pricing architecture?

Ergo Hestia achieved 20-millisecond latency and 400 requests per minute for real-time pricing queries after migrating to Databricks Lakebase and Serving Endpoints. Their previous architecture required nightly batch loads and could not serve pricing requests in real time.

How does Ergo Hestia manage over 100 pricing models in production?

Ergo Hestia manages their 100-plus pricing models using MLflow for model lifecycle management, Unity Catalog for governance, and Databricks Model Serving Endpoints for deployment. Azure DevOps CI/CD pipelines automate model promotion so data scientists can push updates to production without manual overnight operations.

What was Ergo Hestia's baby-steps rollout strategy for Lakebase?

The baby-steps strategy involved gradually migrating live pricing traffic to the new Lakebase and Serving Endpoints architecture, starting with a small percentage and progressively increasing as validation confirmed the system was performing correctly. This approach allowed Ergo Hestia to build confidence and catch issues early without risking a full production cutover.

Full transcript

[00:08] My name is Claudia Rutkowska. I am data engineer working at pricing department in Ergo Hestia. And today I will present you about lake base and serving endpoints implementation in real-time pricing. Uh so first some business context.
[00:24] First I will say few words about Ergo Hestia. So Ergo Hestia is an insurance company operating since 1991 in Poland. We offer life and non-life insurances. Currently we are at number three
[00:42] based on gross written premium, but we have huge beliefs to be number two soon, so keep your fingers crossed. Uh provide comprehensive solutions for retail and commercial with national coverage.
[00:58] Um so what is pricing you may ask. So pricing in Poland is super competitive. Uh to be number three or soon number two in Poland, our our pricing has to be at the state-of-the-art. It has to be great. So in pricing department we work
[01:16] on maintaining and developing models and data processes. So modeling scale, this is basically uh maintaining more than 100 models that use more than 1,000 variables.
[01:33] Our models uses used GLM and machine learning techniques. Modeling scope, basically I can divide it into four groups. So first one is technical price. This is basically estimating
[01:49] uh the price of claim in the insurance you're taking. Uh second group is demand. Third group is street price. So, this is basically price calculating based on technical price and the demand that is
[02:05] then being optimized. And the fourth group is predicting and helping clients claim handlers to resolve claim claims that are being reported to us. Uh our models can work perfectly if we
[02:22] provide great data. So, we use more than 15 external data sources. Uh most of those data sources are provided from our government that are regulated. Uh so, in our governmental databases,
[02:37] you can find information about uh history of insurance of our clients and also a lot of data about our vehicles. Uh we also use our internal data, which I can divide into three groups.
[02:54] Those are policies that are ever taken out, claims reported to us, and all the calculations that agent agent made. Um okay. How the previous architecture looked like? On the right side, you see pricing
[03:09] engine. And this is basically a system where all the calculations from agent come. Pricing engine contain models. But how does models get data? Well, on the left side, you see Databricks, which is our platform where we replicate all
[03:26] the data we use. So, from external and internal sources. Then we we run all the transformations and processes, and then we load this data to Postgres external database. Uh you also see adapter application,
[03:43] which is basically an application for which pricing engine can get data from this Postgres database. In this adapter application, all the SQL queries are being formed. Plus, it also work as a cache because we don't want to
[03:59] stress our Postgres database too much. So, adapter layer is here. Works as cache and as SQL forming tool. Um It didn't work perfectly. So, when we heard about Lake Base and serving
[04:17] endpoints in Databricks, we thought, "Oh, well, maybe this is a great solution. Maybe this is way to go." Uh so, four things that we wanted to improve was and change was, first of all, eliminating external database,
[04:32] which didn't work as well as we wanted because we operate with huge volume of data and this huge volume of data we could load just at night because during the day it was being stressed too much, even though we have this adapter layer.
[04:47] Second one, we wanted to consolidate storage and serving. We wanted to have our data being served and stored in this one Databricks platform. Third one, simplifying architecture. We didn't want to maintain this external
[05:04] Postgres database. Plus, we didn't want to maintain adapter application. And last but not least last least but last but not least, we wanted to reduce latency. We want our models to use just the most fresh data we could provide.
[05:25] Another solution overview. So, this is our target target architecture. Uh on the right side, you see pricing engine. And what's the difference now that this pricing engine doesn't contain models inside anymore. So, now I I also have to mention that in
[05:41] pricing uh major number of calculation are being processed dynamically. So, we use models to process them, but some of them are still being processed uh without models, not dynamically. So,
[05:57] when we use models, then basically pricing engine has to send free request. First request is request for data to price point to Databricks. Uh so, basically in Databricks, we use lake
[06:13] base to provide data back. And then, based on this data, pricing engine creates request to send it to model endpoint in Databricks. Uh
[06:28] then, based on the return, we send the first request from pricing engine, and this is a request to optimize the price. So, it's really great for client. Uh when we cal- when we have calculations
[06:46] that is not dynamic, we just send just one request, and this is just request for data. So, this is a static model, basically. Um okay. Now, end-to-end data flow. How it looks like.
[07:02] So, if it comes to serving data, first, of course, we have to ingest data. We have to replicate all these our data to Databricks. Then, we process it and transform it. And then, those final tables we synchronize to the lake base.
[07:23] Then, we create uh endpoint, uh which we then can govern and monitor to to see how many requests we get, what's the latency, if we should we should have higher or lower compute.
[07:38] Um then, end-to-end model flow. So, first three steps are steps being made by our data scientist. Our data scientist first extract features. Then, they create a model using MLflow.
[07:54] They create it, they experiment it, they train it, and validate it. If it works well, then this is task for data engineers. Data engineer engineers register this model in Unity Catalog.
[08:10] Uh and then they create certain endpoints, which they can serve multiple versions of this model if they want to. They choose the right compute. And then they also monitor the performance of new of the new model.
[08:30] How the implementation look like? So, of course we wanted uh to avoid um huge business impact in case of in case of failure. So, we had this baby steps strategy. It basically means that we
[08:46] started with small portfolio and smaller data volume. So, in case in any failure, our CEO wouldn't be upset too much. And so, we wanted uh to validate four things. First, we wanted to confirm that the performance in latency isn't higher
[09:04] than it was before. Uh then we wanted to check if Lakehouse is stable even when it's when we send a lot of requests for each tables, if it works well. Uh then we wanted to check certain endpoints
[09:19] uh if we don't have any problem with REST API. Uh if the connection works well uh from our pricing engine. And then we wanted to reduce migration risk. It turns out that it works perfectly.
[09:35] We were really happy with the results. Um how the Lake Base implementation look like? Well, to implement Lake Base uh and third data, uh first we needed to start Databricks
[09:51] Lake Base instance. Uh we started with provision Lake Base, in which uh you can set compute units, but then we migrated to auto scaling one, and it works perfectly. Why? Because if we have uh instance in our
[10:08] uh testing environment, we can scale it to zero at night when we run no tests. Uh but on production, when we have a a lot of requests from agents during the day, we can scale it up and then down at night. So, this is perfect. Plus, we can
[10:24] also use branching. So, cool. Then we created synchronized tables. Those are tables that are being stored in Lake Base. From those tables, uh data endpoint is taking all the data. Then feature spec. You can think about
[10:41] it about like a a function in which you can define from which Delta table you want to get the data, uh what are the key lookups, what features you want to return. Plus, also you can define some quite simple functions.
[10:58] Uh then Lake Faults support declarative pipelines. Those are Those are pipelines that we use to synchronize the data from Delta tables to the Lake Base. Uh it works pretty fine because we can trigger them when we want. They can be
[11:14] triggered in the our Delta table detects any change. And also, we can synchronize this continuously. So, basically, we have real-time data that is being saved. And at the end, sorry, endpoints. So, entities where you can serve your
[11:30] feature spec, uh where you can define how we compute it is, and also where you can govern who can send requests to this endpoint. Uh this is how this architecture look like. So, yeah, on the left side you
[11:46] have you see data tables, so those are our final tables from all our processes. Then you see uh the pipeline that is synchronizing this uh data newly added data to this table to the lake base.
[12:02] And then you see serving endpoint that is directly directly taking the data from the lake base. Uh at the bottom you see feature spec. So, this is this function that is defining which table should be being read
[12:17] uh based on which which column. I think my microphone going correct. And this is our model serving implementation. Uh to start serving our models,
[12:35] uh we implemented those five components. So, first of all, we use MLflow uh to uh create our models, to write them, to track experiments, uh and to register them. Uh then
[12:51] you need to catalog is a place where we register versions of our models, and where also we can tag the versions. For example, if I see that this version is the right one and I want to run it to production, I tag it. Also, I I can give all the
[13:07] permissions for the right people, the data scientists work on this model. Then uh serving model endpoints, entities in which I define which versions of which model I want to serve, how big computes I want to I want to
[13:24] set, uh who will have permissions, and this also this so compartment which I can monitor the performance of this endpoint. Then we use Azure DevOps. In Azure DevOps we created CICD
[13:42] deployment pipelines and those pipelines work really cool. On the left side you see YAML file and in a file like this we can define that if in this directory you detect any change please run deployment production
[13:59] of this model to this endpoint. And indeed inference tables those are tables in which we store all the requests and responses to model endpoints. Also in those tables
[14:14] you see if there was any error or it was code 200 so model responded correctly. Also see latency also you see who sent the request so you can monitor performance perfectly.
[14:33] And this is how the schema look like. So on the left side you see ML flows so this is where data scientist register the new the model validated and if everything works works fine they just create pull request in our Azure DevOps in our Azure DevOps repository.
[14:50] So when it detects the change it is asking data engineer for approval and then when they click approve then base this pipeline automatically registers the new version in Unity catalog it sets the right permissions for the right
[15:06] groups and then it it deploys the new version of the model to the serving endpoint. At the bottom you see inference table so those are the tables as as I said that in thanks to them we can uh
[15:21] monitor the performance. And this is how our our timeline looks like. So, we uh created this new solution in Q4 2025. Uh as I said, it was proven a concept
[15:39] and baby steps strategy. So, first the traffic wasn't high. It was around eight requests per minute. And we started with five data endpoints and two model endpoints. Uh but by the Q2 2026, the traffic
[15:54] reached around 400 requests per minute because we had already seven data endpoints and six model endpoints. Um and what's the plan? Well, since we are happy with the results, since it it works really well,
[16:11] uh we want to migrate more models. We want to serve more data endpoints. So, estimated traffic is around several thousand requests per minute.
[16:28] And the evaluation. So, in previous data architecture, which worked okay, but not the best, uh data of course is replicated, ingested, and transformed in Databricks, and then loaded to external Azure Postgres database.
[16:43] Then we needed to use this uh adapter layer to cache it and to access the data. So, this is how pricing engine was taking the data. Current up we have direct access to those tables.
[17:00] Uh there is no end adapter layer anymore. So, also the latency is way lower. Uh we can use continuous uh synchronization of those tables, so we can have fresh data in real time,
[17:16] which is perfect. Um then pricing engine engine can just send uh REST API request to our to our data endpoint. Uh so, again, latency is super low. And um
[17:32] and in the end, we have fully managed solution within Databricks without any external components. We don't have any Postgres database anymore, so we can load really huge uh values of data even during the
[17:48] day. Um model architecture. Um so, before every time we wanted to make any change to deploy new model to production, uh it needed to be during the night. Why?
[18:04] Because during the day our pricing engine has to it has to answer thousands of requests per minute uh from from our agents. So, just at night uh our data scientist and another software
[18:20] engineer needed to deploy the new model to to our pricing engine. We didn't have any real-time monitoring of this of this model. Uh so, so, it was just problematic for our data
[18:37] scientist. Um well, in current architecture, we don't have this problem anymore because thanks to inference tables, we can uh we can get all the calculations now. So, if they see that the perfect performance of
[18:52] the model isn't the best, it's really easy for them to deploy the new version. They don't need to wait uh for the night. They don't need to wait for the software engineer. They can just do it now. Um so, this is really good. This also answers the problem. So,
[19:10] in Poland, uh things can change quickly. So, if they if they need to change model now, they can do it now. Um and our performance results. So. Um when I was talking about this
[19:25] latency, this is latency of our data endpoints. Uh it reaches around 20 milliseconds. And as you see, there is this point of time in time when our synchronizing the table. As you see, the average latency
[19:43] remained the same. Uh which is pretty cool. Uh we can synchronize those tables all the time. If you want to, we can trigger synchronizations if you want to. And also, uh no matter if it's night,
[19:59] morning, middle of the day, uh there is uh latency becomes the same. I mean, it's always around 20 milliseconds, no matter how if it's peak of the day or not.
[20:19] And here are three key benefits that I want to talk about. First one, uh now we have unified Databricks platform for receiving data, for serving data, and and models. Uh which is pretty cool because if our data scientist needs new features, can easily deliver
[20:35] it for them. Uh then the second one, uh we reduced uh the architecture, and now our solution is really flexible. So, again, if any change change need to be done, it is
[20:51] being done as quick as they want. And the third one, uh latency is really low. We are really happy with the time. And also, for the models, uh Uh, it just works perfectly.
[21:11] Yeah, and this is the resource deck. So, I would like uh you to read our article that we published last week on Databricks blog. This is about earning solution um which used in real-time pricing in Ergohestia. So,
[21:27] thank you for your attention. Uh if you have any questions, then I'm I'm happy to answer. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.