How Serverless Empowered Nationwide to Build Cost-Efficient and World Class BI
Summary
- Serverless transformation: Nationwide transformed BI workloads from performance bottlenecks and unpredictable costs to reliable, scalable analytics by migrating to Databricks Data and AI platform serverless compute.
- Unified governance: Features like AI/BI Dashboards, Genie, Materialized Views and Lakehouse Federation—all powered by serverless—enable scalable governance and self-service BI on Databricks Data and AI platform.
- Enterprise scale: Nationwide now runs 40-50% of their BI workloads on serverless across SQL warehouses, notebooks and jobs, delivering world-class analytics performance.
How Serverless Empowered Nationwide to Build Cost-Efficient and World Class BI

Talk by Ananya Ghosh, Lead Data Engineer, Nationwide
Databricks’ Serverless compute streamlines infrastructure setup and management, delivering unparalleled performance and cost optimization for Data and BI workflows. In this presentation, we will explore how Nationwide is leveraging Databricks’ serverless technology and unified governance through Unity Catalog to build scalable, world-class BI solutions. Key features like AI/BI Dashboards, Genie, Materialized Views, Lakehouse Federation and Lakehouse Apps, all powered by serverless, have empowered business teams to deliver faster, scalable and smarter insights. We will show how Databricks’ serverless technology is enabling Nationwide to unlock new levels of efficiency and business impact, and how other organizations can adopt serverless technology to realize similar benefits.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
What challenges did Nationwide face with Databricks?
According to this video, Nationwide initially faced performance bottlenecks, scalability issues, and unpredictable high spending when running BI workloads on classic Databricks clusters.
How did serverless compute help?
Serverless compute eliminated the need for manual cluster upgrades, improved scalability, reduced costs, and enabled automatic resource management for Databricks BI workloads.
What percentage of workloads are now serverless?
Currently, 40-50% of Nationwide's workloads leverage serverless compute across SQL warehouses, notebooks, jobs, and other Databricks Data and AI platform components.
What features powered the transformation?
Features like AI/BI Dashboards, Genie, Materialized Views and Lakehouse Federation—all running on serverless—enabled world-class BI that scales with business needs on Databricks Data and AI platform.
Full transcript
Good afternoon everyone. Uh welcome to my session here today and thank you for joining. Um how is everyone doing? Good. Excellent. Nice to hear that. So um let's start with a little Q&A. So by show of hands, can you tell me how many of you have recently uh made upgrades to your compute, your databicks compute or cluster or had some kind of a code change because of the DBR upgrade? Okay, a few of you u do you guys agree that u upgrading the datab bricks runtime is like putting a duct tape around your spaceship where you're trying to cover up for the OS vulnerabilities only for a short time until it's again ready for another DBR upgrade.
It's just one instance of um manual upgrades that are required to keep our clusters the classic clusters up to date in terms of the latest Spark versions and vulnerabilities. So about four years ago at Nationwide uh one of our largest consumers property casualty team um began building out their ETL workloads on data bricks primarily from um the objective of modernizing their ETL frameworks using the latest and greatest features um and also being able to cater to BI needs. uh when we started running uh the BI workloads, this is way back like in 2020 2021, we found the BI workloads were like uh reaching their performance bottlenecks.
They had scalability issues and of course the expenses, the spend was quite unpredictably high. Um cut down to June 2025. Uh currently we have at least 40 to 50% of our workloads catering to BI. uh running on serverless compute serverless across uh SQL warehouses, notebooks, jobs, DT whatnot. Now this journey from um the BI workload threshold issue to the current state did not happen overnight. So in today's session data bricks um we can talk about how data bricks serless helped nationwide build a world-class BI.
This is the functionary forward-looking statement. I would let you guys read through it and please don't forget to get those brownie points if you complete your survey for this session. All right. Um a little bit about myself real quickly. I am a passionate data warehousing and data engineering professional uh within the enterprise data office at nationwide. Um I lead the platform engineering team which where uh wherein we uh support large scale data analytics and data management platform um on boarding and education monitoring governance.
We are the account admins for data bricks at nationwide essentially um those two links that you see down there just quick trivia uh nationwide won the data democratization award in 2023. uh during the time you can see the link where I presented our unity catalog journey of uniting hive meta stores and migrating them into unity catalog. So nationwide is one of the largest uh insurance and financial services provider in the US.
We began our journey way back in 1926 when uh Nationwide was just a small insurance provider for farm owners uh helping them get the best value for their auto insurance. And since then over the next hundred years we have grown in terms of our products and service offerings. Some of our major products um would be in the space of personal auto, homeowners, commercial businesses, workers compensation. We also include investments, uh annuities, retirement funds, and we also have a bank.
Um you can see some of our rankings over there. And we are a 100 company. All right, so let's get back to the topic. Datab bricks foundation at nationwide. I wanted to give a quick timeline around our data bricks adoption at nationwide. We are one of the early early adopters of data bricks when uh it used to be a monolithic platform like a single workspace uh where the data plane was um deployed at one of our AWS accounts and we would communicate with the control plane over VPC pairing.
Um, data bricks was introduced as um like this new little bright shine little toy for our um business engineers and uh analytical scientists to run their uh machine learning training uh some of their feature engineering workloads, exploratory data analysis and things of that nature. Now very soon the capabilities of data bricks kind of spread across our organization and by the end of 2020 we had close to a few hundreds of uh users and multiple diverse use cases being built.
Now note all of these are happening within the same monolithic environment. So we soon reached all our limits in terms of um scalability in terms of concurrency uh thresholds we were at full throttle. So 2021 is when we deployed enterprise version two which was this federated model to deploy separate federated um workspaces, compute and storage by each business area. That helped us kind of relieve the pain point with a single monolithic structure.
Uh and it really expanded our use cases users by about 20 times or so. uh we did at that point which was around 2022 face start facing some issues with our classical compute uh computes around all-purpose clusters and SQL warehouses. I'll come to that on my next next slide. But this also uh opened scope uh to the point that we could do BI on data bricks as well. It's not just for your engineering pipelines or advanced analytics use cases. you could potentially look at it from a BI perspective to run exploratory use cases or connect your uh BI platforms.
So in 2023 when Unity catalog went GA that is when we began adopting UC across our federated workspaces uh with an intent of migrating off from the legacy hive metal store into Unity catalog. Um also uh it was around the time when we had when we were when we began consciously onboarding new business use cases purely on unit catalog specific workspaces as well. We were not really setting up folks on hive metas store anymore around that time frame.
Now where does uh SQL serverless or serverless uh fit in all this? That's what this slide and the next one we'll talk about. So this slide basically talks about the numerous challenges we have faced with classical compute. I'm not going to go through all the details. I have broadly classified them into three categories. Infrastructure overhead, high amount of cost and finally security updates. So infrastructure overhead was more painful for my team.
As I said, I represent the platform engineering team. We are the account admins. Uh so us and then the workspace admin we had we spent hundreds and thousands of dollars just setting up of new uh compute infrastructure monitoring um the overages monitoring looking for scope for optimization um right sizing clusters we've spent over the past I would say three or four years we've every year we look at look at opportunities to optimize our workloads um non-standard compute configuration which means that we have to educate every new engineer around please do not go and create a new compute for yourself.
Please try and reuse let's use shared compute mode. So um that was another um thing around education and configuration and maintenance of the cluster. Uh I'm not even going to delve into the scalability issues and the startup and termination times. I'm sure folks who have worked on classic computes do you know the pain. Uh recently we were analyzing debugging a particular job where it it it took the job like 10 minutes to just spin up the job cluster and the job finished in under 60 seconds.
So all the time that we're spending with this infrastructure overhead of course leads to high cost as well. So they're kind of tied together. Sluggish autoscaling, high uh startup uh times, redundant usage or spinning up of clusters. U we have teams where every developer has a single node cluster for themselves. um on on on the classical world and then finally security updates. I talked about the DBR upgrades which was manual, laborious, time consuming and it's repeating every time we have to keep on doing it.
Every time there's a new DBR uh version uh available u and then additional firewall rules are public IB exposures. That point is important because to connect to many of our application servers whether it's internal onrem cloud or external to nationwide uh we have to always consider okay what um what kind of firewall rules do we need to set is it even required it was like a there's there's an amount of guesswork there trial and error again wasting our time there and then the third point final point here on security is that we assumed that with uni catalog upgrades we will be able to kind of uh secure our workspaces in such a you don't have to worry about the manual cluster upgrades and stuff like that.
But you see adoption for some of our larger major um business areas have been a little sluggish because of the whole nature around you know you have to upgrade your code a lot of manual testing involved things of the nature. So that has not uh picked up as much as we would have expected to get rid of all the security uh aspects. Now around the same time 2023 24 um there was also two major factors which I've kind of spread it out in details but there are two major factors that were that was leading towards propelling uh use of serverless.
Uh one of them is from our technology leadership. There was a push to uh implement budget or sort of like a target um spend for all of our workloads to try and see how we can optimize how we can scale down how we can rationalize the tools that we're using. We have just like any other company mentioned it uh today during the keynote that we have uh plenty of tools at nationwide doing similar or overlapping capabilities.
So tools r r r r r r r r r r r r r r r r r r r r r r r rationalization is another aspect which is kind of pushing us towards adopting serverless as well. Uh I can uh give you more examples if you're interested after the talk. Um so that technology uh driver around budget and cost control and then there's another driver from our business leadership where they're trying to see a lot of their manual workloads which typically takes them like weeks and months to fulfill using Excel spreadsheets, access databases.
They want to see how we can automate simplify those business processes using data bricks and in such a way that it's easy for the business who are typically non-technical in nature they can probably understand SQL uh the procedure but not really write a lot of code for them uh can we build applications that will simplify their manual their weekly jobs monthly jobs so that drive towards adopting these new features which are genai and NLP based both of these along with the limitations that we saw with classic compute kind of drove us towards adopting serverless.
Now our serverless journey in terms of um testing out performance and everything as you can see here it began around 2023. So this was the same time when we also began uh adopting Unity catalog at nationwide. we didn't want to um implement or enable serverless um at that point without UC being the backbone for your data governance. So um initially the first year in 23 we partnered with a very large business acturial team at Nationwide again under the under our property and casualty um umbrella to run a few volumetric performance testing on different types of uh SQL warehouses that was available and also compare those performances with our at that point in time our nationwide enterprise standard for cloud data warehouse.
So after our volutric analysis running concurrent concurrency testing we figured out that the serverless version of SQL warehouse not classic or compute but the serverless version with all of its bells and whistles in terms of um um adapt adaptive query routing with its optimization caching options uh intelligent autoscaling was matching the same performance benefits performance uh marks as well as the cost was comparable to our uh nationwide cloud data warehouse standards.
So that those results were so optimistic, they were so positive that helped our leadership kind of give a direction that if your data is within the Delta Lakehouse, you don't need to move to another platform for your BI reporting. Try and see if you could leverage our serverless um SQL warehouse for your BI workloads. Now that led to us developing really quickly what the standards, the governance, the observability we need in place in order to roll this out across like 15 plus data teams um or lines and over 2,000 active users at that point in time.
So we were quickly scrambling making sure we are educating our users uh to use serverless SQL warehouses uh properly. We also during that time we also integrated with our enterprise data governance tools like think about anomal uh elation and the likes over serverless um warehouses. We integrated Tableau, PowerBI etc et all of our BI tools um data prep platforms. All of these integrations were tested out during that time and they worked pretty seamlessly um giving pretty um pretty high performance in terms of both performance and cost.
Now move over to 2025. 25 uh is the year where we started seeing new use cases coming up that would leverage the new um features from data bricks which includes like lakeflow connect which would include data bricks apps materialized views streaming tables um whatnot I think I have listed some of them over here dashboards were of course common so some of these new features required us to enable the other versions of serverless notebooks DT and jobs um so we began and testing with selected workspaces um which resulted in extremely positive um um I'm sorry the the ticker is not going the timer still shows me 20 minutes so I'm not sure how many how much time is thank you okay so I'm going to quickly walk through um our top three use cases that I wanted to share with you guys in terms of our serverless uh adoption at nationwide.
Some of these teams have been spearheading our journey uh or and our performance testing uh with different types of serverless computes. So our first use case is with the financial team. Our financial team has been building their financial data lake for the last two years um bringing in um data from multiple sources. One of their use case was to provide the business users a very easy simple to use lightweight application that would let the customers uh come and review their uh reference data and also make updates real time.
Uh note that this process in the current state required several spreadsheets and several weeks of work. So with uh by using materialized views, streaming tables and lakehouse apps, we were able to build this simple lightweight low latency uh application and uh users are able to make their updates real time pretty much near real time. So we were able to save more than 50%. I've mentioned 50% here but it's much more than that in terms of saving of cost, effort, um performance of course.
Our second serverless use case is from the marketing business area. So marketing team typically they run the entire marketing life cycle for nationwide um and they have been also building out a marketing data hub for the past couple of years kind of they manage all the campaigns uh social media uh stuff etc. So uh they have close to 100 jobs which have been running on classic compute. This year they took up a um a project to try and see how they can minimize this and uh we saw close to about 60% performance gains.
There were jobs that were running for like two hours which reduced down to less than 5 minutes. I have given an example here of top three jobs uh where we saw a significant amount of uh performance gain. My third use case is for a specific data governance uh scenario where we are able to monitor data quality uh over our entire data lakehouse which comprises of delta tables comprises of uh foreign tables that we are uh bringing in through lakehouse federation and also delta share uh tables.
So, we're able to use our enterprise standard data quality tool to measure the quality of all of these different types of heterogeneous uh sources and be able to spit out their um their accuracy, their completeness in terms of the quality of data and make the data ready for BI consumption. These are some of our upcoming uh futuristic use cases. I'm not going to go through the details of it. Um I actually wanted to bring this slide up sooner but the main takeaway from this this slide would be the top three best practices for implementing serverless in your organizations.
There are several other governance rules that uh we have been documenting and capturing but definitely unity catalog framework as your governance baseline making sure you're using budget policy. Budget policy does have some limitations right now but making sure that is a requirement to use serless of any type. And then finally making sure that we have observability alerting through our AIBI dashboards and based on system tables, billing and usage tables.
So those are the prerequisites key benefits and um I just want to call out that we have seen significant amount of uh performance boost as you can see here um loaded but one of the major opportunity areas or challenges would be we could not use budget policy for um setting up thresholds or uh shutting down jobs in case we exceed a certain threshold. So we are still working with data bricks to try and see how we can implement this in the future. right now none of the features support this requirement.
Um and then these are the last three takeaways. So our serverless enterprise rollout has been tremendous and it has boosted our business capabilities and new use cases being built out. Uh we have definitely driven uh efficiency at scale being able to reduce uh a lot of the manual work uh in terms of seconds and real time. And finally, we wouldn't have been able to roll out serverless at an enterprise level if we didn't have the governance controls in place.
System tables have been of hu huge uh benefit for us. We're able to exactly measure how much uh we're able to slice down the serverless consumption by um by a cost center by resources by teams business areas. Um as I mentioned, we're not able to apply a threshold to turn down serverless usage if we reach those uh pain points. Um we do have enough governance and observability in place in place for us to monitor that. With that um thank you and happy learning.
Any thank you
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.