Nationwide's Serverless Transformation: 29% Cost Reduction

Summary

Nationwide migrated to Databricks Serverless in three phases — SQL warehouses, Lakeflow Connector, and all-purpose compute — achieving 100% adoption in core areas, a 29% overall cost reduction, and a 33% reduction in idle compute costs.
The primary motivation was eliminating 8-minute compute startup times that imposed an infrastructure tax slowing development, freeing engineers to focus on business outcomes instead of cluster management.
The video covers where serverless creates the most value within the Databricks Data and AI platform, what new capabilities it unlocks such as predictive optimization and automated table-level TTL, and which workloads require more intentional planning before migration.

Nationwide's Serverless Transformation: 29% Cost Reduction

Nationwide's data team discovered that powerful architecture alone doesn't drive faster innovation. Despite strong governance with Unity Catalog and Delta Lake, 8-minute compute startup times created an infrastructure tax slowing development. Serverless shifts infrastructure management from teams to Databricks, freeing engineers to focus on delivering business outcomes.
Learn Nationwide's three-phase migration to Databricks Serverless, from SQL warehouses to Lake Flow Connector to all-purpose compute. See how they achieved 100% adoption in core areas, cut costs 29%, reduced idle costs 33%, and unlocked capabilities like predictive optimization and automated table-level TTL while building enterprise security into the platform.
🤝

Chapters

FAQs

What cost savings did Nationwide achieve by migrating to Databricks Serverless?

Nationwide achieved a 29% overall cost reduction and a 33% reduction in idle compute costs after completing their three-phase migration to Databricks Serverless. They also reached 100% adoption in their core areas of platform use.

What was the biggest infrastructure problem that serverless solved for Nationwide?

Nationwide's teams faced 8-minute compute startup times with classic clusters, creating an infrastructure tax that slowed development velocity and required engineers to spend time managing compute rather than delivering business outcomes. Serverless eliminates this overhead by shifting infrastructure management from the team to Databricks.

What are the three phases of Nationwide's serverless migration?

Nationwide's migration proceeded in three phases: first SQL warehouses, then the Lakeflow Connector for data ingestion workloads, and finally all-purpose compute for interactive development. This phased approach let the team build confidence and address governance and security requirements before expanding to more complex workload types.

Is serverless the right choice for all Databricks workloads?

According to Nationwide's experience shared in this video, serverless is most valuable for workloads where compute startup latency causes friction and where idle cluster costs are significant. Some workloads with specific infrastructure requirements need more intentional planning before migration, and the team explicitly cautions against treating serverless as a universal solution.

Full transcript

[00:08] Hello everyone. Thank you for joining us today. I have Richa here with me from Databricks team and we are talking about less infrastructure and higher impact. How serverless transform our customer experience. Before we get into the presentation, let me ask you all a question.
[00:24] How many of your team have built something you were really proud of only to watch your customers struggle to use it? Sounds familiar? I see quite a lot of hands. That was our experience last year. In case
[00:40] by seeing this title, if you thought we were going to come and stand here and tell all of you serverless solves everything and you all need to migrate immediately if you haven't yet. That is not what this presentation is about.
[00:56] What we want to talk about is where serverless makes more sense, where it reduces operational friction, where it improves adoption, and where it may require much more intentional planning and strategy. So in this presentation, you will hear
[01:13] five important areas. What is serverless and why it matters. What it unlocks in the Databricks platform. Our Nationwide serverless transformation journey. What coming next in the serverless space and one key takeaway that you want to
[01:30] leave all of you from this presentation. All right. Before we get into the topic, let us quickly introduce ourselves. I'll give a quick snapshot of my background. Hello again everyone. I'm Pravina Edward, a lead data engineer on
[01:46] the data management and governance team from security infrastructure department at Nationwide based in Columbus, Ohio. In my role, I primarily manage Databricks workspace for our S&I organization. And
[02:02] I'm very passionate about the helping the teams to utilize full platform capabilities to enable them move faster and deliver meaningful impact. And outside of the work, I'm also a dancer and a bookworm. If any of the bookworms here, high five. That definitely brings
[02:20] influence on how I show up at the work. And I'll turn it over to Richa for her introduction. Thanks, Praveena. Hi, everyone. I'm Richa Sethi. I'm a delivery solutions architect at Databricks. And I sit at the intersection of
[02:36] business strategy and technology execution. I help organizations like Nationwide that have ambitious data and and AI goals and help turn those ambitious goals into production outcomes. The favorite part of my job is
[02:51] to help customers navigate through the change, like helping them move from legacy platform to a modern platform like Databricks, help with serverless adoption, helping build a solid foundation for UC
[03:07] governance, all doing it while safely and responsibly. So, today Praveena is going to talk about Nationwide's journey of serverless adoption, but I'm here to give you some context on why organizations across industries are making similar decisions.
[03:24] And what are the possibilities that open up for you when the burden of infrastructure management is taken away from you? So, when we talk to organizations, one common theme across them is that their teams spend majority of the time
[03:41] upgrading their up upgrading their platform, doing security patches, tuning Spark configuration, managing capacity, managing clusters, and so on. Now, all this is super important, right? I mean, that keeps our jobs and work
[03:56] going, but none of these activities actually helps solve a business problem, improves customer experience, or helps deliver a use case. Right? So, the common theme across customers is that they have highly skilled engineers, but
[04:12] they are spending majority of their times operating on the platform and not innovating on the platform, and that's exactly what I mean by infrastructure tax. It's the cost of complexity that slows down innovation, right? And serverless removes that tax.
[04:30] But, what exactly is serverless? Serverless, as at its core, is shifting the responsibility of managing the infrastructure from you to Databricks, right? So, again, that means you don't have to worry about upgrading your DVRs
[04:46] or waiting for clusters to start or managing those security patches or thinking about capacity, right? Databricks manages that for you. Also, the compute scales as your workload changes, right? And the compute also optimizes behind the scenes, so that you
[05:02] don't have to worry about it. And guess what? When this uh when this burden of you managing the platform, you managing the infrastructure goes away, you get faster access to data, that means faster experimentation, and faster delivery of
[05:18] use cases. So, now your engineers can spend more time building pipelines, dashboards, genie spaces, and apps without worrying about the resources that's helping them build those uh assets on top of the data that you already have in Databricks.
[05:39] So, we get it, right? The technical benefits of serverless are super compelling. You get less than 10 seconds cluster startup. You get 50 to 100 times speed up faster than classic performance. There is better price performance and you get this enterprise grade reliability. But these metrics
[05:55] only matter for what they bring to each persona. So for example, with serverless data engineers, their pipelines start in seconds, so they can build their pipelines without waiting for compute to start. The data scientists and ML engineers can
[06:11] get access to resources as soon as they wait without having to wait for the infrastructure. SQL analysts can quickly get answers to their questions from Genie spaces and dashboards, again without waiting for compute to start up. And platform teams,
[06:27] well, for them it means fewer tickets, less operational burden, and less things to manage overall. So each persona, as you think, as you see here, gets back some time. And when they get back time, it means that innovation increases, right? Because there is less complexity,
[06:43] they have more time to work on real things. And it's not just the operational simplicity of serverless, right? I mean, yes, there is operational simplicity, you also have more efficient workloads. But as you can see here, there are a plethora of capabilities that you get
[07:00] only with serverless. Just to name a few from here, you have model serving AI gateway, vector search, Databricks apps, materialized views, Lake Flow Connect, anomaly detection, data classification. The list is endless. Right? We are able to provide you these
[07:15] capabilities because Databricks manages the compute for you without you having to manage the compute. Right? So the What What is clear here is that when you adopt serverless, you not just get operational simplicity, but then you are also positioning yourself
[07:32] to adopt the capability for next generation of data and AI. And let's talk about the security story because that is super important. Because traditionally, when you think about security, security relies heavily on the use of configurations in the platform.
[07:50] Right? Today, platform team or or maybe the security teams, they create these policies and create configurations and attach those configurations to the cluster. But guess what? These configurations are only make They only make sense if each
[08:06] of the clusters are actually using these configurations correctly and each end user is using the best practices. Right? With serverless, that's not the case. Serverless shifts the model from configuration driven to platform driven.
[08:21] Because with a configuration driven platform, there is a scope for misconfiguration. There there is drift. There are inconsistency. And because serverless runs on a Databricks managed platform, it already has a hardened compute. You have network isolation. You get automated encryption.
[08:39] You get centralized auditing. You have Unity Catalog governance. You get all the security that you need for your team right away. There are no security hatches, either. It's not like someone from the team can simply bypass these policies and start and not use these policies. Security is
[08:56] built in and not bolted on in serverless. Right? It's not just again reduced infrastructure operational simplicity. It's also that we are reducing the attack surface of your organization when you use serverless.
[09:12] With that, I'll pass it over to Praveena to help us translate the story to us for Nationwide. Thank you. All right. Richard just covered what is serverless and why it matters. And I'm going to talk about what it feels like
[09:27] to live inside that transformation. But before we get into the problem statement that led us to a serverless, let me give a high-level overview of our team. My team, data management and governance team, exists to enable our S&I organization to make informed decision by providing
[09:44] trustable data available in the centralized location. We have 14 data engineers, nine data solution analysts, and one governance specialist to support that mission. So far, we are managing 140 plus data sources and 1,000 plus data objects,
[10:01] which includes tables and views, and also 400 plus jobs that brings data from various sources to a centralized location in the Databricks in Delta Lake. And that work so far enabled 55 plus teams to consume the data from the
[10:16] centralized location supported by 50 plus SQL warehouses, 30 plus all-purpose compute. And also, we are supporting 20 plus data products. All right. Let's get into the problem statement.
[10:37] The hidden cost of scale. When we migrated to Unity Catalog last year, we thought we had solved the hard part. The architecture was stronger, governance improved, and the platform capabilities expanded. From the surface, everything looked good, kind of like an
[10:53] iceberg. What you see above the water looks strong and complete, but what we reminded of is that the part that are not visible is equally important as the part that are visible. Powerful architecture alone does not
[11:08] make team move faster. Our customers kept telling us the same thing. We got this powerful, great architecture, but we still don't feel like we can move faster enough. When we looked closer, the issue was no
[11:24] longer an architecture. It was an infrastructure underneath it because our productivity was still waiting on compute. With cluster backed SQL warehouses and workflows, customers often had to wait for at least 8 minutes for them to get
[11:40] started. It felt like we built a Ferrari and then we need to wait for at least 8 minutes for the engine to start for every single time you wanted to drive it. To reduce that wait, we increased auto stop time. That definitely helped our experience a
[11:55] little bit, but it introduced another problem, idle compute cost. Now, imagine that idle compute cost multiplied by 55 plus teams. For a while, we thought maybe this is how the workflow had to work.
[12:10] But when the feedback kept coming, that's when we reached out Databricks team in our monthly touch base and that is how serverless became important for us. Because serverless gave us faster access, better developer experience, and
[12:26] the ability to run our workflows in a more secure and scalable way. We didn't move to serverless just because it was a new feature or looked shiny, but we moved it because we needed our productivity to scale without infrastructure becoming the bottleneck.
[12:42] Now, our Ferrari starts whenever we want to drive it and we pay for only what we drive. All right. Here is how we made that happen. We moved it to Unity Catalog last year September and we decided to move to
[12:58] serverless in October last year. Our plan is not to migrate everywhere at once. We were very intentional about what we prioritized and we prioritized SQL warehouses over all purpose and job compute because that's where a 90% of
[13:15] our customers consumption were running on SQL, which includes BI tools, AI, cyber SOC investigations, and internal reporting. So, that the prioritized and we hit 100% adoption and uh by February this year.
[13:33] That is our phase one. In phase two, we prioritized um Lake Flow Connector, workflow, uh job computer bring service now data to support our enterprise level reporting. Um then we hit it 100% adoption there in March 2026. And then we prioritized lot
[13:50] all purpose compute and job compute because that we took that last because it's internal so only for our team. And also, it is complex than SQL warehouses. So, we have listed down all the dependencies that we would need and we are trying to find an alternative
[14:06] solution for all the dependencies that we identified. So, we are in exploration phase and we are estimating to hit 100% adoption for the in-scoped items by September 2026. And I also want to call out our plan is not to go serverless in the entire
[14:21] workspace. We de-scoped streaming jobs, monthly and weekly jobs because either it runs 24/7 or it has very low idle cost.
[14:37] And here is what that strategy delivered. We saw five immediate impact. One, our startup time went down from 8 minutes to seconds. And two, our SQL adoption hit 100%. The feedback we got from our customers from data collector meetings and working session told us prioritizing
[14:54] SQL warehouse over all purpose and job compute was the right move. And three, Lake Flow Connector adoption hit 100%. All those three had immediate impact on another important area that we are focusing on this year, cost optimization. Our overall cost went down
[15:12] by 29%. Our idle cost went down by 33%. I also want to call out two important feature that serverless unlocked in our space. One, predictive optimizer, now we don't have to think about the vacuum, one less thing to work think about in
[15:28] our workflow. And two, auto TTL feature. Especially the data team exists in the cybersecurity area, it is crucial for us to have control over the retention policy at the table level. And this auto TTL feature helped us to have control
[15:43] over in our workspace. At the table level. And but if you ask me to call out one biggest win from our journey, it would be our customers finally work at the speed they need without paying the price of waiting.
[16:01] All right, I'll turn it over to Richa to talk about what's coming next in the serverless space. All right, so what's coming on the platform? The future is clear. We you'll you'll continue to see that Databricks will have serverless first philosophy on all their features going
[16:16] forward. Some of the top features that are coming and there are tons of break breakout sessions planned for this. So I highly recommend go attend these sessions. They are headed by our PMs who'll tell you what these upcoming capabilities are, what the
[16:32] timelines are. So please go look out for them. But the four common themes, four themes that they are that we are working on are cost controls, um governance, migration acceleration, and security.
[16:47] So something that's coming in cost controls is rate limits rate limits spending controls because that's something that we have heard a lot from you all. So that's coming very very soon. Um we recently launched migration migrator acceleration. So we recently
[17:03] launched the skill that helps you migrate from classic to serverless. So if you haven't seen it, go check out our docs. It's a great skill. Use GD code help convert your classic workloads to serverless workloads. And then there are tons, as you can see, df.cached and
[17:19] support wasn't supported for the longest time. That's coming up. Log delivery, again, that we heard was a common gap or a common thing that our customers wanted. They wanted delivery logs in their platform to analyze those logs. So, that's coming. There is cost
[17:35] optimized mode coming mode coming with fallback, so you don't have to think about spot instances. Insights enhanced insights observability genie debugging optimization, and then configurable publication publication delay before package availability. And that's related
[17:50] to the recent supply chain attacks that we saw. So, with that, I will pass it over to Pravina to wrap it up. All right. We are in the end. As we're wrapping up this one, here is
[18:06] one thing we want to leave you all with from this presentation. A powerful platform creates infinite possibilities. It is also very easy to get lost in everything that's now available. But, here is what this journey reminded
[18:22] us. Possibilities alone don't create impact. Possibilities and intentional choices do. Our customers saw the value because we chose what mattered most to them from the platform offered. It's not about having access to
[18:38] everything. It is about choosing the right thing for the right reason with your customers in mind. And when that happened, here is what that impact looks like. When we sent out a survey to get to know our customers serverless experience, we got 100% positive
[18:55] response. And that speaks more than a metric. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.