Modernize ITSM with Databricks Lakeflow Connect: Nationwide Case Study

Summary

Nationwide modernized a decade-old, heavily customized ServiceNow deployment by replacing a fragmented Oracle-based data path with a unified lakehouse built on Databricks Lakeflow Connect, enabling adoption of ServiceNow out-of-the-box features for the first time.
Lakeflow Connect captures ServiceNow configuration management database auto-discovery data through incremental CDC pipelines with automated quality controls and delete capture, all governed by Unity Catalog in a medallion architecture.
The modernized platform supports executive reporting, enterprise-wide IT metrics, and full historical auditability, shifting Nationwide's IT operations from reactive incident management to proactive decision-making.

Modernize ITSM with Databricks Lakeflow Connect: Nationwide Case Study

Nationwide modernized a decade-old customized ServiceNow system by moving from fragmented Oracle data stores to a unified lakehouse built with Databricks Lakeflow Connect. The new architecture captures configuration management database auto-discovery, uses incremental CDC pipelines, and enables governed access through Unity Catalog.
Learn how Nationwide eliminated manual API orchestration, connection drops, and data quality gaps by adopting ServiceNow out-of-the-box features with the lakehouse. The session covers the medallion architecture, automated quality controls, delete capture, three-phase implementation roadmap, and how modern data foundation enabled proactive decision-making for IT operations. Real results include executive reporting, enterprise-wide metrics, and full historical auditability.
🤝

Chapters

FAQs

Why did Nationwide decide to modernize its ITSM data architecture?

Nationwide had used ServiceNow for approximately 10 years but accumulated so many customizations that upgrading to new out-of-the-box features became extremely difficult. The legacy Oracle-based data path also introduced significant technical debt through manual API orchestration, connection drops, and data quality gaps that created ongoing operational risk.

What is Databricks Lakeflow Connect and how does it enable ITSM modernization?

Lakeflow Connect is a connector framework that enables incremental change data capture pipelines from source systems like ServiceNow into the Databricks lakehouse. This video describes how Nationwide uses it to ingest ServiceNow CMDB auto-discovery data, apply automated quality controls, and capture deletes—capabilities that were manual and error-prone in the prior Oracle-based architecture.

How does Unity Catalog support governance in Nationwide's new ITSM architecture?

Unity Catalog provides centralized access control and discoverability for the ITSM data stored in Nationwide's lakehouse, enabling governed access for different teams across the security and infrastructure organization. This video explains that unified governance was a key design requirement to ensure the new architecture met both security and auditability standards.

What business outcomes did Nationwide achieve from modernizing its ITSM data platform?

The modernized platform enabled executive reporting on IT operations metrics, enterprise-wide visibility that was previously unavailable, and full historical auditability across the CMDB. This video describes the shift from reactive incident management—where data limitations delayed awareness of problems—to proactive decision-making supported by current, governed data.

Full transcript

[00:08] Good afternoon everyone and welcome. My presentation today is ITSM modernization with Lake Flow Connect. I'm excited to share Nationwide's modernization journey and what we have learned along the way. Before I begin, I just wanted to have a quick show of hands of people who have
[00:25] tried to modernize their ITSM in the recent time using the latest features from ServiceNow. Yeah, I think quite a few of them. So, I hope with this session, you know,
[00:41] you'll have some good insights in terms of, you know, how to design and implement using the Lake Flow Connect. A quick introduction about me. I'm a data architect at Nationwide in the security and infrastructure organization. I've been with Nationwide as a full-time associate for the past 3
[00:56] years and before that I worked as a consultant for about 6 years. Throughout my time here, I've been at the intersection of business and technology trying to build systems, you know, that give good business outcomes along with them being secure and stable.
[01:14] On a personal note, I also coach the high school students on the SAT math and so that has helped me shape some of the way I communicate in a simpler and practical way. And for balance, I play badminton. So,
[01:30] that keeps me grounded, resilient and also reminds me that I have to take the wins and losses, you know, with the same mindset. So, with that, I'll get started.
[01:47] So, I want to start off with the big picture. What has been our Why has been our re-implementation objective, you know, at the top of some of the transformations that we are doing. So, we have been having ServiceNow for quite a number of years now, close to about 10 years I would say. And so, what
[02:03] has happened is over the number of years what we have done is we have done a lot of customization and basically that has led us to not adopt any out-of-the-box features that are available, right? So, without the out-of-the-box feature, what happens is every time you know, we need to make some upgrades, it makes that journey
[02:20] very very difficult. Uh the next shift is that for the data path. Uh so, we've had our legacy ITSM data in Oracle and the way Oracle solution was orchestrated was there was a lot of hands-off in the
[02:36] middle and that was causing a lot of you know, technical debt and operational risk. The third capability is uh with the with the with the platform and the data foundation that's using the latest and greatest features of
[02:51] ServiceNow. So, that you know, we have better operational awareness and then proactive decision-making. I also want to talk about the business impact. So, uh with with the re-implementation, our objective is to actually make the
[03:08] troubleshooting and validation easier. Uh we also want to actually create trustworthy data so that you know, there is a unified way in which you can look at the data and then be able to consume it. So, what has happened is data has got scattered in multiple places and there is
[03:24] non-alignment with the medallion architecture. So, this was an opportunity for us to actually pivot back into the reference data architecture. Uh the next uh uh shift is of governing the data. So, when the solution was in Oracle, we really did not have a way in which we
[03:40] could manage the access, the ownership, or the standards across the environment were all different because people were taking the data, they were creating their own data store and data marts, etc. So, we really were not able to govern the data. And then the question of auditability
[03:56] comes where we want to actually go back in history and check all of the change details. Those capabilities were also, you know, not available. So, that has actually forced us to actually go into the re-implementation objective trying to use the out of the box feature.
[04:16] So, some of the current pain points as I was mentioning was the data silos, which we all are familiar with, right? All the fragmented data that's there. So, there's data duplication, etc. When it came to the high volume refreshes, what was happening was Oracle had got connection drops and that was actually causing the data not to be
[04:32] refreshed, right? So, we had to definitely modernize our architecture because these incomplete data sets were causing a lot of disruption in the reporting space. And then the quality controls were also not there inside the Oracle solution because we really were not
[04:47] running any of the quality control tests or processes so that we can understand, you know, whether the data is complete and, you know, whether it is standardized and is there any sort of a schema drift inside the data, right? So, we know that, you know, ServiceNow can
[05:03] add more number of tables or existing structurally the table can be expanded with the columns, etc. And so, you need to have a way in which you can actually try and understand that and that was not available for us in the world of Oracle.
[05:24] So, what are the core benefits and outcomes of using the out of the box feature? So, the out of the box feature provides a lot of uh lot of features, but the most important one that comes to the mind is the configuration management database.
[05:40] So, with the ServiceNow latest implementation, you would have noticed that they have a common services data model, right? That basically bolts on the reference architecture. So, all of the configuration items can be inferred. Now, you can run an item auto discovery in which there will be utilities that
[05:57] will go and find out in your infrastructure what are all the configurable items and then bring it back. Now, when this whole process happens, what happens is you also bring it back into the configuration management database. So, typically what would happen is the discovery instructions would be there inside the
[06:12] ECCQ. The ECCQ will in turn communicate with the mid server and mid server will do the probes and patterns and then bring back all of the data. So, what happens is, you know, as we are all in cloud, there is infrastructure that gets created. But then you really want to have some sort of an automated method by
[06:29] which you can get all of this data. Be it in cloud, be it on your on prem. So, that is the, you know, feature that has really helped us. And then the identification reconciliation engine also ensures that the data quality is really good. So, we know exactly what is
[06:45] the metadata of that configuration item and so that way we can monitor it and then be able to provide reports. So, overall, I think the shift has been we needed a strong data foundation
[07:01] and also the capability to use the platform feature so that, you know, we can do the reporting really well. Now, there are reporting features available in ServiceNow such as platform analytics in which you could do the reporting. But there is going to be situations where you are going to actually pull the data out because you
[07:16] could have some enterprise systems such as Jira or any other systems through in which you need to mash up the data and ensure that you have that correlation with that cross-functional system and then produce the reports. So, those are the places where you want to minimally extract your data, bring it on to your
[07:33] lake house and then be able to provide metrics to the consum- for the consumption. So, what has been our current state? So, our current state, as I was telling you earlier, was having uh
[07:49] the ServiceNow 1.0, and uh there were Kubernetes which was uh running all of the flows, making those API calls, staging the data inside Oracle. And from the staging, what was happening was people were really not consuming from the data mart, so they had made their own copies, and you know,
[08:06] each one had their own method of summarizing their metrics. So, when that happens, what what what what it impacts is that you do not get a consistent view of all of the operational details as it has to be. So, we we saw a lot of inconsistencies in that, and then
[08:23] obviously from a consumption standpoint of view also, there are several ways in which the consumers were consuming the data. So, we had to take care of that when we did the re-implementation, making sure there is no data disruption for the consumption that's happening, but at the same time ensure that
[08:38] whatever consumption is happening stays as is. So, the customer should not feel any impact of the change that we are trying to make. So, what did we do inside the modernized architecture? We simplified it because Databricks gave us the capability of
[08:54] Lake Flow Connect and ServiceNow connector, right? So, when you use the ServiceNow connector along with the Lake Flow, you can orchestrate that whole process. So, in simple terms, you should be able to make that connection to the ServiceNow schema, pop out all the tables, look at the tables that you
[09:10] need, and then bring them over. When you do that, you know, you you try to do it using the medallion architecture. So, what you see is there is a formatted and standardization that happens inside the harmonized layer, and then we build the curated views based on the reporting
[09:26] needs. So, there is a centralized team that is basically taking care of ensuring like they use the data flow connector, make copy of the data once in raw, once in harmonized, and then you know, for curation they can have their own version of the curated data. And
[09:43] overall, this is having the Unity Catalog, you know, for the governance part of it. So, we know exactly, you know, how the access is controlled, right? So, for instance, let's say there is a you know, there's a different team that wants to consume the data, right? So, you provision through
[09:58] the you provision the access through the Delta Sharing. So, that way you know, based on your enterprise ownership model where and all the data is going. So, you can audit all of that data that's going out to the end consumer.
[10:17] I want to zoom in on the Lake Flow Connect. I think some of you might have explored this. With the Lake Flow Connect, what happens is you are able to store the source credentials, which is using the OR 2.0, that is in turn making connection to the Service Now. And the ingestion pipelines
[10:34] are able to maintain the checkpoint. So, you know exactly where you left and where it has to pick up the data from. So, that those sort of capabilities are all already orchestrated. So, you spend very less amount of time developing all of these things. In the previous world, what you would
[10:50] see is all of this data moment you had to orchestrate. Let's say the API fails and then you have to recover from that. You know, there's a lot of work you needed to do. But with this component that Databricks has given us, it has definitely ensured faster development of
[11:05] time and a consistent way in which the data can be delivered to the downstream consumers. Moving on to the next uh slide on the Lake Flow connection benefits. As I was mentioning, you know, we have the built-in CDC. We can
[11:22] understand the schema evolution. And then most importantly, the delete capture. This was also something that was lacking in the previous system. So, there were users or the admins inside the service now who used to delete the data, and then there was no traceability to understand
[11:38] because as an analytical system, you don't have any other way to track it. Now, what happens is with the Lake Flow Connect, we have that capability. We really know that when you probe the sys_audit_delete table, and you see that there is not a connection available with the incident and the sys_audit_delete table, you know
[11:54] that something has happened, you know, behind the scenes where people have not gone and deleted the data the right way it has to be deleted so that, you know, you you're not seeing that deletion cascade down to all of that dependent tables. So,
[12:09] while we were doing this, we were able to point out this to the ServiceNow team, and then ask them some questions in terms of why the data is in an inconsistent state. So, the next benefit is has been like the automated pipelines are reducing the
[12:24] operational risks. So, the way we thought through this was we thought we looked at our executive reporting which was more critical. So, that got the first order of business followed by our enterprise reporting, and then followed by our compliance reporting. So, these
[12:40] pipelines are something that, you know, you can have other teams also run. And the use cases that we had was we needed a refresh every Friday at the afternoon, right? Because there was some executive reporting that had to be refreshed at that time. So, the team could actually share those pipelines,
[12:56] and those pipelines could be run as and when needed so that you really have the latest data coming in so that the reporting metrics are accurate, and so that way the leadership can trust the data. As far as the governance is concerned, Unity Catalog definitely gives you all of those capabilities in terms of the
[13:12] access, lineage, and stewardship. So, you know, overall, we are able to provide the fresh data through this governed ingestion. As far as the analytical consumption is concerned, the end user will will see any difference because they have been consuming using the Power BI or the
[13:29] sequel and now they have the capability to write their own notebooks and consume the data. So, overall I think it has been a flexible consumption. It's just that we had it to we had to build out the harmonized view so that there's one version of that data available for all the downstream consumers. And so that
[13:45] way it is all consistent and now, you know, we don't have any sort of errors inside the reporting. I'll move with my next slide which is talking about the the data strategy road map.
[14:03] When we started on this journey, we just wanted to make sure we get a consolidation of all of the stakeholders. So, that took a while because article data was existing, a lot of people were consuming. So, we really wanted to understand what is the scope. And so, we consolidated all of those things.
[14:19] It took the alignment from the stakeholder to use the lake flow connect and bring the data inside the lake lake house and looked at you know, how we can lock the requirements. So, as I was explaining, the executive reporting is of you know, higher importance. So, it had a set of tables
[14:34] followed by the enterprise reporting. Overall, both of these two these two together comprised of about 150 tables that we targeted and once that was done, we were able to bring the data into the raw layer and harmonize it. As far as you know, the other modules
[14:50] are concerned like the service level optimization which monitors like the third party for the risk etc. Those those are modules that are coming later and we have to tag along with the releases from the service now team. So, we've aligned our road map with theirs
[15:06] release date and so that way we have alignment. And so, during the third quarter is where we'll start realizing the benefits of this modern data lake house that we are building because that's the time major incidents are going to come and that's the time we are cutting over the migration. We will be left with archiving of the
[15:23] data. Uh, so there is historical data that's there inside Oracle. We definitely want to bring that over. There is also going to be some legal hold data that we need. So that is also going to be brought and uh, some data that we need from ServiceNow 1.0 also something that we
[15:38] are going to bring. So the lake house will consist of all the ServiceNow 2.0 that will come through the latest out-of-the-box feature. And whatever was uh, done inside 1.0 is also going to be migrated from Oracle and it is also going to be available. So in terms of any pattern reporting going back in time
[15:55] and then trying to analyze, we will still have the consumer able to tap into the data using the lake flow. Obviously the last phases are also important where we want to do the legacy clean up and decommissioning uh, and then we'll go into a mode of hypercare monitoring and a steady state. So uh,
[16:13] basically our quarter three goal will actually uh, do more do most of what we need for the IT service management, but Q4 will focus on the remaining modules uh, whatever is left over and whatever the ServiceNow team is going to do. So this knowledge management, CMDB, etc.
[16:30] have already got migrated. So we are consuming the data as I'm talking today. Uh, major incidents are going to come to us uh, in the end of June and so we've already had the pipelines and uh, you know, we are ready to cut over.
[16:49] So this last slide is actually going to talk about the path to success. So um, I know that whenever we have the modernization effort, there's always question of funding. You know, how do we plan on funding? You know, how should we align our product owner and capacity? So this has been a learning curve for us um,
[17:06] where we've tried to actually lock in the scope and that has really helped us understand like what is the funding effort that we require. So there has been uh, you know, some of the aspects of trying to understand how much of the engineering effort will be needed, how long it will be needed. And I think
[17:23] overall, it's just not about defining the strategy, right? You have to execute on that strategy and more importantly, you have to operationalize that. Operationalizing that means that, you know, you are not consuming a lot of time from your run team to be able to support all of these things.
[17:38] And so, our success metrics has been defined by a successful execution of executive reporting. We've completed that. We are on the journey for the enterprise and department specific reporting and compliance reporting. I think I'm close to the time, so I want to actually show you how the pipeline
[17:55] looks so that you have a visual of what you can see inside the Lake Flow Connect. So, here you're seeing the example of two tables, CMDB CI and then business unit staging. So, it's a very simple user interface where you can see how much records are updated or updated and
[18:12] inserted and then how much are deleted. So, this is like a single pane of glass where you can monitor all of these details. There is another view which is the performance view and that in turn will tell you how much of data that you have read, you know, and all of the run time details. So, you can monitor all of
[18:29] these things, look for optimizations and then be able to make all of these changes. Overall, I think, you know, for us in this journey, we've been able to simplify the architecture and also get a lot of benefits by using the Lake Flow Connect along with the service now
[18:45] connector. So, with that, I want to conclude the presentation.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.