End-to-End DevOps for Databricks: Asset Bundles, GitHub, CI/CD

Summary

Nationwide Insurance, a Fortune 100 company managing thousands of Databricks pipelines, replaced manual custom deployment scripts with Databricks Asset Bundles, GitHub Actions, and Harness, achieving four times faster deployments and 80% risk reduction in production.
A Git-driven, declarative deployment model uses Asset Bundles to package notebooks, workflows, and Delta Live Tables pipelines with environment-aware YAML configuration, enabling consistent promotion across dev, test, and production environments at a scale of 6,000 pipelines and 100,000 tables.
Unity Catalog integration into the same CI/CD pipeline provides full audit trails and role-based access, ensuring consistent governance across every deployment and operationalizing analytics as secure, production-ready data products.

End-to-End DevOps for Databricks: Asset Bundles, GitHub, CI/CD

P&C insurance data platforms must deliver analytics rapidly while maintaining stringent governance and regulatory requirements. Organizations often struggle with inconsistent deployments, manual promotions, and fragmented DevOps practices across environments. This lightning talk demonstrates how Nationwide Insurance, a Fortune 100 company managing thousands of Databricks pipelines, implemented true end-to-end DevOps using Asset Bundles, GitHub Actions, and integrated governance.
Learn how a Git-driven, declarative deployment model uses Asset Bundles to package notebooks, workflows, and DLT pipelines with environment-aware configuration, enabling consistent promotion across dev, test, and production. See how GitHub Actions orchestrates CI/CD pipelines with static code analysis, unit testing, and automated deployment validation. Discover how Databricks Apps and Unity Catalog integrate into the same pipeline to operationalize analytics as secure, production-ready data products with full audit trails and role-based access.
🤝

Chapters

FAQs

How did Nationwide Insurance implement CI/CD for Databricks pipelines?

Nationwide implemented end-to-end DevOps by replacing manual deployment scripts with Databricks Asset Bundles for packaging and GitHub Actions for the CI pipeline, with Harness handling continuous deployment. This created a single pipeline that promotes notebooks, workflows, and Delta Live Tables pipelines with environment-aware configuration consistently across dev, test, and production.

What is a Databricks Asset Bundle and how does it work?

A Databricks Asset Bundle is a declarative, Git-managed package that defines notebooks, workflows, and Delta Live Tables pipelines with environment-specific YAML configuration files. It enables teams to promote code consistently across environments and integrates directly with standard version control workflows on platforms like GitHub, as demonstrated by Nationwide in this video.

What results did Nationwide achieve after adopting Asset Bundles and GitHub Actions?

Nationwide achieved four times faster deployment speeds and an 80% reduction in production risk after moving to the Asset Bundle and GitHub Actions approach. The team also established consistent governance and full audit coverage at a scale of 6,000 pipelines and 100,000 tables across the organization.

What challenges did Nationwide face before implementing DevOps for Databricks?

Before the migration, Nationwide relied on custom manual scripts that each engineer wrote individually, which did not scale to hundreds or thousands of pipelines and led to config inconsistencies between dev and production environments. The lack of a standardized rollback and recovery process also made it difficult to resolve production incidents quickly, including failures that occurred on Friday evenings.

Full transcript

[00:08] So before we start, we'll do a little bit of stretching exercise because this is a session after lunch. So raise your hand if you have built a pipeline that was running perfectly in dev, but when you went to production it just blasted. Started failing.
[00:27] So and then raise both of your hands if it was on a Friday evening. So that was us, but hopefully towards the end of this session, you will get something to avoid those scenarios. Uh So we'll get started.
[00:44] Uh little bit about Nationwide. So Nationwide is a Fortune 100 company. We are like 73.2 billion in total sales. Uh and and investment port- investment portfolio is pretty large, 172 billion. We are A+ rating towards all the companies within Nationwide. So pretty
[00:59] large company company within PNC. And when you have a company this large scale, it's difficult to scale things and then build hundreds of pipelines and don't have a process to follow. Uh so we'll go over some of it uh next of the slides. Quickly a little bit
[01:15] about myself. Uh I am Harshit Mishra and the data engineering lead within Nationwide PNC dots organization. It's a new data organization. Uh 15-plus years experience uh across multiple data projects. Seven-plus years experience working in Databricks towards
[01:30] infrastructure best practices but working on all these different data pipelines. Uh currently working as a data engineering lead. Passionate towards building data modernization and efficiency towards data pipelines.
[01:46] So before we go to the solution, let's go over all the challenges that we were facing in our current uh scenario when we are building the pipelines. Uh when you work on Databricks pipelines, right? When there are fewer pipelines, it's easy to manage, but when you are trying to deploy things at a large scale, that's where you see critical
[02:03] gaps. So, the first thing was manual deployments. Because we didn't have a process or an automated thing to deploy pipelines, we had custom scripts written, there were manual deployments. So, when you're doing it like for 10 pipelines, it's manageable. When when you scale it towards hundreds and
[02:19] thousands of pipelines, that's where you start facing these challenges. So, that was the biggest thing I would say, the manual deployments, because each person was writing their own things to deploy things from dev to test to prod. The second thing was config inconsistency. So, this is where that
[02:35] example I was telling you, when the things were running in dev, but they were in failing in production. Maybe because somebody forgot to add a permission, or somebody forgot to add a library that was needed in production. It was working fine, but because you need different accesses or different
[02:50] versions of things when you go to production, a small change can cause catastrophe in production. Governance and compliance. So, that was another area because things were manual, um people were doing things using their own IDs, it was very difficult to trace
[03:06] back what changes went into production. Uh things were just going on with their particular IDs and things like that. So, governance was an issue. And then roll back and recovery. So, when you went into production and something failed into production, that's
[03:22] where I think it was very difficult to roll back. And I remember once we spent around 8 hours to roll back things which you just deployed. So, it was a lot of things. When something fails in production, it you have to spend hours and hours to roll back. These were some of the challenges that
[03:37] we were facing initially before we implemented DevOps. So, let's dive in into the architecture that we used for this. So, if you see on the left side, that's where there's a Streamlit app that was built. And this is an optional thing you
[03:53] don't have to do it for every use case, but this one primarily built so that it's easy to deploy things because in our area the data engineers were new, so they didn't have that much capability to create asset bundles. So, we are primarily using asset bundles for deployment. So, this
[04:09] provided a streamlined way, a standard boilerplate code based on certain templates or parameters to create a asset bundle package. And then we had VS code that that was primarily used by developers for development. You can also use Databricks workspaces, but the primary
[04:26] template that was used to deploy was using asset bundles template, and that was created through the Streamlit app. So, once you are done with the development, you can test your changes using from your local user folder using bundle.deploy command using VS code or
[04:42] any CLI. That will just deploy your changes into your user folder. But once you are done with development, you are done with the unit testing, then you can create a PR, a pull request to merge the changes from your branch to the main branch. So, we followed the GitHub flow, and we'll go through in detail of what's
[04:59] the CI portion. But we were following the GitHub flow. So, you had one branch which you can use to do development, and then you can merge that branch into the main branch. And that triggered your the pipeline in the back end. So, we use Harness within Nationwide as a
[05:14] deployment tool. And then Vault was a place where we use store secret, and then Liquibase was mostly used for deployment of DDS. And the main thing here to notice is your the upper box. So, we had one pipeline, one Harness pipeline that was
[05:30] used to deploy end to end from dev to test to production. So, everything is in sync. There's no surprises you are seeing directly in production. So, if you create separate pipelines for each deployment, you might again face the same scenario that this particular environment is out of sync because it
[05:45] just went directly in production. So, that's how we created just one pipeline and that goes through these approval stages before going to any new environment from dev to test prod. So, diving deeper into the CI portion. So,
[06:01] for our CI we use GitHub actions and on the left side left hand side you can see the repo structure. So, the first folder is the GitHub actions folder where we had a standard template for the CI and then you have uh the primary folders are the resources and
[06:17] the source folder. So, the resources folder will have the YAML files which are related to your jobs. And how we deployed is we created one YAML file for each job. So, you can combine multiple YAML files also in the into a single but to be um more clear we separated each job into a
[06:34] separate YAML file and each of them they had their own parameters that can be used. And then the source folder is where you can have your DDLs, you can have your Python scripts, your SQL scripts that can be deployed along with your job code. So, it was doing both jobs as well as
[06:51] your source kit together into using the Databricks asset bundles. And then test is primarily for your unit testing. And then the databricks.yaml is the uh primary file that will contain uh the parameters which will be used across the
[07:06] jobs. So, if you are deploying 100 jobs, then the databricks.yaml file will have the parameters for all those 100 jobs. Maybe it's a uh secret you want to use, a service principle which is common for all the pipelines. You can specify that in the databricks.yaml.
[07:22] So, that was the primary structure of the repo that we were using. And on the right side you can see uh like I mentioned we were using the GitHub flow. The good thing on the GitHub flow is you don't have to maintain so many branches like for each environment dev, test, and prod branches. You don't need those stale things. So, as long as soon as
[07:39] you're done with your your you can merge with the main branch, and that's what will trigger your uh uh SonarQube checks, which is more of static code analysis, that will trigger your unit testing. So, it's more cleaner than maintaining so many different branches for each environment. And we
[07:54] use standard triggers. So, with with every pull request, you will have a trigger which will actually run all these uh you know, unit testing, the scanning, and everything.
[08:10] So, uh the next is the CD part. So, once you're done with your development, you did a PR request, it's approved, then uh the pipeline will get triggered, which is a tool which is specific to Nationwide. You can use any tool for deployment. You can even use GitHub Actions for deployment. Uh but in uh within Nationwide, we use the standard Harness tool. So, if you see, like I
[08:27] mentioned, we had one pipeline which goes from end to end, from dev to test to prod. And the important thing to note there is within a stage in Harness, uh there were two steps. The first step was a planning step. So, it gets changes from your Git repo, and then it will run
[08:42] the bundle.validate command. So, that will tell you all the changes that will be deployed. Because if you directly deploy, you'll not know, okay, what's actually getting deployed. You have You have tested, but this is primarily for the tech leads and all to approve and see, okay, these are the changes that
[08:58] will get deployed once you go from one environment to another. So, there's a approval step in the middle, if you see, between those two stages. So, once the tech lead approves that, then only it goes to the next step, and it will run a Databricks bundle. deploy, which will deploy your changes to the higher environment.
[09:14] And that is the same consistent process across each of the environment as you go. We had a optional pre-prod, so you can add as many environment you want. Uh we use generally the three environments, but uh in certain cases, we had the fourth environment, which was called as pre-prod.
[09:34] So, this was all CI and CD uh and the architecture that we deployed. Now, when you are doing this in a large scale, right? Like I mentioned in the beginning, Nationwide is a big company. We are combining data from different areas, different data marts for PNC insurance. So, this is an actual case that we had
[09:50] within PNC. So, we had around 6,000 pipelines that were running in production. We had 100K tables which were there within different schemas, and then we had 1,000 plus schemas that had to be deployed. So, it's it's not a small
[10:05] thing and at scale when you're doing this scale of implementation, it's not easy to do. You cannot do a big bang implementation for all this together. So, you have to follow a step-by-step process, phased approach. That's what we'll talk next. So, this is our implementation roadmap. Like
[10:21] I said, it's a phased approach so that we can most do it sustainably than just doing a ad hoc thing. And whenever you're trying to do things in a large scale, you have to follow that step-by-step process. So, firstly, we did a current state assessment which included benchmarking of our
[10:38] current pipelines, which pipelines have CI/CD, which didn't have, how to prioritize if there's a critical pipeline which might need regulatory needs. And also, we prioritized and we looked into all the pipelines using our current state assessment. The next step is standards and best
[10:54] practices. This is very important because when you start something new and you don't have standards and best practices around that, every developer will try to do their own thing. So, that was the next important step that we did. And that's where we created the Databricks app that can be used to create a boilerplate YAML template that
[11:11] can be used by each developers. Once we have those standards and best practices, the next phase is your internal training which will be training our resources to know, okay, this is the templates that we've created. This is how you use Databricks start time. This is the GitHub flow that you have to use.
[11:27] So, all that was done in the part of training. And once the training was done, then we adopted a standardized approach. So, anything which was developed new, that was following this new template. But anything which is existing, if we are touching those pipelines, we were implementing the changes for DevOps. So,
[11:44] like I said, the same thing, it's not everything together, but anything new, if you start doing that, that will also help you to even learn as well as adopt. Uh after the adoption, the next thing is improvement because this is a new process and then you need to get the
[11:59] feedback. So, we got the feedback from the users who has who started using the template or started implementing. One of the feedback which we received was the template is like large, it has lot of parameters, so we had to reduce that and made it more consistent. And then
[12:14] finally, once you are doing that adoption, the final thing was measuring and adjusting. So, we did like uh um like a copybook or something that measures, okay, these are the pipelines which have completed the DevOps, how many pipelines are remaining. And then this is a continuous process, it goes as
[12:32] you move to different phases. So, it's it's not a one-time thing that you do and it's completed, it's like continuously evolving. And then finally, the outcomes and benefits. So, once you are done with all your pipelines and developer, these are some
[12:48] of the benefits that we started receiving. Um first thing is faster deployment, so we received four times faster deployment. The number is bigger, but this kind of an average benefit that we started getting. We used to spend hours, sometimes even days for deployment when
[13:03] you have like hundreds and thousands of jobs which are going in a single release, uh which turned out to be in within hours, so we can do now deployments within hour. And for smaller pipelines, it was within minutes. So, that was a huge improvement on the deployment. Uh risk reduction, so we have 80% reduction
[13:20] uh risk reduction. Most of the issues we still found were like mostly defects, data defects, but primarily uh things like things are not running in production. It was working fine in dev. So, all that was reduced because you have this moving the same thing across pipelines. It's everything is in sync. So, the
[13:36] operational risk reduced by 80%. And then the consistent dev test and prod environment. So, it's very easy to get the insights. So, because we are following a modular approach, you can go back and do reporting on how many pipelines are
[13:52] moving into dev test and prod and following the structure. And the audit coverage, so the governance part that we talked about. Because we are using service principle and it was getting deployed through the service principles. It was much easier to go back and trace
[14:08] what happened, who deployed the code. You can go back and then do a rollback based on that. So, it was very easy for the governance because we are following the same process. So, that was all for today's session. It was a quick thing to go over the
[14:24] architecture. Maybe I'll wait for 5 minutes for any questions. If anybody has any questions, you can raise your hand and can talk about it.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.