Automating Data Pipelines with Software Development Best Practices

Summary

Intermountain Health transformed its data operations by replacing GUI-based ETL tools with Git version control, trunk-based development, CI/CD with GitHub Actions, Databricks Asset Bundles, and Spark Declarative Pipelines.
Parameterization ensures development and production environment parity, while service principals are used to test permissions before any deployment reaches production, eliminating the failed deployments and emergency hotfixes that plagued the old approach.
Cultural adoption among 50-plus data architects was achieved through a Future Experts Group of community advocates who built trust and shared ownership alongside the technical transformation.

Automating Data Pipelines with Software Development Best Practices

Most data teams rely on GUI-based ETL tools that make change tracking impossible, enable anyone to deploy directly to production without review, and lock engineers into specific vendor tools. This fragility leads to failed deployments, missed SLAs, and late-night hotfixes when changes break production at 2 a.m. on a Saturday.
See how Intermountain Health transformed their data operations by applying software engineering best practices: Git version control, trunk-based development, CI/CD with GitHub Actions, Databricks Asset Bundles, and Spark Declarative Pipelines. Learn how parameterization enables dev and prod parity, how service principals ensure permission testing before production, and how cultural adoption through community advocates builds trust and ownership among 50+ architects.
🤝

Chapters

FAQs

Why are GUI-based ETL tools problematic for production data pipelines?

GUI-based ETL tools make change tracking impossible, allow anyone to deploy directly to production without code review, and lock engineering teams into specific vendor tooling. This video describes how these characteristics lead to fragile pipelines, missed SLAs, and emergency hotfixes when undocumented changes break production environments.

What is trunk-based development and why did Intermountain Health choose it over GitFlow?

Trunk-based development is a strategy where engineers commit to a single main branch with short-lived feature branches, rather than maintaining long-lived parallel branches as in GitFlow. This video explains that Intermountain Health chose trunk-based development because it reduces integration complexity and keeps the codebase continuously deployable, which is essential when coordinating 50-plus architects.

How do Databricks Asset Bundles support CI/CD for data pipelines?

Databricks Asset Bundles provide a configuration-as-code framework that parameterizes pipeline definitions so the same code runs correctly in both development and production environments. This video shows how Intermountain Health combined Asset Bundles with GitHub Actions to automate testing and deployment, using service principals to validate permissions before code reaches production.

How did Intermountain Health manage cultural change alongside the technical transformation?

Intermountain Health established a Future Experts Group of community advocates—data architects who embraced the new practices and helped colleagues adopt them—rather than mandating change top-down. This video emphasizes that earning trust and building shared ownership among 50-plus architects was the harder challenge compared to the technical implementation.

Full transcript

[00:08] Okay, I am so glad that you guys decided to come today. Uh, this presentation is on automating Spark data pipelines, um, applying software development principles to a data-driven world. I want to start off today's presentation with a story. I live in Michigan
[00:24] and in Michigan woodpeckers can be a big problem. Here is Mr. Woodpecker hanging out with his in his favorite hole in my home. To fix the problem, we needed to patch the hole. Unfortunately, this hole was pretty high
[00:40] up, uh, but we could reach it through the attic. So, I got on my gloves, uh, got out my dusty ladder and I climbed up into the attic to try and fill this hole. How many of you here have ever been in an attic before? Okay, maybe 1/3 to 1/2, okay.
[00:58] When you're at it, you have to make sure that you are stepping in the right spot. Like you have to step on the support beams cuz everybody everywhere else it's just, you know, this drywall and you're just going to fall through. It's not going to hold your weight. Um, so last year I was in the attic and I was doing some blowing insulation
[01:13] and I was navigating no problem, right? I was extremely careful and I made sure to step on these on the support beams. Um, this time, however, my experience was a little bit different. The ins- the insulation was obfuscating where the
[01:29] support beams were and the insulation was a little bit slippery. If you put all this together, you end up with a big hole in the ceiling, a closet covered in insulation, and various wood slivers throughout your body.
[01:44] Now, first of all, I'm totally fine. I'm okay. Don't worry about me. Um, and on one hand I feel pretty fortunate because I didn't fall very far. Uh, the closet had these shelves to catch me. Uh, but on the other hand, it was a really big mess. Um, so not only was there a huge gaping hole in the ceiling to fix, but the insulation came
[02:01] through the closet. Um And this is the closet that my wife so neatly organized just a week before. So, it was all messy. Um I had wood splinters and I was pretty embarrassed. Point of this unfortunate adventure
[02:17] is to point out that occasionally, when we try to solve a problem, we may intentionally or perhaps unintentionally create a bigger mess. This presentation I want to help you avoid these problems entirely, especially when it comes to deploying
[02:33] code and pipelines. You may think that your solution will work, uh but you might end up falling through the ceiling like I did. Okay, real quick I want to share a little bit about myself. My name is Ben Whatley. Hi, glad you're here.
[02:50] I've been with Intermountain Health for about 5 years. And I've led efforts to bring software engineering best practices um into Databricks. And if you saw me yesterday, uh software engineering best practices to our data analytics platform. If you saw me yesterday, I was wearing my BYU
[03:06] T-shirt. Go Cougs. And after this presentation, I'll probably putting on my brand new Mario T-shirt. I got it at the Nintendo store earlier. So, that's a big thing for me. Um I really like playing tennis and I like watching sports except for golf. And I love working for Intermountain
[03:21] Health. Uh the people I work with are really amazing. I get to live close to family and I because I work remote I work remotely in Michigan, whereas Intermountain Health is based mostly in Utah. I have two young kids and they love to play outside.
[03:37] And on top of all of this, I get to work in the Databricks ecosystem and it's a lot of fun. Okay. A little bit about Intermountain Health. Um I'll just do a quick little intro here. Intermountain Health is one of the largest health systems um out in the
[03:52] kind of the Intermountain, Utah region. Um, under the Intermountain umbrella, there's also SelectHealth, uh, the insurance arm. And as a nonprofit, we strive to help people live the healthiest lives possible. Slide. Yay.
[04:08] Um, here's another little visual. Uh, we have lots of hospitals and clinics across our three regions, canyons, desert, and peaks. Um, and on the right-hand side over there, um, you can see all the different services we provide. I'm most proud of the way that we try and give back to our community
[04:24] the best that we can. I believe Intermountain Health is doing a lot of things really, really well and serving the community that the best that it can. Now, as we begin, I want to ask you this question. Why are you here?
[04:39] I'm not going to I'm just going to throw out some ideas. You can raise your hand if it's applicable. Um, the first one, you live and breathe health care and data. Anybody in health care data world? Okay. Uh, number two, you experience resistance from your team when you're
[04:55] trying to do something new with a new data tool feature. Okay. You struggle to deploy declarative pipelines, declarative automation bundles, or Git. That's hard. Okay, a few hands.
[05:10] You want to grab a hammer and smash a window because the process that worked in dev is now failing in production. Anybody? Okay. I know, I know, I've been there. Um, maybe you know me personally and you came here to support me. Hey.
[05:26] Glad you're here. Um, next, you might want to sell me something. I don't know, maybe you're a vendor. If this is you, just know I don't have any purchasing power. Uh, the exception to that is chocolate. I will buy your chocolate. So, if you have that, please come up afterwards.
[05:42] Uh, the last the last option is maybe you You a good power nap. It's been a long day. If this is you, there's no judgement, but I've sat in those chairs, they're not very comfortable. So, hopefully you get a good nap otherwise. Oh boy, I made a mistake. You know, one of those. Okay.
[05:58] Just a second. So, in this session, I want to open the door of knowledge to you. Specifically going to when it comes to appliance software development engineering best practices into a data platform like Databricks. If you're trying to fix problems, the
[06:14] holes, I hope to equip you with tools and processes to not fall through the ceiling as you patch those holes. If we remember that progress precedes perfection, then we'll be more successful in our problem solving. To begin, I will first explain some of
[06:31] the issues that Intermountain has faced both technically and culturally. Then, I'll explain how we executed solving both the technical and cultural challenges that we face. Lastly, I'll share some of the results and the impact that we've had from using documentation.
[06:47] Okay. The old way, um I want to talk about the from a technical perspective how we've talked um overcome some problems. First and foremost, our ETL tools were gooey. So, you know, the click and drag kind. Now, these tools aren't inherently bad,
[07:02] but this approach led us to having a repeated problem, change tracking. It was very difficult to know which tables were updated, who changed the ETL, or what specific change broke the whole thing at 2:00 a.m. on a random Saturday. So, this made our platform uh a lot more fragile, and it led to a lot more
[07:19] weekend and late night hours. Another issue uh with how we use our tools was every architect or engineer had access to deploy to production at any time. And there was little process as far as I know for peer review.
[07:35] So, in their defense, many of you know that you haven't really truly lived until you've truncated a a row table, right? That That's really living, right? So, even though Access was broad, we were limited to specific tools and vendors. This not only limited our ability to
[07:51] innovate, but also limited the engineers' ability to grow and to learn. You wrote a SQL script, dragged it into the tool, and you just kind of hoped and prayed that it would work. Um these technical shortcomings made it difficult to keep a stable production and stable production
[08:06] environment. And it also prevented us from rapidly adapting to new demands from the organization. Fortunately, we had challenges as well. Over the last 5 years, we've implemented a new EMR.
[08:21] We've migrated to a new data platform with Databricks. We merged with another large health system, and we did it all through COVID-19 and the introduction of AI.
[08:37] To put it simply, we were dealing with change fatigue. Not only were the tools changing, but the teams were changing. We had to merge tools, teams, processes, thought patterns, merge everything. Um and knowing what we know now, we probably would never want to do another merger again.
[08:52] Um but there's a lot that was out of our control from a cultural perspective, right? But we had an opportunity. We had an opportunity to avoid falling in the attic. Or maybe we could avoid attics entirely. Uh we have problems that we can control and others that we can't. So, how do we
[09:08] ensure success in solving what we can control? So, my approach is by applying software development best practices to a data platform, we can solve the most painful problems and avoid falling through the ceiling. So, to do that, we had to implement some technical updates. We had to upskill
[09:25] engineering. Uh we had to do it very carefully. Um and in our case, progress in the right direction was much better than immediate perfection, right? So, the new way. First, we prioritize moving our
[09:41] processes. So, you create tables, ETLs, everything we could, we prioritize moving it into Git and the GitHub. Um we wanted to deploy a a cur- a code-first architecture. I wanted to say curd-first because, you know, I live in Michigan, but whatever. Um it allows us
[09:57] to track changes, uh deploy code, and roll back any mistakes quickly. When we set up our projects in Git, we use trunk-based Git development over GitFlow. Now, this is one of the very first avoid falling in the attic moments for us. If you aren't aware, GitFlow
[10:13] developments have a have a dev branch, a test branch, and a prod branch. And from my experience in the software world, there is a lot of overhead that comes when you when you implement a GitFlow methodology. You end up with something called merge hell. With declarative automation bundles, uh
[10:30] you also can see the dev and the prod targets, right? And so, it's very tempting to use the GitFlow because it seems to match perfectly. Don't fall in that attic. It's not worth it. We also wanted to implement CI/CD.
[10:45] Um I love In-N-Out, and especially their sauce, right? In our case, the special sauce for deploying CI/CD was the heavy use of parameterization. Um we use parameters in both the bundle code and within GitHub Actions. This gave us the framework for building,
[11:03] promoting, and deploying pipelines automatically. Finally, automated testing and deployments gave us confidence that the requested changes, whatever they were, were going to be successful. I'll add that Spark declarative pipelines was probably the easiest, most
[11:18] efficient way for us to deploy these code changes, but you could do it without that. Like, you could have create table statements or whatever you want. Um but the easiest way for us was with the pipeline approach. And to summarize the technical changes, um these processes help ensure our
[11:34] platform to be significantly more stable. Okay. Um here are some little screenshots. They're heavily redacted. I just wanted to be extra extra careful, so there's nothing sensitive in here. Um so there's code for like a sample code for a GitHub action, a Databricks YAML
[11:51] file, and a pipeline YAML file. So, the point here is I want to highlight how often we are using parameterization within these files. Um this is not a code-heavy session, but if you are interested in some of this code and what it looks like, come talk to me afterwards. I'm happy to show you the
[12:07] ropes. Okay. I want to show you how our team interacts with these flows even more. The first step is we make changes in our repository on a branch. This could be a feature, could be a fix, doesn't really matter what it is. Um and we have a set a GitHub action
[12:24] that is you can click your branch within, you know, the action, and it will deploy your bundle with your branch to your development Databricks workspace. And from there you can you can test your pipeline. One of the keys to this solution is all
[12:39] deployments dev and prod are through service principles. Why? Cuz one of the key issues in moving from dev to prod, especially in the Databricks world of data, is permissions. Your your individual user likely has different permissions than what a
[12:54] service principal does, right? So, if we can test that out beforehand, we are ensure a greater likelihood of success in production. Um however, deploying to dev with a service principal has its own set of issues, too. The main one is that we have
[13:09] various users who need to deploy to dev and test something. Um so unless we put something in place, uh one user will deploy to dev and then another one will override their changes. So, this is why parameterization is really really important. Um especially for bundles, as long as
[13:26] you store it in a new folder, it actually stores it as a whole new declarative automation. I want to say asset bundle, that's the old words. Rebranding is kind of a big thing around here, I guess. So, we deploy it as a new bundle, and it's not going to overwrite the other ones. So, with that knowledge, we
[13:41] prepended the the user's GitHub username to the bundle deployment path, so it's like their own unique bundle that's attached to the service principal. Um we also prepended the GitHub username to everything. So, we prepended it to the jobs, to the pipelines, and even to
[13:58] the tables and materialized views themselves. So, everything can be separated, and that you don't have to worry about cross-contamination or overriding someone else's changes. And really what I'm trying to replicate here is a local environment, right? That's kind of what I want to get at is have some sort of separation where you
[14:14] can really test your stuff in a stable uh predictable way. Um so, the next stage we have review, approval, and merge. So, we have GitHub Copilot running on every pull request, and we have a human reviewer that
[14:30] uh can also that goes and makes sure that all of those comments are good and can do the final check, right? So, when this happens, you can merge and feel confident that everything is going to be successful in production. Um but the cool thing too is like after
[14:46] you deploy to dev and you merge it into main, automation will run and it'll clean up everything you did in dev, which is really nice. Uh the last piece of this is a weekly release schedule. Um I like this that provides a certain level of stability to the platform. It allows for thorough reviews and
[15:02] confidence in our code changes. Of course, we can implement hot fixes and they are necessary, but as you do this and you get used to it and your organization gets used to it, um the amount of hot fixes should decrease.
[15:21] The culture piece is arguably harder to change and influence than any technical implementation that you might ever do. Um with all this change, we had to make sure that we are giving architects and engineers the resources that they need to be successful. Um there's a ton that's out of our control, including organizational policies, politics, economics, all of
[15:37] that, right? So, what made our implementation and transition to these new processes successful? The measure of success is progress, not perfection. It would be very, very tempting for me to set up right at the beginning to set up
[15:52] everything exactly how I wanted it to, right? The CICD, the bundles, the pipelines, it's all really amazing. Why don't we just start with that? But we had to We had to start somewhere, right? We have to give our engineers time to learn. So, we started with just Git.
[16:08] Just put your code in GitHub so we can track it and see it. So, that was step one on the steps to um to be successful here. Okay. The second piece was a very deliberate training approach. So, training over 50 architects takes a
[16:25] lot of time, especially with other priorities and organizational changes. Uh so, we started a group. I called it like the the future experts group. Um so, they're kind of like reduce the stress on, you know, that you need to learn this. So, we trained a smaller subset of architects, you know, five,
[16:40] six, seven, up to 10 architects on how how to do this, right? Um so, in the small setting, it's a lot easier for them to ask questions and to get help and get the hang of things. And they would be the ones that their teams would go to uh when they had questions. So, instead of everybody
[16:56] coming to me for all of these questions about GitHub, they had this group of architects that they could ask. Um now, we still had big training sessions and documentation, all of those things are important. But I think the secret sauce was getting this small group of people trained, right? Um not only were they learning how to do it and
[17:13] being successful and helping each other, but they ended up becoming big proponents and big advocates for these new tools and processes. So it wasn't just me advocating, they became a bigger group advocating to help others learn these new tools.
[17:31] Um I have some quotes here from some from some architects. So she said, Raleigh said, "The code transparency has been a huge benefit for keeping things more standardized, knowing where to find logic, and having production examples." Oh, did I go too far? Okay. Uh these processes have saved us a huge
[17:47] amount of time. It gives us confidence and eliminates the need to manually set up development objects. So with pipelines, you don't have to worry about creating a table and deleting it. It's just created for you, which is really nice. Um This is just a very some very rough
[18:04] metrics around our implementation here. Um I believe that we can take the best of software engineering and apply it to our data platform work. By measuring progress, not perfection, we can achieve the goals for our organization. Even if a woodpecker pecks a new hole,
[18:19] we have the resources we need to patch it up without falling through the attic. And that is my hope for you as you do this. Thank you. If there are any questions, feel free to come up.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.