From Imperative to Declarative Paradigm: Rebuilding a CI/CD Infrastructure Using Hatch and DABs

Summary

• Declarative CI/CD Paradigm: FreeWheel rebuilt their CI/CD infrastructure using a declarative approach, shifting from imperative custom scripts to configuration-driven deployment.
• Databricks Asset Bundles (DAB): DAB simplifies deploying across multiple environments and enables rapid adoption of new Databricks features.
• Boilerplate Reduction: The new approach eliminated thousands of lines of redundant code, enabling easier project onboarding and improved code quality.

From Imperative to Declarative Paradigm: Rebuilding a CI/CD Infrastructure Using Hatch and DABs

Talk by Luigi Di Tacchio, Sr. Software Engineer, FreeWheel, a Comcast Company ; Saswati Bhoi, Sr. Site Reliability Engineer, FreeWheel, a Comcast Company
Building and deploying Pyspark pipelines to Databricks should be effortless. However, our team at FreeWheel has, for the longest time, struggled with a convoluted and hard-to-maintain CI/CD infrastructure. It followed an imperative paradigm, demanding that every project implement custom scripts to build artifacts and deploy resources, and resulting in redundant boilerplate code and awkward interactions with the Databricks REST API. We set our mind on rebuilding it from scratch, following a declarative paradigm instead. We will share how we were able to eliminate thousands of lines of code from our repository, create a fully configuration-driven infrastructure where projects can be easily onboarded, and improve the quality of our codebase using Hatch and Databricks Asset Bundles as our tools of choice. In particular, DAB has made deploying across our 3 environments a breeze, and has allowed us to quickly adopt new features as soon as they are released by Databricks.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is the difference between imperative and declarative paradigms?

Imperative focuses on how to deploy (custom scripts), while declarative focuses on what to deploy (configuration-driven specification).

What were the problems with FreeWheel's old CI/CD system?

The old system was convoluted and hard-to-maintain, requiring every project to implement custom scripts for building artifacts and deploying resources, resulting in redundant boilerplate code.

How does Databricks Asset Bundles help build data pipelines?

DABs provide a declarative framework for defining data pipelines, workflows, and infrastructure-as-code. Instead of writing imperative deployment scripts, teams describe their desired pipeline configuration, and DABs handle deployment, versioning, and environment management automatically.

Can AI agents safely build, test, debug, and deploy data pipelines?

With DABs' declarative interface, AI agents can reliably generate pipeline configurations that are version-controlled, tested, and deployed consistently. The structured format reduces errors compared to ad-hoc script generation.

What are the benefits of declarative pipeline definitions?

Declarative definitions eliminate manual imperative scripts, reducing deployment errors and maintenance burden. With DABs, teams get automatic versioning, environment-specific configuration, and safe rollback capabilities for mission-critical pipelines.

Full transcript

Welcome to the breakout room session where we're going to talk about from imperative to declarative paradigm. And I'll give it off to. Thank you for the introduction and hello everyone. Uh thank you so much for collecting whatever energies you have left at the end of this uh really long second day of uh data summit uh to attend this presentation. Really appreciate it. Uh it's a long title. We have a lot of ground to cover.
In the first part, we're going to look at a case study uh where we rearchitected our CI/CD infrastructure following a declarative paradigm. Declarative means the emphasis is on what to deploy, not how. So let's keep this in mind. uh and then I'll give you a brief overview of Hatch and Data Bricks asset bundles which are the two tools that we used to uh pretty much eliminate most of the boilerplate code from our infrastructure.
Uh I'm originally from Italy. I moved to the States back in 2013. Started working in academia of all places. Um and my research focus for many years has been energy efficient data transfers which is a very niche topic uh really cool topic but very niche uh my advisor didn't know what I was working on most of the time uh which is most common than what you would think uh and after five years I was like you know what research is a bit too abstract I want to build products I want to build data products so I moved to New York City and started working at Freewheel Comcast cast uh and after five and a half years still building data pipelines using spark and data bricks and uh to be frank I'm honest that I made that transition.
So I know if you're here you know what a CI/CD infrastructure is but in case you don't know just a brief overview of what CI/CD means. CI/CD means continuous integration and continuous delivery deployment either delivery or deployment. It's a system to automatically take your source code and integrate your source code and deploy or deliver your application across different environments depending on whether the application is ready for deployment or deployed.
That's where the D again stands for delivery or deployment. It's a small detail but it's it's worth pointing out. And the other concept that uh we are going to emphasize in this presentation is pipeline. What is a pipeline? Is just an assembly line for your application. It's just a series of stages that take your source code, builds the application and then deploys that application across different environments typically the dev stage and production.
Now that we have the basics covered, I want to start with uh a quote from late 2000. This is from, the co-founder of Stack Overflow. The best code is no code at all. How many of you have heard this quote? Raise your hand. Yeah, it's a it's a famous quote. is just stating the importance of uh being concise, striving for simplicity and not overengineering. And I think this is important across software engineering, but particularly for a CI/CD infrastructure.
So I'm going to tweak this quote slightly and state that the best CI/CD code is no code at all. And some reasons why CI/CD is not code that has intrinsic value. It doesn't ship features. It builds and releases other code and CI/CD code tends to be very complex and with complexity comes bugs that are very difficult to identify and address. And a simpler infrastructure is also easier to maintain. It's easier for new developers to learn the infrastructure and contribute and improve it which is very important.
But this is not what we inherited in my demat free will. It's not where we started from. We started from something that looks like this. It it looks complex. We're going to unpack every single component. But for now, let's just focus on the two blocks here. Uh on the left hand side, the pink block, that's the project pipeline. And that's the Jenkins pipeline which runs the Jenkins file and then delegates the actual logic for building and deploying to the box on the right which is the project folder which provides the build and deploy script to build the application.
Now let's unpack each component individually. Starting with the project pipeline on the left hand side we have multiple Jenkins files to choose from. The difference between these Jenkins files is the environment and some differences, minor differences between stages. These are all imperative Jenkins files by the way. Imperative means going to oversimplify it a little bit is uh general purpose Groovy code. They are for all extent and purposes uh just plain Groovy scripts.
They set up the environment and then uh check out the project based on the pipeline name. And that's why we have one pipeline per project and we have many projects at free will as you can imagine. But the actual logic for building and deploying is delegated to the project itself that has to provide a build script with the actual build logic. Most projects would run unit test. some project would skip that step uh and then build a Python egg which is the old way of packaging a Python application and I would then upload that Python egg to JROG artifactory.
Moving on to the deploy step uh that's where the artifact will be transferred from JROG artifactory to S3 to be available for data bricks resources. uh and then they would then deploy the data bricks resources using the rest API and those data bricks resources which in our case is workflows would include the egg previously uploaded to S3. Now tons of areas for improvement starting with the presence of multiple Jenkins files which are not version controlled leading to confusion. the environment being hardcoded in the Jenkins file, which means if you need to change something in the environment, you have to redeploy in your Jenkins file.
There's also multiple pipelines that are doing pretty much the same thing with some differences, but they are doing very similar steps. And the actual deployment is not even handled by the Jenkins file. It's delegated to the project. And the project has to define a build and a deploy script leading to lack of standardization. Every project will do something a little different. There's also data practices. As I mentioned, building a Python egg that's the old standard.
Nowadays, we build wheels and that was very difficult to address because you had to change each project individually. The artifact was being replicated not just from Jrog Artifactory to S3 but also across environments. The artifact was actually being built in dev stage and prod instead of building once and promoting which is the best practice was being built multiple times once per environment and finally dealing with a rest API is very complex and convoluted.
There's a ton of complexity that you have to handle. For example, if you're deploying a workflow, you need to check if it's a new workflow, you have to hit the create endpoint. If the workflow already exists, you have to hit the update endpoint, which means you have to hit the list endpoint first to check if the workflow already exists. And this complexity had to be handled by the developers. So as we approached the process of refactoring the infrastructure and redesigning it from scratch, we wonder what principles, what best practices we can use to rebuild it the right way.
And we looked at a number of different design principles starting with the 12 factor app. How many of you have heard of the 12 factor app? Not many people. So the 12 factor app is a set of 12 principles to design distributed applications and most of these principles apply to all software engineering. Some of them are a really good fit for a CI/CD infrastructure starting with one codebase many deploys. Have one version of your logic and deploy multiple times one per environment and for different projects as well.
Keep your config separated from the code so that if you need to make a change, you don't need to update your logic. The logic is unique. You can just inject the config and change the config. And that means if there's a new use case, you can just inject a new config file. And also keep your build separate from release, separate from run. Don't have everything in one pipeline. If you want to rebuild your application or redeploy your resources, you can do that separately and independently.
Don't repeat yourself. Don't don't be redundant. There was a lot of redundancy in the old infrastructure as I pointed out. Separation of concerns. If you have different responsibilities, keep them in separate components of your infrastructure. Don't have one module trying to do too much. And finally, this is a bit of a a less known maybe underrated uh design principle which is convention over configuration. If you can find a convention that makes sense and that is easy to follow to achieve a certain goal that will allow you to omit some logic and some configuration that is going to over complicate things. and we're going to see how these principles were implemented in in the new infrastructure.
But just wanted to make sure that uh uh we are on the same page and you're all aware of uh of these principles. This is what we came up with and almost no code infrastructure. There is still a little bit of code but for the most part uh is a code-free infrastructure. So we have a unified build pipeline. This is just one pipeline that is used by all projects. Uh we have the Jenkins file, the build Jenkins file that you can see is running on Docker.
Uh we'll see why in a second. There's only two inputs. Only two inputs. The config file depending on whether you're building a snapshot or a final release uh you're going to inject the right config. And the second input is the branch or tag name which is formatted as project name slash version. So the Jenkins file knows what project to check out and deploy and what version to build. This is very important. The actual life cycle of the Python application is handled by Hatch.
How many of you guys have heard or used Hatch? Oh, few people. Okay. Um, not enough people, no hatch. Uh, it's a fantastic tool, very easy to use, can handle pretty much every single step in a Python application life cycle. Uh, we're going to see some uh sample commands that you can run in Hatch. But anything from validating the version, running unit test, linting, building the artifact, publishing the artifact, environment management, that's handled by Hatch.
The only thing you need to provide is the codebase obviously the project codebase and pi projectl which is the manifest file. It's a file simple file that has the project metadata. Hatch will take uh care for you of building the application building the wheel and then publishing the wheel to a private Python package index which in our case is called artifact because it's cheap and very easy to use. This is the deploy pipeline.
This is the unified pipeline that deploys data bricks resources. Very similar in structure. We have the Jenkins file running again on a Docker container. Uh two inputs, the uh config file for the environment and the branch or tag name which is project version. Uh once again the actual logic is handled by data bricks asset bundles. data bricks asset bundles. Uh once you provide the template which describes the workflow and the environment will take care of deploying the resources for you.
We'll figure out okay what's the parameters for the environment that you want to deploy. We'll do the replacement and interpolation of those variables in the workflow templates and we'll deploy those workflow templates for you or or any other resource that you define. Obviously in our case is data bricks workflows and those workflows install the application deployed previously deployed to uh code artifact. Very similar to the deploy pipeline is the run pipeline.
Uh the only difference here is two more parameters besides the branch or tag name which is the workflow name. You need to know what workflow you want to run and the job parameters what parameters you want to run the workflow with. But it's very similar in structure. Now let's look at the principles that we reviewed earlier where those principles were applied in the redesign of this infrastructure. One codebase, many deploys.
We have a single version controlled Jenkins file which is uh deployed multiple times for different environments and different projects. But it's just one codebase. We have separated the configuration from the code. We have a uh config for the snapshot and one config for the final release. And we also have separated the build pipeline from the release pipeline. So that if you want to redeploy your application, you can do so without having to redeploy the data bricks resources.
And vice versa. If you want to redeploy your data bricks resources, you don't have to rebuild your application. Convention over configuration. This is where we designed the branch or tag name to be project. This is not by accident. This matches the folder structure in our monor repo. So that the Jenkins file just by parsing out the tag knows exactly what project to check out and what version to deploy. And the version is a Python compliant version.
Uh it's in its basic form semantic versioning. So it's major. mminoricro major for breaking changes, minor for new features, micro for bug fixes. It's a bit more nuanced Python versioning, but in its basic form is pretty much uh semantic versioning. And it's also easy to validate the version that you're deploying because it can be extracted from the tag or the branch name. Separation of concerns. We separated concerns in two different ways.
The first way is we have different configuration files for different components of the infrastructure. So we have on the left hand side uh the config files for the Jenkins file which define the environment. In this case for the build pipeline we have the snapshot and the release configs. The project has its own config pi project. l that's the manifest file. Hatch has its own config. So we can tune and customize each component individually without breaking another component of the infrastructure.
And the other way we separate the concerns is by running the Jenkins file in a docker container. Which means the docker image is this all responsible for defining the deployment environment. Previously, we were running the Jenkins file directly on the Jenkins virtual machine, which means if there's a security patch or there's a team that wants to make an update that can break another team deployment environment, so it needs additional testing.
By running the Jenkins file in a Docker image, a team can independently update the version of Hatch that they're running or installing a new Python version, without breaking another team's deployment environment. And the data DevOps team can apply security patches without breaking the de deployment environment for any team that is using that virtual machine. All right. So, let's look at Hatch in more details. I'm going to give you a brief overview of Hatch and highlight some of the uh features that uh I think make Hatch stand out compared to other tools.
Uh as I mentioned earlier, Hatch is an all-in-one project management tool. It handles pretty much every step of a Python application life cycle. It's very intuitive to use. Uh I'm going to show you some commands and I think you can understand what each command is doing without even looking at the documentation. Hatch project metadata version. This is retrieving the version from the project metadata. Hatch test parallel cover.
This is running unit testing in parallel and is generating code coverage. Hatch format. This is formatting your code. Is doing linting. Hatch build. This is building the wheel. Hatch publish. This is publishing the application to a Python package index. You see very easy to use, very intuitive by design, but it's also very powerful. It has a a ton of features that other tools don't have. I'm just going to highlight one of them, which is perfecting Python environments.
I think it really makes Python environments which are already great even better. Uh so it it starts with the concept of Python environment which is an isolated environment where you can install the project dependencies. If you want to pin the version of one of those dependencies, you can do so. But it also adds uh the option to add additional extra dependencies. For example, in this case, we see a test environment that has two additional dependencies, piest and piest.
You can also define commands, well actually scripts, entire scripts if you want to, and assign aliases to those scripts. And we see a couple of examples in this case, run coverage and run. And these are simple commands, but you can put entire scripts. And it also has a concept of environment hierarchy. You can build environments from other environments and inherit dependencies as well as commands. So in this case if we were to run something like hatch run name of the environment test in this case column name of the command which is run in this case.
Uh this would create the python environment. It would install the base dependencies for this project which is pi spark and s3fs. It would then install the additional dependencies defined just for the test environment piest and piest cov. And finally it would run the command or scripts which in this case is run is run coverage nov all with one simple command. And obviously you can make it as complicated as you want, but very intuitive, very easy to use, but also uh highly customizable.
There's a couple of different ways you can customize Hatch. Uh my favorite way to customize it is via config file. I think it's the cleanest. It's also the easiest to um update and learn for new developers. Uh in this case we are customizing the hatch built-in environment hatch test uh and we are enabling parallel testing. We are setting up retries uh in case unit tests fail for whatever reason uh we can retry those tests.
Uh we have additional dependencies defined for uh all projects and we also uh define what Python versions we want to use when running unit test. uh in this case it's two versions which are the two versions in the data bricks runtime environments that we use across all projects. Other ways is via environment variables command line arguments. You can customize hatch dynamically if you prefer uh but I think the config file is the cleanest overall.
Now data bricks asset bundles. How many of you guys have used data bricks asset bundles? Oh, nice. Most of you guys. Uh, it's a fantastic tool to define data bricks resources as source files. It's an example of infrastructure as code. You start by defining workflows as YAML. Now there's support for Python as well. uh which uh given the power of a general purpose programming language you can do more but I think there's some advantages in defining workflows as static YAML I think is cleaner and easier to maintain uh but if you need to do more complicated things you can use Python as well but you can parameterize workflows by environment and uh that's that's one of my favorite features of data bricks asset bundles you can define one workflow parameterize it and then define the config for different environments and the data bricks CLI will figure out given the uh target environment that you want to deploy to what's the configuration that needs to be replaced in the workflow.
It's composible. You can break up your bundle by defining for example a YAML for your shared variables and one or more YAML for workflows. And it's also hierarchical. So you can have a YAML with the shared config used by different projects or used by the entire team. And then each project can provide the workflow definition as well as uh variables that are specific to that uh project. And this is actually the setup that we have in our team.
We have a datab bricks. yaml which is shared by all projects. Uh this one has things like the AWS account ID, the data bricks workspaces, the service principle that in Jenkins is going to uh deploy the workflows. Uh group permissions, things like uh the dev group, the um the dev group can have access, read access and run access to all workflows in dev. uh and the default schedule status um for obviously jobs that run on a schedule.
Uh we don't want to accidentally schedule a job in dev. And then each project provides one or more YAML with the workflow definition, data paths, input and output, uh the distribution list for alerts. If something goes wrong, we know who the alert and then service principle, the service principles that are going to run the workflows. Let's look at an example of the shared data bricks. yaml. On top, we have the bundle name which is parameterized.
This is going to be the project name. Uh then we have the variables section. Here's where you have the teamwide config. So shared variables like the AWS account ID, the service principle that is going to deploy the workflows as well as variables that needs to be defined by each project when deploying the bundle. Things like the project name and the project version. Uh and these are things that by the way in our case can be extracted from the branch name or the tag.
So there's really nothing that the project need to provide. include. This is where you add the path that contains all the YAML that you want to include in this bundle. And in our team, we came up with a convention convention over configuration that every YAML that needs to be provided by the project goes into uh the project root under templates/databicks. So any YAML added to that folder gets picked up by the CLI and deployed as part of the same bundle.
And permission uh as I mentioned earlier, this is where we have shared permissions like the group of developers has read and run access for all workflows in dev. For example, this is the first part of uh data bricks. yaml. In the second part, this is where the magic happens. This is where you define different targets. In this case, three targets. One for dev, one for stage, one for prod with uh the values for all the variables previously defined uh for the different environments.
So this is where you have the value for AWS account ID, the value for the service principle, etc. as well as the workspace URL where you want to deploy the bundle which can also be passed as an environment variable. Now that's the shared YAML. Each workflow is going to provide one or more YAML with the workflow definition as well as all the config around the workflow definition. And this is an example. Uh we have this resources job section with the workflow config.
This matches the jobs API. If you use the jobs API, this is exactly one to one match. Uh I'm going to highlight a few uh sections which are particularly important. Tasks. This is where you uh define your DAG your directed as graph. So tasks dependencies between tasks as well as what function each task runs. Job clusters unless you're using serverless. This is where you define the cluster config, how many nodes, autoscaling, etc. , etc.
Parameters. This is the job parameters are shared across all tasks in the workflow. And these are passed as command line variables. So they can be easily parsed uh using the Python standard library. just arc parse uh with a few lines of code you can have a nice uh tidy dictionary with all the parameters available throughout the code and finally this is where you define the user or service principle that is going to run the workflow.
Now, this final section, um, this is where we set up alerts, uh, in prod, uh, in case the job fails. And I know it's a little verbose and it's a little more complicated than just defining a, um, the value of a variable in a particular environment because we're actually adding a section to the workflow in prod and in prod only. Uh, but I think it's very easy uh to understand and very easy to read. So we have targets prod we are adding something in prod and only in prod uh resources jobs workflow name so we are adding in prod to workflow name uh email notifications on failure.
So this is alerts in case the job fails uh to this distribution list which is parameterized. You can hardcode it if you want. uh we have it parameterized uh and this is how we add alerts on failure only in production. Now why did we go through all this work to rearchitect the CI/CD infrastructure? I kind of alluded to it earlier but uh let's highlight some of the benefits. There's very little code now thanks to Hatch.
Most of the logic is handled by Hatch. So we removed in the build pipeline 99. 9% of the code. Uh same thing for the deploy pipeline. Everything is handled by dab and by the CLI data brick CLI. So we have removed on average between a th00and and 3,000 lines of code per project. Uh this is based on the merge commit as we on boarded new projects to this infrastructure. It's a much smaller code base, much easier to maintain.
It's much easier for developers to learn the infrastructure and contribute to the infrastructure and improve it. There's the occasional uh Docker image upgrade that we need to do every now and then, but for the most part, this is a lowmaintenance infrastructure. On boarding new projects is really easy. In the old infrastructure, the project had to define a build script, a deploy script. the developer had to learn uh the Jenkins file to know exactly what the Jenkins file expected from the build and deploy script.
Now you just provide the manifest file, you provide the DAB template and that's it. Uh the application is built, the resources are deployed, no additional code is needed and best practices are not delegated to the project, they are built in the infrastructure by design. So things like testing, linting, those are stages. Those are steps in every single pipeline. They're not left to the project to implement. There's obviously ton of room for improvement.
Uh starting from collecting all the common patterns that have emerged as we uh rearchitected infrastructure and putting them in a shared Jenkins library. uh this would remove some redundancy from the Jenkins files. We would also like to support new resource types. Right now we support data bricks resources obviously uh via asset bundles and AWS Lambda functions but we would like to add uh some new resources in the future and we would also like to automate the pipelines.
Right now there's a a bit of manual work required. somebody has to go to the Jenkins UI and has to click on build. Uh we would like to automate that step. Once a PR is merged, a tag is created that kicks off the pipeline for building and deploying. But I think this is a a really good improvement compared to the old infrastructure. And uh I hope this was uh uh interesting and informative. I hope you learned something whether it's uh the design principles or uh one of the tools that I mentioned Hatch or data bricks asset bundles.
Uh we have uh a few minutes for questions so I'm going to open the floor to questions or comments but thank you for the attention. Yeah.

Learn more about the Databricks Data and AI platform.