How do I set up CI/CD with Databricks?

Summary

• Databricks recommends Declarative Automation Bundles for CI/CD pipelines. A bundle deploys code, jobs and infrastructure as one YAML-defined unit, which the docs say keeps environments consistent.
• The core loop: keep code and bundle configuration in one Git repository, then have your CI system run databricks bundle validate, databricks bundle deploy and databricks bundle run against dev, staging and production targets.
• Authenticate CI as a service principal, not a user. Databricks recommends workload identity federation, which removes the need for Databricks secrets in your CI system.
• Isolate environments with separate workspaces and catalogs, test on every pull request, and keep bundles small so you can roll back one bundle without touching unrelated workloads.

How do I set up CI/CD with Databricks?

Put your code and your Declarative Automation Bundles configuration in one Git repository, define each environment as a bundle target, and have your CI system run the Databricks CLI's bundle commands as a service principal authenticated with workload identity federation. That's the setup Databricks recommends. Bundles deploy code, jobs and infrastructure as a single YAML-defined unit, which the docs say keeps environments consistent.

What a Databricks CI/CD pipeline covers

The CI/CD docs describe a common flow. Here it is with the tool for each stage:

Stage	What happens	Databricks tooling
Version	Store notebooks, .py and .sql files and bundle YAML in Git	Git folders for authoring, one repository, trunk-based branching
Code	Develop and unit test in the workspace or a local IDE	Workspace bundle editor, Databricks IDE extension
Build	Build artifacts such as Python wheels during deployment	The bundle artifacts mapping
Deploy	Push the bundle to a target environment	databricks bundle deploy -t <target-name>
Test	Unit tests on pull requests, integration tests in staging	pytest, databricks bundle validate
Run	Trigger jobs and pipelines defined in the bundle	databricks bundle run
Monitor	Watch production workloads and roll back failures	Jobs monitoring, small bundles

Principles to build in

The docs list a short set of principles that hold whatever CI system you use: version control everything, automate testing, define infrastructure as code, isolate environments, monitor and automate rollbacks, and manage assets as a unit. They also say what not to commit: build artifacts such as .jar or .whl files (upload them to Unity Catalog volumes during CI), tokens or credentials, and local files containing PII.

How Databricks approaches CI/CD, step by step

1. Create the bundle. Run databricks bundle init to start from a template, which creates databricks.yml. Define one target per environment. Deployment modes apply a group of commonly used settings at once: development mode prefixes deployed resources with [dev <user short name>] and pauses schedules and triggers, while production mode can enforce the Git branch you deploy from.

targets:
  dev:
    mode: development
    default: true
  prod:
    mode: production
    git:
      branch: main

2. Isolate environments. Use separate workspaces for development, staging and production. In a single Unity Catalog metastore, create a catalog per environment, give developers personal schemas, and bind the production catalog to the production workspace in ISOLATED mode.
3. Give CI its own identity. Use a service principal for automation and set run_as for production workloads. The docs recommend separate service principals for deployment and runtime, and workload identity federation for authentication: the Databricks CLI exchanges tokens issued by your CI environment for Databricks OAuth tokens, so there's no long-lived secret to rotate. The docs include federation examples for GitHub Actions, Azure DevOps Pipelines, GitLab CI/CD, Jenkins and other CI tools.
4. Wire the pipeline. On each pull request, run unit tests, then deploy the bundle to a non-production workspace. The docs prefer that deploy over validation alone in CI because it catches YAML and resource-mapping issues before production deploys. On merge to main, deploy to staging, run end-to-end jobs with data-quality checks, then deploy to production:

databricks bundle validate
databricks bundle deploy -t prod

5. Version and roll back. Tie artifacts to Git commit hashes or semantic versions. Keep bundles small and owned by one team, so an incident means reverting one bundle to its last known good version.

Lighter-weight options

Not every team needs full bundle-based CI/CD on day one. The docs describe two code-only approaches:

• Production Git folder. An admin creates a Git folder on a deployment branch, and automation pulls it on merge with databricks repos update. Only code is source-controlled; job and pipeline configurations aren't.
• Git with Lakeflow Jobs. Notebook, Python script, SQL file and dbt project tasks check out a branch, tag or commit from your remote repository at run time, and every task in a run uses the same snapshot.

See CI/CD on Databricks, CI/CD workflows on Databricks and Developer best practices.

FAQs

Should I use bundles or Terraform?

Both, for different layers. The docs recommend Terraform for cloud-level and external resources and admin actions such as workspace provisioning, and bundles for all other Databricks resources.

Which CI systems does Databricks document?

The CI/CD overview names Azure DevOps, GitHub Actions and Jenkins as tools for deploying bundles. The bundle commands are plain Databricks CLI calls, so a job in any of them can run them.

How should my CI system authenticate to Databricks?

As a service principal using workload identity federation. Databricks calls it the most secure option because it removes Databricks secrets from your pipeline.

Is there a starting point for machine learning projects?

Yes. MLOps Stacks combines bundles, preconfigured CI/CD workflows and modular ML project templates, and databricks bundle init offers an mlops-stacks template.