Databricks Declarative Automation Bundles (DABs): Pro Tips for Deploying Workflows

Summary

Direct deployment: Removes Terraform dependency with readable JSON deployment state.
Safety gates: Branch protection, CLI version gates, and Python mutators prevent production mistakes.
Bundle templates: UI changes sync back to YAML to simplify deployment as code.

Databricks Declarative Automation Bundles (DABs): Pro Tips for Deploying Workflows

Talk by Hubert Dudek, Data Architect, Databricks MVP community
Databricks bundles (DABs) let you define jobs, pipelines and dashboards as code and deploy them across workspaces, and the direct deployment engine removes the Terraform dependency for faster deploys and a readable deployment state. This video collects production tips for deploying Databricks workflows as code: branch protection so only main reaches production, a minimum CLI version gate, Python mutators that inject central config into jobs, referencing resources instead of lookups, and applying cluster policy defaults.
Hubert Dudek demonstrates live: branch protection so production deploys only come from main, a minimum CLI version check, Python mutators that inject a central config into jobs, referencing resources instead of lookups, passing Terraform outputs in as variables, and applying cluster policy defaults. He also shows syncing dashboard and job changes from the UI back to YAML, bind and unbind for existing schemas, bundle templates with placeholders (also available in the UI), and bundle plan to review changes in a CI/CD pull request before deployment. Useful if you want to know how to build data pipelines and automate Databricks deployments safely.

Chapters

FAQs

What advantages does the direct deployment engine provide?

The direct engine eliminates Terraform dependency, making deployments fast and keeping state in a clear, readable JSON format. This video shows that the state file can be parsed for additional information, keeping everything as code without Terraform complexity.

How does branch protection prevent production accidents?

By specifying branches in targets, you ensure only specific branches can deploy to production. This video demonstrates that with branch protection enabled, all production deployments go through main, preventing mistakes from experimental branches.

What do Python mutators do in bundle deployments?

Python mutators inject central configuration into jobs automatically. This video shows how mutators keep jobs synchronized with a shared config without manual updates across each deployment.

Why is minimum CLI version protection important?

This video recommends always specifying the minimum required CLI version in bundles. This prevents new features from breaking on older CLI versions running on different machines or CI runners.

Full transcript

`00:00` Introduction

How to get your voice out of your voice? Welcome everyone! Are you tired? I'm really tired, like it was so intense in four days and my voice is dying slowly, so be a pardon if my voice is dying and you don't understand me but I will do my best, yes? So first I will introduce myself. I'm Hubert Dudek, MVP. I'm technical advisor to Sare Data. Also I'm working on contracts. So, and probably you know that I'm active on YouTube, on LinkedIn, on Medium blog.
Also I'm really active in community. So in my opinion it was a great event. I have pleasure to have the last talk on the summit. So I'm really happy with that. Just I'm tired, yes? So let's start. So I'm going to present like my favorite features in declarative automation bundles, yes, DApps. Probably who is familiar with DApps, you may know all these tricks, but let's go one by one. So first, first tip, which is really important, is that now we have direct engine in Databricks asset bundles.

`01:13` Direct deployment engine: deploy bundles without Terraform

And what is the benefit? No more Terraform. And when you deploy using direct engine, you don't need to download Terraform. It's really fast. And soon it will be default setup for asset bundles. And also when you use the direct engine, the state is really clear. You have really clear file like deployment JSON. and you can parse these files and get additional information from that files. So thanks to that you have everything set as a code in readable format.
So it's not like Terraform. So if you are using DAPS, switch to direct engine and that's the way to go. OK, let's move to next tip. Wait a second. Another tip in asset bundles is branch protection. When you have targets, you can specify branches. So then it will be not possible to deploy to given target from different branch. And this way you are sure that, for example, all production deployment go from main. And this way, no one will match with your code.

`02:32` Branch protection for production deployments

And you see that if you deploy from different branch, then you have error. Another tip is version protection. So always in bundle, you can specify minimum required CLI version. Now actually, bundle has already version 1. 0. But in every feature, as you see on that recording, in every version, you have a lot of new features. So it can happen that, for example, on runner, there is different version than on your machine.

`03:14` Minimum CLI version protection

And everything can crash. So I really recommend to everyone to include that minimal CLI version. So if you add to your code something which is quite new in DAPS, specify that minimum CLI version. So you will be sure when, for example, your colleague from work or your CI CD process will use that bundles, it will work actually. So it's better to get error that version is incorrect then that everything is crashing. Another one is something which I really like because it's more complicated topic, but you have now mutators.

`04:07` Mutators: inject a central config with Python

So mutators allow you to ingest data to your YAMLs. So basically it's two way, because mutators both read from your YAML, So you can reference using Python variables from YAML, but you can also ingest additional settings. So for example, on that presentation, here on that video, I'm using that bundle. I'm reading variable, resolve variable from YAML, and then based on that variable, I'm using JSON with central config, which can be even in separate repository.
And that settings from that JSON are applied as a global parameters. So in that case, I can have one JSON for whole enterprise, and it will be automatically applied to job as a parameter. For example, default catalog, default schema as a parameter. And we can see now that when it's deploying, actually, it's installing PyDabs library because it's in Python. Actually, CLA is in Go written, but that one is in Python. So it needs to use Python to process that YAMLs.
And one is deployed, so we can see that the bundle was successfully deployed. We can see that in job we didn't specify any parameters, but they are propagated from that central config. So I really recommend that concept to everyone to have one central config with all your parameters. Another thing is that I often see in code that resources are referenced using lookup. Sometimes you have to do that, of course, or hardcoded ID, but always remember that you can reference resources but using resource. syntax.

`06:20` Reference resources instead of lookups

So this way you can reference another resource, which is in the same state. And this way, when you change, for example, name of SQL warehouse, when you have SQL warehouse as a data declarative automation bundle resource, then it doesn't affect, because it's referencing the state files. So just for referencing, when you can, because there are a lot of situations, use resources dot and the type of resource like schemas and then for example dot and my schema as a resource key.

`07:16` Pass Terraform outputs into bundle variables

There are more and more resources under bundles but even if you have everything under bundles you still need to to use Terraform to deploy your architecture like for example Azure storage accounts service principles etc so there is possibility actually in in bundles to pass output from Terraform and read this as a variables from JSON then just to do that we need to we need to put that in specific folder. And then we can reference to that JSON, which is from Terraform using variable syntax.
So then these things from this JSON are seen by bundles as regular variables. Another thing in every enterprise to define policies for your classic compute clusters. And usually in policies you can use something like default values for the policies. And then you can reference that policy and in bundle you can specify apply policy defaults values to true. So this way you don't need to specify anything related to classic compute in your bundles because all defaults are taken from that policy.

`08:22` Apply cluster policy defaults in bundles

And you only specify something like, for example, number of workers if you want to change the setting from policy. So to do that we need to set policy and you can see now that in UI the cluster is set and working based on values from policy, not the values from bundle. And actually, now we have on another slide that now it's possible to think back values from UI to bundles. But also, if you are working in Visual Studio Code, and for example, you want to edit the dashboard, rather, it's not so easy to do that using code because it's not a free-handly format.

`09:41` Sync dashboard and job changes from the UI back to code

So you need to go to UI and edit that. And actually, there are a couple of functions to watch for changes. So if you use Databricks Bundle Generate dashboards, it's generating bundle, but there is also command to watch for changes. So if we update this in UI, then it's propagated down to our code, which we have in that case in Visual Studio code. So in that example, we added that and we can see that the dashboard is written back to our directory on our, for example, laptop.
Another, and it's a new release just now for Summit, is possibility to edit if we are in a source deployment mode and in mode development, this source link deployment, and if we deploy, so usually it's target development, yes, and so it's not production mode. And if we go then in UI and we change something, so for example, in that example here I'm changing the name of the task, then that value, when I update it in UI, is propagated down to YAML.
So we got that message that YAML was updated, and we can see in YAML that the changes from UI are there. So that way is really good for development because especially with jobs, if I have really complicated job, then it's really easy to lost in logic. And this way you can mix both words, UI and code. Of course, when you have really complicated project, for example, using mutators, which I mentioned before, then always review differences in Git to be sure that UI updated what you wanted, because sometimes some edge cases are really complicated.

`12:19` Bind and unbind existing resources

Another thing is that often in bundles we declare resource which already exist. Usually I see this for schema because we declare in bundle schema and already exist and cannot be deployed. So this way you don't need to delete your schema and do everything from scratch. There is comment from CLI, from DApps, which can link existing resource using bind comment to state of your bundle. So this way we are linking actually that my schema from Hub Catalog to the state of Bundle.
And now when we deploy, Bundle knows that it was already deployed. Yes, so we can all, and the same we can do unbind. So for example, if we want to do something manually with our resource, we can unbind, then we can bind. So it's good always to remember about that functionality. Another thing is that especially when you work in big team, you need to remember that you can create templates in bundles. So templates, I can call it variables, but I call it rather placeholders, that this placeholders, which is specified in JSON, are then replied when we use that template. nice thing is that placeholders can be in code but can be also even catalog names.

`13:22` Bundle templates and placeholders

So then when we click Databricks bundle in it and reference to the templates, it's asking us questions to fill all these placeholders with our values. So for example we can we can specify catalog, we can specify different things and actually from last month that option is also available in ui so there is there is ui folder it will go in a moment to that so when we are in in settings of development there in admin we can set folder for templates and we can put our templates there in that folder, and later when, so here we can see that there is JSON, and now when we are in some, it has to be in Git, when we click create and create bundle, so when I click create bundle, actually I can use this, my test template.
So it's integrated with UI, so we choose the template, And then we have the same question as in still eyes. So we feel all the values so we can change them. And then the placeholders are replaced. Another really nice thing is that in bundles, when you are in this direct engine mode, which was in the first slide, we can generate plan. And that plan is JSON with all changes, which will be applied during deployment. So when you create CICD process, because bundles is not CICD tool, it's like tool defining resources.

`15:23` Generate a deployment plan for CI/CD review

But in CICD process, when you have open pull request, you can actually generate that plan. And then when you want to merge that pull request to your target branch, you can review that plan, approve that everything is OK, and then normal deployment is done using that plan. So you see, first I generated plan using bundle plan, and then I used Databricks bundle deploy using that JSON with that plan, which was generated.
So before that, I was able to review the changes. and that's all of my tricks if you if you want to read read more about data brick asset bundles are I like every month I publish some blog posts there is data Brixtel medium medium blog also on sunny data on sunny data blog there is the same content with additional content from them. And I have also my own YouTube channel. I'm on YouTube channel and there are other news.
Yes. So if you want to know what is new in Databricks, I welcome everyone to visit the resources. And I wanted to thank you for attending my talk and also for attending the last talk during the summit. So for me it was a great time and I'm really happy that all of you came to my talk. So I really didn't expect this number of people. So thank you really much. And if you have any question, we can handle them individually.
I have also some sweets and stickers remaining in MVP zone. so we can talk and thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.