CI/CD for Databricks: Advanced Asset Bundles and GitHub Actions

Summary

• DABs for CI/CD: Databricks Asset Bundles enable automated, repeatable deployments of notebooks, DLT pipelines, and workflows across environments.
• GitHub integration: Using GitHub Actions with DABs automates testing, linting, and deployment—ensuring code quality and consistency.
• Advanced patterns: Target-specific overrides, environment variables, and multi-environment support enable scaled CI/CD without duplicating code.

CI/CD for Databricks: Advanced Asset Bundles and GitHub Actions

Talk by Dustin Vannoy, Sr. Specialist Solutions Architect, Databricks
Databricks Asset Bundles (DABs) provide a way to use the command line to deploy and run a set of Databricks assets — like notebooks, Python code, DLT pipelines and workflows. To automate deployments, you create a deployment pipeline that uses the power of DABs along with other validation steps to ensure high quality deployments. In this video you will learn how to automate CI/CD processes for Databricks while following best practices to keep deployments easy to scale and maintain. After a brief explanation of why Databricks Asset Bundles are a good option for CI/CD, we will walk through a working project including advanced variables, target-specific overrides, linting, integration testing and automatic deployment upon code review approval. You will leave the session clear on how to build your first GitHub Action using DABs.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction
• 1:04 Understanding Asset Bundles
• 2:52 Dev lifecycle and DABS
• 5:04 Bundle configuration details
• 6:51 CLI and VS Code tooling
• 7:45 GitHub integration
• 10:55 Branching strategy
• 12:29 CI/CD pipeline logic
• 15:29 Live demo: Git to PR
• 20:40 Automated deployments
• 22:27 Advanced DABS features
• 23:42 Automated testing
• 25:03 DABs workspace demo
• 28:52 Conclusion and resources

FAQs

What are Databricks Asset Bundles (DABs)?

DABs are declarative bundles that define all Databricks assets (notebooks, pipelines, workflows) as code. They enable version control, environment promotion, and automated deployment through CI/CD.

How do GitHub Actions automate Databricks deployments?

GitHub Actions can trigger on pull requests to run linting, testing, and validation. Once approved and merged, Actions automatically deploy the bundle to dev/staging/prod environments using the Databricks CLI.

What advanced DABs features enable multi-environment deployment?

DABs support target-specific overrides, environment variables, resource naming conventions, and cluster configurations. This lets teams deploy the same code to dev/staging/prod with minimal duplication.

Who presents this CI/CD best practice?

Dustin Vannoy (Databricks Sr. Specialist Solutions Architect) shares CI/CD patterns and maintains a YouTube channel with additional data engineering tutorials.

Full transcript

0:00 Introduction

All right, let's get this going. We have a ton of people here to talk about CI/CD at 8 a. m. in the morning. Wonderful job. I barely made it myself and um I pretty much had to be here. So, uh you all saw the title slide, so you're here on purpose, which is great. We're going to talk about a super super short here's what DABS is and here's the fundamentals so that if you are familiar with CI/CD and development processes but aren't really familiar with DABs you'll kind of get a little glimpse.
Um but I apologize if you've never seen asset bundles before. If you've never seen asset bundles or GitHub before, hang with me. I'm going to redo this tomorrow and come watch again and see if it, you know, all catches in. Um, but I am gonna try to share the most important things that actually go across environments from dev to staging or test to production. And that's where the advanced piece of asset bundles comes in is what do we end up doing when we actually want to deploy to a bunch of different environments.
So this is probably most of your first session. So please read this statement in full. Um, the this isformational purposes. Anything I say about what's coming in the future is not a guarantee. I'm speaking on behalf of what my expectations are and all the other legal language that this says. Please do complete the surveys. You should have access to those through the app. We figured out the app and um you can do that when the session's over.

1:04 Understanding Asset Bundles

So, here's essentially what we're going to talk about though. Yeah, I I admitted it. So, quick intro, GitHub actions. little uh demo of GitHub actions though it'll kind of be jumping around because I don't think you want to watch the whole thing that happens. Automated test explanation because I have to talk about automated tests if I'm talking about CI/CD. Sorry if you're already a fan of automated tests. We can hang out later.
Um I'm still going to talk about it for five minutes and then we'll talk about advanced ad usage and talk through common things that I've had people ask about when they get to great we've been using asset bundles for development. We're about to deploy. Can you come help us? I have a YouTube channel which has some CI/CD and asset bundles related things as well as other data engineering uh tutorials and commentary. So go check that out if you want to go deeper into some of this than we have time for right now.
Um though some of these advanced things are definitely not on my YouTube channel. So you're getting some new stuff. Trust me. and I'm a senior specialist at uh data bricks doing a lot of Devx and uh quite a bit of data warehouse and data engineering. My career has been as like a data engineer and I learned CI/CD processes because I was told I was doing everything wrong as a data engineer. Um, I'm gonna say it a lot more politely today about what it is we sometimes do wrong as data professionals that we should be doing a little bit better if you haven't already learned the lessons that were drilled into me when I went for a company that was very software development driven and not so data um friendly.

2:52 Dev lifecycle and DABS

Data professional friendly just writing SQL queries and having fun. All right, just the important parts about asset bundles. We got to start with why any of this matters, which is there's this development life cycle. Obviously, there could be a lot more complex image here than what I'm about to show you or what I'm showing you, but we have some sort of planning step, whether it's a really short cycle or a very long planning.
We have development allong development. There's a lot that happens and asset bundles can fit in there to help us test these things more um fully within a data bricks environment. And then really important in the focus of this session is primarily once we think it works and we're ready to merge that and start deploying to testing and production and get the stuff out there, how do we make sure it actually works and uh get it deployed in a fairly automated fashion.
I don't know if anyone else has like written down the steps of scripts you need to run in what order, sent them to a DBA that has to run them and then at 7 p. m. on the release night they're messaging you, but I used to do that. This is much better process, trust me. So what do we need in order to be successful with this process? We need a single source of truth. That's where GitHub comes in. Keep in mind Azure DevOps, GitLabs, other CI uh other tools that support git type version control are great.
Where you need to be able to find these resources. Um I put easy to understand format of YAML. It's okay if you disagree with YAML being easy, but it is kind of the language of DevOps and I don't have any like surprise you don't have to write YAML. Um, maybe that'll happen in the future, but you're probably still going to want to understand YAML if you're doing much of this DevOps stuff. That doesn't mean everyone on the team has to be a specialist though, but I'm not going to get into that right now.
Um, specialized resources for each target environment. So, when I go to stage in production, I'm probably running different data sets, might need some different node types, other things that are going to change as I go across environment. We need to be able to make sure that we have isolation between these different deployments and they aren't bleeding into each other. And then we need to easily automate this deployment.
So the several minute version of what are asset bundles, they are YAML files with a way to deploy. The first piece that's kind of small but important is that there's going to be a name for your bundle. So it all starts with data bricks. yaml and you're going to have a name in there and that's going to define this bundle's name. And then we're going to combine that with some other things to really make some magic. The next piece that uh comes up is the resources.

5:04 Bundle configuration details

What you'll see in all of my demos, you if you've looked at this in the past, you've probably seen it before, is I'm going to include resources from a folder rather than having a huge list of jobs, DLT pipelines, other types of things that can define. But for simplicity sake, we have a super short job right here. So you can see and then the target is very critical. And this is where a lot of the important stuff happens for actual deployments.
My dev target will look different and probably be useful for each person as they're working to be able to deploy their own version of everything and test it out in the workspace. My test stage and prod are going to run as a service principle. We're not going to all just run that willy-nilly. It's going to happen from a CI/CD tool. Uh so where does it run? Yes, it can run from a C CI/CD tool. Really anywhere you can install the data brick CLI.
So, it's a CLI command and that means we can do it from all sorts of CI tools, not just GitHub actions, not just Azure DevOps and some of the ones you see the most often. Um, quite a few commands. The ones you're really going to focus on if you're new to DABs is validate and deploy. Validate, deploy, and that's probably the only ones out. And then run to run a job or run a DT pipeline. So, those are the three I'm going to show today.
The others are fun. The others are cool. Um, I think the help and the docs are are fine to tell you what those do, though. Now, the databicks CLI is very powerful. It does a lot more than just bundles. The main thing to know is that when you get it installed, you set up authentication. We're going to look at a way to do that from our CI tool. And that authentication is going to work across a bunch of different capabilities that CLI provides.

6:51 CLI and VS Code tooling

Now, when we're in development mode trying to set all of this up and make it ready, I highly recommend the VS Code extension. Um, this has a built-in bundle explorer which can help you really see a bit of what's going on. We'll share another way to do it that's that's quite new. You may have heard of it already. We'll share that later. Uh, in addition, if you're going to use cursor instead of VS Code, same extension works.
I haven't hit problems. If you hit problems with me, let me know. All right, now to GitHub and GitHub actions. Back to version control. So, um, as a developer, I would have some sort of branch where my code lives and that I need to merge. We're going to start with a this simple kind of trunk base, just a single main for this image. I check that out with the feature branch. Nothing really special has to happen on the CI side here.

7:45 GitHub integration

I commit different sets of changes along the way. So I'm doing that good practice of version control. Now I go to peer review or on GitHub we'd call it a pull request, a code review. Right about here is where I as the person that didn't write this code would love some sort of checks to happen and that's what we'll look at. Now next time it happens, someone checks out from, does some work with it and they're going to go through their own peer review to merge.
Okay. Um, at some point, let's say for a moment, as soon as we go to main, we merge it, we've approved it, we're ready to go to production, we would kick off some site sort of process using the GitHub action if we're in the GitHub world, and that will make everything happen in production. I'm going to complicate it a bit, but before I do that, let me just call out um if you don't work from your local machine at all, which is true for quite a few data bricks users, and there's nothing wrong with working from the workspace only.
They they do a lot of work to try and make it easy for us to do that. Um, git folders is available. You can use version control. You can do that um you can do that branch creation, you can commit, you can push it up and you can even like kick off the pull request process from there and you can do the comparison. And so this image here shows you look we can compare the differences in our code here in the workspace as well.
Very easy to create. This is more for reference, but there's a blue button. It's one of the options. It'll walk you through the steps. we have that easy comparison in the workspace. For a few file types, it'll be a little bit easier in the workspace than it will be if you're doing it like command line getit like I always do. Um, but for the most part, it's what you would expect from doing a pull request type of view or using your own, you know, UI like the extension in your IDE to do get comparisons.
All right, first first tip that if you've been doing this a while, you still may not know because I think it's fairly recent. We can authenticate using GitHub secrets when we're doing this from our GitHub actions. And usually that looks like adding uh secrets that map to environment variables for data bricks host and then either a token or an service principles client ID and client secret. And then depending on what you're doing, you might need to add a cluster ID.
So we'll see that in the code. The other thing we can do now though is workload identity federation. So it's basically federation policy for tokenless authentication. It's you use a service principle without having to set these secrets. And this is something I've had people ask me for. I wish I remembered who asked me for it because now that's out. I'd like to message them and tell them, but if you know them, let them know it's out.
Uh I do not have that set up for this demo, but um wanted you to know that's available. So I'm going to talk about branching strategies. Not to convince you of anything. Um I see lots of different branching strategies. It's fine. So here's what I like. So git flow tends to be a bit longived branches, multiple branches before you get to production. Um sometimes can be a bit much for people. Trunkbased is usually single branch.

10:55 Branching strategy

We go to it often. We deploy really often. Lots of good things about both quite frankly. What I'm going to do is I I just kind of call it a modified git flow or abbreviated git flow. So I have more than just a main branch. You could either I've seen people call it main and release. I call it dev and then main. So Maine is ready to go to production once it's there. Maybe there's someone that has to approve, but we're ready to go to production once we merge to.
That's what I'm doing for this demo. You don't have to like it. It's fine. So you can follow along. Okay. All right. So here's what I would do when I am working on my feature branch and merging back to dev. Well, let me set this up first. Sorry. Uh so we have some questions we need to ask. What do I want to do when a PR is created? What's going to give us confidence to complete the merge if the code looks good? What do we want to do?
What do we want to run after we merge to that dev branch? When deploy to test and when to deploy to stage. That might be the same environment. You might have multiple steps. It kind of depends. And then once we go to prod, what do we need to do? So this has been a good not very fun but a good conversation I've had with different people within data bricks and I've heard from customers. So I've come up with a recommendation. uh you could easily convince me that oh we want to do it a little different and be just fine.

12:29 CI/CD pipeline logic

But here's just kind of a place to start thinking if you haven't really thought through this yet and you're just getting into this and it's how my demo works. So it kind of helps you understand the code that I'm going to share. So when we first create that PR to go to dev. So I'm a developer. I've worked on this maybe in isolation depending on your team and I'm ready to say hey other developers on the team is this okay?
Can we merge this and start to do the process to test it and get to production? At that point, I would run some tests and some validation steps, maybe some linting checks if you don't have that built into kind of how you code. Um, from there, once someone's reviewed the code and approves it that it meets all the policies, looks good, that's when we do our merge to dev. So that approval process that kicks off the merge should run the unit test, run the deployment to a workspace, and run some integration test once it's deployed into the workspace.
These integration tests aren't necessarily everything that will run in production unless you really need to do it this early, but it's a pretty good indication that everything's everything's good with this newest code. So at some point, maybe you get a couple of those um those features merged into dev before you go to main. Maybe you do it pretty frequently, but we take what's in dev and we'd go to main. Very lightweight uh review process from the code standpoint because it's really just saying are all these commits that have been through the process already.
If so, we're good. The really important piece is timing for data projects. I found that the idea that we're just going to release our data pipelines anytime without clear communication is not realistic with software. I think you can maybe that's not how they do it, but I feel like you can kind of get away with it in software development with data data projects. I don't think you can. And so there's some a bit bit of all right, are we ready to release a few things?
Yeah, good. You create that PR domain and that's going to kick off a staging deploy. And I'll I'll kind of talk through that more as we look at it. And then once we do that merge to main, we're saying let's go to production. Optional approval steps beyond the code approval, the PR approval if you need. Um, basically for staging, I'm running everything I can and getting as much confidence as I possibly can. And then when I go to production, I don't need to run unit test and integration tests.
All this code in dev has been tested together as long as I didn't start cherrypicking different things in between. Um I don't need to run all the unit tests. You can if they're quick. I don't mind, but you don't have to. We really need to get that thing deployed and run some sort of system check. Unfortunately, I feel bad. I don't have like a great example of a system check, but I'll say real quick when I had the, you know, Kafka as a source and some relational database as like a configuration lookup place to help with metadata driven pipelines, I would do some connections to those tools and make sure that we didn't just break something about how our clients that do that connectivity work.
And if you're thinking that would never break, good for you. We broke it a lot. And so we needed to test that. And to be honest, that's how excuse me, that's how we drive a lot of this is what do we find breaking that we're not catching in our unit integration test. Um, go build some unit integration test if we can. If not, then put that in a system test. So hang with me. I'm going to switch over to somewhat live demo and we are going to start with here.

15:29 Live demo: Git to PR

We'll start with this. I won't spend much time on the development process. Let's see if I lost Okay, I think that's about as zoomed in as I can get with VS Code for managing this. So, I have a branch called Data AI Summit Demo 2025 with a super simple change. I've already pushed it up and I'm about ready to start a pull request. Before I do this next step, we're going to look at this is what we didn't have time for this morning.
Before I do this next step, I would um probably go and take a look at the resources explorer, see what I have. Come on. Um and in that resources explorer, I could run some specific testing myself. Do the deploy. Make sure I didn't break anything. Run some testing myself. This is all going to deploy in dev mode. It'll prefix it with my name. I'll have my own version of it. And um if we start to deploy without the prefix and without our own version, we'll start to have some challenges.
So that's very common, kind of a different topic than my focus for today. Just wanted to call out that's step one. I feel good about it. I've run my unit test. This is here is the bundle explorer in case um where I'm moving around. So we've got play buttons. If we go down to DT, by the way, we've got the ability to run specific steps in a DT pipeline rather than the full thing. run the full thing or run DT validate. We can do a little bit of um coding and checking our DT from our uh ID as well.
So remember what I'm saying is I already pushed this branch up. I'm on the pull request screen in GitHub. Let me see. We're going to try a little bit more zoomed in and make sure everyone can see. So I've got some sort of title, not the important part today. You'll do this however you feel is appropriate. And I'm going to create a PR to dev. As soon as I create that thing, I'm kicking off some sort of CI action. In this case, it is a GitHub action.
And I will see it running right here. Okay. So, this is that first pull request to dev. So, I am just doing my unit test and validation right now. I'm not going to deploy the bundle. I'm not really ready to do that. I've probably already tested the important stuff as an individual developer. So, we do the setup. Um, eventually we'll do bundle run validates the important thing here. So, that bundle data bricks bundle validate.
Sorry about that. Datab bricks bundle validate command is a command line thing. You could have done it from your local machine. All the permissions you need for the environment. and we do that. Make sure that our YAML is passing the checks of being valid. That does not mean it's going to deploy fine. Um I wish it did. I want it to. That's not what it means. It means your YAML is valid. You don't have mistakes in it. Okay.
And then that's pretty much it. We return that as success. And that means we're allowed to merge if the code looks good. Go through the normal process. Um I would review all of these changes. All right. This guy wants to add I don't know why he's doing a display in production, but he's gonna add a limit. Cool. Works for me. Approve this thing. Uh oh, it's working up there. Okay, cool. My screen is freaking out, so we might be done with the live part of this demo.
One second. Okay, this is really just to show. So if you're new to GitHub, just type lgtm and let people figure out that means looks good to me. Submit review. It's easier to me that's easier than pluck, which confuses me a little bit. Um, all right. Squash and merge in case I have a bunch of history that I don't care about. Click the button and eventually it's going to go. We'll see if my buddy in Europe approves this one also like he did the in the middle of the night for my other test.
But it doesn't matter. We're gonna move without any more approvals. What we are going to check though is what's happening in this. I'm gonna call this my test deployment. Okay. What I I typically do is I've got this like nonpro development workspace. Not as clean as staging, not as locked down as staging, but some kind of workspace where we're doing a bunch of work. I could deploy to that same workspace as a service principle, a very controlled area with permissions around it enough to feel comfortable and make sure that this code works and things are going smoothly.
All right, we're going to jump straight to the actions because I clicked the wrong button. I'm not sure what I did. So, now that I'm doing this test deployment, I did the validation. Same thing I just saw. Now it's running deploy could have you could have more to it but what I've got is I've got the datab bricks um CLI running a bundle deploy with auto approve and the most important piece can I zoom in this part kind of the most important piece is I am using a different target remember that one of those early slides I showed you there's these targets of dev staging or test and production I'm using a different target because this is not a dev deployment and I need to isolate it in that way that can drive a lot of things which is kind of the subject of the last half of this session we'll get to shortly for integration test I'm running a notebook um with a set of steps around it I'm running this as a data bricks workflow I'll show you an image of that and just make sure we kind of follow what what I use there all right and then you can probably picture uh I could then do a poll request from dev to main as I'm doing that it would kick off a staging run.

20:40 Automated deployments

And that's going to look like this. It looks very similar at the high level. Um the main difference, I actually didn't put it in the demo, but the main difference is I run all the steps together. I don't have this like I go and just deploy the whole thing. We're ready to go to. We better be ready to deploy and test it. Uh I also would probably have some sort of additional system test that's appropriate with the quality of data we have in that's looking similar to production now.
Whereas in my test and dev environments, my data is probably a bit smaller, a little bit more messy. My developers might have enough control that we can't trust everything there. Uh system test isn't quite as useful there. And then finally, when I deploy to production, you can do it a lot of ways. You could do all the steps if you really want to, but I would have confidence in saying, let's go ahead and do the validation.

22:27 Advanced DABS features

But for the validation, it's probably hard for some of you to see, but for the validation, I still did a bundle validate. Why not? super quick. I didn't bother with all the unit tests because I just tested it twice really along this flow. Uh and then on the deploy step that looks similar. Instead of running an integration test, I run I called it the same thing, but it's actually running a file called flight system test prod.
Okay. Um in case it's not clear, let's see if this will remember me. All right, we we'll skip that part right now. Um, and essentially that's going to run on whatever workspace your target points to and so each of those was set differently. Okay. All right. I think that's good for now. I might come back and show just a little bit more of the GitHub action code. And then in the resources, I believe I have the link to a public facing version of the scripts that I'm using. moving on to a little bit of explanation about tests.

23:42 Automated testing

So, there's basically three types of tests that I've been using this whole time. Just to clarify in case you've been wondering what I mean with unit test, we're really talking about usually we're talking about Python functions or Scala functions, like very small bits of code that do a single thing, very focused. Sometimes I can test these without Spark being involved at all. Sometimes I do need to create a data a small small data frame and do some checks.
Integration is more like a step in a pipeline like loading a table or maybe it's multiple steps together like loading our sales data which is not hundreds of tables but you know a good group of them. And then the end toend test is where you're kind of making sure does this look like everything's still working now that I'm in production. So, when it comes to automating tests, while we're developing, we're running these unit tests.
Did I say that a couple times already? I hope I did. While we're developing, we can run a single unit test that impacts the code I'm changing right now. Have quick feedback that I didn't break everything before I go through this whole process we just saw. All right. Assuming you're doing that, then once I'm doing the um the merging, I want to build and uh once I complete merges, I want to build and test automatically.
We saw that in action. And then we have additional end to end tests when appropriate. So here's a glimpse. This is like a whole session in itself, but here's a glimpse at what I like for using pi test to do pi spark unit testing. Um create a small data frame, create an expected result, apply your transformations, and then assert dataf frame equal, which is part of the pi spark test utils library. I think I have the import on one of these other slides.

25:03 DABs workspace demo

And so you kind of have a test per function, maybe multiple tests per function depending on how um serious you get about full coverage. That was unit test. Now we're in integration test. So special setup task. Run some sort of piece of code that's under test. In this case, I call it notebook job. It's a single notebook that does a bit of work. Pulling together a bunch of libraries and stuff to do it. And then we run our validation step to make sure whatever I just did worked.
Okay, so the quick view is that we have our setup step is making sure I've got a fresh table and in this case I go ahead and create the schema too in case I need it in a very special spot. This is kind of your indicator by the way that these things are these this code is going to be parameterized to go to a different catalog possibly a different schema but definitely well catalog is kind of the main point if you have UC enabled use different cataloges for dev staging production.
When you're in dev, you tend to use schemas as well. So that's why I usually have that in there. All right, the code being tested. Bunch of ways you could do this. I think I switched it to be a notebook task, so it speeds up a little bit, but you could just run a full job. If you've got this workflow that does a small enough set of work to make an integration test around it, just kick off that job with the run job task and data rich workflows.
I'll note that we have a for each task that you could use to loop through a bunch of different things that you want to test together. That might be more it sounds a little bit big. That sounds more like a system test, but you could do something like that. And then when we talk about parameterization, this is the dbutils widgets um for parameterizing. If you're writing this in notebooks, if you're using SQL, there's a few ways to do parameterization.
The validation step looks like this. um it really came out quite bad the uh pispark testing utils at the very very top is where I got that assert data frame equal if using those capabilities that are built into spark aren't enough then go check out the cheesebust library uh and that's got it's kind of done this for a little bit longer and has some more options for you okay all right so what about DLT uh what about SQL SQL I would do this at same way I did my notebooks run the SQL statements in the with some assertion uh setup and validation at the end.
With DLT, we could do something like what I did for my regular notebooks or we could use this uh superpower called expectations with within DT set up a bunch of expectations. And in this case, I would say for automated deployments, expectations that will fail if they're they're uh not met properly, let them fail if my code's not matching what it should. And that will be my cue that I should stop this deployment and I'll get back this big red failed message.
You can have a pipeline set up just for integration test which includes this and you can leave it out of your production runs because you don't you may if you don't want to fail every time those conditions get hit. So you could have two different pipelines. One of them includes a bunch of extra tests just for this automated deployment. Uh and there you go. That's kind of the image of two notebooks um in my integration test one.
And just imagine one of those isn't there if it's not an integration test. All right, so let's get to advanced dabs features that have been utilized. Most of them have been utilized in this demo without me highlighting it yet. Um there's a bunch here. Sorry if I don't cover what you want to see. There's a library called data brick at github data bricks bundle examples that has like the knowledgebased section with a bunch of small examples like data bricks apps and things like that and then the resource I linked to which is data bricks solution data bricks dab examples has more of the complete project style that I'm using here all right so first off let me highlight that when we are doing our deploy our data bricks bundle deploy we can p we can override variables for the target we're deploying to.

28:52 Conclusion and resources

And that gives us a lot of power to dynamically set certain things at deploy time. When we are going to run that DT pipeline or run that notebook, we could pass in parameters to override that catalog and uh schema parameter. If you set those up with the DB utils like I showed, you can override those at runtime as well. So be aware of that. could be pretty useful depending on how many different variations of the same deploy you need to do.
Complex variables. These have been around a little bit, so I hope those that are using asset bundles have already started using these. I use these a lot. Um, I probably use them a lot more if I had like a true big production environment I manage. And so this is an example of setting up a full cluster, assuming we're using classic cluster compute. set up the full configuration as a variable and then I can fairly easily have different ones that I use.
Uh the next go the next one will be our uh lookups. And so for lookups, we're going to actually it's the situation where I've got a warehouse or a job policy and the data bricks APIs want the ID for that. If I have a staging and production workspace that are separate, the ID is different. I can put that ID as a variable and just use it. That's fine. Sometimes we know that another team's going to redeploy this thing. The name's consistent.
The ID is not. That's where lookups come in. And so this example is as a variable, you can just add this lookup syntax and give it a name. And then in this case, I would have a small job uh policy across every single workspace. And then uh another thing that comes up is I need to exclude certain folders. And in order to do that, we can use the sync option. And you can see the exclude block below saying, hey, these folders have files that maybe are a lot of data.
They don't need to go to the workspace. Perhaps it's taking a while because I have like CSVs and stuff in there, too. You can exclude those with sync. The other thing that comes up is we have like shared library code. It's not within my bundle route. bundle root is like the folder that data bricks yaml exists in. If this is maybe a level above it, it's not going to be included by default, but we can use sync and paths to include some shared code.
Or there's other use cases, but that's the one I see all the time. Okay. Uh something that's made it to public preview fairly recently is Python support for dabs. Public preview, right? I don't know where my database. Um Python support for dabs. This is an SDK where I can define these asset bundle resources on the fly. Uh or I can use it along with a bunch of YAML I've already got to uh mutate what's there or generate additional jobs.
So examples would be I've got a folder structure with like a folder for each client, each customer of mine and I want to go through and generate a special job for each customer. The mutator would be I want to add a a tag or a certain cluster policy on every single job. Just force this across the board. I could set up a mutator with Python code using this SDK. And that's going to apply to everything in my deployment. And then these are these are kind of small.
I almost didn't put them in there, but just to call it out because they're powerful. I use these mostly for development. When it comes to staging production, I like to deploy my unit catal catalog objects like my cataloges, my schemas where all my tables are going. I like to do it with like Terraform or my infrastructure as code for staging or production. Well, I don't like to do it. I like someone else to do it with Terraform.
I don't like to do Terraform. So, when I'm doing dev, I actually dynamically generate it using asset bundles um using the volume or the schema. And you can also do this with all-purpose clusters. And then sort of finally, this one, it doesn't have much to do with getting in production, but I think it's super important, is dabs in the workspace. So, you can now uh go into the workspace. If you have a project folder with uh databicks. yml, open that up, and if you've clicked the right settings, you get this special way of working with that project.
And so, I think with in hand, I can sort of demo that. So, let's give this a shot. Okay, so I have my demo flights advanced. This is the same folder I was using from GitHub actions. And I'm open to that folder. And it automatically picked up that I have a target that matches the workspace I'm in. And so because it matches this workspace, not another one. I'm not deploying across workspaces. This is for building this all out within within the workspace I'm in.
Um I can see this resource explorer a little bit like VS Code, right? So we've got some options here. I can do runs of what's been deployed. I can redeploy it. I can even kind of get started with uh u well I can destroy everything. So be careful. I think it gives you a prompt still. So just be aware. I can do some variable overrides. But the other thing is I can go and create additional job definitions and things from here.
Okay. So to deploy I basically use this rocket button. Hopefully you can see that. I'm not sure if we'll follow this whole cycle through. Um but what that's going to do is it's going to take a look at my dev target. It's going to find the complex variable for the cluster, the overrides I have for variables for my dev target. It's going to generate a schema for me if it doesn't already exist prefixed with my username.
And let me talk about like where that's happening in the bundle code instead of showing you the UI say yes it works. It worked yesterday. It should still work. Let's look at the code which is probably what you're more interested in. Uh oh, another advanced thing I forgot to put a slide for. If you're deploying wheels or if you're deploying wheels, deploying artifacts and packaging up your Python libraries, you can set dynamic version.
It'll automatically add automatically add like this timestamp versioning, this sequential versioning to it for you. So you don't have to do that yourself. All right. Uh here's a bunch of variables. And this might literally be what was on the slide, which I know isn't the most exciting demo ever, but just to kind of call it out, what I've got is a cluster configuration. I've labeled it complex so I know it's more than just a single string.
And then I'm going to use that same job policy that I did a lookup on. So since I'm deploying to dev, it's going to go to my dev workspace. It's going to look for the small job policy. It's going to return that ID and shove it into the API calls that actually make all this happen. Same idea with warehouse. Um in addition, let me go down. And I think in test I start to override some things. So still in dev this is my uh schema creation.
Like I said I do this for te dev. I don't do it other places because I only want to do this in dev. I shoved it under the dev target definition. Most everything else is shared and I do it in every single place I deploy. This is only for dev. So it's underneath the dev target. It's also how we would go and override certain attributes. All right. coming down to each of the really important things that change across workspace.
In this case, it's really just database and schemas for me for test. When I get to test automated, maybe I need to bump up my node size to something bigger, or maybe I sometimes switch to Azure, so I uncomment and use a different node type because those are different across clouds. Um, and then yeah, those are pretty much the overrides I have set here. And then don't forget when I'm doing my deploy, I could actually override some of these different variables as I go.
Okay, the last thing I will do is come back to my staging job that had everything in it and just do this is kind of my recap here before we wrap up is um in my in my action code I do my setup. I told you I triggered when we open or push to a pull request that has not been merged yet. So that's what this trigger is doing. I come down and I do uh my and I use my environment to pick my secrets from GitHub action. So I've got secrets for staging, secrets for production, secrets for test.
Um I would do all of my Python installs and setup and unit testing as needed. And then when it comes to use the data brick CLI, oh there's two one more thing I need to go back to on the unit test. So when it comes to use the data brick CLI, I'm mapping environment variables that the CLI is ready to work with. They're just built into the CLI. It checks for these. I'm mapping those to the secrets and that's how we're authenticating in this in this way.
The other thing you'll see is when I do my piest test, if you're new to this, this is going to run all the unit test in this particular folder. I'm going to authenticate there as well. The reason I do this is I find it after years of doing this, I find it much easier to use data bricks connect for any of my Spark tests rather than trying to have some local open-source PIS spark version that I'm running these tests against.
Um, it makes it saves me developer time overall and is worth it to me. So, I actually use data bricks connect for all of those unit tests and so I need to authenticate in order to run those from my GitHub action. Okay. Uh, oh, and then finally on the deploy, it's really not that special, I guess, but notice that I set my target on the deploy. And, um, I found that auto approve is good. As long as you are confident in this code, auto approving things like recreating resources that need recreated is good.
Um, be careful with auto approve though. Make sure you know like which things are possible to re redo there. And then finally, here's one version of the integration test with a few helpful tips at the end about how we um make sure it's clean output for us uh here. Okay, so to wrap up, here is my contact info. Um definitely I show some of this on YouTube, so you can check that out. And if you follow me on LinkedIn, uh you'll see more messages as I release new things.
Like a lot of this advanced stuff isn't out there yet. And then uh you should have access to be able to get access to this deck. And so I've linked to code examples that I referenced. The data bricks dab examples is the most like what I showed. And the bundle examples has a lot of really good stuff for you as well. Uh thank you. Enjoy the rest of your summit.

Learn more about the Databricks Data and AI platform.