Get Data Into Databricks - Asset Bundles

Summary

• Unified CI/CD approach: Databricks Asset Bundles (DABs) provide a standard, native way to deploy Databricks artifacts (jobs, models) and assets through a simple YAML configuration.
• Multi-workspace deployment: DABs use Terraform for state tracking and the Databricks CLI, enabling teams to deploy code consistently from feature branches to production.
• Simplicity advantage: Unlike complex alternatives (Terraform, dbx), DABs offer formal native support and accessibility for ML engineers and data teams without DevOps expertise.

Get Data Into Databricks - Asset Bundles

Try Databricks today: https://dbricks.co/3EAWLK6.
This video introduces Databricks Asset Bundles (DABs) as a solution to simplify and standardize CI/CD pipelines on Databricks. It highlights the challenges of existing tools, which were often complex, incomplete, or lacked native support. The video explains how DABs address these issues by providing a unified approach to managing Databricks artifacts (e.g., jobs, ML models) and assets (e.g., Python files, notebooks) through a configuration file that integrates seamlessly into CI/CD workflows.
Key points include:

• Challenges before DABs: Existing methods like Terraform, dbx, or REST APIs were either too complex, lacked formal support, or were error-prone.
• What is a DAB?: A collection of co-versioned artifacts and assets that leverages Terraform for state tracking and can be deployed across multiple workspaces using the Databricks CLI.
• Benefits: Simplifies deployment, improves consistency, and supports native integration for CI/CD pipelines.

The video concludes by emphasizing how DABs empower teams to efficiently deploy data products on Databricks.

Chapters

• 0:00 Introduction to dabs
• 2:58 What are bundles
• 3:49 Configuration file structure
• 5:08 Bundle CLI commands
• 7:08 State management and usage
• 8:29 CICD integration
• 9:33 Demo and resources

FAQs

What problem do Databricks Asset Bundles solve?

Before DABs, existing CI/CD solutions were either too complex (Terraform), lacked formal support (dbx), or were error-prone (REST APIs). DABs provide a unified, natively supported approach to deploying Databricks artifacts.

How do DABs support deployment across multiple workspaces?

DABs allow you to co-version your deployment configuration with your code. You can move through feature branches, dev branches, and production branches while deploying to corresponding workspaces (dev, staging, production) consistently.

What does a DAB configuration file look like?

DABs use YAML format for configuration, making them accessible and easy to understand. The configuration specifies your Databricks artifacts and assets in a readable format integrated into CI/CD workflows.

How does Terraform support fit into DABs?

DABs wrap around Terraform for state tracking and management, providing an easier interface on top of infrastructure-as-code while maintaining the benefits of state file support.

Full transcript

0:00 Introduction to dabs

Hey everybody I'm a Solutions architect within the DMB or digital natives business and today we're here to talk about data bricks asset bundles it's a standard unified approach to deploying data products and ml products for datab bricks DMB customers so what we hope to do with dabs is to bring Simplicity and support for cicd processes on data bricks so before building cicd pipelines could be complex incomplete and inconsistent tools were either not natively supported so we're looking at kind of dbx on the Spectrum here dbx was a datab bricks Labs product and it wasn't natively supported it could be too complex if we don't have any devops Engineers on the team so terraform comes to mind and infrastructure is code tool and sometimes is missing capabilities so as you can see on this kind of Matrix we have where dabs fits it's got fully native support um actually wraps around terraform so it's an easier interface for machine learning engineer Ing and data engineering teams beforehand getting to production was really challenging of course we could use IAC such as the data bricks terraform provider it's powerful in general however um if we didn't have devops rocket scientists or Engineers on the team it could be tough so if your persona was more of a machine learning engineering team getting ml flow based models on deploying and running those tasks could have been a little harder and prior to that we also had dbx which sat above our rest apis it was more accessible than terraform but there was no formal support and we've actually kind of sunsetted it uh it's now not not being actively maintained and we will link the migration plan uh underneath and then we also have datab bricks rest apis which are lowlevel scriptable but kind of more do-it-yourself so they're error prone and brittle and it's generally not recommended doesn't have any state file type of support so we thought about how can we simplify this process we wanted to write the code once and deploy it to multiple workspaces easily so if you think about this if you put your deployment process right next to your code you can go across maybe your feature Branch to your Dev Branch to your main branch or maybe a release Branch even and go across multiple workspaces from your de Dev workspace to your staging workspace to your production workspace and that's kind of what we're talking about here Co version your code with simple configuration files and that's why they're in yl format we wanted toine dat bricks resources using existing rest API parameters so we don't have to learn new lingo around the arguments and then we also can specify environment based overrides and variables so if you think about it in terms of a GitHub type environment if you have a GitHub action you can also specify environment secrets that can pass along a specific Target and this ensures user isolation you might be wondering also what exactly is a datab bricks asset bundle or dab it's a collection of datab bricks artifacts such as Jobs machine learning ml flow models DLT pipelines and clusters also assets such as python files or even Python scripts for your datab bricks app notebooks SQL queries and also dashboards dabs does wrap around terraform which enables State tracking and we'll kind of cover this in detail in a later slide um it does generate a terraform dotf State and these dabs or bundles are represented in a configuration file and like I said can be covert version in the same repository as the assets and artifacts reference in the bundle thus you can kind of use local references and paths using the data brick CLI these bundles can then be materialized across multiple workspaces enabling customers to integrate these into the CD workflows so GitHub actions here so we wanted to First cover what a datab bricks yaml or the configuration setting at the very top level looks like and then we'll kind of demo this so here we have the bundle definition which is the name it also uses the include argument to show which resources so if there are those who are familiar with terraform you can also use resources here but you could divy this up any which way you like maybe apps in one folder models in another folder um workflows in another or workflows or jobs in another folder that kind of thing you want to Define your workspace and in this case we also Define the resource right next to it however generally we don't recommend for this we can split them up in like a resources folder and again we can use local relative paths to code as you can see this notebook task and its path is relative to where the code sits and then we can also deploy targets so this is kind of what we're talking about in terms of going multi workspace deployments maybe across branches for your code so in this case if you want to go from maybe a feature Branch or Dev Branch you would develop using the development mode and go to the dev Target next is like production Target or a staging Target and you can kind of Define these and you have per Target overrides so for example here you can have multiple workspaces so before we go to the CLI commands we'll pivot to our demo or data break Amo this is an examle example datab brakes yaml configuration as you can see here very akin to terraform we can also def find variables this is the bundle name I've defined and then we can also include the resources again you can split this however which way you want it's very configurable in this example demo we only have the development workspace but as you can see you can Define development staging and production very easily here inside of these resources I'd like to cover just a few of them here we actually have a data bricks app which actually interacts with our chatbot which will be deployed from and as you can see you can add permissions as well this job this job is a notebook task single job that basically passes in the un uh Unity catalog catalog Unity catalog schema and the model name and deploys a model serving endpoint with which the app will interact with these are the two um bundles which we will deploy and run so switching back to our CLI commands here are the bundle CLI commands we can validate or check for issues with the bundle configur uration so this is very akin to a terraform plan we just want to make sure our configuration can be run before we deploy and run next is deployment we synchronize the code to the workspace and materialize the F resources so this is where that terraform tfd really comes into play we see what resources have drifted how we can adapt for them and then apply our resources next we can run or interact with the deployed resources so for example in our job we deployed the job first and that artifact gets created within the workspace but next we need to click run so to speak and enable another job run and covering into definition of the bundle State bundles do use the datab bricks workspace for file and State Storage it defaults to a secret bundle file as you can see here a bundle folder and then follows this naming convention um if it doesn't exist it will be a full file sync and create these underlying State resources and if it does exist it's an incremental file sync very akin to terraform and creates updates deletes resources and the default can be modified to incorporate additional Dimensions so you might be asking where can we use bundles bundles can be used to deploy and run your project tweak configs and deploy and test changes as you can see here we use the CLI command data bricks bundle deploy Target development now I will give the caveat that before you start development and start using a datab brick yaml you can refer refer to my peer van Noise video on how to initialize a datab breaks asset bundle template it would be datab breaks bundle a nit we won't cover that in this video but please refer to that one here you would deploy to the Target of the development workspace or the development configuration settings and then you would also run a DT pipeline to refresh all and using that same Target and how could this deploy to multiple workspaces you're wondering so as you can see here you can use it as part of a cicd process so in this case maybe you have a Dev branch which automatically is coinciding with your Dev workspace but from a pull request you validate first upon creation of that poll request if that validation passes from a databas bundle then you deploy when you merge the poll request and go to your main branch your main branch would then correlate with your staging workspace or your staging Target as you can see here data brakes bundle deploy Target staging run that Pipeline and then similarly when you want to go to your release Branch you could deploy it to production this is all executed on cic servers in this case G of actions and it's triggered by a CI release by pipelines again ideally from an authentication standpoint this demo will show using a personal access token which may be fine for development reasons however ideally you would run as a service principle and use a service principle secret for M to n authentication so we'll go back to I'd like to show exactly how we coordinate this so as you can see here when we open a poll request on the branch Dev we deploy using a datab braks bundle validate we tell it the environment that we're Target the Target environment we're going to which is Dev we use a data brakes token in this case a personal authentication token but also commented out you can use a client ID and client secret which are akin to the service principle once this passes then we can hit the data breakes bundle deploy action here you can deploy the same exact target in case case Dev and then run a job so in this case this is my rag chatbot notebook task job and then following that depending on that job we also run the app which interacts with the workspace I'd love to show that so this is what it looks like generally from a GitHub actions perspective we deploy the bundles first so all the artifacts kind of sync with the workspace and then we run our jobs associated with the deployed artifacts so what does this look like within our UI I'd love to show you so in this case you might see hey this is a moniker for a datab bricks asset bundle this is my Target and this is the user who deployed it this is exactly what I mentioned in terms of the job name as you can see it's connected to datab Bricks asset bundles you can disconnect it from The Source but when you deploy again it will overwrite next you see we've gone ahead and deployed an app in this uh notebook job we deployed a model serving endpoint as you can see here using the name of this model serving endpoint we're able to interact with this in the data braks act this this app was this third step in the Run GitHub action pipeline when you run the app you can then launch the app and chat with your existing model endpoint I hope this was helpful um please refer to resources uh that we have additional so from Dab's documentation bundles examples and substitutions and of course feel free to reach out if there's any further questions

Learn more about the Databricks Data and AI platform.