Scaling Data Pipelines with DABS: MGM's CI/CD Framework for Enterprise

Summary

MGM Resorts consolidated 10 legacy data platforms into a unified Databricks environment on Azure, onboarding 85 data sources across five business domains with Unity Catalog governance and a standardized Databricks Asset Bundles framework.
By implementing Databricks Asset Bundles with GitHub Actions using a mono-repository pattern and tag-based release versioning, MGM reduced deployment time by 67 percent and cut deployment code by 75 percent.
The framework enables 50-plus production pipelines with deterministic behavior, full audit trails, and approval gates, giving domain teams independent deployability while the central team enforces consistent structure across all pipelines.

Scaling Data Pipelines with DABS: MGM's CI/CD Framework for Enterprise

Managing multiple data pipelines consistently across business domains at scale presents extraordinary challenges. MGM Resorts consolidated 10 legacy data platforms into a unified Databricks environment, on-boarding 85 data sources across five business domains. Without standardization, pipeline deployment involved manual effort, inconsistent governance, and version control fragmentation. The company needed a framework ensuring deterministic deployments, audit trails, and multi-environment consistency from development through production.
Learn how MGM implemented Databricks Asset Bundles with GitHub Actions to achieve infrastructure as code for data. Using a mono-repository pattern with centralized configuration, domain-specific bundles, and pre-built deployment scripts, MGM reduced deployment time by 67 percent and cut deployment code by 75 percent. The framework enforces consistent structure across all pipelines, enables independent deployability, and integrates serverless compute with table-update triggers. Results: 50-plus production pipelines, deterministic behavior, full audit trails, and approval gates that maintain governance while empowering team velocity.
🤝

Chapters

FAQs

What are Databricks Asset Bundles and why did MGM Resorts use them?

Databricks Asset Bundles are an infrastructure-as-code framework that allows data engineering teams to define, version, and deploy Databricks resources consistently across environments. MGM adopted them to replace manual, inconsistent pipeline deployment that lacked audit trails and version control as they consolidated 10 legacy platforms into a unified Databricks environment.

How did MGM structure their Databricks Asset Bundle framework?

MGM used a mono-repository pattern with centralized configuration, domain-specific bundles, and pre-built deployment scripts, organizing codebases into configuration, source, transform, and driver layers. Bundles are deployed through a tag-based release process with GitHub Actions, and the team transitioned from Git flow to a trunk-based development methodology.

What performance improvements did MGM achieve with Databricks Asset Bundles?

MGM reduced deployment time by 67 percent and cut deployment code by 75 percent after implementing the Databricks Asset Bundles framework. The standardized approach enabled over 50 production pipelines with deterministic behavior and full audit trails across five business domains.

How does MGM maintain governance and compliance in data pipeline deployments?

MGM's framework enforces governance through approval gates in the GitHub Actions CI/CD workflow, ensuring changes are reviewed before reaching production. Unity Catalog governs all data assets flowing through the pipelines, and the tag-based release model provides full version history and reproducibility required for compliance across MGM Resorts' multi-domain entertainment operations.

Full transcript

[00:08] Uh so, thanks for joining us today. I'm Jennifer Young. I'm the VP of data at MGM Resorts. Um and I am actually co-presenting with Ajay. He'll be up here on the stage in just a minute. Um but we wanted to walk today through our implementation of dabs
[00:24] at MGM Resorts. So, uh today we're going to cover So, we'll go through the what we've been doing at MGM so that you guys can just kind of get an idea of of what we're building. Um we'll go through the challenges that we were facing and then how um we actually implemented uh our
[00:40] the dab design. Um we'll also go through our CD a CICD pipelines. Um and uh Ajay will actually do a demo of the solution as well. So, to get started. Uh so, this is um a very very high-level view of what we're
[00:56] doing at MGM. So, um we've been on a journey at MGM of trying to combine uh multiple data platforms. Uh so, we have about 10 different legacy platforms um at MGM today, which of course means there's a lot of data sprawl. We have a lot of duplicate information and we
[01:11] actually have no uh visibility into lineage or anything else. So, um it's been quite the journey. We've been moving over to Azure Data Bricks for the last year and a half. Um we're about uh 50% done and uh we've already retired uh three of our legacy systems and uh they
[01:27] are all within our um what we're calling our Enterprise Data Hub or EDH is what everybody calls it. Um so, as you can see, this is really more of just our it's a medallion architecture. We're on on boarding about uh I think we've already on boarded about 85 sources um
[01:43] systems um into our platform. Uh we still have quite a few to go, probably about another 50. Um and then we are uh pushing it through our pipelines and of course we are using Unity Catalog to govern all of that. Um that's been uh I know this is not a
[01:59] unity catalog session but that has been something that's been incredibly useful to us as we have started to build out this platform. So um So the challenge that we've been having right as we are building out this platform how do we really
[02:15] to deliver multiple pipelines actually significant amount of data pipelines consistently across right now we're looking at five business domains but if you kind of think through when you go and I don't know if I assume everybody's been to Las Vegas yes?
[02:30] Okay if you remember going to Las Vegas if it was a memorable trip you know the the one thing that we're really trying to focus on is the customer experience and so when we talk about the customer experience it's really about you walk into a casino and you actually interact with so many
[02:45] different lines of business right so everything from when you go to check in at the front desk you meet the valet you go to a spa you go to a show just imagine right the entire kind of your your journey and that's what's why we're building our data platform
[03:01] because it's really important to understand our customers journey so that we could then use that information to provide much better experience for our guests. So that's really the purpose of our enterprise data hub. So when you think about having to tie
[03:17] all of those data sources together so I mentioned we're about 85 right now but really we had a lot of when we start talking about building our data pipelines it was really difficult right we're talking about a lot of inconsistency nobody was really doing anything the
[03:33] same way it took a lot of time to actually deliver lots of manual deployments which is of course prone to errors and then we had a lot of gaps on the governance right so we all have multiple environments there's no gates
[03:48] and it was really difficult to manage this type of environment and when you talk about having to scale, right? Um at the level that we need to scale, um this is a challenge that we were facing. And so, um Ajay and the engineering team uh decided to implement DABS. Uh so,
[04:04] declarative automation bundles. Uh you might have heard it before it was called uh asset bundles. It's now called automation bundles. Um but ultimately, what this has really led to is for us to actually deploy infrastructure as code. So, now we have everything is consistent. Um we have a
[04:21] multi-environment native implementation where we could actually move everything very seamlessly from our dev uh QA and a prod environments. Um and then each uh pipeline now is independently deployable. So, this has made a a huge
[04:36] difference um in the way that we operate. Um we have also integrated this with GitHub actions. So, now we're able to trigger our our jobs. Um and Ajay's actually going to go through all of that in detail, so I am not going to uh uh probably say anymore, cuz he's
[04:52] going to show all of this in action. So, um I'm going to invite Ajay up here. Uh and um so, Ajay is um our principal data engineer at MGM. Uh he is kind of the mastermind of how all of this has been implemented. Uh so, I'm leaving you in
[05:07] good hands. Thank you. Can everyone hear me okay? Yes. Okay. So, now we all know why we needed to change pretty much. So, uh I'm going
[05:22] I'm going to get started with explaining the architecture and the framework design and how this kind of solved the problem that Jennifer just described. And how we turned that into a framework, which is building and running 50-plus pipelines every single day.
[05:44] In a nutshell, uh this slide pretty much tells the whole story. Before DABS, we were doing a lot of things. Each developer were doing their own things. Some would create notebooks, some would directly run it in the Databricks workspace itself. Yeah, sometimes it could be prod, too. And some would have different flavors of
[06:02] development, right? You know, it could have It could be PySpark, Scala. They produce wheel packages, jars. Sometimes it's plain SQL. We have a lot of combinations there. And it's all glued through uh maybe job CPI, shell scripting, using the personal
[06:18] access token shared somewhere, right? So, now those things are being used to manually deploy into Databricks workspaces, and the results are pretty much, you know, inconsistent pipelines. There is no governance, and there is a lot of manual toil around it, right? So, there's no
[06:35] CI/CD, no audit trails, no hard gates. Everything is done in silos there. Now, if you move on to the right-hand side, that's the after DABS approach. With this, we were able to standardize things. It's about standardizing how people would start their day-to-day
[06:51] activities and start to intake features and start to develop things and promote it to next environments, right? So, we have redefined that in a way that each developer would go ahead and create a feature branch out of main. So, we'll get to all the low-level details later, but this is just to provide a high-level
[07:06] picture of what what it looks like. So, each developer would end up creating a feature branch, and once he is done with the changes that he's supposed to be doing as per the requirements, they would use the existing deploy tooling that we pre-built for them to use them and test it in the lower environments from their local machine.
[07:22] Once it is done, they would go ahead and raise a PR. Once a PR is raised, GitHub Actions would automatically run a CI. It has the intelligence We built that intelligence to identify that, you know, identify where the change exactly is. It could be in a bundle, or it could be
[07:38] at a higher level, that is, at the domain. We'll get to that part. But ultimately, it has the intelligence built in to identify where the change is, and it would deploy the CI pipeline, which would on a high level does three important things. Running the unit test cases to ensure that functionality-wise everything is running fine. And also running the
[07:55] deploy validate command to ensure that those who, you know, dabs, you all know that. Databricks.yaml is the heart of it. If there are any issues in there, it's going to fail. So, the bundle validate is kind of, you know, does that
[08:11] validation. So, pretty much we kind of shifted all these things to left, so we don't have to discover all these things at a later and with the time of deployment. So, all these are taken care at the CI step itself. When a PR is raised, all these things happen, and if it is failed, it's blocked right there.
[08:27] It wouldn't go to the next step. And after that, if PR, if everything is good there, then pretty much the domain lead or tech lead or peer review will happen, and that is where PR will get a review, approve, and merge, right? Once it is merged, that is where
[08:42] it does a lot of steps, right? All of those things that pre-built into a declarative scripts. Whatever you're seeing in the blue box on the right-hand side here. So, pretty much all these things are baked into a pre-built deployment tooling, so that
[08:57] engineers does not have to invoke all of those things, you know, in a repeated way. And it gets to it gets deployed to dev automatically because we do not have any any gate set up on the dev. And for UAT, it needs a human approval, wherein the domain lead goes in and
[09:13] approves it and then takes it to the next step. And for prod, prod is kind of isolated from this space. Prod is kind of triggered from a tag-driven approach. We'll get to the We'll also see that in the demo, so we'll get to that part later, but prod is also governed through a tag-based
[09:30] approach. And underneath environment, it's all Databricks, serverless table triggers, validations, and Unity Catalog governance. All of those things are baked into the CI/CD process itself.
[09:47] So, pretty much with that, we went from chaos to a fully governed end-to-end pipelines, and we did all of that through existing Databricks native tooling, that is Databricks asset bundles, and GitHub actions on top of that. Now, I'm going to talk about the repo structure a bit here.
[10:02] We use a mono repo model, wherein we have one repo, and then each domain, like you see there, loyalty, entertainment, food beverage, each domain will get its own folder, and the things that could be shared
[10:17] amongst them are kind of embedded into the common module. So, they those could be any sort of utilities which are used for reading the inbound tables, writing the outbound tables, what not. All those things are baked into the common module, so that they can they all can use that and then render them accordingly.
[10:35] And within each of these domains, there are three standard things. One is the deployment manifest, which talks about what is getting deployed where, right? Uh we'll get to the details of that, and then it also has the high-level bundle
[10:51] common config, uh wherein all the common things like workspace host, service principal, all of those those things are kind of brought in one level up, so that everything within that domain can share. There could be a lot of bundles which which could be sharing all those, so we don't have to duplicate all that.
[11:08] And the scripts, these are the things that I was talking about, the pre-built scripts, where it's being used by engineers, GitHub actions, all of it's it's the same process which is which is being used everywhere, in every step of the way.
[11:25] So, the key principle here is each bundle is an independently deployable unit, and it can generate one or more gold assets. With that, I'll dive into uh the bundle anatomy. Now, what do we have in each of these
[11:41] bundles? Now, we have talked enough about domain. Now, what do we have in each of these bundles? Each bundle follows this exactly the same structure. I'm kind of emphasizing the word exactly because I'll get to that part a little bit later, but there are no exceptions to it. Every bundle has to follow the
[11:56] structure. It has databricks.yaml. If you look at it on the right-hand side, it contains all the job definitions, tasks, whatever the task it could be. It could be SQL task, it could be a wheel package, a jar running a jar with an entry point. It could be any of those
[12:12] things. And it contains the compute details, be it serverless. In this example, it's shown as serverless, but it could be classic, it could be all of those, whatever it supports. And after that, we have the source code that each bundle kind of, you know,
[12:27] looks into. It has three components to it. It has config.yaml, where all the configuration is sitting in. Think of it as anything that you want to read from inbound, anything that you want to write to the target table, right? And be it setting the properties afterwards, be it delta optimization,
[12:43] change data feed, liquid clustering, whatnot. All of those things are baked into the config.yaml. And the transform.py, this has the core business logic, where developer spends time to write the logic. Be it developer, be it AI, that's the focus
[12:58] area where they need to be focusing on to write the actual logic and do not worry about anything else. And coming to the driver, which is entry point, we'll get to that in more detail later, but it is like an entry point and it also uses a base job. That's an abstraction utility that we built out,
[13:15] which kind of does the low-level orchestration. I would get to that part in reality a little bit. And the last two are kind of, you know, supporting scripts, which in which kind of tracks the requirements intake process and also tracks the acceptance criteria to ensure whatever is being
[13:31] built out of this process is meeting the requirements and the acceptance criteria. A little bit ago I talked about the bundle common bundle config, right? Imagine a scenario where you have a 50
[13:48] plus bundles. And you have to update this 50 plus bundles for every change, right? For example, without this config file, let's assume you have to you have these 50 plus bundles, you have to repeat the workspace host URLs. You have to repeat
[14:03] the service principles, variable definitions, environment target definitions. All of these things are repeated for each and every bundle. What if we can just abstract it one level up and then include them in each and every bundle using just one command? That is each bundle include. This is the
[14:19] specific command that we use to kind of, you know, abstract things that are repeated and take it one level up and have the low-level bundles just included. So, tomorrow if we have to make any change, for example, rotate a service principal altogether, for example, just as an example, right? So, we just make the change in one place and
[14:36] it gets affected to all. This slide talks about the framework. How many of you have worked with Databricks apps using the app kit?
[14:55] Okay. So, those uh who do not know about it, you know, it's it's think of it as a framework, right? Where it does everything. It does react on the on the front end, Node.js on the back end, and it's super easy to build an app using it. And you just have to, you know, be specific about your app specific things and be done with it, right? It does It
[15:11] does take care of everything else. We kind of, you know, follow the similar approach. We kind of, you know, wanted to abstract things out from the engineering so that they can only focus on what actually matters, that is the transform logic, and then everything else, right? Be it Be it the common
[15:28] utilities, CICD process, everything else, they don't need to worry about it. Everything else is baked baked into this framework for them. So it has four steps into it. The first one is the configuration wherein you define all the configuration. As I mentioned earlier,
[15:43] you know, it has inbound, outbound configuration, delta optimizations, liquid clustering, and any sort of PKs, target table columns, comments, all these things are baked into config. And the second step, which is the source. The source has the intelligence to kind
[16:00] of understand the config, whatever you put in the config, and translate it into a dictionary, and map it to data frames wherever, you know, you read the input tables, it maps it to a data frame, and applies all the ETL filters. If your requirement is to read a full table to perform some XYZ aggregation,
[16:17] then you would just end up saying that, you know, ETL active uh if you have a type two uh table that you're reading from, you can just put the active filters in there, and it would just go ahead and read it. But if you are reading an incremental data where the change data feed is enabled on the silver side, for example, you can change
[16:33] the read mode to incremental, then it has the intelligence to kind of go ahead and read the incremental data itself. So all of that is baked into the source. Now, transform. Here is an example of transform on the right side. This is This example is super simple.
[16:48] Wherein you have three tables that you're in taking from sources through config, and then you're returning a joint data frame. But in practice, it could be a lot complex than that, right? It could have a number of aggregations, you could invoke in a lot of other functions outside of this. It
[17:04] could be all of that, right? In this example, I've kind of, you know, made it super simple. But the base job here is like an orchestrator, like I mentioned in the earlier earlier slide. What happens is as and when you instantiate it and pass it as a parameter, it expects a data frame to be
[17:19] written. When a data frame is returned, that is where, you know, that data frame is taken. It is caught on the driver side, and it is using the config, whatever you put in config. For example, in the config you might say, "Hey, I want this table to be
[17:35] written in SCD type two. And these are the columns, these are our comments, and I want to enable change data feed on this, liquid clustering on this columns." All all these things you might be defining in config, right? Pretty much it will take all of those things into effect and then handles it at the very end.
[17:51] So, developer doesn't need to worry about any of these things. So, this is a consistent structure we enforced across all the bundles that we are using.
[18:06] Now, I'm going to quickly talk about how we take any code change from our local machine to production, all the way. This is the process that we follow. Before I talk about this, I would just step back a little and talk about
[18:23] Git Flow. How many people here are very familiar or used the Git Flow, traditional Git Flow? Yeah, I see many hands. We started with that. And soon after we realized that the number of domains are increasing, we just have a mono repo, and each many
[18:40] teams are using this mono repo, right? We ended up maintaining a lot of things. There were a lot of PRs, and we have two long-lived branches we had to maintain, right? And sometimes we had to roll back things as well, and then we which created a very messy Git commit history, right?
[18:56] And then we decided, "Okay, maybe this is not scaling well for us." And then we chose trunk-based methodology. With trunk-based, you know, it has few things. It has few pros and cons, but for for for our needs, trunk-based really did wonders for us. And I'm going
[19:11] to give another example as well in the latest slides, but with trunk base each developer will be cutting a feature branch out of main. And then works on the logic using the deployment tooling, whatever the logic the developer has worked on, they'll go ahead and test it against the dev environment. They don't have to worry
[19:29] about anything, they just need to invoke the pre-built scripts for them where all the dabs commands are abstracted. So, they just need to invoke the scripts. And pretty much, you know, use it. In this case, they are using the deploy bundle.sh, which we are going we are going to talk about in a little
[19:44] while, but that's the script that's invoked to deploy to development. And test it iteratively. Once it is working fine, right? Then they go ahead and raise a PR. As soon as they raise a PR, GitHub actions will identify which domain it is and picks up the change and then does three solid things.
[20:01] Identifying which mod which domain it is. Ensuring the unit test cases are passing. And ensuring that the bundle structure is the same. We are saying that, you know, teams, please do follow this bundle structure, but how how do we know if they are actually
[20:17] following or not, right? This is where we kind of, you know, put a put a hard gate with another GitHub actions workflow saying that anytime you detect any change, identify which module it is and see if it is following the same structure that we have. And we'll go to
[20:32] that in I'll show that in the demo, but you know, it kind of tells us it gives a matrix where it says, okay, if something is not meeting the criteria, you know, it is warning, you know, it is successful, everything is displayed in that particular PR workflow. And then, of course, the last but but
[20:48] the important one is the bundle validate command, which will ensure that the variables that are being passed to the databricks.yaml are fine and there are no issues with the YAML files at all. And whoever is reviewing the PR, they will still be reviewing the actual code because everything in here that you're talking
[21:04] about is is deterministic except the transform. The transform.py can be creative because it can be probabilistic even if the AI agent were to fill in that function, right? It has to be probabilistic. It can be creative. So, except that everything else is deterministic in nature there.
[21:21] Then, uh once the PR is merged, that is when the auto deploy process will today will happen where using the pre-built tooling, whatever the developer has used to test from their local machine, those those exact same scripts will be used by GitHub
[21:37] Actions to test and deploy into the dev environment. This way we don't have any inconsistency. So, same process, same things are being handled by users and by GitHub Actions. So, it's all consistent there. And it does a lot of other things too, right? By uh
[21:53] we are heavy on PySpark and wheel packages and all that. So, it builds a wheel package, it runs the test cases and all, it builds a wheel package and pushes it to the artifact uh repository and downloads it from the artifact repository into the dev workspace workspace, and using the bundle commands, be it bundle validate, bundle deploy, bundle
[22:09] run, optionally, it will run in dev as well. So, it will create the jobs. And for UAT, it kind of waits for the approval because the domain leads uh are expected to go into the the
[22:25] GitHub Actions UI workflow and click on that UAT and then uh review the deployment and then click on the approve. That's a hard gate. So, until the they approve, you know, pretty much it stays there. And once they approve, it goes ahead and you it use the exact same tooling that
[22:40] it used for dev with a different set of parameters and then creates the job in UAT. And once it is done, then the expectation for production, as I said a little while ago, it's a bit different than the when the CD workflow that you triggered before, it ends at
[22:57] UAT. And for production, now for production, you need actual artifact versioning, right? You need to kind of, you know, deploy only things that that have changed. You cannot just deploy everything, right? In lower environments you can do whatever you
[23:12] want. In production you cannot do that. So, in production the way that we have designed this is tag-based approach. Every time something needs to be released to production, it follows this specific naming convention. A tag needs to be created with It starts with
[23:28] release {{}slash} and the module name here, which is same as domain, which you saw earlier like loyalty, food and beverage, entertainment. All those could be modules. So, with this it will know which CI pipeline which CI/CD pipeline it needs to invoke exactly.
[23:45] That and also each each domain in there will will have its own versioning. It's a mono repo. We're not forcing all the domains or all themes to follow the same versioning and then increment upon it. That That That will not scale well. So, each domain will have their own versioning. And when this is created, it
[24:01] triggers a production workflow. And there might be a question, right? You know, how within a bundle, how are you identifying the changes between the bundles, right? Within the domain, how are you identifying the changes within the of the bundles? A domain may consist
[24:18] 20 bundles, for example. Every release you might you might not be redeploying all 15 bundles, right? There could be an addition of new bundle or there could be an update to an existing bundle. We built in the intelligence into our CD process, which will identify what has
[24:35] actually changed. It will compare the current tag with the previous tag and do a get does a get diff to understand what has actually changed. And then there is a second layer to it. The first one is to identify what has actually changed. And the second layer is to identify
[24:51] whatever has been changed, do I have your approval to go ahead and deploy it? We control that through deployment manifest file. That we're going to get to that uh in the next slide, but every change that happens travels
[25:06] through the exact same government path. Now, coming to the deployment manifest file. This is the file which says what is getting deployed where. Imagine a scenario where you have two features that are sitting in the
[25:22] UAT. And feature one is ready to go. You got the business approval and then it's ready to go. And feature two it's not. Uh business is still taking time, but you want to cut a release, right? Uh in a trunk-based model, everything is sitting in one branch. You don't have
[25:37] like multiple branches where uh you have it in traditional Git flow. So, now how do you kind of handle these situations, right? The answer to that is the deployment manifest file. In deployment manifest file, we have sections like development section, UAT section, production
[25:52] section. In the example that I gave earlier, we could enable that feature one in UAT section and say yes, it's ready to go. We cannot Yeah, the same bundle we can add it into production and say yes, this is ready to go to production.
[26:09] And the second feature which is not ready, we can we can just, you know, ignore it uh and do not add it to the production. That way GitHub actions know it has two gates, right? One is to understand what has actually changed by doing the Git diff. And after that, it will verify against
[26:25] the Databricks manifest file, deployment manifest file, to understand yes, is this ready to go? Should I be deploying this? If the answers to both of them are yes, then it will go ahead and proceed to the deployment. And this file, like any other file, will follow the same exact path to go to
[26:42] production. And if you have new domains, they are expected to have the sim the same files, exact same template, so that GitHub will automatically detect all these things and provide these things to them.
[26:58] Now, I did talk about deploy bundle.sh earlier. Uh think of it as something like this, right? You have bunch of uh developers wherein if they are using DABS, each could be using the same repetitive uh things, right? Validate, uh deploy, optionally run,
[27:15] building the wheel package, testing, all all those things, right? We kind of understood that pattern and pretty much baked in all of that information into a deployment utility tooling. That way they don't It's all abstracted to them. They can just go ahead and invoke this thing with the respective parameters,
[27:32] and then this would also ensure that it is abiding by whatever you have in the deployment manifest file that I showed you earlier. So, accidentally even the nothing goes uh into the environment without abiding by the deployment manifest file. Everything uh is honored
[27:48] by the deployment manifest file. The two uh I would say, you know, the two things which made the framework even more powerful are serverless compute and the table update base triggers.
[28:03] With serverless compute, of course, you know, now we do not have to worry about maintaining a cluster, uh picking a type of node, right? Memory optimized, compute optimized, and how do I scale, you know, that be dynamic scaling, what is my minimum, what is my maximum, now I need to worry about my Databricks runtime, all of these things.
[28:20] With serverless, we do not We kind of, you know, got past all of that. Uh of course, you know, not always, right? Uh sometimes uh your code might not be compatible with serverless, right? If I have to take a quick example, data frame.cache or data frame.persist, they wouldn't gel
[28:36] well with serverless. So, you have to tweak your code to kind of meet the serverless needs, but uh um fortunately, we did not had too many things to tweak. So, we were able to, you know, get it work with serverless, and the results are fruitful. And on the right-hand side you have table update based triggers. Sorry.
[28:52] You have table update based triggers. Now, absence of this we would have ended up doing an event event-driven triggers outside of Databricks, right? So, our requirement was as soon as silver tables updated, we
[29:09] want to run this particular XYZ job. And this XYZ job is let's say if it is subscribed to table 1 2 3 on the silver as soon as they got they get updated, it can be any or it can be, you know, all, right? Depending on the configuration. As soon as they get updated, you want
[29:24] this job this job to be run. So, this is what we got through table update based triggers without having to manage external orchestration even events and all that stuff. Uh we were able to get the most out of the table update this based triggers.
[29:40] And we are able to control the burst runs as well, right? Because if our requirement is to run twice or thrice a day and if the silver tables are being updated every hour, for example, right? We do not want to run
[29:55] let's say the gold requirement for that particular object might not be like, okay, I don't want to run I don't want to waste the compute and all to run every 1 hour, right? So, how do you control that? There are two one key thing that you can do. You can use this mean time between triggers. Right now it is set to 12 hours in this example. So, this will control that
[30:12] okay, even though there is a change in the silver object, I will not be running it because now I need to wait 12 hours between runs. So, this will control the burst runs. If silver tables are updated quite often. And you can also have some cool off
[30:28] period here in the sense uh if a table is updated, you if you might want to give it another uh 10 In this example, we gave it 15 minutes. So, you can say that, you know, do not go right off after uh silver is updated, you know, maybe give it 15 minutes and then, you know, start it. Some cases it helps because of
[30:45] uh, the garbage collection and things that happen behind the scenes, you know, some something it helps to add little bit of buffer there. And the benefits with this we did not need to have an external orchestration. And we did not need to
[31:02] manage uh, the state, the you know, the events and all of those things ourselves. Everything was baked in and we just, you know, started using it. And we defined the data quality we ensured data quality
[31:19] in three different uh, uh, classes, I would say. Each layer here detects a different class of problem. One is unit testing, another one is common data quality, and third one is uh, in the interest of time I'll spend time on the layer three, the specific
[31:34] data quality. This is where the domain specific validations are done. If you look at loyalty, tier credit match, uh, match checks, and reward point uh, balance checks, these are things that are critical for that particular domain and they want the data quality monitoring to be enabled. So, of course,
[31:49] we did use the data quality monitoring feature which used to be called as lakehouse monitoring. Uh, we were able to you make use of snapshot and the time series uh, only by picking selective at uh, selective uh, columns because yes, it's going to get a bit expensive. Uh, the DBU cost is a
[32:06] little bit on the higher end uh, for uh, DB data quality monitoring when compared to everything else.
[32:33] Okay. So, in the interest of time I already uh, have an open PR. So, this is an open PR. It was raised a while ago. And if you look at the checks, I just wanted to show Oops.
[32:48] Thank you. Okay. Now, uh here is an open PR. and look at the checks, I will ignore the code QL and code scanning results in
[33:03] the interest of time, but if you look at the loyalty, this is the uh changes in the loyalty module. And uh pretty much it detected that change and invoked the build command, which also runs the uh pytest. And then it invoked the bundle validate, which
[33:18] will ensure that, you know, bundle uh database.yaml is successful and I it's error-free, basically. And then it's kind of, you know, waiting. And if I have to show one thing, earlier I was talking about the structural checks, right? I wanted to ensure that it's following
[33:35] everything is following the same structure. So, through a Git a GitHub uh actions workflow, we have introduced that check. So, pretty much this validates on every PR if everything is meet meeting uh if everything is following the same structure that we expect it to be.
[33:50] And in the interest of time, I'm already going to show and a a pre-run. So, this is one of those things that uh PR is approved, it got merged, and then it got successfully deployed into dev, and it got deployed into UAT uh by following an approval.
[34:06] And uh did who approved it, you know, for example, uh that's those are the things that were lacking behind uh that were lacking uh earlier, right? So, you could get all of that information inside this deployment prediction rules, wherein, you know, you can see who approved this, you know, and all that
[34:22] information through it, right? Once it is approved, it goes into UAT by following the same standard scripts. And here, if you look at it, uh the difference between the earlier one and uh this one.
[34:42] This is triggered from main. And the production one got triggered from a release tag. So, this is where we control the module name and the versioning of it. So, through which it will identify what should be the artifact version based on this and it will go ahead and publish that into JFrog and downloads it into workspace and then pretty much goes
[34:59] on from there. One thing I wanted to kind of quickly show inside this is earlier I did mention about Git diff, right? How it kind of identifies if only that particular module has changed and nothing nothing else has changed, right? It should It should have the intelligence to deploy only that bundle
[35:17] but do not touch anything else, right? So, this is the step that would identify by doing the Git diff by by doing the Git diff comparing the latest version with the prior version which got released into production and do the diff and identify what has changed. Sometimes there could be change
[35:32] in the There could be changes in the common config, right? Which might force to redeploy everything in that particular domain, right? It's It's possible. So, in which cases we have a custom workflow dispatcher which to which you can feed the tag and the build
[35:48] number and all those things and then you can kind of, you know, force it. You can use the same workflow to do any rollbacks as well if things did not go well after going to production, you might have to definitely roll back. So, you can use the workflow dispatcher to roll things back.
[36:13] Now, uh what are the key takeaways from this? These are the real results right from our production at this time and it's growing as we speak. The consistency at scale. Right? With this we are focusing heavily on the structure. Pretty much, you know ensuring that we have the deterministic behavior on
[36:30] everything except the transform which can be probabilistic and which can have some creativity in it. With this we are able to position ourselves in a way that it would become easy even for AI agentic tools to come in and follow this exact same structure because it knows what to
[36:46] do through all the markdown files that we listed down and then it would go from there. In fact you know that's what we're doing at this time. So we were able to build in uh 50 plus pipelines across the five plus domains without any variances. And
[37:02] coming to the speed and reliability with this custom deploy tooling and the data bricks declarative automation bundles pretty much we were able to cut down the time, right? The development time and the testing time and the time it takes to go from
[37:17] one environment to another environment. It has drastically reduced. And the governance and safety. This This is handled in two different layers. One is Unity Catalog. We ensure that it's not user to machine. We used OIDC and then we used M2M
[37:34] uh service principles wherever it's applicable. So all the jobs are being created through service principles and the permissions for the rest of the folks are being added through the permission section of the DAPs wherein you can say, "Hey, you know, this is my AD group of the users. They want to have read access to these things." All of those things are managed
[37:50] through that part of the DAP section. And of course the GitHub actions wherein in the environments you go ahead and say, "Hey, this person is supposed to be approving this." You could be having domain specific leads. Not everyone should have same level of access across
[38:05] the domains because if you have a multi multi teams working on things, you know, it would they might accidentally approve someone else's domain so which can happen. So that can be handled through code owners. Within each domain you can have code owners which will say, "Hey, any changes in this particular domain,
[38:22] this particular set of people needs to approve. Without them approving any anything in this domain, do not promote anything from thereafter. So, that's how it's been handled, and of course, deployment manifest with trunk-based develop trunk-based methodology, it's doing wonders.
[38:46] And I would request everyone to please go ahead and complete the service. It's on the app. Any feedback from you would help shape the future sessions better.
[39:04] And myself and Jennifer will be outside. If there are any questions, we would love to chat. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.