Building a Seamless Multi-Cloud Platform for Secure Portable Workloads

Summary

• Cloud portability: Deploy the same workloads across multiple clouds and regions without code changes or migration effort.
• Compliance automation: Built-in security controls and credential management simplify compliance across different cloud environments.
• Operational resilience: Move workloads between clouds during outages to maintain service continuity and minimize disruptions.

Building a Seamless Multi-Cloud Platform for Secure Portable Workloads

Talk by James Burns, Staff Software Engineer, Databricks ; Scott Reynolds, Staff Software Engineer, Databricks
There are many challenges to making a data platform actually a platform, something that hides complexity. Data engineers and scientists are looking for a simple and intuitive abstraction to focus on their work, not where it runs to maintain compliance, what credentials it uses to access data or how it generates operational telemetry. At Databricks we’ve developed a data-centric approach to workload development and deployment that enables data workers to stop doing migrations and instead develop with confidence. Attend this video to learn how to run simple, secure and compliant global multi-cloud workloads at scale on Databricks.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why would a company run on multiple clouds?

Organizations use multiple clouds to avoid vendor lock-in, optimize costs, meet regional compliance requirements, and ensure business continuity. Multi-cloud deployments reduce the risk of single-cloud outages.

How does Databricks simplify multi-cloud workloads?

The Databricks Data and AI platform abstracts cloud-specific complexity. Engineers define workloads once, and Databricks handles cloud-specific details like credentials and configurations, reducing migration effort and operational overhead.

Can data scientists focus on code instead of infrastructure?

Yes, Databricks hides cloud infrastructure complexity. Data engineers and scientists write workloads without worrying about which cloud they'll run on, cloud-specific APIs, or compliance details.

What happens if a cloud region goes down?

With Databricks, teams can redeploy workloads to a different cloud or region. This portability ensures service continuity and minimizes downtime during regional outages.

Full transcript

A cloud outage. What? A cloud? Look, look. Super important VP. I promise. But we're going to bring it up in just a minute. I promise. I thought you were the TL of all this. I thought clouds didn't go down. How could it be a cloud outage? But they're they're cloud. I mean, there's still computer. Look, look, this is just really complicated. Let me just push a few buttons and I'm going to try and bring it up. A cloud outage and see.
So, I'm not really a VP and I actually am a TL. Hi, And Uh, and uh, and we are um, members of the data platform team here at Data Bricks. And we're here to kind of tell a bit of a story and introduce you all to a few concepts that we that helps us build an every single regent and every single cloud as customer zero on data bricks every single feature. Uh in our years of working in software, we've always been in the spot where there's been a major outage and you know you're on the phone call and that very important salesperson is here to tell you like I need an RCA for my very important cousin, very important customer right now.
This is really important. Or that CTO is like, I'm going to run the post incident review because I want to talk about it. Um, and we want to talk to you all about how to build resiliency and systems into place so that way you can deploy your data bricks workloads not just in a new region in the same cloud but in different clouds different regions and move workloads from place to place to place and a regular basis. This is what we do multiple times a year for all the new regions.
Uh, to start I would actually like to do a quick poll of you all. So I am interested in how many of you all use different clouds. So how many of you all use GCP? Okay. All right. And AWS. All right. Wow. Okay. Azure. I promise this wasn't like a sorted list of my opinion on different qualities of different clouds, but this is just how I I put it up on the slide deck. I swear. Uh. All right. And how many in more than one cloud?
Okay. So now I saw a few people with like wavy hands etc. Right? I think what I found to be true which surprised me when I started datab bricks is that they're actually quite a few f quite a few customers actually deploy on more than one but they're kind of like discreet workloads like different jobs and different clouds. That's not what and I do. We deploy the same stuff in the same cloud the same time every single day.
I'm interested. Is anybody else an intrepid explorer like us? All right. That's what I thought. And so I want to convince you all that it's possible and I want to introduce you all to a few different concepts of what that's going to be like. So this is an extension of something that we started talking about last year at Summit. Um we talked a little bit about like our path to Unity catalog and Unity catalog is a clear important abstraction to the systems that we're building.
Uh and in this talk and I will walk you through what we've been building since. Uh something that makes meeting compliance standards easy. lawyers happy and engineers as productive as possible. Now what we did uh you know like we deploy in more than 100 different workspaces. We have workspaces in every single region uh bunch of different regulatory domains and all these different clouds. And as we sort of started to grow everyone started adding like little if statements like well if it's it's definitely this way if it's GCP it's definitely this way.
And so we found there was an awful lot of friction as new regions came up. which things needed to be deployed in which order, what was the right way to do things? And so we started talking with people and what they're really struggling with and what they actually wanted. And when you talk to them, you always learn so much like why can't I just log into the workspace? That's all I want to do. Why isn't my data there? What data is this?
Um, and as we started diving deeper, we started to come up with what we think is like four core concepts that really help users get to that point that makes this really easy and fast. So the first one is people always care about which data is available. Now you may think well like I just use Unity catalog I search I can find it right well you have to have access to be able to see the schema that's there and we have a 100 workspaces.
Are you going to search every single workspace to go find it? So we need a way to give people the ability to say okay I want to run my job whenever this Unity catalog table is available wherever workspace that is wherever that happens to be. Okay, they need to be able to get access to the data. It's not enough to be able to say like I know the data is there. Can you please please please just click the button and say yes.
Uh Unity catalog you can request the owner to say can for this schema can I please get access to it and the UI will federate some events back and forth and user can log in and click yes you can get access to it. That doesn't work at scale. It doesn't work with 100 workspaces right you can't do that. So, we need a way to give the jobs and the teams the ability to run and get access to that data right away, just in time when they need it, right?
They don't get access to all the data, right? That would be the easy way out. Just like, okay, you get all the access, but they need to be able to get the fine grain access they need just in time for their particular workloads. Need to test the changes. My years working with data teams, this theory, this comes up all the time. The staging data is not like the production data. It doesn't have the same velocity. It doesn't have the same shape.
It's not quite the same skew. I just I need to test on real production data. Sometimes it's not even there, right? Uh so we need ways in which we can allow folks to safely deploy their stuff to all these workspaces, all these production workspaces and do soap tests where they run their job to see if like the RAM if we if we need to run on bigger instances, right? Or if there's any regressions. You be able to see if the data matches.
And you'll be able to look at two different data sets and see if there's any diffs that happen, right? You need the ability to abstract some of that stuff away. And the thing that underlines all this is speed. You need to be able to do this quickly because if you make this hard, you make this arterous, you make it a Jira ticket, they have to file and they have to walk themselves through it. They have to do it every single workspace, this just isn't going to work.
So has to be fast. Everything has to be fast, fast, fast, fast. Okay. It's also important to hear what they're not asking for. They're not asking for VPC configurations. They're asking for IP alkaliths, cloudnetss, V-Net setups, IM ROS. Well, sometimes IM roles. Sometimes they're just like, "Yeah, I want this god role. Why can't you just give it to me?" Uh, you know, they're not thinking about vacuum settings. uh they're not thinking about like predictive optimization which is can be really powerful when used the right way, right?
Um they're just not thinking about these things. It's not because it's too hard. It's just not something that comes in their brains, right? What they want to do is they want to just to understand and transform the information that is there. Uh that's it. Uh it's not that they're not smart enough to care about cloudnetss or VPC settings or IM roles. They just don't have enough time, right? Their brains can only fit so much stuff in their heads.
And so we really need a platform that automates a way for over a 100 workspaces and all these different regions. We need to automate a way so that way we can actually move them from one place to another place, which is something we do on a regular basis. We move jobs from different places. That way we get access to different quotas, different resources, different other things. All right. So this is hard uh details of all these clouds they matter you know ADOSs is subtly different in different regions even right like there's one region that requires you to set up an SDS endpoint I don't know um sometimes you need to make sure you have the VPC endpoint set up the right way is the S3 bucket in a different IM account if it's in a different IM account does it have the right bucket policy that allows me to do the stuff I have to do is it SNS or should I use SQS in this region does event group event grid work in Azure in this spot Look, don't get me wrong, like all this stuff is interesting.
I just don't want everyone have to think about it. I want to automate it away. I want people to be amazed. All I have to do is tell me about it and then I figure it out for them. Right? Dat provides some of this stuff and cloud provider provides some of this stuff. Um, you know, the workspace setup gets you what what users want. They want SQL endpoints. They want job scheduling. They want notebooks. Uni catalog gets me what I want.
I want to be able to restrict what jobs can access which things, what humans can access which things, service principles, cloud identity mappings, all this stuff unique catalog gives me gives me a nice abstraction for cloud providers give me a whole buckets and the VMs and the different configurations and quotas. But what I need and what we've been building is something that blends all these things together. It's able to look holistically at all these different cloud accounts, at all these different data bricks accounts, at all these different workspaces, and make sure that each job that lands in these places has the right things, the right access control, and it can do it fast.
So, that is what we're building. Now, this is hard and it's a little scary. We've been doing it for a little bit. There's been a lot of rough edges. We've learned an awful lot and I've been lucky enough to actually work with who's here up here with me two different companies and so he's going to walk us through what we've been building and how we think about how that stuff works. Thanks. I it's sort of funny mentioned an issue that happened in the build uh just a couple days ago.
It's like yeah I recognize that these are real things that happen. So uh our journey on this starts with our story of what became challenging for us. So we started out and part of what we do is we we run in every region that data bricks has a presence in and that list is continually growing. Um when we start out it was a day to set up a new region because it was a handful of jobs that was supporting a particular use case and then we supported more use cases in more regions and then it took two days and three days and five days and two weeks and that cost that operational cost beyond any financial cost just started adding up and we just couldn't keep up with it.
We couldn't dedicate person weeks of time every quarter just to bring up new regions. So we tried to ask how can we make scaling free? How can we remove this cost that was incrementally added with every single workspace and make it so that when we brought up a new workspace, it was free. When we had new workloads that need to run on every workspace, it was free for us operationally and also very cheap for our customers uh for the data platform.
So we're starting as mentioned on top of the data bricks platform doing data bricks on data bricks. If we weren't we'd be spinning up VMs we'd be setting up IM policies. We'd be putting them in Kubernetes clusters. We'd be running Spark. We'd be doing scheduling on Spark. There's a lot of things that data bricks gives you an abstraction over these different clouds that let you have a standard definition for what a job is, where it runs, how it runs with a schedule.
But what we found is that wasn't enough to make it easy for our customers. They were still having to think about where they ran and what the particular configurations were. And since things change all the time, they'd be like, "Okay, I need to roll out a change. Okay, that's a quarter long migration to make a change everywhere." That was just too expensive for our customers, too. So, how do we go about making scaling free with data bricks at data bricks?
What started with this key recognition that there the two things that customer wants. They want to know what data is there and they want to be able to define work that acts against that data. So let's dive into a data registry. The first part of this data registry has three parts. The first part is the data type. So you have um you can have documents that are say a bucket. You can have tables. You can have streams. kind of other sorts of primitives as they roll out.
And so that defines like what the data is the and then on top of those we have the data taxonomy that annotates within that particular data set. What sort of data is it? And this taxonomy is a whole other talk itself. But it talks about what it is and then what purposes can it be used for. Can it move across regions? Can it does it have to stay in that region? can't be accessed by certain types of jobs but not other types of jobs.
This taxonomy defines what our obligations are mostly from a compliance and legal perspective but also more broadway in how it gets handled. And then the third part is the actual instances of the data that live in different places. So if you've got something that's a particular type of log, you don't want to track all those differently. You want to say I have this log in this region and this region and this region and then on top of that you want to define a workload against it.
So that workload has the dependencies of what it inputs and outputs. It has the artifact that allows you to do that processing and that can be notebooks, it can be jars. We have a lot of jar jobs can also be SQL. And then in addition to that we have our the people who are defining these workloads define the dependencies. So things like secrets they need to access APIs, things like AM roles they need to access data or other endpoints and also the networking requirements.
So sometimes often in our case there's cues that are only accessible via private networking and so for that job to run in a particular place that networking needs to exist before that job is run. So uh that's you saw a single definition of this. This is a definition of how it federates out. Um so you would have a a particular set of tables that in every region were transformed into their other table and then you have another uh workload that collects all those together and has another output.
So for things that need to stay in region, they stay in region. For things that have compliance requirements that are allowed to move across regions, they move like that. So we've got this idea and it's it's a nice abstraction. The first problem is where does it run and that's where we came up with the idea of baseline workspaces. So as part of the first part of this operational challenge that we were originally seeing was that we had created workspaces and each workspace was created at the time it was created with the defaults that happened to be the defaults then and if you ever want to update these it was a migration.
It was a project that could be a quarter long. It's like let's turn on Unity Catalog. I know a lot of people are still facing that migration. That was a significant level of effort. And so we we saw that this was going to happen, going to keep happening. And we decided that we need to make it so that this was free to scale. So in building that, how did we want to expose it? So the first thing we want to have is a use case and that's the core and it's sort of a vague idea.
Um but the idea is that when you have a group of things that have a similar purpose and are going to be subject to similar compliance requirements, you call that a use case. And so when someone wants a use case, that means that they're going to be responsible for owning the budget of that use case. That means they're going to be responsible for owning the administration and like higher level business compliance requirements.
But we wanted to make it easy and that's where we came up with this definition. All you need an owner, a use case name, a use case description, and who pays for it. And what this does underneath it is says, okay, that goes everywhere by default. If you you're going to get a workspace that you can run your workloads in everywhere that we have a managed footprint and if you want to control it there's options like this where you say I want in prod I want it everywhere but for the staging version I only want it in these three regions and then in GCP I need it in all regions.
So we had this this opt-in complexity that we added to this interface so that we could keep it simple for the people who want it simple and add those options for the people who didn't. So how does this actually go together? What's what are the actual phases of building this? Well, the first phase is the cloud account phase and there's where does where does this workload actually run? It runs in a cloud account still for non-serverless workloads which we have a fair number of.
So what do you need to have a a cloud account that can support baseline? You're going to need service accounts for some clouds and not others. Um you're going to need the IM roles that data bricks uses to manage resources in that cloud. And one of the other things that we looked at as we were looking at how to make scaling free. We thought what if we just pre-allocate all the mess or storage saves us time later. Empty storage doesn't cost anything.
So that's what we do. We do pre-allocation. After you have a cloud account, you have a datab bricks account. And this data bicks account, our our interface to this is a little bit different because we we have a button that we click to get a new account. But it's still a process of making sure that account that's associated with that uh that particular use case is set up to our baseline standard. That baseline standard starts with unified login.
So, um, that has some cloud variation as well, but unified login, uh, allows you to log into the account and be able to access any workspace in the account that your user has been assigned to. And I'll talk to you about how that actually works in a couple slides. We manage all this with service principles. We don't want any kind of userbased or service accountbased way of configuring these. So we provision account admin and metas store admin service principles that are the credentials that are used to manage this.
And then we also have this concept of account grant groups. So what these groups do is they provide a place for you to incorporate um a just in time group. So there's there's a whole set of things that we actually do. Um, but what we what we want to do and what we do in fact do is make sure that there's no one who needs standing administrative access to anything ever. Everyone gets the access when they need it with a justification for getting that access.
So if you need account admin, you can get it, but you get it through this group that's dynamically um updated with the people who have requested and gotten approval for this. And there is also the account group setup. So um we also have a standard for importing accounts from our identity provider that we then use for um assigning additional permissions to particular data sets. So those need to be set up and then other defaults uh like network policies for serverless um compliance security profiles and of course um IPACL's.
And if if you had taken the shortcut to um pre-create all the storage, you can also pre-register all the meta stores so that those are available to you in in a single pass in a handful of minutes. So we've got account that's raised support baseline. Now we need to build workspaces inside of those, but those still have their own set of prerequisites. And a lot of this is is basically in the Terraform documentation. Um if if you need to go through and like actually read how it's done um there's in each cloud's version there's a layout of this but you need you need a place for the stuff the compute to run and networking you need gateway still for most people and then you we also set up private link um for the cloud service provider endpoints as well as the data bricks endpoints um so that uh so that we can use the internal and not go across the internet for are um it's called secure cluster connect in two of the three clouds it's called something else in the third one but that's the idea so when we have those cloud cloud ones we also need the data bricks resources um that's the workspace admin service uh principle the workspace admin just in time group and the workspace access group so I talked about how uh with unified login um you log in one place you don't have to set up skim in every single workspace which does not scale.
Um the workspace access group when you assign that to a workspace the people in it get access as a user. And so then we use the same process of including another group inside of that group that's provisioned by the identity provider that allows people to get access to that workspace when they're part of that group and then you create the workspace. That's quite fast actually. Uh and you go on to then the phase after that which is assignment.
So you you can't do these things until the workspace exists and then once the workspace exists you uh assign a meta store to it. Um there could already be a metas store in that region in which case it's not a new one. If it's a new uh workspace for that region then it's a new metas store. You assign the workspace uh the workspace administrator security principle, the just in time groups, the access group and you can also then assign the network policies for the serverless egress gateway.
So we we've been talking about the operational free part. Uh we haven't talked about the security part of it. And so this was one of the tricky things for us because we had um we had policies. They were written in words and it was unclear how those words map to actual API calls. And if we wanted this to run and we did want it to run continuously to be able to assert that everything was still matching, we had to turn those documents into something that we could actually say this is the setting in the API and we have verified that setting is set.
So we had long conversations with both legal and uh trust and safety engineering and got the actual list that we apply. Um so that comes to the security and compliance phase that we we do these settings inside the workspace. Um so these are some of the settings that we do. There are a couple others too. Um disable HMS, disable DBFS, disable support access because if data bricks is accessing data bricks on data bricks, it should be through the product and our management not through support.
Uh we enable verbose audit logs. We enable all system tables because they're invaluable for understanding what's going on in the in the environment, including have the having the exposure of the audit logs. And uh we still I wanted to have this not be on the slides, but we still have pat tokens in some cases. We're trying to get all the way over, but we do assert lifetime setting on those so that they're they have a a specific maximum expiration.
And then we also set the workspace ACL. And then after that because it is security and you you trust but verify we have a separate implementation written in Python. So Terraform says it's all done. We're going to check in Python every one of those settings and then generate a report that says yes this workspace has met the security compliance requirements. Um and then be able to provide that to any anyone who wants to make assertions about the state of the workspaces and then we hand those over.
So in many cases we hand the workspaces over to ourselves um which we'll I'll talk about the workload side of it but we can also hand these over as their own product to other parts of the organization. So if it needs a workspace they shouldn't have to do it from scratch. If HR needs a workspace and they do it from scratch it's it's a whole I mean all these things I've just been talking about it's a whole bunch of things that you need to know.
So what we also support handing these over and what that means is they get account admin, they get met store admin, they get workspace admins uh for the places that require they get the CSP service accounts but we also retain our separate separate set of these that are audited trivial to us so that we can continue to assert the correctness of the configuration and update it as as uh additional controls become available as the security requirements evolve.
So that's how you do it. You you have to make this process something that you can run over and over and over again and so that it continuously converges on on the state that you want it to be in. But there's there's one more tricky part here. Um I don't know how many of you work with Terraform on a regular basis but uh Terraform saying that something is still uh in the same state that it should be and it's not the same as Terraform actually taking the action to create it.
So we had a separate process that takes uh takes a workspace brings it up from scratch and tears it down from scratch in every cloud and every regulatory domain because it's possible it has happened that some policy change then the create part of it doesn't work anymore. We want to find that out right away and not when we actually have to bring it up to meet some deadline. So that's that's how we add this additional process to make sure that we can continue to be able to scale anytime we need to.
So we've talked about making workspaces uh free to scale. Let's talk about what it takes to make workloads free to scale. And that's a lot of the stuff that was talking about before about what users want. There's two different kinds of scaling here. There's federation and this is what people talked about for a long time. So in our particular case, it may be different for you all. Um, but federation is is a big part of what we do.
We need to run, we bring up a new region, we need to run one more of all the workloads that need to run in every region in that new region. And so there's a whole uh long bring up process where lots of people have to make changes to their code and they have to deploy those that code in a particular order so that the data is available for the people who came before them. And we wanted to make that free. We wanted to make it so when you bring up a new region, a new data set was available in that region, your stuff just runs.
But there's also portability. And this was something that we needed to be successful as a platform. So when people are running these workloads, you may need to change something if you're a platform provider, which we are. Um, so say there's a different way that fundamentally data bricks workspaces get set up. It's happened a few times. Um, and you can't make that change on running workspace. Maybe it will never happen again, but we needed to make sure that even with for some of the settings that can't change that we could take their workload, move it to a new workspace, and they would have to make no code changes for that to happen.
That could be to free up capacity. That could be whatever else, but as long as every workload that runs is hardcoded, look up what workspace I'm in and run with these settings, it's not going to work. So, we also need to achieve portability. The process to do this had two parts. Uh the first part was figuring out how to actually put workloads in the place that they need to be in at the right time and that is called the placement platform.
The first step of that process is figuring out where it can run. So we have those annotations on the different data types and they'll say if we're going to process this workload, it needs to be handled here. And so here can be more than one place after you have these baseline workspaces. But you you say you subset all the workspaces everywhere that could be run and you say this one this workload can run with this particular set of workspaces that meet the particular requirements in the data taxonomy and then of those you can you can make choices about which has the most capacity which has the lowest cost all these other sorts of things.
The second part of it is just in time dependency fulfillment. So you could in theory say hey we're going to put all the dependencies everywhere and then no one will have that problem but that would be complicated to maintain. Uh instead when we have selected a workspace we have a process that goes through and says okay is the data set it needs actually present in that particular workspace. So that could mean attaching meta stores, that could mean doing delta shares, that could be mean setting up credentials to be able to access different streams.
But all this allows us to when it when we have placed the workload, we know that it will run there. And the output of that is where those things are actually in that workspace. So the problem that you run into um that you may have run into too is when you run these things everywhere, you end up with this giant config file that says this region, this string, this region, this string, this region, this string. Still sort of have to do that, but we want to do that for them.
Um so that when we place something in a particular place they we say this is where we put this this data, this is where we put this credential, this is where we put this access process and so then that all that information about the particular settings for that workspace goes back and gets handed to the workload runtime. So this is the hard part of this entire thing. The workload runtime is what provides an abstraction for the workload that lets it access its dependencies semantically.
So instead of saying I need to access this this data set at this catalog and schema and table, you say I need to access this data set. So it looks like this. You say I want a dataf frame. You don't say where. You just say I want the data frame for regional usage logs. And then we underneath that because of how we placed it, we can give you something that's portable. The same for a secret. I need the secret to the SAS.
Where is that secret actually in the secret store? When secrets are available in UC, when secrets change, again, doesn't matter. They ask for a secret, they get a secret. That's the right one for that environment. And I don't know if you have streaming uh things that look like this, but if you do autoloader, you can that's where it can really become uh long where you have all these different options that change in every region.
So underneath that, we just say read stream load my input and that's it. So putting that all together, we've provided federation and portability while giving people the thing that they need. They want to talk about their data and they want to talk about their workload. And underneath that, we do all the rest. Back to you. All right. Thank you. Uh, so, you know, I hope hope to leave you all with at the end of this talk with a little bit of inspiration, right?
Probably can't copy and paste everything we did. Probably don't want to. I mean, who wants to run hundreds of workspaces in all these different places with all these different paints? But what I'm trying to leave you all with, I'd love to leave you all with is a bit of inspiration. Think about like who the users are and how you might be able to help them move around in different places. We do this all the time. We've moved major workloads from a different workspace into in the same region into another workspace, another region.
We do this all the time. We did it last week. Uh we work really hard to make sure this stuff happens. And so what we care about very much is making scaling free for our users and automating away a bunch of the different parts and pieces that matter but only matter to those of us that in platform. We really want to avoid people having to write a bunch of big if statements in their job configurations. Uh we like people to really kind of avoid this magical Spark configurations, right?
We've all run into it where there's like Spark configuration slashjira tickets from like four years ago that set the Spark configuration, right? uh we want to help people avoid some of those those wonderful things. So, thank you all so much for attending. I know this is at lunchtime. Uh and I deeply appreciate all happening uh all coming. Uh please fill out a survey uh and uh and I will be off to the side here if you'd like to ask us any questions or anything like that.
Please thank you so much for coming. Thank you.

Learn more about the Databricks Data and AI platform.