Skip to main content

Convert to a VNet Injected Workspace

Summary

  • Managed to VNet conversion: Databricks now enables converting existing Azure Managed Workspaces to VNet Injected Workspaces, allowing enhanced security and flexibility without requiring a complete migration.
  • Network infrastructure setup: The conversion process involves creating network security groups, configuring virtual networks with appropriate subnets, and properly sizing IP allocations for your workspace requirements.
  • Zero-migration transition: This new feature allows seamless updates to existing workspaces, eliminating the operational burden of full migration while preserving workspace settings and configurations.

Convert to a VNet Injected Workspace

Watch: Convert to a VNet Injected Workspace
Join Kiran Anand, a Senior Specialist Solutions Architect at Databricks, as he guides you through the process of converting an Azure Databricks Managed Workspace into a VNet Injected Workspace. This comprehensive demonstration covers the essential steps, from creating network security groups to deploying a virtual network with subnets. Learn how to enhance your workspace's security and flexibility with this new feature, allowing for seamless updates without the need for a complete migration.

Chapters

FAQs

What is the difference between a Managed Workspace and VNet Injected Workspace?

A Managed Workspace is fully managed by Azure/Databricks with locked resource groups where you cannot make network changes. A VNet Injected Workspace allows you to control the underlying network infrastructure, enabling custom configurations and enhanced security controls.

What is the conversion process?

The conversion process involves stopping all running clusters and workloads, creating network security groups, configuring a new virtual network with appropriate subnets, sizing the IP allocation based on workspace needs, and then deploying the conversion.

Can I keep my existing workspace configurations?

Yes, this conversion feature allows seamless updates to existing workspaces. You preserve your workspace settings and configurations while transitioning to VNet Injection, eliminating the need for data migration.

Why would I want to convert to VNet Injection?

VNet Injection provides enhanced security through custom network controls, allows you to add more security policies, and provides greater flexibility for networking configurations. This is especially important for enterprise deployments with strict security requirements.

Full transcript

0:00 Understanding managed workspaces

Hello everyone. I'm Kiran Anand, a senior specialist solutions architect in the cloud infrastructure and security specialist team in datab bricks. Today I'm going to do a quick overview or a quick demonstration of how to convert an Azure data bricks managed workspace into a vnet injected workspace. Let's quickly talk about what exactly is a Azure data bricks managed workspace. So Azure datab bricks managed workspace is a fully managed kind of Azure service.
And when we when you deploy this data bricks um datab bricks creates and deploys a new virtual network on its own and the corresponding subnets that it require for functioning properly all this are deployed into a man logged u resource group. So essentially that resource group where it is all these subp parts are deployed is locked. So you don't have access to change it and it's it's essentially like a black box where you cannot do anything after after it's deployed.
So it's it's a good good thing to use for any kind of initial um testing purposes and stuff like that. But if you want to further update your workspace, add more security, you would want to convert such workspaces into a vener injected workspace. We have introduced this feature recently wherein you can actually update this existing managed workspace and convert that into a vin injected workspace. So now let's look at it how how to go about this process.
Right? One thing that I would like to highlight is what when you are trying to do this process like converting from a managed workspace to a viner injected workspace make sure that you do not have any clusters up and running. Nothing should be up. No workloads should be running. Any SQL warehouses that's u that should be up and running should be closed. Also try to figure out what would be the number of IPs that you would need on your current workspace and based on that create the new virtual network that will that you are trying to inject.

1:18 Preparation and verification

Now let's get into the process. So I have already created a managed datab bricks workspace. So let's go and check what this is. So I'm sharing my screen where you can see my datab bricks workspace which I have created in Australia central true region. And how do you identify that this is actually a managed workspace. So when you look at the u workspace go to the settings expand the settings on the left hand side and you will see that the networking section of that settings is actually grayed out.
There is you cannot click on it, you cannot get into it. So you are not allowed to do any changes on networking. Right? Furthermore, if you want to further look at it, there will be something called a managed resource group within any workspace. Click on the managed resource group and you will find that there will be a VNET that was created by databicks and the name is always workers VNET, right? And you can see that once you see that workers vet if you go further into the details of the workers vet there is a specific IP space that we use temp 139 and there will be like two subnets that we use.
So all those are very specific things that you will be created public subnet private subnet if you see this then you are actually looking up at a managed workspace. Now let's go back and try to convert this into a vit injected workspace. So to do that, the first thing that you should do is to create a network security group which is empty. So I'm going back to portal. I'm going to search for network security groups. Here you go.

3:04 Creating network infrastructure

And then I'm going to create a new network security group. So you don't need to put in any rules or anything. It should be an empty network security group. In the next step, we'll be using this network security group. Give it a name. I'm going to create it in the resource group that I already have. This is again in Australia central 2. So make sure when you're creating this it should be in the same region as your workspace.
So our workspace is in Australia central 2. I'm creating it in Australia central 2. Now I'm create click next. I'm going to I'm not going to add any tags for now. Click on review. It's going to generate a template behind the scenes and validate if it's all good. Once it's validated it'll give you a um create option. Yes, it passed. And now we can go ahead and create it. So let's create the network security group. So you can see that it's running at this point and once this is done what we will do as the second step would be we will create a VNET which which is a virtual network.
So if you already have a virtual network can I use it? Yes, you can use an existing virtual network but you need to create two sub two new subnets in that virtual network. But in our case we would be creating a fresh new virtual network and then we will be creating two subnets within that and the NSD that we are trying to create right now will be used. Well, that is the NSD that will be attached to the subnets that will be created.
So, that is where you will be using it. So, it's getting deployed right now. Looks like it's done. So, go to resource. Yep, all good. It has created the default Azure NSG rules. Looking good. Let's go back now. The next step. So, step one, create a blank NSG. Step two, create the virtual network and the corresponding subnets. Right? Pay attention where how much is your current usage and things like that. And based on that you should decide you should size your VNET as well as your subnets right so for that like look at look at your current usage model try to figure out that how many how many IPs would you need and based on that size your new virtual network so let's you can search for virtual network here I already have virtual network so I'm going to click on it here and then create a new virtual network same thing you have to give a name and I'm going to put it in the same resource group for now search my resource group Australia central 2 put it in the same region.

4:45 Configuring the virtual network

Give it a name. Click on next. I'm not going to add any specific um configurations here. It's click on next and then you will it'll be taken to the like where you have to define your address space for the vinet. So for the address space currently by default there is a default subnet that it's been adding. I'm going to delete that. I don't need it. And we'll create two new subnets. But the default address space right now is 100/16.
It is like 60 64k. IP addresses 65 K. We don't need those many. So, um I'm going to make it smaller. So, I'm going to use a different IP range and a much smaller one. I'm going to change this and then I'm going to use /24. I don't need SL 2016. So, I'm going to use SL24. So, SL24 means you have 256 IP addresses. So, yeah, this is where I said you you have to find your current workloads how much you are using based on that size it.
So it could be SL24, it could be SL25, whatever you need. So I'm going to use for now SL24. And after that I need to create a new subnet. So you have an option to add a subnet here. Click on that. And the subnet purpose is default. Give it a name. If you remember when we looked at the uh default VNET that databets, one was called a public subnet and another one was called a private subnet. Public is used by hosts and the private is used by containers.
So to make the name much more easier, I'm not going to use public and private. I'm going to use host and container. So let me use my names. Going to use the I'm going to use first one which is the host subnet. This is equivalent to the public subnet that you had. And then I'm going to I'm not going to have all the 24. I'm going to only have six. I I don't need I can just stick with 64. So I'm going to use /26. So it's it is ranging from 7900 to 79255.
Uh now if it is IP range this is going to be up to 63. So I'm going to use 64 IPs. Great. This is something where you have to pay attention again. So there are a couple of things that we need to add uh or select while adding the subnets. Right? The first one is if you remember we had created a network security group. You have to select the network security group u for this particular subnet. So what you do you click on this and you will see that it's in the same uh we here created that select that network security group then another important factor is you need to make sure that this subnet is delegated to datab bricks so for that if you really look at it there is a subnet delegation option right here click here and search for datab bricks so you will find this delegation option called Microsoft databicks/worksp choose that two things and like important things is finish like fill in the IP details.
In this case, I've used the 64 IPs. So, I used the /26. Then choose the network security group that you selected. And then delegate the subnet to data bricks. Now you can add it and then we will add one more subnet. We'll do the same process and I'm going to rename it as container. Don't want the name default. I'll put container subnet. I'm going to use again /26. But the IP range I'm going to stick to 64 because the previous one was using till 63.
I can start from 64 and do the same things here. You have to make sure you select the network security group and then you go here do the delegation search for datab bricks and then Microsoft. Bricks/workspaces that's all and click on add. Now if you look at it you create you had asked like you we have still not created the virtual network. While creating the virtual network, we are given the address space and we created to we asked we have putting in the template two new subnets that needs to be created where we added the corresponding NSG as well as the like delegation for Microsoft uh Microsoft data bricks.
So we are good there and now let's click on next. I don't need any tags for now. So I'm going to click next again and then it's going to generate the template and validate it. Now let's go ahead and create this virtual network. It's going to take few few minutes maybe few few seconds to few few minutes and once uh so it's deploying a virtual network behind the scenes right now with the address space that we gave which had 256 IPs and then in that virtual network it's going to create two subnets a host subnet and another container subnet both of these were using the same NSG and then we delegated both of these to Azure datab bricks essentially using that Microsoft databicks/workspaces option and once it get deployed now we have a VNET that we are planning to inject into our managed workspace.
So let's wait for this process to complete and once it's done we have all the prerexs done. This is the first first and second step in the in the document that's done. Now the third step is where we actually convert the convert the existing managed workspace to Vet injection. We Yep, it is done. Let's go to the resource. Take a quick look at the resource. Make sure everything is in place. So you can see that address space is what we gave.
There are two subnets that it has created. Let's check on the subnets. Yes, there is a host subnet and a container subnet. Both of them as are 26/26 which is like yeah 64 IPs. Five IPs would be used by Azure. So 59 IPs and then it's been delegated. It's using the security group. Looking all good. Perfect. So that brings us back to our workspace. And now as the third step, we're going to convert this particular workspace into VNET injected.

10:51 Preparing the workspace template

Like we already created the VNET that we're going to use and now we're going to inject that VNET into this and once it's done we should be able to see the networking tab and we should see that like in the managed resource group it's not using that workers VET anymore. So let's do that. So what's the first step that you should be doing to do that right like again this is a process that you have to follow. So go to the workspace go to the on the left hand side navigation you click on the automation expand the automation and you see something called export template.
Click on export template so that it'll generate a template of the current workspace. So behind the scenes right now it's generating a template of the current current workspace which is actually a managed workspace. Once we have that template what we're going to do we're going to edit this template and put in the values that we want like the vnet the subnet all these details we're going to put it. So while it's generating I'm going to I needed to copy one more.
So I'm going to duplicate the tab and copy one parameter which was the vnets like you had to take the resource ID of the VNET. I'm probably going to show that before I go there. So this is one thing that before you start changing the template you need to go to the vnet that you had created and you have to ex uh you have to get the resource ID of the vinet because this is needed when we are actually going to inject. This is something that we need to supply in the template.
We saw the template that we generated. We need to supply this is the VNET that you're going to use and the whole resource ID need to be supplied there. How do you get the resource ID? You go to the VNET that you created. There are two ways you can do this. Uh the first option is like click on the JSON view right here on the right top corner. If you click on it, the first thing that it shows on top is the resource ID.
You can copy that. That's one option. Or second option is to go here in the properties. If you click on the properties on the left hand side in settings, you will see a resource ID. Either of this are the same. So you can go I already copied it. So I'm saving it in my notepad here. Now let's go back and do the template. Now you go to the workspace. If you remember the steps that I talked about, expand the automation and then click on export template.
And it let it generate the template. And now that the template is generated, you are seeing some kind of a uh highlighted um there are no errors here because this workspace is not having any private endpoints. So it was saying like it cannot be exported. There isn't any private endpoint in this workspace. So we we can ignore this completely. Now once this is done, the template is exported. You click on the deploy. We are not deploying anything at this point.
We want to edit it. So to get to the edit, you have to select the deploy option. Now you get the edit template option. Then you go ahead and uh select the edit template option. Now you have a template that can be edited. So this is where you're going to make some changes. What are the changes that we're going to make? Uh there are three types of changes that we're going to make. So the first one is uh the API version.

13:32 Deploying the conversion

So the current the API version that is being used by the template here is 25 2025 0301 preview. And you have if you look at the like documentation for this vinet injection conversion process it states that the this API version has to be a very specific one which is 2025. So I'm going to change this. So this is something that you will have to edit and replace it with 821. So I changed the uh older version to 821 which is what is we need then.
So this is one change. So this is update a current one. Second one is there could be some values in your template in s certain cases. In certain cases you may not see it, right? If you don't see it, you can ignore it. But if you see these values that I'm listing out, right? I'm going to talk about right now. Delete it. Delete the whole block from your template. So the first one would be vinet address prefix. So what you need to do is search for this in the template.
So I'm going to search for vet address. There is nothing like that in my template. If you had one, you might have seen something like this, right? In the whole template here. So, you had to delete the whole piece in that case. But here in this case, we don't have any. If you had one, you would have seen a VNET address prefix here and the corresponding value or whatever. So, you would delete the whole block. That was what we need to do.
So, remember this. It's it's all documented in the um in in our official documentation. So, you don't have to remember these. You can copy it from there. Vet address prefix is one that you need to remove if it is there. Second one is NAT gateway name. So let's search for that. NAT gateway name is also not here. So we're good with there. And the same third one is public IP name. So if you find any public IP name or NAT gateway name or uh Vinnet address prefix, make sure to delete the entire block from there.
Right? Like make sure that the commas and everything are in proper. So if you're deleting, make sure that you delete the whole block like that and you should be good. So in our case, we don't have any of those. So we don't have to worry. Usually these three values would come in. If this manage workspace are older, you would see these in the newer workspaces. If the workspace is deployed in the last 1 year or so, you would usually not find find these values in there.
If you find it, you can delete it. If you don't find it, ignore that. So the first piece is to change the API version. Second part is to look for this three values and delete. Right? if there if there are anything else. Now comes the third part where we going to update the template and include the vinet and the subnet related stuff. So for that I there is a there is a some template piece that's already provided in the documentation which I have copied in my notes.
I'm going to use that and put it in here. So you can use the same thing. So and so what I'm going to do here is it has to be in the JSON format. So copy it from your from the documentation, make edits to it based on what you have and then put put the value in here. Whatever you are going to add should go into resources. Under the resources there will be uh properties. So essentially you have to look for resources, properties and parameters.
So this is the first section and under that you have properties and under that you have parameters and within the parameter section is where you should be adding it. So I'm going to add it at the bottom. So you need to put a comma first and then I'm going to I've copied it so I'm going to paste it here. So this is the section you get this whole section. This is the section that we just pasted. Right? So let's go through that.
So what does it have? Custom private subnet name which is the container subnet which we created. Custom public subnet name is the host subnet and the custom custom virtual network ID. And this is the resource ID I copied. So this whole section JSON section is provided in our docs. Just copy it. Edit the values with whatever you're giving. Like if you're using public and public subnet and private subnet, use private subnet and public subnet in these corresponding locations, right?
And then use your whatever is your Vet ID, VNET's resource ID. Copy that and put it in here. And make sure that you put a comma if you're adding it at the end. And there there is no comma at the end here. Once it's done, that's all you need to do, right? Like don't change any other values. Make these three different types of changes. one update for the API version, delete if present for three different values, and fourth one is to update and include this section, include the section for the VNET and subnets.
Now, once it's done, let's go ahead and save it. So, it's saved. Click on next. Make sure when before clicking next, make sure everything is filled in. So, the source group is the same. I have a relay central. All good. Sometimes if it's not filled in, make sure that you fill it and then click on next. It's going to validate the template. Make sure that things are looking good. All good here. Now let's go ahead and create it.
So it's initializing the deployment and now it's going to start once it's all done it's going to start the deployment process. So it's submitting the deployment at this point and once it's done like you will see that yeah running the deployment the screen will uh show you there it is. So now it's deploying. So now it's going to take a few minutes. So what is happening behind the scenes is it's deploying the workspace again.
So at this point it's actually upgrading or updating the workspace. you have the same workspace and removing all the existing vnet like parameters and stuff like that and injecting the new vinet parameters. So that way you don't have to worry about creating like all the workloads and everything all all those things whatever you had in your workspace is remaining intact. You are just flipping it over to any injected workspace.

18:38 Post-deployment validation

That is what is happening behind the scenes. This process will take probably a few minutes like depending on your region and uh what is the load there and stuff like that. Okay, looks like the deployment is now complete. So, it's all looking good so far. So, let's go to the resource and check how it is looking. So, go to resource. So, here you see essentially the workspace. So, let's click on the workspace and see how it is.
So, now if I expand the settings, you see that the networking is no longer grayed out. So, that shows that you have the networking tag enabled. Okay, that's looking good. So, you can make some changes. You can add private endpoints. This is this is what I was talking about earlier. You can actually make it more secure by adding more and more features to it like private endpoints and stuff which wasn't possible in the earlier model.
So now it's allowing all this. It's giving me options to choose which which is great. Awesome. If you click on see more here under the details you will see that it's actually listing the new virtual network that's being used by the datab bricks workspace. So you can see that this is what we created the K databicks VNET and if you look at the private subnet name it's container subnet public subnet name is host subnet.
So everything is looking good. This is exactly what we passed into this datab bricks vet injection. And we see that it's all gone. And now you can check the manage resource group. Even if you see some extra things in there, it will all go away. It's behind the scenes. It will delete. It might take some time to delete some of the entries and stuff like that. I've seen that happen couple of times when when I'm doing this conversion.
If you go and check after a few minutes, you will see that this is gone. You will see only these three entries. You should essentially see um in a in a just a unit injected workspace, you will see a bridge uh account. You will see a managed identity and access connector. So after a few minutes, you should see this is also gone. So far so good. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.