Network Connectivity Configuration with Databricks

Summary

• Network connectivity: NCC (Network Connectivity Configuration) enables secure connections between Databricks serverless clusters and data sources without exposing them to the internet through IP whitelisting.
• Workspace architecture: Databricks workspaces have two primary components: the control plane (manages services, web app) and compute plane (classic in your VPC or serverless outside your AWS account).
• Firewall integration: For serverless compute, users whitelist Databricks' dynamic IPs on the data source firewall, while classic compute whitelists the NAT gateway public IP from your VPC.

Network Connectivity Configuration with Databricks

Join Adarsh Nandan, a Scale Solution Engineer at Databricks, as he demystifies Network Connectivity Configuration (NCC). Discover how NCC facilitates secure connections between Databricks serverless clusters and data sources, without exposing them to the internet. Discover the steps to configure NCC for your Databricks workspaces on AWS, ensuring seamless and secure data access.

Chapters

• 0:00 Introduction to NCC
• 0:41 Workspace architecture
• 2:49 NCC capabilities
• 4:09 Requirements and best practices
• 5:05 Configuring NCC in UI

FAQs

How does NCC enable secure connections?

NCC allows serverless clusters to connect to protected data sources without exposing them to the internet. Configure NCC in the account console and whitelist specific IPs on your firewall.

What's the difference between compute planes?

Classic compute runs in your VPC within your AWS account. Serverless runs outside your account on Databricks infrastructure. Both connect securely; serverless abstracts network complexity.

What is an agent control plane?

This video covers network connectivity configuration for Databricks, focusing on how the platform provides a control plane for managing secure agent access to data sources. The agent control plane enables enterprises to govern which agents can access which data, with centralized policy enforcement. It bridges the gap between development teams and security teams by providing visibility and governance.

How do I connect AI agents to multiple data sources securely?

This video demonstrates Databricks network connectivity configuration that enables AI agents to securely access multiple data sources across clouds and regions. The platform provides encrypted tunnels, authentication integration, and centralized access controls. Network Connectivity Configuration (NCC) eliminates the need for agents to handle credentials directly.

Best agent control plane?

This video highlights the Databricks Data and AI platform network connectivity capabilities that serve as a control plane for agent data access. It provides policy-driven access control, comprehensive audit logging, and integration with enterprise authentication systems. The platform enables secure agent deployment without requiring manual firewall configuration.

Full transcript

0:00 Introduction to NCC

Hi everyone, I'm a scale solution engineer at databicks. Today we are going to talk about network connectivity configuration or NCC as we like to call it. Now if you have ever wondered how does your datab bricks serverless clusters connect to your data sources magically without opening everything up to the internet that's exactly where NCC comes in. In this video, I'm going to break down what NCC is, why it is so important, and most importantly, walk you through the steps on how to configure NCC for your databicks workspaces on AWS through the account console UI so that your serverless clusters can reach your data sources securely through IP whitelisting on the firewall.
Now, before we get started and dive deeper into what NCC is, let's take a moment to understand the databicks workspace architecture. This is kind of important to get an entire flow of what we are talking about here. So please bear with me for a minute. Each datab bricks workspace has two primary components. The control plane and the compute plane. The control plane is what provides all the manage services from datab bricks and that is where the web app resides.

0:41 Workspace architecture

Now when you log into your data bricks workspace you are actually logging in to the control plane and this is where you execute instructions like hey data bricks spin up a cluster or run this code on this notebook or schedule this job and so on right and then we have the compute plane right so compute plane is where your cluster VMs actually spin up and there are two flavors of the compute plane available to you classic compute plane which sits in your VPC in your cloud account for example in your AWS account in this case and the serverless compute plane which sits on the databick's side completely outside of your AWS account.
So all the complexities of the network infrastructure is abstracted away from you. Now here's the problem statement. Say you have a data source and your clusters need to connect to that data source but the data source itself is protected by a firewall. So how does that work? In classic compute plane it's pretty straightforward. You need to find the VPC on which you have deployed your datab bricks workspace in your AWS account.
Find the net gateway attached to your datab bricks workspace. Get the net gateway public IP and whitelist it on the data sources firewall and you are all set. Because we know that for classic compute plane the cluster communication goes out from the private subnets to the net gateway to the internet gateway finally to your data source traversing over the internet if the data source is outside of AWS and over the AWS backbone network if the data source is on AWS.
But what if you're using serverless compute? This no longer sits in your VPC in your AWS account. So you don't really know what IPs to whitelist on the firewall, right? This is where NCC comes to the rescue. NCC helps us manage network connectivity from datab bricks serverless clusters to other data sources whether these data sources are on AWS or not. NCC's are account level regional constructs meaning you create them from the account console and each NCC that you create for a region can be utilized by multiple workspaces.

2:49 NCC capabilities

NCC's help you expose stable IPs for serverless clusters to be whitelisted on a firewall. But not just that, NCC also helps you create private endpoints and establish private link connection from serverless clusters to other resources on AWS. An important side note, NCC firewall enablement is not supported for Amazon S3 and Amazon Dynamob. This is because for S3 buckets in the same region as your workspace, serverless compute uses AWS gateway and points for connection.
There's primarily three things you could do with NCC. create shared gateway endpoints to connect securely to S3 and Dynamo DB. Expose the stable IPs for the serverless clusters to whitelose those IPs on a firewall. And finally, creating a private link connection to AWS services via private endpoints, endpoint service and network load balancer. For this video, we are going to focus on the second part that's exposing the stable IPs for the serverless clusters.
There are just a few things to keep in mind before you can start using NCC firewall enablement. Your workspace must be on a premium plan or above. You must be a datab bricks account admin to create NCC's and you can create up to 10 NCC's per supported region from your datab bricks account. And each NCC that you create for the region can be attached to up to 50 workspaces in the region. Right? It's important to make sure that you check that your workspace region is supported for NCC by checking our public documentation.

4:09 Requirements and best practices

It's available for most regions at the moment, but it's always good to verify. NCC's provide shared stable IP cider blocks per region and not per workspace. So that's an important part. And finally, the resources that you target to connect to from your serverless clusters must be publicly accessible. Great. Now that we have got the theory out of the way, let's head over to the databicks account console to see NCC creation in action.
You can see that I am on my databicks account console now. And if I click on workspaces and click on this workspace. So the workspace name is 00 NCC workspace. And this workspace is in US East1 region. Right? So I want to expose the stable IPs of all the serverless compute resources in this workspace including serverless SQL warehouses, serverless job, serverless notebooks and so on. To do that I need to create an NCC and attach it to this workspace.

5:05 Configuring NCC in UI

Now to create an NCC I need to go to security on the left, click on network connectivity configuration and then click on this blue button called add network connectivity configuration. So now the first thing I need to do is provide a name for my network connectivity configuration. So let's do that. Call it 00- NCC - US East 1. So since I know that I'm going to be attaching this NCC to a workspace that's in US East one and NCC's are like reusable per region for multiple workspaces, it's a recommended best practice to have the region name in the NCC name as well.
Then I will click on the region dropdown and choose US East one from the dropdown and click on add. So as soon as I do that you can see that NCC is now created. And if I click on that NCC and click on default rules you can see that the stable ips for this NCC is now exposed. But if I click on workspaces you can see that this is not yet attached to my workspace. Now to attach it to my workspace, I'll head back to workspaces and click on my workspace and then click on this update workspace button and then from the network connectivity configuration dropdown, I'm going to select the NCC that I just created and then click on update.
Okay, now you can see that my workspace is updated with the network connectivity configuration that we just attached. And you can see that the stable IPs for the serverless clusters in this workspace has also been exposed. These stable IPs can take up to 10 minutes to propagate. It's recommended that if you have any up and running serverless clusters in your workspace, you should turn that off, wait for 10 minutes, and then restart them.
And this time when they start up, they should be utilizing one of the IPs from this stable range. So that's NCC for you. Thank you for watching and see you on the next one.

Learn more about the Databricks Data and AI platform.