Skip to main content

How can I define dependencies using the Databricks SSH tunnel?

Summary

  • Choose by compute type. For serverless compute and AI Runtime, pass a workspace base environment with databricks ssh connect --base-environment <name>. For classic dedicated clusters, install cluster libraries.
  • Workspace base environments are YAML specifications that list Python dependencies, and the SSH tunnel supports them only with serverless environment version 4 or below.
  • For project-scoped packages, run a setup notebook with %pip install at the start of each session. %pip propagates packages to Spark executors, so UDFs can use them.
  • Anything installed during a session to local paths is lost when compute restarts, so keep requirements files in /Workspace and use base environments, cluster libraries or init scripts for anything that must persist.
  • For non-Python packages on classic compute, use an init script; serverless compute doesn't support init scripts.

How can I define dependencies using the Databricks SSH tunnel?

Define them where the compute gets its environment, not in your local IDE. The SSH tunnel runs Python and the kernel on Databricks compute, so your packages have to exist there. The SSH tunnel docs give one recommended method per compute type: a workspace base environment for serverless compute and AI Runtime, cluster libraries for classic dedicated clusters, and a %pip setup notebook for anything scoped to one project. Pick the one that matches how long the dependency should live. The SSH tunnel is in Beta.

Dependency options at a glance

Method Compute Persists across sessions Where it's defined
Workspace base environment (--base-environment) Serverless, AI Runtime Yes A YAML spec that workspace admins register; all users can select it
Cluster libraries Classic dedicated cluster Yes, across restarts Compute > Libraries on the cluster
Setup notebook with %pip Any No, rerun each session A notebook in your project
Init script Classic only Yes, runs at startup A shell script that runs when the compute starts

What to weigh when you choose

  • Persistence. Files in /home, /root and other local paths are ephemeral and lost on restart. Files in /Workspace, /Volumes and /dbfs persist, which makes /Workspace/Users/<your-username> the place for requirements files and setup scripts.
  • Reach to Spark executors. %pip commands "propagate dependencies to Spark executor nodes," which is what lets UDFs use custom packages.
  • Sharing. Multiple SSH sessions on the same cluster share one virtual environment, so an install in one session shows up in your other sessions on that cluster.
  • Reproducibility. The %pip docs recommend pinning versions (==<version>), because unpinned installs aren't compatible with the serverless compute architecture.

How to set up each method

Serverless and AI Runtime: workspace base environments

A base environment is "a shareable YAML specification that defines a serverless environment version and a set of additional Python dependencies." For the SSH tunnel, the version must be 4 or below. The docs state that the tunnel doesn't support --base-environment with serverless environment version 5 or above. A minimal spec, following the base environment docs:

environment_version: '4'
dependencies:
  - -r "/Workspace/Shared/requirements.txt"
  - my-library==6.1

A workspace admin registers it under Workspace admin > Compute > Base environments for serverless compute > Manage, or you can create it with the databricks environments create-workspace-base-environment CLI command.
For AI Runtime, build a GPU version of the environment. Then connect with it:

databricks ssh connect --base-environment my-workspace-env

The flag accepts a display name, a workspace YAML path or a resource ID, and you can't combine it with --cluster. In the IDE, pick the longer virtual environment name under Python: Select Interpreter. This route also needs the "Serverless workspace base environment support in Jobs" preview enabled. After editing the YAML, refresh the base environment so new sessions get the update.

Classic dedicated clusters: cluster libraries

Install packages in the workspace UI under Compute > Libraries. The cluster library docs list sources including PyPI, Maven, CRAN, workspace files and Unity Catalog volumes. These libraries persist across restarts and appear in the pythonEnv-xxx environment you select in the IDE. If a compute policy enforces libraries, you can't add or remove them yourself.

Project-scoped: a setup notebook

For dependencies that belong to one project, run a notebook like this at the start of each session:

# Install from pyproject.toml
%pip install .

# Install from a requirements file
%pip install -r requirements.txt

# Install a wheel from Volumes or Workspace
%pip install /Volumes/catalog/schema/volume/your_library.whl

You don't need to rerun it if you reconnect before the tunnel shuts down. By default that happens 10 minutes after the last client disconnects, and --shutdown-delay changes it.

Non-Python packages

On classic compute, use an init script that installs the packages when the compute starts, optionally pulling them from a Unity Catalog volume. Serverless compute doesn't support init scripts.

FAQs

Why did my packages disappear after a restart?

Dependencies installed during a session are lost when the cluster restarts. Move them into cluster libraries, an init script or a workspace base environment, and keep requirements files under /Workspace.

Can I use --base-environment with a dedicated cluster?

No. The docs say --base-environment can't be combined with --cluster. On a dedicated cluster, use cluster libraries instead.

The tunnel's Python environment doesn't show up in my IDE. What do I do?

Run echo $DATABRICKS_VIRTUAL_ENV in the IDE's terminal and paste the output as the interpreter path in Python: Select Interpreter.

Which serverless environment version works with the SSH tunnel?

Version 4 or below. The SSH tunnel docs say --base-environment doesn't work with serverless environment version 5 or above.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.