How do I automate hyperparameter tuning at scale?

Summary

• Automating tuning means defining a search space and an objective metric, then letting an optimizer propose and evaluate trials while an experiment tracker records every run. On Databricks, Optuna drives the search and MLflow tracks the results.
• "At scale" means running many trials in parallel across a cluster. Databricks recommends Optuna for the search and Ray Tune for distributed tuning, both included in Databricks Runtime for Machine Learning.
• MLflow autologging captures each trial's parameters, metrics, and model automatically, so you can rank runs and pull the best one programmatically instead of by hand.
• Match parallelism to your cluster: CPU clusters can run more concurrent trials per node, GPU clusters run one trial per node to avoid contention, and distributed trials shouldn't run on autoscaling clusters because parallelism is fixed when the run starts.
• Hyperopt is deprecated and has been removed from recent Databricks Runtime for ML releases. Register the winning model in Unity Catalog and deploy it with Model Serving.

How do I automate hyperparameter tuning at scale?

Automating hyperparameter tuning means turning a manual trial-and-error loop into a repeatable search: you define the search space and the objective metric, an optimizer proposes configurations, each trial trains and scores a model, and an experiment tracker records every result so the best configuration is selected programmatically. Doing it at scale adds one requirement: run many trials in parallel across a cluster instead of one at a time. On Databricks, Optuna drives the search, MLflow tracks and compares every trial, and Ray Tune distributes trials across the cluster when a single node isn't enough.

What automating tuning at scale actually requires

Tuning is only automated if the whole loop runs without hand-holding and the results are reproducible. These are the capabilities to look for, on any platform.

Capability	Why it matters
Flexible search strategy	Define dynamic search spaces and use modern samplers and pruning instead of brute-force grid search, so compute goes to promising configurations.
Distributed, parallel trials	Run many trials at once across cluster nodes to cut wall-clock time, with the scheduler placing trials without manual coordination.
Automatic experiment tracking	Log every trial's parameters, metrics, and artifacts automatically, so runs are comparable, reproducible, and independent of notebook state.
Programmatic model selection	Query tracked runs to find and load the best model by metric, rather than reading a results table by eye.
Governed promotion path	Register the winning model with versioning, lineage, and access control, then deploy it behind an endpoint.
Repeatable orchestration	Schedule or trigger tuning as a job so it reruns on new data without manual steps.

How Databricks approaches hyperparameter tuning at scale

Databricks recommends Optuna for hyperparameter optimization and Ray Tune for distributed tuning, both preinstalled in Databricks Runtime for Machine Learning. Optuna is a lightweight framework that defines dynamic search spaces and applies current optimization algorithms, and it integrates with MLflow to log hyperparameters and metrics for each trial. It replaces Hyperopt, which is no longer maintained and has been removed from Databricks Runtime for ML after version 16.4 LTS ML.
Automation comes from wiring three pieces together. First, you write an objective function that trains a model on trial-suggested parameters and returns the metric to optimize. Second, you turn on MLflow autologging (for example mlflow.sklearn.autolog()) and open a nested MLflow run per trial, so every configuration's parameters, metrics, and model are captured without extra logging code. Third, you let Optuna run the study for a set number of trials. To scale out, MlflowSparkStudy distributes Optuna trials across Spark workers, and the n_jobs parameter sets how many run in parallel; for larger distributed workloads, Ray Tune uses Ray as its backend on the same cluster.
Getting parallelism right is where scale pays off. On CPU clusters, each node can run several trials at once, so you can push concurrency higher. On GPU clusters, run one trial per node so trials don't contend for the GPU. Avoid running distributed trials on an autoscaling cluster, because the level of parallelism is chosen when the run starts and won't adjust as the cluster resizes. Size the cluster to the number of parallel trials you actually want.
When the study finishes, every trial is already in MLflow. Use mlflow.search_runs, ordered by your objective metric, to find and load the best run's model, then register that model in Unity Catalog for versioning, lineage, and access control, and deploy it as a REST endpoint with Model Serving. To keep tuning automated over time, wrap it in a Lakeflow Job so it reruns on a schedule or when new data lands.

Use cases

• Large model-selection sweeps: compare model families and hyperparameters in one Optuna study, distributing many trials across a cluster and letting MLflow rank them.
• Recurring retraining: schedule a tuning job with Lakeflow Jobs so models are re-tuned on fresh data, with each run tracked for audit and comparison.
• GPU-bound deep learning: tune on a GPU cluster with one trial per node, using Ray Tune to coordinate distributed trials.

FAQs

What hyperparameter tuning library does Databricks recommend?

Optuna for the search itself and Ray Tune for distributed tuning, both included in Databricks Runtime for Machine Learning. Optuna integrates with MLflow so trials are tracked automatically.

Is Hyperopt still supported?

No. Hyperopt is deprecated and no longer maintained, and it has been removed from Databricks Runtime for ML after version 16.4 LTS ML. Move tuning workloads to Optuna, and use Ray Tune where you previously used Hyperopt's distributed trials.

How do I run tuning trials in parallel?

Use MlflowSparkStudy to distribute Optuna trials across Spark workers and set n_jobs to the number of parallel trials, or use Ray Tune for distributed tuning. Match parallelism to your cluster: more concurrent trials per node on CPU, one per node on GPU.

How do I pick and deploy the best model after tuning?

Every trial is logged to MLflow, so query runs with mlflow.search_runs ordered by your metric, load the best model, register it in Unity Catalog, and deploy it with Model Serving.