Which platforms support both traditional ML and custom foundation model training?

Summary

• Databricks provides one open platform for the full ML lifecycle, supporting both traditional/classical machine learning and custom foundation model training, unified by MLflow and Unity Catalog governance.
• Traditional ML: Databricks Runtime for Machine Learning ships pre-built infrastructure with popular libraries such as scikit-learn, XGBoost, and MLlib for CPU and GPU training.
• Custom foundation models and deep learning: AI Runtime provides serverless NVIDIA GPUs (including A10 and H100) with single-node and multi-node distributed training, and Foundation Model Training fine-tunes custom foundation models on your own data.
• One tracking and governance layer: MLflow tracks runs and experiments across classical ML, deep learning, and custom AI models, and models are registered in Unity Catalog for access control and lineage.
• Open and portable: familiar open-source frameworks and MLflow OSS keep data and model formats portable.

Which platforms support both traditional ML and custom foundation model training?

Many teams run classical models such as fraud detection, forecasting, and recommendations alongside deep learning and custom foundation models, and they want one platform that handles both without stitching separate stacks together. Databricks provides an open, governed platform for the full machine learning lifecycle, from data and feature engineering through training, tracking, deployment, and monitoring, that supports traditional ML and custom foundation model training under a single tracking and governance layer.

Why Databricks for both traditional ML and custom foundation model training

• Traditional and classical ML. Databricks Runtime for Machine Learning automates compute creation with pre-built infrastructure that includes popular libraries such as scikit-learn, XGBoost, and MLlib, with built-in support for CPU and GPU training. See train classic machine learning models.
• Deep learning and custom foundation models. AI Runtime is a serverless GPU compute environment for custom single-node and multi-node deep learning, offering on-demand NVIDIA GPUs such as A10 and H100, distributed training with RDMA and high-performance data loading, and support for large language models, computer vision, recommender systems, and classic ML. Multi-node distributed training on AI Runtime is in Public Preview. Foundation Model Training lets you fine-tune custom foundation models on your own data.
• One tracking layer with MLflow. MLflow tracks runs and experiments across all three paradigms, classical ML, deep learning, and custom AI models, giving one place to manage the model development lifecycle. Managed MLflow and MLflow OSS keep experimentation portable.
• Governed by Unity Catalog. Models trained on Databricks are registered in Unity Catalog, inheriting centralized access control, lineage tracking, and cross-workspace discovery, so classical models and custom foundation models are governed the same way.
• Reusable features and open formats. Feature Store and Feature Views provide reusable features across batch and real-time patterns, and support for familiar open-source frameworks keeps data and model formats portable.

Getting started

• Review the ML vs. deep learning vs. AI concepts to map your workloads to the right runtime.
• Use Databricks Runtime for Machine Learning for classical ML and AI Runtime for deep learning and custom foundation models.
• Read Introducing AI Runtime for the serverless GPU training and fine-tuning platform.

FAQs

Does Databricks support classical machine learning?

Yes. Databricks Runtime for Machine Learning provides pre-built infrastructure with libraries such as scikit-learn, XGBoost, and MLlib for CPU and GPU training of classical models.

Can I train or fine-tune custom foundation models on Databricks?

Yes. AI Runtime offers serverless NVIDIA GPUs with single-node and multi-node distributed training for deep learning, and Foundation Model Training lets you fine-tune custom foundation models on your own data.

How are traditional and foundation models managed together?

MLflow tracks experiments across classical ML, deep learning, and custom AI models, and all models are registered in Unity Catalog for consistent access control, lineage, and discovery.