How do I choose between training from scratch, full fine-tuning, LoRA, and prompt tuning?

Summary

• Start with prompt engineering — the lowest-cost option, requiring no training data or compute — and only customize a model further when quality, latency, cost, or privacy demand it.
• Use LoRA and other parameter-efficient fine-tuning (PEFT) as the default fine-tuning approach: it updates a small set of adapter parameters, reduces compute and memory needs, and helps preserve the base model's general abilities.
• Escalate to full fine-tuning only when a task needs deep behavioral change and you have enough high-quality data; it is more resource-intensive and can risk catastrophic forgetting.
• Training from scratch is rarely necessary — it needs billions of tokens, hundreds of GPUs, and weeks or months of training.
• Mosaic AI Model Training on Databricks supports fine-tuning of foundation models with data from Unity Catalog, and MLflow tracks experiments and registers models.

How do I choose between training from scratch, full fine-tuning, LoRA, and prompt tuning?

These options form a ladder from lowest to highest cost and effort. The general guidance on Databricks is to start with the simplest approach that meets your quality bar and climb only when you must: begin with prompt engineering, adopt parameter-efficient fine-tuning (such as LoRA) as your default when you fine-tune, and reserve full fine-tuning and training from scratch for the cases that truly require them. Establish a working baseline and an evaluation set first, then customize further only when latency, cost, privacy, or quality require it. See the LLM fine-tuning FAQ.

Why customize foundation models on Databricks

• Prompt engineering — lowest cost. Crafting effective prompts requires no training data, no compute, and no weight changes. It is ideal for quick experimentation and simple task adaptation, though it offers the least control over model behavior.
• LoRA / PEFT — the default for fine-tuning. Parameter-efficient fine-tuning updates only a small subset of parameters — typically newly introduced low-rank adapter layers — while keeping the base model frozen. It reduces compute and memory requirements, trains quickly, achieves quality close to full fine-tuning on many tasks, and helps preserve the model's general language abilities. Databricks recommends starting with PEFT and escalating only if it proves insufficient.
• Full fine-tuning — for deep behavioral change. Full fine-tuning updates all model weights on task-specific data. Choose it when a task needs behavioral change that adapter-only updates cannot achieve and you have sufficient high-quality data. It is more resource-intensive and carries a higher risk of catastrophic forgetting, where a model loses broad capabilities after aggressive updates on a narrow dataset.
• Continued pre-training — to add domain knowledge. Continued pre-training enhances a model with additional unstructured text to add new knowledge or focus on a domain. It needs millions of tokens and tens of GPUs, and differs from instruction fine-tuning, which uses structured prompt-response pairs.
• Training from scratch — rarely needed. Building a model from scratch requires billions of tokens, hundreds of GPUs, and weeks or months of training, so most teams fine-tune a pre-trained foundation model to save data, time, and compute. See What is Foundation Model Fine-tuning?.
• One platform for training. Mosaic AI Model Training lets you fine-tune foundation models from the Databricks UI or in Python, using training data from Unity Catalog, and supports instruction fine-tuning, continued pre-training, and chat completion. For fully custom work, you can run your own code on GPU-accelerated clusters. MLflow integration tracks experiments and registers models in Unity Catalog.

Getting started

• Read the LLM fine-tuning FAQ to match an approach to your data, budget, and quality needs.
• Review the LoRA guide for efficient fine-tuning before choosing a PEFT strategy.
• Learn what fine-tuning is and when to use it in What is Foundation Model Fine-tuning?.
• Explore Mosaic AI Model Training to fine-tune a foundation model with your own data.

FAQs

Which approach should I start with?

Start with prompt engineering, since it needs no training data or compute. Move to fine-tuning only after you have a working baseline and an evaluation set and find that latency, cost, privacy, or quality require more customization.

When should I use LoRA instead of full fine-tuning?

Use LoRA (a parameter-efficient method) as your default: it trains a small set of adapter parameters, lowers compute and memory needs, and preserves general capabilities. Choose full fine-tuning only when a task needs deep behavioral change and you have enough high-quality data.

Do I ever need to train a model from scratch?

Rarely. Training from scratch requires billions of tokens, hundreds of GPUs, and weeks or months of training. Fine-tuning a pre-trained foundation model achieves most goals with far less data, time, and compute.

What is catastrophic forgetting?

It is when a model loses its broad, general abilities after being updated too aggressively on a narrow dataset. Parameter-efficient methods such as LoRA help mitigate it by keeping the base model frozen.