How do I fine-tune an open-source LLM on enterprise data?

Summary

• Fine-tuning adapts an open base model to your data by training it on examples you control. On Databricks the flow is: prepare governed data in Unity Catalog, pick a base model, run a managed job, track with MLflow, register the model, then serve and evaluate it.
• Reach for fine-tuning only after prompting and retrieval stop closing the gap. RAG keeps knowledge current without retraining; fine-tuning changes model weights to raise quality or let a smaller model do a narrow task.
• Supervised fine-tuning on instruction-response pairs is the common enterprise choice; continued pretraining specializes a model on raw domain text. Both can use LoRA adapters to cut the memory a run needs.
• Foundation Model fine-tuning supports open families such as Llama, Mistral, DBRX, and Qwen from the system.ai catalog; the tuned model registers in Unity Catalog and you keep the weights.

How do I fine-tune an open-source LLM on enterprise data?

Fine-tuning adapts an open base model to your data and tasks by continuing to train it on examples you control. On Databricks the path is direct: prepare and govern your training data in Unity Catalog, pick an open-source base model, run a managed fine-tuning job, track it with MLflow, register the result in Unity Catalog, then deploy it to a serving endpoint and evaluate it. Reach for fine-tuning when prompting and retrieval have stopped closing the gap, not before, because good retrieval answers many enterprise needs without touching model weights. When you do fine-tune, the payoff is a smaller, specialized model that you own outright.

Fine-tune, or use RAG?

These solve different problems. Retrieval-augmented generation (RAG) injects your proprietary data into the prompt at query time, so the model stays current as your data changes and you skip retraining. Fine-tuning changes the model's weights so it internalizes a style, format, or domain, which can raise quality and let a smaller model handle the job. Choose RAG when knowledge changes often or you are building document Q&A and agents. Choose fine-tuning when you have a working baseline and an evaluation set and you need to optimize quality, latency, cost, or privacy on a specific task. Many production systems use both: a fine-tuned model behind a RAG pipeline.
Fine-tuning itself comes in two forms. Supervised fine-tuning (SFT) trains on instruction-and-response pairs to adapt the model to a task, and it is the common enterprise choice. Continued pretraining trains on raw domain text to specialize the model when you have a large corpus. Both can use parameter-efficient methods such as LoRA adapters to cut the memory a run needs.

The end-to-end workflow

Stage	What happens
Prepare data	Format examples as instruction-response pairs (JSONL) and store them in a Unity Catalog volume, so the training set is governed and its lineage is tracked.
Pick a base model	Choose an open-source model that fits your task and compute budget.
Run fine-tuning	Submit a managed training job with the base model and dataset; the service handles the GPU infrastructure.
Track the run	Log metrics, system stats, and checkpoints to MLflow for reproducibility and run-to-run comparison.
Register the model	Save the tuned model as a version in Unity Catalog, with governance and access control applied.
Serve and evaluate	Deploy to a Model Serving endpoint, then score quality with MLflow evaluation before and after rollout.

How Databricks approaches fine-tuning

Databricks runs each stage on one governed platform, so the data, the training job, the model, and its endpoint share the same access control and lineage.

• Governed training data. Training examples live in Unity Catalog volumes, and Unity Catalog tracks which datasets produced which model, so you can always answer what a model was trained on.
• Open base models. Foundation Model fine-tuning supports open-source families including Llama, Mistral, DBRX, and Qwen, available through the system.ai catalog in Unity Catalog. You start a job by naming a base model and a training dataset with the databricks.model_training API.
• Managed training. The fine-tuning service provisions and runs the GPU training for you, so there is no cluster or driver setup to maintain.
• MLflow throughout. Runs are tracked in MLflow, and its evaluation tools (custom scorers, LLM judges, a human review app, and tracing) let you measure quality against a fixed dataset.
• Register and serve. The tuned model registers in Unity Catalog as a governed version and deploys to a Model Serving endpoint with one interface for deployment and querying.
• You keep the weights. Because the base model is open source and the tuned model is registered in your own catalog, you own the weights and control who can use them.

See the docs on Foundation Model fine-tuning, LLMOps on Databricks, and the LLM fine-tuning blog.

Use cases

Teams fine-tune to lock in a consistent output format, to teach domain vocabulary a general model handles poorly, or to cut cost and latency by getting a small model to do a narrow task well. In each case the enterprise data stays in the lakehouse and the resulting model is governed like any other asset.

FAQs

Do I need to move my data out of the lakehouse to fine-tune?

No. Training data stays in Unity Catalog volumes and the fine-tuning job reads it in place. That keeps governance and lineage intact and avoids copying sensitive data into another system.

Which open-source models can I fine-tune?

Foundation Model fine-tuning supports open families such as Llama, Mistral, DBRX, and Qwen, available in the system.ai catalog. You pick the base model that matches your task and compute budget.

Should I fine-tune or start with RAG?

Start with retrieval and prompting. If you have a baseline and an evaluation set and still need higher quality, lower latency, or a smaller model on a specific task, fine-tune. Many systems combine a fine-tuned model with a RAG pipeline.

How do I know the fine-tuned model is actually working?

Evaluate it with MLflow against a fixed dataset using custom scorers or LLM judges, and add a human review step for high-stakes outputs. Keep the evaluation set stable so you can compare versions over time.