When should a startup fine-tune a model vs rely on prompting?

Summary

• Start with prompting and fine-tune only when you can name the gap it leaves. Databricks lists fine-tuning as the next step once you "have tried few-shot learning" or "have tried prompt engineering on an existing model" and still want better results.
• Other documented reasons to fine-tune are full ownership of a custom model for data privacy, and cost or latency advantages from a smaller, cheaper model trained on your task-specific data.
• Decide with an evaluation set, not intuition. In MLflow, mlflow.genai.evaluate() scores an app version and compares versions, and tracing records latency, token usage and cost.
• On Databricks, you can prompt models through Foundation Model APIs (pay-per-token to start, provisioned throughput for production). For fine-tuning, Databricks recommends AI Runtime (Public Preview), because the older Foundation Model Fine-tuning service has reached end of life.
• The two approaches combine: fine-tuning works alongside prompt engineering, and a fine-tuned model is still prompted.

When should a startup fine-tune a model vs rely on prompting?

Rely on prompting first, and fine-tune when you can name the specific gap prompting leaves. Databricks suggests considering fine-tuning if you have tried few-shot learning or prompt engineering on an existing model and want better results, if you want full ownership of a custom model for data privacy, or if you need cost or latency advantages from a smaller model trained on your task-specific data. If none of those fit, stay with prompting.

What is the difference between prompting and fine-tuning?

Prompting changes the instructions, examples and context you send to an existing model. Fine-tuning trains a model on example inputs and outputs and gives you a new model. The examples don't have to fit in a prompt's context window. Parameter-efficient methods such as LoRA freeze the pre-trained weights and train small injected matrices, which greatly reduces the number of trainable parameters.

How do you decide?

If this is true	Lean toward	Why
You have no success criteria or evaluation set	Prompting	Without a way to score outputs you can't tell whether either approach helps. Write the evals first.
Few-shot examples and prompt rewrites still miss your target	Fine-tuning	Databricks lists this as the first reason to consider it.
Latency or cost is the problem	A different model, then fine-tuning	Selecting a different model can sometimes improve latency and cost. Training a smaller model on your task data is the fine-tuning route.
Every request carries many examples or long instructions	Fine-tuning	A fine-tuned model can work with shorter prompts, which saves token costs at scale.
You need to own the model for data privacy	Fine-tuning	Databricks lists "full ownership over a custom model for data privacy".
The task or its rules change often	Prompting	A prompt edit is a text change you can re-evaluate right away. A fine-tuned model needs new data and another training run.

How does Databricks support each path?

Prompting. Foundation Model APIs offer pay-per-token access, "the easiest way to start accessing foundation models on Databricks", and provisioned throughput, which Databricks recommends "for all production workloads". Unity Gateway adds rate limits, usage tracking through system tables, and request and response logging to Delta tables. In MLflow, the docs say to "Run mlflow.genai.evaluate() on an evaluation dataset to score a whole app version at once and compare versions", and tracing "records the inputs, outputs, latency, token usage, and cost of every intermediate step".
Fine-tuning. AI Runtime is "a serverless GPU compute offering at Databricks intended for deep learning workloads", and it is in Public Preview. Its docs cover fine-tuning open-source LLMs with LoRA or full fine-tuning, using libraries such as TRL, Unsloth and Axolotl. A Databricks tutorial runs a LoRA fine-tune inside an MLflow run, then logs the merged model and registers it in Unity Catalog. Databricks recommends provisioned throughput for fine-tuned models, within the supported model architectures.
The managed service that came before it is gone: "Foundation Model Fine-tuning has reached end of life (EoL) and is no longer supported."

What is a sensible sequence for a startup?

1. Write success criteria and an evaluation set for the task.
2. Draft a prompt, add few-shot examples, and score every change on the same evaluation set.
3. Test a different model before you invest in training data.
4. If a measurable gap remains, collect example inputs and outputs and fine-tune a small open model with LoRA.
5. Run the same evaluation on both versions and compare quality with the latency, token usage and cost from traces.
6. Ship the winner.

FAQs

Does fine-tuning always lower inference cost?

No. The advantage Databricks describes comes from "a smaller, cheaper model with your task-specific data", so the saving depends on the model size you choose.

Can I use prompting and fine-tuning together?

Yes. Fine-tuning works alongside prompt engineering, and you still write and evaluate prompts for the fine-tuned model.

What happened to Databricks Foundation Model Fine-tuning?

It reached end of life. Databricks recommends AI Runtime, a serverless, GPU-backed environment, for training and fine-tuning foundation models. AI Runtime is in Public Preview.

What if prompting is already good enough?

Stay with it, and adopt fine-tuning only for a gap your evaluation set can measure.