Fine-Tuning an LLM
— and when you even need it
Before you fine-tune a model — stop. In 90% of cases good prompting or RAG will solve the problem faster and cheaper. But when you genuinely need consistent behavior, a fixed format or a unique style — fine-tuning is the tool. In this guide: when to fine-tune and when not, what LoRA and QLoRA are, how to prepare a dataset, and with which tools.
What fine-tuning actually is
A language model comes pre-trained on a vast amount of text — it "knows" a lot. Fine-tuning is an additional stage where you keep training the model on your own small, focused dataset, to shape its behavior : the tone, the format, the domain or the specific task.
It is important to understand the distinction: fine-tuning teaches the model how to behave — not what to know. If your goal is to feed in current or private knowledge (company documents, a product catalog), RAG is the right tool, not fine-tuning.
When to fine-tune — and when really not to
The rule: always start with the cheap and fast option. Climb the ladder only when the previous step was not enough.
When you have a clear, repeated task with hundreds-to-thousands of quality examples; when you want a small, local model (7B) to behave like a large one on a narrow task; or when the prompt has become long and expensive and every request pays for it. Then tuning saves tokens and money over time.
The most common mistake. Fine-tuning on facts causeshallucinations (the model "fills in" what it did not remember) and goes stale the moment the information changes. For knowledge — always RAG.
LoRA & QLoRA — tuning without giant hardware
Full fine-tuning of all the model weights requires very expensive hardware. LoRA (Low-Rank Adaptation) solves this: instead of updating all the weights, you freeze the original model and train only small adapter layers. The result — less than 1% of the parameters are trained, yet the quality is very close to full training.
QLoRA adds quantization: you load the base model in 4-bit and train the adapters on top of it. This lets you fine-tune a 7B–13B model on a single GPU (even a consumer card with 16–24GB). This is why tuning became accessible to everyone.
| Method | GPU memory | When |
|---|---|---|
| Full fine-tune | Very high | Very few cases |
| LoRA | Medium | A good default |
| QLoRA | Low (single GPU) | Budget / limited hardware |
Dataset preparation — this is where success is decided
90% of fine-tuning success is data quality, not the hyperparameters. A small, clean dataset beats a large, messy one every day.
The common format is chat-style conversations — each example is a pair of input and desired output:
{"messages": [
{"role": "system", "content": "You are a polite support agent for company X."},
{"role": "user", "content": "How do I return a product?"},
{"role": "assistant", "content": "Happy to help! The return is simple..."}
]}- Quality over quantity — 500–1,000 excellent examples beat 10,000 mediocre ones. Every example should be exactly the "behavior" you want to see.
- Consistency — if you wanted a certain format, make every example demonstrate it exactly. Contradictions in the data = a confused model.
- Diversity — cover the real range of inputs, including edge cases, so the model generalizes.
- Splitting — keep 10–15% of the examples for validation to measure and detect overfitting.
Tools & process — how tuning is actually done
- Unsloth — the popular library in 2026 for fast, memory-efficient tuning. ~2× faster, beginner-friendly, with ready-made Colab notebooks.
- Axolotl — a config-based framework (YAML) for production-grade tuning, flexible and supporting many models.
- Hugging Face TRL / PEFT — the official libraries for LoRA and training. See the Hugging Face guide.
- Managed cloud — OpenAI and Google offer managed fine-tuning via API: you upload a dataset and they train. Less control, but zero DevOps.
The typical process: prepare a dataset → choose a base model (e.g. Llama/Qwen/Mistral) → train LoRA/QLoRA → evaluate against validation → merge the adapter or run it on top of the model → deploy (e.g. via Ollama locally).
Tips & common pitfalls
- Start small — tune on a 7B model before you spend money on a large one. If it does not work on the small one, the problem is the data.
- Measure, do not guess — define a clear success metric and compare against the un-tuned model and against RAG. Sometimes you will discover you did not need to tune at all.
- Beware of overfitting — too many epochs and the model "memorizes" the data and loses its ability to generalize. Stop when the validation loss stops dropping.
- Catastrophic forgetting — aggressive tuning can make the model "forget" general abilities. LoRA eases this because the original model is frozen.
- Track the base model — document which model, which version and which hyperparameters. Reproducibility is half the work in production.
Most winning systems do not fine-tune at all, or they combine: RAG for knowledge + light fine-tuning for behavior. RAG brings the current facts, and a small amount of tuning polishes the tone and format. Start with RAG (RAG guide), and consider tuning only when the behavior is still not right.