Fine-tuning Guide

TL;DR

• Fine-tuning continues training a pre-trained model on task-specific data so it internalizes behavior, style, or domain patterns - not just retrieves them at inference time.

• Use prompting first, RAG for knowledge, fine-tuning for behavior - most teams over-index on fine-tuning when a better prompt or retrieval pipeline would solve the problem cheaper and faster.

• Full fine-tuning updates every weight and requires significant GPU memory; parameter-efficient methods like LoRA train 0.1–1% of parameters with comparable results on many tasks.

• Data quality dominates model choice - 500 well-curated, diverse examples beat 50,000 noisy ones; garbage in produces a model that confidently does the wrong thing.

• Fine-tuning is not a one-time fix - you need evaluation harnesses, versioning, regression tests, and a plan for when base models update or your data drifts.

Why This Matters

Prompt engineering gets you surprisingly far. RAG handles knowledge that changes weekly. But some problems require the model to behave differently - consistently following a JSON schema, speaking in your brand voice, classifying intents with 99% accuracy, or routing tool calls in a specific format.

Fine-tuning bakes these patterns into the model's weights. At inference time, you get lower latency (shorter prompts), lower cost (fewer tokens), and more reliable outputs than asking a base model to follow complex instructions every time.

The mistake most teams make is reaching for fine-tuning before exhausting simpler options, or fine-tuning when they actually need retrieval. Understanding where fine-tuning fits - and where it doesn't - saves months of engineering time and thousands in GPU costs.

The Problem Fine-Tuning Solves

Base LLMs are generalists. They were trained on internet-scale text to predict the next token, not to classify your support tickets, extract entities from medical notes, or generate SQL in your proprietary schema dialect.

Three gaps remain even with good prompting:

Behavioral consistency. A base model might follow your output format 85% of the time. In production, 85% is a outage. Fine-tuning on examples of correct behavior pushes reliability toward 98%+.

Domain language and reasoning patterns. Legal, medical, and financial text use vocabulary and reasoning structures underrepresented in pre-training. Fine-tuning on domain examples teaches the model how experts in that field write and think.

Latency and cost at scale. A 2,000-token system prompt with 20 few-shot examples works in a demo. At 10,000 requests per minute, you're paying to re-send that context on every call. A fine-tuned model may need a 200-token prompt instead.

Fine-tuning does not solve knowledge freshness (use RAG), one-off creative tasks (use prompting), or problems where you have no labeled data (collect data first).

What Is Fine-Tuning?

Fine-tuning takes a pre-trained language model and continues training it on a smaller, curated dataset for a specific purpose. The pre-trained weights provide general language understanding; the fine-tuning data teaches task-specific behavior.

Common variants:

The process is standard supervised learning: forward pass, compute loss (usually cross-entropy on target tokens), backpropagate, update weights. The difference from training from scratch is that you start from weights that already understand language, so you need far less data and compute.

How Fine-Tuning Works

At a high level, fine-tuning is next-token prediction on your dataset. For an instruction-tuning example:

{
  "messages": [
    {"role": "system", "content": "Extract entities as JSON."},
    {"role": "user", "content": "John Smith ordered 3 widgets on Jan 5."},
    {"role": "assistant", "content": "{\"person\": \"John Smith\", \"quantity\": 3, \"date\": \"Jan 5\"}"}
  ]
}

During training, the model sees the system and user messages as context and learns to predict the assistant tokens. Loss is computed only on the assistant's response tokens (not the prompt), so the model learns to generate correct outputs given inputs.

Key hyperparameters that matter in practice:

LoRA injects trainable low-rank matrices into attention layers while keeping the base model frozen, dramatically reducing fine-tuning memory and compute.

Source: Microsoft Research

Architecture

A fine-tuning pipeline has four subsystems:

The training and serving paths must agree on base model version, tokenizer, chat template, and (for PEFT) adapter configuration. A mismatch between training and inference chat templates is one of the most common silent bugs.

Step-by-Step Flow

Step 1: Decide if you need fine-tuning. Run this decision tree before touching GPUs:

1. Can prompting + few-shot solve it? → Try that first.
2. Is the problem missing knowledge? → Use RAG.
3. Is the problem inconsistent behavior/format on a stable task? → Fine-tuning candidate.
4. Do you have 200+ high-quality labeled examples? → Proceed. Fewer than 100 → collect more data.

Step 2: Define the task and success criteria. Write 20 golden examples with expected outputs. Define metrics: exact match, F1, JSON validity rate, human preference score. Without this, you cannot tell if training helped.

Step 3: Prepare the dataset. Format as chat messages or instruction-response pairs. Split 90/10 train/eval. Deduplicate near-duplicates. Remove examples with errors - the model will learn them.

Step 4: Choose full fine-tuning vs. PEFT. Full fine-tuning for small models (<7B) when you have GPU budget and need maximum task performance. LoRA or QLoRA for models >7B or limited hardware.

Step 5: Train with conservative hyperparameters. Start with 1 epoch, low learning rate, evaluate after each epoch. Watch eval loss - if train loss drops but eval loss rises, you're overfitting.

Step 6: Evaluate rigorously. Run your golden set, compare against base model + prompt baseline. Test edge cases, adversarial inputs, and out-of-distribution examples.

Step 7: Merge, quantize, deploy. For LoRA, merge adapters into base weights or serve adapter separately. Quantize (GPTQ, AWQ) for inference if latency/cost matters.

Step 8: Monitor in production. Log inputs, outputs, latency. Sample for human review. Retrain when base model updates or performance drifts.

Fine-Tuning vs. RAG vs. Prompting

This is the decision every AI team faces. Here is an honest comparison:

Use prompting when: You're prototyping, the task changes weekly, or you need results today.

Use RAG when: Answers must cite sources, knowledge updates frequently, or you have large document corpora the model shouldn't memorize.

Use fine-tuning when: You need reliable structured output, domain-specific classification/extraction, consistent tone, or you want to eliminate a 2,000-token system prompt.

Combine them. Production systems often fine-tune for behavior and use RAG for knowledge. A fine-tuned model that outputs valid JSON and follows your schema, plus RAG for current product documentation, is a common and effective pattern.

Real Production Example

A fintech company needs to classify transaction dispute emails into 12 categories with 97%+ accuracy. Prompting GPT-4 achieves 91% with a 1,500-token prompt. They fine-tune Llama 3.1 8B with LoRA on 3,200 labeled emails.

from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

# Load and format data
dataset = load_dataset("json", data_files={
    "train": "disputes_train.jsonl",
    "test": "disputes_eval.jsonl",
})

def format_example(example):
    return {
        "text": tokenizer.apply_chat_template(
            [
                {"role": "system", "content": "Classify the dispute email. Reply with only the category name."},
                {"role": "user", "content": example["email_body"]},
                {"role": "assistant", "content": example["category"]},
            ],
            tokenize=False,
        )
    }

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype="bfloat16",
    device_map="auto",
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: ~41M || all params: ~8B || trainable%: 0.51%

train_dataset = dataset["train"].map(format_example)
eval_dataset = dataset["test"].map(format_example)

training_args = TrainingArguments(
    output_dir="./dispute-classifier",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    warmup_ratio=0.05,
    logging_steps=10,
    eval_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    bf16=True,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    processing_class=tokenizer,
)

trainer.train()
model.save_pretrained("./dispute-classifier/lora_adapter")

After training: 97.3% accuracy on holdout, 40ms inference latency (vs. 800ms for GPT-4 API), and a 150-token prompt instead of 1,500. They serve with vLLM and A/B test against the prompt baseline for two weeks before full rollout.

Design Decisions

⚠ Common Mistakes

1. Fine-tuning when RAG would work. If your problem is "answer questions about our docs," fine-tuning memorizes a snapshot of docs that goes stale. Use RAG.

2. Training on too few examples. Under 100 examples rarely produces reliable results. Collect more data or use few-shot prompting instead.

3. Including prompt tokens in the loss. Loss should be computed only on the response tokens. Training on the prompt teaches the model to predict the prompt, not generate good outputs.

4. Mismatched chat templates. Training with one template and serving with another causes garbled outputs. Pin the tokenizer chat template in your training config.

5. No eval set. Training without a held-out eval set means you cannot detect overfitting until production users complain.

6. Chasing benchmark scores over business metrics. 95% on your eval set means nothing if the 5% failures are on your highest-value customer segments.

7. Catastrophic forgetting from high learning rates. Learning rates above 5e-5 on full fine-tuning often destroy general capabilities. Start conservative.

Where It Breaks Down

Small or noisy datasets. Fine-tuning amplifies patterns in your data - including errors, biases, and annotation inconsistencies. With under 200 examples, variance between runs is high.

Rapidly changing tasks. If your output schema changes monthly, you'll spend more time retraining than you would maintaining a prompt.

Knowledge-intensive Q&A. Fine-tuning cannot inject 50,000 pages of product documentation into model weights efficiently. The model will hallucinate details it half-remembered from training.

Multi-turn conversations with long context. Fine-tuning on single-turn examples doesn't teach multi-turn coherence. You need multi-turn training data or RAG with conversation history.

Regulatory and audit requirements. Weight changes are opaque. If you need to prove why the model produced a specific output, RAG with citations is more auditable.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Always maintain a prompt-based baseline and A/B test fine-tuned models against it. Fine-tuning is not automatically better - measure it.

Ecosystem

Related Technologies

• LoRA: The default parameter-efficient fine-tuning method - start here for most projects.

• QLoRA: Fine-tune 70B models on a single GPU with 4-bit quantization.

• PEFT: HuggingFace library unifying LoRA, adapters, and prefix tuning.

• RLHF: Align model outputs to human preferences using reinforcement learning.

• DPO: Simpler preference alignment without a reward model.

• RAG: Retrieve knowledge at query time instead of baking it into weights.

• Prompt Engineering: The first tool to try before fine-tuning.

Learning Path

Prerequisites: Large Language Models · Tokens · Prompt Engineering

Next topics: LoRA · PEFT · RLHF · DPO

Estimated time: 55 min · Difficulty: Intermediate

FAQs

When should I fine-tune instead of using RAG?

Fine-tune when you need consistent behavior (output format, classification, tone) on a stable task. Use RAG when you need current knowledge from a large, changing document corpus. They solve different problems and are often combined.

When should I fine-tune instead of prompting?

Fine-tune when prompting with few-shot examples doesn't reach your reliability threshold, your system prompt is too long (adding latency and cost), or you need the behavior baked in for offline/edge deployment. Prompt first - always.

How much data do I need to fine-tune?

For classification and extraction: 200–1,000 examples per class often suffices. For open-ended generation: 1,000–10,000 high-quality examples. More data helps, but quality matters more than quantity.

How long does fine-tuning take?

LoRA on a 7B model with 5,000 examples: 1–4 hours on a single A100. Full fine-tuning on a 70B model: 12–48 hours on 8×A100. QLoRA reduces memory requirements but not always wall-clock time.

Will fine-tuning cause catastrophic forgetting?

Yes, especially with full fine-tuning and high learning rates. Mitigate with lower learning rates, fewer epochs, LoRA (which freezes base weights), and evaluation on general-capability benchmarks.

Can I fine-tune an API model like GPT-4?

OpenAI and other providers offer managed fine-tuning APIs. You upload JSONL data, they handle training and hosting. Tradeoff: less control, higher per-token cost, but no GPU infrastructure.

Should I merge LoRA weights or serve the adapter separately?

Merge for single-model deployment simplicity. Serve separately when you need multiple adapters on one base model (multi-tenant) or want to swap adapters without reloading the full model.

How do I evaluate a fine-tuned model?

Hold out 10% of data for automated metrics. Run a golden test set against both the fine-tuned model and your prompt baseline. Include human evaluation for generative tasks. A/B test in production before full rollout.

Can I fine-tune for multiple tasks at once?

Yes - multi-task fine-tuning mixes examples from different tasks in one dataset. Works well for general assistant behavior. For specialized tasks, single-task fine-tuning usually performs better.

What base model should I choose?

Match model size to task complexity and latency budget. Llama 3.1 8B handles most classification and extraction tasks. Use 70B+ only when smaller models fail eval after proper fine-tuning. Start small.

How often should I retrain?

Retrain when: base model version updates, your eval metrics drift in production, your task definition changes, or you accumulate 20%+ new training data. Monthly or quarterly is common for stable tasks.

Is fine-tuning safe for models handling PII?

Fine-tuned models can memorize training data. Audit your training set for PII, run membership inference tests, and consider differential privacy techniques for sensitive domains. RAG with access controls may be safer for PII-heavy workloads.

References

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• Hugging Face PEFT Documentation
• Microsoft LoRA Paper

Further Reading

• QLoRA Paper
• bitsandbytes GitHub
• Hugging Face Fine-tuning Guide

Summary

• Fine-tuning teaches the model how to behave; RAG provides what to know; prompting is the fastest way to prototype. - Start with prompting, add RAG for knowledge, fine-tune only when you need reliable behavior on a stable task with sufficient labeled data. - Data quality and evaluation matter more than model size or training tricks.

• Parameter-efficient methods (LoRA, QLoRA) make fine-tuning accessible on consumer hardware for models up to 70B. - Production fine-tuning requires versioning, regression testing, and ongoing monitoring - it's not a one-time training run.