Fine-Tuning Guide

TL;DR

• Fine-tuning changes model parameters during training. Inference later uses those changed parameters; it does not search the training dataset.
• Prompting changes request-time context, not weights. It is the cheapest, fastest, and most reversible first intervention.
• RAG supplies retrieved knowledge at inference time. Use it for private, attributable, access-controlled, or changing facts. Fine-tuning is not a database update mechanism.
• Supervised fine-tuning (SFT) is mainly a behavior tool: format compliance, classification boundaries, tool-call conventions, domain language, style, and shorter prompts.
• Full fine-tuning updates all trainable weights. LoRA, QLoRA, and other PEFT methods update small adapters and are usually the practical starting point for open-weight LLMs.
• A training run is not evidence of improvement. Compare against an unchanged base-model prompt baseline on frozen, representative, contamination-free evaluation sets.
• Production work is mostly data and lifecycle engineering: provenance, privacy, splits, deduplication, artifact versioning, canary deployment, monitoring, rollback, and retraining policy.

On this page

• Why This Matters
• The Problem Fine-Tuning Solves
• How We Got Here
• What Is Fine-Tuning?
• How Fine-Tuning Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Use Fine-Tuning
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

Base large language models are general-purpose next-token predictors. An application, however, may require a narrow contract: map support messages into a fixed taxonomy, produce a canonical JSON structure, follow a tool protocol, write in a regulated template, or execute a stable transformation with predictable latency. Repeating dozens of examples in every prompt can work, but it consumes context, increases time to first token, and leaves behavior sensitive to prompt wording.

Fine-tuning can move a repeated instruction-and-example pattern from request context into model parameters. That can make a smaller model viable, shorten prompts, improve task consistency, and permit local or private deployment. It can also make failures harder to diagnose because the behavior is distributed across weights rather than visible in a prompt.

The distinction between behavior and knowledge is therefore operationally important. A fine-tuned model may learn how an insurance adjuster structures a decision. It should not be trusted to memorize the current policy wording, customer balance, price list, or compliance rule. Those facts belong in an authoritative system, tool call, or RAG pipeline where they can be updated, authorized, cited, and audited.

The Problem Fine-Tuning Solves

Fine-tuning addresses persistent mismatch between a model's default probability distribution and the behavior an application needs. Typical symptoms include:

• a classifier confuses organization-specific categories even after clear few-shot examples;
• output follows a schema most, but not all, of the time;
• a long system prompt and many demonstrations dominate input cost;
• a general model uses the wrong terminology or document structure;
• a small deployable model needs to imitate a validated behavior from a larger model;
• tool-selection or refusal boundaries are stable but poorly represented in pretraining.

These are statistical behavior gaps. Fine-tuning provides many labeled demonstrations and updates parameters so desired tokens become more probable in the relevant contexts.

It does not inherently provide truth, citations, authorization, or freshness. Training on a product catalog may increase recall of catalog text, but it creates stale parametric memory with uncertain provenance. Retrieval keeps the catalog outside the model and injects the authorized current record when a request arrives.

Another boundary is deterministic enforcement. If output must always match a JSON Schema, constrained decoding and application validation are stronger guarantees than training. Fine-tuning may reduce retries, but code should still enforce the contract.

How We Got Here

Early NLP systems trained separate task-specific models for classification, translation, or extraction. Transfer learning changed that workflow: pretrain a broad model once, then adapt it with a smaller labeled dataset. Transformer language models made this especially effective because next-token pretraining produced reusable language representations.

The first common adaptation pattern updated every parameter. As model sizes grew from millions to billions of parameters, full fine-tuning required increasingly expensive optimizer state, gradients, distributed checkpoints, and model copies. Adapter methods then inserted small trainable modules while freezing the base model. LoRA represented weight updates as low-rank matrices, substantially reducing trainable parameters and storage.

QLoRA further reduced training memory by keeping the frozen base model quantized while computing gradients through LoRA adapters. Frameworks such as Hugging Face PEFT made these methods composable. Managed fine-tuning APIs provided another path: upload validated examples and let a provider operate the training and serving infrastructure.

Instruction tuning broadened SFT from single tasks to collections of instruction-response pairs. Preference methods followed. RLHF learns a reward signal from human comparisons and optimizes a policy; DPO directly trains on chosen and rejected responses. These techniques solve alignment objectives beyond ordinary SFT, but they do not remove the need for task-specific evaluation.

Diagram: Evolution of model adaptation

timeline
    title From task models to parameter-efficient alignment
    Task-specific era : Separate model per NLP task
    Transfer learning : Pretrain once, fine-tune downstream
    Full LLM tuning : Update all model parameters
    PEFT : Adapters and low-rank updates
    QLoRA : Quantized base with LoRA training
    Preference tuning : RLHF and DPO pipelines

Adaptation methods evolved primarily to reduce compute and improve control, not to turn weights into a reliable knowledge store.

What Is Fine-Tuning?

Fine-tuning is additional optimization of a pretrained model on a target dataset. Given parameters (\theta), input tokens (x), and target tokens (y), supervised training minimizes token-level negative log likelihood:

[ \mathcal{L}(\theta) = -\sumt m_t \log p\theta(yt \mid x, y{<t}) ]

The mask (m_t) controls which tokens contribute to loss. For assistant-style SFT, teams commonly mask system and user tokens and train on assistant tokens. Some objectives deliberately train on the entire sequence, but that must be an explicit choice rather than an accidental collator setting.

Training versus inference

Training performs forward passes, computes loss, backpropagates gradients, and updates model or adapter parameters. It is expensive, stateful, stochastic, and produces a versioned artifact.

Inference freezes those parameters and performs forward generation for each request. Inference can use a prompt, retrieved context, tools, caches, and decoding constraints, but normally performs no gradient updates.

Confusing the phases leads to bad designs. Adding a document to a fine-tuning dataset does not make a running deployment immediately know it. The model must be retrained, evaluated, registered, and redeployed. Adding a document to a retrieval index can make it available without changing model weights.

Parametric versus retrieved knowledge

Parametric knowledge is represented indirectly in model weights. It is compressed, difficult to enumerate, difficult to delete selectively, and not inherently attributable. Fine-tuning changes this parametric state.

Retrieved knowledge remains in an external source and enters the context for a specific inference request. It can carry source IDs, timestamps, permissions, and citations. That makes retrieval the default for facts that change or require governance.

Fine-tuning can teach the model to use retrieved evidence, cite in a required format, or abstain when no evidence exists. It should not replace retrieval itself.

Main adaptation families

How Fine-Tuning Works

An SFT example is serialized using the exact tokenizer and chat template expected at serving time:

{
  "messages": [
    {
      "role": "system",
      "content": "Classify the dispute. Return one allowed label."
    },
    {
      "role": "user",
      "content": "The ATM dispensed no cash but my account was charged."
    },
    { "role": "assistant", "content": "cash_withdrawal_issue" }
  ]
}

The tokenizer maps text and control tokens into IDs. Batches are padded or packed. The model predicts a distribution over the vocabulary for each target position. Cross-entropy loss measures the difference between predicted and expected next tokens. Backpropagation computes gradients, and an optimizer applies updates.

Full fine-tuning updates the original matrices. For a weight matrix (W), LoRA freezes (W) and learns a low-rank delta:

[ W' = W + \frac{\alpha}{r}BA ]

where rank (r) is much smaller than the dimensions of (W). The trainable matrices (A) and (B) are the adapter. QLoRA stores the frozen base weights in low precision, dequantizes as needed for computation, and trains LoRA parameters at higher precision.

Source: LoRA: Low-Rank Adaptation of Large Language Models

Important controls include learning rate, effective batch size, sequence length, rank, target modules, warmup, epochs, weight decay, gradient clipping, and precision. Their correct values depend on model, dataset size, task, and framework. Published ranges are starting hypotheses, not production defaults.

Architecture

A production fine-tuning platform separates data, training, evaluation, registry, deployment, and feedback. The training job must not silently read mutable production tables; it should consume immutable, versioned snapshots with documented provenance.

Diagram: Fine-tuning system architecture

flowchart LR
    S[Approved sources] --> P[Data pipeline]
    P --> V[Versioned dataset]
    V --> T[Training job]
    B[Pinned base model] --> T
    T --> A[Model or adapter]
    A --> E[Evaluation gates]
    G[Golden sets] --> E
    E --> R[Artifact registry]
    R --> C[Canary serving]
    C --> M[Monitoring]
    M --> F[Reviewed feedback]
    F --> P

Immutable lineage connects each deployment to source data, code, base weights, tokenizer, hyperparameters, and evaluation results.

The artifact manifest should contain:

• base model identifier, revision, license, and checksum;
• tokenizer revision and chat-template hash;
• dataset snapshot, schema, and filtering code revision;
• training code/container version and random seeds;
• full or PEFT configuration and numerical precision;
• evaluation suite revision, thresholds, and results;
• adapter checksum, merged-model checksum if applicable, and quantization recipe.

Serving must reconstruct the same input format used for training. A tokenizer or chat-template mismatch can add, omit, or reorder control tokens and produce a large regression with no infrastructure error.

Step-by-Step Flow

1. Define the behavioral contract. State the input population, allowed outputs, latency budget, unacceptable errors, and business metric. “Improve responses” is not measurable.
2. Build a baseline before collecting training data. Evaluate the base model with a strong zero-shot and few-shot prompt. Add constrained decoding or deterministic validation where applicable.
3. Classify the gap. Missing current facts indicate RAG or tools. An unstable schema may need prompt and decoder changes. A stable behavior gap may justify fine-tuning.
4. Create frozen evaluation sets. Include ordinary, difficult, safety, multilingual, long-tail, and out-of-distribution examples. Keep them out of prompts, training data, and synthetic generation inputs.
5. Collect and govern demonstrations. Define annotation instructions, adjudication, provenance, consent, retention, PII handling, and class balance.
6. Clean and split by entity or time. Remove exact and semantic duplicates. Split related conversations, customers, documents, and templates together to prevent leakage.
7. Choose an adaptation method. Start with LoRA for many open-weight SFT tasks. Use QLoRA when base-weight memory is the constraint. Use full fine-tuning only when evidence justifies its compute and operational cost.
8. Run a small experiment. Train a short run, inspect samples, verify loss masks, and confirm that checkpoints can be loaded by the intended serving stack.
9. Sweep only meaningful variables. Compare learning rate, rank, epochs, and data mixtures against the same frozen evaluations. Avoid selecting on one aggregate score.
10. Apply release gates. The candidate must beat the baseline on primary metrics while staying within safety, latency, cost, and general-capability regression budgets.
11. Deploy to shadow or canary traffic. Preserve request routing so candidate and baseline see comparable populations. Define automatic rollback conditions.
12. Monitor and curate feedback. Review failures before adding them to training. Production thumbs-up data without sampling controls is biased and vulnerable to manipulation.

Diagram: Fine-tuned model lifecycle

stateDiagram-v2
    [*] --> Baseline
    Baseline --> DatasetReady: gap is behavioral
    DatasetReady --> Training
    Training --> Evaluation
    Evaluation --> Rejected: gate fails
    Rejected --> DatasetReady: correct data or recipe
    Evaluation --> Registered: all gates pass
    Registered --> Canary
    Canary --> RolledBack: SLO or quality breach
    Canary --> Production: rollout succeeds
    Production --> DatasetReady: reviewed drift

A candidate is releasable only after offline gates and an online rollout; low training loss is not a deployment state.

Real Production Example

Consider a payments company classifying dispute messages into 12 operational queues. The current large API model with a 1,400-token few-shot prompt reaches 94.1% macro F1. The target is at least 96% macro F1, no class below 90% recall, malformed-label rate below 0.1%, p95 latency under 120 ms, and no material regression on an adversarial safety set.

Dataset preparation

The team exports adjudicated cases only. Raw messages are scrubbed for account numbers and direct identifiers. Each example carries a stable case-group ID, timestamp, language, channel, label, annotator confidence, and source policy version. Threads from the same case stay in one split. The test set uses the newest time window to detect temporal drift.

They remove duplicate templates and near-duplicates before splitting, because repeated bank notification text would otherwise inflate accuracy. Rare classes are sampled deliberately, but the natural-distribution test set remains untouched. Two reviewers adjudicate ambiguous labels using the routing policy. The final snapshots contain 7,600 training examples, 950 validation examples, 1,400 natural-distribution test examples, and 300 challenge examples.

from datasets import load_dataset
from transformers import AutoTokenizer

MODEL_ID = "meta-llama/Meta-Llama-3.1-8B-Instruct"
ALLOWED = {
    "cash_withdrawal_issue", "card_not_present", "cash_advance",
    "duplicate_charge", "fraud", "merchant_dispute",
    "refund_missing", "subscription", "transfer_issue",
    "cash_deposit_issue", "card_present", "other",
}

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, revision="pinned-revision")
dataset = load_dataset(
    "json",
    data_files={"train": "train.jsonl", "validation": "validation.jsonl"},
)

def validate_and_format(row):
    label = row["label"].strip()
    if label not in ALLOWED:
        raise ValueError(f"Unknown label: {label}")
    messages = [
        {
            "role": "system",
            "content": "Route the dispute. Return exactly one allowed label.",
        },
        {"role": "user", "content": row["redacted_message"]},
        {"role": "assistant", "content": label},
    ]
    return {
        "text": tokenizer.apply_chat_template(
            messages, tokenize=False, add_generation_prompt=False
        )
    }

formatted = dataset.map(validate_and_format, remove_columns=dataset["train"].column_names)

Before training, a test confirms that no case-group ID crosses splits and no normalized test message appears in training. The team also hashes the formatter and rendered chat template.

LoRA/PEFT training

The initial recipe freezes the 8B base model and targets attention projection modules. The team trains one epoch first, then compares a second epoch only if validation and per-class metrics improve.

import torch
from peft import LoraConfig
from transformers import AutoModelForCausalLM, TrainingArguments
from trl import SFTTrainer

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    revision="pinned-revision",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

args = TrainingArguments(
    output_dir="artifacts/dispute-router",
    num_train_epochs=1,
    learning_rate=2e-4,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    warmup_ratio=0.05,
    max_grad_norm=1.0,
    bf16=True,
    eval_strategy="steps",
    eval_steps=100,
    save_steps=100,
    save_total_limit=2,
    load_best_model_at_end=True,
    report_to=["mlflow"],
)

trainer = SFTTrainer(
    model=model,
    args=args,
    peft_config=peft_config,
    train_dataset=formatted["train"],
    eval_dataset=formatted["validation"],
    processing_class=tokenizer,
    dataset_text_field="text",
    max_seq_length=1024,
)
trainer.train()
trainer.save_model("artifacts/dispute-router/adapter")

Library APIs change, so the training container pins compatible Transformers, TRL, PEFT, Accelerate, CUDA, and driver versions. A smoke test loads the saved adapter into the exact serving image.

Evaluation gates

The evaluator decodes greedily, normalizes only surrounding whitespace, and treats every other variation as malformed. It reports macro F1, per-class precision and recall, calibration proxies, malformed rate, latency, token count, and safety failures. Bootstrap confidence intervals compare the candidate with the prompt baseline.

Release gates are conjunctive:

1. macro F1 is at least 96% and improves over baseline by a predeclared margin;
2. every class has at least 90% recall;
3. no language or channel slice regresses beyond its budget;
4. challenge-set and safety failures do not increase;
5. p95 latency and projected cost satisfy service limits;
6. a blinded reviewer accepts sampled disagreements;
7. the adapter passes extraction and memorization probes for sensitive strings.

The first candidate reaches 96.4% macro F1 but only 84% recall on cash-advance disputes. It is rejected. Review shows inconsistent annotation between “cash advance” and “cash withdrawal.” The team corrects the guideline, adjudicates affected records, and retrains. The next candidate reaches 96.8% macro F1 with all class-recall gates satisfied.

The model runs in shadow mode, then receives 5%, 25%, and 50% of traffic. Routing errors and low-confidence disagreements go to human review. The baseline remains available for rollback. This is a successful migration because the candidate passes explicit gates—not because its aggregate benchmark looks better.

Design Decisions

Full fine-tuning or PEFT

Full fine-tuning offers maximum freedom to change the model but requires gradients and optimizer state for all parameters, larger checkpoints, and a complete model artifact per variant. It can be justified for smaller models, large high-quality datasets, substantial domain shift, or when adapter results demonstrably plateau below requirements.

LoRA is the usual first experiment for open-weight instruction tuning. Adapter artifacts are small and can share a base model. QLoRA helps when GPU memory is insufficient for the unquantized frozen base, though quantization kernels and numerical behavior add complexity. PEFT also includes prompt tuning, prefix tuning, and adapters; method choice should follow measured quality and serving support.

Open-weight or managed API

Open-weight training provides control over weights, data locality, serving, decoding, and per-request cost. It also transfers licensing, GPU operations, patching, security, and capacity planning to your team.

A managed API reduces infrastructure work and usually accepts provider-specific JSONL. Production patterns remain the same: validate records locally, keep immutable dataset IDs, create the job idempotently, poll with backoff, capture job and model identifiers, run external evaluations, and promote the result through your own registry. Provider training success is not your quality gate.

Adapter serving or merging

Separate adapters support multiple tasks or tenants on a shared base and simplify storage. They require serving software that can load, cache, route, and evict adapters safely. Merging an adapter into the base simplifies single-model serving and may improve compatibility, but creates a full artifact and loses cheap adapter swapping. Quantize after merging only with an evaluation of the resulting artifact.

Data mixture

Oversampling rare classes may improve macro metrics while damaging natural-distribution calibration. Replaying general instruction data can reduce forgetting but may dilute task learning. Keep both balanced challenge evaluations and natural traffic-weighted evaluations. Record mixture weights as part of the model version.

When should I use this?

Comparisons

Common Mistakes

1. Fine-tuning to update facts. Prices, policies, inventory, customer records, and current research belong in RAG or tools. Weight updates are stale, non-attributable, and difficult to delete.
2. Skipping the prompt baseline. Without a strong baseline, a team cannot show that training complexity bought meaningful quality, latency, or cost improvement.
3. Leaking evaluation examples. Near-duplicate templates, shared threads, synthetic paraphrases, or entity overlap can make test scores meaningless.
4. Splitting rows randomly. Group by customer, conversation, source document, or time when those relationships can leak signal.
5. Training on unreviewed production feedback. Feedback is selection-biased, can encode user attacks, and often lacks a reliable target response.
6. Using the wrong loss mask. Inspect tokenized examples and labels. Do not assume a data collator masks prompt tokens as intended.
7. Changing templates between training and serving. Pin tokenizer and chat-template revisions and test rendered inputs byte-for-byte.
8. Selecting by training loss. Lower loss can coincide with memorization, worse safety, or poorer out-of-distribution behavior.
9. Reporting only an aggregate metric. Per-class and per-slice failures can be operationally unacceptable despite a higher average.
10. Merging or quantizing without re-evaluation. The deployed binary is the product; every transformation can change outputs.
11. Treating LoRA as risk-free. Frozen base weights reduce some forgetting but adapters can still produce unsafe, memorized, or task-overfit behavior.
12. No rollback artifact. Keep the previous model, prompt, routing config, and tokenizer deployable until the canary observation window closes.

Where It Breaks Down

Fine-tuning breaks down when the task changes faster than the training lifecycle. A weekly taxonomy change turns every update into dataset migration, training, evaluation, and deployment. A prompt or deterministic rules engine may be more appropriate.

Small or inconsistent datasets create unstable gradients and teach annotation errors. Repeated templates can produce impressive test scores while failing on novel language. Synthetic data can expand coverage, but correlated teacher errors and low diversity require human review and a real-data evaluation set.

Knowledge-heavy question answering remains a poor fit. Even continued pretraining cannot guarantee exact recall, provenance, or deletion. It may improve domain fluency while making unsupported answers sound more convincing.

Fine-tuning also cannot guarantee deterministic constraints, remove hallucinations, or fix an underspecified product requirement. It may reduce failure probability, but authorization, schema validation, factual verification, and policy enforcement remain application responsibilities.

Finally, a base model may lack enough capability. Adapting a small model cannot reliably create reasoning, language coverage, or context handling absent from the base. Evaluate a stronger base before attempting increasingly aggressive training.

When NOT to Use Fine-Tuning

Do not fine-tune when:

• a clear system prompt and a handful of examples already meet quality and latency gates;
• requirements are changing quickly and reversibility matters;
• the missing information is fresh, private, attributable, or access-controlled;
• deterministic code, constrained decoding, or a classifier solves the contract;
• there is no trustworthy labeled data or evaluation set;
• request volume cannot amortize training and operating cost;
• the organization cannot govern training-data rights, PII, retention, and deletion;
• failures require source-level explanations that parametric behavior cannot provide.

Prefer prompt engineering first. Add RAG or tools for external knowledge. Fine-tune only the residual stable behavior. A common production design combines all three: a concise prompt defines request-specific intent, retrieval supplies current evidence, and a fine-tuned model applies a consistent response or tool-use policy.

Running in Production

Treat the model as a versioned dependency with a supply chain. Scan training data for secrets and PII; document legal basis and licenses; restrict dataset and checkpoint access; encrypt artifacts; and verify checksums at deployment. Fine-tuned models can memorize rare strings, so run extraction-oriented probes and avoid training on secrets even when serving is private.

Track four metric groups:

• quality: task score, per-slice score, malformed outputs, abstention, human disagreement;
• safety: policy violations, sensitive-data leakage, jailbreak outcomes, unauthorized behavior;
• service: p50/p95 latency, throughput, queue time, GPU utilization, errors, adapter load time;
• economics: tokens, accelerator hours, utilization, storage, retraining cost, cost per accepted output.

Online labels arrive late and are biased. Build a reviewed sampling process across ordinary traffic, low-confidence cases, disagreement cases, high-value segments, and suspected abuse. Do not automatically train on all failures; first distinguish model, retrieval, prompt, taxonomy, and upstream-data defects.

Use staged rollout with a predeclared stop policy. Shadowing detects service compatibility but not user impact. Canary traffic measures real behavior. Roll back on metric thresholds, not operator intuition. Preserve trace fields for base model, adapter, prompt, retrieval index, decoder settings, and request cohort.

Retrain because measured drift or approved requirements changed—not merely because a calendar date arrived. A retraining trigger should create a new candidate and repeat every gate. It should never overwrite the production artifact in place.

Important

A provider job marked “succeeded” or an open-source trainer reaching its final step means only that optimization completed. Promotion requires independent quality, safety, privacy, latency, and cost gates.

Related Guides

Start with Large Language Models for the training and inference foundation and Prompt Engineering for the first intervention. Use RAG for retrieved knowledge. Study LoRA, QLoRA, and PEFT for efficient adaptation. Continue to RLHF and DPO for preference alignment.

Interview Questions

What is the difference between fine-tuning and prompting?

Prompting changes the tokens supplied for an inference request while model parameters stay frozen. Fine-tuning runs optimization and changes full-model or adapter parameters, producing a new artifact used by later inference requests.

Why is fine-tuning not the default way to add company knowledge?

Parametric knowledge is difficult to update, cite, authorize, and delete. RAG or tools retrieve current records with provenance and access controls. Fine-tuning is better suited to stable behavior such as format, classification, style, or evidence-use policy.

How do LoRA and QLoRA differ?

LoRA trains low-rank adapter matrices while freezing base weights. QLoRA also stores the frozen base in low precision to reduce memory during training. Both normally produce adapters, but QLoRA adds quantization dependencies and numerical trade-offs.

What would you include in a release gate?

Compare against a predeclared baseline on frozen data; require primary and per-slice task metrics, safety and privacy limits, service SLOs, cost limits, artifact-load tests, and human review where labels are subjective. All critical gates should pass.

How do you prevent evaluation leakage?

Deduplicate before splitting, group related entities and conversations, use time-based holdouts where appropriate, isolate golden sets from synthetic-data generation, and search for exact and semantic overlap. Preserve split code and hashes.

When would full fine-tuning be justified?

When a smaller model, substantial domain shift, or a large high-quality dataset requires capacity that PEFT cannot deliver, and controlled experiments show material gains worth the compute, storage, and deployment cost.

What must match between training and inference?

The base-model revision, tokenizer, chat template, special tokens, adapter configuration, and input formatting must be compatible. Decoding and quantization settings also require evaluation because they affect the deployed behavior.

How would you debug a regression after deployment?

Identify the exact artifact and request cohort, compare with the previous model using captured inputs, separate data/template/serving changes from weight changes, inspect per-slice metrics and traces, roll back if thresholds are breached, and reproduce offline before retraining.

Key Takeaways

• Fine-tuning changes parametric behavior during training; prompting and retrieval change inference context.
• Use prompting first and RAG or tools for changing, private, citable, or permissioned facts.
• Start open-weight SFT with LoRA in many cases; use QLoRA for memory pressure and full fine-tuning only when measured evidence supports it.
• Dataset provenance, grouping, deduplication, evaluation isolation, and annotation quality matter more than running more epochs.
• Release the exact deployed artifact through explicit task, safety, privacy, latency, and cost gates.
• Operate adapters and models with immutable lineage, canaries, monitoring, rollback, and reviewed feedback.

FAQs

How much data is required?

There is no universal threshold. A narrow deterministic classification task may improve with hundreds of diverse, correct examples; open-ended behavior may require thousands or more. Build learning curves: train on increasing subsets and measure held-out performance and variance. Stop relying on raw count when examples are duplicated or mislabeled.

Can fine-tuning eliminate a system prompt?

Usually not entirely. Stable demonstrations may move into weights, but the system prompt still defines request-specific policy, versioned constraints, and context. Keep it concise and evaluate attempts to remove it.

Does fine-tuning reduce hallucinations?

It can improve task behavior or teach abstention, but it does not guarantee factual grounding. It may make domain-sounding hallucinations more fluent. Use retrieval, tools, citations, and verification for factual claims.

Should LoRA adapters be merged?

Merge when one adapter is deployed with one base and serving simplicity or runtime compatibility matters. Keep adapters separate for multi-tenant routing, rapid swapping, or storage efficiency. Evaluate the merged and possibly quantized artifact again.

Can a managed API replace the evaluation pipeline?

No. It replaces training and serving infrastructure, not application-specific evaluation. Your team still owns data governance, baselines, release thresholds, canaries, monitoring, and rollback.

What is catastrophic forgetting?

It is degradation of previously learned capabilities while optimizing a narrower dataset. Full tuning, aggressive learning rates, narrow mixtures, and repeated epochs can increase risk. Measure general and safety capabilities alongside the target task; PEFT reduces but does not eliminate regressions.

Is continued pretraining the same as supervised fine-tuning?

No. Continued pretraining usually applies a language-model objective to unlabeled domain text to adapt vocabulary and distributions. SFT uses desired input-output demonstrations. Some programs use continued pretraining followed by SFT, but each stage needs separate evidence.

How should class imbalance be handled?

Use stratified analysis, per-class metrics, intentional sampling or loss weighting, and challenge sets. Preserve a natural-distribution test set so balancing decisions do not hide production calibration or volume effects.

How often should a model be retrained?

Retrain when reviewed data shows material drift, requirements change, a base-model migration is approved, or sufficient high-value corrections accumulate. Every retrain creates a candidate that must pass the full evaluation and rollout process.

Can fine-tuning support RAG?

Yes. It can teach query rewriting, evidence selection, citation format, abstention, or answer synthesis from supplied context. Retrieval still owns the source facts and permissions.

References

• LoRA: Low-Rank Adaptation of Large Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• Hugging Face PEFT documentation
• Hugging Face Transformers training documentation
• Hugging Face TRL SFT Trainer documentation
• OpenAI model fine-tuning guide
• Direct Preference Optimization
• Training language models to follow instructions with human feedback

Further Reading

• Parameter-Efficient Transfer Learning for NLP
• AdapterHub documentation
• PyTorch distributed training documentation
• MLflow model registry documentation
• NIST AI Risk Management Framework