RLHF Guide

TL;DR

• RLHF aligns LLM behavior to human preferences through a three-stage pipeline: supervised fine-tuning (SFT), reward model (RM) training, and reinforcement learning optimization (typically PPO). It changes what the model prefers to say — not the knowledge stored in its weights.

• Stage 1 (SFT) teaches the response format, Stage 2 (RM) learns to score responses by human preference, and Stage 3 (PPO) optimizes the policy to maximize reward while a KL penalty keeps it close to the SFT reference.

• RLHF is how base models became assistants. GPT-4, Claude, Gemini, and Llama chat variants all pass through some form of preference alignment. Without it, a base model predicts plausible internet text, not helpful answers.

• RLHF is expensive and unstable. It needs human labelers, a separate reward model, four model copies in GPU memory during PPO, and constant monitoring for reward hacking and distribution shift.

• DPO is the simpler default. It skips the reward model and PPO loop, reaching comparable alignment quality with far less engineering. Reach for RLHF when you need online RL, multi-objective rewards, or fine-grained reward control.

On this page

• Why This Matters
• The Problem RLHF Solves
• How We Got Here
• What Is RLHF?
• How RLHF Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Use RLHF
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

A base large language model trained on internet text will happily generate toxic content, hallucinate confidently, ignore instructions, and pad responses with filler. None of these are bugs. They are correct predictions of what text statistically follows a prompt on the open web. Pretraining optimizes for likelihood, not helpfulness.

RLHF is the technique that turned raw next-token predictors into usable assistants. It is why ChatGPT follows instructions, refuses clearly harmful requests, and produces answers people actually prefer in side-by-side comparisons. Every major production LLM ships some form of preference alignment on top of pretraining and SFT.

If you build a model that must follow specific behavioral rules — a support-desk tone, safety constraints, a domain-appropriate register — you need to understand RLHF or its simpler successor, DPO. Even if you only consume aligned models through an API, understanding RLHF explains their quirks: sycophancy, over-refusal, verbosity, and the gap between "the model knows X" and "the model is willing to say X."

This guide sits in the LLM Concepts cluster alongside generative AI and fine-tuning. It assumes you already understand training vs inference and parameter-efficient tuning like LoRA and PEFT.

The Problem RLHF Solves

Supervised fine-tuning teaches a model what to say but not how humans want it said. Consider two responses to "Explain quantum computing":

Response A: "Quantum computing uses qubits that hold superpositions, enabling interference across probability amplitudes. Key algorithms include Shor's (factoring) and Grover's (search)."

Response B: "Quantum computing is a fascinating field! Let me explain it in simple terms. Imagine a regular computer bit is like a light switch — it's either on or off. A quantum bit (qubit) is like a dimmer switch that can be partially on and partially off at the same time! This is called superposition. Now, there are many interesting aspects to explore. First, let's talk about entanglement..."

Both are factually acceptable. Many readers prefer A — concise, informative, no throat-clearing. SFT alone cannot reliably encode that preference because both are valid demonstrations. You cannot write a demonstration for every possible prompt, and you cannot express "prefer A's density over B's padding" as a single gold answer. Preference is a ranking over outputs, not a label on one output.

RLHF encodes that ranking directly into the optimization objective. Instead of "here is the correct answer," the signal becomes "here is the better of two answers." That reframing lets the model learn subtle, hard-to-specify qualities:

• Safety — refusing harmful requests without becoming uselessly evasive.
• Honesty / calibration — saying "I don't know" instead of fabricating.
• Instruction following — doing what was asked, not what was statistically likely.
• Tone — professional, casual, or technical as appropriate.
• Conciseness — resisting the reward-free verbosity that pretraining encourages.

Crucially, RLHF aligns behavior, not knowledge. It cannot teach the model a fact it never learned in pretraining; it can only shift the probability mass toward responses humans rate higher. If you need new facts, you want retrieval or fine-tuning on domain data — not alignment.

How We Got Here

Preference-based reinforcement learning predates LLMs. The lineage that produced modern RLHF looks like this.

Diagram: Evolution of preference alignment

timeline
    title From RL preferences to production alignment
    2017 : Christiano et al.
         : Deep RL from human preferences
    2019-2020 : OpenAI summarization
              : Reward models for text quality
    2022 : InstructGPT (Ouyang et al.)
         : SFT + RM + PPO becomes the recipe
    2022 : ChatGPT ships
         : RLHF goes mainstream
    2023 : Constitutional AI / RLAIF
         : AI feedback replaces some human labels
    2023-2026 : DPO and offline methods
              : Simpler alternatives to PPO

Capability came from scale; usefulness came from alignment. RLHF was the bridge, and the field is now simplifying the recipe.

The pivotal paper was Christiano et al. (2017), which showed you could train agents from pairwise human comparisons instead of hand-designed reward functions. OpenAI applied the idea to text summarization, then generalized it in InstructGPT (Ouyang et al., 2022): take a pretrained GPT-3, apply SFT, train a reward model on human comparisons, and optimize with PPO. That three-stage recipe — SFT → RM → PPO — is what people mean by "RLHF" today, and it is the pipeline ChatGPT launched on.

What Is RLHF?

RLHF (Reinforcement Learning from Human Feedback) is a training methodology that uses human preference judgments to align model behavior. The core move: instead of training on correct answers (SFT), train on preferred answers using reinforcement learning. Humans compare pairs of model outputs and mark which is better. A reward model learns to reproduce those judgments as a scalar score. Then an RL algorithm optimizes the LLM (the policy) to produce high-reward outputs, constrained so it does not drift into gibberish.

Three roles matter throughout:

• Policy (π_θ) — the model being optimized. It generates responses and receives gradient updates.
• Reference model (π_ref) — a frozen copy of the SFT model. It anchors the policy via a KL-divergence penalty so alignment does not destroy fluency.
• Reward model (r_φ) — a separate network that maps (prompt, response) to a scalar preference score.

RLHF is alignment, which is why it belongs beside DPO and downstream of fine-tuning in the learning graph. It is not a knowledge-injection technique.

How RLHF Works

RLHF has three sequential stages. Each stage consumes the output of the previous one.

Diagram: The three-stage RLHF pipeline

flowchart LR
    Base[Base LLM] --> SFT[Stage 1: SFT]
    SFT --> RMinit[Init reward model]
    Prefs[Human preference pairs] --> RM[Stage 2: Reward model]
    RMinit --> RM
    SFT --> Policy[Stage 3: PPO policy]
    SFT --> Ref[Frozen reference]
    RM --> PPO[PPO loop]
    Policy --> PPO
    Ref -->|KL penalty| PPO
    PPO --> Aligned[Aligned model]

SFT produces both the policy's starting point and the frozen reference; the reward model provides the signal PPO optimizes against.

Stage 1: Supervised Fine-Tuning (SFT)

Fine-tune the base model on high-quality instruction–response demonstrations. This teaches the format of being an assistant: follow instructions, structure answers, respect the chat template.

• Data: 1,000–100,000 curated instruction–response pairs.
• Method: Standard supervised fine-tuning, often with LoRA.
• Goal: The model can follow instructions at a basic level.
• Duration: Hours to days.

SFT alone yields a helpful-ish model that still hedges, rambles, or wanders off-topic — because it learned from demonstrations, not preferences.

Stage 2: Reward Model (RM) Training

Train a separate model to predict human preferences. Given a prompt and two responses, the RM should score the human-preferred one higher. It is usually the SFT model with the language-modeling head replaced by a scalar value head.

• Data: 10,000–100,000 preference pairs (prompt, chosen, rejected).
• Method: Bradley–Terry loss — maximize the score gap between chosen and rejected.
• Goal: RM scores correlate with human judgment on held-out pairs (target > 70% agreement).
• Duration: Hours.

The Bradley–Terry reward-model loss:

L_RM = -log( σ( r(x, y_chosen) - r(x, y_rejected) ) )

where r(x, y) is the scalar reward for response y to prompt x, and σ is the sigmoid. The RM learns relative quality; its absolute scale is arbitrary, which matters when you tune the KL coefficient later.

Stage 3: RL Optimization (PPO)

Use the reward model as the signal to optimize the SFT model via Proximal Policy Optimization. The policy generates responses, the RM scores them, and PPO updates the policy to raise expected reward — with a KL penalty against the frozen reference.

The PPO objective, at a high level:

maximize  E[ r(x, y) ]  -  β · KL( π_θ(y|x) || π_ref(y|x) )

• Key hyperparameters: KL coefficient β, learning rate, PPO clip range, target KL.
• Goal: Higher human-preferred quality while staying fluent.
• Duration: Days — the most unstable stage.

The KL penalty is the safety rail. Without it, the policy discovers reward hacking: degenerate text that scores high on the RM but reads as garbage, sycophancy, or repetition. The β term keeps the policy in a neighborhood of the reference where the RM's scores are still trustworthy.

Diagram: PPO policy state during optimization

stateDiagram-v2
    [*] --> Generating
    Generating --> Scoring: policy emits responses
    Scoring --> Updating: reward model + KL penalty
    Updating --> Generating: gradient step
    Updating --> RewardHacking: KL too low / trained too long
    RewardHacking --> Recover: raise β, restore checkpoint
    Recover --> Generating
    Updating --> [*]: converged, checkpoint passes eval

Reward hacking is not a rare edge case — it is the default destination if the KL penalty is too weak or training runs too long.

Architecture

A production RLHF system has five components. The PPO stage is the memory-hungry one because it holds several model copies at once.

Memory is the dominant constraint. Classic PPO keeps four models resident: policy, reference, reward, and (for advantage estimation) a value/critic. For a 7B model in FP16 that is roughly 56GB+ before activations and the KV cache. LoRA and QLoRA cut this dramatically because the reference is the frozen base and the policy is base + adapter — the base weights are shared, so you pay for one set of weights plus small adapters instead of two full copies.

Diagram: PPO memory topology

flowchart TB
    subgraph GPU[GPU memory during PPO]
        B[Shared frozen base weights]
        P[Policy = base + LoRA adapter]
        V[Value head]
        R[Reward model + head]
        B --> P
        B --> Ref[Reference = base, no adapter]
    end
    P -->|responses| R
    R -->|scores| ADV[Advantage estimation]
    Ref -->|logprobs| KL[KL penalty]
    V --> ADV
    ADV --> UPD[PPO update to adapter only]
    KL --> UPD

With LoRA sharing the base, only the adapter, value head, and reward model add material memory — the practical reason RLHF is feasible on modest GPU budgets.

Step-by-Step Flow

1. SFT on demonstrations. Collect 5,000–50,000 high-quality instruction–response pairs. Fine-tune the base model with LoRA. Evaluate instruction following on a held-out set before proceeding.
2. Generate comparison data. Sample 2–4 responses per prompt from the SFT model. Have human labelers rank or compare them. Target 10,000–100,000 preference pairs with clear rubric-driven guidelines.
3. Train the reward model. Initialize from the SFT model, add a value head, train with Bradley–Terry loss. Measure held-out agreement with humans (target > 70%). If train accuracy ≫ eval accuracy, the RM is memorizing.
4. Configure PPO. Set β (0.01–0.1), learning rate (1e-6–1e-5), clip range (~0.2), and a target_kl. Use LoRA on the policy to control memory.
5. Run PPO. Per batch: generate → score with RM → compute advantages → apply PPO loss with KL penalty → update. Monitor reward, KL, response length, and refusal rate live.
6. Evaluate alignment. Run safety prompts, instruction-following suites, and helpfulness benchmarks (MT-Bench, AlpacaEval). Compare against the SFT-only baseline. Human evaluation is mandatory — see LLM Evaluation.
7. Iterate. RLHF is rarely one-shot. Mine failure cases, add preference data, retrain the RM, and run another PPO round.

Real Production Example

Aligning a customer-support model to prefer concise, empathetic, action-oriented replies using TRL and LoRA. Stages 2 and 3 are shown; Stage 1 (SFT) is assumed complete from the fine-tuning guide.

# Stage 2: Reward Model Training
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from trl import RewardTrainer, RewardConfig
from datasets import load_dataset

RM_MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(RM_MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

rm_model = AutoModelForSequenceClassification.from_pretrained(
    RM_MODEL_ID,
    num_labels=1,
    torch_dtype="bfloat16",
    device_map="auto",
)

# Preference data: {prompt, chosen, rejected}
prefs = load_dataset("json", data_files="support_preferences.jsonl")

def preprocess_prefs(example):
    chosen = tokenizer(
        example["prompt"] + example["chosen"],
        truncation=True, max_length=1024, padding="max_length",
    )
    rejected = tokenizer(
        example["prompt"] + example["rejected"],
        truncation=True, max_length=1024, padding="max_length",
    )
    return {
        "input_ids_chosen": chosen["input_ids"],
        "attention_mask_chosen": chosen["attention_mask"],
        "input_ids_rejected": rejected["input_ids"],
        "attention_mask_rejected": rejected["attention_mask"],
    }

prefs = prefs.map(preprocess_prefs)

rm_config = RewardConfig(
    output_dir="./reward_model",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    learning_rate=1e-5,
    bf16=True,
    eval_strategy="steps",
    eval_steps=100,
)

rm_trainer = RewardTrainer(
    model=rm_model,
    args=rm_config,
    train_dataset=prefs["train"],
    eval_dataset=prefs["test"],
    processing_class=tokenizer,
)
rm_trainer.train()
rm_model.save_pretrained("./reward_model")

# Stage 3: PPO Training
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from peft import LoraConfig, get_peft_model

ppo_config = PPOConfig(
    output_dir="./ppo_aligned",
    learning_rate=1.4e-5,
    batch_size=16,
    mini_batch_size=4,
    gradient_accumulation_steps=4,
    ppo_epochs=4,
    init_kl_coef=0.05,   # starting KL coefficient (β)
    target_kl=6.0,       # adaptive KL target — safer than a fixed β
    cliprange=0.2,
    log_with="wandb",
)

# Policy model with a value head, LoRA for memory
policy = AutoModelForCausalLMWithValueHead.from_pretrained(
    RM_MODEL_ID, torch_dtype="bfloat16", device_map="auto",
)
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
policy = get_peft_model(policy, lora_config)

# Frozen reference (SFT model) — the KL anchor
ref_model = AutoModelForCausalLMWithValueHead.from_pretrained(
    "./sft_adapter", torch_dtype="bfloat16", device_map="auto",
)

ppo_trainer = PPOTrainer(
    ppo_config, model=policy, ref_model=ref_model,
    tokenizer=tokenizer, dataset=prompts_dataset,
)

for batch in ppo_trainer.dataloader:
    query_tensors = batch["input_ids"]
    response_tensors = ppo_trainer.generate(query_tensors, max_new_tokens=256)
    rewards = [
        rm_model(**tokenizer(r, return_tensors="pt")).logits.item()
        for r in response_tensors
    ]
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
    ppo_trainer.log_stats(stats, batch, rewards)  # watch mean_reward AND kl

In practice this team found PPO unstable: mean reward kept rising, but human-rated quality peaked around step 800 and then degraded — a textbook reward-hacking curve. They checkpointed every 100 steps, evaluated each checkpoint on a golden set, and ultimately switched to DPO, matching quality in roughly half the engineering time. That outcome is common enough that DPO is the recommended starting point for most teams.

Design Decisions

Common patterns

• Adaptive KL over fixed β. A target_kl controller nudges β up when divergence spikes, which is far more robust than guessing a constant.
• RM ensembles. Averaging two or three reward models reduces single-RM blind spots that PPO would otherwise exploit.
• Reward normalization / whitening. Standardize rewards per batch so the RL signal is scale-stable across the run.
• Golden-set gating. Never promote a checkpoint on mean reward alone; gate on a fixed human-rated evaluation set.

When should I use this?

Comparisons

RLHF versus the two techniques it is most confused with — and its main alternative.

Common Mistakes

1. Skipping SFT and running PPO on a base model. PPO on a model that cannot follow instructions wastes compute. SFT is a prerequisite, not optional.
2. Too few preference pairs. Under ~5,000 pairs yields a noisy reward model, and PPO amplifies RM error. Volume and label quality both matter.
3. Reward model overfitting. Train accuracy > 95% with held-out < 65% means memorization. Add diversity and regularization; a bad RM teaches bad preferences.
4. Ignoring KL divergence. Without a meaningful KL penalty the policy drifts to high-reward gibberish. Monitor KL every step and stop when it spikes.
5. Training PPO too long. Quality usually peaks at 500–2,000 steps then degrades. Checkpoint frequently and evaluate each checkpoint.
6. Trusting reward, not humans. RM scores do not perfectly track human preference. Run human evaluation at every stage.
7. Using RLHF where DPO fits. For standard alignment, DPO reaches comparable quality with roughly 10× less engineering. Start there unless you have a specific reason for RL.
8. Expecting RLHF to add knowledge. Alignment shifts behavior, not facts. Confusing the two leads to endless preference collection for problems that need retrieval or data fine-tuning.

Where It Breaks Down

Engineering complexity. Four models, three stages, and RL instability. Most teams underestimate the debugging effort by an order of magnitude.

Reward hacking. The policy finds ways to score high without being good — verbose filler, sycophantic agreement, formatting tricks, repeated high-reward phrases.

Distribution shift. The RM was trained on SFT-model outputs. As PPO moves the policy, generated text drifts off the RM's training distribution and the reward signal becomes unreliable exactly when you rely on it most.

Human-labeling scalability. Quality preferences require trained labelers following detailed rubrics. At scale, labeling cost dominates the training budget and inter-annotator disagreement caps achievable RM accuracy.

Multi-objective tension. Helpfulness, harmlessness, and honesty pull in different directions. Optimizing one can regress another; a single scalar reward flattens genuine trade-offs.

Preference variance. "Better" depends on user, culture, and context. One reward model encodes one preference distribution and cannot serve everyone equally.

When NOT to Use RLHF

Prefer a simpler path when:

• DPO would suffice — standard alignment (tone, safety, conciseness, instruction following) with a fixed preference set. This covers most cases.
• You lack preference data — with only demonstrations, do SFT; RLHF needs comparisons.
• The problem is missing knowledge — use RAG or fine-tune on domain data; alignment cannot inject facts.
• You have no RL expertise or GPU headroom — PPO's instability and 4-model memory footprint punish teams without infrastructure.
• You need deterministic guarantees — RLHF reduces but never eliminates unsafe outputs; pair any aligned model with guardrails rather than treating alignment as a hard filter.

RLHF earns its cost when you need online RL (the model generates its own training data during optimization), multi-objective weighted rewards, or reward shaping that pairwise preferences cannot express.

Running in Production

Best Practice

Instrument every stage, version the SFT model / reward model / preference dataset as one bundle, keep the SFT model as a rollback, and gate promotion on a fixed golden evaluation set — never on reward alone.

Important

Consider DPO before committing to RLHF. It removes the reward model and PPO loop, cutting engineering complexity substantially while matching alignment quality on most tasks.

Related Guides

Inside LLM Concepts:

• Generative AI — where alignment fits in training vs inference
• Large Language Models — the models RLHF aligns
• Fine-tuning — SFT is Stage 1 of RLHF
• DPO — the simpler alignment default
• LoRA · PEFT — memory-efficient training for every stage

Adjacent clusters:

• LLM Evaluation — how to measure whether alignment worked
• Guardrails — runtime safety that alignment does not replace

Tools: Hugging Face · TRL · OpenAI · Anthropic

Interview Questions

1. What are the three stages of RLHF, and what does each contribute? SFT teaches response format; the reward model learns to score preferences; PPO optimizes the policy against the RM with a KL penalty. Each consumes the previous stage's output.

2. Why is a KL penalty necessary in the PPO stage? Without it, the policy drifts far from the reference into high-reward but degenerate text (reward hacking). The KL term keeps the policy where the RM's scores are still valid.

3. Does RLHF add new knowledge to a model? No. It aligns behavior — shifting probability toward preferred responses. New facts require retrieval or fine-tuning on data, not alignment.

4. Why can SFT alone not achieve alignment? SFT trains on single gold demonstrations. Preference is a ranking over outputs that cannot be expressed as one correct answer, so it needs a reward or preference signal.

5. What is reward hacking and how do you detect it? The policy maximizes RM score without genuine quality (verbosity, sycophancy, repetition). Detect it by watching rising reward alongside falling human-eval scores and spiking KL.

6. How many models are in memory during PPO, and how does LoRA help? Up to four: policy, reference, reward, value. LoRA lets the policy and reference share frozen base weights, so you pay for one base plus small adapters instead of two full copies.

7. When would you choose RLHF over DPO? When you need online RL (self-generated training data), multi-objective weighted rewards, or reward shaping that pairwise preferences cannot capture.

8. What is RLAIF? Reinforcement Learning from AI Feedback — an LLM generates the preference labels instead of humans (e.g., Constitutional AI). Cheaper and faster, but it encodes the judge model's biases.

Key Takeaways

• RLHF aligns behavior through three stages: SFT → reward model → PPO. It does not inject knowledge.
• The frozen reference and its KL penalty are what keep alignment from destroying fluency — they are your anchor and your rollback.
• Reward hacking and distribution shift are the default failure modes, not rare edge cases; checkpoint often and gate on human evaluation.
• Memory is the binding constraint; LoRA/PEFT make PPO practical by sharing frozen base weights.
• For most teams, DPO reaches comparable quality with far less engineering — try it first, and reserve RLHF for genuine online-RL or multi-objective needs.

FAQs

What is RLHF in simple terms?

RLHF trains a model to produce responses humans prefer. Humans compare pairs of AI responses and pick the better one; a reward model learns those preferences; reinforcement learning then optimizes the model to produce higher-scored responses without drifting from fluent text.

What are the three stages of RLHF?

1. SFT — fine-tune on instruction–response demonstrations.
2. Reward model — train a model to predict human preferences from comparisons.
3. PPO — use RL to optimize the policy against the reward model, with a KL penalty against the frozen reference.

Does RLHF teach the model new facts?

No. RLHF aligns behavior, not knowledge. It shifts probability toward preferred responses but cannot add information the model never learned in pretraining. Use RAG or fine-tuning for new facts.

Why is RLHF necessary?

Base models predict likely text, not helpful text. Without alignment they produce verbose, hedging, toxic, or off-topic responses. RLHF encodes human preferences about quality, safety, and helpfulness directly into the objective.

How much data does RLHF need?

SFT: ~5,000–50,000 instruction–response pairs. Reward model: ~10,000–100,000 preference pairs. PPO: 10,000+ prompts (no labels needed — the RM supplies the signal). Total human effort spans weeks to months.

What is reward hacking?

The policy exploits weaknesses in the reward model to score high without producing good responses — excessive verbosity, sycophancy, or repeated high-reward phrases. The KL penalty mitigates but does not eliminate it.

What is the KL penalty in PPO?

A term that penalizes the policy for diverging from the frozen SFT reference: reward - β · KL(π_policy || π_ref). It keeps outputs fluent and keeps the policy in a region where the reward model is still reliable.

Is RLHF better than DPO?

On most benchmarks they reach comparable alignment quality. RLHF is more flexible (online RL, multi-objective rewards) but roughly 10× more complex and less stable. Most teams should start with DPO.

Can I do RLHF with LoRA?

Yes, and you should. LoRA on the policy cuts PPO memory 3–4×, and sharing the frozen base makes the reference nearly free. Reward-model training benefits from LoRA too.

What is RLAIF?

Reinforcement Learning from AI Feedback — an LLM generates preference labels instead of humans. Anthropic's Constitutional AI uses this approach. It is cheaper than human labeling but inherits the judge model's biases.

How do I evaluate RLHF quality?

Automated: MT-Bench, AlpacaEval, reward scores. Human: side-by-side comparisons and Likert ratings on helpfulness, safety, and honesty. Production: A/B test against the SFT-only model. See LLM Evaluation.

How long does PPO training take?

For a 7B model with LoRA, roughly 500–2,000 PPO steps over 4–12 hours on 4× A100. Quality peaks early, so checkpoint every ~100 steps and evaluate; training too long causes reward hacking.

Can RLHF make a model worse?

Yes. Over-trained PPO degrades quality, a bad reward model teaches bad preferences, and an insufficient KL penalty produces incoherent text. Always keep an SFT fallback and compare at every checkpoint.

References

• Training language models to follow instructions with human feedback (Ouyang et al., 2022)
• Deep reinforcement learning from human preferences (Christiano et al., 2017)
• OpenAI InstructGPT overview
• Hugging Face TRL Documentation

Further Reading

• Anthropic Constitutional AI
• DeepMind Sparrow (alignment)
• TRL Library on GitHub
• Illustrating Reinforcement Learning from Human Feedback (Hugging Face)