RLHF (Reinforcement Learning from Human Feedback) - Complete Guide

TL;DR

RLHF aligns LLMs to human preferences through a three-stage pipeline: supervised fine-tuning (SFT), reward model training, and reinforcement learning optimization (typically PPO).
Stage 1 (SFT) teaches the model the task format; Stage 2 (reward model) learns to score responses by human preference; Stage 3 (PPO) optimizes the model to maximize reward while staying close to the SFT model.
RLHF is how ChatGPT, Claude, and Gemini became useful - base models predict text; RLHF-trained models produce helpful, harmless, honest responses.
RLHF is expensive and unstable - it requires human labelers, a separate reward model, careful hyperparameter tuning, and monitoring for reward hacking.
DPO is a simpler alternative that skips the reward model and PPO loop, achieving comparable alignment quality with far less engineering complexity.

A base LLM trained on internet text will happily generate toxic content, hallucinate confidently, ignore instructions, and optimize for verbosity. None of these behaviors are bugs - they're correct predictions of what text appears next on the internet.

RLHF is the technique that transformed raw language models into useful assistants. It's the reason ChatGPT follows instructions, refuses harmful requests, and produces responses humans prefer. Every major production LLM (GPT-4, Claude, Gemini, Llama chat versions) goes through some form of preference alignment.

If you're building a custom model that needs to follow specific behavioral guidelines - customer service tone, safety constraints, domain-appropriate responses - you need to understand RLHF (or its simpler successor, DPO).

The Problem RLHF Solves

Supervised fine-tuning teaches a model what to say but not how humans want it said. Consider two responses to "Explain quantum computing":

Response A: "Quantum computing uses qubits that can be in superposition, enabling parallel computation across probability amplitudes. Key algorithms include Shor's (factoring) and Grover's (search)."

Response B: "Quantum computing is a fascinating field! Let me explain it in simple terms. Imagine a regular computer bit is like a light switch - it's either on or off. A quantum bit (qubit) is like a dimmer switch that can be partially on and partially off at the same time! This is called superposition. Now, there are many interesting aspects to explore. First, let's talk about entanglement, which is when qubits become correlated..."

Both are factually correct. Humans overwhelmingly prefer A - concise, informative, no filler. SFT alone can't reliably teach this preference because both are valid text. RLHF encodes human preferences directly into the optimization objective.

RLHF also addresses:

Safety: Refusing harmful requests without being uselessly evasive
Honesty: Saying "I don't know" instead of hallucinating
Instruction following: Doing what was asked, not what was statistically likely
Tone calibration: Professional, casual, or technical as appropriate

What Is RLHF?

RLHF (Reinforcement Learning from Human Feedback) is a training methodology that uses human preference judgments to align model behavior. Introduced in Christiano et al. (2017) and popularized by Ouyang et al. (2022) (InstructGPT), it became the standard approach for making LLMs useful.

The core idea: instead of training on "correct answers" (SFT), train on "preferred answers" using reinforcement learning. Humans compare pairs of model outputs and indicate which is better. A reward model learns these preferences. Then RL optimizes the LLM to produce high-reward outputs.

How RLHF Works

RLHF has three distinct stages. Each stage builds on the previous one.

Stage 1: Supervised Fine-Tuning (SFT)

Fine-tune the base model on high-quality instruction-response demonstrations. This teaches the model the basic format of being an assistant - how to follow instructions, structure responses, and use the chat format.

Base Model → SFT on (instruction, response) pairs → SFT Model

Data: 1,000–100,000 curated instruction-response pairs
Method: Standard supervised fine-tuning (often with LoRA)
Goal: Model can follow instructions at a basic level
Duration: Hours to days

SFT alone produces a model that's helpful but not aligned - it will still generate verbose, hedging, or off-topic responses because it learned from demonstrations, not preferences.

Stage 2: Reward Model (RM) Training

Train a separate model to predict human preferences. Given a prompt and two responses, the reward model scores which response humans would prefer.

Prompt + Response → Reward Model → Scalar Reward Score

Data: 10,000–100,000 preference pairs (prompt, chosen_response, rejected_response)
Method: Bradley-Terry loss - maximize the score gap between chosen and rejected
Architecture: Often the SFT model with a scalar value head replacing the language modeling head
Goal: Reward model correlates with human judgments
Duration: Hours

The reward model loss:

L_RM = -log(σ(r(x, y_chosen) - r(x, y_rejected)))

Where r(x, y) is the reward for response y given prompt x, and σ is the sigmoid function.

Stage 3: RL Optimization (PPO)

Use the reward model as a reward signal to optimize the SFT model via Proximal Policy Optimization (PPO). The model generates responses, the reward model scores them, and PPO updates the model to increase expected reward.

SFT Model → Generate Response → Reward Model → PPO Update → Aligned Model

Method: PPO with KL divergence penalty to prevent drifting too far from the SFT model
Key hyperparameters: KL coefficient (β), learning rate, PPO clip range
Goal: Model produces responses humans prefer while maintaining fluency
Duration: Days (most unstable stage)

The PPO objective:

L_PPO = E[r(x, y)] - β · KL(π_θ || π_SFT)

The KL penalty (β) prevents the model from "reward hacking" - exploiting the reward model's weaknesses to get high scores while producing degenerate text.

Direct Preference Optimization aligns models from pairwise human preferences without training a separate reward model or running RL.

DPO alignment overview

Source: Stanford / CRFM

Architecture

A production RLHF system has five components:

Component	Role	Typical Implementation
SFT model	Starting point for alignment	LoRA fine-tuned base model
Reward model	Scores response quality	SFT model + value head, or standalone classifier
PPO trainer	RL optimization loop	TRL `PPOTrainer` or custom implementation
Reference model	KL penalty anchor (frozen SFT model)	Copy of SFT model, no gradients
Human labeling pipeline	Generates preference data	Label Studio, Scale AI, Surge AI, or internal annotators

Memory requirements are substantial. PPO needs four model copies in memory simultaneously: policy (being trained), reference (frozen SFT), reward model, and value model. With a 7B model, that's ~56GB+ in FP16. LoRA and QLoRA reduce this significantly.

Step-by-Step Flow

Step 1: SFT on demonstrations. Collect 5,000–50,000 high-quality instruction-response pairs. Fine-tune the base model with LoRA. Evaluate instruction-following on a held-out set.

Step 2: Generate comparison data. Use the SFT model to generate 2–4 responses per prompt. Have human labelers rank or compare responses. Aim for 10,000–100,000 preference pairs.

Step 3: Train the reward model. Initialize from the SFT model. Add a value head. Train on preference pairs with Bradley-Terry loss. Evaluate reward model accuracy on a held-out preference set (target: >70% agreement with humans).

Step 4: Configure PPO. Set KL coefficient (β=0.01–0.1), learning rate (1e-6 to 1e-5), PPO clip range (0.2). Use LoRA for the policy model to reduce memory.

Step 5: Run PPO training. For each batch: generate responses → score with reward model → compute PPO loss with KL penalty → update policy. Monitor reward, KL divergence, and response quality.

Step 6: Evaluate alignment. Test on safety prompts, instruction following, helpfulness benchmarks (MT-Bench, AlpacaEval). Compare against SFT-only model. Human evaluation is essential.

Step 7: Iterate. RLHF is rarely one-shot. Collect failure cases, add preference data, retrain reward model, run another PPO round.

Real Production Example

Aligning a customer support model to prefer concise, empathetic, action-oriented responses using TRL and LoRA:

# Stage 1: SFT (already completed - see fine-tuning guide)
# sft_model loaded from "./sft_adapter"

# Stage 2: Reward Model Training
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from trl import RewardTrainer, RewardConfig
from datasets import load_dataset

RM_MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(RM_MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

rm_model = AutoModelForSequenceClassification.from_pretrained(
    RM_MODEL_ID,
    num_labels=1,
    torch_dtype="bfloat16",
    device_map="auto",
)

# Preference data: {prompt, chosen, rejected}
prefs = load_dataset("json", data_files="support_preferences.jsonl")

def preprocess_prefs(example):
    chosen = tokenizer(
        example["prompt"] + example["chosen"],
        truncation=True, max_length=1024, padding="max_length",
    )
    rejected = tokenizer(
        example["prompt"] + example["rejected"],
        truncation=True, max_length=1024, padding="max_length",
    )
    return {
        "input_ids_chosen": chosen["input_ids"],
        "attention_mask_chosen": chosen["attention_mask"],
        "input_ids_rejected": rejected["input_ids"],
        "attention_mask_rejected": rejected["attention_mask"],
    }

prefs = prefs.map(preprocess_prefs)

rm_config = RewardConfig(
    output_dir="./reward_model",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    learning_rate=1e-5,
    bf16=True,
    eval_strategy="steps",
    eval_steps=100,
)

rm_trainer = RewardTrainer(
    model=rm_model,
    args=rm_config,
    train_dataset=prefs["train"],
    eval_dataset=prefs["test"],
    processing_class=tokenizer,
)
rm_trainer.train()
rm_model.save_pretrained("./reward_model")

# Stage 3: PPO Training
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from peft import LoraConfig, get_peft_model

ppo_config = PPOConfig(
    output_dir="./ppo_aligned",
    learning_rate=1.4e-5,
    batch_size=16,
    mini_batch_size=4,
    gradient_accumulation_steps=4,
    ppo_epochs=4,
    init_kl_coef=0.05,
    target_kl=6.0,
    cliprange=0.2,
    log_with="wandb",
)

# Policy model with LoRA
policy = AutoModelForCausalLMWithValueHead.from_pretrained(
    RM_MODEL_ID,
    torch_dtype="bfloat16",
    device_map="auto",
)
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
policy = get_peft_model(policy, lora_config)

# Reference model (frozen SFT)
ref_model = AutoModelForCausalLMWithValueHead.from_pretrained(
    "./sft_adapter",
    torch_dtype="bfloat16",
    device_map="auto",
)

ppo_trainer = PPOTrainer(
    ppo_config,
    model=policy,
    ref_model=ref_model,
    tokenizer=tokenizer,
    dataset=prompts_dataset,
)

# PPO training loop
for batch in ppo_trainer.dataloader:
    query_tensors = batch["input_ids"]
    response_tensors = ppo_trainer.generate(query_tensors, max_new_tokens=256)
    rewards = [rm_model(**tokenizer(r, return_tensors="pt")).logits.item()
               for r in response_tensors]
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)
    ppo_trainer.log_stats(stats, batch, rewards)

In practice, this team found PPO training unstable - reward increased but response quality peaked at step 800 then degraded (reward hacking). They switched to DPO and achieved better results in half the engineering time.

Design Decisions

Decision	Option A	Option B	When to choose
Alignment method	RLHF (PPO)	DPO	DPO for most teams; RLHF when you need online RL or have RL infrastructure
SFT data size	5K examples	50K+ examples	5K for focused tasks; 50K+ for general assistant behavior
Preference data	Human labelers	AI-generated (RLAIF)	Human for safety-critical; AI for scale (Constitutional AI, Claude's approach)
Reward model	Separate model	Implicit (DPO)	Separate for RLHF; DPO eliminates this entirely
Policy training	LoRA	Full fine-tuning	LoRA for models >7B; full FT only for small models with ample GPU
KL penalty	Fixed β	Adaptive KL target	Adaptive (target_kl) is more stable; fixed β is simpler

⚠ Common Mistakes

Skipping SFT and going straight to RLHF. PPO on a base model that can't follow instructions wastes compute. SFT is a prerequisite, not optional.
Too few preference pairs. Under 5,000 preference pairs produces a noisy reward model. The PPO stage amplifies reward model errors.
Reward model overfitting. If the RM achieves >95% accuracy on the training set but <65% on held-out preferences, it's memorizing, not generalizing. Add regularization and more diverse data.
Ignoring KL divergence. Without the KL penalty, the model drifts from coherent text to high-reward gibberish. Monitor KL throughout PPO training and stop when it spikes.
Training PPO too long. PPO often peaks early (500–2000 steps) then degrades due to reward hacking. Checkpoint frequently and evaluate at each checkpoint.
Not evaluating with humans. Automated reward scores don't correlate perfectly with human preference. Run human eval at every stage.
Using RLHF when DPO would work. For most alignment tasks, DPO achieves comparable quality with 10x less engineering. Start with DPO unless you have a specific reason for RL.

Where It Breaks Down

Engineering complexity. RLHF requires managing four models, hyperparameter tuning across three stages, and debugging RL instability. Most teams underestimate the effort.

Reward hacking. The model finds ways to maximize reward without producing good responses - verbose filler text, sycophantic agreement, or repetitive patterns that score well with the reward model.

Distribution shift. The reward model is trained on SFT model outputs. As PPO changes the policy, generated text drifts from the reward model's training distribution, making rewards unreliable.

Scalability of human labeling. Quality preference data requires trained labelers following detailed guidelines. At scale, labeling costs dominate the training budget.

Multi-objective alignment. Balancing helpfulness, harmlessness, and honesty requires careful reward model design. Optimizing for one can degrade others.

Cultural and individual preference variance. What's "better" varies by user, culture, and context. A single reward model encodes one preference distribution.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Dimension	Consideration
Scaling	PPO requires 4 model copies in memory; use LoRA for policy, offload reference model to CPU
Latency	RLHF is a training-time technique; inference latency is unchanged from SFT model
Cost	Human labeling: $0.50–$5 per preference pair; GPU: 4–8× A100 for days; total: $10K–$100K+ for a full pipeline
Monitoring	Track reward scores, KL divergence, response length, refusal rates, human eval scores over time
Evaluation	MT-Bench, AlpacaEval, custom safety evals, human A/B testing; evaluate at every PPO checkpoint
Security	Aligned models can still be jailbroken; RLHF reduces but doesn't eliminate safety failures
Versioning	Version SFT model, reward model, PPO checkpoint, and preference dataset as a pipeline bundle

Important

Consider DPO before committing to RLHF. DPO eliminates the reward model and PPO loop, reducing engineering complexity by 10x while achieving comparable alignment on most tasks.

Ecosystem

Tool	Role
TRL (HuggingFace)	SFT, reward model training, PPO, DPO trainers
PEFT	LoRA adapters for memory-efficient RLHF stages
OpenAI / Anthropic APIs	Production models aligned with RLHF/RLAIF
Scale AI / Surge AI	Human preference data collection
Argilla / Label Studio	Internal preference labeling tools
W&B / MLflow	Track multi-stage training experiments
AlpacaEval / MT-Bench	Automated alignment evaluation benchmarks
DeepSpeed / FSDP	Multi-GPU training for large model RLHF

DPO: Simpler alignment without reward model or PPO - start here for most projects.
Fine-Tuning: SFT is Stage 1 of RLHF.
LoRA: Reduces memory for all RLHF stages.
PEFT: Library for efficient adapter training in RLHF pipelines.

Learning Path

Prerequisites: Fine-Tuning · LoRA · PEFT

Next topics: DPO · LLM Evaluation

Estimated time: 60 min · Difficulty: Advanced

FAQs

What is RLHF in simple terms?

RLHF trains a model to produce responses humans prefer. Humans compare pairs of AI responses and pick the better one. A reward model learns these preferences, then reinforcement learning optimizes the AI to generate higher-scored responses.

What are the three stages of RLHF?

SFT: Fine-tune on instruction-response demonstrations.
Reward Model: Train a model to predict human preferences from comparison data.
PPO: Use RL to optimize the model against the reward model, with a KL penalty to prevent drift.

RLHF aligns LLMs through three stages: SFT → reward model → PPO optimization.
It's how production LLMs (ChatGPT, Claude, Gemini) became useful assistants instead of text predictors.
RLHF is powerful but expensive, unstable, and complex - reward hacking and distribution shift are constant risks.
Use LoRA/PEFT for all stages to manage GPU memory requirements.
For most teams, DPO achieves comparable alignment with far less engineering - try DPO first.

Next Topics

Learning Path

Continue Learning

Foundations

Production

Related Tools