QLoRA Guide

TL;DR

• QLoRA combines 4-bit quantized base model weights with BF16 LoRA adapters so you can fine-tune a 70B model on a single 48GB GPU - something that requires 8+ GPUs with standard LoRA.

• Three techniques make this work: 4-bit NormalFloat (NF4) quantization, double quantization of the quantization constants, and paged optimizers that offload to CPU RAM when GPU memory is exhausted.

• Quality is nearly identical to 16-bit LoRA on most benchmarks - the quantization noise affects base weights (which are frozen), not the trainable adapters.

• Setup requires bitsandbytes + PEFT + the right config flags - missing bnb_4bit_compute_dtype or using FP16 instead of BF16 are common setup errors.

• QLoRA is for memory-constrained training, not faster training - wall-clock time is similar to or slower than 16-bit LoRA; the win is fitting the model in GPU memory at all.

Why This Matters

A 70B parameter model in FP16 requires ~140GB just for weights. Add optimizer states and activations, and you need 320GB+ of GPU memory for full fine-tuning - eight A100 80GB GPUs at minimum. Most teams, researchers, and individual developers don't have that.

QLoRA, introduced by Dettmers et al. (2023), reduced the memory footprint of fine-tuning a 65B model from ~780GB to ~48GB. This democratized large model fine-tuning: a single RTX 4090 (24GB) can fine-tune a 7B model, and a single A100 80GB can fine-tune 70B.

If you want to fine-tune models above 13B and don't have a multi-GPU cluster, QLoRA is how you do it.

The Problem QLoRA Solves

Standard LoRA still loads the full base model in FP16/BF16 into GPU memory. For large models, the base weights alone exceed available VRAM:

LoRA reduces trainable parameters but not the memory needed to store the base model. QLoRA quantizes the frozen base weights to 4-bit, cutting base model memory by ~4x, while keeping LoRA adapters in higher precision for accurate gradient updates.

What Is QLoRA?

QLoRA is LoRA applied to a 4-bit quantized base model. The base model weights are frozen and stored in 4-bit NormalFloat (NF4) format. LoRA adapter weights train in BF16 (or FP16). During the forward pass, 4-bit weights are dequantized to BF16 for computation, gradients flow through the adapters, and the frozen quantized weights are never updated.

The three components:

1. 4-bit NormalFloat (NF4). A quantization data type optimized for normally distributed neural network weights. NF4 uses quantile-based binning rather than uniform spacing, preserving more information at 4 bits than standard INT4.

2. Double quantization. The quantization constants themselves (scaling factors) are quantized to 8-bit, saving an additional ~0.37 bits per parameter. Small per-parameter saving, but ~3GB on a 65B model.

3. Paged optimizers. Uses NVIDIA unified memory to page optimizer states to CPU RAM when GPU memory spikes (e.g., during gradient checkpointing). Prevents OOM crashes at the cost of occasional CPU-GPU transfers.

How QLoRA Works

The Forward Pass

Input → Dequantize 4-bit W to BF16 → Compute h = Wx + BAx → Output
                                    ↑                    ↑
                              frozen (4-bit)      trainable (BF16)

The base weight matrix W is stored in NF4 format. For each forward pass, W is dequantized to BF16, the computation runs in BF16, and gradients are computed only for the LoRA matrices B and A. The 4-bit weights are never modified.

NF4 vs. Standard INT4

Standard INT4 quantization uses uniformly spaced bins. Neural network weights follow a normal distribution - most values cluster near zero. NF4 places quantization bins at the quantiles of a standard normal distribution, allocating more precision where weights are dense:

Standard INT4:  |----|----|----|----|----|----|----|----|  (uniform)
NF4:            |-|--|---|----|----|---|--|-|              (quantile-based)

This means NF4 preserves more information for typical weight distributions at the same bit width.

Memory Breakdown (70B model)

LoRA injects trainable low-rank matrices into attention layers while keeping the base model frozen, dramatically reducing fine-tuning memory and compute.

Source: Microsoft Research

Architecture

A QLoRA training setup requires four components working together:

The critical configuration is in BitsAndBytesConfig:

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NormalFloat quantization
    bnb_4bit_compute_dtype=torch.bfloat16,  # Dequantize to BF16 for compute
    bnb_4bit_use_double_quant=True,      # Double quantize the constants
)

Missing or incorrect any of these flags is the most common QLoRA setup failure.

Step-by-Step Flow

Step 1: Install dependencies.

pip install transformers peft bitsandbytes accelerate trl datasets

Ensure bitsandbytes ≥ 0.41.0 and CUDA is available. QLoRA does not work on CPU or MPS (Apple Silicon) as of 2026.

Step 2: Configure 4-bit quantization.

Create a BitsAndBytesConfig with NF4, double quantization, and BF16 compute dtype.

Step 3: Load the model in 4-bit.

Pass quantization_config=bnb_config to from_pretrained(). The model loads directly into 4-bit - do not load in FP16 and then quantize.

Step 4: Prepare for k-bit training.

Call prepare_model_for_kbit_training(model) to enable gradient checkpointing and cast layer norms to FP32 for training stability.

Step 5: Add LoRA adapters.

Standard LoRA config - r=16, alpha=32, target attention + MLP modules.

Step 6: Train.

Use the same hyperparameters as 16-bit LoRA. Learning rate 2e-4, 1–3 epochs. Enable gradient checkpointing to reduce activation memory.

Step 7: Save adapter only.

model.save_pretrained() saves only the LoRA adapter (~50–500MB), not the 4-bit base model.

Step 8: Deploy.

Load base model in FP16/BF16 (or quantized for inference), load adapter, merge or serve separately.

Real Production Example

Complete QLoRA setup for fine-tuning Llama 3.1 70B on a single A100 80GB:

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import (
    LoraConfig,
    get_peft_model,
    prepare_model_for_kbit_training,
)
from trl import SFTTrainer
from datasets import load_dataset

MODEL_ID = "meta-llama/Llama-3.1-70B-Instruct"

# Step 1: 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# Step 2: Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token

# Step 3: Prepare for k-bit training
model = prepare_model_for_kbit_training(
    model,
    use_gradient_checkpointing=True,
)

# Step 4: Add LoRA adapters
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 335,544,320 || all params: 69,762,123,776 || trainable%: 0.48%

# Step 5: Load dataset
dataset = load_dataset("json", data_files={
    "train": "instructions_train.jsonl",
    "test": "instructions_eval.jsonl",
})

def formatting_func(example):
    return tokenizer.apply_chat_template(
        example["messages"], tokenize=False
    )

# Step 6: Train
training_args = TrainingArguments(
    output_dir="./llama70b-qlora",
    num_train_epochs=1,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=100,
    optim="paged_adamw_8bit",
    gradient_checkpointing=True,
    max_grad_norm=0.3,
    group_by_length=True,
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    formatting_func=formatting_func,
    processing_class=tokenizer,
    max_seq_length=2048,
)

trainer.train()
model.save_pretrained("./llama70b-qlora/adapter")

# Step 7: Inference - load in higher precision for serving
from peft import PeftModel

base_for_inference = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
inference_model = PeftModel.from_pretrained(
    base_for_inference, "./llama70b-qlora/adapter"
)

Peak GPU memory during training: ~42GB on A100 80GB. Training 5,000 examples took ~14 hours at seq_len=2048.

For a 7B model on a 24GB GPU (RTX 4090), the same config works with per_device_train_batch_size=4 and gradient_accumulation_steps=4.

Design Decisions

⚠ Common Mistakes

1. Loading model in FP16 then trying to quantize. Use quantization_config in from_pretrained() to load directly in 4-bit. Post-hoc quantization breaks PEFT integration.

2. Forgetting prepare_model_for_kbit_training(). Without this, gradient checkpointing doesn't work and layer norm precision causes training instability.

3. Using FP16 compute dtype on Ampere GPUs. BF16 has better dynamic range and is natively supported on A100/RTX 30xx+. Use bnb_4bit_compute_dtype=torch.bfloat16.

4. Not using paged optimizer. optim="paged_adamw_8bit" prevents OOM during optimizer step. Standard AdamW will OOM on tight memory budgets.

5. Training the base model weights. Ensure LoRA config has bias="none" and base model parameters have requires_grad=False. Only adapter weights should update.

6. Expecting QLoRA to be faster. QLoRA is slower per-step than 16-bit LoRA due to dequantization overhead. The benefit is memory, not speed.

7. Serving in 4-bit without testing quality. Inference with the 4-bit base model + adapter may differ from BF16 base + adapter. Evaluate before deploying quantized inference.

8. Running on Apple Silicon (MPS). bitsandbytes 4-bit quantization requires CUDA. Use standard LoRA in FP16 on M-series Macs, or rent a cloud GPU.

Where It Breaks Down

Non-CUDA hardware. QLoRA requires NVIDIA GPUs with bitsandbytes. No Apple Silicon, no AMD ROCm (limited support), no CPU training.

Extremely long sequences. At seq_len=8192+ on a 70B model, even QLoRA may not fit on a single GPU. Multi-GPU with DeepSpeed ZeRO or reduce sequence length.

Tasks sensitive to weight precision. Quantization introduces noise in the frozen weights. Most tasks are unaffected, but some precision-critical tasks (arithmetic, code generation with exact syntax) may show small degradation vs. 16-bit LoRA.

Very large models (405B+). Even 4-bit quantization of 405B needs ~100GB+ for weights alone. Multi-GPU QLoRA is possible but less well-tested.

Training speed at scale. If you have ample GPU resources, 16-bit LoRA trains faster and may produce marginally better results. QLoRA is a memory solution, not a performance optimization.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

For inference, load the base model in BF16 (not 4-bit) and apply the adapter. The 4-bit quantization is a training-time optimization. Serving in 4-bit is a separate decision (use GPTQ/AWQ for inference quantization).

Ecosystem

Related Technologies

• LoRA: The adapter method QLoRA builds on - understand LoRA first.

• PEFT: HuggingFace library providing QLoRA integration via bitsandbytes.

• Fine-Tuning: When to fine-tune vs. RAG vs. prompting.

• Llama Models: The most common target for QLoRA fine-tuning.

Learning Path

Prerequisites: LoRA · Fine-Tuning · Large Language Models

Next topics: PEFT · RLHF · DPO

Estimated time: 50 min · Difficulty: Advanced

FAQs

What is QLoRA?

QLoRA is LoRA fine-tuning applied to a 4-bit quantized base model. It reduces GPU memory requirements by ~4x, enabling fine-tuning of 70B models on a single 48GB GPU.

How much GPU memory do I need for QLoRA?

7B model: ~6–10GB (fits RTX 3060 12GB). 13B: ~12–16GB. 34B: ~24–28GB. 70B: ~40–48GB (needs A100 80GB or 2×RTX 4090). These assume gradient checkpointing and r=16.

Is QLoRA worse than regular LoRA?

On most benchmarks, QLoRA with NF4 matches 16-bit LoRA within 1–2% on task accuracy. The quantization noise affects frozen weights, not the trainable adapters. For most practical purposes, they are equivalent.

What is NF4 quantization?

NormalFloat 4-bit is a data type where quantization bins are placed at quantiles of a standard normal distribution. It preserves more information than uniform INT4 for the normally distributed weights found in neural networks.

What is double quantization?

The scaling constants used in 4-bit quantization are themselves quantized to 8-bit. This saves ~0.37 bits per parameter - about 3GB on a 65B model. Enable with bnb_4bit_use_double_quant=True.

Can I use QLoRA on Apple Silicon?

No. bitsandbytes 4-bit quantization requires CUDA. On M-series Macs, use standard LoRA in FP16 (7B models fit in 16–24GB unified memory) or use cloud GPUs.

What optimizer should I use with QLoRA?

paged_adamw_8bit from bitsandbytes. It quantizes optimizer states to 8-bit and pages to CPU RAM when GPU memory is full. Standard AdamW will likely OOM.

How do I deploy a QLoRA model?

Train with QLoRA (4-bit base + adapter). For serving, load the base model in BF16, apply the adapter, and optionally merge. For cost optimization, quantize the merged model with GPTQ or AWQ for inference.

Does QLoRA work with Flash Attention?

Yes. Pass attn_implementation="flash_attention_2" when loading the model. Flash Attention reduces activation memory, which helps fit longer sequences.

What learning rate should I use for QLoRA?

Same as LoRA: 1e-4 to 3e-4. QLoRA does not require a different learning rate. Start with 2e-4 and adjust based on eval loss.

Can I fine-tune a model already quantized with GPTQ?

QLoRA expects to quantize the base model itself with NF4. Loading a pre-GPTQ model and applying QLoRA is not standard practice. Start from the original BF16/FP16 checkpoint.

How does Unsloth relate to QLoRA?

Unsloth provides optimized kernels for QLoRA training - 2–5x faster with less memory. It uses the same NF4 + LoRA approach but with custom Triton kernels. Compatible with PEFT adapters.

References

• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• Hugging Face PEFT Documentation
• Microsoft LoRA Paper

Further Reading

• QLoRA Paper
• bitsandbytes GitHub
• Hugging Face Fine-tuning Guide

Summary

• QLoRA = 4-bit NF4 base model + BF16 LoRA adapters - fits 70B on one A100 80GB.
• Three key techniques: NF4 quantization, double quantization, paged optimizers.
• Setup requires bitsandbytes, correct BitsAndBytesConfig, and prepare_model_for_kbit_training().
• Quality matches 16-bit LoRA on most tasks; the tradeoff is memory, not accuracy.
• For inference, load base model in BF16 and apply adapter - 4-bit is a training-time optimization.