Vector Quantization Guide

TL;DR

• Vector quantization compresses embedding vectors into lower-bit representations (scalar, product, or binary codes) so indexes use less RAM and scan faster.

• Product Quantization (PQ) splits each vector into subspaces and replaces each subspace with a codebook index - the standard path to billion-scale search.

• Scalar Quantization (SQ) maps each float dimension to int8 (or similar) - simple, fast, and often enough for tens of millions of vectors.

• Binary Quantization collapses dimensions to bits - extreme compression with larger recall risk; usually paired with rescoring over full-precision candidates.

• This is embedding compression, not model weight quantization - unrelated to QLoRA or INT4 LLM weights. Do not conflate the two.

Why This Matters

A 1536-dimensional float32 embedding is about 6 KB. Ten million chunks need roughly 60 GB just for vectors - before graph edges, metadata, or replicas. At that size, keeping the full index in RAM becomes the dominant infrastructure cost, and cold storage makes query latency unacceptable.

Quantization is how dedicated vector databases keep search in-memory (or mostly in-memory) at scale. Teams that skip it either overspend on RAM, accept slow disk-backed search, or silently run with under-provisioned indexes that page and thrash. Understanding PQ, SQ, and binary codes lets you choose a compression budget that matches your recall SLOs instead of accepting vendor defaults blindly.

The Problem Vector Quantization Solves

Memory grows linearly with corpus size and dimensions. Exact float32 storage is fine for prototypes. At 50M–1B vectors, memory - not CPU - becomes the bottleneck for vector search.

Distance computation is bandwidth-bound. Comparing a query against millions of float32 vectors moves a lot of data through the CPU cache. Compact codes improve throughput because more candidates fit in L1/L2 cache per cycle.

ANN indexes need resident graphs and codes. HNSW and IVF both assume you can traverse neighbors quickly. If the vectors themselves do not fit, the index cannot deliver millisecond latency regardless of graph quality.

Quantization solves the storage and bandwidth problem for dense retrieval: store approximate vectors, search on codes, optionally rescore a shortlist with full precision.

How We Got Here

Embedding compression followed the same scaling curve as ANN indexes. As corpora grew from thousands to billions of vectors, storing float32 embeddings became the dominant infrastructure cost.

Diagram: Evolution of vector quantization

timeline
    title Embedding compression
    2011 : Product Quantization (Jégou et al.)
    2014 : Faiss IndexIVFPQ at scale
    2018 : Binary and scalar variants in ANN libs
    2020 : Vector DBs add int8 SQ
    2022 : Binary quantization in Qdrant/Weaviate
    2024 : ColBERT + PQ for multi-vector

Compression moved from research-only Faiss demos to first-class features in production vector databases paired with ANN indexes.

Do not confuse this history with LLM weight quantization (GPTQ, AWQ, QLoRA). Those compress model parameters at inference time. Vector quantization compresses stored document embeddings in your search index — a different layer entirely.

Important

Vector quantization compresses stored embeddings used for nearest-neighbor search. It is not the same as quantizing neural network weights (GPTQ, AWQ, QLoRA). Mixing those concepts leads to wrong tooling and wrong expectations.

What Is Vector Quantization?

Vector quantization maps a continuous embedding (\mathbf{x} \in \mathbb{R}^d) to a discrete code (c(\mathbf{x})) from a finite codebook (or set of codebooks). Search then approximates distances using those codes instead of the original floats.

Three families dominate modern retrieval systems:

The goal is not perfect reconstruction. The goal is ranking fidelity: after ANN search over codes, the true nearest neighbors should still appear in the top-(k) (or in a slightly larger candidate set that you rescore).

How Vector Quantization Works

Scalar Quantization (SQ)

SQ learns (or assumes) a range per dimension - or a shared range - and maps floats to integers:

[ q_i = \mathrm{round}\left(\frac{x_i - \min}{\max - \min} \cdot (2^b - 1)\right) ]

Common production choice: int8 ((b=8)). Some systems support uint8 asymmetric quantization where the query stays float32 and documents are int8.

Why it works well for embeddings: many embedding dimensions have roughly similar magnitudes after training. A careful calibration pass over a sample of vectors sets min/max (or percentiles) so clipping is rare.

Tradeoff: only ~4× compression vs float32. Often enough below ~50–100M vectors; insufficient alone for billion-scale RAM budgets.

Product Quantization (PQ)

PQ is the workhorse for large-scale ANN (especially IVF-PQ):

1. Split the (d)-dimensional vector into (m) equal subspaces of size (d/m).
2. For each subspace (j), train a codebook of (k) centroids (usually (k=256), so each subspace index fits in one byte).
3. Encode a vector as (m) bytes (or (m \log_2 k) bits): the nearest centroid ID in each subspace.
4. Approximate distance with precomputed lookup tables between the query (or query subvectors) and each codebook.

Memory for a PQ-encoded vector is roughly (m) bytes when (k=256). For (d=1536), (m=96) yields 96-byte codes (~64× smaller than float32).

Diagram: Product quantization pipeline

flowchart LR
    V["Float vector d dims"] --> S["Split into m subvectors"]
    S --> C1["Codebook 1"]
    S --> C2["Codebook 2"]
    S --> Cm["Codebook m"]
    C1 --> Code["m centroid IDs"]
    C2 --> Code
    Cm --> Code
    Code --> Dist["ADC / SDQ distance approx"]
    Dist --> Rank["Top-k candidates"]
    Rank --> Rescore["Optional float rescore"]

PQ splits vectors into subspaces, assigns codebook IDs, and approximates distance via lookup tables.

Diagram: Quantized search with rescoring

sequenceDiagram
    participant Q as Query
    participant I as Quantized index
    participant S as Full-precision store
    participant R as Ranker
    Q->>I: ANN over codes (k prime)
    I-->>R: candidate IDs
    R->>S: fetch float vectors
    S-->>R: exact distances
    R-->>Q: top-k results

Search on compact codes for speed; rescore a shortlist on full vectors for recall.

Asymmetric Distance Computation (ADC): keep the query in float (or partially quantized) and compare against PQ codes via lookup tables. This usually beats quantizing both sides symmetrically for recall.

Binary Quantization (BQ)

Binary methods map each dimension (or a random/learned projection) to ({0,1}) or ({-1,+1}). Distance becomes Hamming distance (POPCOUNT + XOR), which is extremely fast on modern CPUs.

Examples in practice:

• Sign bit / dimension-wise binary - take (\mathrm{sign}(x_i)).
• Random projection binary codes - project then threshold (related to LSH ideas).
• Vendor-specific binary indexes in Qdrant, Weaviate, and others.

Binary codes are aggressive. Large-scale deployments almost always:

1. Search over binary codes for a large candidate set.
2. Rescore those candidates with float16/float32 vectors stored separately (or fetched from disk).

Without rescoring, binary-only ranking often fails recall SLOs for RAG.

Rescoring Pattern

Compression for candidate generation + precision for final ranking is a common operational pattern:

query → search quantized index (large k') → fetch full vectors → exact rescore → return top-k

This is the same two-stage idea as retrieve-then-rerank, but the second stage here is cheap exact distance, not a cross-encoder.

Engineering Insight

💡 Key Idea - Quantization is a candidate generator. Treat compressed search as a filter that must keep the true neighbors in a shortlist; restore ranking quality with rescoring or a higher-precision stage.

Architecture

In a production vector database, quantization sits inside the index layer:

Typical layouts:

• HNSW + SQ/BQ - graph in RAM, neighbors store quantized payloads; common in Qdrant/Weaviate-style systems.
• IVF + PQ - coarse quantizer (centroids) + PQ residuals; classic Faiss / Milvus path for huge corpora.
• DiskANN-style - graph on SSD with compressed vectors in RAM; quantization keeps the in-memory footprint small.

For selection across engines, see Best Vector Databases and comparisons such as Qdrant vs Pinecone, Milvus vs Qdrant, and Qdrant vs Weaviate.

Diagram: Quantization lifecycle

stateDiagram-v2
    [*] --> Calibrate: SQ or PQ training
    Calibrate --> Encode: upsert vectors
    Encode --> Index: HNSW or IVF
    Index --> Query: ADC or Hamming
    Query --> Rescore: if aggressive
    Rescore --> [*]
    Calibrate --> Retrain: model change
    Retrain --> Calibrate

Codebook calibration must rerun when embedding models change.

Step-by-Step Flow

1. Baseline without compression. Index a representative sample in float32. Measure recall@10 and p95 latency. This is your quality ceiling.

2. Choose a compression target. Example: fit the index in 64 GB RAM with 30% headroom. Back-calculate bytes per vector (including graph overhead).

3. Pick a method. Start with SQ (int8) if 4× is enough. Move to PQ when you need 16–64×. Consider binary only with mandatory rescoring.

4. Train the quantizer on your data. PQ codebooks must be fit on a sample of your embeddings (same model, same preprocessing). Do not reuse codebooks across embedding models.

5. Encode and build the ANN index. Upsert quantized codes; configure IVF nlist / HNSW M as usual.

6. Tune over-fetch and rescoring. Search for k' = 5–20 × k on codes, then rescore to k. Raise k' until recall matches your SLO.

7. Evaluate on a golden set. Compute recall@k against exact search. Watch for segment-specific failures (short queries, rare entities).

8. Deploy with monitoring. Track recall proxy metrics, empty-result rates, and memory. Retrain PQ when you change embedding models.

Real Production Example

A support-knowledge RAG corpus grows to 40M chunks embedded with a 1536-dim model (~230 GB float32). The team targets a single 128 GB search node class and needs recall@10 ≥ 0.95 vs exact.

"""
Illustrative PQ + IVF pipeline with Faiss-style APIs.
Managed engines (Milvus, Qdrant, etc.) wrap similar steps.
"""
from __future__ import annotations

import numpy as np

# --- toy stand-ins for faiss.IndexIVFPQ ---
class ProductQuantizer:
    def __init__(self, d: int, m: int, nbits: int = 8):
        assert d % m == 0
        self.d, self.m, self.nbits = d, m, nbits
        self.ks = 1 << nbits
        self.dsub = d // m
        self.codebooks: list[np.ndarray] = []

    def train(self, x: np.ndarray, iters: int = 10) -> None:
        """Fit per-subspace k-means (simplified)."""
        self.codebooks = []
        for j in range(self.m):
            sub = x[:, j * self.dsub : (j + 1) * self.dsub]
            # random init centroids
            idx = np.random.choice(len(sub), self.ks, replace=False)
            cents = sub[idx].copy()
            for _ in range(iters):
                # assign
                dists = ((sub[:, None, :] - cents[None, :, :]) ** 2).sum(-1)
                assign = dists.argmin(axis=1)
                for c in range(self.ks):
                    members = sub[assign == c]
                    if len(members):
                        cents[c] = members.mean(axis=0)
            self.codebooks.append(cents)

    def encode(self, x: np.ndarray) -> np.ndarray:
        codes = np.empty((len(x), self.m), dtype=np.uint8)
        for j in range(self.m):
            sub = x[:, j * self.dsub : (j + 1) * self.dsub]
            dists = ((sub[:, None, :] - self.codebooks[j][None, :, :]) ** 2).sum(-1)
            codes[:, j] = dists.argmin(axis=1).astype(np.uint8)
        return codes

    def adc_l2(self, query: np.ndarray, codes: np.ndarray) -> np.ndarray:
        """Asymmetric distance: float query vs PQ codes."""
        tables = []
        for j in range(self.m):
            qsub = query[j * self.dsub : (j + 1) * self.dsub]
            # distance from query subvector to each centroid
            tables.append(((self.codebooks[j] - qsub) ** 2).sum(axis=1))
        tables = np.stack(tables, axis=0)  # (m, ks)
        # gather and sum
        return tables[np.arange(self.m)[:, None], codes.T].sum(axis=0)


def search_with_rescore(
    query: np.ndarray,
    pq: ProductQuantizer,
    codes: np.ndarray,
    full_vectors: np.ndarray,
    k: int = 10,
    overfetch: int = 80,
) -> list[int]:
    approx = pq.adc_l2(query, codes)
    cand = np.argpartition(approx, overfetch)[:overfetch]
    # exact L2 rescore on shortlist
    exact = ((full_vectors[cand] - query) ** 2).sum(axis=1)
    order = exact.argsort()[:k]
    return cand[order].tolist()


# Usage sketch
d, n, m = 1536, 50_000, 96
rng = np.random.default_rng(0)
xb = rng.standard_normal((n, d)).astype(np.float32)
xq = rng.standard_normal(d).astype(np.float32)

pq = ProductQuantizer(d=d, m=m)
pq.train(xb[:10_000])
codes = pq.encode(xb)
top = search_with_rescore(xq, pq, codes, xb, k=10, overfetch=80)
print("top ids", top)

In a real deployment you would replace the toy k-means with Faiss/milvus training, persist codebooks beside the index, and keep full-precision vectors on cheaper storage for rescoring only.

Design Decisions

Common patterns

| Pattern | Description | | ---------------------------- | --------------------- | -------------------------------------- | | SQ-first | int8 scalar before PQ | Simplest 4× win; try before complex PQ | | IVF-PQ + rescore | Faiss/Milvus classic | Billion-scale with float shortlist | | HNSW + on-disk originals | Qdrant-style | Quantized graph; originals for rescore | | Binary two-stage | Hamming then float | Maximum RAM savings with safety net |

Comparisons

SQ vs PQ vs Binary vs float16

Quantization vs more RAM vs more shards

Decision tree: compression level

Decision tree: choosing compression

flowchart TD
    A[RAM fits float32?] -->|Yes| B[float16 or none]
    A -->|No| C[float16 enough?]
    C -->|Yes| D[Deploy float16]
    C -->|No| E[int8 SQ + eval]
    E --> F{Recall OK?}
    F -->|Yes| G[Ship SQ]
    F -->|No| H[PQ + over-fetch rescore]
    H --> I{Still tight?}
    I -->|Yes| J[Binary + mandatory rescore]

Introduce the mildest compression that meets the RAM budget; never skip recall@k measurement.

Common Mistakes

1. Confusing embedding quantization with LLM weight quantization. QLoRA/INT4 model loading does not compress your vector index. Different problem, different tools.

2. Training PQ on the wrong distribution. Codebooks from a different embedding model (or an old version) silently tank recall.

3. Skipping rescoring after binary or aggressive PQ. Fast Hamming search that returns the wrong top-10 is still a failed retrieval.

4. Optimizing compression ratio instead of recall@k. 64× looks impressive in a slide; 0.80 recall@10 fails the product.

5. Ignoring graph memory. HNSW edges can rival vector bytes. Quantizing vectors alone may not hit your RAM target.

6. Changing embedding dimensions without rebuilding. New (d) invalidates PQ layouts and codebooks.

7. Using uncalibrated SQ min/max. Outliers stretch the range; most dimensions collapse into few buckets.

Where It Breaks Down

Very short or highly specific queries. When relevance hinges on a few dimensions (rare tokens reflected weakly in the embedding), coarse codes erase the signal.

Cross-modal or mixed embedding spaces. PQ assumes a relatively homogeneous vector distribution. Mixing image and text vectors in one index without separate codebooks hurts.

Frequent embedding model upgrades. Each model change forces re-embed + retrain + reindex. Heavy quantization increases the cost of experimentation if you keep many full-precision copies.

Filtered search with tiny result sets. Aggressive quantization plus selective metadata filtering can leave too few survivors after approximation error. Raise over-fetch or relax compression for filtered collections.

Late-interaction / multi-vector retrieval. ColBERT-style stores many vectors per document; quantization helps, but parameters must be chosen for token-level corpora, not single-vector chunks.

When NOT to Use Vector Quantization

Skip compression when:

1. Corpus fits comfortably in RAM as float32 — below ~1–5M vectors, float16 or uncompressed is simpler.
2. You cannot run a recall baseline — aggressive PQ without measurement will silently degrade RAG answers.
3. Embedding model changes weekly — retrain/reindex cost dominates; stabilize the model first.
4. Latency budget cannot absorb rescoring — binary or aggressive PQ without rescore time in the SLA will miss recall targets.
5. You need bit-exact reproducibility — quantized ADC distances introduce approximation; use exact search for audit oracles.

Running in Production

Best Practice

✅ Best Practices - Baseline float recall first, introduce the mildest compression that meets the RAM budget, always rescore under aggressive schemes, and gate deploys on recall@k + p95 latency.

Common Mistake

Shipping PQ with default m/nbits copied from a blog without measuring your recall. Defaults are starting points for Faiss demos, not guarantees for your embedding model.

Related Guides

Concept guides

• ANN Indexes: HNSW, IVF, IVF-PQ — structures that consume quantized codes.
• Embeddings: Source vectors; dimension and metric dictate quantization design.
• Vector Search: Similarity search problem quantization accelerates.
• Vector Databases: Systems that operationalize quantization + ANN.
• Hybrid Search: Quantization affects only the dense side.
• Late Interaction Retrieval: Multi-vector indexes that especially need compression.
• Metadata Filtering: Filtered search with aggressive compression needs higher over-fetch.
• Retrieval Evaluation: Gate every compression change on recall@k.
• RAG: Downstream consumer where recall loss shows up as wrong answers.

Rankings

• Best Vector Databases

Comparisons

• Qdrant vs Pinecone · Milvus vs Qdrant · Qdrant vs Weaviate

Tools

• Qdrant · Milvus · Weaviate · LanceDB · pgvector

If you understood this topic, read next:

Diagram: Compression learning path

flowchart LR
    A[Embeddings] --> B[ANN Indexes]
    B --> C[Quantization]
    C --> D[Hybrid Search]
    D --> E[Eval]

Prerequisites: Embeddings · Vector Search

Next topics: ANN Indexes · Metadata Filtering · Late Interaction Retrieval

Interview Questions

1. What is vector quantization vs LLM weight quantization?

   • Expected: compresses stored embeddings for ANN search, not model parameters (QLoRA/GPTQ).

2. How does product quantization work?

   • Expected: split into m subspaces, codebook per subspace, ADC distance via lookup tables.

3. Why use asymmetric distance computation?

   • Expected: float query vs quantized docs preserves recall better than quantizing both sides.

4. What is the over-fetch + rescore pattern?

   • Expected: retrieve k′ from codes, exact distance on shortlist, return top-k.

5. When would you choose binary quantization?

   • Expected: extreme RAM pressure with mandatory float rescoring; not binary-only for RAG.

6. What happens if you reuse PQ codebooks across embedding models?

   • Expected: silent recall collapse; must retrain on new model's vector distribution.

7. How does quantization interact with HNSW?

   • Expected: quantized payloads on graph nodes; poor distance estimates can hurt graph quality — validate recall.

8. How do you evaluate a compression change?

   • Expected: golden queries, recall@k vs exact float baseline, p95 latency with rescoring included.

Key Takeaways

• Vector quantization compresses embeddings for memory- and bandwidth-efficient ANN search.
• SQ (int8) is the simplest win; PQ unlocks large-scale compression; binary needs rescoring.
• This is not LLM weight quantization — do not confuse it with QLoRA.
• Always baseline float recall, then introduce the mildest compression that fits RAM.
• Over-fetch + exact rescore is the production safety net for aggressive codes.
• Retrain codebooks when embedding models change; compare engines in Best Vector Databases.

FAQs

Is vector quantization the same as QLoRA?

No. QLoRA quantizes model weights for fine-tuning large language models. Vector quantization compresses embedding vectors in a search index. Same word family, different layers of the stack.

When should I enable quantization?

When vector RAM (plus index overhead) approaches your node budget, or when distance computation is clearly memory-bandwidth bound. Below ~1–5M vectors, float16/float32 is often simpler.

Does quantization change my distance metric?

You still choose cosine / dot / L2 at the index level. Quantization approximates that metric. Some pipelines normalize vectors before SQ/PQ so cosine ≈ dot product in the quantized space.

How much recall will I lose?

With int8 SQ and light rescoring, many text corpora stay within 1–3% recall@10 of float32. Aggressive PQ or binary without rescoring can lose much more. Measure on your golden set.

What is an over-fetch factor?

You retrieve more candidates from the quantized index than you ultimately return (k' > k), then rescore. Typical starting points: 5–20× depending on compression aggressiveness.

Can I quantize the query as well?

Symmetric quantization is possible but asymmetric (float query vs quantized docs) usually preserves recall better for PQ/SQ.

Do I need to retrain PQ when I add documents?

Incremental upserts can use existing codebooks. Retrain when the embedding distribution shifts materially (new model, new domain, major corpus replacement).

How does quantization interact with HNSW?

HNSW can store quantized payloads on nodes. Graph connectivity is separate from compression, but poor distance estimates during construction/search can hurt graph quality - another reason to validate recall.

Is float16 enough without PQ?

Often yes for medium scale. Float16 halves memory vs float32 with tiny quality loss for many embeddings. Use PQ/BQ when float16 still does not fit.

Does binary quantization work for RAG?

Yes, if you rescore. Binary-only top-k is risky for answer quality. Treat BQ as a first-stage candidate generator.

How do I choose PQ m?

Pick the smallest compression that meets RAM limits, then increase over-fetch until recall recovers. Prefer more bytes per vector over heroic over-fetch that blows latency.

Will quantization break hybrid search?

No. BM25/sparse retrieval is unchanged. Only dense scores are approximated. Fusion still works; validate that dense contribution remains useful after compression.

Can pgvector do PQ?

pgvector focuses on PostgreSQL-native indexes (e.g. HNSW/IVFFlat). Advanced PQ workflows are more mature in Faiss/Milvus/Qdrant. For huge corpora, dedicated vector DBs are usually the better fit.

How do I A/B test quantization in production?

Shadow traffic: run quantized search beside float search on a sample of queries, compare overlap@k and downstream answer ratings. Roll forward only if both online and offline gates pass.

References

• Faiss wiki: Product Quantization / IndexIVFPQ
• Jégou et al., Product Quantization for Nearest Neighbor Search (IEEE TPAMI)
• Qdrant documentation: Quantization
• Milvus documentation: Index types (IVF_PQ, SQ)
• Weaviate documentation: Vector quantization / compression

Further Reading

• ANN Indexes guide
• Vector Search guide
• Best vector databases ranking
• Qdrant vs Pinecone comparison
• Milvus vs Qdrant comparison
• Pinecone Learn: vector search concepts