Vector Quantization Guide

TL;DR

• Vector quantization compresses embedding vectors into lower-bit representations (scalar, product, or binary codes) so indexes use less RAM and scan faster.

• Product Quantization (PQ) splits each vector into subspaces and replaces each subspace with a codebook index - the standard path to billion-scale search.

• Scalar Quantization (SQ) maps each float dimension to int8 (or similar) - simple, fast, and often enough for tens of millions of vectors.

• Binary Quantization collapses dimensions to bits - extreme compression with larger recall risk; usually paired with rescoring over full-precision candidates.

• This is embedding compression, not model weight quantization - unrelated to QLoRA or INT4 LLM weights. Do not conflate the two.

Why This Matters

A 1536-dimensional float32 embedding is about 6 KB. Ten million chunks need roughly 60 GB just for vectors - before graph edges, metadata, or replicas. At that size, keeping the full index in RAM becomes the dominant infrastructure cost, and cold storage makes query latency unacceptable.

Quantization is how production vector databases keep search in-memory (or mostly in-memory) at scale. Teams that skip it either overspend on RAM, accept slow disk-backed search, or silently run with under-provisioned indexes that page and thrash. Understanding PQ, SQ, and binary codes lets you choose a compression budget that matches your recall SLOs instead of accepting vendor defaults blindly.

The Problem Vector Quantization Solves

Memory grows linearly with corpus size and dimensions. Exact float32 storage is fine for prototypes. At 50M–1B vectors, memory - not CPU - becomes the bottleneck for vector search.

Distance computation is bandwidth-bound. Comparing a query against millions of float32 vectors moves a lot of data through the CPU cache. Compact codes improve throughput because more candidates fit in L1/L2 cache per cycle.

ANN indexes need resident graphs and codes. HNSW and IVF both assume you can traverse neighbors quickly. If the vectors themselves do not fit, the index cannot deliver millisecond latency regardless of graph quality.

Quantization solves the storage and bandwidth problem for dense retrieval: store approximate vectors, search on codes, optionally rescore a shortlist with full precision.

Important

Vector quantization compresses stored embeddings used for nearest-neighbor search. It is not the same as quantizing neural network weights (GPTQ, AWQ, QLoRA). Mixing those concepts leads to wrong tooling and wrong expectations.

What Is Vector Quantization?

Vector quantization maps a continuous embedding (\mathbf{x} \in \mathbb{R}^d) to a discrete code (c(\mathbf{x})) from a finite codebook (or set of codebooks). Search then approximates distances using those codes instead of the original floats.

Three families dominate production systems:

The goal is not perfect reconstruction. The goal is ranking fidelity: after ANN search over codes, the true nearest neighbors should still appear in the top-(k) (or in a slightly larger candidate set that you rescore).

How Vector Quantization Works

Scalar Quantization (SQ)

SQ learns (or assumes) a range per dimension - or a shared range - and maps floats to integers:

[ q_i = \mathrm{round}\left(\frac{x_i - \min}{\max - \min} \cdot (2^b - 1)\right) ]

Common production choice: int8 ((b=8)). Some systems support uint8 asymmetric quantization where the query stays float32 and documents are int8.

Why it works well for embeddings: many embedding dimensions have roughly similar magnitudes after training. A careful calibration pass over a sample of vectors sets min/max (or percentiles) so clipping is rare.

Tradeoff: only ~4× compression vs float32. Often enough below ~50–100M vectors; insufficient alone for billion-scale RAM budgets.

Product Quantization (PQ)

PQ is the workhorse for large-scale ANN (especially IVF-PQ):

1. Split the (d)-dimensional vector into (m) equal subspaces of size (d/m).
2. For each subspace (j), train a codebook of (k) centroids (usually (k=256), so each subspace index fits in one byte).
3. Encode a vector as (m) bytes (or (m \log_2 k) bits): the nearest centroid ID in each subspace.
4. Approximate distance with precomputed lookup tables between the query (or query subvectors) and each codebook.

Memory for a PQ-encoded vector is roughly (m) bytes when (k=256). For (d=1536), (m=96) yields 96-byte codes (~64× smaller than float32).

flowchart LR
    V["Float vector d dims"] --> S["Split into m subvectors"]
    S --> C1["Codebook 1"]
    S --> C2["Codebook 2"]
    S --> Cm["Codebook m"]
    C1 --> Code["m centroid IDs"]
    C2 --> Code
    Cm --> Code
    Code --> Dist["ADC / SDQ distance approx"]
    Dist --> Rank["Top-k candidates"]
    Rank --> Rescore["Optional float rescore"]

Asymmetric Distance Computation (ADC): keep the query in float (or partially quantized) and compare against PQ codes via lookup tables. This usually beats quantizing both sides symmetrically for recall.

Binary Quantization (BQ)

Binary methods map each dimension (or a random/learned projection) to ({0,1}) or ({-1,+1}). Distance becomes Hamming distance (POPCOUNT + XOR), which is extremely fast on modern CPUs.

Examples in practice:

• Sign bit / dimension-wise binary - take (\mathrm{sign}(x_i)).
• Random projection binary codes - project then threshold (related to LSH ideas).
• Vendor-specific binary indexes in Qdrant, Weaviate, and others.

Binary codes are aggressive. Production deployments almost always:

1. Search over binary codes for a large candidate set.
2. Rescore those candidates with float16/float32 vectors stored separately (or fetched from disk).

Without rescoring, binary-only ranking often fails recall SLOs for RAG.

Rescoring Pattern

Compression for candidate generation + precision for final ranking is the standard production pattern:

query → search quantized index (large k') → fetch full vectors → exact rescore → return top-k

This is the same two-stage idea as retrieve-then-rerank, but the second stage here is cheap exact distance, not a cross-encoder.

Engineering Insight

💡 Key Idea - Quantization is a candidate generator. Treat compressed search as a filter that must keep the true neighbors in a shortlist; restore ranking quality with rescoring or a higher-precision stage.

Architecture

In a production vector database, quantization sits inside the index layer:

Typical layouts:

• HNSW + SQ/BQ - graph in RAM, neighbors store quantized payloads; common in Qdrant/Weaviate-style systems.
• IVF + PQ - coarse quantizer (centroids) + PQ residuals; classic Faiss / Milvus path for huge corpora.
• DiskANN-style - graph on SSD with compressed vectors in RAM; quantization keeps the in-memory footprint small.

For selection across engines, see best vector databases and comparisons such as Qdrant vs Pinecone and Milvus vs Qdrant.

Step-by-Step Flow

1. Baseline without compression. Index a representative sample in float32. Measure recall@10 and p95 latency. This is your quality ceiling.

2. Choose a compression target. Example: fit the index in 64 GB RAM with 30% headroom. Back-calculate bytes per vector (including graph overhead).

3. Pick a method. Start with SQ (int8) if 4× is enough. Move to PQ when you need 16–64×. Consider binary only with mandatory rescoring.

4. Train the quantizer on your data. PQ codebooks must be fit on a sample of your embeddings (same model, same preprocessing). Do not reuse codebooks across embedding models.

5. Encode and build the ANN index. Upsert quantized codes; configure IVF nlist / HNSW M as usual.

6. Tune over-fetch and rescoring. Search for k' = 5–20 × k on codes, then rescore to k. Raise k' until recall matches your SLO.

7. Evaluate on a golden set. Compute recall@k against exact search. Watch for segment-specific failures (short queries, rare entities).

8. Deploy with monitoring. Track recall proxy metrics, empty-result rates, and memory. Retrain PQ when you change embedding models.

Real Production Example

A support-knowledge RAG corpus grows to 40M chunks embedded with a 1536-dim model (~230 GB float32). The team targets a single 128 GB search node class and needs recall@10 ≥ 0.95 vs exact.

"""
Illustrative PQ + IVF pipeline with Faiss-style APIs.
Production systems (Milvus, Qdrant, etc.) wrap similar steps.
"""
from __future__ import annotations

import numpy as np

# --- toy stand-ins for faiss.IndexIVFPQ ---
class ProductQuantizer:
    def __init__(self, d: int, m: int, nbits: int = 8):
        assert d % m == 0
        self.d, self.m, self.nbits = d, m, nbits
        self.ks = 1 << nbits
        self.dsub = d // m
        self.codebooks: list[np.ndarray] = []

    def train(self, x: np.ndarray, iters: int = 10) -> None:
        """Fit per-subspace k-means (simplified)."""
        self.codebooks = []
        for j in range(self.m):
            sub = x[:, j * self.dsub : (j + 1) * self.dsub]
            # random init centroids
            idx = np.random.choice(len(sub), self.ks, replace=False)
            cents = sub[idx].copy()
            for _ in range(iters):
                # assign
                dists = ((sub[:, None, :] - cents[None, :, :]) ** 2).sum(-1)
                assign = dists.argmin(axis=1)
                for c in range(self.ks):
                    members = sub[assign == c]
                    if len(members):
                        cents[c] = members.mean(axis=0)
            self.codebooks.append(cents)

    def encode(self, x: np.ndarray) -> np.ndarray:
        codes = np.empty((len(x), self.m), dtype=np.uint8)
        for j in range(self.m):
            sub = x[:, j * self.dsub : (j + 1) * self.dsub]
            dists = ((sub[:, None, :] - self.codebooks[j][None, :, :]) ** 2).sum(-1)
            codes[:, j] = dists.argmin(axis=1).astype(np.uint8)
        return codes

    def adc_l2(self, query: np.ndarray, codes: np.ndarray) -> np.ndarray:
        """Asymmetric distance: float query vs PQ codes."""
        tables = []
        for j in range(self.m):
            qsub = query[j * self.dsub : (j + 1) * self.dsub]
            # distance from query subvector to each centroid
            tables.append(((self.codebooks[j] - qsub) ** 2).sum(axis=1))
        tables = np.stack(tables, axis=0)  # (m, ks)
        # gather and sum
        return tables[np.arange(self.m)[:, None], codes.T].sum(axis=0)


def search_with_rescore(
    query: np.ndarray,
    pq: ProductQuantizer,
    codes: np.ndarray,
    full_vectors: np.ndarray,
    k: int = 10,
    overfetch: int = 80,
) -> list[int]:
    approx = pq.adc_l2(query, codes)
    cand = np.argpartition(approx, overfetch)[:overfetch]
    # exact L2 rescore on shortlist
    exact = ((full_vectors[cand] - query) ** 2).sum(axis=1)
    order = exact.argsort()[:k]
    return cand[order].tolist()


# Usage sketch
d, n, m = 1536, 50_000, 96
rng = np.random.default_rng(0)
xb = rng.standard_normal((n, d)).astype(np.float32)
xq = rng.standard_normal(d).astype(np.float32)

pq = ProductQuantizer(d=d, m=m)
pq.train(xb[:10_000])
codes = pq.encode(xb)
top = search_with_rescore(xq, pq, codes, xb, k=10, overfetch=80)
print("top ids", top)

In a real deployment you would replace the toy k-means with Faiss/milvus training, persist codebooks beside the index, and keep full-precision vectors on cheaper storage for rescoring only.

Design Decisions

⚠ Common Mistakes

1. Confusing embedding quantization with LLM weight quantization. QLoRA/INT4 model loading does not compress your vector index. Different problem, different tools.

2. Training PQ on the wrong distribution. Codebooks from a different embedding model (or an old version) silently tank recall.

3. Skipping rescoring after binary or aggressive PQ. Fast Hamming search that returns the wrong top-10 is still a failed retrieval.

4. Optimizing compression ratio instead of recall@k. 64× looks impressive in a slide; 0.80 recall@10 fails the product.

5. Ignoring graph memory. HNSW edges can rival vector bytes. Quantizing vectors alone may not hit your RAM target.

6. Changing embedding dimensions without rebuilding. New (d) invalidates PQ layouts and codebooks.

7. Using uncalibrated SQ min/max. Outliers stretch the range; most dimensions collapse into few buckets.

Where It Breaks Down

Very short or highly specific queries. When relevance hinges on a few dimensions (rare tokens reflected weakly in the embedding), coarse codes erase the signal.

Cross-modal or mixed embedding spaces. PQ assumes a relatively homogeneous vector distribution. Mixing image and text vectors in one index without separate codebooks hurts.

Frequent embedding model upgrades. Each model change forces re-embed + retrain + reindex. Heavy quantization increases the cost of experimentation if you keep many full-precision copies.

Filtered search with tiny result sets. Aggressive quantization plus selective metadata filtering can leave too few survivors after approximation error. Raise over-fetch or relax compression for filtered collections.

Late-interaction / multi-vector retrieval. ColBERT-style stores many vectors per document; quantization helps, but parameters must be chosen for token-level corpora, not single-vector chunks.

Running in Production

Best Practice

✅ Best Practices - Baseline float recall first, introduce the mildest compression that meets the RAM budget, always rescore under aggressive schemes, and gate deploys on recall@k + p95 latency.

Common Mistake

Shipping PQ with default m/nbits copied from a blog without measuring your recall. Defaults are starting points for Faiss demos, not guarantees for your embedding model.

Ecosystem

• Qdrant: Scalar and binary quantization with optional rescoring; HNSW-centric deployments.

• Milvus: IVF-PQ, SQ, and related index types for large-scale clusters.

• Weaviate: Vector compression options alongside hybrid search.

• LanceDB: Disk-oriented storage with quantization-friendly ANN paths for analytical + retrieval workloads.

• pgvector: PostgreSQL vectors; quantization support is more limited - often pair with careful dimension choice or external ANN for huge corpora.

Compare engines on the best vector databases ranking, and dig into tradeoffs via Qdrant vs Pinecone and Milvus vs Qdrant.

Related Technologies

• ANN Indexes: HNSW, IVF, IVF-PQ, DiskANN - the structures that consume quantized codes.

• Embeddings: Source vectors; dimension and metric dictate quantization design.

• Vector Search: Similarity search problem quantization accelerates.

• Vector Databases: Systems that operationalize quantization + ANN.

• Hybrid Search: Keyword + dense; quantization affects only the dense side.

• RAG: End application where recall loss from bad quantization shows up as wrong answers.

• Late-Interaction Retrieval: Multi-vector indexes that especially need careful compression.

Learning Path

Prerequisites: Embeddings · Vector Search

Next topics: ANN Indexes · Metadata Filtering · Hybrid Search

Estimated time: 50 min · Difficulty: Intermediate

FAQs

Is vector quantization the same as QLoRA?

No. QLoRA quantizes model weights for fine-tuning large language models. Vector quantization compresses embedding vectors in a search index. Same word family, different layers of the stack.

When should I enable quantization?

When vector RAM (plus index overhead) approaches your node budget, or when distance computation is clearly memory-bandwidth bound. Below ~1–5M vectors, float16/float32 is often simpler.

Does quantization change my distance metric?

You still choose cosine / dot / L2 at the index level. Quantization approximates that metric. Some pipelines normalize vectors before SQ/PQ so cosine ≈ dot product in the quantized space.

How much recall will I lose?

With int8 SQ and light rescoring, many text corpora stay within 1–3% recall@10 of float32. Aggressive PQ or binary without rescoring can lose much more. Measure on your golden set.

What is an over-fetch factor?

You retrieve more candidates from the quantized index than you ultimately return (k' > k), then rescore. Typical starting points: 5–20× depending on compression aggressiveness.

Can I quantize the query as well?

Symmetric quantization is possible but asymmetric (float query vs quantized docs) usually preserves recall better for PQ/SQ.

Do I need to retrain PQ when I add documents?

Incremental upserts can use existing codebooks. Retrain when the embedding distribution shifts materially (new model, new domain, major corpus replacement).

How does quantization interact with HNSW?

HNSW can store quantized payloads on nodes. Graph connectivity is separate from compression, but poor distance estimates during construction/search can hurt graph quality - another reason to validate recall.

Is float16 enough without PQ?

Often yes for medium scale. Float16 halves memory vs float32 with tiny quality loss for many embeddings. Use PQ/BQ when float16 still does not fit.

Does binary quantization work for RAG?

Yes, if you rescore. Binary-only top-k is risky for answer quality. Treat BQ as a first-stage candidate generator.

How do I choose PQ m?

Pick the smallest compression that meets RAM limits, then increase over-fetch until recall recovers. Prefer more bytes per vector over heroic over-fetch that blows latency.

Will quantization break hybrid search?

No. BM25/sparse retrieval is unchanged. Only dense scores are approximated. Fusion still works; validate that dense contribution remains useful after compression.

Can pgvector do PQ?

pgvector focuses on PostgreSQL-native indexes (e.g. HNSW/IVFFlat). Advanced PQ workflows are more mature in Faiss/Milvus/Qdrant. For huge corpora, dedicated vector DBs are usually the better fit.

How do I A/B test quantization in production?

Shadow traffic: run quantized search beside float search on a sample of queries, compare overlap@k and downstream answer ratings. Roll forward only if both online and offline gates pass.

References

• Faiss wiki: Product Quantization / IndexIVFPQ
• Jégou et al., Product Quantization for Nearest Neighbor Search (IEEE TPAMI)
• Qdrant documentation: Quantization
• Milvus documentation: Index types (IVF_PQ, SQ)
• Weaviate documentation: Vector quantization / compression

Further Reading

• ANN Indexes guide
• Vector Search guide
• Best vector databases ranking
• Qdrant vs Pinecone comparison
• Milvus vs Qdrant comparison
• Pinecone Learn: vector search concepts

Summary

• Vector quantization compresses embeddings for memory- and bandwidth-efficient ANN search.
• SQ (int8) is the simplest win; PQ unlocks large-scale compression; binary needs rescoring.
• This is not LLM weight quantization - do not confuse it with QLoRA.
• Always baseline float recall, then introduce the mildest compression that fits RAM.
• Over-fetch + exact rescore is the production safety net for aggressive codes.
• Retrain codebooks when embedding models change; gate deploys on recall@k.