Vector Search Guide

TL;DR

• Vector search finds the k vectors closest to a query vector in embedding space — the core operation behind semantic search and RAG retrieval.

• Exact k-NN compares against every vector — accurate but O(n) per query. Exact search becomes increasingly impractical as corpus size grows into the hundreds of thousands or millions of vectors, depending on hardware and latency requirements.

• Approximate Nearest Neighbor (ANN) indexes trade recall for speed — HNSW and IVF reduce search to milliseconds over large corpora.

• Distance metric must match your embedding model — cosine similarity for normalized vectors; for OpenAI text-embedding-3 models (L2-normalized), cosine similarity and dot product produce equivalent rankings.

• Always measure recall@k on your data — ANN parameters that work for one corpus fail on another. Default configs are starting points, not answers.

Why This Matters

Every RAG pipeline depends on vector search. When a user asks "Why am I getting a 429 error?", your system embeds the query, searches millions of chunk vectors, and returns the top matches. If search is slow, users wait. If search misses the right chunk, the LLM hallucinates — no prompt fixes bad retrieval.

Understanding vector search — not just which vector database to use — lets you debug recall failures, tune latency, and choose between exact and approximate search for your scale. The difference between 85% and 99% recall@5 is the difference between a useful product and a demo.

Vector search is also the foundation for hybrid search (dense path), metadata filtering (constrained ANN), and re-ranking (precision after ANN recall). Master this layer before optimizing upstream or downstream.

The Problem Vector Search Solves

Semantic matching requires comparing vectors. Embeddings map text to high-dimensional vectors (typically 384–3072 dimensions). Finding relevant documents means finding vectors close to the query vector. Without efficient search, you brute-force compare against every vector in your corpus.

Brute force doesn't scale. Cosine similarity between one query and 1 million 1536-dimensional vectors requires 1 billion multiply-add operations per query. At 10 QPS, that's unsustainable on CPU without approximation.

Similarity is not keyword matching. Users phrase questions differently than documents. Vector search finds "rate limit exceeded" when the user types "429 error" — something BM25 handles differently. Production systems often combine both via hybrid search.

Vector search solves the scale and semantics problem for retrieval — fast, meaning-aware lookup over embedding indexes.

How We Got Here

Nearest neighbor search predates modern AI by decades. What changed in the 2020s was combining ANN algorithms with dense embedding models at production scale:

Diagram: Evolution of vector search

flowchart LR
    A[k-NN brute force] --> B[LSH / PQ]
    B --> C[HNSW graphs]
    C --> D[Vector DBs at scale]
    D --> E[Filtered ANN]
    E --> F[Hybrid + rerank]

The industry moved from exact comparison to graph-based ANN, then added metadata-constrained search for multi-tenant RAG.

FAISS from Meta AI democratized ANN research implementations. Managed vector databases productized HNSW and IVF with APIs, metadata filtering, and horizontal scaling. The algorithm is mature; operational quality lives in parameter tuning, index hygiene, and evaluation.

What Is Vector Search?

Vector search (also called similarity search or nearest neighbor search) is the operation of finding the k vectors in a collection that are most similar to a query vector, according to a distance or similarity metric.

Given:

• Query vector q ∈ ℝᵈ
• Corpus of n vectors v₁, v₂, ..., vₙ ∈ ℝᵈ
• Similarity function sim(q, v)

Return the k vectors with highest sim(q, v).

This is distinct from a vector database, which is storage plus indexing plus filtering. Vector search is the algorithm; the database is the system that runs it at scale.

How Vector Search Works

Similarity Metrics

Warning

Mismatching metric and embedding model destroys recall. OpenAI text-embedding-3-small and text-embedding-3-large outputs are L2-normalized, so cosine similarity and dot product rank identically. Using L2 distance without normalization can rank differently — check model docs.

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def brute_force_search(
    query: np.ndarray,
    corpus: np.ndarray,
    k: int = 5,
) -> list[int]:
    """Exact k-NN — baseline for eval, not production at scale."""
    scores = np.array([cosine_similarity(query, v) for v in corpus])
    return np.argsort(scores)[::-1][:k].tolist()

Exact k-NN vs Approximate Nearest Neighbor (ANN)

Exact k-NN computes similarity against every vector. Simple, correct, slow.

ANN builds an index structure that navigates to likely neighbors without scanning the full corpus. Returns approximate nearest neighbors — usually correct, occasionally misses.

Decision Trade-off

ANN trades recall for speed. At 95% recall@10, you miss the right document in 5% of queries. For high-stakes retrieval, increase ef_search or use exact search on a filtered subset.

HNSW (Hierarchical Navigable Small World)

HNSW is the dominant ANN algorithm in production vector databases. It builds a multi-layer graph where each node connects to nearby neighbors.

Query flow:

1. Start at the top layer entry point.
2. Greedy walk to the nearest neighbor at each layer.
3. Drop to the next layer and repeat.
4. At layer 0, expand local neighborhood to find top-k.

Key parameters:

• M — max connections per node (higher = better recall, more memory)
• efConstruction — build-time search width (higher = better index quality, slower build)
• efSearch — query-time search width (higher = better recall, slower queries)

See ANN Indexes for deep comparison of HNSW, IVF, and disk-based alternatives.

IVF (Inverted File Index)

IVF partitions the vector space into clusters (Voronoi cells). At query time, search only the nearest clusters — not the full corpus.

Key parameters:

• nlist — number of clusters (more = finer partitions)
• nprobe — clusters to search at query time (higher = better recall)

IVF is memory-efficient and fast for very large corpora. Often combined with product quantization (IVF-PQ) for billion-scale search with compressed vectors — see Vector Quantization.

Diagram: HNSW query navigation

sequenceDiagram
    participant Q as Query vector
    participant L2 as Layer 2 entry
    participant L1 as Layer 1
    participant L0 as Layer 0
    Q->>L2: greedy walk to nearest
    L2->>L1: drop to next layer
    L1->>L0: drop to base layer
    L0->>L0: expand neighborhood
    L0-->>Q: top-k neighbors

Higher layers provide coarse navigation; layer 0 returns final candidates— efSearch controls expansion width.

Architecture

Vector search sits between embedding and generation in a RAG pipeline:

Diagram: Vector search in RAG stack

flowchart TB
    Docs[Documents] --> Chunk[Chunk]
    Chunk --> Embed[Embed]
    Embed --> Index[ANN index]
    Query[User query] --> QEmbed[Embed query]
    QEmbed --> Index
    Index --> Filter[Metadata filter]
    Filter --> Fetch[Fetch payloads]
    Fetch --> Rerank[Reranker]
    Rerank --> LLM[LLM]

ANN search returns chunk IDs and scores; text payloads are fetched separately in most vector databases.

Source: Meta AI — RAG paper

Latency ranges are representative of typical deployments; actual performance depends on hardware, corpus size, deployment architecture, vector database, embedding model, and workload.

Step-by-Step Flow

Step 1: Choose embedding model and metric. Match distance function to model training objective. Record model version in index metadata for re-index triggers.

Step 2: Build ANN index. Select HNSW for general text RAG; IVF-PQ for billion-scale with memory constraints. Set M and efConstruction at build time.

Step 3: Benchmark exact vs ANN recall. Run brute-force k-NN on a sample as ground truth. Measure recall@k at various ef_search values.

Step 4: Tune query-time parameters. Increase ef_search until recall@5 meets target (typically ≥90%). Document latency/recall tradeoff.

Step 5: Set retrieval k. Retrieve 15–50 candidates; rerank to 3–7 for generation. ANN recall gaps are recovered by reranking, not by retrieving only top-5.

Step 6: Add metadata pre-filtering. For multi-tenant apps, constrain ANN search with metadata filters — never post-filter at scale.

Step 7: Instrument and evaluate. Log chunk IDs, similarity scores, and latency per query. Run weekly retrieval evaluation on golden set.

Real Production Example

Tuning HNSW for a 2M-chunk documentation index on Qdrant:

from dataclasses import dataclass
import time
import numpy as np

@dataclass
class SearchConfig:
    ef_search: int = 128
    top_k: int = 20
    metric: str = "cosine"

class VectorSearchBenchmark:
    def __init__(self, index, queries: list, ground_truth: dict):
        self.index = index
        self.queries = queries
        self.ground_truth = ground_truth  # query_id -> relevant chunk_ids

    def recall_at_k(self, retrieved: list[str], relevant: set[str], k: int) -> float:
        if not relevant:
            return 1.0
        return len(set(retrieved[:k]) & relevant) / len(relevant)

    def benchmark_ef_search(self, ef_values: list[int]) -> list[dict]:
        results = []
        for ef in ef_values:
            self.index.set_ef_search(ef)
            recalls, latencies = [], []
            for qid, query_vec in self.queries:
                start = time.perf_counter()
                hits = self.index.search(query_vec, top_k=20)
                latencies.append((time.perf_counter() - start) * 1000)
                recalls.append(self.recall_at_k(
                    [h.id for h in hits],
                    self.ground_truth[qid],
                    k=5,
                ))
            results.append({
                "ef_search": ef,
                "recall_at_5": np.mean(recalls),
                "p95_latency_ms": np.percentile(latencies, 95),
            })
        return results

# Example output:
# ef=64  → recall@5=0.91, p95=8ms
# ef=128 → recall@5=0.96, p95=14ms  ← chosen
# ef=256 → recall@5=0.98, p95=28ms

The team chose ef_search=128 — 96% recall@5 at 14ms p95. Increasing to 256 gained 2% recall but doubled latency. They added Cohere reranking to recover precision on the fused top-20.

Design Decisions

Comparisons

Vector search vs keyword search (BM25)

HNSW vs IVF

Vector search vs late interaction retrieval

Late interaction (ColBERT) compares token-level embeddings at query time — higher quality, 10–100× query cost. Use ANN bi-encoder search first; escalate when reranking still misses targets.

Vector search vs full corpus in context

Stuffing entire corpora into 128K+ context windows avoids ANN infrastructure but scales poorly on cost and latency. Vector search retrieves 2–8K tokens of relevant context — practical for unlimited corpus sizes.

Decision tree: exact vs approximate search

Decision tree: ANN configuration

flowchart TD
    A[Corpus size?] -->|Under 50K| B[Exact k-NN OK]
    A -->|50K–10M| C[HNSW default]
    A -->|Over 10M| D{Memory constrained?}
    D -->|Yes| E[IVF-PQ + quantization]
    D -->|No| C
    C --> F[Benchmark ef_search]
    F --> G{Recall@5 ≥ 90%?}
    G -->|No| H[Increase ef_search or M]
    G -->|Yes| I[Add reranker]
    I --> J[Monitor weekly evals]

Tune recall before latency—a fast index that misses documents wastes everything downstream.

Head-to-head vector database tooling

Compare ANN implementations in Best Vector Databases: Qdrant vs Pinecone · Qdrant vs Weaviate · Milvus vs Qdrant · pgvector vs Pinecone.

Common Mistakes

1. Using default ANN parameters. Out-of-box ef_search values optimize for generic benchmarks, not your corpus. Benchmark recall@k on your golden set.

2. Wrong distance metric. L2 on L2-normalized text-embedding-3 vectors can rank differently than cosine or dot product. Match the metric to your model's normalization and index configuration.

3. Skipping reranking. ANN returns approximate neighbors with imperfect ordering. A cross-encoder reranker is one of the highest-ROI quality fixes.

4. Post-filtering at scale. Filtering after ANN search when 90% of vectors are excluded wastes compute. Use native pre-filtered indexes.

5. Mixing embedding models in one index. Vectors from different models occupy different semantic spaces. Search quality collapses.

6. Ignoring normalization. Cosine similarity assumes comparable vector magnitudes. Normalize if your pipeline doesn't already.

7. Confusing vector search with keyword search. SKU-9284 won't match semantically unless the embedding space encodes it. Use hybrid search for exact identifiers.

Where It Breaks Down

• Rare tokens and IDs — Embeddings compress meaning; exact codes and SKUs need sparse retrieval alongside dense search.

• Negation and fine distinctions — "refundable" vs "non-refundable" may be close in embedding space. Reranking and metadata help.

• Corpus scale vs index memory — HNSW memory grows with M × n. A 10M-vector index can require 30–60GB RAM.

• Cold start — New documents aren't searchable until embedded and indexed. Plan incremental upsert latency.

• Adversarial queries — Crafted inputs can retrieve unrelated content. Monitor similarity score distributions; set minimum thresholds.

• Filtered search on small subsets — When metadata filters reduce searchable set below k documents, ANN has nothing to rank.

When NOT to Use Vector Search Alone

Skip dense-only vector search when:

1. Queries are predominantly exact-match — SKUs, CVE IDs, legal citations, error codes. Add hybrid search or BM25.

2. Corpus is under 1,000 chunks — brute-force exact k-NN is fast enough and gives 100% recall without index complexity.

3. You need multi-hop entity reasoning — "Which suppliers of our Tier-1 vendor had violations?" requires GraphRAG, not similarity search.

4. Sub-10ms search latency is mandatory — aggressive ANN settings or heavy filtering may exceed budget; cache frequent queries.

5. Embedding model cannot represent your domain — jargon-heavy corpora may need domain fine-tuned embedders before ANN tuning matters.

Prefer keyword-only when you have no embedding budget and queries are purely lexical. Prefer hybrid + rerank as the production default for RAG.

Running in Production

Best Practice

✅ Best Practices — Log top-k results with scores, version embedding models per index, enforce metadata filters at ANN query time, and benchmark recall@k before changing ANN parameters.

Production Tip

Log top-k results with similarity scores for every query. When users report bad answers, you need retrieval traces — not just the final LLM output.

Diagram: Index lifecycle

stateDiagram-v2
    [*] --> Build: initial index
    Build --> Serve: queries
    Serve --> Upsert: doc changes
    Upsert --> Serve
    Serve --> Rebuild: embedding model change
    Rebuild --> Build
    Serve --> Tune: recall regression
    Tune --> Serve

Full rebuild is required when embedding model version changes—incremental upsert handles document edits.

Related Guides

• Foundations: Embeddings · Embedding Models · Chunking Strategies · RAG

• Retrieval stack: Hybrid Search · Metadata Filtering · ANN Indexes · Vector Quantization · Re-ranking

• Quality & ops: Retrieval Evaluation · Vector Databases · Late Interaction Retrieval

• Vector stores: Qdrant · Weaviate · Pinecone · Milvus · pgvector — compare in Best Vector Databases.

• Head-to-heads: Qdrant vs Pinecone · Qdrant vs Weaviate · Milvus vs Qdrant · pgvector vs Pinecone

If you understood this topic, read next:

Diagram: Recommended learning path

flowchart LR
    A[Embeddings] --> B[Vector Search]
    B --> C[Vector DBs]
    C --> D[Hybrid]
    D --> E[Rerank]

Prerequisites: Embeddings

Next topics: Vector Databases · Hybrid Search · Re-ranking

Estimated time: 45 min · Difficulty: Intermediate

Interview Questions

1. What is the difference between exact k-NN and ANN?

   • Expected: exact compares all n vectors (100% recall); ANN uses index structure for sub-linear search with 90–99%+ recall tradeoff.

2. Why must distance metric match the embedding model?

   • Expected: for L2-normalized models like text-embedding-3-small, cosine and dot product rank identically; mismatch between metric and normalization silently degrades ranking.

3. What does ef_search control in HNSW?

   • Expected: query-time candidate list size; higher = better recall, slower queries — tune via benchmark on golden set.

4. Why retrieve 20–50 if you only need 5 for the LLM?

   • Expected: ANN recall gaps; cross-encoder reranker fixes ordering within larger candidate pool.

5. When would you choose IVF-PQ over HNSW?

   • Expected: billion-scale, memory-constrained indexes where slight recall loss is acceptable (Vector Quantization).

6. What triggers a full index rebuild vs incremental upsert?

   • Expected: embedding model version change → full rebuild; single document edit → incremental upsert.

7. Why doesn't vector search alone handle SKU queries?

   • Expected: dense embeddings lose exact token info; use hybrid search with BM25.

8. Name three production monitoring signals for vector search.

   • Expected: recall@k on golden set, p95 ANN latency, similarity score distribution, empty-result rate, index freshness.

Key Takeaways

• Vector search finds semantically similar content by comparing embedding vectors — the core of RAG retrieval.
• Exact k-NN is correct but slow; ANN (HNSW, IVF) trades small recall loss for orders-of-magnitude speedup.
• Match your distance metric to your embedding model. Mismatch silently degrades ranking.
• Tune ANN parameters on your data with recall@k benchmarks — defaults are not enough.
• Retrieve many, rerank few. ANN gets candidates; cross-encoders fix ordering.
• Compare vector stores in Best Vector Databases before committing to infrastructure.

FAQs

What's the difference between vector search and a vector database?

Vector search is the algorithm (find nearest neighbors). A vector database stores vectors, builds ANN indexes, handles filtering, replication, and APIs. You can run vector search with FAISS in memory; a vector DB adds production operations.

When should I use exact k-NN instead of ANN?

On smaller corpora with modern hardware, exact search is often fast enough and gives 100% recall. Use exact search as an eval baseline to measure ANN recall loss.

HNSW vs IVF — which should I choose?

HNSW is the default for text RAG at millions of vectors — best recall/latency balance. IVF-PQ suits billion-scale when memory is constrained and slight recall loss is acceptable.

What is ef_search?

HNSW query-time parameter controlling how many candidates are explored. Higher ef_search = better recall, slower queries. Tune by benchmarking on your data.

Why do my search results seem irrelevant?

Common causes: wrong metric, mixed embedding models, stale index, corpus doesn't contain the answer, or ANN parameters too aggressive. Check similarity scores — low scores indicate poor match.

How many results should I retrieve?

Retrieve 15–50, rerank to 3–7 for generation. Retrieving only top-5 without reranking misses ANN recall gaps.

Does vector search work for multi-tenant apps?

Yes — use metadata filters (tenant_id) with pre-filtered indexes. Never rely on post-hoc filtering of results.

How do I re-index when changing embedding models?

Full rebuild required. Vectors from different models are incompatible. Plan blue-green index migration.

What's a good recall@5 target?

90%+ for production RAG. Below 85%, fix chunking and embeddings before tuning ANN parameters.

Can I combine vector search with SQL filters?

Yes — production vector DBs support metadata filtering during ANN search. pgvector uses natural SQL WHERE clauses.

References

• Efficient and Robust Approximate Nearest Neighbor Search Using HNSW (Malkov & Yashunin, 2018)
• FAISS: A Library for Efficient Similarity Search (Johnson et al.)
• OpenAI Embeddings Guide
• Qdrant Vector Search Documentation
• Pinecone Approximate Nearest Neighbor Explained

Further Reading

• Meta AI — Retrieval-Augmented Generation (2020)
• Cohere Rerank Documentation
• Milvus Index Documentation