Vector Search Guide

TL;DR

• Vector search finds the k vectors closest to a query vector in embedding space - the core operation behind semantic search and RAG retrieval.

• Exact k-NN compares against every vector - accurate but O(n) per query. Impractical beyond ~100K vectors on CPU.

• Approximate Nearest Neighbor (ANN) indexes trade recall for speed - HNSW and IVF reduce search to milliseconds over millions of vectors.

• Distance metric must match your embedding model - cosine similarity for normalized vectors, dot product for OpenAI embeddings, L2 for some image models.

• Always measure recall@k on your data - ANN parameters that work for one corpus fail on another. Default configs are starting points, not answers.

Why This Matters

Every RAG pipeline depends on vector search. When a user asks "Why am I getting a 429 error?", your system embeds the query, searches millions of chunk vectors, and returns the top matches. If search is slow, users wait. If search misses the right chunk, the LLM hallucinates - no prompt fixes bad retrieval.

Understanding vector search - not just which database to use - lets you debug recall failures, tune latency, and choose between exact and approximate search for your scale. The difference between 85% and 99% recall@5 is the difference between a useful product and a demo.

The Problem Vector Search Solves

Semantic matching requires comparing vectors. Embeddings map text to high-dimensional vectors (typically 384–3072 dimensions). Finding relevant documents means finding vectors close to the query vector. Without efficient search, you brute-force compare against every vector in your corpus.

Brute force doesn't scale. Cosine similarity between one query and 1 million 1536-dimensional vectors requires 1 billion multiply-add operations per query. At 10 QPS, that's unsustainable on CPU without approximation.

Similarity is not keyword matching. Users phrase questions differently than documents. Vector search finds "rate limit exceeded" when the user types "429 error" - something BM25 handles differently. Production systems often combine both via hybrid search.

Vector search solves the scale and semantics problem for retrieval - fast, meaning-aware lookup over embedding indexes.

What Is Vector Search?

Vector search (also called similarity search or nearest neighbor search) is the operation of finding the k vectors in a collection that are most similar to a query vector, according to a distance or similarity metric.

Given:

• Query vector q ∈ ℝᵈ
• Corpus of n vectors v₁, v₂, ..., vₙ ∈ ℝᵈ
• Similarity function sim(q, v)

Return the k vectors with highest sim(q, v).

This is distinct from a vector database, which is storage plus indexing plus filtering. Vector search is the algorithm; the database is the system that runs it at scale.

How Vector Search Works

Similarity Metrics

Warning

Mismatching metric and embedding model destroys recall. OpenAI text-embedding-3 models are trained for dot product. Using L2 distance without normalization can rank incorrectly.

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def brute_force_search(
    query: np.ndarray,
    corpus: np.ndarray,
    k: int = 5,
) -> list[int]:
    """Exact k-NN - baseline for eval, not production at scale."""
    scores = np.array([cosine_similarity(query, v) for v in corpus])
    return np.argsort(scores)[::-1][:k].tolist()

Exact k-NN vs Approximate Nearest Neighbor (ANN)

Exact k-NN computes similarity against every vector. Simple, correct, slow.

ANN builds an index structure that navigates to likely neighbors without scanning the full corpus. Returns approximate nearest neighbors - usually correct, occasionally misses.

Decision Trade-off

ANN trades recall for speed. At 95% recall@10, you miss the right document in 5% of queries. For high-stakes retrieval, increase ef_search or use exact search on a filtered subset.

HNSW (Hierarchical Navigable Small World)

HNSW is the dominant ANN algorithm in production vector databases. It builds a multi-layer graph where each node connects to nearby neighbors.

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Query flow:

1. Start at the top layer entry point.
2. Greedy walk to the nearest neighbor at each layer.
3. Drop to the next layer and repeat.
4. At layer 0, expand local neighborhood to find top-k.

Key parameters:

• M - max connections per node (higher = better recall, more memory)
• efConstruction - build-time search width (higher = better index quality, slower build)
• efSearch - query-time search width (higher = better recall, slower queries)

IVF (Inverted File Index)

IVF partitions the vector space into clusters (Voronoi cells). At query time, search only the nearest clusters - not the full corpus.

The generator is a seq2seq model that consumes retrieved passages and produces the final answer token by token.

Key parameters:

• nlist - number of clusters (more = finer partitions)
• nprobe - clusters to search at query time (higher = better recall)

IVF is memory-efficient and fast for very large corpora. Often combined with product quantization (IVF-PQ) for billion-scale search with compressed vectors.

Architecture

Vector search sits between embedding and generation in a RAG pipeline:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Source: Meta AI

Real Production Example

Tuning HNSW for a 2M-chunk documentation index:

from dataclasses import dataclass
import time
import numpy as np

@dataclass
class SearchConfig:
    ef_search: int = 128
    top_k: int = 20
    metric: str = "cosine"

class VectorSearchBenchmark:
    def __init__(self, index, queries: list, ground_truth: dict):
        self.index = index
        self.queries = queries
        self.ground_truth = ground_truth  # query_id -> relevant chunk_ids

    def recall_at_k(self, retrieved: list[str], relevant: set[str], k: int) -> float:
        if not relevant:
            return 1.0
        return len(set(retrieved[:k]) & relevant) / len(relevant)

def benchmark_ef_search(self, ef_values: list[int]) -> list[dict]:
        results = []
        for ef in ef_values:
            self.index.set_ef_search(ef)
            recalls, latencies = [], []
            for qid, query_vec in self.queries:
                start = time.perf_counter()
                hits = self.index.search(query_vec, top_k=20)
                latencies.append((time.perf_counter() - start) * 1000)
                recalls.append(self.recall_at_k(
                    [h.id for h in hits],
                    self.ground_truth[qid],
                    k=5,
                ))
            results.append({
                "ef_search": ef,
                "recall_at_5": np.mean(recalls),
                "p95_latency_ms": np.percentile(latencies, 95),
            })
        return results

# Example output:
# ef=64  → recall@5=0.91, p95=8ms
# ef=128 → recall@5=0.96, p95=14ms  ← chosen
# ef=256 → recall@5=0.98, p95=28ms

The team chose ef_search=128 - 96% recall@5 at 14ms p95. Increasing to 256 gained 2% recall but doubled latency. They added reranking to recover precision.

Decision Matrix

⚠ Common Mistakes

1. Using default ANN parameters. Out-of-box ef_search values optimize for generic benchmarks, not your corpus. Benchmark recall@k on your golden set.

2. Wrong distance metric. L2 on unnormalized OpenAI embeddings ranks differently than dot product. Check model docs.

3. Skipping reranking. ANN returns approximate neighbors with imperfect ordering. A cross-encoder reranker is the highest-ROI quality fix.

4. Post-filtering at scale. Filtering after ANN search when 90% of vectors are excluded wastes compute. Use native pre-filtered indexes.

5. Mixing embedding models in one index. Vectors from different models occupy different semantic spaces. Search quality collapses.

6. Ignoring normalization. Cosine similarity assumes comparable vector magnitudes. Normalize if your pipeline doesn't already.

Common Mistake

Confusing vector search with keyword search. "SKU-9284" won't match semantically unless the embedding space encodes it. Use hybrid search for exact identifiers.

Where It Breaks Down

• Rare tokens and IDs - Embeddings compress meaning; exact codes and SKUs need sparse retrieval alongside dense search.

• Negation and fine distinctions - "refundable" vs "non-refundable" may be close in embedding space. Reranking and metadata help.

• Corpus scale vs index memory - HNSW memory grows with M × n. A 10M-vector index can require 30–60GB RAM.

• Cold start - New documents aren't searchable until embedded and indexed.

Plan incremental upsert latency.

• Adversarial queries - Crafted inputs can retrieve unrelated content. Monitor retrieval scores; set minimum similarity thresholds.

Running Vector Search in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Production Tip

Log top-k results with similarity scores for every query. When users report bad answers, you need retrieval traces - not just the final LLM output.

Ecosystem

Related Technologies

• Vector Databases: Storage systems that run vector search at scale with filtering and ops tooling.

• Embeddings: The vectors being searched - model choice determines search quality ceiling.

• Embedding Models: Model selection and dimension tradeoffs.

• Hybrid Search: Combining dense vector search with sparse BM25 for production recall.

• Re-ranking: Cross-encoders that fix ANN ordering imprecision.

• Retrieval Evaluation: Measuring whether vector search finds the right documents.

Learning Path

Prerequisites: Embeddings

Next topics: Vector Databases · Hybrid Search · Re-ranking

Estimated time: 45 min · Difficulty: Intermediate

FAQs

What's the difference between vector search and a vector database?

Vector search is the algorithm (find nearest neighbors). A vector database is a system that stores vectors, builds ANN indexes, handles filtering, replication, and APIs. You can run vector search with FAISS in memory; a vector DB adds production operations.

When should I use exact k-NN instead of ANN?

Under ~50K vectors on modern hardware, exact search is fast enough and gives 100% recall. Use exact search as an eval baseline to measure ANN recall loss.

HNSW vs IVF - which should I choose?

HNSW is the default for text RAG at millions of vectors - best recall/latency balance. IVF-PQ suits billion-scale when memory is constrained and slight recall loss is acceptable.

What is ef_search?

HNSW query-time parameter controlling how many candidates are explored. Higher ef_search = better recall, slower queries. Tune by benchmarking on your data.

Why do my search results seem irrelevant?

Common causes: wrong metric, mixed embedding models, stale index, corpus doesn't contain the answer, or ANN parameters too aggressive. Check similarity scores - low scores indicate poor match.

How many results should I retrieve?

Retrieve 15–50, rerank to 3–7 for generation. Retrieving only top-5 without reranking misses ANN recall gaps.

Does vector search work for multi-tenant apps?

Yes - use metadata filters (tenant_id) with pre-filtered indexes. Never rely on post-hoc filtering of results.

How do I re-index when changing embedding models?

Full rebuild required. Vectors from different models are incompatible. Plan downtime or blue-green index migration.

What's a good recall@5 target?

90%+ for production RAG. Below 85%, fix chunking and embeddings before tuning ANN parameters.

Can I combine vector search with SQL filters?

Yes - production vector DBs support metadata filtering during ANN search. See Metadata Filtering.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Retrieval Documentation
• LlamaIndex RAG Documentation
• Pinecone RAG Guide

Further Reading

• OpenAI Embeddings Guide
• Cohere Rerank Documentation
• RAGAS Evaluation Framework
• Meta AI RAG Paper Repository

Summary

• Vector search finds semantically similar content by comparing embedding vectors - the core of RAG retrieval. - Exact k-NN is correct but slow; ANN (HNSW, IVF) trades small recall loss for orders-of-magnitude speedup. - Match your distance metric to your embedding model. Mismatch silently degrades ranking. - Tune ANN parameters on your data with recall@k benchmarks - defaults are not enough. - Retrieve many, rerank few.

ANN gets candidates; cross-encoders fix ordering. - Log retrieval traces in production. Debugging bad answers starts with what was retrieved.

Production Checklist

• Distance metric matches embedding model documentation
• ANN recall@k benchmarked on golden set (target ≥ 90% at k=5)
• ef_search or nprobe tuned - documented with latency/recall tradeoff
• Single embedding model version per index
• Metadata pre-filtering for multi-tenant isolation
• Top-k set to 15–50 with reranker to top-5
• Retrieval logged with chunk IDs and similarity scores
• Incremental upsert pipeline for document changes
• Full re-index plan for embedding model upgrades
• Minimum similarity threshold to reject low-confidence matches