Semantic Search Guide

TL;DR

• Semantic search retrieves documents by meaning - it compares query and document embeddings in vector space instead of matching exact keywords.

• It solves vocabulary mismatch - a query for "login failure" can retrieve docs about "authentication errors" because their vectors are nearby.

• Keyword search (BM25) still matters - exact IDs, SKUs, error codes, and rare terms often need lexical matching; production systems combine both via hybrid search.

• The pipeline is index → embed → store → query → rank - offline embedding of chunks, online nearest-neighbor lookup, optional reranking before the LLM sees results.

• Retrieval quality is the ceiling for RAG - semantic search is the retrieval layer in most RAG systems; bad search means bad answers regardless of model size.

Why This Matters

Users do not write queries the way documents are authored. Support tickets say "can't sign in"; runbooks say "OAuth token refresh failure." Keyword search returns nothing useful. Semantic search closes the gap by encoding meaning into dense vectors.

Every production AI search system - enterprise knowledge bases, legal discovery, code search, customer support RAG - depends on semantic retrieval at some layer. The engineering work is not "call an embedding API"; it is choosing models, chunking strategy, index parameters, hybrid weighting, and evaluation metrics that hold up under real traffic.

If you ship semantic search without measuring recall on representative queries, you will discover failures in production when users ask about topics your docs cover under different terminology.

The Problem Semantic Search Solves

Traditional keyword retrieval (BM25, TF-IDF, inverted indexes) ranks documents by term overlap and frequency. It excels when query terms appear verbatim in documents. It fails when:

1. Synonyms and paraphrases differ - "rate limiting" vs "429 throttling."

2. Users ask questions, documents state facts - "How do I reset my password?" vs a procedural paragraph that never uses the word "reset."

3. Cross-lingual retrieval is required - query in English, corpus in Spanish.

4. Conceptual similarity matters more than lexical overlap - finding "refund policy" content when the user asks about "getting money back."

Semantic search maps text to vectors where proximity reflects meaning. Similar concepts cluster regardless of surface form. Combined with approximate nearest neighbor (ANN) indexes, you can compare a query against millions of vectors in tens of milliseconds.

What Is Semantic Search?

Semantic search is retrieval where ranking is driven by vector similarity between a query embedding and document (or chunk) embeddings. The core operation:

score(query, document) = cosine_similarity(embed(query), embed(document))

Higher scores mean more semantically related content. Unlike keyword search, there is no requirement for shared tokens.

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(text: str) -> list[float]:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return resp.data[0].embedding

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

query = "How do I fix memory leaks in Python?"
docs = [
    "Debugging heap allocation issues in CPython applications.",
    "Best pizza restaurants in downtown Chicago.",
    "Using tracemalloc to profile memory usage in Python services.",
]

query_vec = embed(query)
for doc in docs:
    score = cosine_similarity(query_vec, embed(doc))
    print(f"{score:.3f}  {doc[:60]}...")
# 0.72  Debugging heap allocation...
# 0.31  Best pizza restaurants...
# 0.81  Using tracemalloc to profile memory...

In production, you never brute-force compare against every document. You store precomputed embeddings in a vector database and use ANN algorithms (HNSW, IVF) for sub-linear lookup. See vector search for index internals.

Semantic Search vs Keyword Search

Neither approach wins alone. Production systems use hybrid search - weighted combination of BM25 and vector scores - to get recall from semantics and precision on exact terms.

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Important

Semantic search is not magic - it retrieves statistically similar text, not guaranteed truth. Always validate retrieved content before passing to an LLM, and add citations so users can verify sources.

How Semantic Search Works in RAG

Semantic search is the retrieval step in most RAG pipelines:

The generator is a seq2seq model that consumes retrieved passages and produces the final answer token by token.

Source: Meta AI

Indexing (offline): chunk documents → embed each chunk → store vector + text + metadata.

Querying (online): embed query → ANN retrieve → optional metadata filter → rerank → pass to LLM.

The embedding model used at query time must match the model used at index time. Mixing models without re-indexing collapses retrieval quality.

Architecture

Real Production Example

class SemanticRetriever:
    def __init__(self, store, embed_fn, reranker=None):
        self.store = store
        self.embed = embed_fn
        self.reranker = reranker

    def search(
        self,
        query: str,
        tenant_id: str,
        top_k: int = 20,
        top_n: int = 5,
    ) -> list[dict]:
        vector = self.embed(query)
        candidates = self.store.query(
            vector=vector,
            filter={"tenant_id": tenant_id},
            top_k=top_k,
        )
        if self.reranker:
            return self.reranker.rerank(query, candidates, top_n=top_n)
        return candidates[:top_n]

Multi-tenant isolation happens at the index filter layer - never rely on the LLM to exclude other customers' documents.

Decision Matrix

⚠ Common Mistakes

1. Skipping evaluation - Deploying without a golden query set means you discover synonym gaps from user complaints, not metrics.

2. Chunks too large - 2,000-token chunks dilute embedding signal; one topic per chunk improves precision.

3. No metadata filters - Semantic similarity ignores access control; filter by tenant, doc type, and date at query time.

4. Re-embedding without versioning - Changing embedding models without full re-index creates a mixed vector space where search fails silently.

5. Vector-only for technical docs - Error codes (ECONNREFUSED), API paths, and version numbers need BM25 or hybrid search.

6. Ignoring query-document asymmetry - Some models offer separate query/passage encoders (E5 query: / passage: prefixes). Use them correctly.

Where It Breaks Down

Semantic search retrieves locally similar passages. It does not:

• Traverse multi-hop relationships - "Which suppliers of our Tier-1 vendor had violations?" needs graph traversal, not vector lookup.

• Handle precise negation - Embeddings often treat "refundable" and "non-refundable" as similar.

• Guarantee numeric accuracy - "$4.2M" and "$42M" may embed similarly.

• Replace freshness guarantees - Stale index means stale answers; monitor index lag.

For relationship-heavy queries, combine semantic search with GraphRAG. For exact-match-heavy corpora, prioritize hybrid retrieval.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Production Checklist

• Golden query set (50–200 representative questions) with expected document IDs
• Same embedding model for indexing and querying; version tracked in index metadata
• Chunk size and overlap tuned against recall@k - not arbitrary defaults
• Metadata filters for tenant, document type, status, and date range
• Hybrid search enabled if corpus contains IDs, codes, or rare proper nouns
• Cross-encoder reranker on top-20 candidates before LLM context injection
• Incremental re-index pipeline for document updates
• Full re-index runbook when changing embedding models
• Retrieval logs with scores for debugging bad answers
• Citations mapped from chunk metadata to source URLs in the UI

Warning

Never ship semantic search as the sole retrieval mechanism for regulated or security-sensitive content without access-control filters enforced at the index layer.

Ecosystem

• Embedding APIs: OpenAI text-embedding-3, Cohere embed-v3, Voyage AI

• Open models: BGE-M3, E5, Nomic Embed, GTE - self-host for data residency

• Vector stores: Pinecone, Qdrant, Weaviate, Milvus, pgvector

• Hybrid search: Elasticsearch dense_vector, OpenSearch k-NN, Weaviate hybrid

• Evaluation: RAGAS retrieval metrics, custom recall@k scripts, LangSmith

Related Technologies

• Embeddings - The vectors semantic search compares.

• Vector Search - ANN indexes and similarity algorithms.

• Hybrid Search - Combining semantic and keyword retrieval.

• RAG - The pattern that consumes semantic search results.

• Re-ranking - Cross-encoders that improve precision after ANN retrieval.

• Chunking Strategies - How you split documents affects embedding quality.

Learning Path

Prerequisites: Embeddings · Large Language Models

Next topics: Hybrid Search · RAG · Vector Databases

Estimated time: 45 min · Difficulty: Intermediate

FAQs

When should I use semantic search instead of Elasticsearch keyword search?

Use semantic search when users and documents use different words for the same concepts. Keep keyword search (or hybrid) when exact term matching matters - SKUs, error codes, legal citations, API endpoint paths.

How many documents can semantic search handle?

Millions to billions of vectors with dedicated vector databases and ANN indexes. Bottleneck is embedding cost at index time and recall quality at scale, not storage capacity.

What similarity threshold should I use?

There is no universal cutoff. Measure on your data: plot score distributions for relevant vs irrelevant pairs. Use reranking rather than hard thresholds when possible.

Do I need to re-embed everything when documents change?

Only changed chunks. Upsert new vectors and delete stale ones. Full re-embed required when switching embedding models.

Why does semantic search miss obvious matches?

Common causes: chunk fragmentation (answer split across chunks), wrong embedding model for domain, query-document asymmetry (question vs statement style), or stale index.

How does semantic search relate to RAG?

Semantic search is typically the retrieval component in RAG. RAG adds chunking, indexing, reranking, and LLM generation on top of vector retrieval.

Can semantic search work across languages?

Yes, with multilingual embedding models (BGE-M3, Cohere embed-multilingual). Query in one language can retrieve documents in another if semantics align.

What embedding dimensions should I use?

Higher dimensions (1536) capture more nuance; lower (256–512) reduce storage and latency with modest quality tradeoff. OpenAI supports dimension reduction at embed time - evaluate on your corpus.

Should I embed full documents or chunks?

Chunks. Full-document embeddings average away specific facts; retrieval returns imprecise passages. Chunk 256–512 tokens with overlap.

How do I debug a bad search result?

Trace: Was the correct chunk embedded? Is it in top-k? What was its score vs winners? Is metadata filtering excluding it? Log all four.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Semantic search finds content by meaning using embedding similarity - essential when queries and documents use different vocabulary. - Keyword search still wins on exact identifiers; production systems use hybrid retrieval. - In RAG, semantic search is the retrieval layer - its quality caps answer quality. - Evaluate recall@k on a golden query set before tuning models or prompts.

• Enforce access control via metadata filters; chunk appropriately; rerank before generation. - Re-index fully when changing embedding models; monitor retrieval metrics continuously.