Embedding Models Guide

TL;DR

• Embedding models convert text to dense vectors that capture semantic meaning — the foundation of all vector-based retrieval in RAG.

• Model choice affects retrieval quality more than vector database choice — a better embedding model on a simple index outperforms a worse model on an optimized index.

• Example starting points (current as of writing): OpenAI text-embedding-3-small and bge-large-en-v1.5 are common production choices for English RAG — evaluate on your domain before committing.

• Changing embedding models requires full re-indexing — vectors from different models are not comparable.

• Match embedding dimensions, distance metric, chunk size, and query prefixes to your model's training characteristics.

Why This Matters

The embedding model is the lens through which your RAG system "sees" your documents. It determines whether a query about "rate limiting" finds your document about "API throttling" or returns irrelevant results about "network bandwidth."

Teams often spend weeks evaluating vector databases and LLMs while using whatever embedding model their framework defaults to. This is backwards. Embedding model quality is one of the highest-leverage decisions in the indexing pipeline — it affects every retrieval query until you re-index.

A domain-specific embedding model can improve recall@5 materially over a general-purpose model on specialized corpora (legal, medical, code). The cost difference between models is negligible compared to the downstream impact on answer quality — benchmark on your corpus; published leaderboard gaps do not always transfer.

The Problem Embedding Models Solve

Text is unstructured and unsearchable by meaning. Keyword search matches characters; embedding models match concepts.

The core challenge: compress variable-length text into a fixed-size vector that preserves semantic similarity. "Dog" and "puppy" should be close. "Dog" and "car" should be far apart. "Rate limit exceeded" and "API throttling error" should be close despite sharing no words.

Embedding models solve this by training neural networks on massive text corpora to produce vectors where semantic similarity corresponds to geometric proximity. This enables finding documents by meaning, cross-lingual retrieval, and powering the retrieval step in every RAG pipeline. The vectors themselves are defined in Embeddings; this guide focuses on choosing and operating the models that produce them.

How We Got Here

Embedding models evolved from research benchmarks to production API defaults in under five years:

Diagram: Evolution of embedding models

timeline
    title Embedding model milestones
    2019 : Sentence-BERT contrastive training
    2020 : DPR bi-encoder retrieval
    2022 : OpenAI ada-002 API default
    2023 : BGE / E5 open-source surge
    2024 : Matryoshka + multilingual v3
    2025 : Domain-specific + long-context embedders

Production teams moved from self-hosted SBERT to API embeddings, then back to open-source when data residency and cost at scale mattered.

The MTEB leaderboard standardised comparison across 56 tasks. Use it for initial screening — then validate on your corpus with retrieval evaluation.

What Is an Embedding Model?

An embedding model is a neural network (typically a transformer encoder) trained to map text into a fixed-dimensional vector space. Similar texts produce vectors that are close together by cosine similarity or dot product.

from openai import OpenAI

client = OpenAI(timeout=30.0)
response = client.embeddings.create(
    model="text-embedding-3-small",
    input="How do I configure rate limiting for the API?",
)
vector = response.data[0].embedding  # list of 1536 floats

Key properties:

How Embedding Models Work

Bi-encoder architecture

Most modern text embedding models use a bi-encoder: a transformer encoder followed by mean pooling and optionally a projection layer.

Diagram: Bi-encoder vs cross-encoder

flowchart TB
    subgraph bi [Bi-encoder — indexing + first-stage retrieval]
        Q1[Query text] --> TE1[Transformer encoder]
        D1[Document text] --> TE2[Transformer encoder]
        TE1 --> VQ[Query vector]
        TE2 --> VD[Doc vector]
        VQ --> SIM[Cosine similarity]
        VD --> SIM
    end
    subgraph cross [Cross-encoder — reranking only]
        QD[Query + doc concatenated] --> TE3[Single transformer]
        TE3 --> SCORE[Relevance score]
    end

Bi-encoders pre-compute document vectors; cross-encoders score pairs jointly — too slow for full-corpus indexing.

Bi-encoder models embed queries and documents independently. This enables pre-computing document embeddings (fast indexing) and embedding only the query at search time (low latency). The trade-off: query and document embeddings are computed separately, so the model cannot capture fine-grained query-document interactions.

Cross-encoder models (used in re-ranking, not initial retrieval) concatenate query and document and run them through the transformer together. More accurate but too slow for indexing entire corpora.

Source: Sentence Transformers Documentation

Training methods

Matryoshka embeddings

OpenAI's text-embedding-3 and Nomic's models support Matryoshka Representation Learning — vectors where the first N dimensions are useful on their own:

# Full 1536-dim embedding
full_vector = embed("query text", dimensions=1536)

# Truncated 256-dim still works (lower quality, less storage)
compact_vector = embed("query text", dimensions=256)

This lets you trade quality for storage and speed without re-indexing the full corpus — validate recall after truncation with retrieval evaluation. Pair with vector quantization for further compression.

Diagram: Model selection workflow

sequenceDiagram
    participant Eng as Engineer
    participant Eval as Eval harness
    participant API as Embed API / GPU
    participant VS as Vector store
    Eng->>Eval: define 100 query-doc pairs
    Eng->>API: embed corpus with candidate A
    API->>VS: upsert index A
    Eval->>VS: recall@5 for model A
    Eng->>API: embed corpus with candidate B
    Eval->>VS: recall@5 for model B
    Eval-->>Eng: pick winner + document choice

Never ship a model without recall@k on domain-specific pairs — MTEB ranks candidates, your corpus decides.

Architecture

Embedding models sit in both the indexing and query paths of a RAG pipeline:

Diagram: Embedding model in the RAG stack

flowchart TB
    Docs[Documents] --> Chunk[Chunker]
    Chunk --> EM1[Embedding model]
    EM1 --> VDB[(Vector DB)]
    Query[User query] --> EM2[Same embedding model]
    EM2 --> VS[Vector search]
    VDB --> VS
    VS --> HF[Hybrid fusion]
    HF --> RR[Reranker]
    RR --> LLM[Generator]

One model version serves both paths; rerankers and LLMs are separate model choices downstream.

Source: Survey on RAG (arXiv:2312.10997)

Critical rule: the same embedding model must be used for indexing and querying. Vectors from different models occupy different semantic spaces and cannot be compared. Store embedding_model and embedding_version in vector metadata.

Step-by-Step Flow

Step 1: Define requirements. Language(s), domain specificity, latency budget, data residency, and cost constraints.

Step 2: Select candidates. Start with 2–3 models from the comparison table below — typically one API and one self-hosted option.

Step 3: Build an evaluation set. 50–200 query-document pairs from your actual corpus. Include failure cases from production logs.

Step 4: Benchmark. Compute recall@k, MRR, and nDCG for each model on the same chunks.

Step 5: Consider operational factors. API vs self-hosted, cost at volume, latency, dimension constraints, ANN index compatibility.

Step 6: Index and deploy. Embed the full corpus. Version the model name in metadata. Compare vector DB options in Best Vector Databases.

Step 7: Monitor. Track retrieval quality in production. Re-evaluate when new models release or corpus drifts.

Real Production Example

Evaluating three embedding models on a legal document corpus with proper E5 prefixes and metadata versioning:

import numpy as np
from dataclasses import dataclass
from openai import OpenAI

client = OpenAI(timeout=30.0)

@dataclass
class EvalPair:
    query: str
    relevant_doc_id: str

def embed_openai(text: str, model: str = "text-embedding-3-small") -> list[float]:
    resp = client.embeddings.create(model=model, input=text)
    return resp.data[0].embedding

def embed_e5(text: str, is_query: bool) -> list[float]:
    prefix = "query: " if is_query else "passage: "
    return hf_embed(prefix + text, "intfloat/e5-large-v2")

def recall_at_k(
    embed_query_fn,
    embed_doc_fn,
    corpus: dict[str, str],
    eval_pairs: list[EvalPair],
    k: int = 5,
) -> float:
    doc_ids = list(corpus.keys())
    doc_vectors = np.array([embed_doc_fn(corpus[did]) for did in doc_ids])
    hits = 0
    for pair in eval_pairs:
        q_vec = np.array(embed_query_fn(pair.query))
        sims = doc_vectors @ q_vec / (
            np.linalg.norm(doc_vectors, axis=1) * np.linalg.norm(q_vec)
        )
        top_k_ids = [doc_ids[i] for i in np.argsort(sims)[-k:][::-1]]
        if pair.relevant_doc_id in top_k_ids:
            hits += 1
    return hits / len(eval_pairs)

models = {
    "text-embedding-3-small": (
        lambda q: embed_openai(q),
        lambda d: embed_openai(d),
    ),
    "bge-large-en-v1.5": (
        lambda q: hf_embed(q, "BAAI/bge-large-en-v1.5", prompt="Represent this sentence for searching relevant passages:"),
        lambda d: hf_embed(d, "BAAI/bge-large-en-v1.5"),
    ),
    "e5-large-v2": (
        lambda q: embed_e5(q, is_query=True),
        lambda d: embed_e5(d, is_query=False),
    ),
}

for name, (q_fn, d_fn) in models.items():
    score = recall_at_k(q_fn, d_fn, corpus, eval_pairs, k=5)
    print(f"{name}: recall@5 = {score:.3f}")

Run this on your domain before choosing. MTEB leaderboard scores are directional — your corpus is ground truth.

Model comparison table

Tip

For E5 models, prefix queries with "query: " and documents with "passage: ". This matches their training format and improves retrieval quality significantly.

Design Decisions

Common patterns

• API-first, self-host later — Ship with OpenAI v3; migrate to BGE when volume or privacy thresholds hit.

• Dual-index migration — Index new model in parallel; A/B test recall before decommissioning old index.

• Query prefix discipline — Centralize prefix logic in one embed wrapper; silent prefix bugs are common production failures.

• Title injection — Prepend document title to chunk text before embedding; often +10% recall on structured docs.

Comparisons

API vs self-hosted embedding models

Small vs large embedding models

Bi-encoder vs late interaction (ColBERT)

Decision tree: choosing an embedding model

Decision tree: Embedding model selection

flowchart TD
    A[Start model selection] --> B{Data leaves network OK?}
    B -->|No| C[Self-host BGE / E5]
    B -->|Yes| D[API candidate list]
    C --> E{Multilingual corpus?}
    D --> E
    E -->|Yes| F[multilingual-e5 / Cohere v3]
    E -->|No| G{text-embedding-3-small recall@5 >= 0.75?}
    G -->|Yes| H[Ship API small]
    G -->|No| I[Try large / domain model / fine-tune]
    F --> J[Match chunk size to max tokens]
    H --> J
    I --> J
    J --> K[Weekly eval on golden set]

Evaluate on your corpus at the target chunk size — model quality and chunking interact.

Compare vector DB fit in Best Vector Databases: Qdrant vs Pinecone · Pinecone vs Weaviate · Milvus vs Qdrant.

Common Mistakes

1. Using different models for indexing and querying. Vectors from different models are incomparable. Always use the same model on both paths.

2. Choosing based on MTEB leaderboard alone. Benchmark scores do not predict performance on your specific domain. Always evaluate on your corpus.

3. Ignoring input length limits. Embedding a 2000-token chunk in a 512-token model truncates silently. Match chunk size to model max tokens.

4. Not versioning the model. When you migrate models, you need to know which vectors were created with which model. Store model name/version in metadata.

5. Skipping query preprocessing. E5 models need "query: " prefix. BGE recommends query instructions. Read the model card.

6. Embedding raw HTML/PDF text. Garbage input produces garbage vectors. Clean and parse documents before embedding.

7. Optimizing embedding model before fixing chunking. A perfect embedding model cannot retrieve a complete answer from a badly chunked document.

Where It Breaks Down

Exact matching — Embeddings compress meaning and lose exact token information. Product codes, error IDs, and version numbers need hybrid search alongside embeddings.

Negation and logic — "Refundable" and "non-refundable" may embed similarly. Embeddings capture topic, not logical structure.

Numerical precision — "$4.2 million" and "$42 million" may be indistinguishable in vector space. Do not rely on embeddings for numeric comparison.

Domain shift — A model trained on web text may underperform on internal jargon, abbreviations, or domain-specific terminology. Fine-tune or evaluate carefully.

Multilingual mixing — Using an English-only model on a multilingual corpus silently degrades non-English retrieval. Use multilingual models or language-specific indexes.

Model deprecation — API providers retire models (ada-002 → v3). Plan migration windows and dual-index cutovers before forced shutdown.

When NOT to Use a Dedicated Embedding Model

Skip investing in embedding model selection when:

1. Retrieval is purely structured — SQL, Elasticsearch keyword, or graph queries answer the question; embeddings add no signal.

2. Corpus fits in context — Under ~50 pages of static text, long-context prompting avoids index infrastructure entirely.

3. Real-time factual lookup is external — Live APIs (stock prices, weather) should not be embedded; call tools or APIs directly.

4. You need guaranteed exact match only — SKU lookup with no paraphrase requirement; BM25 alone suffices.

5. Regulatory block on any ML encoding — If neither API nor self-hosted embedding is permitted, use lexical search only.

Prefer fine-tuning an embedding model only after general models fail eval — not as the first step.

Running in Production

Best Practice

✅ Best Practices — Pin model versions, batch index offline, cache hot query embeddings, and block deploys that regress recall@k.

Important

Store the embedding model name and version in your vector database metadata. You will need this for re-indexing, debugging, and compliance audits.

Diagram: Embedding model version lifecycle

stateDiagram-v2
    [*] --> Active: v1 in production
    Active --> Evaluating: v2 candidate
    Evaluating --> DualIndex: recall improves
    DualIndex --> Cutover: traffic shift
    Cutover --> Active: v2 primary
    Active --> Deprecated: v1 drained
    Deprecated --> [*]
    Evaluating --> Active: no improvement

Never swap models in-place — dual-index until eval confirms quality on production queries.

Related Guides

• Foundations: Embeddings · RAG · Chunking Strategies

• Retrieval stack: Vector Search · Hybrid Search · Metadata Filtering · ANN Indexes · Vector Quantization · Re-ranking · Late Interaction Retrieval

• Quality & ops: Retrieval Evaluation · Vector Databases

• Tools: Hugging Face Transformers · LangChain · LlamaIndex

• Vector stores: Pinecone · Qdrant · Weaviate · Milvus — Best Vector Databases.

• Head-to-heads: Qdrant vs Pinecone · Pinecone vs Weaviate · Milvus vs Qdrant

If you understood this topic, read next:

Diagram: Learning path for embedding models

flowchart LR
    A[Embeddings] --> B[Embedding Models]
    B --> C[Chunking]
    C --> D[Vector DB]
    D --> E[Eval]

Prerequisites: Embeddings · RAG

Next topics: Vector Databases · Re-ranking · Retrieval Evaluation

Estimated time: 45 min · Difficulty: Intermediate

Interview Questions

1. Why does embedding model choice matter more than vector DB choice for recall?

   • Expected: model defines semantic space; bad vectors cannot be fixed by faster ANN or better sharding.

2. What happens if you index with BGE and query with text-embedding-3-small?

   • Expected: incomparable vector spaces; meaningless similarity scores; must re-embed entire corpus.

3. When do E5 query/passage prefixes matter and why?

   • Expected: asymmetric training; missing prefixes misalign query and document regions of vector space.

4. How do Matryoshka embeddings help in production?

   • Expected: truncate dimensions for storage/speed without full retrain; validate recall after truncation.

5. API vs self-hosted — decision criteria?

   • Expected: data residency, volume/cost crossover (~10M+ embeddings/month), ops capacity, latency SLAs.

6. Bi-encoder vs cross-encoder — where does each run?

   • Expected: bi-encoder for index + first-stage retrieval; cross-encoder for reranking top-k only.

7. What triggers a full re-index vs incremental upsert?

   • Expected: model version or dimension change → full re-index; new documents → incremental upsert.

8. How do you evaluate a new embedding model before cutover?

   • Expected: same chunks, golden query-doc pairs, recall@k/MRR/nDCG, dual-index A/B, production shadow traffic.

Key Takeaways

• Embedding model choice is one of the highest-leverage decisions in the RAG indexing pipeline.
• Always evaluate on your domain — leaderboard scores are directional, not definitive.
• Use the same model for indexing and querying. Version it in metadata.
• Match chunk size to model input limits. Apply query prefixes per model instructions.
• API models for speed; self-hosted for data residency and cost at scale.
• Plan for full re-indexing when changing models — vectors are not transferable.
• Compare infrastructure in Best Vector Databases after model eval stabilizes.

FAQs

Which embedding model should I use?

For English RAG, common starting points (current as of writing): OpenAI text-embedding-3-small (API) or bge-large-en-v1.5 (self-hosted). Evaluate both on your domain. Use multilingual-e5-large for non-English corpora.

Can I mix embeddings from different models?

No. Vectors from different models occupy different semantic spaces. Mixing them in one index produces meaningless search results.

How do I switch embedding models?

Create a new collection/index with the new model's dimensions. Re-embed all documents. Run both indexes in parallel during migration. Cut over when eval confirms quality.

API or self-hosted?

API for speed to market and low ops. Self-hosted for data residency, high volume (cost savings above ~10M embeddings/month), or offline environments.

Do I need the largest model?

Not necessarily. text-embedding-3-small often matches text-embedding-3-large on domain-specific eval sets. Larger models cost more storage and compute. Measure, do not assume.

What input length should I use?

Match your chunk size to the model's max input tokens. Most models handle 512 tokens; Nomic and Jina handle 8192. Truncation loses information silently.

How do I fine-tune an embedding model?

Use sentence-transformers with contrastive loss on query-document pairs from your domain. Requires 1K–10K labeled pairs. Worth it when general models score below 0.7 recall@5.

What distance metric should I use?

Cosine similarity for most text models. For OpenAI text-embedding-3-small and text-embedding-3-large, outputs are L2-normalized — cosine similarity and dot product produce equivalent rankings. Match your vector database metric configuration to the model.

How often do new models come out?

Major releases every 3–6 months. Re-evaluate quarterly. Do not chase every release — switch when eval shows meaningful improvement on your corpus.

Can one embedding model handle text and images?

Multimodal models (CLIP, Cohere embed-v3, Voyage multimodal) embed both. Useful for documents with diagrams. Text-only models ignore visual content.

What is the E5 query/passage prefix?

E5 models are trained with "query: " before queries and "passage: " before documents. Using these prefixes at inference time significantly improves retrieval quality.

How do embedding costs compare to LLM costs?

Embeddings are 10–100× cheaper than generation. Indexing 1M chunks costs ~$10 with OpenAI small. The same text through GPT-4o costs ~$1,250. Embed freely; generate carefully.

How do I batch embed efficiently?

Most APIs support batching 100–2,000 texts per request. Batch during offline indexing. For self-hosted models, use sentence-transformers with encode(texts, batch_size=64).

What is the MTEB benchmark?

Massive Text Embedding Benchmark evaluates models across 56 datasets. Useful for initial selection — always validate on your domain with retrieval evaluation.

Should I embed the document title separately?

Prepend title and section heading to each chunk before embedding — more effective than metadata alone. The model encodes the title into the vector, improving topic-name queries.

References

• Sentence-BERT (Reimers & Gurevych, 2019)
• Dense Passage Retrieval (Karpukhin et al., 2020)
• OpenAI Embeddings Guide
• Cohere Embed Documentation
• MTEB Leaderboard

Further Reading

• BGE model card (BAAI/bge-large-en-v1.5)
• E5 model card (intfloat/e5-large-v2)
• LlamaIndex Embedding Models Guide