Embeddings Guide

TL;DR

• An embedding is a fixed-size vector of numbers that represents the meaning of text — similar texts produce vectors that are close together in high-dimensional space, measured by cosine similarity or dot product.

• Embeddings enable semantic comparison — you can find "fixing bugs" when searching for "debugging" because their vectors are nearby, even though the words differ.

• They are the foundation of RAG, semantic search, and recommendations — every retrieval pipeline starts with converting text to embeddings and comparing vectors at query time.

• Embedding models are separate from LLMs — you do not use GPT-4 to generate embeddings; you use dedicated models such as OpenAI text-embedding-3-small or open-source alternatives like bge-large-en-v1.5 (examples current as of writing).

• Quality depends on the model, domain, and task — always evaluate embeddings on your specific data before committing to a model in production.

Why This Matters

Keyword search fails when users and documents use different words for the same concept. A support engineer searching for "login failure" will miss articles titled "authentication errors" or "sign-in issues." Embeddings solve this by encoding meaning, not surface form.

Every modern AI retrieval system — RAG pipelines, enterprise search, recommendation engines, duplicate detection, clustering — depends on embeddings. If you are building anything that finds, groups, or compares text by meaning, embeddings are your core primitive.

The engineering challenge is not generating embeddings (that is a single API call). It is choosing the right embedding model, managing the index lifecycle, handling domain-specific vocabulary, and measuring retrieval quality before it reaches your LLM.

The Problem Embeddings Solve

Traditional information retrieval matches exact keywords. BM25 and TF-IDF rank documents by term frequency and inverse document frequency — effective for exact matches, brittle for semantic variation.

Embeddings solve the vocabulary mismatch problem: the gap between how users query and how documents are written. They also enable entirely new capabilities:

1. Cross-lingual search — Query in English, find documents in Spanish, because multilingual embedding models map semantically equivalent text to nearby vectors regardless of language.

2. Similarity at scale — Compare a query against millions of documents in milliseconds using approximate nearest neighbor (ANN) algorithms.

3. Clustering and classification — Group documents, detect duplicates, and classify text by comparing embeddings to prototype vectors — no labeled training data required.

4. RAG retrieval — Find the most relevant document chunks to inject into an LLM prompt, grounding generation in real data.

Without embeddings, vector search and modern RAG do not exist. Hybrid pipelines still use dense vectors alongside sparse keyword signals — see hybrid search.

How We Got Here

Embeddings evolved from word vectors to sentence-level dense retrieval — the stack that powers production RAG today:

Diagram: Evolution of text embeddings

flowchart LR
    A[Word2Vec / GloVe] --> B[Sentence-BERT]
    B --> C[DPR / bi-encoders]
    C --> D[OpenAI ada-002]
    D --> E[Matryoshka v3]
    E --> F[Multilingual + domain models]

Representation learning moved from static word vectors to contrastive sentence encoders, then to API-scale and domain-specific embedding models.

The RAG paper (Lewis et al., 2020) made dense retrieval practical at scale by pairing a bi-encoder retriever with a generator. Today's deployed systems add hybrid search, metadata filtering, re-ranking, and vector quantization — but embeddings remain the semantic layer underneath all of it.

What Is an Embedding?

An embedding is a dense vector — typically 384 to 3,072 floating-point numbers — that represents text in a continuous vector space. The key property: texts with similar meanings map to vectors that are close together, measured by cosine similarity or Euclidean distance.

"How do I fix a memory leak?"     → [0.12, -0.34, 0.56, ..., 0.78]  (1536 dims)
"Debugging memory allocation"     → [0.11, -0.33, 0.55, ..., 0.77]  ← close
"Best pizza recipes in NYC"       → [-0.45, 0.67, -0.12, ..., 0.23] ← far

Embedding models are neural networks (usually transformer encoders) trained with contrastive objectives: pull semantically similar text pairs close together, push dissimilar pairs apart. The sentence-transformers framework popularized this approach for production use.

Embeddings are not LLM outputs. They are produced by dedicated, smaller models optimized for representation quality, not text generation. A 1536-dimensional embedding from text-embedding-3-small captures semantic content in a fixed-size vector — you cannot decode it back to text.

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="How do I optimize PostgreSQL query performance?",
    dimensions=1536,
)

embedding = response.data[0].embedding
print(f"Dimensions: {len(embedding)}")  # 1536

How Embeddings Work

Training objective

Embedding models learn through contrastive training on pairs of related texts:

• Contrastive loss — Minimize distance between similar pairs, maximize distance between dissimilar pairs.

• Multiple Negatives Ranking (MNR) — Use other items in the batch as negatives (efficient at scale).

• Matryoshka Representation Learning — Train embeddings that work at multiple dimensionalities (768, 512, 256) without retraining — supported by OpenAI v3 and Nomic models.

Source: Sentence Transformers Documentation

Similarity search

Once texts are embedded, finding similar items is a nearest neighbor search. Cosine similarity is the standard metric when vectors are normalized:

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

query = embed("How to reduce API latency")
doc1 = embed("Optimizing request response times")
doc2 = embed("Best hiking trails in Colorado")

print(f"Relevant doc similarity: {cosine_similarity(query, doc1):.3f}")
print(f"Irrelevant doc similarity: {cosine_similarity(query, doc2):.3f}")
# Relevant doc similarity: 0.847
# Irrelevant doc similarity: 0.312

At scale, brute-force comparison is replaced by ANN indexes (HNSW, IVF) inside vector databases. Compare engines in Best Vector Databases.

Diagram: Query embedding and retrieval sequence

sequenceDiagram
    participant U as User
    participant API as Search API
    participant E as Embedder
    participant V as Vector DB
    participant R as Reranker
    U->>API: query text
    API->>E: embed query
    E-->>API: query vector
    API->>V: ANN search + filters
    V-->>API: top-k chunks
    API->>R: optional rescore
    R-->>API: ranked passages
    API-->>U: results / RAG context

The query path embeds once, searches the index, optionally reranks, then returns passages to the application or LLM.

Architecture

An operational embedding pipeline has four layers:

Diagram: Production embedding architecture

flowchart TB
    subgraph offline [Offline indexing]
        L[Loaders] --> C[Chunker]
        C --> EM[Embedding model]
        EM --> VS[Vector store]
        VS --> ANN[ANN index]
    end
    subgraph online [Online query]
        Q[Query] --> EQ[Embed query]
        EQ --> SR[Similarity search]
        SR --> MF[Metadata filter]
        MF --> RR[Reranker]
    end
    ANN --> SR

Offline indexing embeds chunks once; online query embeds the question and searches the same vector space.

Source: Meta AI — RAG paper (arXiv:2005.11401)

Critical rule: use the same embedding model for indexing and querying. Switching models invalidates your entire index — vectors from different models occupy different semantic spaces and are not comparable.

Diagram: Embedding index lifecycle

stateDiagram-v2
    [*] --> Ingest
    Ingest --> Chunk
    Chunk --> Embed
    Embed --> Index
    Index --> Ready
    Ready --> Query: online
    Query --> Ready
    Ready --> Reembed: model change
    Reembed --> Embed

Model version changes force a full re-embed; document updates can be incremental upserts.

Step-by-Step Flow

Indexing (offline)

1. Load documents — Ingest from files, databases, APIs, or web crawlers. Preserve structure (headings, page numbers).

2. Chunk — Split into passages sized for your embedding model's token limit (typically 512 tokens). See chunking strategies.

3. Embed — Pass each chunk through the embedding model. Batch for efficiency (OpenAI supports up to 2,048 inputs per request).

4. Store — Write vectors + metadata (source, title, date, tenant ID, access level) to a vector database.

5. Index — Build ANN index (HNSW, IVF) for fast similarity search.

Querying (online)

1. Receive query — User question or search string.

2. Embed query — Same model, same dimensions as indexing. Apply model-specific prefixes if required (E5: "query: ").

3. Search — Find top-k nearest vectors using ANN algorithm, optionally fused with BM25 in hybrid search.

4. Filter — Apply metadata filters (date, department, tenant, access level).

5. Rerank (optional) — Re-score with a cross-encoder for higher precision. See re-ranking.

6. Return — Pass retrieved text to the application or LLM in a RAG pipeline.

Real Production Example

Building an embedding index for a company knowledge base with batch processing, tenant isolation, and similarity search:

import os
import time
import numpy as np
from openai import OpenAI
from dataclasses import dataclass, field

client = OpenAI(timeout=30.0)

EMBEDDING_MODEL = "text-embedding-3-small"
EMBEDDING_DIMS = 1536
BATCH_SIZE = 100
MAX_RETRIES = 3

@dataclass
class Document:
    id: str
    text: str
    metadata: dict = field(default_factory=dict)
    embedding: list[float] | None = None

def embed_texts(texts: list[str]) -> list[list[float]]:
    """Batch embed with exponential backoff."""
    all_embeddings = []
    for i in range(0, len(texts), BATCH_SIZE):
        batch = texts[i:i + BATCH_SIZE]
        for attempt in range(MAX_RETRIES):
            try:
                response = client.embeddings.create(
                    model=EMBEDDING_MODEL,
                    input=batch,
                    dimensions=EMBEDDING_DIMS,
                )
                all_embeddings.extend([item.embedding for item in response.data])
                break
            except Exception:
                if attempt == MAX_RETRIES - 1:
                    raise
                time.sleep(2 ** attempt)
    return all_embeddings

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def build_index(documents: list[Document]) -> list[Document]:
    texts = [doc.text for doc in documents]
    embeddings = embed_texts(texts)
    for doc, emb in zip(documents, embeddings):
        doc.embedding = emb
        doc.metadata["embedding_model"] = EMBEDDING_MODEL
        doc.metadata["embedding_dims"] = EMBEDDING_DIMS
    return documents

def search(
    query: str,
    index: list[Document],
    tenant_id: str,
    top_k: int = 5,
    min_score: float = 0.65,
) -> list[tuple[Document, float]]:
    query_embedding = embed_texts([query])[0]
    query_vec = np.array(query_embedding)

    scored = []
    for doc in index:
        if doc.metadata.get("tenant_id") != tenant_id:
            continue
        doc_vec = np.array(doc.embedding)
        score = cosine_similarity(query_vec, doc_vec)
        if score >= min_score:
            scored.append((doc, score))

    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

For production, replace the in-memory index with a vector database (Qdrant, Weaviate, Milvus), add hybrid search, reranking, and log every query for retrieval evaluation.

Design Decisions

Common patterns

• Bi-encoder retrieval — Pre-compute document embeddings; embed query at search time. Fast, scales to billions of vectors.

• Asymmetric encoding — Separate query/passage prefixes (E5, BGE) to align short queries with long documents.

• Dual-write migration — Run old and new embedding indexes in parallel when changing models; cut over after eval confirms quality.

• Embedding cache — Cache frequent query embeddings in Redis to shave 20–80ms off hot paths.

Comparisons

Dense embeddings vs sparse (BM25)

Embeddings vs LLM hidden states

Embeddings vs fine-tuning for knowledge

Decision tree: do you need custom embeddings?

Decision tree: Embedding approach

flowchart TD
    A[Need semantic search or RAG?] -->|No| B[Keyword / structured query]
    A -->|Yes| C[Sensitive data leaves network?]
    C -->|Yes| D[Self-host BGE / E5]
    C -->|No| E[Evaluate API vs self-host]
    E --> F{Recall@5 below 0.7?}
    F -->|Yes| G[Domain model or fine-tune]
    F -->|No| H[Ship API default]
    D --> I[Add hybrid if exact IDs matter]
    H --> I
    G --> I
    I --> J[Measure weekly on golden set]

Start with a strong general model; escalate to domain-specific or fine-tuned embeddings only when eval shows a gap.

Compare vector stores in Best Vector Databases: Qdrant vs Pinecone · Chroma vs Pinecone · LanceDB vs Chroma.

Common Mistakes

1. Using different models for indexing and querying — Embeddings from text-embedding-3-small and text-embedding-3-large are not comparable. Changing models requires re-embedding the entire corpus.

2. Embedding entire documents without chunking — Embedding models have token limits (512–8,192 tokens). Long documents get truncated, losing most content. Always chunk first.

3. Ignoring domain mismatch — General-purpose embeddings underperform on specialized domains (legal, medical, code). Evaluate on your data; consider domain-specific models.

4. Not normalizing vectors — Some indexes assume normalized vectors for cosine similarity. If your model does not normalize, compute cosine explicitly or normalize before indexing.

5. Skipping evaluation — Deploying embeddings without measuring retrieval recall on a test set. You will not know if search quality is acceptable until users complain.

6. Embedding queries and documents differently — Models like E5 and BGE require prefix instructions: "query: " for queries and "passage: " for documents. Missing prefixes degrade retrieval significantly.

Tip

Models like multilingual-e5-large and bge-large-en-v1.5 require instruction prefixes. Check the model card before deploying — this is the most common open-source embedding mistake.

Where It Breaks Down

Embeddings are lossy compressions of meaning. Specific failure modes:

• Negation — "The drug is safe" and "The drug is not safe" may have similar embeddings because most tokens overlap. Cross-encoder reranking helps.

• Numbers and codes — Product SKUs, error codes, and serial numbers embed poorly. Semantic search misses exact identifiers; combine with keyword search via hybrid search.

• Recency — Embeddings capture semantic content, not temporal relevance. "Latest pricing" and "2022 pricing" may embed similarly. Add metadata filters for date-sensitive queries.

• Short queries vs long documents — Query and document embeddings occupy different regions of vector space. Asymmetric models (E5, GTE) address this with separate encoders or prefixes.

• Language mismatch — Monolingual models fail cross-lingually. Use multilingual models (multilingual-e5, Cohere embed-v3) for multi-language corpora.

• Embedding inversion — Vectors can be approximately inverted to recover source text. Do not treat embeddings as encryption; enforce access control at the metadata filter level.

When NOT to Use Embeddings

Skip dense embeddings as your primary retrieval mechanism when:

1. Exact token match is the requirement — Invoice numbers, UUIDs, legal citations. Use BM25 or structured search; optionally add embeddings for paraphrase coverage via hybrid search.

2. Corpus is tiny and static — Under ~100 documents, full-text search or long-context prompting may be simpler with equivalent quality.

3. Multi-hop relationship queries dominate — "Which suppliers of our Tier-1 vendor had violations?" needs GraphRAG or graph traversal, not cosine similarity alone.

4. Sub-10ms search latency is mandatory — ANN search plus embedding API latency typically exceeds 30ms. Precomputed keyword indexes or caches may be required.

5. You cannot store vectors securely — If embedding sensitive text to a third-party API violates policy and self-hosting is not feasible, do not embed; use on-prem lexical search or structured retrieval.

Prefer late interaction retrieval (ColBERT-style) when bi-encoder recall is insufficient and you can afford higher storage and latency.

Running in Production

Best Practice

✅ Best Practices — Instrument every stage, version embedding models in metadata, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Store embedding_model, embedding_dims, and indexed_at in every vector record. You need this for migrations, debugging, and compliance audits.

Related Guides

• Companies: OpenAI · Google · Hugging Face — embedding APIs and open model hubs.

• Models: GPT-5 · Gemini 3.1 Pro · Llama 4 — generators commonly paired with embedding retrieval in RAG.

• Foundations: Embedding Models · Tokens · Large Language Models

• Retrieval stack: RAG · Vector Search · Hybrid Search · Metadata Filtering · ANN Indexes · Vector Quantization · Re-ranking

• Indexing: Chunking Strategies · Vector Databases · Retrieval Evaluation

• Advanced: Late Interaction Retrieval · GraphRAG · Semantic Search

• Vector stores: Qdrant · Weaviate · Pinecone · Milvus · pgvector · Chroma — compare in Best Vector Databases.

• Head-to-heads: Qdrant vs Pinecone · Chroma vs Pinecone · LanceDB vs Chroma

If you understood this topic, read next:

Diagram: Learning path for embeddings

flowchart LR
    A[Tokens] --> B[Embeddings]
    B --> C[Embedding Models]
    C --> D[Vector Search]
    D --> E[RAG]

Prerequisites: Large Language Models · Tokens

Next topics: Embedding Models · Vector Search · RAG

Estimated time: 45 min · Difficulty: Beginner

Interview Questions

1. What is an embedding and how does it differ from an LLM output?

   • Expected: fixed-size dense vector for similarity search vs generated text tokens; different models and training objectives.

2. Why must indexing and querying use the same embedding model?

   • Expected: different models occupy incomparable vector spaces; mixed indexes produce meaningless similarity scores.

3. When would you add hybrid search over dense-only retrieval?

   • Expected: SKUs, error codes, rare tokens, exact identifiers — BM25 + dense fusion (Hybrid Search).

4. What causes "similar" embeddings for opposite meanings?

   • Expected: negation overlap, lossy compression; mitigated by reranking and hybrid search.

5. How do you migrate to a new embedding model in production?

   • Expected: dual-index migration, full re-embed, eval on golden set before cutover, version metadata.

6. What metadata should travel with every vector?

   • Expected: tenant_id, ACL, source URL, chunk_id, embedding_model version, indexed_at.

7. Dense vs sparse retrieval — trade-offs?

   • Expected: semantic paraphrase vs exact token match; production uses both via hybrid fusion.

8. How do you monitor embedding retrieval quality in production?

   • Expected: recall@k on golden set, score distribution drift, latency per stage, user feedback on bad answers.

Key Takeaways

• Embeddings convert text to vectors that capture semantic meaning — enabling search by concept, not keyword.
• They are the foundation of RAG, vector search, clustering, and recommendations.
• Use dedicated embedding models, not LLMs, and always use the same model for indexing and querying.
• Chunk documents before embedding, evaluate on your domain, and combine with keyword search for production retrieval.
• Embedding cost is low; the engineering challenge is index management, quality evaluation, and integration with downstream systems.
• Compare vector stores in Best Vector Databases before committing to infrastructure.

FAQs

What is the difference between an embedding and an LLM output?

An embedding is a fixed-size vector representing meaning — used for comparison and search. An LLM output is generated text — used for answering questions and completing tasks. Different models, different purposes.

How many dimensions should my embeddings have?

1536 (OpenAI default) is a strong starting point. Reducing to 512 or 256 dimensions shrinks index size and speeds search with minimal quality loss for most applications. Evaluate on your data before reducing.

Can I use ChatGPT to generate embeddings?

No. ChatGPT is a text generation model. Use dedicated embedding models: OpenAI's text-embedding-3-small/large, Cohere's embed-v3, or open-source alternatives. They are cheaper, faster, and produce better vectors for search.

How do I choose an embedding model?

Evaluate on your domain with a retrieval test set. For English RAG, common starting points (current as of writing) include text-embedding-3-small (API) or bge-large-en-v1.5 (self-hosted). For multilingual: multilingual-e5-large or Cohere embed-v3. See Embedding Models.

Do I need to re-embed when I add new documents?

Only the new documents. Append new vectors to your index. Re-embed the entire corpus only when you change embedding models or significantly change chunking strategy.

How similar is "similar enough"?

Cosine similarity thresholds depend on your model and domain. Scores above 0.8 typically indicate strong relevance; 0.6–0.8 is moderate; below 0.5 is usually noise. Calibrate on your test set — do not use universal thresholds.

Can embeddings capture sentiment?

Partially. "Great product" and "Terrible product" will have different embeddings, but embeddings optimize for semantic content, not sentiment. Use dedicated sentiment models if sentiment is your primary task.

What is the difference between dense and sparse embeddings?

Dense embeddings (neural, fixed-size vectors) capture semantic meaning. Sparse embeddings (BM25, SPLADE) capture keyword relevance with high-dimensional sparse vectors. Hybrid approaches combine both — see hybrid search.

How do I handle embedding model updates?

When your provider releases a new model, evaluate it against your current model on your test set. If quality improves, re-embed the full corpus (plan dual-index migration) and switch query embedding to the new model.

Are embeddings deterministic?

API embeddings are deterministic for the same input and model version. Self-hosted models may have slight variation with different hardware or batch sizes. Pin model versions in production.

Should I normalize vectors before indexing?

If your vector database uses dot product on L2-normalized vectors, cosine and dot product rank identically. OpenAI text-embedding-3-small and text-embedding-3-large outputs are normalized; verify normalization for self-hosted models. Mismatch between metric and normalization causes silent ranking bugs.

How do embeddings relate to vector quantization?

Vector quantization compresses float vectors to int8 or binary codes for smaller indexes and faster search. Quantization adds recall loss — validate after compression on your eval set.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• Dense Passage Retrieval (Karpukhin et al., 2020)
• Sentence-BERT (Reimers & Gurevych, 2019)
• OpenAI Embeddings Guide
• LangChain Embeddings Documentation

Further Reading

• MTEB Leaderboard
• LlamaIndex Embeddings Guide
• Pinecone — What are embeddings?