Embeddings Guide

TL;DR

• An embedding is a fixed-size vector of numbers that represents the meaning of text - similar texts produce vectors that are close together in high-dimensional space.

• Embeddings enable semantic comparison - you can find "fixing bugs" when searching for "debugging" because their vectors are nearby, even though the words differ.

• They are the foundation of RAG, semantic search, and recommendations - every retrieval pipeline starts with converting text to embeddings and comparing vectors.

• Embedding models are separate from LLMs - you do not use GPT-4 to generate embeddings; you use dedicated models like OpenAI text-embedding-3-small or open-source alternatives like bge-large.

• Quality depends on the model, domain, and task - always evaluate embeddings on your specific data before committing to a model in production.

Why This Matters

Keyword search fails when users and documents use different words for the same concept. A support engineer searching for "login failure" will miss articles titled "authentication errors" or "sign-in issues." Embeddings solve this by encoding meaning, not surface form.

Every modern AI retrieval system - RAG pipelines, enterprise search, recommendation engines, duplicate detection, clustering - depends on embeddings. If you are building anything that finds, groups, or compares text by meaning, embeddings are your core primitive.

The engineering challenge is not generating embeddings (that is a single API call). It is choosing the right model, managing the index lifecycle, handling domain-specific vocabulary, and measuring retrieval quality before it reaches your LLM.

The Problem Embeddings Solve

Traditional information retrieval matches exact keywords. BM25 and TF-IDF rank documents by term frequency and inverse document frequency - effective for exact matches, brittle for semantic variation.

Embeddings solve the vocabulary mismatch problem: the gap between how users query and how documents are written. They also enable entirely new capabilities:

1. Cross-lingual search - Query in English, find documents in Spanish, because multilingual embedding models map semantically equivalent text to nearby vectors regardless of language.

2. Similarity at scale - Compare a query against millions of documents in milliseconds using approximate nearest neighbor (ANN) algorithms.

3. Clustering and classification - Group documents, detect duplicates, and classify text by comparing embeddings to prototype vectors - no labeled training data required.

4. RAG retrieval - Find the most relevant document chunks to inject into an LLM prompt, grounding generation in real data.

What Is an Embedding?

An embedding is a dense vector - typically 384 to 3,072 floating-point numbers - that represents text in a continuous vector space. The key property: texts with similar meanings map to vectors that are close together, measured by cosine similarity or Euclidean distance.

"How do I fix a memory leak?"     → [0.12, -0.34, 0.56, ..., 0.78]  (1536 dims)
"Debugging memory allocation"     → [0.11, -0.33, 0.55, ..., 0.77]  ← close
"Best pizza recipes in NYC"       → [-0.45, 0.67, -0.12, ..., 0.23] ← far

Embedding models are neural networks (usually transformer encoders) trained with contrastive objectives: pull semantically similar text pairs close together, push dissimilar pairs apart. The sentence-transformers framework popularized this approach.

Embeddings are not LLM outputs. They are produced by dedicated, smaller models optimized for representation quality, not text generation. A 1536-dimensional embedding from text-embedding-3-small captures semantic content in a fixed-size vector - you cannot decode it back to text.

from openai import OpenAI

client = OpenAI()

response = client.embeddings.create(
    model="text-embedding-3-small",
    input="How do I optimize PostgreSQL query performance?",
    dimensions=1536,  # Optional: reduce dimensions for smaller index
)

embedding = response.data[0].embedding
print(f"Dimensions: {len(embedding)}")  # 1536
print(f"First 5 values: {embedding[:5]}")
# Dimensions: 1536
# First 5 values: [0.023, -0.041, 0.018, 0.052, -0.009]

How Embeddings Work

Training Objective

Embedding models learn through contrastive training on pairs of related texts:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Common training approaches:

• Contrastive loss - Minimize distance between similar pairs, maximize distance between dissimilar pairs.

• Multiple Negatives Ranking (MNR) - Use other items in the batch as negatives (efficient at scale).

• Matryoshka Representation Learning - Train embeddings that work at multiple dimensionalities (768, 512, 256) without retraining.

Similarity Search

Once texts are embedded, finding similar items is a nearest neighbor search:

The generator is a seq2seq model that consumes retrieved passages and produces the final answer token by token.

Cosine similarity is the standard metric:

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

query = embed("How to reduce API latency")
doc1 = embed("Optimizing request response times")
doc2 = embed("Best hiking trails in Colorado")

print(f"Relevant doc similarity: {cosine_similarity(query, doc1):.3f}")
print(f"Irrelevant doc similarity: {cosine_similarity(query, doc2):.3f}")
# Relevant doc similarity: 0.847
# Irrelevant doc similarity: 0.312

Architecture

A production embedding pipeline has three components:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Source: Meta AI

Critical rule: use the same embedding model for indexing and querying. Switching models invalidates your entire index - vectors from different models are not comparable.

Step-by-Step Flow

Indexing (offline)

1. Load documents - Ingest from files, databases, APIs, or web crawlers.

2. Chunk - Split into passages sized for your embedding model's token limit (typically 512 tokens). See chunking strategies.

3. Embed - Pass each chunk through the embedding model. Batch for efficiency (OpenAI supports up to 2,048 inputs per request).

4. Store - Write vectors + metadata (source, title, date, access level) to a vector database.

5. Index - Build ANN index (HNSW, IVF) for fast similarity search.

Querying (online)

1. Receive query - User question or search string.

2. Embed query - Same model, same dimensions as indexing.

3. Search - Find top-k nearest vectors using ANN algorithm.

4. Filter - Apply metadata filters (date, department, access level).

5. Rerank (optional) - Re-score with a cross-encoder for higher precision. See re-ranking.

6. Return - Pass retrieved text to the application or LLM.

Real Production Example

Building an embedding index for a company knowledge base with batch processing and similarity search:

import os
import numpy as np
from openai import OpenAI
from dataclasses import dataclass

client = OpenAI()

EMBEDDING_MODEL = "text-embedding-3-small"
EMBEDDING_DIMS = 1536
BATCH_SIZE = 100

@dataclass
class Document:
    id: str
    text: str
    metadata: dict
    embedding: list[float] | None = None

def embed_texts(texts: list[str]) -> list[list[float]]:
    """Batch embed texts with retry logic."""
    all_embeddings = []
    for i in range(0, len(texts), BATCH_SIZE):
        batch = texts[i:i + BATCH_SIZE]
        response = client.embeddings.create(
            model=EMBEDDING_MODEL,
            input=batch,
            dimensions=EMBEDDING_DIMS,
        )
        batch_embeddings = [item.embedding for item in response.data]
        all_embeddings.extend(batch_embeddings)
    return all_embeddings

def build_index(documents: list[Document]) -> list[Document]:
    texts = [doc.text for doc in documents]
    embeddings = embed_texts(texts)
    for doc, emb in zip(documents, embeddings):
        doc.embedding = emb
    return documents

def search(
    query: str,
    index: list[Document],
    top_k: int = 5,
    min_score: float = 0.7,
) -> list[tuple[Document, float]]:
    query_embedding = embed_texts([query])[0]
    query_vec = np.array(query_embedding)

    scored = []
    for doc in index:
        doc_vec = np.array(doc.embedding)
        score = cosine_similarity(query_vec, doc_vec)
        if score >= min_score:
            scored.append((doc, score))

    scored.sort(key=lambda x: x[1], reverse=True)
    return scored[:top_k]

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

For production, replace the in-memory index with a vector database (Pinecone, pgvector, Qdrant) and add metadata filtering, reranking, and monitoring.

Design Decisions

⚠ Common Mistakes

1. Using different models for indexing and querying - Embeddings from text-embedding-3-small and text-embedding-3-large are not comparable. Changing models requires re-embedding the entire corpus.

2. Embedding entire documents without chunking - Embedding models have token limits (512–8,192 tokens). Long documents get truncated, losing most of their content. Always chunk first.

3. Ignoring domain mismatch - General-purpose embeddings underperform on specialized domains (legal, medical, code). Evaluate on your data; consider domain-specific models or fine-tuned embeddings.

4. Not normalizing vectors - Some indexes assume normalized vectors for cosine similarity. If your embedding model does not normalize, compute cosine similarity explicitly or normalize before indexing.

5. Skipping evaluation - Deploying embeddings without measuring retrieval recall on a test set. You will not know if search quality is acceptable until users complain.

6. Embedding queries and documents differently - Some models (E5, BGE) require prefix instructions: "query: " for queries and "passage: " for documents. Missing prefixes degrade retrieval quality significantly.

Tip

Models like multilingual-e5-large and bge-large-en-v1.5 require instruction prefixes. Check the model card before deploying - this is the most common open-source embedding mistake.

Where It Breaks Down

Embeddings are lossy compressions of meaning. Specific failure modes:

• Negation - "The drug is safe" and "The drug is not safe" may have similar embeddings because most tokens overlap. Cross-encoder reranking helps.

• Numbers and codes - Product SKUs, error codes, and serial numbers embed poorly. Semantic search misses exact identifiers; combine with keyword search (hybrid search).

• Recency - Embeddings capture semantic content, not temporal relevance.

"Latest pricing" and "2022 pricing" may embed similarly. Add metadata filters for date-sensitive queries.

• Short queries vs long documents - Query embeddings and document embeddings occupy different regions of vector space. Asymmetric models (E5, GTE) address this with separate query/document encoders.

• Language mismatch - Monolingual embedding models fail cross-lingually.

Use multilingual models (multilingual-e5, Cohere embed-v3) for multi-language corpora.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Ecosystem

• OpenAI - text-embedding-3-small (1536d), text-embedding-3-large (3072d). Industry default for English RAG.

• Cohere - embed-v3 with multilingual support and compression. Strong reranker ecosystem.

• Hugging Face - bge-large-en-v1.5, multilingual-e5-large, nomic-embed-text. Best open-source options.

• Voyage AI - Domain-specific models for code, finance, and legal text.

• Vector databases - Pinecone, Weaviate, Qdrant, pgvector, Milvus, Chroma.

• Frameworks - LangChain, LlamaIndex with built-in embedding integrations.

Related Technologies

• Vector Search - How to find similar vectors at scale using ANN algorithms.

• Semantic Search - Search by meaning, powered by embeddings.

• RAG - The dominant pattern that uses embeddings for document retrieval before LLM generation.

• Tokens - Embedding models have their own token limits and tokenizers.

• Embedding Models - Choosing and evaluating specific embedding models for production.

• Hybrid Search - Combining embeddings with keyword search for better retrieval.

• Large Language Models - LLMs consume retrieved context; embeddings find it.

Learning Path

Prerequisites: Large Language Models · Tokens

Next topics: Vector Search · RAG · Semantic Search

Estimated time: 45 min · Difficulty: Beginner

FAQs

What is the difference between an embedding and an LLM output?

An embedding is a fixed-size vector representing meaning - used for comparison and search. An LLM output is generated text - used for answering questions and completing tasks. Different models, different purposes.

How many dimensions should my embeddings have?

1536 (OpenAI default) is a strong starting point. Reducing to 512 or 256 dimensions shrinks index size and speeds search with minimal quality loss for most applications. Evaluate on your data before reducing.

Can I use ChatGPT to generate embeddings?

No. ChatGPT is a text generation model. Use dedicated embedding models: OpenAI's text-embedding-3-small/large, Cohere's embed-v3, or open-source alternatives. They are cheaper, faster, and produce better vectors for search.

How do I choose an embedding model?

Evaluate on your domain with a retrieval test set. For English RAG: text-embedding-3-small (API) or bge-large-en-v1.5 (self-hosted). For multilingual: multilingual-e5-large or Cohere embed-v3. For code: Voyage voyage-code-2.

Do I need to re-embed when I add new documents?

Only the new documents. Append new vectors to your index. Re-embed the entire corpus only when you change embedding models or significantly change chunking strategy.

How similar is "similar enough"?

Cosine similarity thresholds depend on your model and domain. Scores above 0.8 typically indicate strong relevance; 0.6–0.8 is moderate; below 0.5 is usually noise. Calibrate on your test set - do not use universal thresholds.

Can embeddings capture sentiment?

Partially. "Great product" and "Terrible product" will have different embeddings, but embeddings optimize for semantic content, not sentiment. Use dedicated sentiment models if sentiment is your primary task.

What is the difference between dense and sparse embeddings?

Dense embeddings (neural, fixed-size vectors) capture semantic meaning. Sparse embeddings (BM25, SPLADE) capture keyword relevance with high-dimensional sparse vectors. Hybrid approaches combine both for best retrieval. See hybrid search.

How do I handle embedding model updates?

When your provider releases a new embedding model, evaluate it against your current model on your test set. If quality improves, re-embed the full corpus (plan for downtime or dual-index migration) and switch query embedding to the new model.

Are embeddings deterministic?

API embeddings are deterministic for the same input and model version. Self-hosted models may have slight variation with different hardware or batch sizes. Always use the same model version in production.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Retrieval Documentation
• LlamaIndex RAG Documentation
• Pinecone RAG Guide

Further Reading

• OpenAI Embeddings Guide
• Cohere Rerank Documentation
• RAGAS Evaluation Framework
• Meta AI RAG Paper Repository

Summary

• Embeddings convert text to vectors that capture semantic meaning - enabling search by concept, not keyword. - They are the foundation of RAG, semantic search, clustering, and recommendations. - Use dedicated embedding models, not LLMs, and always use the same model for indexing and querying. - Chunk documents before embedding, evaluate on your domain, and combine with keyword search for production retrieval.

• Embedding cost is low; the engineering challenge is index management, quality evaluation, and integration with downstream systems.