Re-ranking Guide

TL;DR

• Reranking re-scores retrieved documents using a cross-encoder that jointly evaluates query-document relevance - far more accurate than bi-encoder similarity alone.

• Retrieve many (top-20 to top-50), rerank to few (top-3 to top-5) - reranking improves precision without sacrificing recall.

• Cross-encoders are slow but accurate - they can't pre-compute document scores, so reranking runs at query time on a small candidate set.

• One of the highest-ROI retrieval improvements for many RAG pipelines — reranking frequently improves precision with added latency that is often on the order of ~100–300ms in typical deployments (measure on your stack).

• Example production models (current as of writing): Cohere Rerank and bge-reranker-v2-m3 — API for simplicity, self-hosted for data residency; re-evaluate as new releases ship.

Why This Matters

Initial retrieval (vector search, BM25, hybrid) optimizes for recall - getting the right document somewhere in the top-k results. But k=20 results often contain 15 irrelevant chunks. Passing all 20 to the LLM wastes context window, adds noise, increases cost, and degrades answer quality.

Reranking solves the precision problem. A cross-encoder reads the query and each candidate document together, producing a highly accurate relevance score. It reorders the initial results so the most relevant documents rank first.

Across many deployed RAG systems, adding a reranker is often one of the highest-impact quality improvements — more than switching embedding models, more than tuning chunk size, and more than prompt engineering. The latency cost is often ~100–300ms in typical deployments — acceptable for many applications, but benchmark on your SLA.

The Problem Re-ranking Solves

Bi-encoder embedding models (used in initial retrieval) encode queries and documents independently. The model never sees the query and document together, so it can't capture fine-grained interactions:

• Does this document actually answer this specific question?
• Is the relevant paragraph buried in a long chunk?
• Does the document contradict the query's assumptions?

A query "How do I reset my password?" might retrieve a chunk about "password security best practices" (high embedding similarity) ahead of "password reset procedure" (lower similarity but actually answers the question).

Cross-encoder rerankers process [query, document] as a single input sequence. The transformer's attention mechanism captures precise query-document interactions that bi-encoders miss. This produces dramatically better relevance scores - at the cost of running inference on every query-document pair at query time.

How We Got Here

Reranking predates RAG. Learning-to-rank models reordered web search results for decades. Neural IR added cross-encoders that jointly encode query-document pairs, then deployed RAG pipelines adopted them as a common high-ROI precision stage after vector search and hybrid search.

Diagram: Evolution of reranking in retrieval

timeline
    title Reranking in IR
    2000s : Learning to rank (LambdaMART)
    2019 : MS MARCO cross-encoders
    2020 : Cohere Rerank API
    2022 : bge-reranker open source
    2023 : Rerank in vector DB query APIs
    2024 : Late interaction as LI-rerank stage

Cross-encoder rerankers became the default second stage in deployed RAG stacks once teams realized bi-encoder recall alone could not rank the right chunk first.

Managed APIs (Cohere Rerank) and open models (bge-reranker-v2-m3) made reranking accessible without training custom rankers. Vector databases such as Qdrant and Weaviate now integrate rerankers in query pipelines.

What Is Re-ranking?

Re-ranking is a second-stage retrieval step that takes an initial set of candidate documents and reorders them by true relevance to the query. The reranker model (typically a cross-encoder) scores each query-document pair directly.

def rerank(query: str, documents: list[str], top_n: int = 5) -> list[tuple[str, float]]:
    pairs = [(query, doc) for doc in documents]
    scores = cross_encoder.predict(pairs)
    ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_n]

The pipeline pattern:

Most deployed RAG systems separate offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Initial retrieval casts a wide net (high recall). Reranking selects the best catches (high precision). When multi-query retrieval merges candidates from several transformed queries, deduplicate by chunk ID and rerank against the original user question — the union improves recall but increases noise without a rerank stage.

Diagram: Bi-encoder vs cross-encoder

flowchart LR
    subgraph bi [Bi-encoder retrieval]
        Q1[Query] --> EQ[Query embed]
        D1[Docs] --> ED[Doc embeds precomputed]
        EQ --> ANN[ANN similarity]
        ED --> ANN
    end
    subgraph ce [Cross-encoder rerank]
        Q2[Query + doc text] --> CE[Joint transformer]
        CE --> Score[Relevance score]
    end
    ANN --> Shortlist[Top 20-50]
    Shortlist --> CE

Bi-encoders precompute doc vectors; cross-encoders score pairs at query time on a shortlist only.

How Re-ranking Works

Bi-encoder vs Cross-encoder

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Popular Reranker Models

Diagram: RAG query with reranking

sequenceDiagram
    participant U as User
    participant R as Retriever
    participant RR as Reranker
    participant L as LLM
    U->>R: question
    R->>R: hybrid top-25
    R->>RR: query + 25 chunks
    RR->>RR: cross-encoder scores
    RR->>L: top-5 chunks + question
    L-->>U: answer + citations

Reranking sits between retrieval and generation — typically 100–300ms for 25 document pairs.

Architecture

Reranking sits between retrieval and generation in the query pipeline:

Diagram: Production rerank architecture

flowchart TB
    Q[User query] --> H[Hybrid retriever]
    H --> F[Metadata filters]
    F --> C[Candidates k=25]
    C --> D[Deduplicate by doc]
    D --> RR[Cross-encoder rerank]
    RR --> T[Top-n to LLM]
    T --> G[Generate + cite]
    RR --> Log[Score logging]

Apply tenant and permission filters before reranking; log scores for failure diagnosis.

Key parameters:

• Initial retrieval k: How many candidates to fetch (20–50)

• Rerank top-n: How many to pass to the LLM (3–5)

• Score threshold: Minimum reranker score to include (optional, model-dependent)

Step-by-Step Flow

Step 1: Configure initial retrieval for recall. Retrieve top-20 to top-50 using hybrid search. Cast a wide net - reranking handles precision.

Step 2: Choose a reranker model. Cohere Rerank for API simplicity. bge-reranker-v2-m3 for self-hosted multilingual.

Step 3: Score all candidates. Pass each query-document pair through the cross-encoder. Get a relevance score per document.

Step 4: Select top-n. Take the highest-scoring documents (typically 3–5). Optionally apply a minimum score threshold.

Step 5: Pass to LLM. Send reranked chunks as context. Include reranker scores in logs for debugging.

Step 6: Measure impact. Compare precision@k and answer quality with and without reranking on your eval set.

Real Production Example

A customer support RAG system retrieves 25 candidates, reranks to 5, and generates an answer:

import cohere
from dataclasses import dataclass

@dataclass
class RetrievedChunk:
    id: str
    text: str
    metadata: dict
    initial_score: float

class RerankerPipeline:
    def __init__(self, vector_store, cohere_client=None):
        self.store = vector_store
        self.cohere = cohere_client or cohere.Client()

    def retrieve(self, query: str, top_k: int = 25, filters: dict = None) -> list[RetrievedChunk]:
        results = self.store.hybrid_search(query, top_k=top_k, filters=filters)
        return [
            RetrievedChunk(id=r.id, text=r.text, metadata=r.metadata, initial_score=r.score)
            for r in results
        ]

    def rerank(self, query: str, chunks: list[RetrievedChunk], top_n: int = 5) -> list[RetrievedChunk]:
        if not chunks:
            return []

        response = self.cohere.rerank(
            model="rerank-v3.5",
            query=query,
            documents=[c.text for c in chunks],
            top_n=top_n,
        )

        reranked = []
        for result in response.results:
            chunk = chunks[result.index]
            chunk.initial_score = result.relevance_score
            reranked.append(chunk)

        return reranked

    def query(self, question: str, tenant_id: str) -> dict:
        candidates = self.retrieve(
            question, top_k=25, filters={"tenant_id": tenant_id}
        )
        reranked = self.rerank(question, candidates, top_n=5)

        context = "\n\n".join(
            f"[{i+1}] {c.text}" for i, c in enumerate(reranked)
        )
        answer = llm.generate(question, context)

        return {
            "answer": answer,
            "sources": [c.metadata for c in reranked],
            "reranker_scores": [c.initial_score for c in reranked],
        }

# Self-hosted alternative with sentence-transformers
from sentence_transformers import CrossEncoder

local_reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def rerank_local(query: str, documents: list[str], top_n: int = 5):
    pairs = [[query, doc] for doc in documents]
    scores = local_reranker.predict(pairs)
    ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_n]

The reranker pushes the specific "password reset procedure" document above generic "password security" content that the bi-encoder ranked higher.

Design Decisions

Common patterns

| Pattern | Description | | ------------------------------- | ------------------------------------ | ----------------------------------------- | | Retrieve many, rerank few | k=25 → n=5 | Common deployed default | | Conditional rerank | Skip if bi-encoder top-1 > threshold | Latency-sensitive paths | | Score threshold gate | Drop low rerank scores | Trigger "I don't know" | | Dedupe then rerank | One chunk per doc ID | Avoid wasting cross-encoder on duplicates | | Stack with late interaction | Hybrid → MaxSim → cross-encoder | Maximum precision pipelines |

Comparisons

Cross-encoder vs bi-encoder vs late interaction vs LLM judge

Cohere Rerank vs bge-reranker vs Jina

Decision tree: reranker selection

Decision tree: choosing a reranker

flowchart TD
    A[Need reranking?] -->|Recall broken| B[Fix retrieval first]
    A -->|Yes| C[Data leaves VPC?]
    C -->|No| D[Cohere or Jina API]
    C -->|Yes| E[Self-host bge-reranker]
    E --> F{Long chunks?}
    F -->|Yes| G[bge-reranker-v2-m3 8k]
    F -->|No| H[MiniLM for speed]
    D --> I[Measure precision@3]
    G --> I
    H --> I

Do not add reranking until recall@k is acceptable — reranking cannot retrieve missing documents.

Compare retrieval infrastructure in Best Vector Databases: Qdrant vs Pinecone · Pinecone vs Weaviate.

Common Mistakes

1. Passing too many documents to the reranker. Cross-encoders are slow. Reranking 100 documents adds seconds of latency. Retrieve 20–50, rerank to 3–5.

2. Skipping reranking because retrieval "looks fine." Bi-encoder scores are poorly calibrated. A document at rank 1 with score 0.85 may be less relevant than rank 5 with score 0.78. Measure with precision@k, don't eyeball.

3. Reranking before metadata filtering. Apply tenant and permission filters before reranking to avoid scoring documents the user shouldn't see.

4. Not logging reranker scores. When answers are wrong, reranker scores tell you whether the right document was retrieved but scored low (reranker problem) or wasn't retrieved at all (retrieval problem).

5. Using a reranker with mismatched language. English rerankers on multilingual content produce unreliable scores. Match reranker language to your corpus.

6. Reranking without initial retrieval. Cross-encoders can't search a million documents - they're too slow. Always use bi-encoder retrieval first, then rerank.

Where It Breaks Down

Latency-sensitive applications - Reranking 25 documents adds 100–300ms (API) or 200–500ms (self-hosted CPU). For sub-second requirements, consider smaller rerankers or conditional reranking (only when initial confidence is low).

Very long documents - Cross-encoders have input length limits (512–8192 tokens). If retrieved chunks are 1024 tokens, the reranker may truncate, missing relevant content at the end.

Non-text content - Standard rerankers process text only. Images, tables, and code may need domain-specific rerankers or alternative scoring.

Diminishing returns with perfect retrieval - If your initial retrieval already achieves 95%+ recall@5 with high bi-encoder scores, reranking adds marginal value. Measure before adding complexity.

Cost at extreme scale - Cohere Rerank costs ~$1/1000 searches. At 1M queries/day, that's $30K/month. Self-hosted becomes cost-effective above ~100K queries/day.

When NOT to Use Re-ranking

Skip reranking when:

1. Recall@k is below target — fix chunking, embeddings, or hybrid search first; reranking cannot find missing docs.
2. Sub-200ms total latency is mandatory — cross-encoder adds 100–300ms; use conditional rerank or smaller models.
3. Initial retrieval already achieves precision@3 > 0.95 — measure before adding complexity.
4. Candidates exceed reranker context — 1024-token chunks in a 512-token reranker truncate relevant tail content.
5. You cannot send document text to a third-party API — use self-hosted bge-reranker for sensitive corpora.

Prefer Late Interaction Retrieval when compositional token matching matters more than full cross-attention on a shortlist.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Reranking is a common production pattern for RAG when initial retrieval recall is solid but precision@k is weak. Added latency is often ~100–300ms in typical deployments — measure precision@3 on your golden set and against your SLA before skipping it.

Related Guides

Concept guides

• RAG: The pipeline reranking improves.
• Hybrid Search: Initial retrieval that feeds the reranker.
• Embedding Models: Bi-encoders used before reranking.
• Retrieval Evaluation: Measure precision@3 and nDCG impact.
• Late Interaction Retrieval: Alternative or complementary precision stage.
• Metadata Filtering: Apply filters before reranking.
• ANN Indexes: First-stage recall that reranking depends on.
• Vector Search: Bi-encoder retrieval foundation.

Rankings

• Best Vector Databases

Comparisons

• Qdrant vs Pinecone · Pinecone vs Weaviate

Tools

• Cohere · Hugging Face Transformers · Qdrant · Weaviate · Pinecone

If you understood this topic, read next:

Diagram: Reranking learning path

flowchart LR
    A[Hybrid Search] --> B[Re-ranking]
    B --> C[Retrieval Eval]
    C --> D[Late Interaction]
    D --> E[RAG]

Prerequisites: RAG · Embedding Models · Hybrid Search

Next topics: Retrieval Evaluation · Late Interaction Retrieval · Metadata Filtering

Interview Questions

1. Why can't cross-encoders replace first-stage retrieval?

   • Expected: O(N) inference per query; too slow for millions of documents.

2. What is retrieve-many rerank-few?

   • Expected: bi-encoder top-20–50 for recall; cross-encoder to top-3–5 for precision.

3. Bi-encoder vs cross-encoder tradeoff?

   • Expected: independent encoding + ANN vs joint encoding + accurate scores at query time.

4. When do reranker scores indicate retrieval vs ranking failure?

   • Expected: right doc missing = retrieval; retrieved but low rerank score = ranking/model mismatch.

5. Why filter before reranking?

   • Expected: avoid scoring documents user cannot access; security and wasted compute.

6. Cohere vs self-hosted bge-reranker?

   • Expected: API simplicity vs data residency and volume economics.

7. How do you evaluate reranking impact?

   • Expected: precision@3, nDCG@5, end-to-end answer quality on golden set.

8. Reranking vs late interaction?

   • Expected: cross-encoder full attention on shortlist vs MaxSim token matching; can stack both.

Key Takeaways

• Reranking uses cross-encoders to precisely score query-document relevance after initial retrieval.
• Retrieve many (20–50), rerank to few (3–5) — a standard deployed pattern.
• Often one of the highest-ROI retrieval improvements after fixing recall@k.
• Always apply metadata filters before reranking, not after.
• Log reranker scores to diagnose retrieval vs ranking failures.
• Example production models (current as of writing): Cohere Rerank (API) and bge-reranker-v2-m3 (self-hosted).

FAQs

What is the difference between retrieval and reranking?

Retrieval (bi-encoder) finds candidate documents quickly from a large corpus. Reranking (cross-encoder) accurately scores a small set of candidates. Retrieval optimizes recall; reranking optimizes precision.

How many documents should I rerank?

Retrieve 20–50, rerank to 3–5. The exact numbers depend on your eval metrics - increase initial k if recall is low, decrease final n if the LLM gets confused by too much context.

Is reranking worth the latency?

Often yes — many teams accept ~100–300ms of reranking latency when precision@3 measurably improves on their golden set. Users typically prefer a slightly slower correct answer over a fast wrong one, but validate on your own SLA and eval data.

Cohere Rerank vs bge-reranker?

Cohere: easier integration, no GPU needed, ~$0.001/search. BGE: free, self-hosted, data stays local. Similar quality. Choose based on ops preference and data sensitivity.

Can I use an LLM as a reranker?

Yes (LLM-as-judge), but it's 10–50x slower and more expensive than a dedicated cross-encoder. Use LLM reranking only for high-stakes queries where latency and cost are acceptable.

Should I rerank before or after metadata filtering?

After filtering. Apply tenant/permission filters during initial retrieval, then rerank the filtered candidates. Never rerank documents the user shouldn't access.

What if reranking makes results worse?

Rare, but possible if the reranker model doesn't match your domain. Evaluate on your test set. Try a different reranker model. Check if chunks are too long (truncation).

Can I cache reranker results?

Cache reranked results for identical queries (exact or semantic cache). Reranker scores are deterministic for the same query-document pairs.

Do I need reranking with hybrid search?

Yes. Hybrid search improves recall (finding more relevant docs). Reranking improves precision (ranking them correctly). They solve different problems and complement each other.

How do I evaluate reranking impact?

Compare precision@3 and nDCG on your test set with and without reranking. Measure end-to-end answer correctness. A/B test in production with real user queries.

What is a reranker score threshold?

A minimum relevance score below which documents are excluded. If no documents exceed the threshold, return "I couldn't find relevant information" instead of generating from irrelevant context.

Can reranking replace hybrid search?

No. Reranking reorders candidates; it can't find documents that initial retrieval missed. Fix recall with hybrid search and better embeddings first, then add reranking for precision.

How do I choose between reranker models?

Run both Cohere Rerank and bge-reranker-v2-m3 on your golden test set. Compare precision@3 and nDCG@5. If scores are within 2%, choose based on deployment preference (API vs self-hosted). If one model clearly wins on your domain, use it regardless of deployment convenience.

What hardware do I need for self-hosted reranking?

A cross-encoder reranking 25 document pairs requires ~500MB GPU memory for bge-reranker-v2-m3. Throughput varies widely by hardware — illustrative ranges are ~50–200 rerank pairs/sec on a single GPU vs ~300–500ms per batch on CPU. Benchmark on your deployment.

Should I rerank before or after deduplication?

Deduplicate first. If initial retrieval returns five chunks from the same document, reranking all five wastes compute and crowds out diverse sources. Deduplicate by document ID, keeping the highest bi-encoder score per document, then rerank the deduplicated set.

Does reranking help with multilingual RAG?

Yes, but only if you use a multilingual reranker (for example, Cohere Rerank v3.5 or bge-reranker-v2-m3 at time of writing). English-only rerankers on multilingual content produce unreliable scores. Match the reranker language support to your corpus and query languages.

References

• Cohere Rerank Documentation
• BGE Reranker on Hugging Face
• MS MARCO Cross-Encoders

Further Reading

• Re-ranking in RAG survey
• Qdrant reranking documentation