Re-ranking Guide

TL;DR

• Reranking re-scores retrieved documents using a cross-encoder that jointly evaluates query-document relevance - far more accurate than bi-encoder similarity alone.

• Retrieve many (top-20 to top-50), rerank to few (top-3 to top-5) - reranking improves precision without sacrificing recall.

• Cross-encoders are slow but accurate - they can't pre-compute document scores, so reranking runs at query time on a small candidate set.

• The highest-ROI retrieval improvement for most RAG pipelines - often 15–30% precision gain with 100–300ms added latency.

• Cohere Rerank and bge-reranker-v2-m3 are strong defaults - API for simplicity, self-hosted for data residency.

Why This Matters

Initial retrieval (vector search, BM25, hybrid) optimizes for recall - getting the right document somewhere in the top-k results. But k=20 results often contain 15 irrelevant chunks. Passing all 20 to the LLM wastes context window, adds noise, increases cost, and degrades answer quality.

Reranking solves the precision problem. A cross-encoder reads the query and each candidate document together, producing a highly accurate relevance score. It reorders the initial results so the most relevant documents rank first.

In production RAG systems I've evaluated, adding a reranker is consistently the single highest-impact quality improvement - more than switching embedding models, more than tuning chunk size, and more than prompt engineering. The latency cost (100–300ms) is acceptable for most applications.

The Problem Re-ranking Solves

Bi-encoder embedding models (used in initial retrieval) encode queries and documents independently. The model never sees the query and document together, so it can't capture fine-grained interactions:

• Does this document actually answer this specific question?
• Is the relevant paragraph buried in a long chunk?
• Does the document contradict the query's assumptions?

A query "How do I reset my password?" might retrieve a chunk about "password security best practices" (high embedding similarity) ahead of "password reset procedure" (lower similarity but actually answers the question).

Cross-encoder rerankers process [query, document] as a single input sequence. The transformer's attention mechanism captures precise query-document interactions that bi-encoders miss. This produces dramatically better relevance scores - at the cost of running inference on every query-document pair at query time.

What Is Re-ranking?

Re-ranking is a second-stage retrieval step that takes an initial set of candidate documents and reorders them by true relevance to the query. The reranker model (typically a cross-encoder) scores each query-document pair directly.

def rerank(query: str, documents: list[str], top_n: int = 5) -> list[tuple[str, float]]:
    pairs = [(query, doc) for doc in documents]
    scores = cross_encoder.predict(pairs)
    ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_n]

The pipeline pattern:

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Initial retrieval casts a wide net (high recall). Reranking selects the best catches (high precision).

How Re-ranking Works

Bi-encoder vs Cross-encoder

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Popular Reranker Models

Architecture

Reranking sits between retrieval and generation in the query pipeline:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Key parameters:

• Initial retrieval k: How many candidates to fetch (20–50)

• Rerank top-n: How many to pass to the LLM (3–5)

• Score threshold: Minimum reranker score to include (optional, model-dependent)

Step-by-Step Flow

Step 1: Configure initial retrieval for recall. Retrieve top-20 to top-50 using hybrid search. Cast a wide net - reranking handles precision.

Step 2: Choose a reranker model. Cohere Rerank for API simplicity. bge-reranker-v2-m3 for self-hosted multilingual.

Step 3: Score all candidates. Pass each query-document pair through the cross-encoder. Get a relevance score per document.

Step 4: Select top-n. Take the highest-scoring documents (typically 3–5). Optionally apply a minimum score threshold.

Step 5: Pass to LLM. Send reranked chunks as context. Include reranker scores in logs for debugging.

Step 6: Measure impact. Compare precision@k and answer quality with and without reranking on your eval set.

Real Production Example

A customer support RAG system retrieves 25 candidates, reranks to 5, and generates an answer:

import cohere
from dataclasses import dataclass

@dataclass
class RetrievedChunk:
    id: str
    text: str
    metadata: dict
    initial_score: float

class RerankerPipeline:
    def __init__(self, vector_store, cohere_client=None):
        self.store = vector_store
        self.cohere = cohere_client or cohere.Client()

    def retrieve(self, query: str, top_k: int = 25, filters: dict = None) -> list[RetrievedChunk]:
        results = self.store.hybrid_search(query, top_k=top_k, filters=filters)
        return [
            RetrievedChunk(id=r.id, text=r.text, metadata=r.metadata, initial_score=r.score)
            for r in results
        ]

    def rerank(self, query: str, chunks: list[RetrievedChunk], top_n: int = 5) -> list[RetrievedChunk]:
        if not chunks:
            return []

        response = self.cohere.rerank(
            model="rerank-v3.5",
            query=query,
            documents=[c.text for c in chunks],
            top_n=top_n,
        )

        reranked = []
        for result in response.results:
            chunk = chunks[result.index]
            chunk.initial_score = result.relevance_score
            reranked.append(chunk)

        return reranked

    def query(self, question: str, tenant_id: str) -> dict:
        candidates = self.retrieve(
            question, top_k=25, filters={"tenant_id": tenant_id}
        )
        reranked = self.rerank(question, candidates, top_n=5)

        context = "\n\n".join(
            f"[{i+1}] {c.text}" for i, c in enumerate(reranked)
        )
        answer = llm.generate(question, context)

        return {
            "answer": answer,
            "sources": [c.metadata for c in reranked],
            "reranker_scores": [c.initial_score for c in reranked],
        }

# Self-hosted alternative with sentence-transformers
from sentence_transformers import CrossEncoder

local_reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")

def rerank_local(query: str, documents: list[str], top_n: int = 5):
    pairs = [[query, doc] for doc in documents]
    scores = local_reranker.predict(pairs)
    ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_n]

The reranker pushes the specific "password reset procedure" document above generic "password security" content that the bi-encoder ranked higher.

Design Decisions

⚠ Common Mistakes

1. Passing too many documents to the reranker. Cross-encoders are slow. Reranking 100 documents adds seconds of latency. Retrieve 20–50, rerank to 3–5.

2. Skipping reranking because retrieval "looks fine." Bi-encoder scores are poorly calibrated. A document at rank 1 with score 0.85 may be less relevant than rank 5 with score 0.78. Measure with precision@k, don't eyeball.

3. Reranking before metadata filtering. Apply tenant and permission filters before reranking to avoid scoring documents the user shouldn't see.

4. Not logging reranker scores. When answers are wrong, reranker scores tell you whether the right document was retrieved but scored low (reranker problem) or wasn't retrieved at all (retrieval problem).

5. Using a reranker with mismatched language. English rerankers on multilingual content produce unreliable scores. Match reranker language to your corpus.

6. Reranking without initial retrieval. Cross-encoders can't search a million documents - they're too slow. Always use bi-encoder retrieval first, then rerank.

Where It Breaks Down

Latency-sensitive applications - Reranking 25 documents adds 100–300ms (API) or 200–500ms (self-hosted CPU). For sub-second requirements, consider smaller rerankers or conditional reranking (only when initial confidence is low).

Very long documents - Cross-encoders have input length limits (512–8192 tokens). If retrieved chunks are 1024 tokens, the reranker may truncate, missing relevant content at the end.

Non-text content - Standard rerankers process text only. Images, tables, and code may need domain-specific rerankers or alternative scoring.

Diminishing returns with perfect retrieval - If your initial retrieval already achieves 95%+ recall@5 with high bi-encoder scores, reranking adds marginal value. Measure before adding complexity.

Cost at extreme scale - Cohere Rerank costs ~$1/1000 searches. At 1M queries/day, that's $30K/month. Self-hosted becomes cost-effective above ~100K queries/day.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Reranking is not optional for production RAG. The precision improvement (15–30%) far outweighs the latency cost (100–300ms) in virtually every use case.

Ecosystem

• Cohere Rerank: Best API option. Multilingual, long context, simple integration.

• BGE Reranker: Best open-source option. v2-m3 for multilingual, v2-gemma for quality.

• Jina Reranker: API and self-hosted. Long context (8192 tokens).

• Sentence Transformers: Python library for running cross-encoders locally.

• Pinecone, Weaviate, Qdrant: Some vector DBs integrate reranking natively in query pipelines.

Related Technologies

• RAG: The pipeline reranking improves.

• Hybrid Search: Initial retrieval that feeds the reranker.

• Embedding Models: Bi-encoders used in the retrieval stage before reranking.

• Retrieval Evaluation: Measuring reranking impact with nDCG and precision@k.

• Metadata Filtering: Apply filters before reranking.

Learning Path

Prerequisites: RAG · Embedding Models

Next topics: Retrieval Evaluation · Metadata Filtering · RAG Evaluation

Estimated time: 40 min · Difficulty: Intermediate

FAQs

What is the difference between retrieval and reranking?

Retrieval (bi-encoder) finds candidate documents quickly from a large corpus. Reranking (cross-encoder) accurately scores a small set of candidates. Retrieval optimizes recall; reranking optimizes precision.

How many documents should I rerank?

Retrieve 20–50, rerank to 3–5. The exact numbers depend on your eval metrics - increase initial k if recall is low, decrease final n if the LLM gets confused by too much context.

Is reranking worth the latency?

Yes, in virtually every case. 100–300ms added latency for 15–30% precision improvement. Users prefer a slightly slower correct answer over a fast wrong one.

Cohere Rerank vs bge-reranker?

Cohere: easier integration, no GPU needed, ~$0.001/search. BGE: free, self-hosted, data stays local. Similar quality. Choose based on ops preference and data sensitivity.

Can I use an LLM as a reranker?

Yes (LLM-as-judge), but it's 10–50x slower and more expensive than a dedicated cross-encoder. Use LLM reranking only for high-stakes queries where latency and cost are acceptable.

Should I rerank before or after metadata filtering?

After filtering. Apply tenant/permission filters during initial retrieval, then rerank the filtered candidates. Never rerank documents the user shouldn't access.

What if reranking makes results worse?

Rare, but possible if the reranker model doesn't match your domain. Evaluate on your test set. Try a different reranker model. Check if chunks are too long (truncation).

Can I cache reranker results?

Cache reranked results for identical queries (exact or semantic cache). Reranker scores are deterministic for the same query-document pairs.

Do I need reranking with hybrid search?

Yes. Hybrid search improves recall (finding more relevant docs). Reranking improves precision (ranking them correctly). They solve different problems and complement each other.

How do I evaluate reranking impact?

Compare precision@3 and nDCG on your test set with and without reranking. Measure end-to-end answer correctness. A/B test in production with real user queries.

What is a reranker score threshold?

A minimum relevance score below which documents are excluded. If no documents exceed the threshold, return "I couldn't find relevant information" instead of generating from irrelevant context.

Can reranking replace hybrid search?

No. Reranking reorders candidates; it can't find documents that initial retrieval missed. Fix recall with hybrid search and better embeddings first, then add reranking for precision.

How do I choose between reranker models?

Run both Cohere Rerank and bge-reranker-v2-m3 on your golden test set. Compare precision@3 and nDCG@5. If scores are within 2%, choose based on deployment preference (API vs self-hosted). If one model clearly wins on your domain, use it regardless of deployment convenience.

What hardware do I need for self-hosted reranking?

A cross-encoder reranking 25 document pairs requires ~500MB GPU memory for bge-reranker-v2-m3. A single T4 or A10G GPU handles 50–200 rerank requests per second. CPU inference works for development but adds 300–500ms latency - unacceptable for most production SLAs.

Should I rerank before or after deduplication?

Deduplicate first. If initial retrieval returns five chunks from the same document, reranking all five wastes compute and crowds out diverse sources. Deduplicate by document ID, keeping the highest bi-encoder score per document, then rerank the deduplicated set.

Does reranking help with multilingual RAG?

Yes, but only if you use a multilingual reranker (Cohere Rerank v3.5, bge-reranker-v2-m3). English-only rerankers on multilingual content produce unreliable scores. Match the reranker language support to your corpus and query languages.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Reranking uses cross-encoders to precisely score query-document relevance after initial retrieval.
• Retrieve many (20–50), rerank to few (3–5) - the standard production pattern.
• The highest-ROI retrieval improvement: 15–30% precision gain for 100–300ms latency.
• Always apply metadata filters before reranking, not after.
• Log reranker scores to diagnose whether failures are retrieval or ranking problems.
• Cohere Rerank (API) and bge-reranker-v2-m3 (self-hosted) are strong defaults.