Hallucination Detection - Production Guide

TL;DR

Hallucination is when an LLM generates confident, plausible statements that are factually incorrect or unsupported - not random gibberish, but convincing fabrications.
Detection requires comparing claims against a ground truth source - retrieved context for RAG, knowledge base for Q&A, or external APIs for dynamic facts.
Claim decomposition + NLI entailment is the most reliable automated approach - split the answer into atomic claims, verify each against source text.
Citation validation catches a specific hallucination pattern - the model cites a source but the cited text doesn't support the claim.
Layer defenses - retrieval grounding, faithfulness scoring, citation requirements, output validation, and human review for high-stakes outputs.

A healthcare RAG system tells a patient their medication dosage is 500mg. The retrieved document says 50mg. The LLM added a zero. The answer sounds authoritative, includes a citation, and the patient follows it.

Hallucination is the highest-risk failure mode for production LLM systems. Unlike format errors (caught by schema validation) or refusals (visible to users), hallucinations are silent failures - confident, well-formed, and wrong. Users trust them because the model sounds certain.

Detection is not optional for any system where factual accuracy has consequences: medical, legal, financial, compliance, customer support with policy implications. Even in lower-stakes applications, hallucination erodes user trust faster than "I don't know."

The engineering challenge: hallucinations are structurally identical to correct answers. Same grammar, same confidence, same format. You cannot detect them by examining the output alone - you must verify against external ground truth.

The Problem Hallucination Detection Solves

LLMs are trained to produce fluent, plausible text. They optimize for probability, not truth. When context is insufficient, ambiguous, or missing, the model fills gaps with statistically likely - but potentially false - content.

Without detection:

Wrong answers reach users with full confidence
RAG systems appear grounded while inventing details not in retrieved context
Citations reference real documents but misrepresent their content
Compliance and legal exposure from fabricated policy interpretations
No signal to trigger "I don't know" fallback behavior

Hallucination detection provides an automated verification layer that compares generated claims against authoritative sources before delivery.

What Is Hallucination Detection?

Hallucination detection is the process of identifying unsupported or false claims in LLM-generated text by comparing them against ground truth sources. It operates at three levels:

Level	What It Checks	Method
Claim-level	Is each atomic statement supported?	Claim decomposition + NLI/LLM judge
Citation-level	Does the cited source actually support the claim?	Span extraction + entailment
Answer-level	Is the overall answer faithful to context?	RAGAS faithfulness, LLM-as-judge

Hallucination types in production:

Intrinsic hallucination - Contradicts the provided context. "The refund window is 30 days" when context says 60 days.

Extrinsic hallucination - Adds information not in context. "Premium members also receive free shipping" when context doesn't mention shipping.

Citation hallucination - References a real source but misrepresents its content. Cites doc_042 for a claim doc_042 doesn't contain.

Confabulation - Fills gaps with plausible fabrications. Invents specific numbers, dates, or names when context is vague.

# Hallucination detection pipeline
answer = "Premium members receive refunds within 60 days and free expedited shipping."
context = "Premium members receive full refunds within 60 days of purchase."

claims = decompose_claims(answer)
# ["Premium members receive refunds within 60 days",
#  "Premium members receive free expedited shipping"]

for claim in claims:
    supported = verify_entailment(claim, context)
    if not supported:
        flag_hallucination(claim, severity="extrinsic")

How Hallucination Detection Works

Claim Decomposition

Break the answer into atomic, verifiable statements:

DECOMPOSE_PROMPT = """
Break this answer into atomic factual claims. Each claim should be
a single verifiable statement. Return as JSON list.

Answer: {answer}
Claims:"""

# Input: "Refunds are available within 60 days and require manager approval for amounts over $500."
# Output: [
#   "Refunds are available within 60 days",
#   "Manager approval is required for refund amounts over $500"
# ]

LLM-based decomposition is most flexible. Rule-based splitting (sentences, clauses) works for simpler outputs.

NLI Entailment Verification

Natural Language Inference models classify whether a premise (context) entails a hypothesis (claim):

Premise: "Premium members receive full refunds within 60 days of purchase."
Hypothesis: "Premium members receive refunds within 60 days"
Label: ENTAILMENT ✓

Hypothesis: "Premium members receive free expedited shipping"
Label: CONTRADICTION or NEUTRAL ✗

from transformers import pipeline

nli = pipeline("text-classification", model="cross-encoder/nli-deberta-v3-base")

def verify_claim(claim: str, context: str) -> dict:
    result = nli(f"{context} [SEP] {claim}")[0]
    return {
        "claim": claim,
        "label": result["label"],  # ENTAILMENT, CONTRADICTION, NEUTRAL
        "confidence": result["score"],
        "is_supported": result["label"] == "ENTAILMENT" and result["score"] > 0.7,
    }

NLI models are faster and cheaper than LLM judges but less accurate on complex reasoning. Use NLI for high-volume screening; LLM judge for high-stakes verification.

Citation Validation

When the model provides citations, verify the cited span actually supports the claim:

def validate_citation(claim: str, cited_doc_id: str, cited_span: str, corpus: dict) -> bool:
    full_doc = corpus[cited_doc_id]

    # Verify the cited span actually exists in the document
    if cited_span not in full_doc:
        return False  # Citation hallucination - span doesn't exist

    # Verify the span supports the claim
    result = verify_claim(claim, cited_span)
    return result["is_supported"]

Three failure modes:

Cited document doesn't exist (fabricated reference)
Cited span doesn't exist in the document (fabricated quote)
Cited span exists but doesn't support the claim (misrepresentation)

LLM-as-Judge Faithfulness

For complex claims requiring reasoning:

FAITHFULNESS_PROMPT = """
Given the context and a claim from an answer, determine if the claim
is supported by the context.

Context: {context}
Claim: {claim}

Respond with exactly one of:
- SUPPORTED: The context directly supports this claim
- UNSUPPORTED: The context does not mention or support this claim
- CONTRADICTED: The context directly contradicts this claim
"""

Run each claim through the judge. Flag any UNSUPPORTED or CONTRADICTED claims.

Confidence-Based Detection

Some approaches use model internals for detection:

Token probability - low-confidence tokens may indicate hallucination. Unreliable alone but useful as a signal.
Self-consistency - generate multiple answers; disagreement suggests hallucination.
Retrieval confidence - low retrieval scores + confident answer = likely hallucination.

Faithfulness detection decomposes an LLM answer into atomic claims, then verifies each claim against retrieved source context using NLI or an LLM judge before aggregating a final faithfulness score.

RAGAS faithfulness metric - claim verification against context

Source: RAGAS

Architecture

A production hallucination detection system layers multiple checks:

Layer	Check	Latency	Cost	When
Prevention	RAG grounding, strict prompts	0ms (design-time)	Free	Always
Generation	Require citations, temperature=0	0ms	Free	Always
Post-generation	Claim decomposition + NLI	200–500ms	Low	Every response
Post-generation	LLM-as-judge faithfulness	1–3s	Medium	High-stakes or sampled
Delivery gate	Block if faithfulness < threshold	0ms	Free	Always
Monitoring	Sample and human-review flagged	Async	Medium	Continuous

Step-by-Step Flow

Step 1: Prevent at the source. Use RAG to ground answers. Instruct the model to say "I don't know" when context is insufficient. Set temperature=0 for factual tasks.

Step 2: Require citations. Force the model to cite source spans for every factual claim. Citations create verifiable anchors.

Step 3: Decompose answers into claims. Split generated text into atomic, verifiable statements automatically.

Step 4: Verify each claim. Run NLI entailment or LLM-as-judge against retrieved context. Flag unsupported claims.

Step 5: Validate citations. Confirm cited spans exist in source documents and support the associated claims.

Step 6: Apply delivery gate. Block, revise, or flag answers below faithfulness threshold. Default: refuse rather than deliver unverified claims.

Step 7: Monitor and iterate. Log flagged hallucinations. Add patterns to eval test set. Tune thresholds based on false positive/negative rates.

Real Production Example

Production hallucination detection middleware for a RAG system:

import json
from dataclasses import dataclass, field
from openai import OpenAI

client = OpenAI()

@dataclass
class Claim:
    text: str
    cited_span: str | None = None
    cited_doc_id: str | None = None

@dataclass
class VerificationResult:
    claim: str
    status: str  # SUPPORTED, UNSUPPORTED, CONTRADICTED
    confidence: float
    cited_span_valid: bool | None = None

@dataclass
class HallucinationReport:
    answer: str
    faithfulness_score: float
    claims: list[VerificationResult]
    hallucinated_claims: list[str]
    action: str  # DELIVER, REVISE, REFUSE

class HallucinationDetector:
    def __init__(self, faithfulness_threshold: float = 0.85):
        self.threshold = faithfulness_threshold

    def decompose_claims(self, answer: str) -> list[Claim]:
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""Extract atomic factual claims from this answer.
Return JSON list of objects with "text" and optional "cited_span" fields.

Answer: {answer}"""}],
            temperature=0,
            response_format={"type": "json_object"},
        )
        data = json.loads(resp.choices[0].message.content)
        return [Claim(**c) for c in data.get("claims", data.get("result", []))]

    def verify_claim(self, claim: Claim, context: str) -> VerificationResult:
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""Classify whether this claim is supported by the context.
Reply with exactly one word: SUPPORTED, UNSUPPORTED, or CONTRADICTED.

Context: {context[:3000]}
Claim: {claim.text}"""}],
            temperature=0,
        )
        status = resp.choices[0].message.content.strip().upper()
        confidence = {"SUPPORTED": 0.95, "UNSUPPORTED": 0.85, "CONTRADICTED": 0.90}.get(status, 0.5)

        span_valid = None
        if claim.cited_span:
            span_valid = claim.cited_span in context

        return VerificationResult(
            claim=claim.text,
            status=status,
            confidence=confidence,
            cited_span_valid=span_valid,
        )

    def detect(self, answer: str, contexts: list[str]) -> HallucinationReport:
        context_text = "\n---\n".join(contexts)
        claims = self.decompose_claims(answer)
        verifications = [self.verify_claim(c, context_text) for c in claims]

        supported = sum(1 for v in verifications if v.status == "SUPPORTED")
        faithfulness = supported / len(verifications) if verifications else 1.0

        hallucinated = [
            v.claim for v in verifications
            if v.status in ("UNSUPPORTED", "CONTRADICTED")
            or (v.cited_span_valid is False)
        ]

        if faithfulness >= self.threshold:
            action = "DELIVER"
        elif faithfulness >= 0.60:
            action = "REVISE"
        else:
            action = "REFUSE"

        return HallucinationReport(
            answer=answer,
            faithfulness_score=faithfulness,
            claims=verifications,
            hallucinated_claims=hallucinated,
            action=action,
        )

    def guarded_generate(self, rag_pipeline, query: str) -> dict:
        result = rag_pipeline.query(query)
        report = self.detect(result["answer"], result["contexts"])

        if report.action == "REFUSE":
            return {
                "answer": "I don't have sufficient information to answer this accurately.",
                "hallucination_report": report,
                "original_answer": result["answer"],
            }
        elif report.action == "REVISE":
            revised = self._revise(result["answer"], report.hallucinated_claims, result["contexts"])
            return {"answer": revised, "hallucination_report": report}

        return {"answer": result["answer"], "hallucination_report": report}

    def _revise(self, answer: str, bad_claims: list[str], contexts: list[str]) -> str:
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""Rewrite this answer removing ONLY these unsupported claims.
Keep all supported claims. Use only information from the context.

Context: {chr(10).join(contexts)[:3000]}
Original answer: {answer}
Remove these claims: {bad_claims}"""}],
            temperature=0,
        )
        return resp.choices[0].message.content

# Production usage
detector = HallucinationDetector(faithfulness_threshold=0.85)
response = detector.guarded_generate(production_rag, "What is the refund policy for premium members?")
print(f"Action: {response['hallucination_report'].action}")
print(f"Faithfulness: {response['hallucination_report'].faithfulness_score:.2f}")

Design Decisions

Decision	Option A	Option B	When to choose
Verification method	NLI model	LLM-as-judge	NLI for speed/volume; LLM judge for accuracy on complex claims
On failure	Block and refuse	Revise and deliver	Refuse for high-stakes; revise for user experience in lower-stakes
Faithfulness threshold	0.85	0.95	0.85 general; 0.95 medical/legal/financial
Citation requirement	Required for all claims	Required for key claims	All claims for audit trails; key claims for latency
Detection placement	Post-generation middleware	Inline during generation	Middleware for clean separation; inline for streaming
Sampling vs full	Check every response	Sample 10%	Every response for high-stakes; sample for cost control

⚠ Common Mistakes

Detecting without ground truth. You can't detect hallucinations by examining the output alone. Always compare against retrieved context, knowledge base, or external API.
Trusting citations blindly. Models generate plausible-looking citations that reference real documents with fabricated content. Always validate cited spans.
Single-layer defense. Prompts alone don't prevent hallucination. Combine RAG grounding, faithfulness scoring, citation validation, and delivery gates.
Ignoring partial hallucination. An answer that's 80% correct and 20% fabricated is 100% untrustworthy. Verify every claim, not just the overall answer.
No "I don't know" path. Systems that always generate an answer will always hallucinate sometimes. Explicit refusal is a feature, not a failure.
Threshold too low. Faithfulness 0.70 means 30% of claims may be fabricated. Set thresholds based on domain risk, not convenience.
Not logging hallucinations. Every detected hallucination is a test case for your eval suite. Log, analyze, and add to golden test set.

Where It Breaks Down

No ground truth available - Open-ended creative tasks have no source to verify against. Hallucination detection applies to factual/grounded tasks only.

Implicit knowledge - "Paris is the capital of France" is correct but may not appear in retrieved context. NLI marks it UNSUPPORTED even though it's true. Distinguish corpus-grounded from world-knowledge claims.

Nuanced interpretation - Legal and policy text allows multiple valid interpretations. Binary supported/unsupported misses legitimate inference.

Latency cost - Full claim decomposition + verification adds 1–3 seconds per response. Too slow for real-time chat without sampling or async verification.

False positives - Over-aggressive detection blocks correct answers, frustrating users. Tune thresholds on labeled data; measure false positive rate.

Multi-hop synthesis - Answers requiring inference across multiple context chunks may be flagged as hallucinated even when logically derived. LLM judges handle this better than NLI.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Dimension	Consideration
Scaling	NLI verification: 200ms per claim, parallelizable. LLM judge: 1s per claim. Batch claims for efficiency.
Latency	Full verification adds 500ms–3s. Acceptable for async workflows; use sampling or NLI-only for real-time chat.
Cost	NLI model: free (self-hosted). LLM judge: $0.01–0.05 per answer verified. Budget $100–500/month for full coverage.
Monitoring	Track faithfulness score distribution, hallucination rate, false positive rate. Alert on rate spikes. Log every flagged claim.
Evaluation	Maintain hallucination test set with known unsupported claims. Run detection eval in CI. Target > 90% detection recall.
Security	Detected hallucinations may reveal prompt injection attempts (model generating off-context content). Log for security review.

Important

Prefer refusing over delivering unverified claims. A false "I don't know" costs one interaction; a confident hallucination can cost trust, compliance, or safety.

Ecosystem

RAGAS: Faithfulness metric via automated claim verification. Good starting point for RAG hallucination detection.
DeepEval: Hallucination metric with customizable thresholds. Pytest integration.
LangSmith: Trace-linked faithfulness evaluation. Monitor hallucination rates in production traces.
Braintrust: Custom scoring functions for faithfulness. Regression detection on hallucination rate.
Guardrails AI: Validation framework with hallucination detection validators and structured output enforcement.
TruLens: Groundedness feedback function for RAG hallucination detection.
SelfCheckGPT: Sampling-based hallucination detection using response consistency.

RAG Evaluation: Faithfulness is the primary RAG hallucination metric.
LLM Evaluation: General eval framework including faithfulness scoring.
RAG: Retrieval grounding is the first line of hallucination prevention.
Prompt Engineering: Prompt design to reduce hallucination tendency.
Guardrails: Output validation and safety guardrails including hallucination checks.

Learning Path

Prerequisites: RAG · LLM Evaluation · RAG Evaluation

Next topics: Guardrails · Agentic RAG · Prompt Engineering

Estimated time: 50 min · Difficulty: Intermediate

Hallucination detection requires comparing claims against ground truth - you cannot detect it from output alone. - Claim decomposition + NLI entailment is the core automated pipeline: split, verify, aggregate. - Layer defenses: RAG grounding → citation requirements → faithfulness scoring → delivery gate. - Validate citations - models fabricate plausible references that misrepresent source content.
Prefer refusing over delivering unverified claims, especially in high-stakes domains. - Every detected hallucination becomes a test case - log, analyze, and expand your eval suite.

Tool	Category	Purpose	Website	Best For
LangChain	Framework	Framework for building LLM-powered applications and workflows.	langchain.com	RAG systems
LlamaIndex	RAG	Data framework for connecting LLMs to private and structured data.	llamaindex.ai	RAG over documents

Next Topics

Learning Path

Continue Learning

Production

Retrieval

Related Tools