Caching Guide

TL;DR

• LLM caching stores expensive compute results - completed responses, embeddings, and static prompt prefixes - so identical work is not repeated.

• Exact-match caching keys on normalized input - same question + tenant + model + prompt version → cache hit; paraphrases miss unless you use semantic caching.

• Provider prompt caching discounts repeated input prefixes - structure prompts with static system content first, dynamic user content last.

• TTL and invalidation are product decisions - FAQ cache 24h; account-specific answers 60s or no cache; never cache without tenant isolation.

• Caching wrong answers is worse than no cache - version cache keys with prompt_version and index_version; invalidate on deploy.

Why This Matters

Without caching, every "What are your business hours?" costs a full embed + retrieve + generate cycle. At 10K daily repeats, that is wasted thousands of dollars and added seconds of latency for deterministic answers.

Caching is the simplest high-ROI optimization after basic model routing. It appears in cost optimization and latency optimization playbooks because hits are near-free and near-instant.

Done wrong, caches leak data across tenants, serve stale policies after doc updates, or freeze incorrect answers indefinitely. Production caching requires explicit key design, TTL policy, and invalidation hooks.

The Problem Caching Solves

LLM workloads repeat:

• Same FAQ questions phrased identically (or nearly - see semantic cache).
• Same documents embedded on every re-index job.
• Same system prompt prefix on every request.
• Same retrieval query within a session.

Recomputing these burns tokens, GPU/API quota, and milliseconds on the critical path. Caching trades memory and invalidation complexity for repeat lookup - O(1) Redis GET vs O(seconds) LLM.

What Is LLM Caching?

LLM caching is storing and reusing prior computation results in LLM applications to reduce latency, cost, and load.

Cache types:

This article focuses on exact-match and embedding/prompt caching. Semantic similarity caching is covered in Semantic Caching.

How Caching Works

Cache Key Design

A production cache key includes every input that affects output:

def response_cache_key(
    tenant_id: str,
    query: str,
    model: str,
    prompt_version: str,
    index_version: str,
) -> str:
    normalized = " ".join(query.lower().split())
    payload = f"{tenant_id}|{model}|{prompt_version}|{index_version}|{normalized}"
    return "llm:resp:" + hashlib.sha256(payload.encode()).hexdigest()

Missing tenant_id → cross-tenant data leak. Missing prompt_version → stale behavior after deploy. Missing index_version → wrong answers after re-index.

Lookup Flow

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Provider Prompt Caching

OpenAI and Anthropic cache identical input token prefixes:

messages = [
    {"role": "system", "content": STATIC_SYSTEM_PROMPT},  # cacheable
    {"role": "user", "content": f"Context:\n{dynamic_rag}\n\nQ: {query}"},  # not cached across queries
]

Requirements: prefix must be byte-identical across requests. Dynamic timestamps or user IDs in system prompt break cache.

Billing: cached input tokens at reduced rate (provider-specific, often 50–90% off).

Embedding Cache

def embed_with_cache(text: str, embed_fn) -> list[float]:
    key = "emb:" + hashlib.sha256(text.encode()).hexdigest()
    cached = redis.get(key)
    if cached:
        return json.loads(cached)
    vec = embed_fn(text)
    redis.setex(key, 86400 * 7, json.dumps(vec))
    return vec

Critical for re-indexing and duplicate chunks. Use content hash, not chunk ID - text may move.

Retrieval Result Cache

Cache chunk IDs returned for a query - cheaper than full response cache when generation varies (e.g., personalized tone) but retrieval is stable:

retrieval_key = f"ret:{tenant_id}:{index_version}:{query_hash}"
chunk_ids = redis.get(retrieval_key)
if chunk_ids:
    chunks = store.get_by_ids(json.loads(chunk_ids))
else:
    chunks = await store.search(...)
    redis.setex(retrieval_key, 600, json.dumps([c.id for c in chunks]))

TTL shorter than response cache (5–15 min) - documents change more often than FAQ answers.

Cache Stampede Protection

When a hot key expires, thousands of concurrent requests miss and hammer the LLM simultaneously:

async def get_with_singleflight(key, factory, redis):
    cached = redis.get(key)
    if cached:
        return json.loads(cached)
    lock_key = f"lock:{key}"
    if redis.set(lock_key, "1", nx=True, ex=30):
        try:
            value = await factory()
            redis.setex(key, 3600, json.dumps(value))
            return value
        finally:
            redis.delete(lock_key)
    else:
        await asyncio.sleep(0.1)
        return await get_with_singleflight(key, factory, redis)

Use probabilistic early expiration (jitter TTL ±10%) to spread expirations. For high-traffic keys, background refresh before TTL expires.

Multi-Layer Cache Stack

Production systems stack caches - each layer catches what the previous misses:

L0: In-process LRU (embeddings, 1000 entries)     ~0.01ms
L1: Redis exact response cache                      ~1ms
L2: Redis retrieval result cache                    ~1ms
L3: Provider prompt cache (automatic)               ~0ms billing discount
L4: Semantic cache (see semantic-caching.md)        ~50–100ms lookup
L5: Full RAG + LLM pipeline                         ~1–3s

Check layers in order; stop on first hit. Instrument cache_layer tag on every response for tuning.

Architecture

Place cache lookup first in orchestrator - before embed and LLM.

Step-by-Step Flow

Step 1: Classify endpoints - cacheable (public FAQ) vs never-cache (personalized account actions).

Step 2: Define key schema - tenant, model, prompt_version, index_version, normalized query hash.

Step 3: Implement lookup - Redis GET; on hit, validate TTL and version metadata in value.

Step 4: On miss, run pipeline - store response with TTL and metadata {created_at, prompt_version, sources}.

Step 5: Enable provider prompt caching - refactor prompt layout; monitor cached_tokens in API response.

Step 6: Add embedding cache - for indexing workers and repeated query embeds.

Step 7: Wire invalidation - on index promotion, bump index_version globally or flush tenant keys.

Step 8: Monitor hit rate - target 20–40% for FAQ-heavy apps on exact cache; tune TTL.

Real Production Example

Redis response cache with versioning and tenant isolation:

import hashlib
import json
import time
from dataclasses import dataclass
from typing import Optional

@dataclass
class CacheConfig:
    ttl_seconds: int = 3600
    prompt_version: str = "v3.2"
    index_version: str = "2026-07-01"

class LLMResponseCache:
    def __init__(self, redis_client, config: CacheConfig):
        self.redis = redis_client
        self.config = config

    def _key(self, tenant_id: str, query: str, model: str) -> str:
        norm = " ".join(query.lower().split())
        raw = f"{tenant_id}|{model}|{self.config.prompt_version}|{self.config.index_version}|{norm}"
        return "llm:cache:" + hashlib.sha256(raw.encode()).hexdigest()

    def get(self, tenant_id: str, query: str, model: str) -> Optional[dict]:
        key = self._key(tenant_id, query, model)
        raw = self.redis.get(key)
        if not raw:
            metrics.increment("cache_miss")
            return None
        entry = json.loads(raw)
        if entry.get("prompt_version") != self.config.prompt_version:
            metrics.increment("cache_stale_version")
            return None
        metrics.increment("cache_hit")
        return entry

    def set(self, tenant_id: str, query: str, model: str, response: str, sources: list):
        key = self._key(tenant_id, query, model)
        entry = {
            "response": response,
            "sources": sources,
            "prompt_version": self.config.prompt_version,
            "index_version": self.config.index_version,
            "cached_at": time.time(),
        }
        self.redis.setex(key, self.config.ttl_seconds, json.dumps(entry))

    def invalidate_tenant(self, tenant_id: str):
        # Production: use Redis SCAN + delete pattern or per-tenant version bump
        self.redis.incr(f"llm:cache:ver:{tenant_id}")

class CachedRAGService:
    def __init__(self, cache: LLMResponseCache, pipeline):
        self.cache = cache
        self.pipeline = pipeline

    async def query(self, tenant_id: str, query: str, model: str = "gpt-4o-mini"):
        hit = self.cache.get(tenant_id, query, model)
        if hit:
            return {"answer": hit["response"], "sources": hit["sources"], "from_cache": True}

        result = await self.pipeline.run(tenant_id, query, model)
        if result.get("cache_eligible", True):
            self.cache.set(tenant_id, query, model, result["answer"], result["sources"])
        return {**result, "from_cache": False}

Mark cache_eligible=False for responses containing user-specific data or low confidence.

Design Decisions

⚠ Common Mistakes

1. Cache key without tenant_id. Critical security bug.

2. Infinite TTL on policy content. Refund rules change; users get wrong answer for weeks.

3. Dynamic content in system prompt. Breaks provider prompt cache every request.

4. Caching non-deterministic outputs (temperature > 0) without noting variance - users see frozen random answer.

5. No cache bypass for support. Engineers cannot reproduce bugs - add admin no-cache mode.

6. Caching before guardrails. Store only post-validation output.

7. Ignoring memory limits. Redis evicts hot keys - monitor memory and hit rate drops.

Where It Breaks Down

Personalized answers - "What's my order status?" cannot share cache across users. Key must include user-scoped entity IDs; often not worth caching.

Time-sensitive data - stock prices, SLA status. Short TTL or no cache.

Exact cache miss on paraphrases - "hours?" vs "when are you open?" - need semantic layer.

Stale after silent index update - bump index_version on every promotion; automate in CI/CD.

Compliance - regulated industries may prohibit storing LLM outputs - check data retention policy.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Include prompt_version and index_version in every response cache key. Deploy without version bump serves stale wrong answers at lightning speed.

Ecosystem

• Redis / KeyDB / Dragonfly: Primary response and embedding cache stores.

• OpenAI / Anthropic prompt caching: Provider-native prefix caching.

• GPTCache: Semantic + exact caching library (see Semantic Caching).

• Cloudflare AI Gateway: Edge caching for API responses.

• LangChain Cache: Pluggable cache backends for LLM calls.

Related Technologies

• Semantic Caching: Similarity-based hits for paraphrased queries.

• Cost Optimization: Caching is top-tier cost lever.

• Latency Optimization: Cache hits are fastest path.

• Tokens: Prompt caching reduces billed input tokens.

• Embeddings: Embedding cache speeds indexing and retrieval.

Learning Path

Prerequisites: Cost Optimization · Tokens

Next topics: Semantic Caching · Latency Optimization · Cost Optimization

Estimated time: 45 min · Difficulty: Intermediate

FAQs

Should I cache LLM responses?

Yes for deterministic, non-personalized queries with proper TTL and tenant-scoped keys. No for user-specific or time-critical data without short TTL.

What belongs in a cache key?

Tenant ID, normalized query, model ID, prompt version, index/knowledge base version, and any parameter affecting output (temperature if >0).

What is prompt caching vs response caching?

Prompt caching is provider-side discount on repeated input prefixes. Response caching is your infrastructure storing complete outputs - different layers, both valuable.

What TTL should I use?

FAQ/public docs: 1–24 hours. Semi-dynamic: 5–15 minutes. User-specific: avoid or <60 seconds. Policy/legal content: invalidate on doc change, not TTL alone.

How do I invalidate cache on deploy?

Bump prompt_version in key builder on prompt deploy. Bump index_version on vector index promotion. Avoid FLUSHALL in production.

Can I cache at temperature 0?

Yes - output is deterministic for same model/version. Document that model provider updates may change outputs - version model in key.

Does caching help latency?

Yes - Redis hit is ~1ms vs seconds for LLM. Largest latency win after streaming for repeated queries.

What is embedding cache?

Store text hash → vector to avoid re-embedding identical strings during indexing and repeated queries.

What is negative caching?

Caching "no results" or "I don't know" to prevent repeated expensive pipelines for unanswerable spam queries - use short TTL.

How do I measure cache effectiveness?

Hit rate = hits / (hits + misses). Track cost avoided = hits × avg_cost_per_miss. Alert if hit rate drops suddenly (key bug).

When should I not cache?

Medical/legal personalized advice, live account data, responses failing guardrails, experimental prompt versions under A/B test.

How does caching interact with RAG?

Include index_version in key so new documents invalidate stale answers. Optionally cache retrieval results separately with shorter TTL.

What is cache stampede and how do I prevent it?

Thundering herd on hot key expiry - use singleflight locks, jittered TTL, and background refresh for top queries.

Should I cache streaming responses?

Cache complete assembled string after stream ends. On hit, replay as stream with small inter-chunk delay if UX requires streaming appearance - or return instant full response.

How much Redis memory do I need?

Estimate: daily_unique_queries × avg_response_bytes × TTL_factor. 100K entries × 2KB ≈ 200MB plus overhead. Monitor used_memory and eviction rate.

How do I cache multi-turn conversations?

Cache per-turn responses keyed by (tenant, session_turn_hash, prompt_version) where turn hash includes prior turn IDs - not full history text. Alternatively cache only the retrieval portion per turn; regenerate with updated history for coherence.

Should I use Memcached instead of Redis?

Redis preferred for LLM caching - TTL per key, persistence options, and vector search extensions for semantic layers. Memcached works for simple exact response cache if you already operate it at scale.

How do I debug unexpected cache hits?

Log cache key components on hit (tenant, prompt_version, query hash). Compare against miss trace for same user report. Usually stale index_version or missing tenant scoping.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Cache exact responses, embeddings, and provider prompt prefixes - three complementary layers.
• Keys must include tenant, prompt version, and index version - never cache across tenants.
• TTL and invalidation are product policies, not afterthoughts.
• Lookup before embed/LLM in orchestrator; store only post-guardrail output.
• Add semantic caching when exact match hit rate is too low for FAQ traffic.