Caching Guide

TL;DR

• LLM caching stores expensive compute results — completed responses, embeddings, and static prompt prefixes — so identical work is not repeated.

• Exact-match caching keys on normalized input — same question + tenant + model + prompt version → cache hit; paraphrases miss unless you use semantic caching.

• Provider prompt caching discounts repeated input prefixes — structure prompts with static system content first, dynamic user content last. This is not the same as your app's response cache.

• TTL and invalidation are product decisions — FAQ cache 24h; account-specific answers 60s or no cache; never cache without tenant isolation.

• Caching wrong answers is worse than no cache — version cache keys with prompt_version and index_version; invalidate on deploy.

On this page

• Why This Matters
• The Problem Caching Solves
• How We Got Here
• What Is LLM Caching?
• How Caching Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Cache
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References

Why This Matters

Without caching, every "What are your business hours?" costs a full embed + retrieve + generate cycle. At 10K daily repeats, that is wasted thousands of dollars and added seconds of latency for deterministic answers.

Caching is the simplest high-ROI optimization after basic model routing. Hits are near-free and near-instant — the largest latency win on repeated traffic.

Done wrong, caches leak data across tenants, serve stale policies after doc updates, or freeze incorrect answers indefinitely. Production caching requires explicit key design, TTL policy, and invalidation hooks.

The Problem Caching Solves

LLM workloads repeat:

• Same FAQ questions phrased identically (or nearly — see semantic caching).
• Same documents embedded on every re-index job.
• Same system prompt prefix on every request.
• Same retrieval query within a session.

Recomputing these burns tokens, GPU/API quota, and milliseconds on the critical path. Caching trades memory and invalidation complexity for repeat lookup — O(1) Redis GET vs O(seconds) LLM.

How We Got Here

Early LLM apps called the API with no memory of prior answers. Cost crises pushed exact response caches (Redis). Providers then shipped prompt/prefix caching so static system prompts stop billing at full rate. Semantic layers followed for paraphrases.

Diagram: Caching layers arrived in waves

timeline
    title From raw API calls to multi-layer LLM caches
    2022-2023 : App response cache
              : Redis exact keys
    2023-2024 : Embedding cache
              : Hash → vector
    2024 : Provider prompt cache
              : Prefix discounts
    2025-2026 : Tiered stacks
              : Exact → semantic → LLM

App caches and provider prompt caches solve different problems — use both deliberately.

What Is LLM Caching?

LLM caching is storing and reusing prior computation results in LLM applications to reduce latency, cost, and load.

This article focuses on exact-match, embedding, and provider prompt caching. Similarity caching is covered in Semantic Caching.

Engineering Insight

Provider prompt cache ≠ app response cache. Prompt cache discounts repeated input tokens on the provider bill. Response cache skips the LLM call entirely in your infrastructure. Both can apply on the same request.

How Caching Works

Cache key design

A production cache key includes every input that affects output:

def response_cache_key(
    tenant_id: str,
    query: str,
    model: str,
    prompt_version: str,
    index_version: str,
) -> str:
    normalized = " ".join(query.lower().split())
    payload = f"{tenant_id}|{model}|{prompt_version}|{index_version}|{normalized}"
    return "llm:resp:" + hashlib.sha256(payload.encode()).hexdigest()

Missing tenant_id → cross-tenant data leak. Missing prompt_version → stale behavior after deploy. Missing index_version → wrong answers after re-index. Include model ID when routing Luna vs Terra vs Sol — answers differ by tier.

Lookup flow

1. Build key from tenant, normalized query, model, versions.
2. Redis GET — on hit, optionally re-validate versions in payload.
3. On miss, run pipeline; store only post-guardrail, cache-eligible output.
4. Separately, structure prompts for provider prefix cache (static first).

Provider prompt caching

OpenAI, Anthropic, and Google cache identical input token prefixes:

messages = [
    {"role": "system", "content": STATIC_SYSTEM_PROMPT},  # cacheable
    {"role": "user", "content": f"Context:\n{dynamic_rag}\n\nQ: {query}"},
]

Requirements: prefix must be byte-identical across requests. Dynamic timestamps or user IDs in the system prompt break cache. Billing: cached input tokens at reduced rate (provider-specific, often 50–90% off — verify current docs).

Embedding cache

def embed_with_cache(text: str, embed_fn) -> list[float]:
    key = "emb:" + hashlib.sha256(text.encode()).hexdigest()
    cached = redis.get(key)
    if cached:
        return json.loads(cached)
    vec = embed_fn(text)
    redis.setex(key, 86400 * 7, json.dumps(vec))
    return vec

Critical for re-indexing and duplicate chunks. Use content hash, not chunk ID — text may move. Flush or partition when the embedding model version changes.

Retrieval result cache

Cache chunk IDs returned for a query — cheaper than full response cache when generation varies (personalized tone) but retrieval is stable. TTL shorter than response cache (5–15 min).

Cache stampede protection

When a hot key expires, thousands of concurrent requests miss and hammer the LLM:

async def get_with_singleflight(key, factory, redis):
    cached = redis.get(key)
    if cached:
        return json.loads(cached)
    lock_key = f"lock:{key}"
    if redis.set(lock_key, "1", nx=True, ex=30):
        try:
            value = await factory()
            redis.setex(key, 3600, json.dumps(value))
            return value
        finally:
            redis.delete(lock_key)
    else:
        await asyncio.sleep(0.1)
        return await get_with_singleflight(key, factory, redis)

Use probabilistic early expiration (jitter TTL ±10%). For high-traffic keys, background refresh before TTL expires.

Architecture

Place cache lookup first in the orchestrator — before embed and LLM.

Diagram: Multi-layer cache stack

flowchart TB
    Q[Query] --> L0[L0 in-process LRU embeds]
    L0 -->|miss| L1[L1 Redis exact response]
    L1 -->|miss| L2[L2 retrieval result cache]
    L2 -->|miss| L3[L3 provider prompt cache]
    L3 --> L4[L4 semantic cache optional]
    L4 -->|miss| Full[Full RAG + LLM]
    L1 -->|hit| Out[Return]
    L4 -->|hit| Out
    Full --> Out

Check cheap exact layers before paying for embed, retrieval, or generation.

Typical stack timing:

L0: In-process LRU (embeddings)     ~0.01ms
L1: Redis exact response            ~1ms
L2: Redis retrieval result          ~1ms
L3: Provider prompt cache           billing discount
L4: Semantic cache                  ~50–100ms
L5: Full RAG + LLM                  ~1–3s

Step-by-Step Flow

Diagram: Exact cache miss path with invalidation hooks

sequenceDiagram
    participant C as Client
    participant O as Orchestrator
    participant R as Redis
    participant P as Pipeline
    participant Prov as Provider
    C->>O: Query
    O->>R: GET exact key
    alt hit
        R-->>C: Cached answer
    else miss
        O->>P: Retrieve + generate
        P->>Prov: LLM (prompt cache eligible)
        Prov-->>P: Answer + usage
        P->>O: Post-guardrail result
        O->>R: SETEX with versions
        O-->>C: Answer
    end
    Note over O,R: On deploy: bump prompt_version
    Note over O,R: On index promo: bump index_version

Version bumps invalidate logically without FLUSHALL.

1. Classify endpoints — cacheable (public FAQ) vs never-cache (personalized account actions).
2. Define key schema — tenant, model, prompt_version, index_version, normalized query hash.
3. Implement lookup — Redis GET; on hit, validate TTL and version metadata.
4. On miss, run pipeline — store response with TTL and metadata {created_at, prompt_version, sources}.
5. Enable provider prompt caching — refactor prompt layout; monitor cached_tokens.
6. Add embedding cache — indexing workers and repeated query embeds.
7. Wire invalidation — on index promotion, bump index_version or flush tenant keys.
8. Stampede protection — singleflight + jittered TTL.
9. Monitor hit rate — target 20–40% for FAQ-heavy apps on exact cache; tune TTL.

Real Production Example

Redis response cache with versioning and tenant isolation:

import hashlib
import json
import time
from dataclasses import dataclass
from typing import Optional

@dataclass
class CacheConfig:
    ttl_seconds: int = 3600
    prompt_version: str = "v3.2"
    index_version: str = "2026-07-21"


class LLMResponseCache:
    def __init__(self, redis_client, config: CacheConfig):
        self.redis = redis_client
        self.config = config

    def _key(self, tenant_id: str, query: str, model: str) -> str:
        norm = " ".join(query.lower().split())
        raw = (
            f"{tenant_id}|{model}|{self.config.prompt_version}|"
            f"{self.config.index_version}|{norm}"
        )
        return "llm:cache:" + hashlib.sha256(raw.encode()).hexdigest()

    def get(self, tenant_id: str, query: str, model: str) -> Optional[dict]:
        key = self._key(tenant_id, query, model)
        raw = self.redis.get(key)
        if not raw:
            metrics.increment("cache_miss")
            return None
        entry = json.loads(raw)
        if entry.get("prompt_version") != self.config.prompt_version:
            metrics.increment("cache_stale_version")
            return None
        metrics.increment("cache_hit")
        return entry

    def set(self, tenant_id: str, query: str, model: str, response: str, sources: list):
        key = self._key(tenant_id, query, model)
        entry = {
            "response": response,
            "sources": sources,
            "prompt_version": self.config.prompt_version,
            "index_version": self.config.index_version,
            "cached_at": time.time(),
        }
        self.redis.setex(key, self.config.ttl_seconds, json.dumps(entry))

    def invalidate_tenant(self, tenant_id: str):
        # Prefer per-tenant version bump over SCAN deletes at scale
        self.redis.incr(f"llm:cache:ver:{tenant_id}")


class CachedRAGService:
    def __init__(self, cache: LLMResponseCache, pipeline):
        self.cache = cache
        self.pipeline = pipeline

    async def query(self, tenant_id: str, query: str, model: str = "gpt-5.6-luna"):
        hit = self.cache.get(tenant_id, query, model)
        if hit:
            return {"answer": hit["response"], "sources": hit["sources"], "from_cache": True}

        result = await self.pipeline.run(tenant_id, query, model)
        if result.get("cache_eligible", True):
            self.cache.set(tenant_id, query, model, result["answer"], result["sources"])
        return {**result, "from_cache": False}

Mark cache_eligible=False for user-specific data or low confidence. Default volume model to Luna/Flash; include model in the key when Terra/Sol may answer differently.

Design Decisions

Comparisons

Diagram: Exact vs prompt vs semantic

flowchart LR
    subgraph App [Your infra]
        Exact[Exact response cache]
        Sem[Semantic cache]
    end
    subgraph Prov [Provider]
        Pref[Prompt / prefix cache]
    end
    Q[Query] --> Exact
    Exact -->|miss| Sem
    Sem -->|miss| Pref
    Pref --> Gen[Generate]

App caches avoid calls; provider prompt cache only discounts tokens on calls you still make.

Common Mistakes

1. Cache key without tenant_id. Critical security bug.
2. Infinite TTL on policy content. Refund rules change; users get wrong answers for weeks.
3. Dynamic content in system prompt. Breaks provider prompt cache every request.
4. Caching non-deterministic outputs (temperature > 0) without noting variance.
5. No cache bypass for support. Engineers cannot reproduce bugs.
6. Caching before guardrails. Store only post-validation output.
7. Ignoring memory limits. Redis evicts hot keys — monitor memory and hit rate drops.
8. Confusing prompt cache with response cache. Teams "enable caching" and wonder why the LLM still runs.

Where It Breaks Down

Personalized answers — "What's my order status?" cannot share cache across users. Key must include user-scoped entity IDs; often not worth caching.

Time-sensitive data — stock prices, SLA status. Short TTL or no cache.

Exact cache miss on paraphrases — "hours?" vs "when are you open?" — need semantic caching.

Stale after silent index update — bump index_version on every promotion; automate in CI/CD.

Compliance — regulated industries may prohibit storing LLM outputs — check retention policy.

Stampede on expiry — hot FAQ keys without singleflight melt provider quotas.

When NOT to Cache

Do not cache (or use only very short TTL) when:

• Personalized or auth-gated account data — order status, balances, private tickets.
• High-stakes advice — medical, legal, financial recommendations that must reflect latest policy without lag.
• Rapidly changing facts — prices, inventory, live incident status.
• Responses that failed guardrails — never store blocked or toxic outputs.
• A/B experimental prompts — mix versions in one key pollutes experiments.
• Retention policy forbids storing outputs — compliance override.
• Temperature > 0 creative paths where freezing one sample is undesirable.

Warning

Include prompt_version and index_version in every response cache key. Deploy without a version bump serves stale wrong answers at lightning speed.

Running in Production

Best Practice

Centralize key building. Version prompts and indexes. Stampede-protect hot keys. Store only post-guardrail output.

Production checklist

• Centralized key builder with tenant + model + prompt_version + index_version
• TTL policy per endpoint class
• Invalidation on deploy and index promotion (version bump)
• Singleflight / jitter for hot keys
• Provider prompt layout: static prefix first
• Embedding cache keyed by content hash + embed model version
• Admin no-cache bypass
• Hit/miss/stale metrics and memory alerts

Operational notes

Treat cache hit rate as a capacity and FinOps signal, not a vanity metric. A sudden drop often means a broken key builder (missing tenant, forgotten version bump) rather than organic traffic change. A sudden spike after a prompt deploy without a version bump usually means you are serving stale answers. Pair Redis metrics with LLM $/request: if hit rate is healthy but spend is flat, you may be caching the wrong endpoints (low-token paths) while missing fat RAG routes. When routing across GPT-5.6 Luna / Terra / Sol (or Haiku 4.5 / Sonnet 5 / Gemini 3.7 Flash), keep model ID in the key so a cheap-tier FAQ answer is never reused for a Sol-tier escalation path.

Related Guides

Efficiency cluster:

• Semantic Caching — similarity-based hits for paraphrases
• Cost Optimization — caching is a top-tier cost lever
• Latency Optimization — cache hits are the fastest path

Foundations:

• Tokens — prompt caching reduces billed input tokens
• Embeddings — embedding cache speeds indexing
• Prompt Engineering — static vs dynamic prompt layout

Tools: Redis · LangChain

Diagram: Caching learning path

flowchart LR
    T[Tokens] --> CO[Cost opt]
    CO --> C[Caching]
    C --> SC[Semantic cache]
    C --> L[Latency opt]

Exact cache first; add semantic when paraphrase miss rate is high.

Interview Questions

1. What belongs in an LLM response cache key?
   Tenant ID, normalized query, model, prompt version, index/knowledge version, and any parameter that changes output.

2. Prompt caching vs response caching?
   Prompt caching is provider-side discount on repeated input prefixes. Response caching stores complete outputs in your infra and can skip the LLM entirely.

3. How do you invalidate on deploy?
   Bump prompt_version / index_version in the key builder — safer than FLUSHALL.

4. What is a cache stampede?
   Thundering herd on hot key expiry — mitigate with singleflight locks, jittered TTL, and background refresh.

5. Should you cache at temperature 0?
   Yes for deterministic FAQ paths; still version the model ID because provider snapshots change.

6. How does caching interact with RAG?
   Include index_version so new documents invalidate stale answers; optionally cache retrieval separately with shorter TTL.

7. When is negative caching useful?
   Short-TTL cache of "no answer" for repeated unanswerable spam that would otherwise burn full pipelines.

8. Why include model in the key when routing Luna vs Sol?
   Different tiers produce different answers; sharing one entry across tiers serves the wrong quality/cost tradeoff.

Key Takeaways

• Cache exact responses, embeddings, and provider prompt prefixes — three complementary layers.
• Keys must include tenant, prompt version, and index version — never cache across tenants.
• TTL and invalidation are product policies, not afterthoughts.
• Lookup before embed/LLM; store only post-guardrail output.
• Add semantic caching when exact-match hit rate is too low for FAQ traffic.

FAQs

Should I cache LLM responses?

Yes for deterministic, non-personalized queries with proper TTL and tenant-scoped keys. No for user-specific or time-critical data without short TTL.

What belongs in a cache key?

Tenant ID, normalized query, model ID, prompt version, index/knowledge base version, and parameters affecting output.

What is prompt caching vs response caching?

Prompt caching discounts repeated input prefixes on the provider. Response caching stores complete outputs in your infrastructure.

What TTL should I use?

FAQ/public docs: 1–24 hours. Semi-dynamic: 5–15 minutes. User-specific: avoid or <60 seconds. Policy/legal: invalidate on doc change, not TTL alone.

How do I invalidate cache on deploy?

Bump prompt_version on prompt deploy and index_version on vector index promotion. Avoid FLUSHALL in production.

Can I cache at temperature 0?

Yes — output is deterministic for the same model/version. Version the model in the key.

Does caching help latency?

Yes — Redis hit is ~1ms vs seconds for LLM. Largest latency win after streaming for repeated queries.

What is embedding cache?

Store text hash → vector to avoid re-embedding identical strings during indexing and repeated queries.

What is negative caching?

Caching "no results" or "I don't know" with short TTL to prevent repeated expensive pipelines for spam.

How do I measure cache effectiveness?

Hit rate = hits / (hits + misses). Track cost avoided = hits × avg_cost_per_miss.

When should I not cache?

Personalized advice, live account data, failed guardrails, experimental A/B prompts, retention bans.

How does caching interact with RAG?

Include index_version in the key. Optionally cache retrieval results separately with shorter TTL.

What is cache stampede and how do I prevent it?

Thundering herd on expiry — singleflight locks, jittered TTL, background refresh.

Should I cache streaming responses?

Cache the assembled string after stream ends. On hit, replay as stream or return instantly.

How much Redis memory do I need?

Estimate daily_unique_queries × avg_response_bytes × TTL_factor. Monitor used_memory and evictions.

How do I cache multi-turn conversations?

Key by (tenant, session_turn_hash, prompt_version) or cache retrieval only and regenerate with updated history.

Should I use Memcached instead of Redis?

Redis preferred — per-key TTL, persistence options, and vector extensions for semantic layers.

How do I debug unexpected cache hits?

Log key components on hit; compare against miss traces. Usually stale index_version or missing tenant scoping.

References

• OpenAI — Prompt Caching
• Anthropic — Prompt Caching
• Redis Documentation
• LangChain — Caching
• Google AI for Developers

Further Reading

• GPTCache Documentation
• LlamaIndex Documentation
• Cloudflare AI Gateway