Context Windows Guide

TL;DR

• The context window is the maximum number of tokens an LLM processes in a single request - input plus output combined.

• Current models range from 8K to 1M+ tokens, but effective usable context is often much smaller due to quality degradation and cost.

• Longer context is not always better - the "lost in the middle" effect means models ignore information buried in the center of long prompts.

• Production systems use RAG, summarization, and sliding windows to work within context limits rather than stuffing everything into the prompt.

• Context directly drives cost and latency - a 100K-token prompt costs 50x more than a 2K-token prompt and takes longer to process.

Why This Matters

Context window management is the single most impactful engineering decision in LLM applications. Get it wrong and you either overflow the limit (hard error), silently truncate critical information (wrong answers), or burn budget on unnecessary tokens (cost explosion).

Every architecture decision - whether to use RAG or stuff documents into the prompt, how much conversation history to retain, whether to summarize or truncate - flows from context window constraints. A team that understands context management builds systems that are faster, cheaper, and more accurate than one that simply uses the largest available window.

The context window is also where model capabilities diverge most visibly. Claude's 200K window enables full-codebase analysis. GPT-4o's 128K window handles long documents. But both degrade in quality when you approach their limits - understanding these boundaries is essential for production design.

The Problem Context Windows Solve

LLMs process input through self-attention, where every token attends to every other token. For a sequence of length N, attention requires O(N²) computation and memory. A 128K-token input creates a 128K × 128K attention matrix - billions of operations per forward pass.

Context windows exist because:

1. Compute is finite - Attention scales quadratically. Without limits, a single request could consume all available GPU memory.

2. Quality degrades at scale - Models are trained on shorter sequences. Performance on tasks requiring information from the middle of very long contexts is significantly worse than from the beginning or end.

3. Cost must be bounded - API providers price per token. Unlimited context would mean unlimited cost per request.

The engineering problem is not "how do I fit more text in" but "how do I select the most relevant text for this specific query."

What Is a Context Window?

The context window (also called context length or context size) is the maximum number of tokens an LLM can process in one inference call. It includes everything: system prompt, conversation history, retrieved documents, tool results, and the user's message - plus the tokens the model generates in response.

Context Window (128K tokens)
┌─────────────────────────────────────────────────────────┐
│ System Prompt │ History │ Retrieved Docs │ User Msg │ Output │
│    ~200 tok   │ ~2K tok │   ~10K tok     │ ~100 tok │ ~2K tok │
└─────────────────────────────────────────────────────────┘
Total used: ~14.3K / 128K available

Current context window sizes (as of 2026):

Important

Advertised context window size is an upper bound, not a quality guarantee. Most models perform best with 8K–32K tokens of relevant context. Test quality at your target context length before relying on long-context capabilities.

How Context Windows Work

Attention and Memory

The context window is implemented as the maximum sequence length the model's attention mechanism can process:

The original Transformer uses stacked encoder and decoder blocks. Each block combines multi-head self-attention with position-wise feed-forward layers, residual connections, and layer normalization.

During generation, the context grows token by token. Each new output token increases the attention matrix size. This is why long outputs are slower - not just because there are more tokens to generate, but because each token requires attention over an increasingly large sequence.

Positional Encoding

Models track token position through positional encodings (RoPE, ALiBi, or learned embeddings). Extended context windows require extrapolating these encodings beyond training length - which contributes to quality degradation at extreme lengths.

Multi-head attention runs several attention functions in parallel, letting the model attend to information from different representation subspaces.

The Lost-in-the-Middle Effect

Research consistently shows that LLMs prioritize information at the beginning and end of the context, often ignoring content in the middle:

Attention strength by position in context:

High  ████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░████████
      ↑ Beginning (strong)              End (strong) ↑
                        ↑ Middle (weak) ↑

This has direct engineering implications: put the most important context (system instructions, key documents) at the beginning and end of your prompt, not in the middle.

Architecture

Production context management spans multiple layers:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Step-by-Step Flow

Managing context in a production chat application with RAG:

1. Load system prompt (~200 tokens) - Fixed instructions, always included.

2. Load conversation history - Retrieve from session store, count tokens.

3. Retrieve relevant documents - Embed query, search vector DB, get top-k chunks.

4. Calculate token budget - available = context_window - system - reserved_output - user_message.

5. Allocate budget - Prioritize: user message > retrieved docs > recent history > old history.

6. Truncate overflow - Drop oldest history turns first; reduce retrieved chunks if still over budget.

7. Assemble final prompt - System prompt → key context → history → user message.

8. Send to LLM - With max_tokens set to reserved output budget.

9. Store turn - Persist user message and response for next turn's history.

Real Production Example

A customer support bot that manages context across a multi-turn conversation with document retrieval:

import tiktoken
from dataclasses import dataclass, field

MODEL = "gpt-4o"
CONTEXT_WINDOW = 128_000
MAX_OUTPUT = 2_048
SYSTEM_PROMPT = "You are a support agent for Acme Corp. Answer using provided docs."

enc = tiktoken.encoding_for_model(MODEL)

@dataclass
class ContextManager:
    system_prompt: str
    max_output: int = MAX_OUTPUT
    context_window: int = CONTEXT_WINDOW
    history: list[dict] = field(default_factory=list)

    def count_tokens(self, text: str) -> int:
        return len(enc.encode(text))

    def message_tokens(self, msg: dict) -> int:
        return self.count_tokens(msg["content"]) + 4

    def build_messages(
        self,
        user_message: str,
        retrieved_docs: list[str],
    ) -> tuple[list[dict], dict]:
        budget = self.context_window - self.max_output
        system_tokens = self.count_tokens(self.system_prompt)
        user_tokens = self.count_tokens(user_message) + 4
        remaining = budget - system_tokens - user_tokens

        # Allocate retrieved docs (priority: high)
        doc_budget = min(remaining * 0.6, remaining - 1000)  # Reserve 1K for history
        fitted_docs = []
        doc_tokens = 0
        for doc in retrieved_docs:
            dt = self.count_tokens(doc)
            if doc_tokens + dt <= doc_budget:
                fitted_docs.append(doc)
                doc_tokens += dt

        remaining -= doc_tokens

        # Fit conversation history (most recent first)
        fitted_history = []
        history_tokens = 0
        for msg in reversed(self.history):
            mt = self.message_tokens(msg)
            if history_tokens + mt <= remaining:
                fitted_history.insert(0, msg)
                history_tokens += mt
            else:
                break

        context_block = ""
        if fitted_docs:
            context_block = "Reference documents:\n" + "\n---\n".join(fitted_docs)

        messages = [{"role": "system", "content": self.system_prompt}]
        if context_block:
            messages.append({"role": "system", "content": context_block})
        messages.extend(fitted_history)
        messages.append({"role": "user", "content": user_message})

stats = {
            "system_tokens": system_tokens,
            "doc_tokens": doc_tokens,
            "history_tokens": history_tokens,
            "user_tokens": user_tokens,
            "total_input": system_tokens + doc_tokens + history_tokens + user_tokens,
            "docs_included": len(fitted_docs),
            "history_turns": len(fitted_history),
            "budget_used_pct": round(
                (system_tokens + doc_tokens + history_tokens + user_tokens) / budget * 100, 1
            ),
        }

        return messages, stats

Design Decisions

⚠ Common Mistakes

1. Filling the entire context window - Using 120K of a 128K window leaves almost no room for output and maximizes cost. Target 60–80% utilization with relevant content, not 100%.

2. Putting critical instructions in the middle - Due to lost-in-the-middle, system instructions and key context at the center of a long prompt get ignored. Place them at the beginning or end.

3. Not counting tokens before sending - Relying on the API to reject oversized requests. Some providers silently truncate from the start, dropping your system prompt.

4. Keeping all conversation history - A 50-turn conversation can consume 20K+ tokens of history, crowding out retrieved documents. Implement history truncation or summarization.

5. Assuming long context replaces RAG - Stuffing 100 documents into a 128K window is expensive ($0.32/query on GPT-4o) and lower quality than retrieving the 5 most relevant chunks via RAG.

6. Ignoring output token budget - Not setting max_tokens and letting the model generate a 4K-token response when 200 tokens suffice.

Where It Breaks Down

• Needle-in-a-haystack - Models fail to retrieve specific facts buried in long contexts, even within their advertised window. Do not rely on long-context for precise fact lookup - use RAG.

• Multi-document reasoning - Comparing information across 20+ documents in context produces unreliable results. Use iterative retrieval or agentic RAG instead.

• Real-time data - Context window content is static at request time.

For changing data, RAG with fresh retrieval beats a large static context.

• Cost at scale - A 100K-token prompt at $2.50/1M input tokens costs $0.25 per query. At 10K queries/day, that is $2,500/day. RAG with 5K tokens costs $0.0125/query - 20x cheaper.

• Latency - Processing 128K tokens adds 2–5 seconds of prefill latency before the first output token. Users notice.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Ecosystem

• Prompt caching - OpenAI and Anthropic cache repeated prompt prefixes, reducing cost for static system prompts and document sets.

• Context compression - LLMLingua, RECOMP - compress long contexts while preserving key information.

• Summarization chains - LangChain's ConversationSummaryMemory - automatically summarize old conversation turns.

• Long-context models - Claude (200K), Gemini (1M), GPT-4o (128K) - for tasks requiring full document processing.

• RAG frameworks - LangChain, LlamaIndex - manage retrieval to minimize context usage.

Related Technologies

• Tokens - Context windows are measured in tokens. Token counting is how you manage context.

• Large Language Models - Models define the context window size and quality at different lengths.

• RAG - The primary strategy for working within context limits with large knowledge bases.

• Prompt Engineering - Prompt design affects how efficiently you use context budget.

• Agent Memory - How agents manage context across multi-step tasks and sessions.

• Cost Optimization - Context size is the primary driver of LLM input cost.

Learning Path

Prerequisites: Tokens · Large Language Models

Next topics: Prompt Engineering · RAG · Agent Memory

Estimated time: 40 min · Difficulty: Intermediate

FAQs

Does the context window include the response?

Yes. Input tokens plus output tokens must fit within the context window. If you send 126K tokens of input to a 128K model, you only have 2K tokens for the response.

What happens when I exceed the context window?

Most API providers return a 400 error. Some silently truncate from the beginning of the input - which can remove your system prompt. Always pre-count tokens to avoid both outcomes.

Is a 1M token context window actually useful?

For specific tasks - full codebase analysis, processing entire books - yes. For general Q&A and RAG, quality degrades well before 1M tokens. Test on your use case; do not assume bigger is better.

How much conversation history should I keep?

Keep enough for conversational coherence - typically 5–10 recent turns. Summarize older turns if the conversation is long. Monitor token usage and truncate when history exceeds 30–40% of your context budget.

Should I use long context or RAG?

Use RAG for Q&A over large, dynamic corpora (cheaper, more accurate). Use long context for analyzing a specific document in full (code review, contract analysis). Most production systems use both.

What is prompt caching and how does it help?

Prompt caching stores the computed attention states for repeated prompt prefixes (system prompt, static documents). Subsequent requests with the same prefix skip recomputation - reducing latency by 50–80% and cost by 50–90% for the cached portion.

How do I handle context for multi-turn tool-calling agents?

Tool results consume context rapidly. An agent making 10 tool calls might accumulate 20K+ tokens of results. Implement result summarization, store full results externally, and pass summaries to the model.

Does context window size affect embedding models?

Embedding models have separate, smaller context limits (512–8,192 tokens). These are independent of LLM context windows. See embeddings.

How do I test context quality?

Create eval cases with facts placed at the beginning, middle, and end of context. Measure whether the model retrieves each fact correctly. This reveals lost-in-the-middle effects for your specific use case.

Can I extend a model's context window?

Open-weight models support context extension via RoPE scaling or YaRN - but quality is not guaranteed beyond training length. For API models, you are limited to the provider's advertised window.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Context window = max tokens per request (input + output). It is a hard limit with cost and quality implications. - Effective context is often much smaller than the advertised maximum - test quality at your target length. - Use RAG to select relevant content rather than stuffing entire corpora into the prompt. - Place critical instructions and data at the beginning and end of context - not the middle.

• Always count tokens, set max_tokens, truncate history, and monitor context utilization in production.