Tokens Guide

TL;DR

• A token is the atomic unit an LLM processes - not a word, but a chunk of text ranging from a single character to a full word, depending on the tokenizer.

• Tokenization happens before the model sees your text - your prompt is converted to a sequence of integer token IDs, processed by the neural network, then decoded back to text.

• Token count directly determines cost, speed, and context limits - API pricing is per token, generation latency scales with token count, and context windows are measured in tokens.

• English averages ~4 characters per token, but this varies wildly - code, non-English text, numbers, and special characters tokenize less efficiently.

• Always count tokens before sending requests - use tiktoken (OpenAI), the provider's tokenizer, or API usage fields to avoid surprise costs and context overflow.

Why This Matters

Every dollar you spend on LLM APIs, every millisecond of latency your users experience, and every context window limit you hit - all of it is measured in tokens. If you do not understand tokenization, you cannot estimate costs, debug context overflow errors, or optimize prompts.

A common production failure: a team builds a RAG pipeline that retrieves 20 chunks of 512 tokens each, fills 10,000 tokens of context, and wonders why responses are slow and expensive. The fix is not a better model - it is understanding that those 10,000 tokens cost real money on every query and leave less room for the actual answer.

Tokenization also explains seemingly bizarre model behavior: why the model struggles with counting letters in "strawberry," why "hello" and hello tokenize differently, and why JSON with lots of brackets costs more than equivalent plain text.

The Problem Tokens Solve

Neural networks operate on fixed-size numerical inputs - they cannot read raw strings. Tokenization bridges human-readable text and the integer sequences models process.

The challenge is representing the entire vocabulary of human language (plus code, math, emoji, and every language on Earth) as a finite set of token IDs. A model with a 100,256-token vocabulary must encode any input text as a sequence drawn from those 100,256 possibilities.

Good tokenization balances three competing goals:

1. Vocabulary size - Larger vocabularies mean fewer tokens per text (lower cost) but bigger embedding tables (more memory).

2. Compression efficiency - Common words should be single tokens; rare words split into subword pieces.

3. Consistency - The same text must always produce the same token sequence, regardless of surrounding context.

Byte Pair Encoding (BPE) and its variants solve this by learning a vocabulary from training data, starting with individual bytes and iteratively merging the most frequent pairs.

What Is a Token?

A token is an entry in a model's vocabulary - an integer ID mapped to a text fragment. When you send "Hello, world!" to GPT-4o, the tokenizer converts it to something like:

[9906, 11, 1917, 0]  →  ["Hello", ",", " world", "!"]

Four tokens for a four-word sentence. But token boundaries are not intuitive:

The tokenizer is model-specific. GPT-4o uses cl100k_base (via tiktoken). Llama 3 uses SentencePiece BPE. Claude uses its own tokenizer. You cannot count tokens for one model and assume the count applies to another.

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")

text = "ChatGPT tokenizes this differently than you'd expect."
tokens = enc.encode(text)
print(f"Token count: {len(tokens)}")
print(f"Tokens: {[enc.decode([t]) for t in tokens]}")
# Token count: 11
# Tokens: ['Chat', 'G', 'PT', ' token', 'izes', ' this', ' differently', ' than', ' you', "'d", ' expect', '.']

Warning

Never estimate token counts by dividing character count by 4 for non-English text, code, or structured data. Always use the model's actual tokenizer. A JSON payload with nested objects can be 2–3x more token-dense than equivalent prose.

How Tokenization Works

Byte Pair Encoding (BPE)

BPE is the dominant tokenization algorithm for modern LLMs. The training process:

Multi-head attention runs several attention functions in parallel, letting the model attend to information from different representation subspaces.

At inference time, encoding applies the learned merge rules greedily:

1. Split text into bytes/characters.
2. Apply merge rules in order - merge the highest-priority pair first.
3. Map resulting pieces to token IDs.

Scaled dot-product attention maps a query and a set of key-value pairs to an output, scaling dot products by 1/√d_k before softmax.

Tokenizer Variants

Architecture

Tokenization sits at the boundary between your application and the model:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Every production LLM pipeline should include a token counting step before the API call. This prevents context overflow, enables cost estimation, and supports intelligent truncation.

Step-by-Step Flow

Here is the complete token lifecycle in a production request:

1. User submits input - Raw text, potentially with conversation history and retrieved documents.

2. Assemble full prompt - System prompt + history + context + user message.

3. Count input tokens - Use tiktoken or provider SDK to get exact count.

4. Check against budget - Compare to model's context window minus reserved output tokens.

5. Truncate if needed - Remove oldest history turns, summarize context, or reduce retrieved chunks.

6. Send to API - Provider re-tokenizes server-side (your count should match).

7. Model generates output tokens - One token at a time, autoregressively.

8. Receive response + usage - API returns prompt_tokens, completion_tokens, total_tokens.

9. Log and bill - Record token usage for cost tracking and anomaly detection.

Real Production Example

A document Q&A system that must stay within a 128K context window while maximizing retrieved context:

import tiktoken
from dataclasses import dataclass

MODEL = "gpt-4o"
CONTEXT_WINDOW = 128_000
RESERVED_OUTPUT = 4_096  # Max tokens for the answer
SYSTEM_PROMPT_TOKENS = 200  # Pre-counted system prompt

enc = tiktoken.encoding_for_model(MODEL)

@dataclass
class TokenBudget:
    total: int
    available_for_context: int
    used_by_history: int
    used_by_retrieval: int
    remaining: int

def calculate_budget(
    system_prompt: str,
    conversation_history: list[dict],
    retrieved_chunks: list[str],
) -> TokenBudget:
    system_tokens = len(enc.encode(system_prompt))
    history_tokens = sum(
        len(enc.encode(msg["content"])) + 4  # +4 for message overhead
        for msg in conversation_history
    )

    max_context = CONTEXT_WINDOW - RESERVED_OUTPUT - system_tokens
    used_by_history = history_tokens
    available_for_retrieval = max_context - used_by_history

    # Fit as many chunks as possible within budget
    fitted_chunks = []
    retrieval_tokens = 0
    for chunk in retrieved_chunks:
        chunk_tokens = len(enc.encode(chunk))
        if retrieval_tokens + chunk_tokens <= available_for_retrieval:
            fitted_chunks.append(chunk)
            retrieval_tokens += chunk_tokens
        else:
            break

    total = system_tokens + used_by_history + retrieval_tokens

    return TokenBudget(
        total=total,
        available_for_context=max_context,
        used_by_history=used_by_history,
        used_by_retrieval=retrieval_tokens,
        remaining=CONTEXT_WINDOW - total - RESERVED_OUTPUT,
    )

def truncate_conversation(history: list[dict], max_tokens: int) -> list[dict]:
    """Keep most recent turns that fit within token budget."""
    truncated = []
    token_count = 0
    for msg in reversed(history):
        msg_tokens = len(enc.encode(msg["content"])) + 4
        if token_count + msg_tokens > max_tokens:
            break
        truncated.insert(0, msg)
        token_count += msg_tokens
    return truncated

This pattern - pre-count, budget, truncate - prevents the two most common token-related production failures: context overflow and runaway costs.

Design Decisions

⚠ Common Mistakes

1. Using character count as token count - "https://api.example.com/v2/users/12345/profile" is ~10 tokens, not ~45 characters / 4 ≈ 11. URLs, code, and JSON tokenize inefficiently. Always use the tokenizer.

2. Not reserving output tokens - Filling a 128K context window with input leaves zero room for the model's response. Reserve at least 1,024–4,096 tokens for output.

3. Ignoring token overhead in chat messages - Each message in a chat API adds ~4 tokens of formatting overhead (role markers, separators). A 50-turn conversation adds ~200 tokens of overhead beyond message content.

4. Assuming tokenizers are interchangeable - Switching from GPT-4o to Claude changes token counts for the same text. Re-benchmark costs and context budgets when changing models.

5. Not setting max_tokens - Without a cap, a model can generate thousands of tokens on a simple question, especially with high temperature. Always set max_tokens.

6. Counting tokens after the fact only - Logging usage is necessary but insufficient. Pre-flight token counting prevents sending requests that will fail or cost too much.

Where It Breaks Down

Tokenization creates specific failure modes:

• Letter counting and spelling - Models process tokens, not characters. Asking "how many r's are in strawberry?" fails because the model never sees individual letters - it sees token IDs representing subword chunks.

• Precise string manipulation - Tasks requiring exact character-level operations (base64 encoding, hash verification, regex on specific positions) fail because token boundaries do not align with character boundaries.

• Non-English efficiency - Languages with large character sets (Chinese, Japanese, Arabic) often require more tokens per word than English, increasing cost and reducing effective context window.

• Code tokenization - Indentation, brackets, and operators often become separate tokens. A 100-line Python file may be 2,000+ tokens - always count before sending code to an LLM.

• Special tokens - Some tokenizers add special tokens (<|endoftext|>, [INST]) that count toward your budget but are invisible in the decoded text.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Ecosystem

• tiktoken - OpenAI's fast BPE tokenizer library (Python, Rust). The standard for counting GPT model tokens.

• Hugging Face tokenizers - Rust-based library supporting Llama, Mistral, and most open models.

• Anthropic tokenizer - Available via their SDK for Claude token counting.

• OpenAI API usage field - Every response includes prompt_tokens, completion_tokens, total_tokens - the source of truth for billing.

• LangChain token counters - Wrappers around tiktoken and HF tokenizers integrated into chain callbacks.

Related Technologies

• Large Language Models - Models that consume and produce tokens. Understanding LLMs starts with understanding their input format.

• Context Windows - Context windows are measured in tokens. Token counting is how you manage context limits.

• Prompt Engineering - Prompt design directly affects token count.

Shorter, focused prompts cost less and often perform better.

• Cost Optimization - Token management is the primary lever for reducing LLM API costs.

• Embeddings - Embedding models also tokenize input, with their own token limits (typically 512–8,192 tokens per input).

Learning Path

Prerequisites: Large Language Models

Next topics: Context Windows · Prompt Engineering · Cost Optimization

Estimated time: 30 min · Difficulty: Beginner

FAQs

How many tokens is a word?

Roughly 0.75 words per token for English prose (about 1.3 tokens per word). "Hello world" is typically 2 tokens. But this varies: technical terms, abbreviations, and non-English text can be 2–3 tokens per word.

Why does "ChatGPT" use 3 tokens?

BPE learned that "Chat" is a frequent token (from training data), but "G" and "PT" are not common enough together to form a single merge. The tokenizer splits at the most efficient boundaries based on training corpus statistics.

How do I count tokens for Claude?

Use the Anthropic SDK: client.messages.count_tokens(model="claude-sonnet-4-20250514", messages=[...]). Alternatively, use the approximation of ~3.5 characters per token for English text with Claude's tokenizer.

Do tokens include the system prompt?

Yes. Everything sent in the API request counts toward input tokens: system prompt, conversation history, retrieved context, and the user's message. The context window limit applies to the total.

What happens when I exceed the context window?

API providers return an error (400 Bad Request) if input exceeds the limit. Some providers silently truncate from the beginning - losing your system prompt or early context. Always pre-count to avoid both scenarios.

Are whitespace and punctuation tokens?

Often yes. A leading space is frequently part of a token (" world" vs "world" are different tokens). Punctuation attached to words may be merged ("hello!") or separate ("hello", "!") depending on training frequency.

How many tokens can I send per API request?

Depends on the model: GPT-4o supports 128K, Claude supports 200K, Gemini supports 1M+ (with quality degradation at extreme lengths). Check the provider's current documentation - limits increase regularly.

Does token count affect response quality?

Indirectly. More input tokens mean the model has more context to work with (up to a point), but also increase "lost in the middle" effects where information in the center of long contexts gets ignored. More output tokens mean longer generation time.

How do I reduce token usage in prompts?

Remove redundant instructions, use abbreviations in system prompts, compress conversation history (summarize old turns), retrieve fewer/smaller chunks in RAG, and use shorter model names in few-shot examples. Measure before and after.

Can I cache tokens to reduce costs?

OpenAI and Anthropic offer prompt caching - if the same prefix (system prompt + early context) is sent repeatedly, cached tokens are billed at a reduced rate (50–90% discount). Structure prompts with static content first, dynamic content last.

References

• Attention Is All You Need (Vaswani et al., 2017)
• Hugging Face Transformer Architecture
• PyTorch Transformer Tutorial

Further Reading

• Google Research Blog: Transformer
• TensorFlow Transformer Guide
• The Illustrated Transformer (Jay Alammar)

Summary

• Tokens are the fundamental unit of LLM input and output - not words, not characters. - Token count determines cost, latency, and context capacity. Count before every request. - Use the model-specific tokenizer (tiktoken for OpenAI, provider SDK for others) - never estimate with character heuristics alone. - Always set max_tokens, reserve output budget, and truncate input to fit within the context window.

• Tokenization explains many LLM failure modes: letter counting, string manipulation, and non-English inefficiency. - Token management is the highest-leverage cost optimization for LLM applications.