Tokens Guide

TL;DR

• Tokens exist because neural networks operate on bounded integer vocabularies, not arbitrary strings. A tokenizer deterministically maps text to token IDs; an embedding lookup then maps each ID to a vector.
• Tokenization is a compression and representation decision. Whole-word vocabularies cannot cover every name, typo, language, number, and code fragment, while character-level input makes sequences unnecessarily long. Subword tokenization occupies the middle ground.
• Tokenizer training and model training are different stages. Tokenizer training learns a vocabulary and segmentation model from a corpus. Model training consumes the resulting IDs to learn next-token prediction. Inference reuses the frozen tokenizer; it does not relearn merges.
• BPE and SentencePiece are not interchangeable labels. BPE is a family of merge-based algorithms. SentencePiece is a language-independent tokenizer toolkit that can train BPE or Unigram models directly from raw text and explicitly represents whitespace.
• Context, cost, and latency are coupled to tokens. Input and generated output share a context budget. Providers bill by token category, while autoregressive output usually adds sequential latency one token at a time.
• Client-side counts are planning estimates; API usage is the billing record. Message wrappers, tool schemas, images, hidden control tokens, caching categories, and provider revisions can make raw-text counts differ from request usage.
• Do not optimize token count blindly. Removing necessary instructions or examples can reduce tokens while increasing retries, errors, and total cost. Optimize cost per successful task, not tokens per request.

On this page

• Why This Matters
• The Problem Tokens Solve
• How We Got Here
• What Is a Token?
• How Tokenization Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Over-Optimize Tokenization
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

Before asking how a tokenizer splits a word, ask why tokens exist at all. A large language model is a numerical function. It accepts tensors, performs matrix operations, and returns a probability distribution over a finite set of output symbols. Raw Unicode strings have variable length, an enormous and evolving symbol space, and no direct row in a model's embedding matrix. Tokens provide a stable interface: arbitrary text becomes a sequence of integers drawn from a fixed vocabulary.

That interface becomes part of the model contract. It determines the embedding table's size, the sequence length seen during training, how much text fits in a context window, and which output units the model predicts. A tokenizer that represents a domain efficiently can shorten training sequences. A tokenizer poorly matched to a language or codebase can consume more positions for the same information and leave less effective context.

In production, tokens are also an accounting and capacity unit:

• API providers meter input, cached input, reasoning, and output using provider-defined token categories.
• Admission control must reserve enough context for system instructions, conversation state, retrieved evidence, tool definitions, and output.
• Prefill work grows with input length and model architecture; decode work repeats for each generated token.
• Rate limits are often expressed in tokens per minute as well as requests per minute.
• Prompt changes alter both quality and operating cost, so token usage belongs in evaluation telemetry.

The practical lesson is not that every character must be micro-optimized. It is that application boundaries should treat token budgets as explicitly as databases treat connection pools or services treat memory limits.

The Problem Tokens Solve

A language model needs an input alphabet that satisfies conflicting constraints.

A word-level vocabulary is too brittle. It needs entries for inflections, spelling variants, user names, product IDs, source code identifiers, URLs, and every supported language. Unknown words require an out-of-vocabulary symbol, which destroys information. Keeping every observed word instead creates an enormous embedding and output matrix, including many rarely trained rows.

A character-level vocabulary is robust but inefficient. It can represent unseen words, but common concepts require long sequences. Attention and activation memory then operate over more positions, training examples carry less semantic content per sequence, and generation requires more decoding steps.

A byte-level vocabulary is complete but even more granular. Bytes guarantee that any UTF-8 input can be represented without an unknown token. The trade-off is that multibyte characters and common strings would consume many positions if bytes were never merged.

Subword tokenization combines completeness with compression. Frequent strings become single vocabulary entries; rare strings decompose into smaller pieces, characters, or bytes. The vocabulary remains finite while arbitrary input remains representable.

This is not semantic parsing. A tokenizer does not need to understand that authentication and login are related. It learns useful recurring surface patterns. Semantic representation emerges later, when the model learns embeddings and contextual transformations for token sequences.

How We Got Here

Early NLP systems used hand-built rules, words, stems, and task-specific vocabularies. Neural language models initially inherited word-level tokenization and an unknown-word marker. As models moved toward open-domain text, translation, multilingual corpora, and code, out-of-vocabulary handling became a central limitation.

Subword methods addressed that limitation. Byte Pair Encoding originated as a compression procedure and was adapted to learn frequent symbol merges for neural machine translation. WordPiece used a related iterative vocabulary construction objective. SentencePiece removed the requirement for pre-tokenized, whitespace-separated input and provided BPE and probabilistic Unigram algorithms. Byte-level BPE later made arbitrary UTF-8 text representable without an unknown character path.

Diagram: Evolution of model text units

timeline
    title From words to model-specific subwords
    1990s : Word vocabularies and unknown tokens
    2012 : Neural word embeddings
    2016 : BPE for neural translation
    2018 : SentencePiece and Unigram adoption
    2019 : Byte-level BPE in generative models
    2020s : Multilingual, code, and multimodal vocabularies

Tokenization evolved toward reversible, model-specific subword systems that balance sequence length, vocabulary size, and coverage.

The tokenizer is normally finalized before large-scale model pretraining because changing token IDs or vocabulary rows changes the model's input and output interfaces. Vocabulary extension is possible, but new token embeddings and output weights require training; swapping a tokenizer under an already trained model does not preserve behavior.

What Is a Token?

A token is a vocabulary entry identified by an integer. Depending on the tokenizer and surrounding text, one token may represent punctuation, a leading space plus a word, a word fragment, one byte, several characters, or a reserved control symbol.

Three terms must remain distinct:

1. Token string or byte sequence — the fragment associated with a vocabulary entry.
2. Token ID — the integer passed to the model, such as 42 or 100257.
3. Token embedding — the learned vector obtained by indexing the model's embedding matrix with that ID.

The same visible text can tokenize differently across models. Even within one tokenizer, a leading space or neighboring punctuation can change segmentation. Unicode normalization, invalid byte handling, chat templates, and special-token policies also matter. Therefore examples are illustrative unless generated with the exact named tokenizer version.

Tokens are not always valid standalone Unicode strings. A byte-level token may contain only part of a multibyte UTF-8 character. Decoding one token at a time can therefore produce replacement characters or fail to display meaningful text, even though decoding the complete sequence round-trips correctly.

Special tokens are vocabulary entries with protocol meaning rather than ordinary user text. Examples include beginning-of-sequence, end-of-sequence, message-boundary, padding, fill-in-the-middle, or tool-call delimiters. Applications should not concatenate reserved marker text into prompts and assume it behaves like a special token; the tokenizer API usually controls whether special tokens are accepted.

How Tokenization Works

Tokenizer training

Tokenizer training happens before language-model training. Engineers choose a corpus, normalization policy, algorithm, vocabulary size, reserved symbols, and byte fallback behavior. The tokenizer trainer then learns a fixed mapping used to encode the model's entire training set.

For a simplified BPE trainer:

1. Represent training text as initial symbols, often bytes or characters.
2. Count adjacent symbol pairs, usually weighted by corpus frequency.
3. Merge a selected frequent pair into a new symbol.
4. Recount affected pairs and repeat until the vocabulary target or stopping rule is reached.
5. Save the vocabulary, merge ranks, normalization rules, and special-token IDs.

Real implementations differ in pre-tokenization, regex splitting, tie-breaking, byte mapping, and whether merges cross whitespace. Those details are part of compatibility. Two systems both described as "BPE" can produce different IDs.

Inference encoding

At inference, no vocabulary learning occurs. The encoder applies the frozen normalization, pre-tokenization, and segmentation rules, then returns IDs. For rank-based BPE, candidate adjacent pairs are merged according to learned priority until no eligible pair remains. The decoder reverses IDs to token byte sequences and reconstructs text, subject to special-token handling and normalization choices.

Diagram: Training and inference tokenization

flowchart LR
    subgraph train [Tokenizer construction]
        C[Training corpus] --> N[Normalize]
        N --> A[Learn merges or pieces]
        A --> V[Vocabulary + rules]
    end
    subgraph infer [Frozen encoding]
        T[Request text] --> P[Apply fixed rules]
        V --> P
        P --> I[Token IDs]
        I --> E[Embedding lookup]
    end

Tokenizer training creates the vocabulary once; every training and inference request later applies that frozen mapping.

BPE at engineering level

BPE provides deterministic segmentation with a merge vocabulary. Byte-level BPE starts from a complete byte alphabet, so it can encode arbitrary text. Frequent byte sequences become compact tokens. This is useful for mixed prose, code, and noisy web data.

Important operational properties include:

• Vocabulary size versus sequence length: more learned pieces can shorten sequences but enlarge embedding and output projection matrices.
• Corpus dependence: a code-heavy or English-heavy corpus allocates merges differently from a balanced multilingual corpus.
• Boundary policy: regex pre-tokenization can prevent merges across classes such as letters, digits, and whitespace.
• Reversibility: byte-level foundations avoid unknown characters, but displayed per-token fragments may not be valid Unicode.
• Determinism: encoding should be stable for a pinned vocabulary and implementation, which makes cached token counts possible.

SentencePiece and Unigram

SentencePiece is a toolkit and data model, not one segmentation algorithm. It commonly trains either:

• SentencePiece BPE: merge-based segmentation over raw text with whitespace represented by a visible meta-symbol such as ▁.
• SentencePiece Unigram: begins with many candidate pieces, estimates a probabilistic language model over segmentations, and prunes pieces while preserving likely encodings.

Unigram can score multiple possible segmentations and supports subword regularization during training, where alternative segmentations act as noise. At normal inference, applications usually request a deterministic best segmentation. SentencePiece can also configure byte fallback so unseen characters remain representable.

Training versus inference tokenization

The same tokenizer model should encode pretraining data and inference requests, but the surrounding pipeline differs:

The model predicts token IDs, not text strings. A sampling algorithm chooses an ID from logits, the ID joins the context, and the next decode step repeats. Text streaming is a decoding presentation layer over that sequence.

Architecture

Tokenization sits in both the application's control plane and the model's data path. The application assembles structured messages and tool definitions, estimates their token footprint, enforces policy, and sends the request. The provider serializes that request with its model-specific template, tokenizes it, runs prefill and decode, then reports usage.

Diagram: Production token-budget architecture

flowchart TB
    U[User input] --> A[Prompt assembler]
    H[Conversation store] --> A
    R[Retrieved context] --> A
    S[System + tool schemas] --> A
    A --> C[Client token estimator]
    C --> G{Fits budget?}
    G -->|No| T[Trim, summarize, or reject]
    T --> C
    G -->|Yes| API[Model API]
    API --> PF[Prefill]
    PF --> D[Autoregressive decode]
    D --> O[Response + usage]
    O --> M[Cost and quality telemetry]

The budget gate evaluates the complete request, while provider usage closes the loop with authoritative post-request accounting.

Use the inequality:

estimated_input + reserved_output + safety_margin <= supported_context_window

The supported context window is model- and endpoint-specific. It may also interact with output caps. Do not derive it from a model family name or a stale configuration file; keep model capabilities in versioned configuration sourced from provider documentation.

Counting only message text is incomplete. A request can include role wrappers, names, JSON schemas, tool descriptions, response-format schemas, images or audio represented through provider-specific accounting, and internal special tokens. Exact local reproduction may be unavailable for hosted models. In that case, calibrate estimates from observed usage and use the provider's count endpoint if one exists.

Step-by-Step Flow

1. Select the model configuration. Resolve the model ID, tokenizer or count API, context limit, maximum output, and pricing categories from a centrally maintained registry.

2. Assemble the logical request. Include the system instruction, conversation history, current user input, retrieved passages, tool definitions, and structured-output schema. Token budgeting before assembly misses important overhead.

3. Apply deterministic preprocessing. Normalize only if the product requires it. Seemingly harmless whitespace or Unicode changes can alter meaning, signatures, source code, and tokenization.

4. Estimate input tokens. Use the exact local tokenizer when officially supported. Otherwise use a provider count endpoint or a conservative estimator calibrated against actual usage.

5. Reserve output and margin. The output cap is a maximum, not a prediction. Choose it from task requirements and observed completion distributions. Preserve a margin for count mismatch and provider formatting.

6. Enforce a degradation policy. Prefer relevance-aware reductions: remove duplicate retrieval chunks, cap tool descriptions, summarize old turns, or route oversized documents to a staged workflow. Blind character truncation can split Unicode, JSON, or critical instructions.

7. Send with timeout and retry policy. A retry may be billed again. Retry transient failures with idempotency controls where supported; do not automatically retry context-limit or invalid-request errors.

8. Stream and decode safely. Buffer bytes or rely on the SDK's text events. Do not assume each streamed token is a complete character, word, or JSON fragment.

9. Read authoritative usage. Capture provider-reported input, output, cached, and any model-specific categories. Field names differ by endpoint and SDK version.

10. Compute cost from versioned rates. Store the rate-card version and currency with the event. Never hard-code a blog's price into business logic.

11. Evaluate the outcome. Join token and latency telemetry to task success, answer quality, retries, and user feedback. A cheaper request that fails more often may increase total spend.

Real Production Example

The following Python pattern uses tiktoken for an OpenAI-compatible preflight estimate and the OpenAI Responses API for the request. It deliberately keeps prices in configuration and treats API usage as authoritative. Model IDs, tokenizers, SDK fields, context limits, and prices evolve; verify them against current official documentation before deployment.

from __future__ import annotations

import os
from dataclasses import dataclass
from decimal import Decimal

import tiktoken
from openai import OpenAI


@dataclass(frozen=True)
class ModelPolicy:
    model: str
    context_window: int
    max_output_tokens: int
    input_usd_per_million: Decimal
    output_usd_per_million: Decimal
    safety_margin: int = 256


POLICY = ModelPolicy(
    model=os.environ.get("OPENAI_MODEL", "gpt-4o-mini"),
    context_window=int(os.environ["MODEL_CONTEXT_WINDOW"]),
    max_output_tokens=int(os.getenv("MAX_OUTPUT_TOKENS", "800")),
    input_usd_per_million=Decimal(os.environ["INPUT_USD_PER_MILLION"]),
    output_usd_per_million=Decimal(os.environ["OUTPUT_USD_PER_MILLION"]),
)

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    timeout=30.0,
    max_retries=2,
)


def encoding_for(model: str) -> tiktoken.Encoding:
    try:
        return tiktoken.encoding_for_model(model)
    except KeyError as exc:
        raise RuntimeError(
            f"No pinned tokenizer mapping for {model}; configure one explicitly"
        ) from exc


def estimate_text_tokens(parts: list[str], model: str) -> int:
    """Estimate visible text only; request framing may add tokens."""
    enc = encoding_for(model)
    return sum(len(enc.encode(part, disallowed_special=())) for part in parts)


def estimate_cost(input_tokens: int, output_tokens: int) -> Decimal:
    million = Decimal(1_000_000)
    return (
        Decimal(input_tokens) * POLICY.input_usd_per_million / million
        + Decimal(output_tokens) * POLICY.output_usd_per_million / million
    )


def answer(system_prompt: str, user_prompt: str) -> tuple[str, dict]:
    estimated_input = estimate_text_tokens(
        [system_prompt, user_prompt], POLICY.model
    )
    required = (
        estimated_input
        + POLICY.max_output_tokens
        + POLICY.safety_margin
    )
    if required > POLICY.context_window:
        raise ValueError(
            f"Request budget {required} exceeds context "
            f"{POLICY.context_window}; reduce input or output reserve"
        )

    response = client.responses.create(
        model=POLICY.model,
        instructions=system_prompt,
        input=user_prompt,
        max_output_tokens=POLICY.max_output_tokens,
    )

    # Names reflect the Responses API at time of writing. Pin the SDK and
    # update this adapter when upgrading.
    actual_input = response.usage.input_tokens
    actual_output = response.usage.output_tokens
    event = {
        "model": POLICY.model,
        "estimated_input_tokens": estimated_input,
        "actual_input_tokens": actual_input,
        "actual_output_tokens": actual_output,
        "estimate_error_tokens": actual_input - estimated_input,
        "estimated_cost_usd": str(
            estimate_cost(actual_input, actual_output).quantize(
                Decimal("0.000001")
            )
        ),
    }
    return response.output_text, event

This estimator intentionally does not pretend that concatenated text exactly matches provider serialization. For chat messages with tools or structured output, place request construction and estimation behind one adapter per provider. Track actual_input - estimated_input; set the safety margin from a high percentile of observed positive error, not an arbitrary constant.

For multi-tenant systems, add a reservation ledger. Atomically reserve an estimated maximum before the call, then reconcile the reservation with actual usage. This prevents concurrent requests from exceeding a tenant budget. Distinguish a financial budget from a context budget: cached input may cost less but still occupies context positions.

Design Decisions

Context allocation

Do not give every component whatever remains. Define explicit envelopes, for example: system and policy, tool schemas, current turn, recent history, retrieved evidence, and output. The values should be task-specific and evaluated. Retrieval should rank chunks by value per token rather than blindly accepting a fixed number of chunks.

Truncation policy

Truncation is a product decision. Preserve system instructions and the current user intent. For chat, remove or summarize older turns while keeping referenced facts. For RAG, drop duplicate or low-ranked chunks before cutting high-ranked evidence. For code, preserve syntactic units and relevant definitions. Record which content was omitted so failures remain debuggable.

Versioning

Pin tokenizer library versions and record the logical tokenizer name. When changing models, compare token distributions on representative prompts, including supported languages, code, JSON, and tool schemas. A tokenizer migration changes cost and capacity even when answer quality remains constant.

When should I care about tokens?

Comparisons

BPE vs SentencePiece Unigram vs word and character units

SentencePiece can itself train BPE, so "BPE vs SentencePiece" is an imprecise comparison. The useful comparison specifies both algorithm and implementation: for example, byte-level rank BPE with regex pre-tokenization versus SentencePiece Unigram with normalization and byte fallback.

Tokens vs words vs characters for capacity planning

Input tokens vs output tokens

Input processing and output generation have different performance profiles. Input tokens are processed during prefill and can often exploit parallel hardware operations, although attention and memory costs still grow with sequence length. Output tokens are generated autoregressively: each next token depends on the previous sequence. Consequently, reducing an unnecessarily long answer can improve wall-clock latency more directly than removing the same number of prompt tokens. Exact behavior depends on model architecture, batching, serving stack, and caching.

Common Mistakes

1. Explaining tokenization as word splitting. Tokens may contain spaces, punctuation, partial words, bytes, or control symbols. Word-count intuition breaks on code, numbers, URLs, emoji, and many languages.

2. Using a universal characters-per-token rule. Ratios vary by tokenizer, corpus, language, and content type. A ratio is acceptable for a rough capacity sketch, never for request admission or billing.

3. Counting raw text instead of the serialized request. Roles, tool schemas, response schemas, and special tokens add usage. Images and audio may use separate accounting rules.

4. Claiming all GPT-family models use one encoding. Tokenizer mappings differ by model generation and can change for new model IDs. Resolve the exact model through the maintained tokenizer library.

5. Treating a local estimate as the invoice. The server's usage object is authoritative for that provider. Store both estimate and actual count to detect drift.

6. Assuming every token decodes to a visible character. Byte-level fragments may be incomplete Unicode. Decode complete sequences or use the tokenizer's byte-safe APIs.

7. Truncating from the beginning. This can remove system policy, task definitions, or conversation facts. Apply a content-aware policy with protected regions.

8. Filling the whole context with input. Generation needs output capacity, and local counts may be low. Reserve output plus a calibrated margin.

9. Hard-coding prices and context limits in prompt code. Both are operational configuration. Version them, test them, and update them independently of application logic.

10. Optimizing token count while ignoring quality. Removing examples may cause format errors and retries. Compare cost per accepted result and end-to-end latency.

Where It Breaks Down

Tokenization is a lossy boundary for some tasks even though text encoding can be reversible.

• Character-level reasoning: spelling, letter counts, acrostics, and exact offsets do not align with subword boundaries. Use deterministic code for exact string operations.
• Numbers and identifiers: long numbers, UUIDs, hashes, and product IDs may split into many pieces. The model can also make digit-level errors; validate with software.
• Underrepresented languages: a corpus may allocate fewer merged pieces to some scripts or languages, producing longer sequences for equivalent content. Measure token parity across your actual user population.
• Unicode edge cases: visually identical strings can have different code-point sequences. Normalization may improve consistency but can be incorrect for signatures, source code, or security-sensitive identifiers.
• Prompt injection boundaries: tokenization does not create a trust boundary. Special-looking delimiters in user text do not isolate instructions. Enforce roles, data provenance, tool permissions, and output validation.
• Tokenizer/model mismatch: IDs from a different vocabulary point to unrelated embedding rows. Results may be invalid, not merely lower quality.
• Streaming structure: a stream chunk or generated token is not guaranteed to complete a JSON value or Unicode character. Incremental parsers must buffer incomplete data.
• Opaque provider accounting: hosted systems may include tokens not reproducible with public libraries. Use provider counters and empirical margins.

Tokens also do not measure semantic information. Repeated boilerplate can consume many tokens while adding little value; a short equation or identifier can carry high value. Token budgets need relevance and task-aware selection above the tokenizer layer.

When NOT to Over-Optimize Tokenization

Do not spend engineering time shaving tokens when usage is small, prompts are stable, and no context, latency, or budget objective is being missed. Readable prompts are easier to review, secure, evaluate, and maintain.

Avoid manual abbreviations or punctuation tricks that save a few tokens but make instructions ambiguous. The model may need more output, retries, or correction turns. Likewise, do not train a custom tokenizer for an API model: the hosted model's vocabulary is fixed, so your custom IDs cannot replace its expected input.

Do not replace source evidence with aggressive summaries when auditability matters. Legal, medical, financial, and operational workflows may prefer additional tokens for exact quotations and provenance. Reduce duplicate context first.

Do not choose a model solely because its tokenizer produces fewer tokens. Providers can have different prices, caching rules, context limits, output quality, latency, and tool behavior. Compare end-to-end task economics.

Custom tokenizer training is appropriate mainly when training or substantially adapting a model whose embedding and output layers you control. Even then, changing the tokenizer complicates checkpoint reuse, multilingual behavior, data pipelines, and evaluation. Reuse the base tokenizer for ordinary fine-tuning unless there is a measured domain coverage problem and a plan to train the added rows.

Running in Production

Best Practice

✅ Best Practices — Centralize model capabilities, preflight complete requests, reserve output, and reconcile every successful call against provider usage.

Metrics

Record at least:

• estimated and actual input tokens;
• output tokens and configured output cap;
• cached and other provider-specific token categories when available;
• model, endpoint, tokenizer version, tenant, feature, and rate-card version;
• time to first token, total latency, and generated tokens per second;
• truncation action, retry count, response status, and task-quality result.

Use distributions rather than only averages. High-percentile prompt sizes cause context failures; high-percentile outputs drive tail latency. Segment by language and content type to expose inefficient tokenization or product populations receiving less effective context.

Capacity and rate limits

Use a weighted admission controller when providers expose token-per-minute quotas. Reserve estimated input plus expected or maximum output, release the difference after reconciliation, and add backoff based on rate-limit headers. Requests with huge prompts can starve short interactive traffic, so use queues or separate quota pools by workload.

Cost controls

Set tenant and feature budgets, but distinguish hard and soft limits. Hard limits prevent unbounded spend; soft alerts catch regressions before users are blocked. Detect changes in tokens per successful task after prompt, retrieval, model, or tokenizer updates. Prompt caching can reduce billed cost for eligible repeated prefixes, but cached tokens still participate in context and should remain visible in usage telemetry.

Reliability and security

Treat user-controlled length as a resource-exhaustion vector. Bound HTTP body size before tokenization, then enforce token limits after parsing. Tokenization itself consumes CPU and memory, so do not accept unlimited payloads merely because the model API will reject them later.

Validate structured outputs after complete decoding. Never execute a tool call from a partial stream. Keep secrets out of prompts regardless of token accounting, and ensure logs redact sensitive content while retaining numerical usage metadata.

Testing

Build a tokenization regression corpus containing:

• every supported language and script;
• code, JSON, Markdown tables, URLs, and long numeric identifiers;
• combining characters, emoji sequences, and unusual whitespace;
• maximum-size tool and response schemas;
• adversarial repeated input and reserved-token-looking strings.

Snapshot counts for pinned local tokenizers. On dependency or model upgrades, review count deltas rather than mechanically updating snapshots. Integration tests should verify graceful overflow handling and reconciliation with live provider usage in a controlled environment.

Diagram: Request accounting lifecycle

sequenceDiagram
    participant App
    participant Budget as Budget service
    participant API as Model API
    participant Ledger as Usage ledger
    App->>Budget: reserve estimated maximum
    Budget-->>App: admission granted
    App->>API: complete request
    API-->>App: output + usage
    App->>Ledger: actual categories + rate version
    Ledger->>Budget: reconcile reservation
    Budget-->>App: remaining tenant budget

Reservation limits concurrent exposure, while reconciliation replaces estimates with provider-reported usage.

Related Guides

• Foundations: Generative AI · Large Language Models
• Capacity and behavior: Context Windows · Prompt Engineering
• Operations: Cost Optimization
• Representation and retrieval: Embeddings

If you understood this guide, continue through the model input and application stack:

Prerequisites: Generative AI · Large Language Models

Next topics: Context Windows · Prompt Engineering · Cost Optimization · Embeddings

Estimated time: 50 min · Difficulty: Beginner

Interview Questions

1. Why do LLMs use tokens instead of words or raw characters?

   • Expected: finite numerical vocabulary, unknown-word coverage, sequence-length and vocabulary-size trade-off, subwords as the compromise.

2. What is the difference between tokenizer training and model training?

   • Expected: tokenizer training learns vocabulary and segmentation rules; model training uses frozen token IDs to learn embeddings and next-token probabilities.

3. How does byte-level BPE encode unseen text?

   • Expected: starts from a complete byte alphabet, then applies learned merges; unseen strings fall back to smaller byte pieces rather than an unknown word.

4. Why is "BPE versus SentencePiece" an incomplete comparison?

   • Expected: BPE is an algorithm family; SentencePiece is a toolkit that supports BPE and Unigram with raw-text processing and explicit whitespace handling.

5. Why can a client token count differ from API usage?

   • Expected: chat templates, roles, tools, schemas, special tokens, multimodal accounting, and provider-side implementation details.

6. How do input and output tokens affect latency differently?

   • Expected: input is processed during prefill; output is sequential autoregressive decode. Hardware, batching, architecture, and caching affect exact scaling.

7. How would you prevent context overflow in a RAG chat system?

   • Expected: count the assembled request, reserve output and margin, deduplicate and rank evidence, summarize old history, enforce a deterministic degradation policy.

8. What should be stored in a token cost ledger?

   • Expected: provider usage categories, model and endpoint, tenant and feature, pricing version, currency, latency, retry, and quality outcome.

9. When is a custom tokenizer justified?

   • Expected: primarily when training a model whose embedding/output layers are controlled, after measured domain or language inefficiency; not for a fixed hosted model.

10. Why is cost per successful task better than tokens per request?

    • Expected: aggressive compression can reduce quality and increase retries, correction turns, output length, or escalation.

Key Takeaways

• Tokens are the finite symbols that connect arbitrary text to a model's numerical embedding and output layers.
• They exist to balance vocabulary size, text coverage, and sequence length; they are not linguistic words or semantic concepts.
• Train the tokenizer before the model, then freeze and reuse it during pretraining, fine-tuning, and inference.
• Distinguish BPE algorithms from SentencePiece implementations and Unigram models.
• Couple input budgets to context limits, output reserves, latency, rate limits, and cost.
• Count with the exact supported tokenizer where possible, but use provider-reported usage for reconciliation.
• Preserve readability and task quality; optimize cost per accepted outcome rather than minimizing tokens in isolation.

FAQs

How many characters or words are in one token?

There is no universal conversion. Ratios vary with tokenizer, language, whitespace, code, numbers, and punctuation. Use illustrative ratios only for rough planning and run the exact tokenizer for admission control.

Does the context window include output tokens?

Usually the request and generated sequence share a model context constraint, though endpoint-specific limits and separate output caps can apply. Budget input plus reserved output and verify the exact model documentation.

Are tokens the same during training and inference?

The base tokenizer and IDs should be the same. Training pipelines additionally pack documents, insert boundaries, and mask losses. Inference pipelines apply chat or tool templates and enforce context budgets.

Does tokenization understand meaning?

Not by itself. It learns recurring surface pieces. The model learns contextual meaning through embeddings, attention, and training objectives applied to token sequences.

Can I decode each token independently?

Not safely for byte-level tokenizers. One token can contain an incomplete UTF-8 byte sequence. Decode the full sequence or use byte-safe incremental decoding.

Why does adding a space change the token count?

Many vocabularies include leading whitespace as part of a token, and pre-tokenization rules treat boundaries differently. "word" and " word" can therefore map to different pieces.

Can two models use the same tokenizer?

Yes, but never assume they do. Models can share a vocabulary while differing in architecture or training; related model names can also use different vocabularies. Resolve compatibility from official model configuration.

Is provider-reported usage exact?

It is the authoritative accounting value for that request and provider. It may include categories a public tokenizer cannot reproduce. Retain the raw usage fields because billing categories can evolve.

Should I remove whitespace to save tokens?

Only when the content format permits it and evaluation shows a material benefit. Whitespace can carry meaning in code, Markdown, tables, and human-readable instructions. Minifying tool schemas may be safer than minifying user content, but still test behavior.

How should I estimate the cost of a request?

Multiply each provider-reported or estimated token category by its configured per-token rate, then sum the categories. Keep rates outside code, version the rate card, and distinguish estimated preflight cost from reconciled actual cost.

Do embedding models also tokenize input?

Yes. Embedding endpoints use model-specific tokenizers and input limits. Chunking by one generation model's tokenizer may not match the embedding model's limits, so count with the embedding model configuration.

Does prompt caching reduce context usage?

No. Eligible cached prefixes may reduce billed cost and prefill work, depending on provider behavior, but they still represent input context. Do not use discounted price as a reason to exceed the context budget.

References

• Neural Machine Translation of Rare Words with Subword Units (Sennrich, Haddow, and Birch, 2016)
• SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing (Kudo and Richardson, 2018)
• Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018)
• Google SentencePiece repository and documentation
• OpenAI tiktoken repository
• OpenAI API Usage documentation

Further Reading

• Hugging Face Tokenizers documentation
• Hugging Face: Summary of tokenizers
• OpenAI Cookbook: How to count tokens with tiktoken
• SentencePiece model options
• The Unicode Standard