Cost Optimization Guide

TL;DR

• Generation tokens dominate cost — input tokens (prompt + RAG context) and output tokens (completion) bill per request; long contexts and verbose models multiply spend fast. See Tokens.

• Model routing sends easy queries to cheap models — classify intent/complexity, route FAQs to GPT-5.6 Luna, Claude Haiku 4.5, or Gemini 3.7 Flash; reserve GPT-5.6 Sol / Claude Sonnet 5 for hard reasoning; use GPT-5.6 Terra as a balanced middle tier. Never permanently default to one model for all traffic.

• Caching layers (prompt, response, embedding) eliminate redundant compute — see Caching and Semantic Caching.

• Measure cost per tenant, endpoint, and model — you cannot optimize what you do not attribute; log tokens on every span.

• Quality-aware optimization — route down only when eval confirms cheaper models meet SLO; never save 80% cost and lose 40% accuracy silently.

On this page

• Why This Matters
• The Problem Cost Optimization Solves
• How We Got Here
• What Is LLM Cost Optimization?
• How Cost Optimization Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Optimize Aggressively
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References

Why This Matters

A prototype costing $20/day becomes $15,000/month at production traffic without architectural changes. Finance notices. A single agent loop burning 50K tokens on a vague question drains budget. Duplicate RAG context sent on every turn doubles input cost.

Cost optimization is not about using the cheapest model everywhere. It is about spending tokens where they create value and eliminating waste: redundant context, wrong model tier, uncached repeated queries, unbounded agent iterations, and missing max_tokens limits.

Teams that instrument cost early ship sustainable products. Teams that do not get shut down or forced into abrupt model downgrades that break quality. The same levers that cut cost — shorter context, smaller models on easy routes — often improve latency. Quality is the constraint that keeps routing honest.

The Problem Cost Optimization Solves

LLM pricing is usage-based and opaque at scale:

Cost optimization provides routing, caching, compression, and budgets so spend scales sub-linearly with traffic while quality stays within eval thresholds. Without it, every feature launch is a blank check to the provider invoice.

How We Got Here

LLM apps went from fixed SaaS seats to metered tokens almost overnight. Early GPT-3 Completions pricing made every demo expensive; ChatGPT-era APIs multiplied call volume. Teams discovered that context size and model tier dominate bills more than request count.

Diagram: How LLM cost engineering matured

timeline
    title From fixed demos to FinOps for tokens
    2020-2022 : Completions APIs
              : Pay per token, little caching
    2023 : Chat + RAG at scale
              : Context bloat hits invoices
    2024 : Prompt cache + batch APIs
              : Provider-side discounts
    2025-2026 : Workload routing
              : Sol / Terra / Luna + eval gates

Capability shipped first; FinOps for tokens followed once usage-based bills hit finance.

Provider price sheets change. Always verify current rates in official docs before budgeting — treat numbers in this guide as illustrative, not invoices.

What Is LLM Cost Optimization?

LLM cost optimization is the engineering discipline of minimizing API and infrastructure spend for AI features while meeting latency and quality SLOs.

Techniques:

1. Model routing — Select model by task complexity, confidence, or user tier.
2. Prompt optimization — Shorter system prompts, dynamic context, remove redundant few-shot examples.
3. Caching — Prompt cache (provider), response cache (exact), semantic cache (similar queries).
4. Batching — Async batch API for non-real-time workloads (summaries, indexing labels).
5. Token budgets — max_tokens, context truncation, conversation summarization.
6. Architecture — Cheaper retrieval/rerank; expensive generation only on filtered context.
7. Self-hosting — Open models for high-volume simple tasks (see Llama Models).

How Cost Optimization Works

Cost formula (per request)

cost = (input_tokens × price_in + output_tokens × price_out) / 1e6
     + embedding_cost + rerank_cost + tool_side_effects

Track rolling averages and p95 — outliers (agents) dominate bills. Input tokens from fat RAG context often exceed output cost; optimizing only the completion model misses the largest line item.

Model routing by workload

Route by workload class, never a permanent single default:

Escalation pattern: Start cheap → if confidence low or user retries → escalate. Log escalation rate for tuning. Pin model IDs via env; verify current provider IDs and rates before ship.

Router inputs: query length, embedding similarity to FAQ cluster, classifier label, user tier, retrieval confidence score.

Router outputs: model_id, max_tokens, use_tools: bool, rag_top_k.

Provider-specific levers

• OpenAI: Prompt caching on static prefixes; Batch API discounts; route Luna vs Terra vs Sol by workload (verify current IDs and rates).
• Anthropic: Prompt caching; Haiku 4.5 / Sonnet 5 / Opus tier routing.
• Google: Gemini 3.7 Flash vs Pro; context caching on Vertex.

Structure prompts: static system prompt first (cacheable), dynamic user/RAG content last. See Prompt Engineering.

Token budget patterns

Implement budgets in the orchestrator, not in prompts. Log truncated_tokens and budget_exceeded events for tuning.

Architecture

Model routing belongs in the orchestration layer (AI System Architecture), not scattered in handlers.

Diagram: Cost-aware request path

flowchart TB
    subgraph Gateway [Gateway]
        RL[Rate limits + quotas]
    end
    subgraph Router [Router]
        Exact[Exact cache lookup]
        Sem[Semantic cache]
        Class[Complexity classifier]
        Pick[Pick model tier]
    end
    subgraph Pipeline [Pipeline]
        Ret[Retrieval top_k]
        Gen[Generate with max_tokens]
        Esc[Escalate if low confidence]
    end
    subgraph FinOps [FinOps]
        Span[Token spans]
        Dash[Cost dashboards]
    end
    Gateway --> Router
    Exact -->|hit| Out[Return]
    Sem -->|hit| Out
    Class --> Pick --> Ret --> Gen
    Gen --> Esc
    Esc --> Span --> Dash

Cache and classify before you spend; attribute every dollar after.

Hybrid and self-hosted routing

Use a gateway (LiteLLM) with fallback chain: primary self-hosted → secondary API provider. Log which leg served each request for cost reconciliation.

Step-by-Step Flow

Diagram: Cost-aware completion with escalation

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant R as Router
    participant Cache as Cache
    participant LLM as LLM API
    C->>G: Query + tenant
    G->>R: Authenticated request
    R->>Cache: Exact / semantic lookup
    alt cache hit
        Cache-->>C: Cached answer
    else miss
        R->>R: Classify + pick tier
        R->>LLM: Complete (Luna/Terra/Sol)
        alt low confidence
            R->>LLM: Escalate to higher tier
        end
        LLM-->>R: Answer + usage
        R->>Cache: Store if eligible
        R-->>C: Answer
    end

Lookup first; escalate only when cheap paths fail confidence or validation.

1. Instrument tokens on every LLM call. Tags: tenant, endpoint, model, trace_id.
2. Build cost dashboard. Daily spend, cost/request, top tenants, model mix.
3. Identify top 3 cost drivers from data — usually context size, model tier, agent loops.
4. Implement response cache for deterministic FAQ paths (Caching).
5. Deploy router with cheap default + escalation on low confidence — Luna/Flash first, Terra/Sonnet balanced, Sol for complex.
6. Compress prompts — shorten system prompt, summarize old conversation turns.
7. Enable provider prompt caching — verify cache hit metrics; keep static prefix identical.
8. Set budgets and alerts — per-tenant daily cap, kill switch for runaway agents.
9. Run eval on routed traffic — ensure quality within threshold before increasing cheap-model percentage.
10. Re-verify provider prices quarterly — rate cards move; update pricing tables in config, not hardcode forever.

Real Production Example

Model router with cache, escalation, and token budgets. Model IDs are illustrative — override via env and pin current production snapshots.

from dataclasses import dataclass
from enum import Enum
import hashlib
import os

VOLUME_MODEL = os.environ.get("OPENAI_VOLUME_MODEL", "gpt-5.6-luna")
BALANCED_MODEL = os.environ.get("OPENAI_MODEL", "gpt-5.6-terra")
COMPLEX_MODEL = os.environ.get("OPENAI_COMPLEX_MODEL", "gpt-5.6-sol")


class ModelTier(Enum):
    MINI = VOLUME_MODEL       # Luna / Haiku 4.5 / Gemini 3.7 Flash
    STANDARD = BALANCED_MODEL # Terra / Sonnet 5
    FRONTIER = COMPLEX_MODEL  # Sol for hard reasoning


@dataclass
class RouteDecision:
    model: ModelTier
    max_tokens: int
    use_rag: bool
    rag_top_k: int
    reason: str


class CostAwareRouter:
    def __init__(self, classifier, cache, llm_client, embedder):
        self.classifier = classifier
        self.cache = cache
        self.llm = llm_client
        self.embedder = embedder

    def route(self, query: str, tenant_id: str, history_len: int) -> RouteDecision:
        cache_key = hashlib.sha256(
            f"{tenant_id}:{query.lower().strip()}".encode()
        ).hexdigest()
        if self.cache.get(cache_key):
            return RouteDecision(ModelTier.MINI, 0, False, 0, "cache_hit")

        if len(query) < 40 and history_len == 0:
            sim = self._faq_similarity(query)
            if sim > 0.92:
                return RouteDecision(ModelTier.MINI, 256, True, 3, "faq_match")

        label = self.classifier.predict(query)  # simple | complex | extraction
        if label == "simple":
            return RouteDecision(ModelTier.MINI, 512, True, 5, "classifier_simple")
        if label == "complex":
            return RouteDecision(ModelTier.STANDARD, 2048, True, 8, "classifier_complex")
        return RouteDecision(ModelTier.MINI, 1024, True, 5, "default_volume")

    async def complete_with_escalation(self, query: str, ctx, rag_pipeline):
        decision = self.route(query, ctx.tenant_id, ctx.history_len)
        if decision.reason == "cache_hit":
            return self.cache.get_response(query, ctx.tenant_id)

        result = await self._run_pipeline(query, ctx, decision, rag_pipeline)

        if result.confidence < 0.7 and decision.model == ModelTier.MINI:
            escalated = RouteDecision(
                ModelTier.STANDARD, 2048, True, 10, "low_confidence_escalation"
            )
            result = await self._run_pipeline(query, ctx, escalated, rag_pipeline)
            metrics.increment("model_escalation", tags={"from": decision.model.value})

        self._log_cost(ctx, result.usage, decision.model)
        return result

    def _log_cost(self, ctx, usage, model):
        # Illustrative $/1M — load from config / provider pricing API.
        prices = {
            VOLUME_MODEL: (1.0, 6.0),
            BALANCED_MODEL: (2.5, 15.0),
            COMPLEX_MODEL: (5.0, 30.0),
        }
        pin, pout = prices.get(model.value, (1, 1))
        cost_usd = (usage.prompt_tokens * pin + usage.completion_tokens * pout) / 1e6
        metrics.increment(
            "llm_cost_usd",
            cost_usd,
            tags={"tenant_id": ctx.tenant_id, "model": model.value},
        )

Combine with OpenAI Batch API for nightly doc summarization at ~50% discount — separate path from interactive router.

Design Decisions

Common patterns

• Cheap-first + escalate — Luna/Flash → Terra/Sonnet → Sol only when needed.
• Sensitive-intent blocklist — refunds, medical, legal force STANDARD or FRONTIER.
• Offline batch lane — never share rate limits with interactive chat.
• FinOps export — daily cost by tenant to warehouse with $ = f(tokens, model).

Comparisons

Diagram: When to spend more tokens

flowchart TD
    Q[Incoming query] --> Sens{Sensitive intent?}
    Sens -->|Yes| Sol[Sol / Sonnet 5]
    Sens -->|No| Cache{Cache hit?}
    Cache -->|Yes| Hit[Return cached]
    Cache -->|No| Hard{Complex / agent?}
    Hard -->|Yes| Sol
    Hard -->|No| Easy{FAQ / short?}
    Easy -->|Yes| Luna[Luna / Haiku / Flash]
    Easy -->|No| Terra[Terra / Sonnet 5]

Route by risk and complexity — not by habit or a single permanent default.

Common Mistakes

1. Optimizing output model only. Input tokens from fat RAG context often exceed output cost.
2. Routing without eval. Cheap model silently fails on edge cases until customers complain.
3. No max_tokens. Agent generates 4K tokens for a yes/no question.
4. Ignoring prompt cache layout. Dynamic timestamp in system prompt breaks cache every request.
5. Caching personalized responses under shared keys — data leak + wrong answers.
6. Unbounded agent steps. Cap at 5–10; each step is a full LLM call.
7. No tenant quotas. One customer runs load test; everyone pays.
8. Hardcoding last quarter's prices. Provider rate cards change — verify before forecasting.
9. Permanent single default. "We always use Sol" wastes money; "we always use Luna" wastes quality.

Where It Breaks Down

Classifier misroutes high-stakes queries to cheap models — refunds, legal, medical. Use blocklists forcing frontier tier for sensitive intents.

Escalation doubles cost on hard queries — still cheaper than always using frontier, but monitor escalation rate. If escalation exceeds ~30%, fix the classifier or raise the cheap default quality.

Prompt compression loses nuance — aggressive summarization drops constraints. Eval faithfulness after compression.

Multi-model ops complexity — different APIs, rate limits, failure modes. Use a gateway (LiteLLM) for unified interface and fallbacks.

FinOps without eng partnership — finance cuts models arbitrarily; quality collapses. Shared dashboards and eval gates prevent that.

Batch API misuse — putting interactive chat on batch destroys UX; keep lanes separate.

When NOT to Optimize Aggressively

Do not push cost cuts hard when:

• You have no golden set or online quality metrics — you will route into silent failure.
• Traffic is low and support cost > API cost — engineer time on routing may not pay back yet; instrument first.
• Every query is high-stakes and unique — medical triage, legal drafting; prefer quality + verification over mini defaults.
• You are mid-incident or mid-migration — freeze routing experiments; stabilize first.
• Personalization dominates — per-user answers break shared caches; routing still helps, aggressive caching does not.
• You cannot verify provider prices — do not commit finance forecasts to outdated blog numbers.

Warning

Cost saved on paper means nothing if support ticket volume doubles. Eval-gate every tier change.

Running in Production

Best Practice

Instrument tokens and $ per tenant before optimizing. Version prompt and index in cache keys. Eval-gate every route-down.

Production checklist

• Token + cost spans on every LLM/embed/rerank call
• Workload router (Luna/Flash ↔ Terra/Sonnet ↔ Sol) with no permanent single default
• Exact cache + prompt-cache-friendly prompt layout
• max_tokens and agent step caps per route
• Per-tenant daily budget + anomaly alert (2× 7-day MA)
• Eval gate before increasing cheap-model percentage
• Pricing table loaded from config / provider docs (re-verified)

FinOps integration

1. Export daily cost by tenant to warehouse from span metrics.
2. Anomaly alerts page (not email) at 2× moving average.
3. Soft quota → 429 with retry-after; hard cap requires override.
4. Chargeback reports by feature flag, not aggregate invoice.

Related Guides

Foundations:

• Tokens — unit of billing and optimization
• Large Language Models — model tiers and capabilities
• Prompt Engineering — shorter effective prompts

Efficiency cluster:

• Caching — exact-match response and embedding cache
• Semantic Caching — paraphrase-tolerant hits
• Latency Optimization — often aligns (smaller model, less context)
• AI System Architecture — router placement in orchestration

Quality guardrails:

• Evaluation — gate route-downs
• Observability — token and latency spans

Tools: LangChain · OpenAI · Claude · Gemini

Diagram: Efficiency learning path

flowchart LR
    T[Tokens] --> CO[Cost opt]
    CO --> C[Caching]
    C --> SC[Semantic cache]
    CO --> L[Latency opt]
    SC --> L

Instrument tokens, cut waste, then stack exact and semantic cache; latency usually improves with the same levers.

Interview Questions

1. How do you reduce LLM cost without killing quality?
   Instrument spend, eliminate wasteful context, route by workload with eval gates, layer caches, and cap agent loops — never a permanent cheap default.

2. What is model routing?
   Selecting which model handles a request by complexity, intent, tier, or confidence — e.g. Luna/Flash for FAQ, Terra/Sonnet for balanced, Sol for hard reasoning.

3. Why do input tokens matter more than people expect?
   RAG context and system prompts often dominate billed tokens vs short completions.

4. How does provider prompt caching work?
   Identical byte prefixes (static system content first) get discounted; dynamic timestamps in the system prompt destroy hits.

5. When should you use a Batch API?
   Offline/async jobs only — summarization, labeling, bulk eval — not interactive chat.

6. How do you attribute cost to customers?
   Tag every span with tenant_id; aggregate tokens × current price daily for chargeback.

7. What is escalation routing?
   Start cheap; retry with a higher tier on low confidence or validation failure. Pays twice on hard queries, saves on easy ones.

8. How do agent loops affect cost?
   Each step is a full LLM call. Cap steps, disable tools on simple routes, and reserve Sol for final synthesis when needed.

Key Takeaways

• Instrument tokens and cost per tenant before optimizing.
• Route by workload (Luna/Flash ↔ Terra/Sonnet ↔ Sol) — never a permanent single default.
• Layer caching (prompt, exact response, semantic) and batch offline work.
• Set max_tokens, cap agent loops, compress conversation history.
• Optimize with quality metrics — cost reduction that breaks answers is not savings.
• Re-verify provider prices; treat blog numbers as illustrative only.

FAQs

What is model routing?

Selecting which LLM handles a request based on complexity, intent, user tier, or confidence — routing simple tasks to cheaper models (Luna, Haiku 4.5, Gemini 3.7 Flash) and hard work to Sol / Sonnet 5.

How much can model routing save?

Often 40–70% vs a single frontier model for mixed workloads, depending on query distribution. Measure on your traffic.

What is prompt caching?

Provider feature discounting repeated input prefixes (system prompt, static docs). Requires identical byte prefix across requests.

Should I use the Batch API?

Yes for offline/async jobs. Not for interactive chat — latency is hours; cost is ~50% lower.

How do I set max_tokens?

Set from expected output: 256 for FAQ, 1024 for explanations, 4096 for code. Add 20% buffer, not 10× buffer.

How do I attribute cost to customers?

Tag every LLM span with tenant_id; aggregate tokens × price daily. Expose in admin billing or internal chargeback.

When does self-hosting pay off?

Rough heuristic: sustained >1–5M simple queries/month at mini-model quality — GPU amortization beats API. Factor eng ops cost.

Does RAG always increase cost?

Yes — embedding + retrieval + large input context. Optimize with smaller top_k, rerank to fewer chunks, compress context, cache frequent queries.

What is escalation routing?

Start with a cheap model; if confidence is low or validation fails, retry with an expensive model.

How do agent loops affect cost?

Each step = LLM call + tools. Cap steps; use cheap models for planning; expensive only for final answer when needed.

Can semantic caching reduce cost?

Yes — 30–60% hit rates common on FAQ-heavy apps. See Semantic Caching.

How do I prevent denial-of-wallet?

Rate limits, per-tenant budgets, max agent steps, max input length, alert on cost anomaly, require auth for expensive endpoints.

How do I optimize embedding costs?

Batch embed during indexing. Cache embeddings by content hash. Prefer smaller embedding models unless eval shows recall loss.

What is the cost impact of reranking?

Rerank usually worth it for quality. Skip on cache hits and simple routes to save 100% of rerank cost on those paths.

Should I compress RAG context with an LLM?

Summarizing retrieved chunks costs an extra call but can cut input tokens 50%+. Worth it at 10+ long chunks; measure faithfulness after.

Should every request use GPT-5.6 Sol?

No. Sol is for complex workloads. Default volume traffic to Luna/Flash; balanced to Terra/Sonnet; escalate to Sol when needed.

References

• OpenAI API Documentation — Pricing
• OpenAI — Prompt Caching
• OpenAI — Batch API
• Anthropic Documentation — Prompt Caching
• Google AI for Developers
• LangChain Documentation
• LiteLLM Documentation

Further Reading

• Hugging Face Transformers Documentation
• LlamaIndex Documentation
• Helicone — LLM Observability