Latency Optimization Guide

TL;DR

• Users perceive latency at first token, not total completion — streaming cuts perceived wait from 3s to ~500ms even when full response takes longer.

• LLM generation dominates p95 latency — retrieval and reranking are hundreds of ms; generation is seconds. Parallelize everything before the LLM call.

• Smaller models and shorter context reduce TTFT — route latency-sensitive paths to GPT-5.6 Luna, Haiku 4.5, or Gemini 3.7 Flash; keep GPT-5.6 Terra / Claude Sonnet 5 for balanced quality; reserve GPT-5.6 Sol for hard work that can tolerate slower TTFT. Never a permanent single default.

• Never block the critical path on non-essential work — log traces, run eval sampling, and persist sessions asynchronously after streaming starts.

• Define per-stage latency budgets — embed 50ms, retrieve 100ms, rerank 150ms, TTFT 800ms, full response 3s — alert when any stage regresses.

On this page

• Why This Matters
• The Problem Latency Optimization Solves
• How We Got Here
• What Is LLM Latency Optimization?
• How Latency Optimization Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Chase Latency
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References

Why This Matters

A 4-second wait feels broken in chat. A 1-second wait with streaming tokens feels responsive. Latency is a product metric, not just infrastructure — drop-off rates correlate with time-to-first-token (TTFT).

Production incidents often manifest as "the bot is slow" while error rates stay at zero. Without stage-level latency visibility, teams guess: upgrade the model (wrong), add servers (useless if the provider queue is the bottleneck), or cache blindly.

Latency optimization aligns with cost optimization — shorter context and smaller models are faster and cheaper. It conflicts with quality when reranking or large context is skipped — tradeoffs must be measured. Caching and semantic caching are the largest latency wins on repeated traffic.

The Problem Latency Optimization Solves

End-to-end RAG latency stacks sequentially by default:

embed (50ms) + search (80ms) + rerank (200ms) + LLM TTFT (600ms) + generation (2000ms)
  ≈ 2.9s to first token if not streaming
  ≈ 4.9s total

Users experience the sum. Agents multiply this by step count. Synchronous logging, guardrails, and session writes on the critical path add more.

Latency optimization parallelizes independent work, streams partial results, caches hot paths, and trims work on the path to first token — without guessing which stage to cut.

How We Got Here

Early Completions APIs returned full strings. Chat UIs trained users to expect typing. Provider streaming (SSE) made TTFT the product metric. RAG then stacked embed → retrieve → rerank before generation, and agents multiplied round-trips.

Diagram: Latency became a product metric

timeline
    title From full-buffer responses to TTFT SLOs
    2020-2022 : Sync completions
              : Wait for full string
    2023 : Streaming chat UIs
              : TTFT becomes UX
    2024 : RAG + rerank stacks
              : Pre-LLM budgets matter
    2025-2026 : Degraded modes + routing
              : Skip stages under budget

Streaming fixed perceived wait; production systems still needed budgets, parallelism, and routing.

What Is LLM Latency Optimization?

LLM latency optimization is the practice of reducing time-to-first-token and total response time for AI features while meeting quality SLOs.

Techniques: streaming, parallel retrieval, async pipelines, model tier selection, caching, prefetch, edge deployment for embedders, speculative decoding (self-hosted).

How Latency Optimization Works

Critical path analysis

Map the synchronous chain from request to first byte sent to client. Everything else moves off-path. Generation usually dominates; before that, rerank and cold connections are common culprits.

Streaming

OpenAI, Anthropic, and Google support SSE streaming. Send tokens as generated; record TTFT on first delta.

async def stream_answer(llm, messages, start):
    first_token_at = None
    async for chunk in llm.stream(messages):
        if first_token_at is None:
            first_token_at = time.time()
            metrics.record("ttft_ms", (first_token_at - start) * 1000)
        yield chunk.delta

UX pattern: Show retrieved sources immediately while generation streams — masks retrieval + rerank time.

Parallelism

Run independent operations concurrently:

query_vec, history_summary = await asyncio.gather(
    embedder.embed(query),
    session.summarize_if_needed(session_id),
)
chunks = await vector_store.search(query_vec, top_k=20)

Do not parallelize rerank after retrieve — dependency exists. Parallelize embed with session load, or hybrid BM25 with vector search if the index supports it.

Model and context selection for speed

• Shorter context — fewer retrieved chunks, summarized history.
• Skip rerank for cache hits or high-confidence FAQ matches.
• Provider region — nearest API region; avoid cross-region on every request.

Speculative decoding (self-hosted)

A draft model proposes tokens; the target model verifies in batch — higher tokens/sec on your GPUs. Rare in managed API workflows; useful with vLLM / TGI when you control inference. Do not confuse with "faster API tier" marketing.

Connection reuse and warm paths

Cold connections add 50–150ms per request to vector DBs and HTTP LLM clients. Pool per process; reuse AsyncOpenAI / httpx clients. Warm-up on deploy: health check runs embed + dummy vector search + LLM ping before traffic. Readiness probes should hit this path, not just /health.

Architecture

Place time budgets in the orchestrator — if retrieval exceeds 200ms, skip rerank and proceed with top-5 vector results (degraded mode).

Diagram: Latency-optimized RAG architecture

flowchart TB
    subgraph Edge [Edge / Gateway]
        Auth[Auth + rate limit]
        Exact[Exact cache]
    end
    subgraph PreLLM [Pre-LLM parallel]
        Emb[Embed]
        Sess[Session load]
        Ret[Retrieve]
        RR[Rerank optional]
    end
    subgraph Gen [Generation]
        Route[Model route]
        Stream[SSE stream]
    end
    subgraph OffPath [Off critical path]
        Log[Async logs]
        Eval[Eval sample]
    end
    Auth --> Exact
    Exact -->|miss| Emb
    Emb --> Sess
    Emb --> Ret --> RR --> Route --> Stream
    Stream --> Log
    Stream --> Eval

Cache and parallelize before generation; move logging off the path to first token.

Perceived latency UX patterns

Step-by-Step Flow

Diagram: Budgeted stream with degraded retrieval

sequenceDiagram
    participant U as User
    participant API as API
    participant Cache as Cache
    participant Ret as Retriever
    participant LLM as LLM
    U->>API: Query
    API->>Cache: Exact lookup
    alt hit
        Cache-->>U: Replay / stream cached
    else miss
        par Parallel prep
            API->>Ret: Embed + search
            API->>API: Load session
        end
        alt within budget
            API->>Ret: Rerank
        else timeout
            API->>API: Degrade top-k
        end
        API-->>U: sources event
        API->>LLM: Stream (Luna/Terra/Sol)
        LLM-->>U: token deltas
        API->>API: Async log
    end

Emit sources before generation; degrade under budget rather than blocking forever.

1. Measure TTFT and per-stage latency in production (p50, p95, p99).
2. Enable streaming on all user-facing chat endpoints.
3. Show intermediate UI state — "Searching…", source cards, then streamed answer.
4. Parallelize embed + session fetch + cache lookup.
5. Tune retrieval — index params (ef_search), warm connections, regional replica.
6. Conditional rerank — skip when cache hit or top vector score > threshold.
7. Route latency-sensitive endpoints to Luna / Haiku / Flash; escalate only on retry.
8. Move logging, eval, session write to background after stream completes.
9. Set alerts on TTFT p95 regression >20% week-over-week.
10. Sample eval on degraded-mode traffic so quality does not silently rot.

Real Production Example

FastAPI endpoint with streaming, parallel retrieval, and degraded-mode timeout:

import asyncio
import json
import time
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()
RETRIEVAL_BUDGET_MS = 250
RERANK_BUDGET_MS = 150
# Prefer Luna/Flash for TTFT; escalate via router elsewhere.
FAST_MODEL = "gpt-5.6-luna"


class LatencyOptimizedRAG:
    def __init__(self, embedder, store, reranker, llm, cache):
        self.embedder = embedder
        self.store = store
        self.reranker = reranker
        self.llm = llm
        self.cache = cache

    async def answer_stream(self, query: str, tenant_id: str):
        t0 = time.perf_counter()

        cached = await self.cache.get_exact(query, tenant_id)
        if cached:
            for token in cached.split():
                yield f"data: {token} \n\n"
            return

        try:
            query_vec = await asyncio.wait_for(
                self.embedder.embed(query), timeout=0.08
            )
        except asyncio.TimeoutError:
            yield "data: [ERROR: embed timeout]\n\n"
            return

        retrieve_start = time.perf_counter()
        chunks = await asyncio.wait_for(
            self.store.search(query_vec, tenant_id, top_k=15),
            timeout=RETRIEVAL_BUDGET_MS / 1000,
        )

        elapsed_retrieve = (time.perf_counter() - retrieve_start) * 1000
        if elapsed_retrieve < RETRIEVAL_BUDGET_MS and len(chunks) > 5:
            try:
                chunks = await asyncio.wait_for(
                    self.reranker.rerank(query, chunks, top_n=5),
                    timeout=RERANK_BUDGET_MS / 1000,
                )
            except asyncio.TimeoutError:
                chunks = chunks[:5]  # degraded: vector order only

        sources = [c.metadata.get("title", c.id) for c in chunks[:5]]
        yield f"event: sources\ndata: {json.dumps(sources)}\n\n"

        messages = self._build_messages(query, chunks)
        ttft_recorded = False
        async for event in self.llm.stream(model=FAST_MODEL, messages=messages):
            if not ttft_recorded:
                metrics.record("ttft_ms", (time.perf_counter() - t0) * 1000)
                ttft_recorded = True
            if event.delta:
                yield f"data: {event.delta}\n\n"

        asyncio.create_task(self._async_log(query, tenant_id, chunks))


@app.post("/chat/stream")
async def chat_stream(body: ChatRequest):
    async def gen():
        async for chunk in rag.answer_stream(body.query, body.tenant_id):
            yield chunk
    return StreamingResponse(gen(), media_type="text/event-stream")

Streaming masks generation tail latency. Sources event masks rerank wait. Timeouts prevent one slow stage from blocking the fleet.

Design Decisions

Comparisons

Latency vs cost vs quality

Diagram: Trade-off decision tree

flowchart TD
    Need[Latency SLO tight?] -->|No| Quality[Prefer Terra/Sonnet or Sol]
    Need -->|Yes| Cache{Repeat traffic?}
    Cache -->|Yes| Hit[Exact + semantic cache]
    Cache -->|No| Simple{Simple intent?}
    Simple -->|Yes| Fast[Luna / Haiku / Flash]
    Simple -->|No| Bal[Terra / Sonnet + stream]
    Fast --> Eval[Eval sample weekly]
    Bal --> Eval
    Quality --> Eval

Chase TTFT with cache and volume models; keep quality routes explicit and measured.

Common Mistakes

1. Waiting for full LLM response before sending bytes. Users stare at spinner for 3s+.
2. Sequential embed → search → rerank without timeouts. One slow dependency blows p99.
3. Synchronous trace export before response. Export async.
4. Oversized context for speed-critical paths. Every extra 1K tokens adds TTFT.
5. Ignoring provider cold starts. Warm with health pings.
6. No connection pooling to vector DB. TLS handshake per request adds 50–100ms.
7. Running heavy guardrails before first token. Lightweight input checks only; full output rails on buffer or post-stream.
8. Optimizing total HTTP duration while TTFT stays at 3s. Users feel TTFT.

Where It Breaks Down

Streaming complicates output guardrails — buffer for JSON validation or validate incrementally.

Parallelism race conditions — session summary and user message ordering; use version tokens.

Global provider latency — your code is fast; the API queue is not. Multi-provider fallback adds complexity.

Agents — multi-step loops inherit sum of latencies; hard to stream meaningfully mid-plan. Cap steps; stream partial reasoning if product allows.

Mobile/slow clients — backpressure on SSE; cancel in-flight LLM calls on client abort.

Degraded mode without eval — skipping rerank forever becomes the new normal and quality drifts.

When NOT to Chase Latency

Do not cut stages aggressively when:

• Compliance requires full retrieval + verification before any token — legal/medical attestations.
• Structured JSON must be valid before delivery — buffer; do not stream invalid partials to machines.
• Quality SLO is stricter than latency SLO — prefer Terra/Sonnet or Sol with honest wait UI.
• You lack stage metrics — you will cut the wrong stage.
• Traffic is already cache-dominated — invest in invalidation, not micro-optimizing miss path.
• Speculative decoding without GPU ops maturity — managed APIs are simpler until self-host is justified.

Warning

Latency cuts that skip rerank or shrink context may hurt faithfulness. Sample eval on degraded paths.

Running in Production

Best Practice

Instrument TTFT and per-stage histograms. Stream by default. Pool connections. Eval degraded modes.

Production checklist

• Streaming on all chat surfaces; TTFT metric on first token
• Per-stage budgets with degraded fallback
• Connection pools + warm readiness probe
• Workload routing for speed (Luna/Flash) vs quality (Terra/Sonnet/Sol)
• Exact + semantic cache ahead of embed
• Async logging / eval / session write
• Client disconnect cancels in-flight generation
• Weekly eval sample on degraded traffic

Related Guides

Efficiency cluster:

• Cost Optimization — smaller models reduce TTFT and cost together
• Caching — sub-ms hits bypass retrieval and LLM
• Semantic Caching — paraphrase FAQ hits
• AI System Architecture — orchestrator owns budgets

Adjacent:

• RAG — retrieval and rerank dominate pre-LLM latency
• Observability — per-span latency for tuning
• Evaluation — quality on degraded paths

Tools: OpenAI · Claude · LangChain

Diagram: Latency learning path

flowchart LR
    CO[Cost opt] --> C[Caching]
    C --> L[Latency opt]
    L --> SC[Semantic cache]
    L --> Obs[Observability]

Cost and cache levers unlock latency; observe stages so you cut the right work.

Interview Questions

1. What is TTFT and why does it matter?
   Time to first streamed token — primary driver of perceived chat responsiveness.

2. Why is streaming important?
   Users start reading while the model generates; perceived wait drops to roughly pre-LLM work + first token.

3. Which stage is usually slowest?
   LLM generation. Before that, rerank often adds 100–300ms; retrieval 50–150ms with tuned indexes.

4. How do you parallelize RAG safely?
   Concurrent cache/embed/session; rerank only after retrieve; use timeouts and degraded modes.

5. When should you skip reranking?
   Cache hits, high top-1 vector score, or latency SLO breach with measured quality impact.

6. What is speculative decoding?
   Draft model proposes tokens; target verifies in batch — self-hosted inference speedup, not a managed-API default.

7. How do agents affect latency?
   Each step adds a full round-trip. Cap steps; parallelize independent tools; stream final synthesis.

8. Latency vs cost vs quality — how do you choose?
   Route by workload: Luna/Flash for TTFT, Terra/Sonnet balanced, Sol when quality dominates; eval every cut.

Key Takeaways

• Optimize TTFT with streaming — users feel first token, not total generation time.
• Parallelize embed, cache, and session work; set per-stage timeouts and degraded modes.
• Route volume traffic to Luna/Haiku/Flash; escalate to Terra/Sonnet/Sol when needed.
• Move logging, eval, and persistence off the critical path.
• Instrument per-stage latency and alert on TTFT regressions, not just HTTP duration.

FAQs

What is time-to-first-token (TTFT)?

Milliseconds from user request to first streamed token received. Primary driver of perceived chat responsiveness.

Why is streaming important for latency?

Users start reading while the model generates. Perceived wait drops to roughly embed + retrieve + rerank + first token — not full completion time.

Which pipeline stage is usually slowest?

LLM generation (TTFT + token generation). Before that, reranking often adds 100–300ms.

How do I parallelize RAG?

Concurrent: cache lookup + embed start; session load parallel with embed. Rerank after retrieve completes.

When should I skip reranking?

Cache hits, high top-1 vector score, or latency SLO breach with degraded-mode policy. Measure quality impact.

Does a smaller model always reduce latency?

Generally yes for TTFT and tokens/sec. Exception: overloaded mini tier vs idle frontier — measure empirically.

How do I handle a slow vector DB?

Connection pooling, regional replicas, ANN tuning, timeout with degraded top-k, cache hot queries.

What is speculative decoding?

Draft model proposes tokens; target model verifies in batch. Rare in managed API workflows today.

Should guardrails block streaming?

Run fast input rails pre-LLM. Output rails: stream prose after lightweight scan, or buffer JSON until validated.

How do agents affect latency?

Each step adds a full round-trip. Cap steps; parallelize independent tool calls; stream final synthesis only.

How do I cancel in-flight LLM on client disconnect?

Propagate client abort to close SSE and cancel provider generation if available.

What latency SLO is reasonable?

Consumer chat: TTFT p95 <1.5s, total <4s. Internal tools: 2–5s may be acceptable. Define per surface.

How does regional routing affect latency?

Call LLM and vector DB in the same region as API pods. Cross-region adds 100–300ms RTT per hop.

Should I use edge functions for embed?

Only if the embed model runs edge-side (small ONNX). API-based embed still round-trips to cloud.

How do I profile latency in staging?

Replay production payloads with stage timers. Load test at 2× peak with concurrent streams.

How do I reduce rerank latency without skipping it?

Smaller cross-encoders, GPU batching, retrieve top-15 not top-50 before rerank. Validate on your corpus.

What about CDN caching for AI APIs?

Avoid CDN-caching authenticated LLM responses unless keys include auth scope and TTL is short. Prefer Redis origin cache for invalidation control.

Should every latency-sensitive path use GPT-5.6 Sol?

No. Sol is slower and costlier. Prefer Luna/Flash for TTFT-critical volume; escalate when quality fails.

References

• OpenAI API Documentation — Streaming
• Anthropic Documentation — Streaming
• Google AI for Developers
• LangChain Documentation
• vLLM Documentation
• OpenTelemetry Documentation

Further Reading

• Hugging Face Text Generation Inference
• LlamaIndex Documentation
• Qdrant Performance Tuning