AI Observability Guide

TL;DR

• AI observability is tracing non-deterministic pipelines. A request can return HTTP 200 with a hallucinated answer, wrong retrieval, an unauthorized tool call, or 10× the normal cost — and classic APM dashboards stay green. You need to see prompts, retrieved context, tool calls, tokens, and per-stage latency for every request.

• The unit of debugging is the trace: one trace per user request, with child spans for embedding, retrieval, reranking, each LLM completion, each tool invocation, and guardrail checks — carrying model IDs, prompt versions, token counts, and scores as attributes.

• Metrics extend the classic three pillars: alongside QPS and error rate you track tokens per request, cost per tenant, cache hit rate, retrieval zero-result rate, and quality proxies (thumbs-down, refusal rate, guardrail blocks).

• Observability is not evaluation. Observability records what happened; evaluation measures whether it was good. Traces feed eval datasets; eval scores come back as monitored metrics. You need both.

• Instrument before you need it, and redact before you export. Adding tracing after an incident means no baseline for "what changed" — and prompts routinely contain PII that must never land raw in a third-party trace store.

On this page

• Why This Matters
• The Problem Observability Solves
• How We Got Here
• What Is AI Observability?
• How AI Observability Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Rely on Observability Alone
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

A user reports: "the bot lied about our refund policy on Tuesday." To fix it, you need the exact prompt that was sent, which chunks were retrieved and their scores, which model and version generated the answer, whether a tool returned stale data, and whether Tuesday morning's deploy changed the reranker. If you logged only the final answer and an HTTP status code, that incident is unfixable — you will re-tune prompts by guesswork and hope.

Traditional APM answers "is the service up and fast?" That is necessary but wildly insufficient for LLM systems, because the failure modes that matter are semantic: the service was up, fast, and wrong. Quality degradation produces no exceptions. Cost explosions produce no 500s. A retrieval index that silently went stale produces perfectly healthy latency histograms.

Observability designed for AI closes three gaps at once. It makes incidents debuggable (reconstruct any request end to end), makes money legible (attribute tokens and cost to tenants, features, and pipeline stages), and closes the quality loop — sampled traces become evaluation datasets, and eval scores become metrics you alert on. This is the difference between a demo that mostly works and an operated service with SLOs. It is also load-bearing for cost optimization and latency optimization: you cannot optimize what you cannot attribute.

The Problem Observability Solves

The core problem: LLM applications fail in ways classic monitoring cannot see, and the evidence needed to debug them exists only at request time.

Silent quality degradation. A chunking change drops retrieval recall. No errors, no latency shift — just wrong answers, discovered weeks later through support tickets. Without per-request retrieval context in traces, the regression cannot even be dated.

Non-reproducible bugs. The same question yields different answers across runs — temperature, provider-side model updates, cache hits, or a prompt template that interpolates changing state. Reproduction requires full context capture: the exact prompt, model ID, parameters, and retrieved chunks, not the final string.

Cost explosions without error signals. An agent loop runs 14 tool calls instead of 2. A tenant starts pasting 80K-token documents. A retry storm doubles token spend. All invisible to error-rate dashboards; all obvious in span-level token metrics.

Cross-service blindness. A request flows gateway → orchestrator → embedder → vector DB → reranker → LLM provider → guardrails. Without distributed tracing, you know the request took 6 seconds — not that 4.5 of them were the reranker cold-starting.

No audit trail. When compliance asks "what data did the model see for this decision, and which tools did it invoke?", the answer must come from a trace, not from an engineer's recollection.

How We Got Here

LLM observability recapitulated the microservices observability story, compressed into about four years and complicated by payloads that are natural-language text rather than structured RPCs.

Diagram: How AI observability practice evolved

timeline
    title From print statements to LLM-native tracing
    2020-2022 : Logging era
              : Print prompts to stdout, grep later
    2022-2023 : First LLM apps at scale
              : APM dashboards green while answers wrong
    2023-2024 : LLM-native tools emerge
              : LangSmith, Langfuse, Phoenix, Helicone
    2024-2025 : Standards converge
              : OpenTelemetry GenAI semantic conventions
    2025-2026 : Quality loop closes
              : Traces feed evals, eval scores become alerts

Each stage was driven by an incident class the previous stage couldn't explain: wrong answers, then cost blowups, then multi-agent traces nobody could read.

Three lessons from that history shape current practice:

1. General-purpose APM wasn't enough, and wasn't wrong either. Teams first tried forcing LLM calls into standard OpenTelemetry spans — it worked for latency but lost prompts, completions, and token semantics. LLM-native platforms (the LangSmith / Braintrust / Langfuse / Helicone / Arize Phoenix class) added payload capture, prompt-version diffing, and dataset export. The eventual synthesis: OpenTelemetry-style transport and propagation, with GenAI semantic conventions for the LLM-specific attributes.

2. Payload capture forced a privacy reckoning. Prompts contain user PII by construction. Early teams shipped full prompts to third-party trace stores and discovered the GDPR/SOC2 implications later. Redaction-before-export and sampled payload capture became standard.

3. Observability and evaluation converged into a loop. Tracing vendors added eval features; eval vendors added tracing. The stable pattern: traces are the raw material, evals are the judgment, and each feeds the other.

What Is AI Observability?

AI observability is the practice of instrumenting LLM applications so operators can understand production behavior: debug individual failures, attribute cost and latency, detect quality regressions, and audit decisions. It extends the three classical pillars with LLM-specific semantics:

The trace model

A trace represents one user request. Spans are units of work within it, nested to reflect the pipeline:

trace: support_query_8f3a
├── span: api.gateway (12ms)
├── span: orchestrator.route (45ms)
│   ├── span: retrieval.embed_query (38ms)
│   ├── span: retrieval.vector_search (62ms)      k=20, zero_results=false
│   ├── span: retrieval.rerank (180ms)            top_score=0.83
│   ├── span: llm.generate (2100ms)               model=sonnet-5, in=1847 out=312
│   │   └── span: tool.order_lookup (240ms)       status=ok
│   └── span: guardrails.validate (95ms)          verdict=pass
└── span: session.persist (8ms)

Each span carries start/end time, status, and attributes — model ID, prompt version, temperature, tenant ID, token counts, retrieval scores — plus, on sampled requests, inputs and outputs with PII redacted.

Three span kinds do most of the debugging work:

• Prompt/completion spans: the exact rendered prompt (or its hash + version), model and parameters, completion, token usage, finish reason.
• Tool spans: tool name, arguments (redacted), result status, duration — the difference between "the model was wrong" and "the model was fed stale data."
• Retrieval spans: query, top-k, chunk IDs and scores, zero-result flag — the first place to look for any wrong RAG answer.

The feedback loop

Observability's fourth element, beyond the pillars, is the loop back to quality: user feedback (thumbs-down, corrections, escalations) attaches to trace IDs; flagged traces flow into labeling queues and golden sets; evaluation runs produce scores; scores come back as metrics with alert thresholds. Observability without this loop is a very detailed record of problems nobody measures.

Observability is not evaluation

Worth stating precisely because teams substitute one for the other: observability records what happened on each request; evaluation measures whether outputs are correct against ground truth. A tracing dashboard cannot tell you faithfulness dropped — only an eval with labels can. An eval suite cannot tell you which tenant drove yesterday's cost spike — only traces can.

How AI Observability Works

Instrumentation

Instrumentation should be orthogonal to business logic: wrap the clients (LLM provider SDK, vector DB, tool executor) once, rather than scattering log calls through application code. The pattern, whether via an LLM-native SDK or OpenTelemetry directly:

1. A trace ID is generated at the gateway and propagated via headers (traceparent) through every internal call.
2. Wrapped clients open a child span per operation, attach semantic attributes, and record token usage and status.
3. Spans export asynchronously — the user response never waits on the trace backend, and export failures never fail requests.

What to capture (minimum viable)

Deriving metrics and alerts

Metrics come from spans, not separate instrumentation: token counts aggregate into cost-per-tenant-per-day; stage durations become per-stage latency histograms; zero-result flags become a retrieval health rate. On top of these, define SLOs — p99 end-to-end latency, time-to-first-token, error rate, daily cost envelope per tenant — and alert on breaches plus anomalies (cost 2× baseline, guardrail block spike, retrieval zero-result rate climbing). A dashboard without SLOs is decoration; the alert thresholds are the actual operational contract.

Sampling and redaction

Full payload capture at production volume is expensive and risky. The standard policy: capture attributes (IDs, scores, counts) on 100% of requests; capture full prompt/completion payloads on a sample (1–10%) plus 100% of errors, high-latency outliers, and user-flagged requests. Redact PII — emails, account numbers, names — before export, in the instrumentation layer, not in the backend.

Architecture

The reference architecture is a pipeline from the request path to storage and consumption, with the collector as the control point for sampling and redaction.

Diagram: AI observability reference architecture

flowchart TB
    subgraph Path [Request path]
        GW[API gateway\ntrace ID origin] --> App[App / orchestrator]
        App --> Ret[Retriever + reranker]
        App --> LLM[LLM provider]
        App --> Tools[Tool executor]
        App --> Guard[Guardrails]
    end
    subgraph Emit [Instrumentation]
        SDK[Tracer SDK\nwrapped clients, async export]
    end
    subgraph Process [Collection]
        Col[Collector\nsampling, PII redaction, batching]
    end
    subgraph Store [Storage]
        TS[(Trace store)]
        MS[(Metrics store)]
        LS[(Log store)]
    end
    subgraph Consume [Consumption]
        Dash[Dashboards + SLO alerts]
        Eval[Eval pipeline\nsampled traces to golden sets]
        Audit[Audit / forensics]
    end
    App & Ret & LLM & Tools & Guard -.spans.-> SDK --> Col
    Col --> TS & MS & LS
    TS --> Dash & Eval & Audit
    MS --> Dash
    LS --> Audit
    Eval -->|scores as metrics| MS

Spans flow gateway → app → tracer → collector → stores → dashboards, alerts, and the eval pipeline. Sampling and redaction live in the collector so policy changes don't require app deploys.

On the LLM-native vs vendor-neutral question: LLM-native platforms (LangSmith, Langfuse, Braintrust, Helicone, Phoenix — evaluate by pattern, they converge) give you payload-aware UIs, prompt diffing, and dataset export out of the box. OpenTelemetry gives you one pipeline for AI and non-AI services and no vendor lock-in. Mature stacks commonly run both: OTel as transport with GenAI semantic conventions, an LLM-native backend for prompt-level debugging.

Step-by-Step Flow

The flow below traces one RAG request through an instrumented pipeline, including the async export and feedback path.

Diagram: One traced request, end to end

sequenceDiagram
    participant U as User
    participant GW as Gateway
    participant App as Orchestrator
    participant Ret as Retriever
    participant LLM as LLM provider
    participant Col as Collector (async)
    U->>GW: Question
    GW->>App: Request + trace_id in header
    App->>Ret: Embed + search + rerank (child spans)
    Ret-->>App: Chunks + scores
    App->>LLM: Prompt v14, temperature 0
    LLM-->>App: Completion + token usage
    App-->>U: Answer + X-Trace-Id header
    App--)Col: Export spans (never blocks response)
    Col->>Col: Sample decision + PII redaction
    Col--)Col: Write trace, derive metrics
    U->>GW: Thumbs-down
    GW--)Col: Feedback event attached to trace_id
    Note over Col: Flagged trace enters eval labeling queue

The response returns before any trace export happens; the thumbs-down later attaches to the same trace, which is how bad answers become eval cases.

1. Generate the trace ID at the gateway. Propagate via traceparent (or X-Trace-Id) to every internal call, and return it to the client in a response header so support tickets can reference it.
2. Wrap external calls in child spans. Every embedding call, vector search, rerank, LLM completion, and tool invocation gets a span with status and timeout.
3. Attach semantic attributes. Model ID, prompt version, temperature, top-k, filters, cache hit/miss, tenant ID. These attributes are what make traces queryable ("all slow requests using prompt v14 for tenant acme").
4. Capture payloads with redaction and sampling. Chunk IDs and scores on every request; full prompt/completion on the sampled set plus all errors and flagged requests.
5. Export asynchronously. Batch spans; drop on backend failure rather than degrading user requests.
6. Derive metrics from spans. Token counts → cost per tenant; stage durations → latency histograms; zero-result flags → retrieval health.
7. Attach feedback to traces. Thumbs-down, corrections, and escalations reference the trace ID — turning vague complaints into reproducible cases.
8. Route flagged traces to evaluation. A nightly job pulls thumbs-down and low-score traces into a labeling queue; the best cases join the golden set. See Evaluation.
9. Alert on SLOs and anomalies. p99 per stage, time-to-first-token, error rates, daily cost vs baseline, guardrail block spikes, eval-score drops.

Real Production Example

A support assistant runs a multi-step pipeline: query rewrite, retrieval, rerank, generation with an order-lookup tool. Finance asks why Tuesday's LLM bill was 3× normal; support asks why tenant acme got slow answers the same day. Both questions are answered by the same instrumentation — per-span token, cost, and latency attribution.

from __future__ import annotations

import time
import uuid
from contextlib import contextmanager
from dataclasses import dataclass, field

# Per-1M-token pricing, kept in config — model choice is a routing
# decision (fast tier for rewrite, frontier tier for final answers).
PRICING = {
    "fast-tier":     {"in": 0.10, "out": 0.40},   # Haiku 4.5 / GPT-5.6 Luna class
    "frontier-tier": {"in": 3.00, "out": 15.00},  # Sonnet 5 / GPT-5.6 Terra class
}


@dataclass
class TraceContext:
    trace_id: str
    tenant_id: str
    spans: list[dict] = field(default_factory=list)


@contextmanager
def span(ctx: TraceContext, name: str, **attrs):
    record = {"name": name, "trace_id": ctx.trace_id,
              "tenant_id": ctx.tenant_id, **attrs}
    start = time.perf_counter()
    try:
        yield record
        record["status"] = "ok"
    except Exception as e:
        record["status"] = "error"
        record["error_type"] = type(e).__name__
        raise
    finally:
        record["duration_ms"] = (time.perf_counter() - start) * 1000
        ctx.spans.append(record)   # exported async in batches, never inline


def record_llm_usage(rec: dict, tier: str, usage) -> None:
    price = PRICING[tier]
    rec.update(
        model_tier=tier,
        input_tokens=usage.input_tokens,
        output_tokens=usage.output_tokens,
        cost_usd=(usage.input_tokens * price["in"]
                  + usage.output_tokens * price["out"]) / 1_000_000,
    )


class ObservableSupportPipeline:
    def __init__(self, retriever, llm, tools, exporter):
        self.retriever, self.llm, self.tools = retriever, llm, tools
        self.exporter = exporter

    async def answer(self, question: str, tenant_id: str) -> dict:
        ctx = TraceContext(trace_id=str(uuid.uuid4()), tenant_id=tenant_id)

        with span(ctx, "llm.rewrite", prompt_version="rewrite-v3") as s:
            rw = await self.llm.complete(tier="fast-tier",
                                         prompt=REWRITE_TMPL.format(q=question))
            record_llm_usage(s, "fast-tier", rw.usage)

        with span(ctx, "retrieval.search", top_k=20) as s:
            chunks = await self.retriever.search(rw.text, top_k=20,
                                                 filter={"tenant": tenant_id})
            s["chunk_ids"] = [c.id for c in chunks[:5]]
            s["top_score"] = chunks[0].score if chunks else 0.0
            s["zero_results"] = not chunks

        with span(ctx, "tool.order_lookup") as s:
            order = await self.tools.order_lookup(question, tenant_id)
            s["tool_status"] = order.status

        with span(ctx, "llm.generate", prompt_version="answer-v14") as s:
            resp = await self.llm.complete(
                tier="frontier-tier",
                prompt=ANSWER_TMPL.format(q=question, chunks=chunks, order=order),
                temperature=0,
            )
            record_llm_usage(s, "frontier-tier", resp.usage)

        self.exporter.submit(ctx.spans)   # async batch; redaction in collector
        return {"answer": resp.text, "trace_id": ctx.trace_id}

With spans in the store, both Tuesday questions become queries instead of investigations:

• Cost spike: sum(cost_usd) by (tenant_id, name) for Tuesday shows tenant acme at 40× their baseline on llm.rewrite spans — a client-side retry loop was re-sending each question ~12 times. The fix was idempotency keys at the gateway; the evidence was one metrics query.
• Slow answers: the latency histogram by span name shows retrieval.search p99 jumped from 90ms to 2.1s for acme only — their index shard was resharding. Generation was innocent; nobody wasted a day tuning prompts.
• The bad answer in the same window: support pastes the trace_id from the user's session, the trace shows zero_results=true on retrieval with the model answering anyway from parametric memory — which becomes both a guardrail fix (abstain on empty retrieval) and a new golden-set case.

Design Decisions

Common patterns

Decision matrix

Self-hosted vs SaaS trace backends

Prompts and completions are among the most sensitive payloads a company handles. SaaS backends (fastest setup, best UX) fit when redaction is solid and data residency permits. Self-hosted (Langfuse, Phoenix, Tempo) fits regulated industries and strict residency requirements — at the cost of operating the store. The deciding question is usually legal, not technical.

Comparisons

Common Mistakes

1. Logging only final answers. Useless for debugging retrieval failures, tool errors, or prompt regressions. Every stage gets a span keyed to the trace ID.

2. No tenant or cost attribution. Finance asks which customer drove the spike; you cannot answer. Tag every span with tenant ID and derive cost from token counts from day one.

3. Synchronous trace export in the hot path. Awaiting the trace backend before responding couples user latency to observability infrastructure. Export async; drop spans on backend failure.

4. Storing raw PII in traces. Prompts contain emails, account numbers, and health details by construction. Redact before export — a third-party trace store full of raw prompts is a compliance incident in waiting.

5. Metrics without quality signal. Latency stable, error rate flat, quality collapsed after an embedding model change. Wire eval scores and feedback rates into the same dashboards as latency.

6. Inconsistent span naming. llm_call, openai, generate across services makes aggregation impossible. Define a naming convention (llm.generate, retrieval.search, tool.<name>) and enforce it in the wrapped clients.

7. No retention policy matched to incident SLA. Traces expire in 3 days; the weekly quality review finds a Tuesday issue on Friday. Retain at least as long as your investigation window (7–30 days minimum).

8. Dashboards without SLOs. Twelve beautiful charts and no defined threshold means every incident starts with a debate about whether it's an incident. Define SLOs first; dashboards visualize them.

Common Mistake

Instrumenting after the first incident. You'll have no baseline for "what changed," and last week's bad answers are gone forever. Tracing is a day-one dependency, like migrations.

Where It Breaks Down

• Volume and cost. Full payload capture at 10K QPS is expensive in storage and egress. Sampling and selective capture are mandatory at scale — which means some individual bad answers won't have full payloads. Tail-sampling on errors and feedback mitigates but doesn't eliminate this.

• Provider blind spots. Your span measures your client's view of the LLM call — not the provider's internal queue time, silent model updates, or regional degradation. Correlate with provider status feeds and keep per-region latency baselines.

• Deep multi-agent traces. One request spawning dozens of LLM calls across agents produces trace trees too deep to read. Use sub-traces per agent, roll up token/cost to the parent, and build aggregate views — raw waterfalls stop being useful.

• Streaming semantics. A single duration is misleading for streamed responses: time-to-first-token and total stream time are different SLOs. Instrument both, or your latency dashboards will look fine while users stare at blank screens.

• Semantic gaps. Traces show what happened, not why the model behaved that way. A perfectly captured trace of a hallucination still requires evaluation and hallucination detection to characterize the failure.

• Observability of the observers. Collectors crash, exporters drop spans, sampling misconfigurations silently discard the errors you meant to keep. Monitor span-drop rates and collector health, or you'll debug an incident with a trace store that quietly lost the evidence.

When NOT to Rely on Observability Alone

Observability is necessary infrastructure, not a substitute for other disciplines. Do not lean on it when:

• You need to know whether outputs are correct. Observability is not evaluation. Traces record behavior; only labeled evals measure quality. A team with world-class tracing and no golden set will watch quality regress in high definition. Build evaluation alongside, not after.

• Dashboards exist but SLOs don't. Charts without thresholds produce alert fatigue in one direction and missed incidents in the other. If you cannot state "p99 time-to-first-token under 800ms, daily cost within 1.5× baseline per tenant," the dashboards are decoration.

• Payloads contain PII and redaction isn't built. Do not turn on full prompt capture "temporarily" before scrubbing exists. Logging PII prompts without redaction converts a debugging convenience into a GDPR/SOC2 liability — and trace stores are rarely governed as strictly as production databases.

• You expect it to prevent failures. Observability detects and explains; it does not block. Preventing bad outputs is guardrails; preventing injections is AI security. Tracing an attack beautifully is not a defense.

• The system is a low-volume prototype. A notebook demo doesn't need an OTel collector. Structured logging with request IDs is enough until real users and real money arrive — then instrument before launch, not after the first incident.

Running in Production

Best Practice

Define SLOs before dashboards: p99 per stage, time-to-first-token, error rate, cost envelope per tenant, and quality-proxy floors. Every chart should answer "are we inside SLO?" — everything else is a debugging view.

Production checklist

• Trace ID generated at the gateway, propagated everywhere, returned to the client
• Child spans for embed, search, rerank, every LLM call, every tool call, guardrails
• Semantic attributes: model ID, prompt version, tenant, tokens, scores, cache status
• Async batched export; requests never wait on the trace backend
• PII redaction before export; payload capture sampled with tail-keep for errors/flags
• Cost derived from span token counts, attributed per tenant and stage
• SLOs defined and alerting: latency per stage, TTFT, error rate, cost anomaly, quality proxies
• Feedback events linked to trace IDs; flagged traces auto-routed to eval
• Retention ≥ investigation window (7–30 days); RBAC on the trace UI
• Collector health and span-drop rate monitored

Related Guides

Diagram: Where observability sits in the production AI path

flowchart LR
    ARCH[AI System Architecture] --> OBS[AI Observability - you are here]
    LLMg[Large Language Models] --> OBS
    OBS --> EV[Evaluation]
    OBS --> CO[Cost Optimization]
    OBS --> LAT[Latency Optimization]
    OBS --> SEC[AI Security]
    EV --> GR[Guardrails]
    OBS --> GR

Observability supplies the data every downstream discipline consumes: eval datasets, cost attribution, latency breakdowns, and security forensics.

Upstream foundations:

• AI System Architecture — the layer boundaries where instrumentation hooks attach
• Large Language Models — token semantics behind cost and latency metrics
• RAG — retrieval spans are the first stop for any wrong answer

Downstream consumers:

• Evaluation — traces feed golden sets; eval scores return as alertable metrics
• Agent Evaluation — score tool spans, trajectories, and task outcomes on those traces
• Cost Optimization — span-level token attribution drives routing and caching
• Latency Optimization — per-stage histograms and TTFT locate the bottleneck
• Guardrails — verdicts as spans; block-rate alerting
• AI Security — injection forensics and tool-call audit trails

Tools: LangChain · LangSmith

Interview Questions

1. Why is classic APM insufficient for LLM systems?
   APM measures availability and latency, but LLM failures are semantic: HTTP 200 with a hallucinated answer, wrong retrieval, runaway agent cost. Debugging requires request-scoped capture of prompts, retrieved context, tool calls, and token counts — none of which classic APM records.

2. Describe the trace model for a RAG request.
   One trace per user request; child spans for query embedding, vector search, reranking, each LLM completion, each tool invocation, and guardrail checks. Spans carry model ID, prompt version, token counts, retrieval chunk IDs and scores, tenant ID, status, and duration; payloads captured on sampled requests with redaction.

3. What is the difference between observability and evaluation?
   Observability records what happened per request; evaluation measures whether outputs are correct against ground truth. They form a loop: traces feed eval datasets, and eval scores return as monitored metrics. Neither substitutes for the other.

4. How do you attribute LLM cost per customer?
   Tag every LLM span with tenant ID and token usage; multiply by a per-model price table (kept in config since routing tiers change); aggregate per tenant per day. Span-derived cost keeps traces and billing from drifting apart.

5. How would you debug "the bot gave a wrong answer yesterday"?
   Locate the trace (by trace ID from the session, or time/user/query search). Check in order: was retrieval empty or irrelevant (chunk IDs, scores, zero-result flag)? Did a tool return bad data? Did the model ignore context? Did the prompt version or model version change with a deploy? Fix the earliest failing stage; promote the case into the golden set.

6. What sampling strategy would you use at high volume?
   Attributes on 100% of requests (cheap, enables all metrics); full payloads head-sampled at 1–10% of successes; tail-based keep for 100% of errors, latency outliers, guardrail blocks, and user-flagged requests — the traces you'll actually need are the anomalous ones.

7. How do you handle PII in traces?
   Redact before export, in the instrumentation SDK or collector — never rely on the backend. Replace emails, account numbers, and names with stable pseudonyms; store IDs referencing secured systems where full content is required; enforce RBAC and retention on the trace store; audit access.

8. How do you trace streaming responses?
   Open the span at request start; record time-to-first-token when the first chunk arrives; close on stream end with total tokens and duration. TTFT and total time are separate SLOs — a stream can have great TTFT and terrible tail latency, or vice versa.

9. What alerts would you configure for a production LLM service?
   Per-stage p99 latency and TTFT breaches, LLM error/retry rates, retrieval zero-result rate above threshold, daily cost anomaly vs baseline per tenant, guardrail block spikes (possible attack or misconfiguration), and quality-proxy regressions (thumbs-down rate, eval score drops).

10. LLM-native platform vs OpenTelemetry — how do you choose?
    LLM-native (LangSmith/Langfuse class) gives payload-aware debugging, prompt diffing, and eval linkage immediately; OTel gives one pipeline across AI and non-AI services with no lock-in. Common production answer: OTel-style transport with GenAI semantic conventions, plus an LLM-native backend for prompt-level workflows.

11. How does observability support multi-agent systems, and where does it strain?
    Sub-traces per agent with cost/token rollups to the parent make deep trees navigable; consistent span naming enables cross-agent aggregation. It strains on readability (dozens of LLM calls per request) and on attribution when agents recurse — aggregate views replace raw waterfalls.

12. Why should trace export never be synchronous?
    Synchronous export couples user latency and availability to the observability backend — an outage in your trace store becomes an outage in your product. Export must be async and batched, with spans dropped (and drop-rate monitored) rather than requests failed.

Key Takeaways

• LLM failures are semantic — wrong, expensive, or slow while returning HTTP 200 — so observability must capture prompts, retrieval context, tool calls, tokens, and per-stage latency, not just uptime.
• The trace is the unit of debugging: one per request, spans per pipeline stage, semantic attributes that make behavior queryable, and the trace ID returned to the client.
• Derive metrics from spans — cost per tenant, latency per stage, retrieval health — and define SLOs before dashboards.
• Redact PII before export, sample payloads with tail-keep for anomalies, and never block a user response on trace export.
• Close the loop: feedback attaches to traces, flagged traces become eval cases, eval scores come back as alerts. Observability without evaluation is watching quality regress in high definition.

FAQs

What is the difference between observability and evaluation?

Observability records what happened on each request — traces, metrics, logs. Evaluation measures whether outputs are correct against ground truth. Traces feed eval datasets; eval scores become metrics you alert on. See Evaluation.

Do I need an LLM-native tool if I already use OpenTelemetry?

Not strictly. OTel with GenAI semantic conventions covers transport and infra unification. LLM-native platforms add payload-aware UIs, prompt version diffing, and dataset export. Many teams run both: OTel as the spine, an LLM-native backend for prompt-level debugging.

What should every LLM trace include?

Trace ID, tenant ID, model ID, prompt version, per-stage latency, input/output token counts, retrieval chunk IDs and scores, tool names and statuses, guardrail verdicts, and error details. Payloads on sampled requests, redacted.

How do I debug a wrong RAG answer?

Find the trace, then check stages in order: (1) was retrieval empty or irrelevant? (2) did reranking demote the right chunk? (3) did the model ignore the context? (4) did a prompt or model version change that day? Fix the earliest failing stage. Most wrong answers start at retrieval.

How much tracing overhead is acceptable?

Under 5ms of instrumentation per request, excluding export — which must be asynchronous. At high QPS, sample payload capture; attributes are cheap enough for 100%.

How do I attribute cost per customer or feature?

Tag spans with tenant and feature identifiers; record token usage per LLM span; multiply by a per-model price table in config; aggregate daily. This also feeds cost optimization routing decisions.

Can traces be used for compliance audits?

Yes, if retention and access controls meet policy: tool calls, data accessed, and guardrail outcomes are all in the trace. Redact PII and reference secured stores by ID for full content.

How do I trace streaming responses?

Record time-to-first-token on the first chunk and total duration/tokens at stream end. Treat TTFT and total time as separate SLOs.

What alerts should I set first?

Start with five: p99 latency per stage, LLM error rate, retrieval zero-result rate, daily cost anomaly per tenant, and a quality proxy (thumbs-down rate or eval score). Expand from incidents.

How do I handle trace volume at scale?

Attributes on everything; payloads head-sampled at 1–10% with tail-based keep for errors, outliers, and user-flagged requests. Match retention to your investigation window, typically 7–30 days.

Does observability help against prompt injection?

It provides detection and forensics — suspicious input patterns, tool-escalation attempts, guardrail blocks — but does not prevent attacks. Pair with AI Security and Guardrails.

Which spans matter most in an agent pipeline?

Tool spans and per-step LLM spans. Agent failures are usually a bad tool result propagating, or a loop that should have terminated — both visible only with per-call spans and rollup counts per trace.

References

• OpenTelemetry — Generative AI Semantic Conventions
• LangSmith Documentation
• Langfuse Documentation
• Arize Phoenix Documentation
• OpenAI API Documentation

Further Reading

• Google SRE Book — Monitoring Distributed Systems
• Braintrust Documentation
• Helicone Documentation
• Anthropic Documentation