Observability Guide

TL;DR

• AI observability is tracing non-deterministic pipelines - you need to see prompts, retrieved context, tool calls, latencies, token counts, and outputs for every request, not just HTTP 500s.

• LangSmith and similar tools provide LLM-native tracing - parent/child spans for chains, automatic capture of inputs/outputs, and dataset regression against golden examples.

• Structure logs around trace IDs - one trace_id from API gateway through retrieval, generation, and tools lets you reconstruct any bad answer in minutes.

• Metrics alone miss quality regressions - latency and error rate can look fine while retrieval recall collapses; combine traces with automated eval and sampling for human review.

• Instrument before you need it - adding tracing after a production incident means you have no baseline for "what changed."

Why This Matters

Traditional APM tells you p99 latency and error rates. That is necessary but insufficient for LLM systems. A request can return HTTP 200 with a hallucinated answer, retrieve the wrong document, call an unauthorized tool, or cost 10× your average - and your dashboards stay green.

When a user reports "the bot lied about our refund policy on Tuesday," you need: the exact prompt sent, which chunks were retrieved and their scores, which model and temperature were used, whether a tool returned stale data, and whether a deploy changed the reranker that morning. Without observability designed for AI, incident response is archaeology.

Observability also closes the loop with evaluation. Traces feed datasets; datasets drive regression tests; regression tests block bad deploys. This is how AI systems move from "demo" to "operated service."

The Problem Observability Solves

LLM applications fail in ways classic monitoring does not capture:

Silent quality degradation. Retrieval recall drops after a chunking change. No errors - just wrong answers.

Non-reproducible bugs. Same question, different answer (temperature, model updates, cache hits). You need full context capture, not just the final string.

Cost explosions. An agent loop runs 12 tool calls instead of 2. Per-request cost spikes without error signals.

Cross-service blindness. Gateway → orchestrator → vector DB → reranker → OpenAI → guardrails. Without distributed tracing, you know generation was slow but not whether retrieval or reranking caused it.

Observability provides request-scoped visibility into inputs, intermediate steps, outputs, metadata (model, version, tenant), and resource consumption - with retention long enough to debug yesterday's incident.

What Is AI Observability?

AI observability is the practice of instrumenting LLM applications so operators can understand system behavior in production: debug failures, attribute cost, detect quality regressions, and audit decisions.

It extends three pillars from classical observability:

Tools like LangSmith (LangChain ecosystem), Langfuse, Arize Phoenix, Braintrust, and OpenTelemetry with custom semantic conventions address LLM-specific needs: capturing large text payloads, linking spans to eval datasets, and comparing runs across prompt versions.

Observability is not the same as evaluation - eval measures correctness against ground truth; observability captures what happened on each request so you can investigate and feed eval pipelines.

How Observability Works

Trace Model

A trace represents one user request. Spans represent units of work within it:

trace: support_query_8f3a
├── span: api.gateway (12ms)
├── span: orchestrator.route (45ms)
│   ├── span: retrieval.embed_query (38ms)
│   ├── span: retrieval.vector_search (62ms)
│   ├── span: retrieval.rerank (180ms)
│   ├── span: llm.generate (2100ms) ← tokens: 1847 in / 312 out
│   └── span: guardrails.validate (95ms)
└── span: session.persist (8ms)

Each span carries: start/end time, status, attributes (model=gpt-4o, tenant_id=acme), and optionally inputs/outputs (with PII redaction).

LangSmith Integration

LangSmith provides hosted tracing for LangChain and direct SDK usage:

1. Set LANGCHAIN_TRACING_V2=true and LANGCHAIN_API_KEY.
2. Wrap chains or manual calls with @traceable or LangChain runnables.
3. View waterfall traces in the LangSmith UI - inspect prompts, tool outputs, and latency breakdown.
4. Promote traces to datasets for regression testing.

For non-LangChain systems, use the LangSmith Python/JS SDK or OpenTelemetry exporters with equivalent span naming.

What to Capture (Minimum Viable)

Redact PII and secrets before export. Hash prompts if full text retention is restricted.

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Architecture

A production observability stack for AI:

Keep instrumentation orthogonal to business logic - wrap clients (OpenAI, vector DB) rather than scattering log calls everywhere.

Step-by-Step Flow

Step 1: Generate trace ID at gateway. Propagate via HTTP header (traceparent or X-Trace-Id) to all internal calls.

Step 2: Wrap external calls in spans. Each embedding, search, rerank, LLM completion, and tool invocation gets a child span with timeout and status.

Step 3: Attach semantic attributes. Model name, temperature, top_k, filter applied, cache hit/miss.

Step 4: Capture inputs/outputs with redaction. Store retrieval chunk IDs (not full text) if storage is costly; full prompt on sampled requests (1–5%).

Step 5: Export spans asynchronously. Never block user response on trace export failure.

Step 6: Derive metrics from spans. Token counts, stage latencies, tool call counts → Prometheus histograms.

Step 7: Sample for eval. Nightly job pulls traces where user gave thumbs-down or confidence was low → labeling queue.

Step 8: Alert on SLO breaches. p99 generation > 5s, retrieval error rate > 1%, daily cost > 2× baseline.

Real Production Example

Support bot with LangSmith tracing, OpenTelemetry metrics, and retrieval debug fields:

import os
import time
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import Any, Optional

# LangSmith: set LANGCHAIN_TRACING_V2=true in production
from langsmith import traceable
from langsmith.run_helpers import get_current_run_tree

@dataclass
class SpanContext:
    trace_id: str
    tenant_id: str
    attributes: dict = field(default_factory=dict)

@contextmanager
def timed_span(name: str, ctx: SpanContext, **attrs):
    start = time.perf_counter()
    merged = {**ctx.attributes, **attrs, "tenant_id": ctx.tenant_id}
    try:
        yield merged
    finally:
        elapsed_ms = (time.perf_counter() - start) * 1000
        # Emit to metrics backend
        metrics_histogram("ai.span.duration_ms", elapsed_ms, tags={"span": name, **merged})
        run = get_current_run_tree()
        if run:
            run.add_metadata({**merged, "duration_ms": elapsed_ms})

class ObservableRAGPipeline:
    def __init__(self, retriever, llm_client, metrics):
        self.retriever = retriever
        self.llm = llm_client
        self.metrics = metrics

    @traceable(name="rag.query", run_type="chain")
    async def query(self, question: str, ctx: SpanContext) -> dict:
        ctx.attributes["question_length"] = len(question)

        with timed_span("retrieval", ctx, top_k=20):
            chunks = await self.retriever.search(
                question,
                filter={"tenant_id": ctx.tenant_id},
                top_k=20,
            )
            ctx.attributes["retrieval_count"] = len(chunks)
            ctx.attributes["top_score"] = chunks[0].score if chunks else 0.0
            ctx.attributes["chunk_ids"] = [c.id for c in chunks[:5]]

        with timed_span("rerank", ctx):
            chunks = await self.retriever.rerank(question, chunks, top_n=5)

with timed_span("generation", ctx, model="gpt-4o-mini"):
            response = await self.llm.complete(
                model="gpt-4o-mini",
                messages=self._build_messages(question, chunks),
            )
            ctx.attributes["input_tokens"] = response.usage.prompt_tokens
            ctx.attributes["output_tokens"] = response.usage.completion_tokens
            self.metrics.increment(
                "ai.tokens.total",
                response.usage.prompt_tokens + response.usage.completion_tokens,
                tags={"tenant_id": ctx.tenant_id, "model": "gpt-4o-mini"},
            )

        return {
            "answer": response.text,
            "sources": [c.metadata for c in chunks],
            "trace_id": ctx.trace_id,
        }

# Usage in FastAPI middleware
@app.middleware("http")
async def trace_middleware(request, call_next):
    trace_id = request.headers.get("traceparent") or str(uuid.uuid4())
    request.state.trace_id = trace_id
    response = await call_next(request)
    response.headers["X-Trace-Id"] = trace_id
    return response

When a user reports a bad answer, search LangSmith by trace_id, inspect retrieval scores and chunk IDs, compare to a known-good trace for the same question.

Design Decisions

⚠ Common Mistakes

1. Logging only final answers. Useless for debugging retrieval failures. Log each stage with correlation ID.

2. No tenant/cost attribution. Finance asks "which customer drove the spike?" - you cannot answer.

3. Tracing in the hot path synchronously. Export spans async; never await trace backend before responding.

4. Storing PII in traces. GDPR and SOC2 violations. Redact emails, account numbers; use IDs.

5. Metrics without eval. Latency stable, quality collapsed after embedding model change. Run weekly eval; alert on score drops.

6. Inconsistent span names. llm_call, openai, generate across services - impossible to aggregate. Define a naming convention.

7. No retention policy. Traces expire before you investigate weekly reports. Match retention to incident SLA (7–30 days minimum).

Where It Breaks Down

Volume and cost. Full prompt capture at 10K QPS is expensive. Sampling and selective capture are mandatory at scale.

Non-LangChain stacks. LangSmith integrates cleanly with LangChain; custom pipelines need manual @traceable or OTel instrumentation.

Multi-agent systems. One user request spawns dozens of LLM calls across agents - trace trees become deep and hard to read. Use sub-traces per agent and collapse in UI.

Provider blind spots. You see your client's latency, not OpenAI's internal queue time. Combine with provider status pages and regional failover metrics.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Every production incident involving "wrong AI answer" requires retrieval context in the trace. If you cannot see what was retrieved, you cannot fix the pipeline.

Ecosystem

• LangSmith: Native LangChain tracing, datasets, eval runs, prompt hub.

• Langfuse: Open-source alternative, self-hostable, prompt management.

• Arize Phoenix: Open-source, embedding and retrieval visualization.

• Braintrust: Eval-first with trace integration.

• OpenTelemetry: Vendor-neutral spans; define LLM semantic conventions.

• Prometheus / Grafana / Datadog: Metrics and alerting on derived span data.

Related Technologies

• AI System Architecture: Where observability hooks attach at layer boundaries.

• Guardrails: Guardrail results should appear as trace spans and metrics.

• AI Security: Audit logs for tool calls and blocked injections.

• Cost Optimization: Token metrics drive routing and caching decisions.

• RAG: Retrieval spans are the first place to look when answers are wrong.

Learning Path

Prerequisites: AI System Architecture · Large Language Models

Next topics: Guardrails · AI Security · Cost Optimization

Estimated time: 50 min · Difficulty: Intermediate

FAQs

What is the difference between observability and evaluation?

Observability records what happened on each request (traces, logs, metrics). Evaluation measures whether outputs are correct against ground truth. Traces feed eval datasets; eval scores become metrics you alert on.

Do I need LangSmith if I use OpenTelemetry?

Not necessarily. LangSmith adds LLM-specific UX and LangChain integration. OTel works for unified infra. Many teams use both: OTel for infra metrics, LangSmith for prompt-level debugging.

What should every LLM trace include?

Trace ID, tenant ID, model ID, prompt version, latency per stage, input/output token counts, retrieval chunk IDs and scores, tool call names and status, guardrail outcome, and error details if any.

How do I debug a wrong RAG answer?

Find trace by time/user/question. Check: (1) were correct chunks retrieved? (2) reranker scores? (3) did LLM ignore context? (4) did prompt version change? Fix the earliest failing stage.

How much tracing overhead is acceptable?

Under 5ms instrumentation per request. Export asynchronously. At high QPS, sample 1–10% of success traces; 100% of errors and flagged requests.

How do I attribute LLM cost per customer?

Tag spans with tenant_id. Sum input_tokens + output_tokens × price per model daily. Expose in billing dashboard or internal chargeback reports.

Can I use observability for compliance audits?

Yes, if retention and access controls meet policy. Log tool calls, data accessed, and guardrail blocks. Redact PII; store IDs referencing secure stores for full content if needed.

How do I trace streaming responses?

Start span at request; update with first_token_ms metric on first chunk; close span on stream end with total tokens. LangSmith supports streaming runs.

What alerts should I set?

p99 latency per stage, LLM error rate, retrieval zero-result rate > threshold, daily token spend anomaly, eval faithfulness drop, guardrail block spike (may indicate attack or misconfiguration).

How do I handle trace volume at scale?

Head-based sampling for successes; tail sampling for errors, high latency, and user thumbs-down. Store full payloads only for sampled traces.

Does observability help with prompt injection attacks?

It helps detection and forensics - log suspicious input patterns, tool escalation attempts, and guardrail blocks. Pair with AI Security defenses; tracing alone does not prevent attacks.

How do I connect traces to CI/CD?

Export golden trace fixtures from staging. On deploy, run eval suite against prompt/model changes. Compare faithfulness and latency to baseline before promoting to production.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• AI observability requires request-scoped traces through retrieval, generation, tools, and guardrails - not just HTTP metrics.
• LangSmith and OTel provide the spine; define consistent span names and mandatory attributes.
• Capture retrieval context (chunk IDs, scores) - most "wrong answer" incidents start there.
• Redact PII, sample at scale, export asynchronously.
• Connect traces to eval pipelines and alert on quality metrics, not just errors and latency.