Architecture

AI System Architecture Guide

Reference architecture for modern enterprise AI systems, bringing together retrieval, knowledge graphs, AI agents, LLM gateways, caching, observability, security, and production deployment.

55 min readAdvancedLast reviewed: 21 July 2026

Quick Summary

AI System Architecture is the layered blueprint that separates ingestion, retrieval, orchestration, generation, and governance so production LLM applications can scale, fail safely, and evolve independently.

One Analogy

AI System Architecture is an airport control tower — planes (requests) are routed, screened, and logged before they touch any runway (model, retrieval, or tool).

Engineering Rule

Treat orchestration as a control plane with typed contracts, timeouts, and tenant isolation — never collapse retrieval, generation, and auth into a single prompt handler.

AI System Architecture in One Sentence

AI System Architecture =

  • Ingestion & Indexing
  • Orchestration
  • Retrieval
  • LLM Generation
  • Tools / Agents
  • Caching
  • Guardrails
  • Observability
  • Evaluation

TL;DR

  • Production AI systems are layered pipelines, not single API calls — ingestion, indexing, orchestration, retrieval, generation, and post-processing each have different scaling and failure profiles.

  • The orchestration layer is the control plane — it routes requests, manages state, invokes tools, enforces timeouts, and coordinates multi-step workflows without embedding business logic in prompt strings.

  • Separate offline indexing from online query paths — re-embedding documents must not require redeploying the API, and query latency must not depend on batch ingestion jobs.

  • Every layer boundary needs observability, caching, and fallbacks — propagate trace IDs across retrieval and generation, degrade gracefully when a vector store is slow, and never let one failed tool call hang the request path.

  • Specialized architectures compose inside this blueprintEnterprise RAG, GraphRAG, Knowledge Graph + LLM, and AI Agents are patterns within the same layered system, not replacements for it.

Architecture Snapshot

Complexity

★★★★★

Audience

AI Architects, Platform Engineers, AI Engineers, Tech Leads

Difficulty

Advanced

Typical Deployment

Enterprise Production

Typical Latency

1.5–4 seconds (p95 interactive)

Scalability

Independent scale per layer

Availability Target

99.9%

Read Time

~55 min

Last Updated

July 21, 2026

Recommended Stack

  • FastAPI / NestJS orchestrator
  • LangGraph / LlamaIndex (as libraries)
  • AI Gateway (LiteLLM / Portkey)
  • Qdrant / Pinecone / Weaviate
  • Redis + Postgres
  • OpenTelemetry + Langfuse / LangSmith

Why This Matters

Most AI demos are a single script: embed a query, call a model, return text. Production systems serve concurrent tenants, enforce isolation, integrate with CRMs and internal APIs, survive provider outages, and must be debuggable when a customer reports a wrong answer from three days ago.

Without deliberate architecture, teams accumulate spaghetti: prompts embedded in API handlers, retrieval logic duplicated across endpoints, no way to A/B test a reranker, and incident response that starts with "which service even logged this?" The orchestration layer — often dismissed as glue code — is where reliability, cost control, and security actually live.

If you are moving from prototype to production, or inheriting a system that grew organically, this guide is the reference map. Specialized guides deepen individual patterns; this document defines how those patterns fit together.

Who should read this?

Reader Why
AI Architects Define layer boundaries, contracts, and scaling paths across products.
Platform Engineers Operate gateways, indexes, caches, and Kubernetes deployments with clear SLAs.
AI Engineers Implement retrieval, agents, and evaluation without reinventing platform concerns.
Technical Leads Decide monolith vs services, framework vs custom orchestrator, agent vs workflow.
Security / Compliance See where authz, audit, and PII controls must live (not in prompts).

The Problem AI System Architecture Solves

LLM applications combine familiar backend concerns (APIs, databases, auth) with unfamiliar ones (non-deterministic outputs, context windows, embedding drift, prompt injection). Without structure, three problems compound:

Unbounded complexity. A support bot becomes a RAG pipeline, then adds tool calling, then multi-step agents, then human escalation — all in one file. Changes become risky; testing becomes impossible.

Operational blindness. When latency spikes or answers degrade, you need to know: Was retrieval slow? Did the wrong model get routed? Did a tool timeout? Monolithic designs make root-cause analysis guesswork.

Scaling mismatches. Embedding ten million documents is batch work. Answering a user query is latency-sensitive. Coupling them means ingestion jobs starve query traffic, or you over-provision everything.

Architecture solves this by defining layers with clear contracts, async boundaries, and operational hooks (tracing, caching, circuit breakers) at every seam.

Without architecture With layered architecture
Prompt + retrieval + tools in one handler Orchestrator coordinates typed services
One deploy for chunking changes Index versioning independent of API
Auth in system prompt Tenant filters at retrieval and tools
"The LLM is slow" alerts Per-layer latency and error budgets
Framework upgrade = rewrite Frameworks used as libraries behind interfaces

How We Got Here

Production AI architecture did not appear with ChatGPT. It is the composition of earlier systems with an LLM in the generation and decision path.

Diagram: Evolution toward production AI systems

timeline
    title From scripts to production AI platforms
    2015-2018 : Search + ranking stacks
             : Classical NLP pipelines
    2019-2022 : Dense retrieval + vector indexes
             : Early RAG prototypes
    2023 : Chat UIs over single LLM calls
         : Framework chains explode
    2024-2026 : Gateways, agents, eval harnesses
              : Layered platforms with SLOs

Interactive chat demos collapsed into layered platforms once cost, auth, and reliability became requirements.

Era What shipped Gap exposed
Search + ranking Lexical retrieval, feature stores No generative synthesis
Dense retrieval Embeddings + ANN indexes Still mostly search UX
Single-call LLM apps Chat wrappers No grounding, no ops
Framework chains Rapid RAG demos Opaque control flow
Production platforms Orchestration, gateways, eval, guardrails Requires deliberate layering

Public references that shaped practice include Anthropic — Building effective agents, Microsoft's GPT-RAG solution accelerators, and gateway patterns such as LiteLLM / Cloudflare AI Gateway. The consistent lesson: models are interchangeable; architecture is not.

What Is AI System Architecture?

AI system architecture is the structural design of a production LLM application: how data flows from sources to users, which components own which responsibilities, and how the system behaves under load, failure, and change.

It is not a model choice or a framework selection. It is the blueprint that answers:

  • Where does knowledge live, and how is it updated?
  • Who orchestrates multi-step reasoning — application code, a workflow engine, or an agent loop?
  • What happens when the provider returns 503, the vector database is degraded, or retrieval returns zero results?
  • How do you deploy a new reranker without a full system redeploy?

Distinguish three planes that beginners often conflate:

Plane Question it answers Typical owner
Model capabilities What can the LLM do? Model provider / ML
Application engineering How do we call it safely? Product / platform
Knowledge plane What external truth grounds answers? Search / data / KG teams

Parametric knowledge (weights) and retrieved knowledge (indexes, graphs, tools) must stay separable. Fine-tuning changes behavior; RAG and knowledge graphs change what the system can cite. Architecture keeps those concerns from collapsing into one prompt.

How AI System Architecture Works

A production AI system decomposes into layers that can be developed, scaled, and monitored independently.

Layer 1: Ingestion and Indexing (Offline)

Documents, tickets, code, or database rows are parsed, chunked, embedded, and stored. This path is throughput-oriented: batch jobs, queues, retries, idempotent upserts. Latency is measured in minutes or hours, not milliseconds.

Key properties: versioning (embedding model ID in metadata), incremental updates (CDC, webhooks), and separation of raw storage from vector indexes (re-embed without re-parsing). See Chunking Strategies and Embedding Models.

Layer 2: Orchestration (Online Control Plane)

The orchestration layer receives user requests and coordinates execution:

  • Session and state management (history, tenant, user context)
  • Routing (pipeline, model tier, agent vs workflow)
  • Workflow execution (sequential or parallel steps)
  • Tool invocation (schema validation, permissions, timeouts)
  • Error handling (retries, fallbacks, partial responses)
  • Observability emission (spans, cost attribution, structured logs)

Frameworks like LangChain, LangGraph, and LlamaIndex provide primitives. Mature teams often extract a thin custom orchestrator that calls those components as libraries rather than owning the entire stack.

Layer 3: Retrieval (Online Data Plane)

Given a query (or agent-generated sub-query), retrieval returns relevant evidence. Hybrid search, metadata filters, and reranking live here. Authorization must be enforced at the database query level — never rely on the LLM to filter sensitive documents. See Hybrid Search and Metadata Filtering.

Layer 4: Generation (Online Compute Plane)

The LLM synthesizes output from prompt + context + tools. This layer is latency- and cost-sensitive. Streaming, model routing, and prompt caching apply here. See Cost Optimization and Latency Optimization.

Layer 5: Post-processing and Delivery

Output validation, citation formatting, guardrails, logging, and response streaming. Failures caught here prevent bad output from reaching users.

Layer 6: Cross-cutting Platform Services

Caching, AI gateways, evaluation pipelines, and identity are not optional add-ons — they attach at layer boundaries. Model Context Protocol standardizes tool servers; the orchestrator remains the client.

Architecture

Enterprise AI reference architecture showing API gateway, orchestrator, retrieval, model routing, private networking, observability, and governance

Figure 1. Representative enterprise AI platform shape — gateway, authentication, retrieval, AI gateway, orchestration, observability, and governance as separate concerns.

Source: Microsoft Azure GPT-RAG Solution Accelerator

Diagram: Reference AI system architecture

flowchart TB
    Client[Client / Channel] --> GW[API Gateway]
    GW --> Orch[Orchestrator]
    Orch --> Cache[(Cache)]
    Orch --> Ret[Retrieval]
    Orch --> Agent[Agent / Workflow]
    Orch --> Tools[Tool Registry]
    Ret --> VDB[(Vector / Search)]
    Ret --> KG[(Knowledge Graph)]
    Agent --> LLM[AI Gateway / LLM]
    Orch --> LLM
    Orch --> Guard[Guardrails]
    Guard --> Client
    Orch --> Obs[Observability]
    Ingest[Ingestion Workers] --> VDB
    Ingest --> KG

Online path is orchestration-centered; offline ingestion writes indexes the query path only reads.

Layer Owns Does NOT own Scaling pattern
Ingestion Parse, normalize, enqueue Query-time logic Horizontal workers, job queues
Indexing Chunk, embed, upsert User sessions Batch pipelines, scheduled jobs
Orchestration Routing, state, workflows Raw retrieval algorithms Stateless pods, horizontal scale
Retrieval Search, filter, rerank Generation prompts Read replicas, ANN indexes
Generation LLM calls, streaming Document storage Provider limits, regional routing
Post-process Validation, citations, guardrails Business workflows Co-located with API
Platform Auth, cache, gateway, eval Domain prompts Shared services with quotas

Orchestration as control plane

A well-designed orchestrator exposes a stable interface:

from dataclasses import dataclass, field
from typing import Any, Protocol

@dataclass(frozen=True)
class OrchestrationContext:
    trace_id: str
    tenant_id: str
    user_id: str
    session_id: str
    metadata: dict[str, Any] = field(default_factory=dict)

class Orchestrator(Protocol):
    async def handle(self, request: "UserRequest", ctx: OrchestrationContext) -> "Response":
        ...

Concrete shapes behind that interface:

Pattern Control flow Best fit
Fixed pipeline Retrieve → rerank → generate Most document RAG apps
Workflow graph DAG with conditional branches Triage, multi-step forms
Agent loop Plan → act → observe until stop Open-ended tool use — see AI Agents
Multi-agent Handoffs across specialists Large domains — see Multi-Agent Systems

The orchestrator should be framework-agnostic at the interface. Swap LangGraph for custom code without changing your API contract.

Diagram: Orchestration patterns

flowchart LR
    subgraph fixed [Fixed pipeline]
        R1[Retrieve] --> G1[Generate]
    end
    subgraph dag [Workflow DAG]
        C[Classify] --> A[Path A]
        C --> B[Path B]
    end
    subgraph loop [Agent loop]
        P[Plan] --> T[Tool]
        T --> O[Observe]
        O --> P
    end

Choose the least autonomous pattern that meets the task — autonomy increases cost and failure surface.

Contracts between layers

Boundary Contract must include
Gateway → Orchestrator Auth identity, tenant, rate-limit class, trace ID
Orchestrator → Retrieval Query, filters, top_k, timeout, ACL context
Orchestrator → LLM gateway Model tier, max tokens, prompt version, stream flag
Orchestrator → Tools Name, typed args, idempotency key, permission scope
Post-process → Client Validated text, citations, safe fallback on failure

Decision Trade-off

Microservices buy independent scaling and ownership at the cost of distributed tracing and contract versioning. Prefer a modular monolith until retrieval or indexing load diverges from API traffic.

Step-by-Step Flow

Request lifecycle (production RAG + tools)

Diagram: Online request sequence

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant O as Orchestrator
    participant R as Retrieval
    participant L as LLM Gateway
    participant T as Tools
    participant P as Guardrails

    C->>G: Authenticated request
    G->>O: trace_id + tenant
    O->>R: search(filter=tenant)
    R-->>O: chunks
    alt requires tools
        O->>L: complete(tools)
        L-->>O: tool_calls
        O->>T: execute(timeout)
        T-->>O: results
    end
    O->>L: stream(prompt+context)
    L-->>O: tokens
    O->>P: validate
    P-->>C: safe response

Every hop carries tenant and trace context; tool calls are bounded and permission-checked.

  1. API gateway authenticates, rate-limits, attaches trace ID, resolves tenant. Sanitize input before retrieval.
  2. Orchestrator loads session state from Redis/Postgres. Truncate to context budget. Apply prompt-injection defenses on new user turns.
  3. Intent routing (optional) classifies FAQ vs account action vs escalation; routes to model tier (Cost Optimization).
  4. Retrieval runs with tenant filters. Zero results → broaden, clarify, or fallback — do not let the model invent quietly.
  5. Generation builds prompt from template + evidence + history; streams tokens; logs prompt hash and model version.
  6. Tool calls (if agentic) validate schema, check permissions, execute with timeout, inject results; cap iterations.
  7. Post-processing runs guardrails, formats citations, persists the turn, emits observability spans.
  8. Async side effects append to eval datasets, analytics, or human-review queues on low confidence.

Real Production Example

A B2B SaaS company runs a multi-tenant assistant over product docs, API references, and account tools. Shape: FastAPI gateway → custom orchestrator → retrieval (Qdrant) → LiteLLM-style model router → output guardrails.

from __future__ import annotations

import asyncio
import hashlib
from dataclasses import dataclass
from typing import Any, AsyncIterator, Protocol

from pydantic import BaseModel, Field


class UserRequest(BaseModel):
    query: str = Field(min_length=1, max_length=8000)
    session_id: str
    stream: bool = True


@dataclass(frozen=True)
class RequestContext:
    trace_id: str
    tenant_id: str
    user_id: str
    session_id: str


class Retriever(Protocol):
    async def search(self, query: str, *, filter: dict[str, Any], top_k: int) -> list[dict]: ...
    async def rerank(self, query: str, chunks: list[dict], top_n: int) -> list[dict]: ...


class LLMRouter(Protocol):
    async def classify(self, query: str, ctx: RequestContext) -> Any: ...
    def stream(self, model: str, prompt: str) -> AsyncIterator[str]: ...
    async def complete(self, *, model: str, messages: list[dict], tools: list[dict]) -> Any: ...


class ProductionOrchestrator:
    def __init__(
        self,
        retriever: Retriever,
        llm_router: LLMRouter,
        tool_registry,
        guardrails,
        tracer,
        session_store,
    ):
        self.retriever = retriever
        self.llm_router = llm_router
        self.tools = tool_registry
        self.guardrails = guardrails
        self.tracer = tracer
        self.sessions = session_store

    async def handle_query(self, req: UserRequest, ctx: RequestContext) -> AsyncIterator[str]:
        with self.tracer.start_span("orchestrator.handle", attributes={"tenant": ctx.tenant_id}):
            history = await self.sessions.load(ctx.session_id, limit=12)
            route = await self.llm_router.classify(req.query, ctx)

            with self.tracer.start_span("retrieval"):
                chunks = await asyncio.wait_for(
                    self.retriever.search(
                        req.query,
                        filter={"tenant_id": ctx.tenant_id},
                        top_k=20,
                    ),
                    timeout=0.8,
                )
                chunks = await asyncio.wait_for(
                    self.retriever.rerank(req.query, chunks, top_n=5),
                    timeout=0.4,
                )

            if not chunks and getattr(route, "requires_grounding", True):
                yield "I could not find relevant documentation. Rephrase or name a product area."
                return

            prompt = self._build_prompt(req.query, chunks, history)
            prompt_version = hashlib.sha256(prompt.encode()).hexdigest()[:12]

            buffer: list[str] = []
            with self.tracer.start_span(
                "generation",
                attributes={"model": route.model, "prompt_version": prompt_version},
            ):
                async for token in self.llm_router.stream(route.model, prompt):
                    buffer.append(token)
                    if req.stream:
                        yield token

            full = "".join(buffer)
            result = await self.guardrails.validate(
                query=req.query,
                response=full,
                sources=chunks,
            )
            if not result.passed:
                # Replace streamed content only when clients buffer; otherwise emit fallback turn.
                yield result.safe_fallback
                return

            await self.sessions.append(ctx.session_id, user=req.query, assistant=result.text)
            if not req.stream:
                yield result.text

    async def handle_agent_turn(
        self,
        query: str,
        ctx: RequestContext,
        *,
        max_steps: int = 5,
        tool_timeout_s: float = 10.0,
    ) -> str:
        messages = [{"role": "user", "content": query}]
        for step in range(max_steps):
            with self.tracer.start_span(f"agent.step.{step}"):
                response = await self.llm_router.complete(
                    model=os.environ.get("OPENAI_COMPLEX_MODEL", "gpt-5.6-sol"),  # pin current tier via config
                    messages=messages,
                    tools=self.tools.schemas_for(ctx.tenant_id),
                )
                if not response.tool_calls:
                    return response.content

                for call in response.tool_calls:
                    if not self.tools.is_allowed(call.name, ctx):
                        raise PermissionError(f"Tool {call.name} not permitted for tenant")
                    result = await asyncio.wait_for(
                        self.tools.execute(
                            call.name,
                            call.arguments,
                            idempotency_key=f"{ctx.trace_id}:{call.name}:{step}",
                        ),
                        timeout=tool_timeout_s,
                    )
                    messages.append({"role": "tool", "content": result, "name": call.name})

        raise TimeoutError("Agent exceeded max steps")

    def _build_prompt(self, query: str, chunks: list[dict], history: list[dict]) -> str:
        evidence = "\n\n".join(
            f"[{i}] {c.get('text', '')}" for i, c in enumerate(chunks, start=1)
        )
        return (
            "Answer using only the evidence. Cite chunk numbers.\n"
            f"History: {history}\nEvidence:\n{evidence}\n\nQuestion: {query}"
        )

The orchestrator owns flow control. Retrieval, LLM, and tools are injectable services. Every phase emits spans for OpenTelemetry / LangSmith / Langfuse. Timeouts are per dependency, not one global request timeout.

Design Decisions

Decision Option A Option B Choose A when Choose B when
Orchestration Custom code + thin wrappers Full framework graph Strict tests, latency SLOs, long-lived platform Speed to first demo, small team
Deployment Modular monolith Microservices <~100 RPS, one team Independent retrieval/index scale
Control flow Fixed DAG / workflow Agent loop Predictable paths, auditability Open-ended tool decomposition
State Redis sessions Postgres durable history Low-latency chat Compliance / audit trail
Model access Direct provider SDK AI gateway Single provider, simple billing Multi-provider routing, shared cache
Indexing Webhook / CDC upserts Nightly batch Freshness SLAs Cost on huge corpora
Knowledge Vector RAG only Graph + RAG / KG+LLM Document Q&A Multi-hop entity reasoning

Engineering Insight

Frameworks accelerate delivery; they do not define architecture. If you cannot draw layer boundaries on a whiteboard without naming LangChain, the design is incomplete.

Comparisons

Concern Thin orchestrator Framework-centric app Pure agent runtime
Debuggability High (explicit code) Medium (abstractions) Low without traces
Time to demo Slower Fast Fast for tool demos
Latency control Precise Depends on chain design Harder (variable steps)
Cost control Central routing Easy to bury calls Needs hard caps
Best next read This guide LangChain docs AI Agents
Pattern Primary knowledge Typical guide
Enterprise RAG Document corpora Enterprise RAG Architecture
GraphRAG Graph built from documents GraphRAG Architecture
KG + LLM Governed enterprise graph Knowledge Graph + LLM Architecture
Agents Tools + state loop AI Agents
Platform umbrella All of the above This guide

Decision tree: which control flow?

flowchart TD
    Q[Is the path known upfront?] -->|Yes| DAG[Fixed pipeline / DAG]
    Q -->|No| T[Need external side effects?]
    T -->|No| RAG[Retrieve then generate]
    T -->|Yes| Risk[Irreversible actions?]
    Risk -->|Yes| HITL[Agent + HITL gates]
    Risk -->|No| Agent[Bounded agent loop]

Prefer deterministic workflows until tool choice must be dynamic; add HITL when actions mutate money, access, or production systems.

Common Mistakes

  1. No orchestration layer — logic in API handlers. Prompt assembly, retrieval, and tool calls scattered across routes. Extract an orchestrator early.

  2. Coupling indexing and query deployments. Changing chunk size must not require redeploying the API. Version indexes; run blue/green index migrations.

  3. Trusting the LLM for authorization. Tenant filters belong in retrieval queries and tool permission matrices, not in the system prompt.

  4. Missing timeouts on every external call. Vector search, reranker, LLM, and tools each need independent timeouts and budgets.

  5. Framework as architecture. LangChain / LlamaIndex are libraries, not substitutes for defining layers, contracts, and operational boundaries.

  6. No idempotency on write tools. Agent retries can double-charge or duplicate records. Use idempotency keys.

  7. Streaming without backpressure. Unbounded token buffers OOM services under slow clients.

  8. Skipping evaluation gates. Shipping prompt or retrieval changes without golden-set regression is how quality silently decays. See Evaluation and Retrieval Evaluation.

Where It Breaks Down

Cross-cutting experiments resist clean layers. A/B testing a new prompt while comparing retrieval configs needs coordinated experiment infrastructure — architecture helps but does not invent it.

Agent loops blur boundaries. When the model chooses the next step, cost and latency become distributions. Cap steps, budget tokens, and require human approval for destructive actions (Human-in-the-Loop).

Multi-modal and real-time streams break simple ingest-index-query. Live telemetry, video, and collaborative editors need streaming ingestion and different index types.

Org boundaries. Retrieval owned by search, LLM by ML, API by backend — without shared trace IDs and contracts, incidents span three on-call rotations.

Over-centralization. A single "AI platform" team that owns every prompt for every product becomes a bottleneck. Platforms should own gateways, eval harnesses, and shared retrieval; products own domain prompts and tools.

When NOT to Use a Full Platform Architecture

Do not build the full layered platform when:

  • You are validating product-market fit with a single-team prototype under low traffic
  • The use case is offline batch summarization with no interactive SLO
  • You have one data source, one model, and no multi-tenancy requirements
  • Compliance does not yet require audit trails or document-level ACL

In those cases, ship a modular monolith with clear internal modules. Design interfaces as if services will split later — then split when metrics (QPS, index size, team ownership) justify it.

Warning

Premature microservices around a chat demo create distributed failure modes without production traffic to justify them.

Running in Production

Best Practice

Instrument every stage, version embedding models and prompts, enforce access control at retrieval time, and block deploys on golden-set regressions.

Dimension Guidance
Scaling Scale orchestration statelessly. Scale retrieval replicas separately. Scale indexing on queue depth, not request QPS.
Latency Budget per layer (illustrative): gateway ~10ms, retrieval 100–300ms, generation 1–3s. Stream generation. See Latency Optimization.
Cost Orchestrator is the injection point for model routing, caching / semantic caching, and classification. Attribute cost per tenant.
Monitoring Traces across orchestration → retrieval → generation. Alert on p99 per layer, error rate, zero-result retrieval rate. See Observability.
Evaluation End-to-end golden tests at the orchestrator, not only unit tests on prompts.
Security Sanitize at gateway. Tool permission matrix in orchestrator. Secrets never in prompts. See AI Security.
Versioning Log prompt_version, model_id, index_version, embedding_model on every request. Canary at the routing layer.

Important

The orchestration layer is where you enforce timeouts, tenant isolation, and fallbacks. If it is missing, every outage becomes an "LLM outage."

Production checklist

  • Trace ID propagated gateway → orchestrator → retrieval → LLM → tools
  • Per-dependency timeouts and circuit breakers
  • Tenant (or ACL) filters enforced in retrieval queries
  • Tool allow-list + idempotency keys for mutating tools
  • Prompt, model, and index versions logged
  • Streaming with client disconnect / backpressure handling
  • Guardrails on input and output paths
  • Golden eval suite blocking promotion
  • Cost and latency dashboards per tenant / route
  • Runbook for provider 503, empty retrieval, and guardrail failures

Suggested fallbacks

Failure Orchestrator response
Empty retrieval Clarifying question or search broadening
LLM 503 Secondary provider or cached safe response
Tool timeout Partial answer with explicit disclaimer
Guardrail failure Static safe message — never raw blocked text
Cache miss storm Shed non-critical enrichment; protect core path

Ecosystem

Category Representative options Role in the architecture
Orchestration libraries LangChain, LangGraph, LlamaIndex, Haystack, Semantic Kernel Graphs, agents, retrievers as libraries
Workflow / durability Temporal, Inngest, Prefect Long-running jobs, retries, HITL waits
AI gateways LiteLLM, Portkey, Cloudflare AI Gateway Routing, rate limits, provider failover
Retrieval stores Qdrant, Pinecone, Weaviate, OpenSearch, pgvector Online evidence plane
Graph platforms Neo4j, Neptune, Stardog Structured traversal — see KG architectures
Observability OpenTelemetry, Langfuse, LangSmith, Arize Phoenix Traces, eval hooks, cost
Tool protocol MCP servers Standardized tool I/O — orchestrator is client

Tools: LangChain · LangGraph · LlamaIndex · LangSmith

Rankings: Best AI Agent Frameworks · Best Vector Databases

This guide is the platform umbrella. Specialized architecture guides deepen individual patterns without replacing layering.

Architecture series:

Foundations the platform assumes:

Operations:

Diagram: Recommended reading around this guide

flowchart LR
    LLM[LLM Concepts] --> RAG[Retrieval]
    RAG --> KG[Knowledge Graphs]
    KG --> Agents[AI Agents]
    Agents --> Arch[AI System Architecture]
    Arch --> ERAG[Enterprise RAG Arch]
    Arch --> KGLLM[KG + LLM Arch]

Architecture sits above domain guides and fans out into specialized production architectures.

Learning Path

Prerequisites: RAG · Large Language Models · AI Agents

Next topics: Enterprise RAG Architecture · Observability · Cost Optimization · AI Security

Estimated time: 55 min · Difficulty: Advanced

Architecture Series

Production architecture references for designing and operating enterprise AI systems at scale.

Interview Questions

  1. What does the orchestration layer own that the LLM does not?
    Routing, session state, timeouts, tool permissions, fallbacks, and trace emission — control-plane concerns, not token generation.

  2. How do you separate indexing from querying in production?
    Async workers write versioned indexes; the API reads via an alias; promote indexes without redeploying query services; keep raw docs for re-embedding.

  3. Where must multi-tenant authorization be enforced?
    In retrieval filters and tool allow-lists. Prompts are not an access-control mechanism.

  4. When would you choose a fixed DAG over an agent loop?
    When paths are known, latency/cost must be predictable, and auditors need deterministic control flow.

  5. What metrics belong on a per-layer dashboard?
    p50/p95/p99 latency, error rate, timeout rate, zero-result retrieval rate, tokens and $ per request, cache hit rate.

  6. How do gateways change architecture?
    They centralize provider routing, retries, and rate limits so product orchestrators stay provider-agnostic.

  7. What breaks when you skip idempotency on tools?
    Retries duplicate side effects — charges, tickets, emails — especially under agent loops.

  8. How does MCP fit without replacing architecture?
    MCP standardizes tool I/O; retrieval, guardrails, authz, and observability remain first-class layers.

FAQs

What is the orchestration layer in an AI system?

The orchestration layer coordinates request handling: routing, session state, retrieval calls, LLM invocation, tool execution, error handling, and observability. It is application control-plane code, not the model itself.

Should I use LangChain or write custom orchestration?

Use LangChain / LlamaIndex to move fast on prototypes. Extract a custom orchestrator when you need deterministic tests, strict latency budgets, or framework upgrades block releases. Many production teams keep frameworks as libraries behind a thin custom layer.

How do I separate indexing from querying?

Run indexing as async jobs writing to a versioned vector (or graph) index. The query API reads the active alias. Promote new versions without redeploying the API. Store raw documents separately for re-embedding.

When should retrieval be a separate microservice?

When retrieval latency or load diverges from API traffic, when multiple products share one index, or when a search team owns embedding/reranking independently. Before that, a well-modularized monolith is fine.

How do agents fit into layered architecture?

Agents are an orchestration pattern: a bounded loop of plan → tool call → observe. Keep the same layers (retrieval, generation, tools) but replace a fixed DAG with iteration caps, budgets, and approval gates.

What belongs in post-processing vs orchestration?

Orchestration decides what runs and in what order. Post-processing validates and formats output (guardrails, citations, PII redaction) before return. Both can async-log for evaluation.

How do I handle multi-tenancy?

Pass tenant ID through every layer. Enforce metadata filters at retrieval. Separate rate limits and cost attribution per tenant. Never mix tenant data in shared caches without tenant-scoped keys.

What is a reasonable latency budget?

Illustrative interactive RAG: ~50ms embed + 100–300ms retrieve + ~150ms rerank + 1–3s generation. Stream after first token. Budget roughly 2–4s p95 for chat; keep classification/routing steps much faster.

How do I version prompts and models?

Store templates in version control or a config service. Log prompt_version and model_id on every request. Eval before traffic shift. Canary at the orchestrator routing layer.

What fallbacks should the orchestrator implement?

Empty retrieval → clarify or broaden. LLM 503 → secondary provider or cache. Tool timeout → partial answer with disclaimer. Guardrail failure → safe static message.

How does MCP change architecture?

MCP standardizes tool capability exposure. Your orchestrator (or agent runtime) becomes an MCP client. It replaces ad-hoc wrappers but does not replace retrieval, guardrails, or observability.

When is a monolith enough?

Under modest RPS, a single team, and one product surface — a structured monolith with internal modules beats premature microservices. Split when operational or scaling boundaries are clear.

References

Further Reading

Key Takeaways

  • Production AI systems are layered: ingestion, indexing, orchestration, retrieval, generation, post-processing, plus platform services.
  • The orchestration layer is the control plane — routing, state, tools, timeouts, and observability.
  • Separate offline indexing from online queries; version indexes independently of API deploys.
  • Enforce authorization and tenant isolation in retrieval and tools, not in prompts.
  • Specialized architectures (Enterprise RAG, GraphRAG, KG + LLM, Agents) compose inside this blueprint.
  • Start with explicit contracts between layers; add agents and microservices when metrics justify the complexity.

Next Topics

Learning Path

Continue Learning

Related Guides

Related Tools

ToolCategoryPurposeWebsiteBest For
LangChain
PopularOpen SourceAPI
frameworksFramework for building LLM-powered applications and workflows.langchain.comRAG systems
LangGraph
FeaturedOpen SourceAPI
frameworksGraph-based orchestration runtime for long-running, stateful agents.langgraph.devMulti-agent orchestration
LlamaIndex
Open SourceAPI
frameworksData framework for connecting LLMs to private and structured data.llamaindex.aiRAG over documents
LiteLLM
Open SourceAPI
infrastructureUnified API gateway for 100+ LLM providers with routing and fallbacks.litellm.aiMulti-provider routing
Baseten
APICloud
DeploymentPlatform for deploying, serving, and scaling ML and LLM models in production.baseten.coCustom model deployment
OctoAI
APICloud
DeploymentNVIDIA-acquired platform for efficient model deployment and media generation APIs.octo.aiOptimized model endpoints
Beam Cloud
APICloud
DeploymentServerless GPU cloud for running AI workloads, training, and inference.beam.cloudServerless GPU inference
Inferless
APICloud
DeploymentServerless ML inference platform with fast cold starts and custom runtimes.inferless.comLow-latency serverless inference
Anyscale
CloudEnterprise
DeploymentManaged Ray platform for scaling Python, ML, and LLM workloads.anyscale.comDistributed LLM serving
TensorZero
Open SourceCloud
servingOpen-source LLM gateway for routing, observability, and optimization in Rust.tensorzero.comLLM API gateway
Dagster
Open SourceCloud
data processingData orchestrator for building, testing, and monitoring ML and AI pipelines.dagster.ioML pipeline orchestration
Apache Airflow
Open SourceCloud
data processingIndustry-standard workflow orchestration for data and ML pipelines.airflow.apache.orgBatch data pipelines
Windmill
Open SourceCloud
automationOpen-source developer platform for building workflows, scripts, and internal tools.windmill.devInternal tool automation
Trigger.dev
Open SourceCloud
automationOpen-source background jobs platform for long-running AI and TypeScript workflows.trigger.devLong-running AI tasks
Amazon Bedrock
APICloud
cloudManaged multi-model AI platform on AWS with enterprise controls and agents.aws.amazon.comAWS-native GenAI
Google Vertex AI
APICloud
cloudGoogle Cloud AI platform for Gemini, Model Garden, and MLOps pipelines.cloud.google.comGemini on GCP
Groq
APICloud
infrastructureUltra-low-latency LLM inference API powered by custom LPU hardware.groq.comLow-latency chat
Snowflake Cortex
APICloud
cloudLLM and AI functions inside Snowflake for governed enterprise data apps.snowflake.comAI on warehouse data

Related Rankings