AI System Architecture in One Sentence
AI System Architecture =
- Ingestion & Indexing
- Orchestration
- Retrieval
- LLM Generation
- Tools / Agents
- Caching
- Guardrails
- Observability
- Evaluation
TL;DR
-
Production AI systems are layered pipelines, not single API calls — ingestion, indexing, orchestration, retrieval, generation, and post-processing each have different scaling and failure profiles.
-
The orchestration layer is the control plane — it routes requests, manages state, invokes tools, enforces timeouts, and coordinates multi-step workflows without embedding business logic in prompt strings.
-
Separate offline indexing from online query paths — re-embedding documents must not require redeploying the API, and query latency must not depend on batch ingestion jobs.
-
Every layer boundary needs observability, caching, and fallbacks — propagate trace IDs across retrieval and generation, degrade gracefully when a vector store is slow, and never let one failed tool call hang the request path.
-
Specialized architectures compose inside this blueprint — Enterprise RAG, GraphRAG, Knowledge Graph + LLM, and AI Agents are patterns within the same layered system, not replacements for it.
Architecture Snapshot
Complexity
★★★★★
Audience
AI Architects, Platform Engineers, AI Engineers, Tech Leads
Difficulty
Advanced
Typical Deployment
Enterprise Production
Typical Latency
1.5–4 seconds (p95 interactive)
Scalability
Independent scale per layer
Availability Target
99.9%
Read Time
~55 min
Last Updated
July 21, 2026
Recommended Stack
- FastAPI / NestJS orchestrator
- LangGraph / LlamaIndex (as libraries)
- AI Gateway (LiteLLM / Portkey)
- Qdrant / Pinecone / Weaviate
- Redis + Postgres
- OpenTelemetry + Langfuse / LangSmith
Why This Matters
Most AI demos are a single script: embed a query, call a model, return text. Production systems serve concurrent tenants, enforce isolation, integrate with CRMs and internal APIs, survive provider outages, and must be debuggable when a customer reports a wrong answer from three days ago.
Without deliberate architecture, teams accumulate spaghetti: prompts embedded in API handlers, retrieval logic duplicated across endpoints, no way to A/B test a reranker, and incident response that starts with "which service even logged this?" The orchestration layer — often dismissed as glue code — is where reliability, cost control, and security actually live.
If you are moving from prototype to production, or inheriting a system that grew organically, this guide is the reference map. Specialized guides deepen individual patterns; this document defines how those patterns fit together.
Who should read this?
| Reader | Why |
|---|---|
| AI Architects | Define layer boundaries, contracts, and scaling paths across products. |
| Platform Engineers | Operate gateways, indexes, caches, and Kubernetes deployments with clear SLAs. |
| AI Engineers | Implement retrieval, agents, and evaluation without reinventing platform concerns. |
| Technical Leads | Decide monolith vs services, framework vs custom orchestrator, agent vs workflow. |
| Security / Compliance | See where authz, audit, and PII controls must live (not in prompts). |
The Problem AI System Architecture Solves
LLM applications combine familiar backend concerns (APIs, databases, auth) with unfamiliar ones (non-deterministic outputs, context windows, embedding drift, prompt injection). Without structure, three problems compound:
Unbounded complexity. A support bot becomes a RAG pipeline, then adds tool calling, then multi-step agents, then human escalation — all in one file. Changes become risky; testing becomes impossible.
Operational blindness. When latency spikes or answers degrade, you need to know: Was retrieval slow? Did the wrong model get routed? Did a tool timeout? Monolithic designs make root-cause analysis guesswork.
Scaling mismatches. Embedding ten million documents is batch work. Answering a user query is latency-sensitive. Coupling them means ingestion jobs starve query traffic, or you over-provision everything.
Architecture solves this by defining layers with clear contracts, async boundaries, and operational hooks (tracing, caching, circuit breakers) at every seam.
| Without architecture | With layered architecture |
|---|---|
| Prompt + retrieval + tools in one handler | Orchestrator coordinates typed services |
| One deploy for chunking changes | Index versioning independent of API |
| Auth in system prompt | Tenant filters at retrieval and tools |
| "The LLM is slow" alerts | Per-layer latency and error budgets |
| Framework upgrade = rewrite | Frameworks used as libraries behind interfaces |
How We Got Here
Production AI architecture did not appear with ChatGPT. It is the composition of earlier systems with an LLM in the generation and decision path.
Diagram: Evolution toward production AI systems
timeline
title From scripts to production AI platforms
2015-2018 : Search + ranking stacks
: Classical NLP pipelines
2019-2022 : Dense retrieval + vector indexes
: Early RAG prototypes
2023 : Chat UIs over single LLM calls
: Framework chains explode
2024-2026 : Gateways, agents, eval harnesses
: Layered platforms with SLOs
Interactive chat demos collapsed into layered platforms once cost, auth, and reliability became requirements.
| Era | What shipped | Gap exposed |
|---|---|---|
| Search + ranking | Lexical retrieval, feature stores | No generative synthesis |
| Dense retrieval | Embeddings + ANN indexes | Still mostly search UX |
| Single-call LLM apps | Chat wrappers | No grounding, no ops |
| Framework chains | Rapid RAG demos | Opaque control flow |
| Production platforms | Orchestration, gateways, eval, guardrails | Requires deliberate layering |
Public references that shaped practice include Anthropic — Building effective agents, Microsoft's GPT-RAG solution accelerators, and gateway patterns such as LiteLLM / Cloudflare AI Gateway. The consistent lesson: models are interchangeable; architecture is not.
What Is AI System Architecture?
AI system architecture is the structural design of a production LLM application: how data flows from sources to users, which components own which responsibilities, and how the system behaves under load, failure, and change.
It is not a model choice or a framework selection. It is the blueprint that answers:
- Where does knowledge live, and how is it updated?
- Who orchestrates multi-step reasoning — application code, a workflow engine, or an agent loop?
- What happens when the provider returns 503, the vector database is degraded, or retrieval returns zero results?
- How do you deploy a new reranker without a full system redeploy?
Distinguish three planes that beginners often conflate:
| Plane | Question it answers | Typical owner |
|---|---|---|
| Model capabilities | What can the LLM do? | Model provider / ML |
| Application engineering | How do we call it safely? | Product / platform |
| Knowledge plane | What external truth grounds answers? | Search / data / KG teams |
Parametric knowledge (weights) and retrieved knowledge (indexes, graphs, tools) must stay separable. Fine-tuning changes behavior; RAG and knowledge graphs change what the system can cite. Architecture keeps those concerns from collapsing into one prompt.
How AI System Architecture Works
A production AI system decomposes into layers that can be developed, scaled, and monitored independently.
Layer 1: Ingestion and Indexing (Offline)
Documents, tickets, code, or database rows are parsed, chunked, embedded, and stored. This path is throughput-oriented: batch jobs, queues, retries, idempotent upserts. Latency is measured in minutes or hours, not milliseconds.
Key properties: versioning (embedding model ID in metadata), incremental updates (CDC, webhooks), and separation of raw storage from vector indexes (re-embed without re-parsing). See Chunking Strategies and Embedding Models.
Layer 2: Orchestration (Online Control Plane)
The orchestration layer receives user requests and coordinates execution:
- Session and state management (history, tenant, user context)
- Routing (pipeline, model tier, agent vs workflow)
- Workflow execution (sequential or parallel steps)
- Tool invocation (schema validation, permissions, timeouts)
- Error handling (retries, fallbacks, partial responses)
- Observability emission (spans, cost attribution, structured logs)
Frameworks like LangChain, LangGraph, and LlamaIndex provide primitives. Mature teams often extract a thin custom orchestrator that calls those components as libraries rather than owning the entire stack.
Layer 3: Retrieval (Online Data Plane)
Given a query (or agent-generated sub-query), retrieval returns relevant evidence. Hybrid search, metadata filters, and reranking live here. Authorization must be enforced at the database query level — never rely on the LLM to filter sensitive documents. See Hybrid Search and Metadata Filtering.
Layer 4: Generation (Online Compute Plane)
The LLM synthesizes output from prompt + context + tools. This layer is latency- and cost-sensitive. Streaming, model routing, and prompt caching apply here. See Cost Optimization and Latency Optimization.
Layer 5: Post-processing and Delivery
Output validation, citation formatting, guardrails, logging, and response streaming. Failures caught here prevent bad output from reaching users.
Layer 6: Cross-cutting Platform Services
Caching, AI gateways, evaluation pipelines, and identity are not optional add-ons — they attach at layer boundaries. Model Context Protocol standardizes tool servers; the orchestrator remains the client.
Architecture
Figure 1. Representative enterprise AI platform shape — gateway, authentication, retrieval, AI gateway, orchestration, observability, and governance as separate concerns.
Source: Microsoft Azure GPT-RAG Solution Accelerator
Diagram: Reference AI system architecture
flowchart TB
Client[Client / Channel] --> GW[API Gateway]
GW --> Orch[Orchestrator]
Orch --> Cache[(Cache)]
Orch --> Ret[Retrieval]
Orch --> Agent[Agent / Workflow]
Orch --> Tools[Tool Registry]
Ret --> VDB[(Vector / Search)]
Ret --> KG[(Knowledge Graph)]
Agent --> LLM[AI Gateway / LLM]
Orch --> LLM
Orch --> Guard[Guardrails]
Guard --> Client
Orch --> Obs[Observability]
Ingest[Ingestion Workers] --> VDB
Ingest --> KG
Online path is orchestration-centered; offline ingestion writes indexes the query path only reads.
| Layer | Owns | Does NOT own | Scaling pattern |
|---|---|---|---|
| Ingestion | Parse, normalize, enqueue | Query-time logic | Horizontal workers, job queues |
| Indexing | Chunk, embed, upsert | User sessions | Batch pipelines, scheduled jobs |
| Orchestration | Routing, state, workflows | Raw retrieval algorithms | Stateless pods, horizontal scale |
| Retrieval | Search, filter, rerank | Generation prompts | Read replicas, ANN indexes |
| Generation | LLM calls, streaming | Document storage | Provider limits, regional routing |
| Post-process | Validation, citations, guardrails | Business workflows | Co-located with API |
| Platform | Auth, cache, gateway, eval | Domain prompts | Shared services with quotas |
Orchestration as control plane
A well-designed orchestrator exposes a stable interface:
from dataclasses import dataclass, field
from typing import Any, Protocol
@dataclass(frozen=True)
class OrchestrationContext:
trace_id: str
tenant_id: str
user_id: str
session_id: str
metadata: dict[str, Any] = field(default_factory=dict)
class Orchestrator(Protocol):
async def handle(self, request: "UserRequest", ctx: OrchestrationContext) -> "Response":
...
Concrete shapes behind that interface:
| Pattern | Control flow | Best fit |
|---|---|---|
| Fixed pipeline | Retrieve → rerank → generate | Most document RAG apps |
| Workflow graph | DAG with conditional branches | Triage, multi-step forms |
| Agent loop | Plan → act → observe until stop | Open-ended tool use — see AI Agents |
| Multi-agent | Handoffs across specialists | Large domains — see Multi-Agent Systems |
The orchestrator should be framework-agnostic at the interface. Swap LangGraph for custom code without changing your API contract.
Diagram: Orchestration patterns
flowchart LR
subgraph fixed [Fixed pipeline]
R1[Retrieve] --> G1[Generate]
end
subgraph dag [Workflow DAG]
C[Classify] --> A[Path A]
C --> B[Path B]
end
subgraph loop [Agent loop]
P[Plan] --> T[Tool]
T --> O[Observe]
O --> P
end
Choose the least autonomous pattern that meets the task — autonomy increases cost and failure surface.
Contracts between layers
| Boundary | Contract must include |
|---|---|
| Gateway → Orchestrator | Auth identity, tenant, rate-limit class, trace ID |
| Orchestrator → Retrieval | Query, filters, top_k, timeout, ACL context |
| Orchestrator → LLM gateway | Model tier, max tokens, prompt version, stream flag |
| Orchestrator → Tools | Name, typed args, idempotency key, permission scope |
| Post-process → Client | Validated text, citations, safe fallback on failure |
Decision Trade-off
Microservices buy independent scaling and ownership at the cost of distributed tracing and contract versioning. Prefer a modular monolith until retrieval or indexing load diverges from API traffic.
Step-by-Step Flow
Request lifecycle (production RAG + tools)
Diagram: Online request sequence
sequenceDiagram
participant C as Client
participant G as Gateway
participant O as Orchestrator
participant R as Retrieval
participant L as LLM Gateway
participant T as Tools
participant P as Guardrails
C->>G: Authenticated request
G->>O: trace_id + tenant
O->>R: search(filter=tenant)
R-->>O: chunks
alt requires tools
O->>L: complete(tools)
L-->>O: tool_calls
O->>T: execute(timeout)
T-->>O: results
end
O->>L: stream(prompt+context)
L-->>O: tokens
O->>P: validate
P-->>C: safe response
Every hop carries tenant and trace context; tool calls are bounded and permission-checked.
- API gateway authenticates, rate-limits, attaches trace ID, resolves tenant. Sanitize input before retrieval.
- Orchestrator loads session state from Redis/Postgres. Truncate to context budget. Apply prompt-injection defenses on new user turns.
- Intent routing (optional) classifies FAQ vs account action vs escalation; routes to model tier (Cost Optimization).
- Retrieval runs with tenant filters. Zero results → broaden, clarify, or fallback — do not let the model invent quietly.
- Generation builds prompt from template + evidence + history; streams tokens; logs prompt hash and model version.
- Tool calls (if agentic) validate schema, check permissions, execute with timeout, inject results; cap iterations.
- Post-processing runs guardrails, formats citations, persists the turn, emits observability spans.
- Async side effects append to eval datasets, analytics, or human-review queues on low confidence.
Real Production Example
A B2B SaaS company runs a multi-tenant assistant over product docs, API references, and account tools. Shape: FastAPI gateway → custom orchestrator → retrieval (Qdrant) → LiteLLM-style model router → output guardrails.
from __future__ import annotations
import asyncio
import hashlib
from dataclasses import dataclass
from typing import Any, AsyncIterator, Protocol
from pydantic import BaseModel, Field
class UserRequest(BaseModel):
query: str = Field(min_length=1, max_length=8000)
session_id: str
stream: bool = True
@dataclass(frozen=True)
class RequestContext:
trace_id: str
tenant_id: str
user_id: str
session_id: str
class Retriever(Protocol):
async def search(self, query: str, *, filter: dict[str, Any], top_k: int) -> list[dict]: ...
async def rerank(self, query: str, chunks: list[dict], top_n: int) -> list[dict]: ...
class LLMRouter(Protocol):
async def classify(self, query: str, ctx: RequestContext) -> Any: ...
def stream(self, model: str, prompt: str) -> AsyncIterator[str]: ...
async def complete(self, *, model: str, messages: list[dict], tools: list[dict]) -> Any: ...
class ProductionOrchestrator:
def __init__(
self,
retriever: Retriever,
llm_router: LLMRouter,
tool_registry,
guardrails,
tracer,
session_store,
):
self.retriever = retriever
self.llm_router = llm_router
self.tools = tool_registry
self.guardrails = guardrails
self.tracer = tracer
self.sessions = session_store
async def handle_query(self, req: UserRequest, ctx: RequestContext) -> AsyncIterator[str]:
with self.tracer.start_span("orchestrator.handle", attributes={"tenant": ctx.tenant_id}):
history = await self.sessions.load(ctx.session_id, limit=12)
route = await self.llm_router.classify(req.query, ctx)
with self.tracer.start_span("retrieval"):
chunks = await asyncio.wait_for(
self.retriever.search(
req.query,
filter={"tenant_id": ctx.tenant_id},
top_k=20,
),
timeout=0.8,
)
chunks = await asyncio.wait_for(
self.retriever.rerank(req.query, chunks, top_n=5),
timeout=0.4,
)
if not chunks and getattr(route, "requires_grounding", True):
yield "I could not find relevant documentation. Rephrase or name a product area."
return
prompt = self._build_prompt(req.query, chunks, history)
prompt_version = hashlib.sha256(prompt.encode()).hexdigest()[:12]
buffer: list[str] = []
with self.tracer.start_span(
"generation",
attributes={"model": route.model, "prompt_version": prompt_version},
):
async for token in self.llm_router.stream(route.model, prompt):
buffer.append(token)
if req.stream:
yield token
full = "".join(buffer)
result = await self.guardrails.validate(
query=req.query,
response=full,
sources=chunks,
)
if not result.passed:
# Replace streamed content only when clients buffer; otherwise emit fallback turn.
yield result.safe_fallback
return
await self.sessions.append(ctx.session_id, user=req.query, assistant=result.text)
if not req.stream:
yield result.text
async def handle_agent_turn(
self,
query: str,
ctx: RequestContext,
*,
max_steps: int = 5,
tool_timeout_s: float = 10.0,
) -> str:
messages = [{"role": "user", "content": query}]
for step in range(max_steps):
with self.tracer.start_span(f"agent.step.{step}"):
response = await self.llm_router.complete(
model=os.environ.get("OPENAI_COMPLEX_MODEL", "gpt-5.6-sol"), # pin current tier via config
messages=messages,
tools=self.tools.schemas_for(ctx.tenant_id),
)
if not response.tool_calls:
return response.content
for call in response.tool_calls:
if not self.tools.is_allowed(call.name, ctx):
raise PermissionError(f"Tool {call.name} not permitted for tenant")
result = await asyncio.wait_for(
self.tools.execute(
call.name,
call.arguments,
idempotency_key=f"{ctx.trace_id}:{call.name}:{step}",
),
timeout=tool_timeout_s,
)
messages.append({"role": "tool", "content": result, "name": call.name})
raise TimeoutError("Agent exceeded max steps")
def _build_prompt(self, query: str, chunks: list[dict], history: list[dict]) -> str:
evidence = "\n\n".join(
f"[{i}] {c.get('text', '')}" for i, c in enumerate(chunks, start=1)
)
return (
"Answer using only the evidence. Cite chunk numbers.\n"
f"History: {history}\nEvidence:\n{evidence}\n\nQuestion: {query}"
)
The orchestrator owns flow control. Retrieval, LLM, and tools are injectable services. Every phase emits spans for OpenTelemetry / LangSmith / Langfuse. Timeouts are per dependency, not one global request timeout.
Design Decisions
| Decision | Option A | Option B | Choose A when | Choose B when |
|---|---|---|---|---|
| Orchestration | Custom code + thin wrappers | Full framework graph | Strict tests, latency SLOs, long-lived platform | Speed to first demo, small team |
| Deployment | Modular monolith | Microservices | <~100 RPS, one team | Independent retrieval/index scale |
| Control flow | Fixed DAG / workflow | Agent loop | Predictable paths, auditability | Open-ended tool decomposition |
| State | Redis sessions | Postgres durable history | Low-latency chat | Compliance / audit trail |
| Model access | Direct provider SDK | AI gateway | Single provider, simple billing | Multi-provider routing, shared cache |
| Indexing | Webhook / CDC upserts | Nightly batch | Freshness SLAs | Cost on huge corpora |
| Knowledge | Vector RAG only | Graph + RAG / KG+LLM | Document Q&A | Multi-hop entity reasoning |
Engineering Insight
Frameworks accelerate delivery; they do not define architecture. If you cannot draw layer boundaries on a whiteboard without naming LangChain, the design is incomplete.
Comparisons
| Concern | Thin orchestrator | Framework-centric app | Pure agent runtime |
|---|---|---|---|
| Debuggability | High (explicit code) | Medium (abstractions) | Low without traces |
| Time to demo | Slower | Fast | Fast for tool demos |
| Latency control | Precise | Depends on chain design | Harder (variable steps) |
| Cost control | Central routing | Easy to bury calls | Needs hard caps |
| Best next read | This guide | LangChain docs | AI Agents |
| Pattern | Primary knowledge | Typical guide |
|---|---|---|
| Enterprise RAG | Document corpora | Enterprise RAG Architecture |
| GraphRAG | Graph built from documents | GraphRAG Architecture |
| KG + LLM | Governed enterprise graph | Knowledge Graph + LLM Architecture |
| Agents | Tools + state loop | AI Agents |
| Platform umbrella | All of the above | This guide |
Decision tree: which control flow?
flowchart TD
Q[Is the path known upfront?] -->|Yes| DAG[Fixed pipeline / DAG]
Q -->|No| T[Need external side effects?]
T -->|No| RAG[Retrieve then generate]
T -->|Yes| Risk[Irreversible actions?]
Risk -->|Yes| HITL[Agent + HITL gates]
Risk -->|No| Agent[Bounded agent loop]
Prefer deterministic workflows until tool choice must be dynamic; add HITL when actions mutate money, access, or production systems.
Common Mistakes
-
No orchestration layer — logic in API handlers. Prompt assembly, retrieval, and tool calls scattered across routes. Extract an orchestrator early.
-
Coupling indexing and query deployments. Changing chunk size must not require redeploying the API. Version indexes; run blue/green index migrations.
-
Trusting the LLM for authorization. Tenant filters belong in retrieval queries and tool permission matrices, not in the system prompt.
-
Missing timeouts on every external call. Vector search, reranker, LLM, and tools each need independent timeouts and budgets.
-
Framework as architecture. LangChain / LlamaIndex are libraries, not substitutes for defining layers, contracts, and operational boundaries.
-
No idempotency on write tools. Agent retries can double-charge or duplicate records. Use idempotency keys.
-
Streaming without backpressure. Unbounded token buffers OOM services under slow clients.
-
Skipping evaluation gates. Shipping prompt or retrieval changes without golden-set regression is how quality silently decays. See Evaluation and Retrieval Evaluation.
Where It Breaks Down
Cross-cutting experiments resist clean layers. A/B testing a new prompt while comparing retrieval configs needs coordinated experiment infrastructure — architecture helps but does not invent it.
Agent loops blur boundaries. When the model chooses the next step, cost and latency become distributions. Cap steps, budget tokens, and require human approval for destructive actions (Human-in-the-Loop).
Multi-modal and real-time streams break simple ingest-index-query. Live telemetry, video, and collaborative editors need streaming ingestion and different index types.
Org boundaries. Retrieval owned by search, LLM by ML, API by backend — without shared trace IDs and contracts, incidents span three on-call rotations.
Over-centralization. A single "AI platform" team that owns every prompt for every product becomes a bottleneck. Platforms should own gateways, eval harnesses, and shared retrieval; products own domain prompts and tools.
When NOT to Use a Full Platform Architecture
Do not build the full layered platform when:
- You are validating product-market fit with a single-team prototype under low traffic
- The use case is offline batch summarization with no interactive SLO
- You have one data source, one model, and no multi-tenancy requirements
- Compliance does not yet require audit trails or document-level ACL
In those cases, ship a modular monolith with clear internal modules. Design interfaces as if services will split later — then split when metrics (QPS, index size, team ownership) justify it.
Warning
Premature microservices around a chat demo create distributed failure modes without production traffic to justify them.
Running in Production
Best Practice
Instrument every stage, version embedding models and prompts, enforce access control at retrieval time, and block deploys on golden-set regressions.
| Dimension | Guidance |
|---|---|
| Scaling | Scale orchestration statelessly. Scale retrieval replicas separately. Scale indexing on queue depth, not request QPS. |
| Latency | Budget per layer (illustrative): gateway ~10ms, retrieval 100–300ms, generation 1–3s. Stream generation. See Latency Optimization. |
| Cost | Orchestrator is the injection point for model routing, caching / semantic caching, and classification. Attribute cost per tenant. |
| Monitoring | Traces across orchestration → retrieval → generation. Alert on p99 per layer, error rate, zero-result retrieval rate. See Observability. |
| Evaluation | End-to-end golden tests at the orchestrator, not only unit tests on prompts. |
| Security | Sanitize at gateway. Tool permission matrix in orchestrator. Secrets never in prompts. See AI Security. |
| Versioning | Log prompt_version, model_id, index_version, embedding_model on every request. Canary at the routing layer. |
Important
The orchestration layer is where you enforce timeouts, tenant isolation, and fallbacks. If it is missing, every outage becomes an "LLM outage."
Production checklist
- Trace ID propagated gateway → orchestrator → retrieval → LLM → tools
- Per-dependency timeouts and circuit breakers
- Tenant (or ACL) filters enforced in retrieval queries
- Tool allow-list + idempotency keys for mutating tools
- Prompt, model, and index versions logged
- Streaming with client disconnect / backpressure handling
- Guardrails on input and output paths
- Golden eval suite blocking promotion
- Cost and latency dashboards per tenant / route
- Runbook for provider 503, empty retrieval, and guardrail failures
Suggested fallbacks
| Failure | Orchestrator response |
|---|---|
| Empty retrieval | Clarifying question or search broadening |
| LLM 503 | Secondary provider or cached safe response |
| Tool timeout | Partial answer with explicit disclaimer |
| Guardrail failure | Static safe message — never raw blocked text |
| Cache miss storm | Shed non-critical enrichment; protect core path |
Ecosystem
| Category | Representative options | Role in the architecture |
|---|---|---|
| Orchestration libraries | LangChain, LangGraph, LlamaIndex, Haystack, Semantic Kernel | Graphs, agents, retrievers as libraries |
| Workflow / durability | Temporal, Inngest, Prefect | Long-running jobs, retries, HITL waits |
| AI gateways | LiteLLM, Portkey, Cloudflare AI Gateway | Routing, rate limits, provider failover |
| Retrieval stores | Qdrant, Pinecone, Weaviate, OpenSearch, pgvector | Online evidence plane |
| Graph platforms | Neo4j, Neptune, Stardog | Structured traversal — see KG architectures |
| Observability | OpenTelemetry, Langfuse, LangSmith, Arize Phoenix | Traces, eval hooks, cost |
| Tool protocol | MCP servers | Standardized tool I/O — orchestrator is client |
Related Technologies
- RAG — dominant retrieve-then-generate pattern inside the architecture
- Enterprise RAG Architecture — production envelope for document RAG
- GraphRAG Architecture — graph-indexed retrieval over document-derived graphs
- Knowledge Graph + LLM Architecture — governed enterprise graph + LLM
- AI Agents — bounded autonomy as an orchestration pattern
- Observability — tracing and metrics across layers
- Guardrails — policy enforcement before and after generation
- Model Context Protocol — tool server standard
- Tokens / Context Windows — budgets the orchestrator must enforce
Tools: LangChain · LangGraph · LlamaIndex · LangSmith
Rankings: Best AI Agent Frameworks · Best Vector Databases
Related Guides
This guide is the platform umbrella. Specialized architecture guides deepen individual patterns without replacing layering.
Architecture series:
- Enterprise RAG Architecture
- Knowledge Graph + LLM Architecture
- Enterprise Knowledge Graph Architecture
- GraphRAG Architecture
Foundations the platform assumes:
- Large Language Models · Prompt Engineering · Function Calling
- RAG · Hybrid Search · Embeddings
- Knowledge Graphs · AI Agents
Operations:
Diagram: Recommended reading around this guide
flowchart LR
LLM[LLM Concepts] --> RAG[Retrieval]
RAG --> KG[Knowledge Graphs]
KG --> Agents[AI Agents]
Agents --> Arch[AI System Architecture]
Arch --> ERAG[Enterprise RAG Arch]
Arch --> KGLLM[KG + LLM Arch]
Architecture sits above domain guides and fans out into specialized production architectures.
Learning Path
Prerequisites: RAG · Large Language Models · AI Agents
Next topics: Enterprise RAG Architecture · Observability · Cost Optimization · AI Security
Estimated time: 55 min · Difficulty: Advanced
Architecture Series
Production architecture references for designing and operating enterprise AI systems at scale.
Interview Questions
-
What does the orchestration layer own that the LLM does not?
Routing, session state, timeouts, tool permissions, fallbacks, and trace emission — control-plane concerns, not token generation. -
How do you separate indexing from querying in production?
Async workers write versioned indexes; the API reads via an alias; promote indexes without redeploying query services; keep raw docs for re-embedding. -
Where must multi-tenant authorization be enforced?
In retrieval filters and tool allow-lists. Prompts are not an access-control mechanism. -
When would you choose a fixed DAG over an agent loop?
When paths are known, latency/cost must be predictable, and auditors need deterministic control flow. -
What metrics belong on a per-layer dashboard?
p50/p95/p99 latency, error rate, timeout rate, zero-result retrieval rate, tokens and $ per request, cache hit rate. -
How do gateways change architecture?
They centralize provider routing, retries, and rate limits so product orchestrators stay provider-agnostic. -
What breaks when you skip idempotency on tools?
Retries duplicate side effects — charges, tickets, emails — especially under agent loops. -
How does MCP fit without replacing architecture?
MCP standardizes tool I/O; retrieval, guardrails, authz, and observability remain first-class layers.
FAQs
What is the orchestration layer in an AI system?
The orchestration layer coordinates request handling: routing, session state, retrieval calls, LLM invocation, tool execution, error handling, and observability. It is application control-plane code, not the model itself.
Should I use LangChain or write custom orchestration?
Use LangChain / LlamaIndex to move fast on prototypes. Extract a custom orchestrator when you need deterministic tests, strict latency budgets, or framework upgrades block releases. Many production teams keep frameworks as libraries behind a thin custom layer.
How do I separate indexing from querying?
Run indexing as async jobs writing to a versioned vector (or graph) index. The query API reads the active alias. Promote new versions without redeploying the API. Store raw documents separately for re-embedding.
When should retrieval be a separate microservice?
When retrieval latency or load diverges from API traffic, when multiple products share one index, or when a search team owns embedding/reranking independently. Before that, a well-modularized monolith is fine.
How do agents fit into layered architecture?
Agents are an orchestration pattern: a bounded loop of plan → tool call → observe. Keep the same layers (retrieval, generation, tools) but replace a fixed DAG with iteration caps, budgets, and approval gates.
What belongs in post-processing vs orchestration?
Orchestration decides what runs and in what order. Post-processing validates and formats output (guardrails, citations, PII redaction) before return. Both can async-log for evaluation.
How do I handle multi-tenancy?
Pass tenant ID through every layer. Enforce metadata filters at retrieval. Separate rate limits and cost attribution per tenant. Never mix tenant data in shared caches without tenant-scoped keys.
What is a reasonable latency budget?
Illustrative interactive RAG: ~50ms embed + 100–300ms retrieve + ~150ms rerank + 1–3s generation. Stream after first token. Budget roughly 2–4s p95 for chat; keep classification/routing steps much faster.
How do I version prompts and models?
Store templates in version control or a config service. Log prompt_version and model_id on every request. Eval before traffic shift. Canary at the orchestrator routing layer.
What fallbacks should the orchestrator implement?
Empty retrieval → clarify or broaden. LLM 503 → secondary provider or cache. Tool timeout → partial answer with disclaimer. Guardrail failure → safe static message.
How does MCP change architecture?
MCP standardizes tool capability exposure. Your orchestrator (or agent runtime) becomes an MCP client. It replaces ad-hoc wrappers but does not replace retrieval, guardrails, or observability.
When is a monolith enough?
Under modest RPS, a single team, and one product surface — a structured monolith with internal modules beats premature microservices. Split when operational or scaling boundaries are clear.
References
- Anthropic — Building effective agents
- Microsoft Azure GPT-RAG Solution Accelerator
- LangChain Documentation
- LlamaIndex Documentation
- OpenAI API Documentation
- OpenTelemetry Documentation
Further Reading
- Anthropic Documentation
- Google AI for Developers
- LiteLLM Documentation
- Model Context Protocol Specification
- Pinecone Learning Center
Key Takeaways
- Production AI systems are layered: ingestion, indexing, orchestration, retrieval, generation, post-processing, plus platform services.
- The orchestration layer is the control plane — routing, state, tools, timeouts, and observability.
- Separate offline indexing from online queries; version indexes independently of API deploys.
- Enforce authorization and tenant isolation in retrieval and tools, not in prompts.
- Specialized architectures (Enterprise RAG, GraphRAG, KG + LLM, Agents) compose inside this blueprint.
- Start with explicit contracts between layers; add agents and microservices when metrics justify the complexity.