Enterprise RAG Architecture Guide

TL;DR

• Enterprise RAG extends the RAG fundamentals pattern with authentication, tenant isolation, hybrid retrieval, model routing, caching, guardrails, and continuous evaluation — not with more prompt engineering.

• The architecture is a pipeline of independently scalable services, each with clear ownership: gateway, orchestration, retrieval, generation, post-processing, and async evaluation.

• Hybrid retrieval with metadata filtering and re-ranking is the production baseline — vector-only search is a prototype shortcut, not an enterprise design.

• Every request must be traceable end-to-end — trace ID, tenant ID, prompt version, model version, retrieved chunk IDs, latency per stage, and token cost.

• Security and governance are architectural concerns, not afterthoughts: RBAC, PII handling, audit logs, and document-level access control must be enforced before retrieval, not after generation.

On this page

• Why This Matters
• The Problem Enterprise RAG Architecture Solves
• How We Got Here
• What Is Enterprise RAG Architecture?
• How Enterprise RAG Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Use Enterprise RAG
• Running in Production
• Related Guides
• Interview Questions
• FAQs

Why This Matters

Enterprise RAG Architecture is the system design for deploying retrieval-augmented generation in production environments where correctness, security, latency, cost, and compliance are non-negotiable. If you have read the RAG guide, you already understand indexing, chunking, and the retrieve-then-generate pattern. This document answers a different question: how do you engineer a RAG platform that survives enterprise load, passes security review, and improves over time?

The intended readers are AI architects defining boundaries, platform engineers operating gateways and indexes, AI and ML engineers implementing retrieval and evaluation, and technical leads making build-versus-buy decisions. A prototype proves that retrieval can improve an answer. An enterprise architecture proves that the answer came from authorized, current evidence; that the request stayed within latency and cost budgets; and that an operator can reconstruct the decision later.

Reference Architecture

Figure 1. Enterprise Retrieval-Augmented Generation reference architecture showing API gateway, authentication, retrieval, AI gateway, model routing, private networking, orchestration, observability, and governance.

Source: Microsoft Azure GPT-RAG Solution Accelerator

An enterprise RAG system is not a single application. It is a composed architecture of services:

Production teams treat RAG as infrastructure, not a feature flag on a chatbot. The retrieval layer, generation layer, and governance layer each have separate deployment units, on-call ownership, and SLAs.

The reference diagram illustrates a key enterprise principle: no component talks to another over the public internet. API gateways, orchestrators, vector indexes, and LLM endpoints communicate through private endpoints within a controlled network boundary. Your cloud provider may differ; the pattern does not.

The Problem Enterprise RAG Architecture Solves

Moving from a demo to production exposes problems that prompts cannot solve: one tenant can retrieve another tenant's content, index freshness drifts from source systems, provider incidents halt every request, and teams cannot tell whether a bad answer came from retrieval or generation. Enterprise RAG solves those failures by making identity, retrieval policy, routing, and telemetry explicit system boundaries.

The architecture is governed by the following principles:

How We Got Here

Enterprise RAG is one stage in a progression — each level adds capability without replacing what came before.

Diagram: RAG architecture maturity

timeline
    title From basic retrieval to governed knowledge systems
    Basic RAG : Dense retrieval
              : Retrieve then generate
    Hybrid RAG : Vector plus BM25
               : Metadata filters and reranking
    Enterprise RAG : Identity and tenant isolation
                   : Gateways, evals, and SLOs
    GraphRAG : Community and relationship retrieval
             : Multi-hop document synthesis
    KG plus LLM : Governed entities and ontology
                : Deterministic graph traversal

Each stage addresses a failure exposed by the previous one; graph techniques extend rather than replace a sound retrieval baseline.

Basic RAG covers indexing, retrieval, and generation — the pattern in the RAG fundamentals guide. Hybrid Search adds keyword + vector retrieval with metadata filters. Enterprise RAG (this guide) wraps the pipeline in authentication, governance, routing, caching, guardrails, and observability. GraphRAG introduces graph-based community retrieval for complex multi-hop questions. Knowledge Graph + LLM combines structured graph traversal with language model reasoning. Enterprise Knowledge Graph scales graph infrastructure across the organization with ontology governance and production operations.

Teams do not skip levels. A prototype that jumps straight to Enterprise Knowledge Graph without solid hybrid retrieval will fail in production. Each layer assumes the previous one works.

What Is Enterprise RAG Architecture?

Enterprise RAG architecture is the production envelope around the retrieve-then-generate pattern. It combines an offline knowledge pipeline with an authenticated online request path and cross-cutting controls for authorization, quality, resilience, cost, and auditability. Teams that graduate from a proof-of-concept encounter failures that the fundamentals guide does not address. These are architectural problems, not prompt problems.

Important

Enterprise RAG failures are almost always architectural — retrieval boundaries, missing ACL filters, absent observability — not fixable by prompt tuning alone.

Hallucinations at scale

In a demo, a wrong answer is embarrassing. In production, it is a liability. Enterprise RAG must enforce grounded generation — answers cite retrieved evidence, faithfulness checks run before responses reach users, and low-confidence answers trigger escalation rather than fabrication. See Hallucination Detection and Guardrails.

Latency under load

A 3-second response time with ten concurrent users becomes unacceptable at ten thousand. Enterprise architecture separates hot paths (cache hits, pre-filtered retrieval) from cold paths (full hybrid search + re-rank + generation). Streaming and parallel retrieval are mandatory, not optional.

Enterprise security

RAG systems expose proprietary documents to LLM APIs. Enterprise architecture requires:

• Authentication before any retrieval
• Document-level access control enforced at query time
• PII detection and redaction before context reaches the LLM
• Private networking between services
• Secrets managed through vaults, not environment variables in code

See AI Security for threat models specific to RAG.

Important

Security Consideration: Document-level ACLs must be enforced inside the retrieval query. Post-filtering search results after vector similarity scoring is both a security defect and a recall problem.

Governance and compliance

Regulated industries require audit trails: who asked what, what documents were retrieved, which model version generated the answer, and whether the response passed policy checks. Governance is an architectural layer — not a logging statement added after launch.

Scalability

Document corpora grow from thousands to millions of chunks. Indexing pipelines must handle incremental updates without full re-indexing. Retrieval must scale horizontally with read replicas and partitioned indexes. The orchestration layer must remain stateless.

Cost

Uncontrolled LLM usage is expensive. Enterprise RAG implements cost optimization through semantic caching, model routing (cheap models for simple queries), token budgets, and retrieval limits that cap context size before generation.

Document freshness

Stale indexes produce confident wrong answers. Enterprise systems need CDC-driven incremental indexing, freshness metadata on chunks, and monitoring that alerts when indexed content lags source systems by more than a defined SLA.

Authentication and authorization

Every query carries identity context. Retrieval filters must include tenant ID, user role, and document ACLs. Never rely on the LLM to "know" what a user should see.

Multi-model deployments

Enterprise teams rarely depend on a single LLM provider. Architecture must support routing across GPT, Claude, Gemini, and open models with failover, A/B testing, and per-tenant model policies. See Model Routing patterns.

How Enterprise RAG Works

The system has two paths. The offline path parses, classifies, chunks, embeds, and upserts documents with tenant, ACL, source, freshness, and embedding-version metadata. The online path authenticates a caller, rewrites the query when needed, executes hybrid retrieval with ACL filters, re-ranks candidates, assembles bounded context, routes generation by workload, validates output, and emits an auditable trace.

Hybrid retrieval plus ACL filters plus cross-encoder re-ranking is the July 2026 production baseline. Vector-only retrieval misses exact identifiers; lexical-only retrieval misses paraphrases; post-filtered authorization can leak data. The three controls must operate as one retrieval contract.

Architecture

Diagram: Enterprise RAG layers and request pipeline

flowchart TB
    Sources[Enterprise Sources] --> Ingest[Parse and Chunk]
    Ingest --> Embed[Embed and Version]
    Embed --> Search[(Hybrid Search Index)]
    Client[Client] --> Gateway[API Gateway and Auth]
    Gateway --> Orch[Stateless Orchestrator]
    Orch --> Cache[(Tenant Cache)]
    Orch --> Rewrite[Query Rewrite]
    Rewrite --> Retrieve[Hybrid Retrieve plus ACL]
    Retrieve --> Search
    Retrieve --> Rank[Cross-Encoder Rerank]
    Rank --> Router[AI Gateway and Model Router]
    Router --> Models[Workload-Routed Models]
    Models --> Guard[Guardrails and Citations]
    Guard --> Client
    Orch --> Observe[Traces, Evals, and Audit]

Offline ingestion writes versioned evidence; the online path enforces identity before retrieval and observes every boundary.

The complete architecture uses logical service boundaries. Smaller deployments may colocate them in a modular monolith, but their contracts should remain separable.

Important

Running vector search over the full corpus and filtering ACLs in application code is both a data-exposure defect and a relevance defect. Unauthorized candidates must never enter reranking, cache, prompt, or logs.

Workload-based model routing

Use GPT-5.6 Luna or Claude Haiku 4.5 for classification and rewrites; GPT-5.6 Terra, Claude Sonnet 5, or Gemini 3.7 Flash for balanced interactive work; and GPT-5.6 Sol or Claude Opus 4.8 for the hardest synthesis. These are routing tiers, not permanent defaults. Pin deployable IDs in configuration, canary changes, evaluate each route, and retain cross-provider failover.

Step-by-Step Flow

The following flow describes a single user query through the enterprise RAG pipeline. Each step is a separate span in your distributed trace.

Diagram: Authenticated enterprise RAG request

sequenceDiagram
    participant U as User
    participant G as Gateway
    participant O as Orchestrator
    participant R as Retrieval
    participant M as Model Gateway
    participant P as Policy
    participant E as Eval and Audit
    U->>G: Query plus identity token
    G->>O: tenant, roles, trace_id
    O->>R: hybrid search with ACL filters
    R-->>O: reranked authorized chunks
    O->>M: workload tier plus evidence
    M-->>O: streamed grounded answer
    O->>P: citations, PII, faithfulness
    P-->>U: validated answer
    O-->>E: async trace and sample

The identity and trace context cross every hop; evaluation writes remain off the user-facing critical path.

Step-by-step detail:

1. User Request — Client sends question + session ID. Request body is validated against schema.

2. Authentication — Gateway validates JWT. Extracts tenant_id, user_id, roles. Rejects unauthenticated requests with 401.

3. Prompt Management — Orchestrator fetches active prompt template (v2.3.1) from Prompt Manager. Injects tenant-specific instructions.

4. Cache Check — Semantic cache keyed by (tenant_id, embedding(query)). On hit, skip retrieval and generation — return cached response. Typical hit rate: 15–40% for FAQ-heavy workloads.

5. Query Rewriting — Rewrite "What about remote workers?" to "What is the PTO policy for remote employees hired after 2023?" using conversation history.

6. Retrieval — Hybrid retriever runs parallel vector + keyword search with filters: tenant_id=X AND acl_groups INTERSECTS user.roles. Returns 30 candidates.

7. Re-ranking — Cross-encoder scores 30 pairs. Top 5 chunks above relevance threshold proceed to context assembly.

8. Generation — AI Gateway routes to Claude Sonnet 5 (or GPT-5.6 Terra by eval). Prompt contains system instructions + 5 chunks + conversation history. Streaming enabled.

9. Guardrails — Output scanned for PII leakage, faithfulness to retrieved context, and policy compliance. Blocked responses trigger fallback message.

10. Response — Client receives streamed answer with [1][2] citations mapped to source URLs. trace_id included for support escalation.

Production Tip

Production Advice: Emit async observability and evaluation events after the response is delivered. Never block the user-facing path on logging or eval writes.

Real Production Example

Consider a global company operating an HR and policy assistant for 40 business units. Employees ask about leave, travel, benefits, and country-specific employment policy. Documents come from SharePoint and an HRIS; each chunk carries tenant_id, country, employment_type, effective_from, and acl_groups. A US contractor must not retrieve a German works-council policy or an executives-only compensation document.

The online service makes authorization part of the search query and routes by workload. A short policy lookup can use GPT-5.6 Luna, Claude Haiku 4.5, or Gemini 3.7 Flash; nuanced policy reconciliation can use GPT-5.6 Terra or Claude Sonnet 5; only difficult cross-policy synthesis is eligible for GPT-5.6 Sol or Claude Opus 4.8. The router chooses from current evaluated deployments and always has a cross-provider fallback.

from dataclasses import dataclass
from typing import Any

@dataclass(frozen=True)
class Identity:
    tenant_id: str
    user_id: str
    country: str
    employment_type: str
    acl_groups: tuple[str, ...]

async def answer_policy_question(query: str, identity: Identity) -> dict[str, Any]:
    filters = {
        "tenant_id": identity.tenant_id,
        "country": {"$in": [identity.country, "global"]},
        "employment_type": {"$in": [identity.employment_type, "all"]},
        "acl_groups": {"$overlap": list(identity.acl_groups)},
        "status": "active",
    }
    candidates = await retriever.hybrid_search(
        query=query, filters=filters, top_k=30, timeout_s=0.8
    )
    evidence = await reranker.rank(query, candidates, top_n=5, timeout_s=0.4)
    if not evidence:
        return {"answer": "I could not find an applicable policy.", "citations": []}

    route = await model_router.select(
        query=query,
        allowed_tiers=("economy", "standard", "premium"),
        fallback_provider=True,
    )
    result = await generator.generate(
        model=route.model_id,
        evidence=evidence,
        require_citations=True,
        timeout_s=12,
    )
    return await guardrails.validate(result, identity=identity, sources=evidence)

The important detail is not the framework. The retrieval API requires filters, timeouts are dependency-specific, empty evidence produces a refusal, model IDs come from evaluated routing policy, and validation receives the same identity and sources used by generation.

Design Decisions

Key architectural decisions encountered when moving from prototype to enterprise RAG. Document these as ADRs in your repository.

Engineering Insight

Engineering Tip: Write ADRs when you choose — not after the incident. Future engineers (including you in six months) need the why, not just the what.

The decisions interact. Run vector and BM25 retrieval in parallel, apply tenant_id, acl_groups, date, document type, and language filters before scoring, then rerank. Version prompts and indexes independently; never change an embedding model without building and evaluating a new index. Use tenant-scoped semantic caching, stream safely, and attribute cost per route.

For recovery, snapshot indexes and retain raw parsed documents so they can be rebuilt. Keep prompts and configuration in version control, replicate session state only when the product requires continuity, and test restore—not just backup creation. Audit records should include trace, user and tenant, rewritten query, chunk IDs, model and prompt versions, guardrail outcome, latency, tokens, and cost; minimize or hash sensitive text according to retention policy.

Comparisons

Decision tree: choose a retrieval architecture

flowchart TD
    Q[Need grounded answers?] -->|No| LLM[Direct LLM or workflow]
    Q -->|Yes| T[Multi-tenant or regulated?]
    T -->|No| M[Basic or hybrid RAG]
    T -->|Yes| H[Need multi-hop relations?]
    H -->|No| E[Enterprise RAG]
    H -->|Yes| G[Curated graph exists?]
    G -->|No| GR[GraphRAG]
    G -->|Yes| KG[KG plus LLM]
    E --> A[Need dynamic actions?]
    A -->|Yes| EA[Enterprise RAG plus bounded agent]
    A -->|No| E

Choose the simplest architecture that meets authorization and reasoning requirements; add agents only when dynamic actions are part of the job.

Common Mistakes

Example enterprise tech stack

A representative stack for a B2B SaaS company deploying enterprise RAG:

This stack is illustrative. Smaller teams may combine layers (e.g., LiteLLM as both AI Gateway and router). Larger teams may split retrieval into a dedicated microservice with its own on-call rotation.

Production readiness checklist

Use this checklist before promoting an enterprise RAG deployment to production traffic.

• Authentication — OIDC/JWT validated at API gateway on every request
• Authorization (RBAC) — Document-level ACLs enforced in retrieval queries
• Hybrid Retrieval — Vector + keyword search with RRF merge
• Metadata Filtering — Tenant and ACL filters applied before similarity scoring
• Re-ranking — Cross-encoder rescores top-20+ candidates before generation
• Prompt Versioning — Templates versioned, logged, and rollback-tested
• AI Gateway — All LLM/embedding calls routed through a single gateway
• Model Routing — Complexity-based routing with failover model configured
• Semantic Cache — Tenant-scoped cache with appropriate TTL
• Observability — Distributed tracing with per-stage latency and token metrics
• Evaluation Pipeline — Nightly golden-set eval in CI/CD
• Guardrails — Input and output validation before responses reach users
• Disaster Recovery — Index snapshots, prompt rollback, and tested restore procedure
• Cost Monitoring — Per-tenant, per-model cost dashboards with alerts
• Rate Limiting — Per-user and per-tenant limits at the API gateway

Where It Breaks Down

Enterprise RAG is not a universal reasoning engine. It breaks down when the source corpus cannot support the question, authorization metadata is incomplete, or answers depend on relationships that flat chunks cannot preserve.

Retrieval metrics can also look healthy while answer quality degrades. Recall@k does not measure citation faithfulness; faithfulness does not measure policy correctness. Separate retrieval, generation, and human outcome evaluations, then correlate them through one trace.

When NOT to Use Enterprise RAG

Enterprise RAG adds operational complexity that is justified only when production requirements demand it. Do not build this architecture when:

• Personal projects — a script with a vector store and an API call is sufficient
• Quick prototypes — validate retrieval quality with standard RAG first
• Small document collections — under ~1,000 documents rarely needs multi-service orchestration
• Single-user tools — no multi-tenancy, no RBAC, no per-user ACL filtering
• No authentication requirements — internal demos where every user sees the same corpus
• No governance needs — no audit trail, compliance review, or data residency constraints
• No compliance requirements — HIPAA, SOC 2, or GDPR obligations are what drive most enterprise controls

For these cases, use the RAG fundamentals guide pattern: index documents, retrieve with hybrid search if needed, generate with an LLM. Add enterprise layers only when security review, multi-tenancy, or operational SLAs require them.

Running in Production

Operate the system as a collection of explicit SLOs rather than one chatbot uptime number. Track gateway rejection rate, retrieval p95/p99, zero-result rate, reranker score distribution, time to first token, grounded-answer rate, cache hit rate, index freshness, and cost per tenant. Observability explains trace design; Evaluation and Cost Optimization cover release gates and budgets.

Best Practice

Use hybrid retrieval with ACL filters and re-ranking on every production path. Block prompt, model, embedding, index, or reranker changes unless a tenant-representative golden set passes.

Release and operations checklist

• OIDC/JWT validation occurs at the gateway; tenant identity propagates end to end
• ACL and tenant filters are mandatory fields in the retrieval contract
• Hybrid candidate retrieval and re-ranking have independent timeouts and metrics
• Prompt, model, embedding, reranker, and index versions appear in every trace
• Model routing is workload-based, evaluated, and configured with provider failover
• Tenant-scoped cache keys and invalidation behavior have isolation tests
• Golden-set evaluation blocks regressions in retrieval and generation separately
• Index freshness and CDC lag have alerts and an owned runbook
• Guardrails cover prompt injection, PII, grounding, and safe refusal
• Provider 429/503, empty retrieval, and cache-stampede failure drills have been run
• Cost, latency, and quality dashboards support per-tenant drill-down
• Index backup, blue/green promotion, and rollback procedures are tested

Related Guides

This architecture guide builds on the RAG fundamentals. Read that first if you have not already.

Retrieval layer:

• RAG — core pattern and indexing/querying phases
• Hybrid Search — combining vector and keyword retrieval
• Semantic Search — meaning-based document matching
• Vector Search — ANN search mechanics
• Vector Databases — storage and scaling
• Chunking Strategies — index-time text segmentation
• Embedding Models — model selection and versioning
• Metadata Filtering — ACL and tenant isolation
• Re-ranking — precision improvement before generation
• Retrieval Evaluation — measuring recall and relevance

Architecture series:

• AI System Architecture — the broader production platform blueprint
• GraphRAG Architecture — graph-based retrieval for complex questions
• Knowledge Graph + LLM Architecture — governed graphs with language models
• Enterprise Knowledge Graph Architecture — organization-scale graph operations

Operations layer:

• Caching — response and embedding caching
• Semantic Caching — similarity-based cache hits
• Guardrails — input/output safety and policy
• Observability — tracing and monitoring
• Evaluation — quality measurement frameworks
• AI System Architecture — broader AI platform design

Diagram: Enterprise RAG learning path

flowchart LR
    RAG[RAG] --> Hybrid[Hybrid Search]
    Hybrid --> Rank[Reranking]
    Rank --> ERAG[Enterprise RAG]
    ERAG --> Obs[Observability and Eval]
    ERAG --> Graph[GraphRAG Arch]
    ERAG --> KG[KG plus LLM Arch]

Build reliable hybrid retrieval first, then add operations or graph-based reasoning according to measured failure modes.

Production architecture references for designing and operating enterprise AI systems at scale.

Tools: LangChain · LlamaIndex · Pinecone · Weaviate · Qdrant

Ranking: Best Vector Databases

Interview Questions

1. Why is hybrid retrieval plus ACL filtering plus re-ranking a production baseline?
   Hybrid retrieval covers semantic and exact matching, query-time ACLs prevent unauthorized evidence from entering the candidate set, and re-ranking restores precision before generation.

2. Where should document authorization be enforced?
   Inside the retrieval query using tenant and ACL metadata. Prompts and post-retrieval filtering are not security boundaries.

3. How do you separate retrieval failures from generation failures?
   Maintain retrieval-labeled and answer-labeled datasets, record candidate and reranker scores, and evaluate each stage independently under the same trace ID.

4. How should an enterprise RAG system route models?
   Route by workload, risk, latency, cost, tenant policy, and evaluated quality. Never make one model a permanent default; retain cross-provider fallback.

5. What must be versioned to reproduce an answer?
   Prompt, model deployment, embedding model, index, chunking configuration, reranker, guardrail policy, and source document revision.

6. When is a separate retrieval service justified?
   When multiple products share retrieval, query load scales differently from orchestration, or a search team owns relevance and its SLO independently.

7. How do you migrate embedding models without downtime?
   Build a new versioned index from retained source data, evaluate it, dual-read or canary it, then atomically move an alias and retain rollback.

8. What causes cross-tenant leakage besides a missing filter?
   Shared cache keys, copied chunks with stale ACLs, permissive defaults, asynchronous entitlement drift, and traces that log raw sensitive context.

9. When should Enterprise RAG evolve into GraphRAG or KG + LLM?
   Use GraphRAG when document-derived relationships and multi-hop synthesis dominate; use KG + LLM when governed entities and deterministic traversal already exist.

10. Which production metrics matter beyond uptime?
    Zero-result rate, recall@k, reranker score distribution, grounded-answer rate, citation correctness, index freshness, p95/p99 by stage, cache hit rate, and cost per tenant.

Key Takeaways

• Enterprise RAG is a governed production architecture around RAG, not a larger prompt.
• Hybrid retrieval, query-time ACL filtering, and re-ranking form one security-and-quality baseline.
• Separate offline indexing from the online query path and version every artifact needed to reproduce an answer.
• Route among GPT-5.6 Sol/Terra/Luna, Claude Sonnet 5/Opus 4.8/Haiku 4.5, Gemini 3.7 Flash, and other approved deployments by workload and evaluation—not permanent default.
• Trace and evaluate retrieval and generation independently; bad answers have different failure domains.
• Adopt GraphRAG, KG + LLM, or bounded agents only when measured requirements justify their additional complexity.

FAQs

How is enterprise RAG different from the RAG pattern in the fundamentals guide?

The RAG guide explains the retrieve-then-generate pattern, chunking, and retrieval mechanics. Enterprise RAG adds the operational envelope: multi-tenancy, security, governance, caching, model routing, observability, and continuous evaluation. The core pattern is the same; the architecture around it is what changes.

Do I need all 15 components from day one?

No. Start with gateway, orchestrator, hybrid retriever, LLM, and basic logging. Add re-ranking, caching, guardrails, and evaluation as you approach production traffic. The interfaces should be designed for all components from the start, even if implementations are deferred.

Should I use one vector database or multiple?

One vector database per environment is typical. Partition by tenant (separate indexes or metadata filters) within that database. Multiple databases add operational complexity without benefit unless regulatory requirements mandate physical isolation.

How do I handle multi-tenant data isolation?

Enforce tenant_id as a mandatory metadata filter on every retrieval query. Test isolation with automated cross-tenant access tests. For strict isolation requirements, use separate indexes per tenant.

What is the minimum team to operate enterprise RAG?

At minimum: one AI engineer (pipeline + eval), one platform engineer (infra + observability), and one security reviewer. Larger deployments add dedicated retrieval engineers and MLOps.

How often should I run evaluation?

Nightly automated eval on golden datasets. Full eval on every prompt or model version change in CI/CD. Ad-hoc eval when users report quality issues.

When should I add GraphRAG?

When flat chunk retrieval fails on questions requiring multi-hop reasoning across documents — see GraphRAG. Most enterprise Q&A systems do not need GraphRAG on day one.

How do I choose between self-hosted and managed vector databases?

Managed (Pinecone, Azure AI Search) for faster time-to-production. Self-hosted (Milvus, Qdrant, Weaviate) when you need data sovereignty, custom hardware, or cost control at billion-vector scale.

What latency should I target?

How do I migrate from a POC to enterprise architecture?

1. Extract retrieval into a service with ACL filters.
2. Add AI Gateway for LLM calls.
3. Add re-ranking.
4. Add observability.
5. Add eval pipeline.
6. Add caching.
7. Add guardrails.

Each step is independently valuable.

References

• Azure GPT-RAG Solution Accelerator
• Azure AI Search — RAG Overview
• Baseline Microsoft Foundry Chat Reference Architecture
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Production RAG Guide
• LlamaIndex Production RAG

Further Reading

• Pinecone — Enterprise RAG Guide
• Weaviate — RAG Architecture Patterns
• RAGAS Evaluation Framework
• Portkey — AI Gateway Documentation
• LiteLLM — Production Proxy