Knowledge Graph + LLM Architecture Guide

TL;DR

• Knowledge Graph + LLM Architecture combines governed structured knowledge with language model reasoning — the graph holds facts, relationships, and provenance; the LLM interprets intent, synthesizes answers, and explains results. Neither replaces the other.

• Graph-first retrieval is the production baseline — resolve entities, traverse relationships, collect evidence, then generate. Vector search supplements graph traversal; it does not replace it.

• Entity resolution must run before retrieval — ambiguous names, duplicate records, and cross-system identifiers destroy answer quality when skipped. See Entity Resolution patterns.

• Every response must be explainable and auditable — cite graph paths, source systems, ontology versions, and query plans. Regulated enterprises require lineage from answer to authoritative record.

• Never fine-tune or prompt-memorize knowledge that belongs in the graph — governed enterprise facts live in the knowledge graph with schema validation, versioning, and access control. The LLM reasons over retrieved evidence; it is not the system of record.

Why This Matters

Knowledge Graph + LLM Architecture is the production system design for grounding language models in governed enterprise knowledge — not document chunks alone. If RAG retrieves text and GraphRAG Architecture extracts graphs from documents, this guide covers traversable facts from source systems with entity resolution, semantic parsing, and audit-grade explainability.

Who should read this?

Why Vector Search Cannot Replace Graph Traversal

Vector search retrieves semantically similar content—documents or entity descriptions whose embeddings sit close to the query in vector space. That excels at paraphrase matching and unstructured recall. Knowledge graphs retrieve structurally connected information: explicit relationships typed in the schema, traversable paths, and constraints enforced by the ontology.

Similarity is not the same as an explicit relationship. Two entities may embed near each other without sharing a governed edge. A vector index cannot answer "which suppliers of Component X are subsidiaries of a sanctioned entity three hops away?" because that requires following typed paths, applying filters at each hop, and respecting access-controlled subgraphs—not ranking by cosine distance.

Multi-hop reasoning requires graph traversal. Each hop must use authoritative identifiers and relationship types from source systems, not approximate neighborhood search. When entity resolution links CRM, ERP, and compliance records, only a graph query can walk from product → component → supplier → jurisdiction with reproducible evidence paths.

Production AI systems typically combine vector retrieval, graph traversal, and keyword search rather than replacing one with another. Vector search surfaces relevant document passages and candidate entities; graph traversal assembles governed facts and relationship chains; keyword search handles exact identifiers and codes. Collapsing enterprise reasoning into a single vector index loses structure, explainability, and the deterministic guarantees regulated deployments require.

Architecture

System overview

Knowledge Graph + LLM Architecture is the system design for combining governed enterprise knowledge graphs with large language models in production. If you have read the Knowledge Graphs and Enterprise Knowledge Graphs guides, you already understand triples, ontologies, and graph modeling. If you have read Enterprise RAG Architecture, you understand retrieval pipelines, gateways, and operational controls. This document answers a different question: how do you architect a production system where the knowledge graph and the LLM each do what they are best at?

Enterprises adopt this pattern when unstructured document RAG is insufficient:

The LLM's role is intent understanding, query formulation, synthesis, and explanation. The knowledge graph's role is authoritative facts, relationships, constraints, provenance, and access-controlled traversal. Collapsing both into a single vector index or a fine-tuned model violates the engineering rule at the top of this guide.

Reference Architecture

Source: Microsoft GraphRAG

Note: Production KG + LLM systems ground generation in governed graph traversal; the diagram illustrates graph-aware retrieval feeding the LLM context window.

A production Knowledge Graph + LLM system is a composed architecture of services:

Source: Microsoft GraphRAG

Production teams treat the knowledge graph as infrastructure — the semantic layer that outlives any single LLM vendor or prompt version.

Diagram: Knowledge Graph + LLM production architecture

flowchart TB
    subgraph sources [Enterprise sources]
        ERP[ERP / CRM / MDM]
        DOC[Document corpora]
    end
    subgraph graph [Graph platform]
        ING[Ingestion + SHACL]
        KG[Knowledge graph]
        ER[Entity resolution]
    end
    subgraph ai [AI layer]
        SP[Semantic parser]
        RET[Hybrid retriever]
        GW[AI gateway]
        LLM[LLM synthesis]
    end
    ERP --> ING --> KG
    DOC --> ING
    KG --> ER
    ER --> SP
    ER --> RET
    RET --> LLM
    SP --> KG
    GW --> LLM

Facts originate in source systems; the LLM synthesizes and explains — it is not the system of record.

Diagram: Hybrid retrieval — graph + vector

flowchart LR
    Q[Query] --> EL[Entity linking]
    EL --> GT[Graph traversal]
    EL --> VS[Vector search on linked docs]
    GT --> MERGE[Evidence ranking]
    VS --> MERGE
    MERGE --> CTX[Context builder]
    CTX --> LLM[LLM]

Graph traversal is primary; hybrid search supplements unstructured evidence linked to graph nodes.

Architecture principles

How We Got Here

Architecture maturity progression

Knowledge Graph + LLM is one stage in an enterprise AI progression — each level adds capability without replacing what came before.

Enterprise Search
    ↓
RAG
    ↓
Enterprise RAG
    ↓
Knowledge Graph + LLM
    ↓
GraphRAG
    ↓
Enterprise AI Platform

Enterprise Search provides keyword lookup across siloed indexes. RAG adds vector retrieval over document chunks. Enterprise RAG wraps document retrieval in security, governance, and operational controls. Knowledge Graph + LLM (this guide) grounds reasoning in governed structured knowledge with entity resolution and explainable traversal. GraphRAG adds community detection and summarization over graph structure for complex multi-hop discovery. Enterprise AI Platform unifies graph, vector, agent, and governance services under a single operational model.

Diagram: Enterprise AI architecture progression

timeline
    title Path to Knowledge Graph + LLM
    section Retrieval
        RAG : Document chunks
        Enterprise RAG : Hybrid + security
    section Graph
        KG + LLM : Governed traversal
        GraphRAG : Document-derived graph
    section Platform
        Enterprise AI : Unified graph + vector + agents

Teams do not skip Enterprise RAG fundamentals — hybrid retrieval patterns still apply to linked document corpora.

Teams do not skip levels. A team that deploys Knowledge Graph + LLM without Enterprise RAG fundamentals for document corpora will lack hybrid retrieval patterns, caching, and eval infrastructure that this architecture still requires for unstructured evidence.

Comparisons

Knowledge Graph + LLM Architecture solves different problems than Enterprise RAG and GraphRAG Architecture. Use these tables for procurement and ADR decisions.

Traditional RAG vs Knowledge Graph + LLM

Enterprise teams often ask when to move beyond Enterprise RAG. The architectures solve different classes of problems — not a strict upgrade path, but a capability expansion when structured relationships and governed knowledge become requirements.

Traditional RAG

User
 │
Vector Search
 │
LLM
 │
Answer

Knowledge Graph + LLM

User
 │
Entity Resolution
 │
Knowledge Graph
 │
Graph Traversal
 │
Evidence Collection
 │
LLM
 │
Grounded Answer

Move to Knowledge Graph + LLM when questions require joining authoritative records across systems

KG + LLM vs fine-tuning for enterprise knowledge

KG + LLM Architecture vs GraphRAG Architecture

Decision tree: KG + LLM vs RAG vs GraphRAG

flowchart TD
    A[Question type?] --> B{Authoritative system facts?}
    B -->|Yes| C[KG + LLM Architecture]
    B -->|No| D{Document corpus only?}
    D -->|Yes| E{Multi-hop or global themes?}
    E -->|Yes| F[GraphRAG Architecture]
    E -->|No| G[Enterprise RAG]
    C --> H[Hybrid with RAG for linked docs]
    F --> H

Truth location determines architecture — systems of record vs document-derived structure.

, traversing explicit relationships, or audit-grade explainability — not when document search alone suffices.

The Problem Knowledge Graph + LLM Architecture Solves

Enterprises that combine knowledge graphs with LLMs encounter failures that neither graph tutorials nor RAG guides address alone. These are architectural problems spanning data integration, identity, and reasoning boundaries.

Important

Knowledge Graph + LLM failures are almost always integration failures — unresolved entities, stale ontologies, missing provenance, or LLMs asked to answer without graph evidence.

Enterprise data silos

Critical knowledge spans CRM, ERP, data warehouses, MDM systems, and document repositories. No single retrieval index captures cross-system relationships. A knowledge graph unifies these sources into a queryable semantic layer — but only if ingestion, mapping, and entity resolution are architectural concerns, not one-time ETL scripts.

Disconnected systems

When the graph and the LLM operate independently — graph for analytics dashboards, LLM for chatbots — users receive contradictory answers. Architecture must expose a single query interface where graph traversal and LLM synthesis share identity context, authorization, and provenance.

Hallucinations on structured questions

Document RAG hallucinates when chunks are missing or misranked. Graph + LLM systems hallucinate differently: the LLM invents relationships not present in the graph, or semantic parsing generates invalid queries that return empty results — and the LLM fills the gap with fabrication. Grounding means every user-visible fact traces to a graph node or edge with provenance.

Explainability and compliance

Financial services, healthcare, and manufacturing require audit trails. "The model said so" is not acceptable. Architecture must return evidence subgraphs — the entities, relationships, and source records that support each claim. See Knowledge Graph Best Practices.

Data lineage

Answers must trace to source systems: which ERP record, which MDM golden entity, which regulatory filing. Provenance metadata on every triple or edge is not optional for regulated deployments.

Entity ambiguity

"Johnson & Johnson," "J&J," and internal SKU references may denote different entities in different contexts. Without entity resolution, graph traversal starts from the wrong node — producing confident, plausible, and completely wrong answers.

Reasoning across systems

Questions like "Which active contracts involve suppliers with compliance violations in the last quarter?" require joining legal, procurement, and risk data through graph paths — not semantic similarity over text chunks.

Governance

Who can modify the ontology? Who approves new relationship types? Which triples are authoritative vs. inferred vs. extracted-from-documents-pending-review? Governance is an architectural layer with RBAC, audit logs, and approval workflows.

Knowledge freshness

Source databases change continuously. Incremental ingestion, change data capture, and staleness alerts are production requirements. A graph that lags ERP by 48 hours produces liability in operational decisions.

Important

Security Consideration: Graph traversal must enforce the same RBAC and row-level security as source systems. A user who cannot see a supplier record in SAP must not reach it through an unfiltered Cypher or SPARQL path.

Knowledge Ingestion Pipeline

Source Systems
 │
Mapping + ETL
 │
Entity Resolution
 │
Ontology Validation
 │
Knowledge Graph

Component breakdown

The following components form a complete Knowledge Graph + LLM reference architecture. Each is a logical service boundary — interfaces should remain separable even when co-deployed.

Engineering Insight

Engineering Tip: Define the contract between semantic parsing and graph execution before choosing an LLM vendor. The query plan interface should survive a model migration without changing your API surface.

Knowledge Sources

Purpose: Authoritative enterprise systems that supply facts, relationships, and metadata to the knowledge graph.

Responsibilities:

• Provide structured records (SQL, NoSQL, APIs, MDM golden records)
• Supply semi-structured data (JSON exports, data lake tables)
• Feed unstructured corpora (PDFs, wikis, tickets) for extraction and linking
• Expose change events for incremental synchronization

Technology choices: SAP, Salesforce, Snowflake, Databricks, PostgreSQL, REST APIs, Kafka change feeds, MDM platforms (Reltio, Informatica).

Production considerations: Catalog every source with owner, refresh SLA, and data classification. Never ingest PII without classification tags. Federated access (virtual graphs) vs. materialized ingestion is an ADR — see Production Design Decisions.

Common mistakes: Treating all sources as equal authority. Ingesting without lineage metadata. Ignoring source-system deletion events.

Data Ingestion

Purpose: Transform source data into graph-ready entities, relationships, and literals on a defined schedule or event stream.

Responsibilities:

• Execute ETL/ELT pipelines from source to graph
• Apply mapping rules (source schema → ontology)
• Validate against SHACL or custom constraints
• Handle incremental updates and tombstone deletions
• Log ingestion runs with record counts and error rates

Technology choices: Apache Spark, Airflow, dbt, Stardog Virtual Graphs, Neo4j ETL tools, AWS Glue, Fivetran + custom graph loaders.

Production considerations: Separate batch ingestion from online query paths. Idempotent pipelines — re-running must not duplicate nodes. Monitor ingestion lag per source. See Enterprise Knowledge Graphs.

Common mistakes: Full re-ingestion nightly on billion-triple graphs. No validation gate before write. Silent failures that leave stale subgraphs.

Ontology

Purpose: Define the governed schema — classes, properties, constraints, and business meaning — that all graph data must conform to.

Responsibilities:

• Model domain entities and relationships
• Enforce cardinality, data types, and domain constraints
• Version and publish ontology releases
• Document mappings from source systems to ontology terms
• Support extension without breaking existing queries

Technology choices: OWL, SHACL, RDF Schema, Stardog Designer, TopBraid EDG, Protégé, enterprise ontology repositories.

Production considerations: Store ontologies in Git with semantic versioning. Run SHACL validation on every ingestion batch. Notify downstream consumers on ontology changes. See Ontologies and SHACL.

Common mistakes: Ad-hoc property additions without review. No versioning — impossible to reproduce historical queries. Ontology designed by engineers without domain expert validation.

Knowledge Graph

Purpose: The governed semantic layer combining entities, relationships, attributes, and provenance from all enterprise sources.

Responsibilities:

• Store unified enterprise knowledge as nodes and edges (or triples)
• Maintain provenance and source-system references
• Support inference rules where appropriate
• Expose query interfaces (SPARQL, Cypher, Gremlin)
• Enforce data quality constraints at write time

Technology choices: Stardog, Neo4j, Amazon Neptune, Ontotext GraphDB, Azure Cosmos DB (Gremlin/API for Gremlin), Apache Jena Fuseki.

Production considerations: Choose property graph vs. RDF based on ADR-001. Plan for billions of edges. Separate analytical graph workloads from transactional query paths if needed.

Common mistakes: Using the graph as a cache of LLM extractions without grounding. No provenance on edges. Mixing analytical and operational graphs in one database without isolation.

Entity Resolution

Purpose: Link mentions, records, and identifiers across systems to canonical graph entities before retrieval and traversal.

Responsibilities:

• Match user query mentions to graph entities (NER + linking)
• Resolve cross-system duplicates to golden records
• Maintain confidence scores and human-review queues for ambiguous matches
• Propagate resolution context to query planner

Technology choices: Senzing, Tamr, custom ML matchers, LLM-assisted linking with validation, SPARQL sameAs clusters, Neo4j GDS entity resolution.

Production considerations: Entity resolution is on the critical path — budget 100–500ms. Log resolution decisions for audit. Never auto-merge below confidence threshold without review. See Entity Resolution.

Common mistakes: Skipping resolution for "simple" queries. Using LLM entity guesses without graph validation. No feedback loop from wrong answers to resolution model.

Graph Database

Purpose: Persist and query the knowledge graph at production scale with HA, backup, and access control.

Responsibilities:

• Execute graph queries with predictable latency
• Support indexing (full-text, vector on node properties where needed)
• Replicate for HA and read scaling
• Enforce authentication and authorization at query time
• Provide backup, point-in-time recovery, and monitoring

Technology choices: Stardog Cluster, Neo4j Enterprise / AuraDB, Amazon Neptune, Ontotext GraphDB, Azure Cosmos DB.

Production considerations: Set query timeouts (5–30s depending on use case). Monitor slow queries and supernode patterns. Partition large graphs by domain or tenant if required. See Graph Databases.

Common mistakes: No query timeout — one expensive traversal blocks the cluster. Missing indexes on high-cardinality properties. Treating graph DB as a document store with ad-hoc JSON blobs.

Reasoning Engine

Purpose: Apply inference rules, OWL reasoning, or business logic to expand query results beyond explicitly asserted triples.

Responsibilities:

• Execute OWL-RL / RDFS inference where configured
• Apply custom rule engines (SWRL, Datalog, Stardog rules)
• Expand transitive relationships (subsidiary-of, part-of)
• Materialize inferred triples or compute at query time

Technology choices: Stardog reasoning, Apache Jena inference, GraphDB OWL-Horst, Neo4j custom procedures.

Production considerations: Materialize vs. runtime inference is a performance trade-off — document as ADR. Reasoning explosions (combinatorial inference) must be bounded. Test inference impact on query latency before production.

Common mistakes: Enabling full OWL DL on large graphs without performance testing. No cache for materialized inferences. Reasoning rules that contradict source-system authority.

SPARQL / Cypher Layer

Purpose: Translate natural language intent into executable graph queries and validate results against schema constraints.

Responsibilities:

• Semantic parsing: NL → SPARQL or Cypher
• Query validation against ontology and syntax rules
• Query plan optimization and timeout enforcement
• Parameterized query templates for common patterns
• Result serialization for downstream retriever and LLM

Technology choices: Stardog Voicebox text-to-SPARQL, Neo4j Cypher generation, custom fine-tuned parsers, LangChain GraphCypherQAChain (with validation wrapper).

Production considerations: Never execute LLM-generated queries without validation. Parse → validate → execute → verify non-empty or explicit "no results." Log generated queries for debugging. See SPARQL and Cypher.

Common mistakes: Passing raw LLM output to graph DB without syntax check. No allowlist of relationship types per user role. Unbounded variable-length path queries.

Common Mistake

Common Mistake: Treating text-to-Cypher as a demo feature. In production, unvalidated graph queries are both a security risk (data exfiltration via crafted traversals) and a reliability risk (expensive queries, empty results → hallucination).

Retriever

Purpose: Collect evidence from the knowledge graph and supplementary vector indexes for LLM context assembly.

Responsibilities:

• Execute graph traversals from resolved seed entities
• Run hybrid retrieval: graph paths + vector search over linked documents
• Rank and deduplicate evidence subgraphs
• Apply authorization filters before results enter context
• Return structured evidence packages (nodes, edges, source refs, scores)

Technology choices: LangChain GraphRetriever, LlamaIndex KnowledgeGraphQueryEngine, Stardog path queries, Neo4j GraphRAG retriever, custom FastAPI retrieval service.

Production considerations: Cap traversal depth and result count. Parallelize graph and vector retrieval. Return structured JSON evidence — not prose summaries — for the LLM. See Hybrid Search.

Common mistakes: Returning entire subgraphs without ranking. Vector-only retrieval when the question is relational. No ACL filtering on retrieved nodes.

LLM

Purpose: Interpret user intent, synthesize natural language answers from graph evidence, and explain reasoning paths.

Responsibilities:

• Disambiguate underspecified questions (with user confirmation when needed)
• Synthesize answers strictly from retrieved evidence
• Generate human-readable explanations of graph paths
• Refuse to answer when evidence is insufficient
• Produce structured outputs (tables, JSON) when required

Technology choices: Claude Sonnet 5/Opus, GPT-5.6, Gemini 3.5 Pro/Flash, Llama (self-hosted) — route per workload through the AI Gateway rather than hardcoding one provider.

Production considerations: Cap context to relevant evidence — not the full subgraph. Instruct the model to cite entity IDs and source systems. Log model version per request. Streaming for perceived latency.

Common mistakes: Stuffing raw triple dumps into context. No instruction to refuse when graph returns empty. Using LLM to answer factual questions without graph evidence.

Prompt Manager

Purpose: Centralize prompt templates for semantic parsing, synthesis, and explanation with versioning and rollback.

Responsibilities:

• Store system prompts for evidence-grounded synthesis
• Version control with environment promotion
• Inject ontology context snippets and schema summaries
• Log prompt version with every trace

Technology choices: Langfuse Prompt Management, Git-backed store, Humanloop.

Production considerations: Separate prompts for parsing vs. synthesis vs. explanation. Never hot-edit production prompts without version increment.

Common mistakes: One monolithic prompt for all pipeline stages. No schema context in parsing prompts. Hardcoded prompts in application code.

AI Gateway

Purpose: Unified API layer for all LLM calls with provider abstraction, failover, and cost attribution.

Responsibilities:

• Route parsing calls to fast models, synthesis to capable models
• Provider failover and retry logic
• API key management and vault integration
• Per-tenant cost tracking

Technology choices: Portkey, LiteLLM, Azure AI Gateway.

Production considerations: Parsing steps use economy models (Claude Haiku, Gemini 3.7 Flash, GPT mini tiers). Synthesis uses standard tier. Circuit breakers per provider.

Common mistakes: Single model for all stages. No failover. Direct provider calls from multiple services.

Guardrails

Purpose: Validate inputs and outputs against safety, policy, and grounding rules.

Responsibilities:

• Input: prompt injection detection, scope validation
• Output: faithfulness to graph evidence, PII leakage detection
• Policy: block answers that cite unauthorized entities
• Escalation: route low-confidence or policy-violating responses to review

Technology choices: NeMo Guardrails, Guardrails AI, custom faithfulness checks comparing claims to evidence JSON.

Production considerations: Verify every factual claim maps to a retrieved graph entity. Block answers that introduce entities not in evidence. See Guardrails.

Common mistakes: Guardrails only on input. No grounding check against evidence package. Allowing LLM to "helpfully" add facts beyond retrieval.

Observability

Purpose: Full visibility into graph queries, LLM calls, entity resolution, and end-to-end latency.

Responsibilities:

• Distributed tracing with trace_id across all stages
• Log generated SPARQL/Cypher, execution time, result counts
• Track entity resolution confidence and outcomes
• Per-tenant cost and latency dashboards

Technology choices: Langfuse, OpenTelemetry + Grafana, Datadog.

Production considerations: Alert on query latency SLO breaches, resolution failure rate spikes, and empty-result → generation attempts. See Observability.

Common mistakes: Logging LLM calls but not graph queries. No correlation between resolution failures and bad answers.

Evaluation

Purpose: Continuously measure graph retrieval quality, parsing accuracy, and answer faithfulness.

Responsibilities:

• Maintain golden question sets with expected graph paths
• Measure query parsing accuracy, traversal correctness, answer faithfulness
• Run eval on ontology changes and model updates
• Feed failures back to engineering

Technology choices: RAGAS (adapted for graph), DeepEval, custom pytest + LLM judge, Langfuse datasets.

Production considerations: Eval parsing, retrieval, and generation independently. Include multi-hop and ambiguous-entity cases. See Evaluation and Retrieval Evaluation.

Common mistakes: End-to-end eval only — cannot isolate parsing vs. traversal failures. No golden set for entity resolution edge cases.

Explainability Layer

Purpose: Present evidence paths, source provenance, and confidence to users and auditors alongside natural language answers.

Responsibilities:

• Render subgraph visualizations or path summaries
• Map claims to source system records
• Display ontology version and query plan
• Support export for compliance audit

Technology choices: Custom UI components, Neo4j Bloom, Stardog Studio paths, graph visualization libraries (Cytoscape, vis.js).

Production considerations: Explainability is a first-class API response field — not a UI afterthought. Include evidence_graph, source_systems, ontology_version, query_plan in response schema.

Common mistakes: Showing only natural language with no evidence. Evidence paths too large for users to interpret. No export format for auditors.

Request Flow

User
 │
Entity Recognition
 │
Graph Lookup
 │
Traversal
 │
Evidence
 │
LLM
 │
Grounding
 │
Response

Step-by-Step Flow

Diagram: LLM + knowledge graph query execution

sequenceDiagram
    participant U as User
    participant EL as Entity linker
    participant SP as Semantic parser
    participant G as Knowledge graph
    participant R as Retriever
    participant L as LLM
    U->>EL: natural language question
    EL->>G: resolve canonical entities
    EL->>SP: plan query intent
    SP->>G: validated SPARQL/Cypher
    G-->>R: query results + subgraph
    R->>R: ACL filter + rank evidence
    R->>L: structured evidence package
    L-->>U: grounded answer + graph paths

Never execute unvalidated LLM-generated graph queries in production.

End-to-end request flow

The following flow describes a single user query through the Knowledge Graph + LLM pipeline. Each step is a separate span in your distributed trace.

User Question
    ↓
API Gateway              → TLS termination, rate limit, trace_id assignment
    ↓
Authentication           → Token validation, attach user_id + tenant_id + roles
    ↓
Entity Recognition       → NER + entity linking to canonical graph nodes
    ↓
Knowledge Graph Lookup   → Validate entities exist; load local subgraph context
    ↓
Graph Traversal          → Execute SPARQL/Cypher (parsed or templated) with ACL filters
    ↓
Evidence Collection      → Rank subgraphs; parallel vector retrieval for linked documents
    ↓
Retriever                → Assemble structured evidence package (nodes, edges, provenance)
    ↓
Prompt Construction      → Load synthesis prompt version; inject evidence JSON + schema context
    ↓
AI Gateway               → Route to LLM with assembled context
    ↓
LLM Synthesis            → Generate grounded answer with citations to graph entities
    ↓
Grounding Validation     → Verify claims map to evidence; block ungrounded assertions
    ↓
Guardrails               → Policy check, PII scan, faithfulness validation
    ↓
Response                 → Natural language + evidence paths + trace_id
    ↓
Async: Observability     → Emit metrics, log query plan, update cost counters
    ↓
Async: Evaluation        → Sample into eval dataset

Step-by-step detail:

1. User Question — "Which suppliers of Component X have active compliance violations?" Client sends question + session ID.

2. Authentication — Gateway validates JWT. Extracts tenant_id, user_id, roles. Procurement role required for supplier data.

3. Entity Recognition — NER extracts "Component X." Entity linker resolves to ex:Component_X_4421 with confidence 0.94. Ambiguous matches below 0.8 trigger clarifying question.

4. Knowledge Graph Lookup — Verify ex:Component_X_4421 exists. Load 1-hop supplier relationships and compliance status properties.

5. Graph Traversal — Execute parameterized Cypher: MATCH (c:Component {id: $id})-[:SUPPLIED_BY]->(s:Supplier)-[:HAS_VIOLATION]->(v:Violation) WHERE v.status = 'active' AND s.tenant = $tenant RETURN s, v Query timeout: 10s. ACL filter applied in WHERE clause.

6. Evidence Collection — Traversal returns 3 suppliers with 5 violations. Vector retriever fetches linked audit PDF summaries for top 2 violations. Evidence package: 3 nodes, 5 edges, 2 document chunks.

7. Prompt Construction — Prompt Manager loads synthesis template v3.1.0. Injects evidence JSON + instruction: "Answer only from evidence. Cite entity IDs."

8. LLM Synthesis — Claude Sonnet generates answer with supplier names, violation types, dates, and [entity:sup_8821] citations.

9. Grounding Validation — Automated check: every supplier name in answer maps to a node in evidence package. Violation count matches graph result. Pass.

10. Response — Client receives answer, expandable evidence subgraph, source system refs (SAP MM, GRC module), and trace_id.

Production Tip

Production Advice: Return "I cannot find an answer in the knowledge graph" when traversal returns empty — never allow the LLM to fill gaps with general knowledge for regulated queries.

Technology Choices

Knowledge Graph Platforms

Reasoning and Validation

Graph Query Languages

LLMs

Treat this table as workload routing guidance, not a fixed vendor list — model generations turn over quickly, so pin versions per deployment and revisit routing tiers as new releases ship.

Embeddings

Observability

Architecture decision records

Key architectural decisions for Knowledge Graph + LLM deployments. Document these as ADRs in your repository.

Engineering Insight

Engineering Tip: Write ADRs when you choose — not after the compliance audit. Future engineers need the why behind property graph vs. RDF, not just the decision.

Production Stack Layers

Client
 │
Gateway + Auth
 │
Entity Resolution
 │
Graph Query + Retrieval
 │
LLM Synthesis + Grounding
 │
Explainability + Observability

Design Decisions

Production design decisions

Ontology versioning

Store ontologies in Git with semantic version tags (ontology-v2.4.0). Run downstream compatibility tests on every release. Log ontology_version with every query trace. Breaking changes require migration scripts and consumer notification.

Knowledge synchronization

Implement CDC-driven incremental ingestion from source systems. Define per-source freshness SLAs. Alert when ingestion lag exceeds threshold. Tombstone deletions must remove or mark inactive graph nodes — stale nodes produce wrong answers.

Incremental ingestion

Never full-reload billion-edge graphs nightly. Upsert changed entities and relationships. Track last_modified per source record. Reconciliation jobs detect drift between source and graph.

Entity linking

Maintain a resolution service with confidence thresholds. Below 0.8: ask clarifying question. Below 0.5: refuse to traverse. Log all resolution decisions. Human-review queue for borderline merges.

Hybrid retrieval

Graph traversal for relational evidence. Vector search over document chunks linked to graph entities via mentions or describes edges. Merge and rank with RRF. See Hybrid Search and Enterprise RAG Architecture.

Tip

Performance Tip: Execute graph traversal and vector retrieval in parallel. Sequential execution adds full latency of both paths.

Reasoning performance

Profile inference rules on production-scale data before enabling. Materialize expensive inferences offline. Set query timeouts. Cache frequent query patterns (parameterized templates, not LLM-generated).

Caching

Cache parameterized graph query results with TTL keyed by (tenant_id, query_template, entity_ids). Cache entity resolution results. Do not cache LLM synthesis for regulated queries without provenance checksum validation.

Graph partitioning

Partition by domain (products, suppliers, customers) or tenant for very large graphs. Federated queries across partitions add latency — design boundaries around natural organizational splits.

High availability

Graph database with replication factor ≥ 2 across availability zones. Read replicas for query scaling. Automated failover with tested runbooks. Target 99.9% for query path.

Disaster recovery

Security and RBAC

Map identity provider groups to graph-level permissions. Enforce row-level security in query WHERE clauses — not post-filtering. Test with cross-role access attempts quarterly.

Audit logging

Log: timestamp, trace_id, user_id, tenant_id, question, resolved_entities, generated_query, query_result_count, evidence_node_ids, ontology_version, model, prompt_version, grounding_result, latency_ms. Retain per compliance requirements.

Running in Production

Best practices

1. Keep enterprise knowledge in the graph — not in LLM weights or giant system prompts.
2. Resolve entities before traversal — every query starts from canonical graph nodes.
3. Validate every generated graph query — syntax, schema, and authorization checks before execution.
4. Return evidence with every answer — subgraph, source refs, and query plan.
5. Version ontologies in Git — tag releases; notify downstream consumers.
6. Run SHACL validation on ingestion — reject invalid triples at the gate.
7. Maintain provenance on every edge — source system, record ID, ingestion timestamp.
8. Separate ingestion from query paths — different SLAs, different scaling.
9. Use hybrid retrieval — graph traversal + vector search over linked documents.
10. Cap traversal depth and result count — prevent runaway queries.
11. Set query timeouts — 5–30s depending on use case complexity.
12. Route parsing to economy models — reserve capable models for synthesis.
13. Implement refusal behavior — empty graph results → "cannot answer," not fabrication.
14. Log generated queries — essential for debugging parsing failures.
15. Eval parsing, retrieval, and generation independently — isolate failure domains.
16. Maintain golden datasets with expected graph paths — not just expected text answers.
17. Test entity resolution edge cases — ambiguous names, new entities, cross-language mentions.
18. Enforce RBAC in graph queries — not in application post-filters.
19. Monitor ingestion lag per source — stale data is a production incident.
20. Document architecture decisions as ADRs — especially property graph vs. RDF.
21. Use AI Gateway for all LLM calls — provider abstraction and failover.
22. Version every prompt — log version with every trace.
23. Incremental ingestion only — full reloads do not scale.
24. Plan schema evolution explicitly — new relationship types require review.
25. Read Enterprise RAG Architecture for operational patterns — caching, guardrails, and observability apply here too.

Best Practice

Best Practice: Run the Production Readiness Checklist below before exposing any Knowledge Graph + LLM system to regulated data or external users.

Common Mistakes

Real Production Example

Example enterprise stack

A representative stack for a manufacturing enterprise deploying Knowledge Graph + LLM:

This stack is illustrative. Teams on Neo4j substitute Cypher for SPARQL. AWS-centric deployments may choose Neptune. The architecture pattern — graph first, LLM second, evidence always — remains the same.

Production readiness checklist

Use this checklist before promoting a Knowledge Graph + LLM deployment to production traffic.

• Ontology Versioning — Schema in Git, version logged per query, rollback tested
• SHACL Validation — Data quality constraints enforced on every ingestion batch
• Entity Resolution — Canonical linking with confidence thresholds and audit log
• Provenance — Source system and record ID on every graph edge
• Query Validation — Generated SPARQL/Cypher validated before execution
• RBAC — Graph queries enforce tenant and role filters at query time
• Hybrid Retrieval — Graph traversal + vector search over linked documents
• Grounding Validation — Output checked against evidence package before delivery
• Explainability — Evidence subgraph and query plan returned in API response
• AI Gateway — All LLM calls routed through single gateway with failover
• Prompt Versioning — Templates versioned, logged, and rollback-tested
• Incremental Ingestion — CDC-driven updates with lag monitoring and alerts
• Query Timeouts — Configured per query class; slow query logging enabled
• Observability — Distributed tracing with per-stage latency and graph metrics
• Evaluation Pipeline — Golden sets with expected graph paths; nightly CI eval
• Guardrails — Input/output validation and faithfulness checks
• Disaster Recovery — Graph backup, ontology rollback, tested restore procedure
• Refusal Behavior — Empty graph results return explicit "cannot answer"

Where It Breaks Down

KG + LLM Architecture fails when integration is treated as "connect LLM to Neo4j" without entity resolution, ontology governance, and query validation. Unresolved entities and stale ontologies produce confident wrong answers — worse than RAG hallucination because users trust "graph-backed" labels.

See Knowledge Graph + LLM for integration patterns. This guide addresses platform architecture for production scale.

When NOT to Use KG + LLM

Knowledge Graph + LLM Architecture adds significant data engineering and governance overhead. Do not build this architecture when:

• Small datasets — under ~10,000 documents with no cross-system relationships
• No relationships — questions are keyword lookup, not traversal ("find the PDF about X")
• Prototype or demo — validate the use case with Enterprise RAG first
• Single document QA — one manual, one contract, one policy document
• FAQ bots — static Q&A pairs without enterprise data integration
• No governance requirements — no ontology, no provenance, no audit trail needed
• No reasoning across systems — answers live in a single database or document set

For these cases, use Enterprise RAG Architecture or RAG fundamentals. Add a knowledge graph when relationship traversal, entity resolution, explainability, or cross-system reasoning become hard requirements — not because graphs are fashionable.

Related Guides

This architecture guide builds on Enterprise RAG Architecture and Enterprise Knowledge Graphs. Read those first if you have not already.

Knowledge graph foundations:

• Knowledge Graphs — core concepts and modeling patterns
• Enterprise Knowledge Graphs — production graph operations
• RDF — W3C resource description framework
• SPARQL — query language for RDF graphs
• Cypher — query language for property graphs
• Graph Databases — storage, scaling, and operations
• Knowledge Graph Best Practices — governance and quality

AI integration:

• GraphRAG — graph-based retrieval for complex queries
• Knowledge Graph + LLM — integration patterns overview
• Enterprise RAG Architecture — operational envelope for retrieval systems
• Entity Resolution — cross-system identity linking
• Hybrid Search — combining graph and vector retrieval

Operations layer:

• Guardrails — input/output safety and policy
• Observability — tracing and monitoring
• Evaluation — quality measurement frameworks
• Metadata Filtering — ACL and tenant isolation patterns

Cross-cluster retrieval: RAG · Hybrid Search · Agentic RAG · GraphRAG · Knowledge Graphs

Tools: LangChain · LlamaIndex · Neo4j vector

Rankings: Best Vector Databases · Best AI Agent Frameworks

Diagram: Recommended architecture reading order

flowchart LR
    A[Knowledge Graphs] --> B[KG + LLM]
    B --> C[KG + LLM Architecture]
    C --> D[GraphRAG Architecture]

Learning Path

Prerequisites: Enterprise RAG Architecture · Knowledge Graphs · Enterprise Knowledge Graphs · SPARQL or Cypher

Estimated time: 40 min · Difficulty: Advanced

Architecture Series

Production architecture references for designing and operating enterprise AI systems at scale.

Which Architecture Should You Choose?

Use this decision tree to select the architecture pattern that matches your requirements. Each branch adds capability — start at the left and follow the path that fits your situation.

Need enterprise AI?
    ↓
   No → Traditional RAG
    ↓
   Yes
    ↓
Need structured relationships?
    ↓
   No → Enterprise RAG
    ↓
   Yes
    ↓
Need governed enterprise knowledge?
    ↓
   No → GraphRAG
    ↓
   Yes
    ↓
Knowledge Graph + LLM
    ↓
Need enterprise-wide semantic layer across many systems?
    ↓
   Yes → Enterprise Knowledge Graph Architecture

Need enterprise AI? — If the use case is a personal tool, internal demo, or single-document assistant with no authentication, governance, or multi-tenancy, Traditional RAG is sufficient. Stop here.

Need structured relationships? — If questions are answered by searching documents without joining entities across systems ("find the section about X in the handbook"), Enterprise RAG provides the operational envelope: hybrid retrieval, caching, guardrails, and observability.

Need governed enterprise knowledge? — If answers must come from authoritative records with provenance, entity resolution, and ontology validation — not probabilistic chunk retrieval — choose Knowledge Graph + LLM (this guide). If you need graph-based discovery over document corpora without a governed semantic layer, GraphRAG may suffice first.

Need enterprise-wide semantic layer across many systems? — If the knowledge graph itself is the platform — spanning dozens of source systems with federation, ontology governance, and organization-wide consumption — proceed to Enterprise Knowledge Graph Architecture.

Most production deployments combine patterns: Enterprise RAG for document evidence linked to graph entities, with Knowledge Graph + LLM for structured reasoning over governed facts.

Interview Questions

1. What is the engineering rule for KG + LLM Architecture?

   • Expected: never memorize enterprise knowledge in LLM weights; graph is system of record; ground before generate.

2. Why must entity resolution precede retrieval?

   • Expected: unresolved mentions traverse from wrong nodes; confident incorrect answers.

3. How do you safely use LLM-generated SPARQL/Cypher?

   • Expected: validate syntax, read-only, timeouts, allowlists; log queries; refuse on empty results.

4. KG + LLM Architecture vs GraphRAG Architecture?

   • Expected: governed source-system facts vs document-extracted graphs; complementary in hybrid platforms.

5. What belongs in the evidence package for the LLM?

   • Expected: ranked triples/subgraph, source record IDs, ontology version — not raw full graph dump.

6. How does hybrid retrieval work in this architecture?

   • Expected: graph traversal primary; vector search on documents linked to graph entities; merge with ranking.

7. What governance layers are architectural requirements?

   • Expected: ontology versioning, SHACL validation, RBAC on traversal, audit logs, ingestion lineage.

8. Name three metrics for production KG + LLM systems.

   • Expected: entity linking P/R, grounding coverage, ingestion lag, query timeout rate, faithfulness on graph evidence.

FAQs

How is Knowledge Graph + LLM different from GraphRAG?

GraphRAG focuses on community detection and summarization over graph structure for discovery-style questions. Knowledge Graph + LLM Architecture (this guide) focuses on governed enterprise knowledge — authoritative facts from source systems, entity resolution, ontology validation, and explainable traversal. GraphRAG can be a retrieval technique within this architecture; it is not the full production system.

How is this different from Enterprise RAG?

Enterprise RAG centers on document chunk retrieval with hybrid search, caching, and operational controls. Knowledge Graph + LLM adds structured knowledge — relationships, provenance, deterministic traversal, and entity resolution. Most production deployments use both: graph for structured facts, RAG for document evidence linked to graph entities.

Property graph or RDF — which should I choose?

RDF (Stardog, GraphDB, Neptune RDF) when you need OWL reasoning, SHACL validation, standards compliance, and federation. Property graph (Neo4j, Neptune Gremlin) when your team knows Cypher, relationships are primary, and you want GraphRAG ecosystem tooling. Document as ADR-001; either works in production with the right team.

Do I need text-to-SPARQL or text-to-Cypher in production?

You need semantic parsing — NL to graph query — but it must be validated before execution. Options: fine-tuned parsers, template-based planners for common question types, or agent loops with validation. Raw LLM-to-query without validation is a prototype pattern.

How do I handle entity ambiguity?

Run entity resolution on the critical path. Use confidence thresholds: auto-resolve above 0.8, clarify between 0.5–0.8, refuse below 0.5. Log all decisions. Maintain a human-review queue for borderline cases. Test with ambiguous names quarterly.

What latency should I target?

How often should I update the knowledge graph?

Continuously for operational sources (CDC with minutes-level lag). Daily for analytical sources. Alert when any source exceeds its SLA. Never let the graph lag authoritative systems by more than your business can tolerate for wrong decisions.

Can I fine-tune an LLM on our enterprise knowledge instead of building a graph?

No — for governed enterprise knowledge. Fine-tuning lacks provenance, ACL, freshness, cross-system joins, and auditability. Fine-tune for format, tone, or parsing accuracy. Facts belong in the graph.

What is the minimum team to operate this architecture?

At minimum: one knowledge graph engineer (ontology + ingestion), one AI engineer (parsing + synthesis pipeline), one platform engineer (infra + observability). Larger deployments add dedicated entity resolution engineers and data stewards.

When should I add GraphRAG on top of this architecture?

When discovery questions exceed fixed ontology paths — "what themes connect these suppliers?" — and community summarization adds value. See GraphRAG Architecture when published. Most enterprise operational Q&A does not need GraphRAG on day one.

References

• Stardog — Safety RAG Reference Architecture
• Stardog Voicebox Documentation
• Neo4j — GraphRAG for Agentic AI
• Microsoft — Neo4j GraphRAG Context Provider for Agent Framework
• Amazon Neptune — Graph Database Overview
• Ontotext GraphDB Documentation
• W3C — RDF 1.1 Concepts
• W3C — SPARQL 1.1 Query Language
• W3C — SHACL Specification
• IBM — Knowledge Graphs for Enterprise AI
• Google Cloud — Graph Data Science and AI

Further Reading

• Stardog — Designing LLM Applications with Knowledge Graphs and LangChain
• Neo4j — Empowering Microsoft Agent Framework with Knowledge Graphs
• AWS — Neo4j GraphRAG on Bedrock Starter
• NVIDIA — Knowledge Graphs for Generative AI
• Cambridge Semantics — Enterprise Knowledge Graphs
• TopQuadrant — Enterprise Ontology Management

Key Takeaways

• Knowledge Graph + LLM Architecture grounds language model reasoning in governed, traversable enterprise knowledge — the graph holds facts; the LLM interprets and explains.
• Entity resolution, ontology governance, and query validation are architectural prerequisites — not optional enhancements.
• Graph-first retrieval with hybrid document search is the production baseline; vectors supplement traversal, not replace it.
• Every answer must be explainable: evidence subgraphs, provenance, query plans, and ontology versions in the API response.
• Never store governed enterprise knowledge in LLM weights — external knowledge with lineage is the engineering rule.
• Read Enterprise RAG Architecture and Enterprise Knowledge Graphs first; use this document to design the combined system.