GraphRAG Guide

TL;DR

• GraphRAG combines knowledge graphs with vector retrieval to answer questions that require understanding relationships between entities, not just matching similar text.

• It solves multi-hop and global questions that vanilla RAG fails on - "How do these three systems interact?" or "Summarize themes across 10,000 documents."

• Microsoft's GraphRAG pipeline extracts entities, builds a graph, generates community summaries, and uses both local (entity-neighborhood) and global (community) retrieval.

• Trade-off: indexing cost and complexity. Graph construction requires LLM calls per document chunk; indexing is 10–50x more expensive than standard RAG.

• Use GraphRAG when relationship traversal matters - compliance, supply chain, org charts, research synthesis. Use vanilla RAG for straightforward document Q&A.

Why This Matters

Vanilla RAG retrieves text chunks by semantic similarity. This works for "What is our refund policy?" but fails for questions that require connecting information across documents:

• "Which vendors supply components used in products recalled last quarter?"
• "How does the authentication service depend on the billing microservice?"
• "What are the main themes in our customer feedback from the past year?"

These queries need entity resolution, relationship traversal, and summarization over connected subgraphs - capabilities vector search alone cannot provide.

GraphRAG addresses this by building a knowledge graph during indexing, then retrieving structured context (entity neighborhoods, community summaries) alongside or instead of raw text chunks. Organizations with complex, interconnected data - legal, pharma, finance, enterprise IT - increasingly adopt GraphRAG as a complement to standard RAG.

The Problem GraphRAG Solves

Standard RAG has three classes of failure that graph-based retrieval fixes:

Multi-hop queries. The answer requires following a chain of relationships across documents. Vector search returns chunks similar to the query text, not chunks connected through entity links. If the relationship "Vendor A supplies Component X used in Product Y" spans three documents, no single chunk may embed close to the query.

Global summarization. "What are the key risks mentioned across our 5,000 incident reports?" requires synthesizing themes across the entire corpus. Vector retrieval returns a handful of similar chunks - a sample, not a summary. GraphRAG's community detection clusters related entities and pre-generates summaries at each community level.

Entity disambiguation. "Tell me about Apple" could mean the company, the fruit, or a project codename. Knowledge graphs resolve entities to canonical nodes with typed relationships, reducing ambiguity in retrieval.

GraphRAG does not replace vector RAG - it extends it for queries where structure matters.

What Is GraphRAG?

GraphRAG is a retrieval pattern that builds a knowledge graph from your document corpus and uses graph structure - entities, relationships, and community hierarchies - to inform what context gets passed to the LLM at query time.

The approach was popularized by Microsoft's GraphRAG research, which introduced a pipeline:

1. Extract entities and relationships from text using an LLM.
2. Build a graph where nodes are entities and edges are relationships.
3. Detect communities (clusters) in the graph using algorithms like Leiden.
4. Generate hierarchical summaries for each community.
5. At query time, choose between local search (entity-centric neighborhood retrieval) and global search (community summary retrieval).

# Conceptual GraphRAG query routing
def graphrag_query(question: str, graph, vector_store):
    query_type = classify_query(question)  # local vs global

    if query_type == "local":
        entities = extract_entities(question)
        context = graph.get_neighborhood(entities, hops=2)
        chunks = vector_store.search(question, top_k=5)
        return llm.generate(question, context + chunks)

    elif query_type == "global":
        communities = graph.rank_communities(question)
        summaries = [c.summary for c in communities[:5]]
        return llm.generate(question, summaries)

GraphRAG sits alongside other graph-enhanced retrieval approaches: Neo4j + vector indexes, LlamaIndex Knowledge Graph Index, and custom entity-linking pipelines.

How GraphRAG Works

Indexing Phase

GraphRAG builds a knowledge graph from source documents, clusters entities into communities, and retrieves graph-aware context for complex multi-hop questions.

Entity extraction. An LLM reads each chunk and extracts entities (people, organizations, products, concepts) and relationships (works_at, supplies, depends_on). This is the most expensive step - one or more LLM calls per chunk.

Entity resolution. "IBM", "International Business Machines", and "Big Blue" must merge into one node. Fuzzy matching, embedding similarity, and LLM-based deduplication handle this.

Community detection. Algorithms like Leiden partition the graph into densely connected clusters. Each community gets a summary generated by an LLM reading the entities and relationships within it.

Dual indexing. Text chunks still get embedded into a vector store for hybrid retrieval. The graph store holds entities, edges, and community summaries.

Query Phase

Local search - For specific entity questions. Extract entities from the query, traverse the graph 1–2 hops, retrieve connected entity descriptions and relationships, combine with vector-retrieved text chunks.

Global search - For broad thematic questions. Rank community summaries by relevance to the query, pass top summaries to the LLM for synthesis.

The indexing phase extracts entities and relationships, constructs community summaries, and stores graph structures for later retrieval.

Source: Microsoft Research

Architecture

Production GraphRAG systems typically maintain three stores: raw documents, a property graph, and a vector index. The orchestrator routes queries to the appropriate retrieval strategy.

Step-by-Step Flow

Step 1: Chunk documents using the same strategies as vanilla RAG. GraphRAG still benefits from good chunking for entity extraction quality.

Step 2: Extract entities and relationships from each chunk via LLM prompt. Use structured output (JSON schema) for reliable parsing.

Step 3: Resolve entities across chunks. Build a canonical entity registry with aliases, types, and descriptions.

Step 4: Construct the graph. Nodes = entities. Edges = extracted relationships with source chunk references.

Step 5: Run community detection. Partition the graph. Generate LLM summaries for each community at multiple hierarchy levels.

Step 6: Embed text chunks into a vector store (standard RAG indexing continues in parallel).

Step 7: At query time, classify the query as local or global. Retrieve from graph + vector store. Generate answer with citations to both text chunks and graph entities.

Real Production Example

A compliance team needs to answer: "Which third-party processors handle EU customer PII and what certifications do they hold?"

import json
from neo4j import GraphDatabase

EXTRACTION_PROMPT = """Extract entities and relationships from this text.
Return JSON: {"entities": [{"name": "", "type": ""}],
"relationships": [{"source": "", "target": "", "type": ""}]}

Text: {chunk}"""

class GraphRAGPipeline:
    def __init__(self, llm, graph_driver, vector_store):
        self.llm = llm
        self.graph = GraphDatabase.driver(graph_driver)
        self.vector_store = vector_store

    def extract_graph_elements(self, chunk: str) -> dict:
        response = self.llm.generate(
            EXTRACTION_PROMPT.format(chunk=chunk),
            response_format={"type": "json_object"},
        )
        return json.loads(response)

    def index_chunk(self, chunk: str, doc_id: str):
        elements = self.extract_graph_elements(chunk)

with self.graph.session() as session:
            for entity in elements["entities"]:
                session.run(
                    "MERGE (e:Entity {name: $name}) "
                    "SET e.type = $type, e.source_doc = $doc_id",
                    name=entity["name"], type=entity["type"], doc_id=doc_id,
                )
            for rel in elements["relationships"]:
                session.run(
                    "MATCH (a:Entity {name: $src}), (b:Entity {name: $tgt}) "
                    "MERGE (a)-[:REL {type: $rtype, source_doc: $doc_id}]->(b)",
                    src=rel["source"], tgt=rel["target"],
                    rtype=rel["type"], doc_id=doc_id,
                )

        self.vector_store.upsert(chunk, metadata={"doc_id": doc_id})

    def local_search(self, query: str, hops: int = 2) -> str:
        entities = self.extract_graph_elements(query)["entities"]
        entity_names = [e["name"] for e in entities]

        with self.graph.session() as session:
            result = session.run(
                f"MATCH (e:Entity) WHERE e.name IN $names "
                f"MATCH path = (e)-[*1..{hops}]-(connected) "
                f"RETURN e, connected, relationships(path) LIMIT 50",
                names=entity_names,
            )
            graph_context = "\n".join(
                f"{r['e']['name']} -> {r['connected']['name']}"
                for r in result
            )

        vector_chunks = self.vector_store.search(query, top_k=5)
        return f"Graph context:\n{graph_context}\n\nDocuments:\n{vector_chunks}"

    def query(self, question: str) -> str:
        context = self.local_search(question)
        return self.llm.generate(
            f"Answer using this context:\n{context}\n\nQuestion: {question}"
        )

The graph traversal finds processors connected to "EU customer PII" through processes_data_for relationships, even when no single document mentions both concepts together.

Design Decisions

⚠ Common Mistakes

1. Using GraphRAG for simple Q&A. If your queries are "What is X?" on well-structured docs, vanilla RAG is faster and cheaper. GraphRAG adds value for relationship and global queries.

2. Skipping entity resolution. Duplicate entities ("AWS" vs "Amazon Web Services") fragment the graph and break traversals.

3. Under-budgeting indexing. GraphRAG indexing requires LLM calls per chunk for extraction plus community summarization. A 10K-document corpus can cost $50–500 to index vs $5 for vector-only.

4. Ignoring graph maintenance. Documents change; entities merge and split. Without incremental graph updates, the graph drifts from reality.

5. No fallback to vector search. Graph retrieval alone misses chunks that weren't entity-rich. Always combine with vector retrieval.

6. Over-extracting relationships. Noisy extraction creates a hairball graph. Validate extraction quality on a sample before full indexing.

Where It Breaks Down

GraphRAG indexing is slow and expensive compared to vector RAG. Entity extraction quality depends on the LLM - errors propagate into the graph permanently until re-indexed.

Sparse entity documents - Logs, code, and tabular data produce few meaningful entities. GraphRAG adds little value over vector search for these content types.

Dynamic data - Graphs are expensive to update incrementally. A corpus that changes hourly may outpace graph maintenance. Vector RAG with frequent re-indexing may be more practical.

Small corpora - Under 1,000 documents, the graph may be too sparse for meaningful community detection. The overhead isn't justified.

Real-time queries - Graph traversal + vector search + LLM generation adds latency. Expect 3–8 seconds per query vs 1–3 for vanilla RAG.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

GraphRAG is a complement to vanilla RAG, not a replacement. Run both pipelines and route queries based on complexity.

Ecosystem

• Microsoft GraphRAG: Open-source reference implementation with community detection and hierarchical summarization.

• LlamaIndex: Knowledge Graph Index, property graph stores, hybrid retrievers.

• Neo4j: Property graph database with vector index support and GraphRAG integrations.

• LangChain: Graph document transformers, Neo4j graph QA chains.

• FalkorDB: Redis-based graph database optimized for GraphRAG workloads.

Related Technologies

• RAG: The baseline retrieval pattern GraphRAG extends.

• Knowledge Graphs: The structured data model underlying GraphRAG.

• Enterprise Knowledge Graphs: Organization-scale graph deployments.

• Retrieval Evaluation: Measuring multi-hop retrieval quality.

• Agentic RAG: Agents that decide when to traverse graphs vs search vectors.

Learning Path

Prerequisites: RAG · Embeddings

Next topics: Knowledge Graphs · Retrieval Evaluation · Agentic RAG

Estimated time: 55 min · Difficulty: Advanced

FAQs

When should I use GraphRAG vs vanilla RAG?

Use GraphRAG when queries require relationship traversal, multi-hop reasoning, or global summarization across a large corpus. Use vanilla RAG for straightforward document Q&A.

How much more expensive is GraphRAG indexing?

Typically 10–50x vector-only RAG. Entity extraction requires an LLM call per chunk, plus community summarization. A 10K-document corpus might cost $50–500 vs $5 for vector indexing.

Do I need Neo4j?

Not necessarily. NetworkX works for prototyping. Neo4j, FalkorDB, or Amazon Neptune for production persistence and Cypher queries. Choose based on scale and existing infrastructure.

Can I use GraphRAG with existing vector RAG?

Yes - this is the recommended approach. Build both indexes in parallel. Route queries to graph retrieval, vector retrieval, or both based on query type.

How do I handle entity extraction errors?

Validate on a sample corpus first. Use structured output with JSON schemas. Run a human review pass on extracted entities before full indexing. Plan for periodic re-extraction.

What is community detection?

An algorithm (Leiden, Louvain) that partitions the graph into densely connected clusters. Each cluster gets an LLM-generated summary, enabling global queries like "What are the main themes?"

How do local and global search differ?

Local search retrieves an entity's neighborhood (1–2 hops) for specific questions. Global search retrieves community summaries for broad thematic questions across the corpus.

Does GraphRAG work with code repositories?

Limited value. Code has few traditional entities/relationships. Vector RAG with code-aware chunking (by function/class) usually performs better.

How often should I rebuild the graph?

Full rebuild when changing extraction models or community detection parameters. Incremental updates when documents are added/changed - merge new entities and edges.

Can GraphRAG replace a data warehouse?

No. GraphRAG is a retrieval pattern for LLM context, not an analytics engine. Use it to find and synthesize information, not to run aggregations or reports.

What query classification works best?

Start with an LLM prompt: "Is this query about a specific entity (local) or a broad theme (global)?" Refine with rules as you learn your query distribution.

How do I evaluate GraphRAG?

Build a test set of multi-hop questions with known entity paths. Measure whether the correct subgraph is retrieved. Compare end-to-end answer quality against vanilla RAG.

References

• From Local to Global: A Graph RAG Approach (Microsoft, 2024)
• Microsoft GraphRAG Documentation
• Neo4j LLM Knowledge Graph Builder

Further Reading

• LangChain GraphRAG Integration
• Apache Jena Documentation
• OpenAI Structured Outputs

Summary

• GraphRAG adds knowledge graph structure to RAG for relationship-aware and global retrieval. - Use it when queries require multi-hop reasoning or corpus-wide summarization - not for simple document Q&A. - Indexing is significantly more expensive due to LLM-based entity extraction and community summarization. - Always combine graph retrieval with vector search - neither alone covers all query types.

• Entity resolution quality determines graph utility; invest in deduplication and validation. - Route queries between local (entity-centric) and global (community summary) search strategies.