Chunking Strategies Guide

TL;DR

• Chunking splits documents into retrievable passages — chunk quality directly determines retrieval recall and answer accuracy in RAG.

• Wrong chunk size is the silent killer — too small loses context, too large adds noise and dilutes embedding signal.

• Start with recursive splitting at 512 tokens with 50-token overlap — measure recall@k before optimizing.

• Structure-aware chunking (by heading, section, page) outperforms blind fixed-size splitting for technical docs and manuals.

• Parent-child and semantic chunking address advanced cases but add complexity — use only when eval shows basic chunking fails.

Why This Matters

Chunking is the most underestimated step in a RAG pipeline. Teams spend weeks tuning LLM prompts and evaluating embedding models while their retriever returns fragments of answers split across chunk boundaries.

Consider a refund policy where the eligibility criteria are in paragraph 2 and the exceptions are in paragraph 3. Fixed-size chunking at 256 tokens might put each in a separate chunk. A user asking "Can I get a refund after 45 days?" retrieves the eligibility chunk (which says "30 days") but misses the exception chunk (which says "45 days for premium members"). The LLM gives a wrong answer from incomplete context.

Across deployed RAG systems, failures often start with chunking — roughly 40% of retrieval issues originate here. Getting chunking right before investing in rerankers, hybrid search, or agentic RAG saves months of iteration.

The Problem Chunking Solves

Documents are too large to embed and retrieve as single units. A 50-page PDF might be 30,000 tokens — far beyond what a single embedding vector can represent faithfully, and too large to pass as context to an LLM alongside other retrieved chunks.

Chunking solves the granularity problem: it determines the unit of retrieval. The chunk is both the atom of search (what the retriever finds) and the atom of context (what the LLM reads). Its size and boundaries affect both retrieval precision and generation quality.

Bad chunking creates three failure modes:

• Context loss — critical information split across chunks, none retrievable alone.

• Signal dilution — chunks too large, embedding represents a vague average of multiple topics.

• Noise injection — chunks contain irrelevant content alongside the answer, confusing the LLM.

How We Got Here

Chunking for RAG evolved from naive character splits to structure-aware and semantic pipelines:

Diagram: Evolution of document chunking

flowchart LR
    A[Fixed char split] --> B[Recursive splitter]
    B --> C[Heading-aware MD]
    C --> D[Semantic boundaries]
    D --> E[Parent-child + GraphRAG]

Splitting moved from blind token counts to respecting document structure, then to embedding-based topic boundaries and hierarchical indexes.

Frameworks such as LangChain text splitters and LlamaIndex node parsers standardised splitter APIs — but chunk parameters remain domain-specific. A stable starting pattern: start recursive at 512/50, prepend headings, measure retrieval evaluation metrics, then escalate strategy only when eval fails.

What Is Document Chunking?

Document chunking (text splitting) is the process of dividing documents into smaller passages optimized for embedding, storage, and retrieval. Each chunk becomes an independent unit in the vector database with its own embedding, metadata, and text payload.

The core parameters:

# Chunking transforms this document:
document = """
# Refund Policy
## Eligibility
Customers may request refunds within 30 days...
## Exceptions
Premium members have 45 days...
## Process
Submit via the support portal...
"""

# Into these retrievable chunks:
chunks = [
    {"text": "Refund Policy > Eligibility\nCustomers may request refunds within 30 days...",
     "metadata": {"section": "Eligibility"}},
    {"text": "Refund Policy > Exceptions\nPremium members have 45 days...",
     "metadata": {"section": "Exceptions"}},
    {"text": "Refund Policy > Process\nSubmit via the support portal...",
     "metadata": {"section": "Process"}},
]

How Chunking Works

Different strategies make different trade-offs between simplicity, context preservation, and retrieval quality.

Fixed-size chunking

Split every N tokens/characters regardless of content structure.

def fixed_chunk(text: str, size: int = 512, overlap: int = 50) -> list[str]:
    tokens = text.split()  # use tiktoken in production
    chunks = []
    for i in range(0, len(tokens), size - overlap):
        chunks.append(" ".join(tokens[i:i + size]))
    return chunks

Pros: Simple, predictable chunk count, fast. Cons: Ignores document structure, splits mid-sentence and mid-paragraph.

Recursive character splitting

Try splitting by the largest separator first, then fall back to smaller ones.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document)

Pros: Respects paragraph and sentence boundaries. Good default for most text. Cons: Still ignores semantic topic boundaries within paragraphs.

Semantic chunking

Group sentences by embedding similarity — split when the topic shifts.

def semantic_chunk(sentences: list[str], embed_fn, threshold: float = 0.5) -> list[str]:
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_similarity(embed_fn(sentences[i-1]), embed_fn(sentences[i]))
        if sim < threshold:
            chunks.append(" ".join(current))
            current = [sentences[i]]
        else:
            current.append(sentences[i])
    chunks.append(" ".join(current))
    return chunks

Pros: Chunks align with topic boundaries. Better retrieval for multi-subject documents. Cons: Requires embedding during indexing (slower, costlier). Threshold tuning needed.

Document-aware chunking

Split by document structure — headings, sections, pages, code functions.

def heading_aware_chunk(markdown: str, max_size: int = 512) -> list[dict]:
    sections = markdown.split("\n## ")
    chunks = []
    for section in sections:
        heading = section.split("\n")[0]
        body = "\n".join(section.split("\n")[1:])
        if len(body.split()) <= max_size:
            chunks.append({"text": f"{heading}\n{body}", "metadata": {"heading": heading}})
        else:
            sub = RecursiveCharacterTextSplitter(
                chunk_size=max_size, chunk_overlap=50
            ).split_text(body)
            for sc in sub:
                chunks.append({"text": f"{heading}\n{sc}", "metadata": {"heading": heading}})
    return chunks

Pros: Preserves document hierarchy. Headings provide context in every chunk. Cons: Requires structured input (markdown, HTML). Parser-dependent.

Parent-child chunking

Index small chunks for retrieval but return larger parent chunks for LLM context.

Small chunks embed precisely (better retrieval). Large parent provides full context (better generation). Store parent_id in child metadata and resolve at query time.

Diagram: Parent-child chunk retrieval sequence

sequenceDiagram
    participant Q as Query
    participant VS as Vector store
    participant P as Parent store
    participant L as LLM
    Q->>VS: ANN on child chunks
    VS-->>Q: top child IDs + scores
    Q->>P: fetch parent by parent_id
    P-->>L: full section context
    L-->>Q: grounded answer

Retrieve small for precision; generate from large parent for completeness.

Architecture

Chunking sits in the indexing pipeline between document loading and embedding:

Diagram: Chunking in the RAG indexing pipeline

flowchart TB
    subgraph ingest [Ingestion]
        PDF[PDF / HTML / MD] --> Parse[Layout parser]
        Parse --> Clean[Clean + normalize]
    end
    subgraph chunk [Chunking layer]
        Clean --> Router{Content type?}
        Router -->|Docs| Rec[Recursive splitter]
        Router -->|MD| Head[Heading-aware]
        Router -->|Code| AST[AST splitter]
        Rec --> Meta[Enrich metadata]
        Head --> Meta
        AST --> Meta
    end
    subgraph index [Indexing]
        Meta --> Embed[Embedding model]
        Embed --> VDB[(Vector DB)]
    end

Route by content type; unify metadata schema before embedding.

Source: Survey on RAG (arXiv:2312.10997)

Key design choices at this layer:

• One chunker or many? Different document types (PDF, code, markdown) often need different strategies.

• Where to store chunk metadata? Section headings, page numbers, and parent IDs must travel with the chunk to the vector DB for metadata filtering.

• Re-chunking trigger? When chunk parameters change, the entire corpus must be re-chunked and re-embedded with the same embedding model.

Step-by-Step Flow

Step 1: Analyze your corpus. What document types? Average length? Structure (headings, tables, code)? Query types (factoid vs explanatory)?

Step 2: Choose a strategy. Recursive for general text. Document-aware for structured docs. Semantic for multi-topic documents.

Step 3: Set chunk size and overlap. Start 512/50. Align max size with embedding model token limit.

Step 4: Enrich chunks with metadata. Prepend section headings. Attach source file, page number, section ID, and parent chunk ID.

Step 5: Index and evaluate. Embed chunks, run retrieval evaluation. Measure recall@k on your golden test set.

Step 6: Iterate. Adjust size, overlap, or strategy based on failure analysis. Re-chunk and re-embed.

Chunk size guide

Diagram: Chunk lifecycle in production

stateDiagram-v2
    [*] --> Parsed: document ingested
    Parsed --> Chunked: splitter applied
    Chunked --> Embedded: vectors stored
    Embedded --> Indexed: ANN ready
    Indexed --> Retrieved: query match
    Retrieved --> [*]
    Chunked --> Rechunk: param change
    Rechunk --> Embedded

Parameter changes force re-chunk and re-embed — version chunk config in metadata.

Real Production Example

A developer documentation site with 3,000 markdown pages — document-aware chunking with heading injection and tenant metadata:

import hashlib
import tiktoken
from dataclasses import dataclass, field
from pathlib import Path

enc = tiktoken.get_encoding("cl100k_base")

@dataclass
class Chunk:
    id: str
    text: str
    metadata: dict = field(default_factory=dict)

class DocAwareChunker:
    def __init__(self, max_tokens: int = 512, overlap: int = 50):
        self.max_tokens = max_tokens
        self.overlap = overlap

    def count_tokens(self, text: str) -> int:
        return len(enc.encode(text))

    def chunk_markdown(self, content: str, source: str, tenant_id: str) -> list[Chunk]:
        chunks = []
        current_heading = ""
        current_section: list[str] = []
        section_tokens = 0

        for line in content.split("\n"):
            if line.startswith("#"):
                if current_section:
                    chunks.extend(self._flush_section(
                        current_heading, current_section, source, tenant_id
                    ))
                current_heading = line.lstrip("#").strip()
                current_section = []
                section_tokens = 0
            else:
                current_section.append(line)
                section_tokens += self.count_tokens(line)
                if section_tokens >= self.max_tokens:
                    chunks.extend(self._flush_section(
                        current_heading, current_section, source, tenant_id
                    ))
                    overlap_lines = self._tail_overlap_lines(current_section)
                    current_section = overlap_lines
                    section_tokens = sum(self.count_tokens(l) for l in current_section)

        if current_section:
            chunks.extend(self._flush_section(
                current_heading, current_section, source, tenant_id
            ))
        return chunks

    def _tail_overlap_lines(self, lines: list[str]) -> list[str]:
        target, acc = [], 0
        for line in reversed(lines):
            acc += self.count_tokens(line)
            target.insert(0, line)
            if acc >= self.overlap:
                break
        return target

    def _flush_section(
        self, heading: str, lines: list[str], source: str, tenant_id: str
    ) -> list[Chunk]:
        body = "\n".join(lines).strip()
        if not body:
            return []
        text = f"## {heading}\n{body}" if heading else body
        chunk_id = hashlib.sha256(f"{source}:{heading}:{body[:80]}".encode()).hexdigest()[:16]
        return [Chunk(
            id=chunk_id,
            text=text,
            metadata={
                "source": source,
                "heading": heading,
                "tenant_id": tenant_id,
                "chunk_strategy": "heading-aware-v1",
                "chunk_max_tokens": self.max_tokens,
            },
        )]

# Usage
chunker = DocAwareChunker(max_tokens=512, overlap=50)
for doc_path in Path("docs").rglob("*.md"):
    content = doc_path.read_text(encoding="utf-8")
    chunks = chunker.chunk_markdown(content, source=str(doc_path), tenant_id="acme")
    vector_store.upsert_batch(chunks, embedding_model="text-embedding-3-small")

Every chunk includes its section heading, so a query about "authentication middleware" retrieves chunks prefixed with ## Authentication Middleware — even if the exact phrase is not in the chunk body.

Design Decisions

Common patterns

• Heading prepend — One of the highest-ROI changes for structured docs (+10–15% recall@k).

• Chunking router — Route PDF → layout parser, MD → heading-aware, code → AST, chat → time windows.

• Table preservation — Keep tables intact; duplicate headers when splitting long tables.

• Golden-set driven tuning — Change one parameter at a time; log chunk config version in metadata.

Comparisons

Fixed-size vs recursive vs semantic

Chunking vs whole-document indexing

Chunking vs GraphRAG community summaries

Decision tree: chunking strategy

Decision tree: Choosing a chunking strategy

flowchart TD
    A[New RAG corpus] --> B{Avg doc length > 512 tokens?}
    B -->|No| C[Index whole documents]
    B -->|Yes| D{Structured headings?}
    D -->|Yes| E[Heading-aware + recursive fallback]
    D -->|No| F{Multi-topic paragraphs?}
    F -->|Yes| G[Semantic chunking]
    F -->|No| H[Recursive 512/50]
    E --> I[Prepend headings]
    G --> I
    H --> I
    I --> J[Eval recall@k]
    J --> K{Retrieval OK, context thin?}
    K -->|Yes| L[Parent-child index]
    K -->|No| M[Ship + monitor]
    L --> M

Escalate complexity only when baseline recursive + heading-aware fails retrieval evaluation.

Compare vector stores for chunk metadata support in Best Vector Databases: Qdrant vs Pinecone · Pinecone vs Weaviate · pgvector vs Pinecone.

Common Mistakes

1. Using default chunk size without evaluation. 512 tokens is a starting point, not an answer. Measure recall@k on your data.

2. Zero overlap. Without overlap, information at chunk boundaries is lost. Always use 10–15% overlap unless chunks are naturally self-contained.

3. Splitting tables across chunks. A table header in one chunk and rows in another makes both chunks useless. Keep tables intact or duplicate headers.

4. Ignoring document structure. Blind fixed-size splitting on markdown, HTML, or code destroys the semantic units users query against.

5. Not prepending context. A chunk saying "It supports OAuth 2.0 and SAML" is unretrievable without knowing "It" refers to the authentication system. Include headings and preceding context.

6. Chunking before cleaning. Headers, footers, page numbers, and navigation elements from PDFs pollute chunks. Clean before chunking.

7. Same strategy for all document types. PDFs, code, markdown, and chat logs need different approaches. Build a chunking router by content type.

8. Mismatch with embedding model token limit. Chunks longer than embedding model max tokens truncate silently — the retriever never sees the full passage.

Where It Breaks Down

Highly structured short documents — If most documents are under 512 tokens, chunking adds complexity without benefit. Index whole documents instead.

Cross-reference heavy content — Chunks that say "see Section 4.2" lose meaning without the referenced section. Consider GraphRAG or parent-child chunking.

Real-time content — Chat logs and streaming data need window-based chunking with time boundaries, not static document splitting.

Multimodal content — Images, charts, and diagrams within documents lose meaning when only surrounding text is chunked. Multimodal embeddings or caption extraction are needed.

Non-text content — Audio transcripts, spreadsheets, and JSON documents require domain-specific chunking logic that general text splitters do not provide.

Highly redundant corpora — Duplicate chunks across versions inflate index size and confuse ranking; deduplicate by content hash before embedding.

When NOT to Use Advanced Chunking

Skip semantic or parent-child chunking when:

1. Baseline recursive passes eval — recall@5 ≥ 0.8 on golden set; do not add complexity without measured gain.

2. Documents are uniformly short — Wiki pages, ticket titles, FAQ entries under 256 tokens index whole.

3. Latency budget forbids semantic indexing — Semantic chunking doubles index-time embedding cost.

4. Structure is unavailable — Raw OCR garbage without headings; fix parsing before investing in heading-aware splitters.

5. Graph queries dominate — Relationship traversal needs GraphRAG, not finer text chunks.

Prefer hybrid search when failures are exact-token misses, not boundary fragmentation — chunking will not fix SKU lookup.

Running in Production

Best Practice

✅ Best Practices — Version chunk config in metadata, parallelize chunking offline, deduplicate by hash, and A/B test size changes on a golden set before full re-index.

Tip

Prepend section headings to chunk text before embedding. This single change often improves recall@k by 10–15% on structured documentation.

Important

Store chunk_strategy, chunk_max_tokens, and chunk_version in vector metadata. Debugging retrieval failures without knowing how text was split is nearly impossible.

Related Guides

• Foundations: RAG · Embeddings · Embedding Models

• Retrieval stack: Vector Search · Hybrid Search · Metadata Filtering · ANN Indexes · Re-ranking

• Quality & ops: Retrieval Evaluation · Vector Databases

• Advanced: GraphRAG · Agentic RAG · Late Interaction Retrieval

• Frameworks: LangChain · LlamaIndex

• Vector stores: Qdrant · Pinecone · Weaviate · Chroma · pgvector — Best Vector Databases.

• Head-to-heads: Qdrant vs Pinecone · Pinecone vs Weaviate · pgvector vs Pinecone

If you understood this topic, read next:

Diagram: Learning path for chunking

flowchart LR
    A[RAG] --> B[Embeddings]
    B --> C[Chunking]
    C --> D[Embed Models]
    D --> E[Eval]

Prerequisites: RAG · Embeddings

Next topics: Embedding Models · Retrieval Evaluation · Re-ranking

Estimated time: 40 min · Difficulty: Intermediate

Interview Questions

1. Why is chunk size the ceiling on RAG answer quality?

   • Expected: chunk is both retrieval unit and LLM context; wrong boundaries → missing or noisy evidence.

2. What overlap percentage do you start with and why?

   • Expected: 10–15% of chunk size to preserve boundary information; 50 tokens at 512 default.

3. When does parent-child chunking beat flat chunks?

   • Expected: retrieval finds correct region but child too small for generation; fetch parent for LLM context.

4. How do chunk size and embedding model max tokens interact?

   • Expected: chunks exceeding model limit truncate silently; align size with embedding model cap.

5. Recursive vs semantic chunking — trade-offs?

   • Expected: recursive cheap and good default; semantic aligns topics but adds index-time embed cost.

6. How do you chunk code differently from prose?

   • Expected: AST/function boundaries, include imports, not token-based splits mid-function.

7. What metadata must every chunk carry in multi-tenant SaaS?

   • Expected: tenant_id, ACL, source URL, heading, chunk_version, parent_id if hierarchical.

8. How do you evaluate a chunking change before full re-index?

   • Expected: sample corpus slice, golden queries, recall@k comparison, then staged rollout.

Key Takeaways

• Chunking determines the unit of retrieval — its quality is the foundation of RAG performance.
• Start with recursive splitting at 512 tokens with 50-token overlap.
• Prepend section headings to chunks for better embedding and retrieval.
• Use document-aware chunking for structured content; semantic chunking for multi-topic documents.
• Always evaluate chunking choices with retrieval evaluation — do not guess.
• Different document types need different chunking strategies — build a router.
• Compare vector infrastructure in Best Vector Databases after chunking and embed choices stabilize.

FAQs

What chunk size should I use?

Start with 512 tokens and 50-token overlap. Evaluate recall@k. Decrease to 256 for factoid Q&A; increase to 768–1024 for analytical questions needing context.

How much overlap do I need?

10–15% of chunk size. For 512-token chunks, use 50–75 token overlap. Overlap prevents information loss at boundaries.

Should I chunk by tokens or characters?

Tokens — because embedding models and LLMs operate on tokens. Use tiktoken or the model's tokenizer for accurate counts.

Does chunk size affect embedding quality?

Yes. Very long chunks produce vague embeddings averaging multiple topics. Very short chunks may lack enough signal. Match chunk size to the granularity of your expected queries.

When should I use semantic chunking?

When documents cover multiple topics within paragraphs and recursive splitting creates mixed-topic chunks. Test on your data — semantic chunking adds indexing cost.

How do I chunk PDFs?

Use layout-aware parsers (Unstructured, Docling) that detect headings, tables, and columns. Chunk by section after parsing. Never chunk raw PDF text without parsing.

How do I chunk code?

Split by AST nodes — functions, classes, methods. Each chunk should be a compilable unit with its imports. LlamaIndex CodeSplitter and tree-sitter-based tools handle this.

What is parent-child chunking?

Index small child chunks (256 tokens) for precise retrieval. Store a link to a larger parent chunk (1024 tokens). When a child matches, return the parent to the LLM for full context.

How often should I re-chunk?

Re-chunk when changing chunk size, overlap, or strategy. Re-chunking requires re-embedding. Plan for a full re-index pipeline, not ad-hoc changes.

Can I use different chunk sizes for different document types?

Yes — and you should. Policies might use section-based chunking; chat logs might use time-window chunking; code uses function-based splitting. Route by content type.

How do I handle tables in documents?

Keep tables as single chunks. If a table exceeds max chunk size, split by rows but repeat column headers in each sub-chunk.

Does chunking strategy affect hybrid search?

Indirectly. Better chunks improve both vector and keyword retrieval. Chunk boundaries affect which terms appear together in BM25-indexed text — see hybrid search.

How does chunking relate to reranking?

Chunking affects what candidates reach the reranker. Good chunks improve bi-encoder recall; reranking fixes ordering within retrieved sets — neither compensates for missing chunks.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Text Splitters Documentation
• LlamaIndex Node Parser Guide
• Survey on RAG (Gao et al., 2023)

Further Reading

• Pinecone — Chunking strategies for RAG
• OpenAI Cookbook — Chunking text
• Unstructured.io Documentation