Chunking Strategies Guide

TL;DR

• Chunking splits documents into retrievable passages - chunk quality directly determines retrieval recall and answer accuracy in RAG.

• Wrong chunk size is the silent killer - too small loses context, too large adds noise and dilutes embedding signal.

• Start with recursive splitting at 512 tokens with 50-token overlap - measure recall@k before optimizing.

• Structure-aware chunking (by heading, section, page) outperforms blind fixed-size splitting for technical docs and manuals.

• Parent-child and semantic chunking address advanced cases but add complexity - use only when eval shows basic chunking fails.

Why This Matters

Chunking is the most underestimated step in a RAG pipeline. Teams spend weeks tuning LLM prompts and evaluating embedding models while their retriever returns fragments of answers split across chunk boundaries.

Consider a refund policy where the eligibility criteria are in paragraph 2 and the exceptions are in paragraph 3. Fixed-size chunking at 256 tokens might put each in a separate chunk. A user asking "Can I get a refund after 45 days?" retrieves the eligibility chunk (which says "30 days") but misses the exception chunk (which says "45 days for premium members"). The LLM gives a wrong answer from incomplete context.

Every RAG failure I've debugged in production traces back to chunking at least 40% of the time. Getting this right before investing in rerankers, hybrid search, or agentic retrieval saves months of iteration.

The Problem Chunking Solves

Documents are too large to embed and retrieve as single units. A 50-page PDF might be 30,000 tokens - far beyond what a single embedding vector can represent faithfully, and too large to pass as context to an LLM alongside other retrieved chunks.

Chunking solves the granularity problem: it determines the unit of retrieval. The chunk is both the atom of search (what the retriever finds) and the atom of context (what the LLM reads). Its size and boundaries affect both retrieval precision and generation quality.

Bad chunking creates three failure modes:

• Context loss - critical information split across chunks, none retrievable alone.

• Signal dilution - chunks too large, embedding represents a vague average of multiple topics.

• Noise injection - chunks contain irrelevant content alongside the answer, confusing the LLM.

What Is Document Chunking?

Document chunking (text splitting) is the process of dividing documents into smaller passages optimized for embedding, storage, and retrieval. Each chunk becomes an independent unit in the vector database with its own embedding, metadata, and text payload.

The core parameters:

# Chunking transforms this document:
document = """
# Refund Policy
## Eligibility
Customers may request refunds within 30 days...
## Exceptions
Premium members have 45 days...
## Process
Submit via the support portal...
"""

# Into these retrievable chunks:
chunks = [
    {"text": "Refund Policy > Eligibility\nCustomers may request refunds within 30 days...", "metadata": {"section": "Eligibility"}},
    {"text": "Refund Policy > Exceptions\nPremium members have 45 days...", "metadata": {"section": "Exceptions"}},
    {"text": "Refund Policy > Process\nSubmit via the support portal...", "metadata": {"section": "Process"}},
]

How Chunking Works

Different strategies make different trade-offs between simplicity, context preservation, and retrieval quality.

Fixed-Size Chunking

Split every N tokens/characters regardless of content structure.

def fixed_chunk(text: str, size: int = 512, overlap: int = 50) -> list[str]:
    tokens = text.split()  # simplified; use tiktoken in production
    chunks = []
    for i in range(0, len(tokens), size - overlap):
        chunks.append(" ".join(tokens[i:i + size]))
    return chunks

Pros: Simple, predictable chunk count, fast. Cons: Ignores document structure, splits mid-sentence and mid-paragraph.

Recursive Character Splitting

Try splitting by the largest separator first, then fall back to smaller ones.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document)

Pros: Respects paragraph and sentence boundaries. Good default for most text. Cons: Still ignores semantic topic boundaries within paragraphs.

Semantic Chunking

Group sentences by embedding similarity - split when the topic shifts.

def semantic_chunk(sentences: list[str], embed_fn, threshold: float = 0.5) -> list[str]:
    chunks, current_chunk = [], [sentences[0]]

    for i in range(1, len(sentences)):
        sim = cosine_similarity(embed_fn(sentences[i-1]), embed_fn(sentences[i]))
        if sim < threshold:
            chunks.append(" ".join(current_chunk))
            current_chunk = [sentences[i]]
        else:
            current_chunk.append(sentences[i])

    chunks.append(" ".join(current_chunk))
    return chunks

Pros: Chunks align with topic boundaries. Better retrieval for documents with multiple subjects. Cons: Requires embedding during indexing (slower, costlier). Threshold tuning needed.

Document-Aware Chunking

Split by document structure - headings, sections, pages, code functions.

def heading_aware_chunk(markdown: str, max_size: int = 512) -> list[dict]:
    sections = markdown.split("\n## ")
    chunks = []
    for section in sections:
        heading = section.split("\n")[0]
        body = "\n".join(section.split("\n")[1:])
        if len(body.split()) <= max_size:
            chunks.append({"text": f"{heading}\n{body}", "metadata": {"heading": heading}})
        else:
            sub_chunks = RecursiveCharacterTextSplitter(
                chunk_size=max_size, chunk_overlap=50
            ).split_text(body)
            for sc in sub_chunks:
                chunks.append({"text": f"{heading}\n{sc}", "metadata": {"heading": heading}})
    return chunks

Pros: Preserves document hierarchy. Headings provide context in every chunk. Cons: Requires structured input (markdown, HTML). Parser-dependent.

Parent-Child Chunking

Index small chunks for retrieval but return larger parent chunks for LLM context.

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Small chunks embed precisely (better retrieval). Large parent provides full context (better generation).

Architecture

Chunking sits in the indexing pipeline between document loading and embedding:

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Source: Survey on RAG

Key design choices at this layer:

• One chunker or many? Different document types (PDF, code, markdown) often need different strategies.

• Where to store chunk metadata? Section headings, page numbers, and parent IDs must travel with the chunk to the vector DB.

• Re-chunking trigger? When chunk parameters change, the entire corpus must be re-chunked and re-embedded.

Step-by-Step Flow

Step 1: Analyze your corpus. What document types? Average length? Structure (headings, tables, code)? Query types (factoid vs explanatory)?

Step 2: Choose a strategy. Recursive for general text. Document-aware for structured docs. Semantic for multi-topic documents.

Step 3: Set chunk size and overlap. Start 512/50. See the sizing guide below.

Step 4: Enrich chunks with metadata. Prepend section headings. Attach source file, page number, section ID, and parent chunk ID.

Step 5: Index and evaluate. Embed chunks, run retrieval eval. Measure recall@k on your golden test set.

Step 6: Iterate. Adjust size, overlap, or strategy based on failure analysis. Re-chunk and re-embed.

Chunk Size Guide

Real Production Example

A developer documentation site with 3,000 markdown pages:

import hashlib
from dataclasses import dataclass, field

@dataclass
class Chunk:
    id: str
    text: str
    metadata: dict = field(default_factory=dict)

class DocAwareChunker:
    def __init__(self, max_tokens: int = 512, overlap: int = 50):
        self.max_tokens = max_tokens
        self.overlap = overlap

    def count_tokens(self, text: str) -> int:
        return len(text.split())  # use tiktoken in production

    def chunk_markdown(self, content: str, source: str) -> list[Chunk]:
        chunks = []
        current_heading = ""
        current_section = []
        section_tokens = 0

for line in content.split("\n"):
            if line.startswith("#"):
                if current_section:
                    chunks.extend(self._flush_section(
                        current_heading, current_section, source
                    ))
                current_heading = line.lstrip("#").strip()
                current_section = []
                section_tokens = 0
            else:
                current_section.append(line)
                section_tokens += self.count_tokens(line)

                if section_tokens >= self.max_tokens:
                    chunks.extend(self._flush_section(
                        current_heading, current_section, source
                    ))
                    current_section = current_section[-self.overlap:]
                    section_tokens = sum(self.count_tokens(l) for l in current_section)

        if current_section:
            chunks.extend(self._flush_section(current_heading, current_section, source))

        return chunks

    def _flush_section(self, heading: str, lines: list[str], source: str) -> list[Chunk]:
        body = "\n".join(lines)
        text = f"## {heading}\n{body}" if heading else body
        chunk_id = hashlib.md5(f"{source}:{heading}:{body[:50]}".encode()).hexdigest()
        return [Chunk(
            id=chunk_id,
            text=text,
            metadata={
                "source": source,
                "heading": heading,
                "chunk_type": "section",
            },
        )]

# Usage
chunker = DocAwareChunker(max_tokens=512, overlap=50)
for doc_path in glob("docs/**/*.md"):
    content = open(doc_path).read()
    chunks = chunker.chunk_markdown(content, source=doc_path)
    vector_store.upsert_batch(chunks)

Every chunk includes its section heading, so a query about "authentication middleware" retrieves chunks prefixed with ## Authentication Middleware - even if the exact phrase isn't in the chunk body.

Design Decisions

⚠ Common Mistakes

1. Using default chunk size without evaluation. 512 tokens is a starting point, not an answer. Measure recall@k on your data.

2. Zero overlap. Without overlap, information at chunk boundaries is lost. Always use 10–15% overlap unless chunks are naturally self-contained.

3. Splitting tables across chunks. A table header in one chunk and rows in another makes both chunks useless. Keep tables intact or duplicate headers.

4. Ignoring document structure. Blind fixed-size splitting on markdown, HTML, or code destroys the semantic units users query against.

5. Not prepending context. A chunk saying "It supports OAuth 2.0 and SAML" is unretrievable without knowing "It" refers to the authentication system. Include headings and preceding context.

6. Chunking before cleaning. Headers, footers, page numbers, and navigation elements from PDFs pollute chunks. Clean before chunking.

7. Same strategy for all document types. PDFs, code, markdown, and chat logs need different approaches. Build a chunking router by content type.

Where It Breaks Down

Highly structured short documents - If most documents are under 512 tokens, chunking adds complexity without benefit. Index whole documents instead.

Cross-reference heavy content - Chunks that say "see Section 4.2" lose meaning without the referenced section. Consider graph-based retrieval (GraphRAG) or parent-child chunking.

Real-time content - Chat logs and streaming data need window-based chunking with time boundaries, not static document splitting.

Multimodal content - Images, charts, and diagrams within documents lose meaning when only surrounding text is chunked. Multimodal embeddings or caption extraction are needed.

Non-text content - Audio transcripts, spreadsheets, and JSON documents require domain-specific chunking logic that general text splitters don't provide.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Tip

Prepend section headings to chunk text before embedding. This single change often improves recall@k by 10–15% on structured documentation.

Ecosystem

• LangChain: RecursiveCharacterTextSplitter, MarkdownHeaderTextSplitter, semantic chunkers.

• LlamaIndex: SentenceSplitter, SemanticSplitterNodeParser, HierarchicalNodeParser.

• Unstructured: Layout-aware parsing that preserves document structure before chunking.

• Docling: IBM's document parser with structure-preserving chunking for PDFs.

• chonkie: Lightweight chunking library with semantic and sentence-based splitters.

Related Technologies

• RAG: The pipeline chunking feeds into.

• Embedding Models: What converts chunks to vectors - model choice interacts with chunk size.

• Retrieval Evaluation: How you measure whether chunking works.

• Vector Databases: Where chunks are stored and searched.

Learning Path

Prerequisites: RAG · Embeddings

Next topics: Embedding Models · Retrieval Evaluation · Re-ranking

Estimated time: 40 min · Difficulty: Intermediate

FAQs

What chunk size should I use?

Start with 512 tokens and 50-token overlap. Evaluate recall@k. Decrease to 256 for factoid Q&A; increase to 768–1024 for analytical questions needing context.

How much overlap do I need?

10–15% of chunk size. For 512-token chunks, use 50–75 token overlap. Overlap prevents information loss at boundaries.

Should I chunk by tokens or characters?

Tokens - because embedding models and LLMs operate on tokens. Use tiktoken or the model's tokenizer for accurate counts.

Does chunk size affect embedding quality?

Yes. Very long chunks produce vague embeddings averaging multiple topics. Very short chunks may lack enough signal. Match chunk size to the granularity of your expected queries.

When should I use semantic chunking?

When documents cover multiple topics within paragraphs and recursive splitting creates mixed-topic chunks. Test on your data - semantic chunking adds indexing cost.

How do I chunk PDFs?

Use layout-aware parsers (Unstructured, Docling) that detect headings, tables, and columns. Chunk by section after parsing. Never chunk raw PDF text without parsing.

How do I chunk code?

Split by AST nodes - functions, classes, methods. Each chunk should be a compilable unit with its imports. Tools like LlamaIndex's CodeSplitter handle this.

What is parent-child chunking?

Index small child chunks (256 tokens) for precise retrieval. Store a link to a larger parent chunk (1024 tokens). When a child matches, return the parent to the LLM for full context.

How often should I re-chunk?

Re-chunk when changing chunk size, overlap, or strategy. Re-chunking requires re-embedding. Plan for a full re-index pipeline, not ad-hoc changes.

Can I use different chunk sizes for different document types?

Yes - and you should. Policies might use section-based chunking; chat logs might use time-window chunking; code uses function-based splitting. Route by content type.

How do I handle tables in documents?

Keep tables as single chunks. If a table exceeds max chunk size, split by rows but repeat column headers in each sub-chunk.

Does chunking strategy affect hybrid search?

Indirectly. Better chunks improve both vector and keyword retrieval. Chunk boundaries affect which terms appear together in BM25-indexed text.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Chunking determines the unit of retrieval - its quality is the foundation of RAG performance.
• Start with recursive splitting at 512 tokens with 50-token overlap.
• Prepend section headings to chunks for better embedding and retrieval.
• Use document-aware chunking for structured content; semantic chunking for multi-topic documents.
• Always evaluate chunking choices with recall@k - don't guess.
• Different document types need different chunking strategies.