Chunking Strategies for RAG, Benchmarked on Real Documents

Most RAG postmortems start in the wrong place. Someone swaps the embedding model, adds a reranker, tunes the prompt, and the retrieval quality barely moves — because the actual damage happened earlier, at the chunking step, before any of those components ever saw the text. Chunking is usually the first thing implemented and the last thing revisited, which is backwards: it’s the one decision that determines what’s even possible to retrieve. If a chunk boundary splits a table from its caption, or a definition from the sentence that uses it, no embedding model and no reranker can put it back together at query time.

The reason chunking gets skipped over is that most tutorials treat it as a solved, boring step — split every 500 tokens with 50 tokens of overlap and move on to the interesting parts. But chunk size, overlap, and boundary logic have measurable effects on both recall (did you retrieve the chunk with the answer at all?) and precision (how much of what you retrieved was actually relevant?). This post runs four chunking strategies — fixed-size, recursive, semantic, and hierarchical — against the same small corpus, with a methodology you can reuse on your own documents, plus worked example numbers from that test set to show what the differences look like in practice.

You’ll learn:

• The real mechanical difference between fixed-size, recursive, semantic, and hierarchical chunking, with runnable Python for each
• How chunk size and overlap trade off recall against precision, with worked numbers from a small test corpus
• Why semantic chunking helps meaningfully on some documents and does almost nothing on others
• How hierarchical chunking preserves context flat chunking throws away, and when that’s worth the added complexity
• A reproducible methodology for benchmarking chunkers against your own corpus and your own queries
• How to pick a sane default chunking strategy for a new RAG pipeline without over-engineering the first version

Table of Contents

1. Why Chunking Is the Retrieval Decision Everyone Skips
2. Four Chunking Strategies, Defined
3. Building a Small, Honest Benchmark
4. Results: Recall and Precision by Strategy
5. Chunk Size and Overlap: What Actually Moves the Needle
6. When Semantic Chunking Earns Its Cost
7. Hierarchical Chunking for Long, Structured Documents
8. Choosing a Default for Your Own Corpus
9. Where to Go From Here

Why Chunking Is the Retrieval Decision Everyone Skips

Retrieval quality is bounded by chunking before it’s bounded by anything else. An embedding model can only encode what’s inside a chunk’s boundaries, and a reranker can only re-score chunks that made it into the candidate set. If your chunk boundaries cut a paragraph in half so the sentence containing the answer sits alone with no surrounding context, the embedding for that chunk represents a fragment, not the idea — and it will rank lower for a query that’s really asking about the whole idea.

There are two failure modes people conflate under “bad retrieval,” and they need different fixes:

• Recall failures: the chunk containing the answer was never retrieved at all, because it wasn’t split out as a coherent unit, or it’s buried inside a chunk so large that the embedding is diluted by unrelated content.
• Precision failures: the chunk was retrieved, but it’s noisy — padded with unrelated boilerplate from neighboring sections, so the LLM has to do extra work separating signal from filler, which increases hallucination risk even when the right information is technically present.

Fixed-size chunking is prone to both. Splitting purely by token count with no regard for document structure will, on a long enough corpus, sometimes split a table row from its header, sometimes split a code block mid-function, and sometimes land a boundary in the middle of a sentence. Every one of those is a chunk that’s harder to retrieve correctly and harder to reason over once retrieved. The strategies below are, in order, increasingly aware of document structure and semantic content — and increasingly expensive to compute.

Four Chunking Strategies, Defined

Fixed-Size Chunking

Split the raw text into windows of N tokens (or characters), usually with some overlap so an idea that straddles a boundary appears in full in at least one chunk.

def fixed_size_chunks(text: str, chunk_size: int = 400, overlap: int = 50) -> list[str]:
    words = text.split()
    chunks = []
    start = 0
    while start < len(words):
        end = start + chunk_size
        chunks.append(" ".join(words[start:end]))
        start += chunk_size - overlap
    return chunks

This is fast, deterministic, and completely blind to structure. It’s the right baseline to measure everything else against, not the strategy to ship without comparison.

Recursive Chunking

Recursive splitters (the default in LangChain’s RecursiveCharacterTextSplitter) try a list of separators in priority order — paragraph breaks, then sentence breaks, then word breaks — and only fall back to a harder split when a piece is still too large after trying the gentler one.

def recursive_chunks(text: str, chunk_size: int = 400, separators=None) -> list[str]:
    separators = separators or ["\n\n", "\n", ". ", " "]
    if len(text.split()) <= chunk_size or not separators:
        return [text]
    sep, rest = separators[0], separators[1:]
    parts = text.split(sep)
    chunks, buffer = [], ""
    for part in parts:
        candidate = (buffer + sep + part) if buffer else part
        if len(candidate.split()) <= chunk_size:
            buffer = candidate
        else:
            if buffer:
                chunks.append(buffer)
            buffer = part if len(part.split()) <= chunk_size else None
            if buffer is None:
                chunks.extend(recursive_chunks(part, chunk_size, rest))
                buffer = ""
    if buffer:
        chunks.append(buffer)
    return chunks

This respects paragraph and sentence boundaries wherever it can, which alone removes most of the mid-sentence and mid-table splits that plague fixed-size chunking, at essentially zero extra compute cost.

Semantic Chunking

Rather than splitting on structural characters, semantic chunking splits on meaning. Embed each sentence, walk through the document comparing the embedding similarity between consecutive sentences, and cut a new chunk boundary wherever similarity drops below a threshold — the signal that the topic just shifted.

import numpy as np

def semantic_chunks(sentences: list[str], embed_fn, threshold: float = 0.75) -> list[list[str]]:
    embeddings = [embed_fn(s) for s in sentences]
    chunks, current = [], [sentences[0]]
    for i in range(1, len(sentences)):
        sim = cosine_sim(embeddings[i - 1], embeddings[i])
        if sim < threshold:
            chunks.append(current)
            current = []
        current.append(sentences[i])
    if current:
        chunks.append(current)
    return chunks

def cosine_sim(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

Semantic chunking produces variable-length chunks that track actual topic boundaries instead of arbitrary token counts, at the cost of an embedding call per sentence during ingestion.

Hierarchical Chunking

Hierarchical (or “parent-child”) chunking keeps two granularities at once: small child chunks for precise embedding and retrieval, linked back to a larger parent chunk (a whole section, or a whole document) that gets passed to the LLM once its child is retrieved. This uses structural signals — markdown headers, section numbering — to build the tree.

def hierarchical_chunks(sections: dict[str, str], child_size: int = 200) -> list[dict]:
    records = []
    for heading, body in sections.items():
        parent_id = heading
        for child in recursive_chunks(body, chunk_size=child_size):
            records.append({"parent_id": parent_id, "parent_text": body, "child_text": child})
    return records

At query time you search over the small child_text embeddings for precision, then hand the associated parent_text to the generation step so the model sees full context rather than an isolated fragment.

Building a Small, Honest Benchmark

To make the tradeoffs concrete, I ran all four strategies against a small, purpose-built test set — twenty markdown documents pulled from internal engineering runbooks and API reference pages, averaging around 1,800 words each, with a mix of prose, tables, and code blocks. This is deliberately a toy corpus, not a large-scale published study: the numbers below are illustrative worked examples meant to demonstrate the methodology on a corpus small enough to inspect by hand, not a claim about chunking performance in general. The point of this section is that you can rebuild this exact pipeline on your own documents in an afternoon.

Building the eval set. For each document, I wrote two to three natural-language questions whose answer was contained in a specific passage, then recorded which sentence(s) in the source document actually answered it — the “gold” span. That produced 47 query/gold-span pairs across the 20 documents.

Indexing. Each strategy chunked all 20 documents, and every resulting chunk was embedded with the same embedding model (keeping the model constant isolates the chunking variable — mixing in a different embedding model per strategy would confound the results).

Metric definitions:

• Recall@5: for each query, retrieve the top 5 chunks by cosine similarity; count it a hit if any retrieved chunk contains the gold span (or a strict superset/subset overlapping it).
• Precision@5: of the 5 retrieved chunks, what fraction contain content that’s actually relevant to the query, judged against the gold span and its immediate context.

def recall_at_k(retrieved: list[str], gold_span: str, k: int = 5) -> int:
    return int(any(gold_span in chunk or chunk in gold_span for chunk in retrieved[:k]))

def precision_at_k(retrieved: list[str], relevant_ids: set[str], k: int = 5) -> float:
    top_k = retrieved[:k]
    hits = sum(1 for c in top_k if c in relevant_ids)
    return hits / k

This is the same skeleton used for full pipeline evaluation, just with the generation step removed so you’re isolating retrieval quality alone.

Results: Recall and Precision by Strategy

Worked numbers from the 47-query, 20-document test set described above, all at chunk sizes tuned per-strategy to their natural defaults (400 tokens, ~50-token overlap for fixed and recursive; a 0.75 similarity threshold for semantic; 200-token children for hierarchical):

Two patterns stood out on this test set, and both matched what the mechanism predicts rather than being surprising in hindsight. First, the jump from fixed-size to recursive was larger than the jump from recursive to semantic — respecting paragraph and sentence boundaries captured most of the easy wins, and semantic chunking’s extra cost bought a smaller, second-order improvement on top of that. Second, hierarchical chunking’s recall advantage came almost entirely from queries whose gold span required context from outside the immediately retrieved sentence — a table’s meaning depending on the paragraph introducing it, for instance — where the parent chunk supplied context the child alone couldn’t.

Precision moved with recall in every case here, which isn’t guaranteed in general — it’s a property of this corpus having reasonably distinct topics per section. A corpus with denser cross-referencing between sections could plausibly show precision diverging from recall in ways this small test set didn’t surface.

Chunk Size and Overlap: What Actually Moves the Needle

Holding recursive chunking constant and sweeping chunk size across the same 47 queries:

The pattern here is the standard chunk-size tradeoff, just made concrete: smaller chunks push precision up (less irrelevant content per chunk) but cap recall on queries whose answer spans more than one chunk’s worth of text; larger chunks recover some recall by keeping more context together, but precision drops because each retrieved chunk drags in more unrelated surrounding text. Overlap is a cheap partial fix for the boundary problem specifically — it makes it less likely that an idea splitting across two chunks is fully lost in both, at the cost of redundant storage and, at high overlap ratios, near-duplicate chunks competing for the same retrieval slots.

The practical takeaway from this sweep: there’s rarely a universally “correct” chunk size — there’s a size that matches how self-contained your source documents’ ideas typically are. Short, list-like reference docs want small chunks; narrative or explanatory prose with dependent context wants larger ones or a hierarchical fallback.

When Semantic Chunking Earns Its Cost

Semantic chunking’s gain over recursive chunking on this test set wasn’t uniform across documents — it concentrated almost entirely in prose-heavy documents with irregular paragraph lengths (a design doc with one three-sentence paragraph followed by a fourteen-sentence one), where recursive chunking’s paragraph-based splits either cut a long paragraph awkwardly or bundled several short, topically distinct paragraphs into one chunk. On the API reference pages, which were already broken into short, single-topic paragraphs by their authors, semantic chunking’s boundaries landed in almost the same places recursive chunking’s did — the extra embedding-per-sentence cost bought close to nothing, because the document was already structurally aligned with its own semantic boundaries.

That’s the actual decision rule: semantic chunking pays off in proportion to how badly structured your source documents are relative to their actual topic boundaries. If your corpus is already well-organized markdown or well-formatted docs, recursive chunking captures most of the same signal for a fraction of the ingestion cost. If your corpus is scraped HTML, OCR output, or free-form user-submitted text with inconsistent formatting, semantic chunking’s independence from document structure becomes a real advantage rather than a redundant one.

Hierarchical Chunking for Long, Structured Documents

Hierarchical chunking’s advantage isn’t really about retrieval precision at the embedding step — child chunks in this benchmark used the same underlying recursive splitting as the recursive-only strategy. The advantage shows up downstream, at generation time: because each retrieved child carries a pointer to its full parent section, the context handed to the LLM after retrieval is a coherent section instead of an isolated 200-token fragment. This is precisely the shape of fix that helps when retrieval keeps returning technically-matching but contextually incomplete chunks — the embedding step was never the problem; the chunk boundary throwing away surrounding context was.

The tradeoff is storage and pipeline complexity: you’re now maintaining two chunk granularities, a parent-child mapping, and usually a two-step retrieval where the child index is searched but the parent text is what actually gets returned. For documents under a few thousand tokens, this is often unnecessary — just embed the whole thing as one chunk. Hierarchical chunking earns its complexity on long, structured documents (specs, runbooks, contracts, multi-section reports) where a query’s answer plausibly depends on context several paragraphs away from the sentence that most directly answers it.

Choosing a Default for Your Own Corpus

Given the results above, a reasonable decision order for a new pipeline:

1. Start with recursive chunking at a size matched to your documents’ natural idea length. For prose-heavy content, 300–500 tokens with 10–15% overlap is a solid starting point; measure before assuming you need to tune further.
2. Add semantic chunking only if your corpus is structurally messy — inconsistent formatting, scraped content, or documents where paragraph boundaries don’t reliably track topic boundaries. Measure the recall lift on your own eval set before committing to the added ingestion cost; it may be smaller than expected, as it was here on the already-clean API docs.
3. Add hierarchical chunking specifically for long, structured documents where context beyond the immediate chunk matters — not as a default for every corpus. The complexity cost is real, and it isn’t refunded on documents short enough to just retrieve whole.
4. Always run the recall@k / precision@k comparison on your own documents and queries before trusting any general benchmark, including this one — the 20-document test set here demonstrates the methodology and the direction of the effects, not a universal ranking for every corpus.

None of these strategies fix a chunking decision made badly at a different level — chunk size and boundary logic won’t help if the metadata needed to filter results correctly (tenant, document version, access level) was never attached to the chunk in the first place, which is its own, separate failure mode.

Where to Go From Here

Chunking sets the ceiling on what retrieval can find; the rest of a RAG pipeline decides how well it uses what’s under that ceiling. If this post solved the “did we split the document sensibly” problem, these cover what to do with the chunks once they exist:

• RAG in Production: Why It Works in Demos, Fails Live — semantic chunking is one of three fixes covered there, with before/after numbers from a live system.
• pgvector or a Dedicated Vector Database? — the storage decision that sits right after chunking, once you know what you’re embedding and how many chunks you’ll have.

Wrapping Up

Chunking earns the amount of engineering attention people give to embedding models and rerankers, and usually gets a fraction of it. The four strategies here aren’t ranked in some absolute order — fixed-size is a reasonable prototype baseline, recursive is a strong general default, semantic chunking pays for itself on structurally messy corpora, and hierarchical chunking earns its complexity on long documents where answers depend on surrounding context. The number that should change your mind about which one to use isn’t a benchmark you read on a blog — including this one — it’s recall@k and precision@k measured on your own documents, with your own queries, because that’s the only test set your production system actually has to answer to.