Chunking Strategies for RAG: How Document Splitting Decides Retrieval Quality

The chunk is the unit of retrieval. Whatever you split a document into is what gets embedded, what gets scored against the query, and what is handed to the model. Nothing downstream repairs a bad split: a reranker cannot promote a passage that was never a candidate, and a bigger context window cannot supply a sentence that was never indexed.

Two hard limits follow. You cannot retrieve information that is not in a chunk. And you cannot reliably retrieve a concept split across two chunks, because each half is a weak match for a query about the whole, so neither surfaces. When a RAG system confidently answers the wrong thing, the cause is usually not the model's reasoning: the retrieved passages did not contain the answer.

So before swapping embedding models or bolting on a reranker, read what you indexed: pull the top-{k} chunks for ten queries your system gets wrong.

The central tension

Chunk size is one dial with an opposing failure mode at each end.

Small enough to mean one thing. An embedding is a single point in vector space. Embed a passage covering three topics and you get one point, in the middle, near none of them, so a query about topic two loses to a tighter chunk about topic two alone. This is semantic dilution: the more distinct ideas packed into one embedding, the weaker its match on any of them. A topic-hopping chunk does not match many queries well. It matches all of them badly.

Large enough to be useful alone. The chunk is also what the model reads. A sentence saying "this increases throughput substantially" is a sharp embedding target and useless evidence: what, under what conditions? At generation time the chunk stands alone, stripped of its document and neighbours, so it must carry its own context. Most strategies below try to escape this tradeoff rather than split it.

Strategy by strategy

Fixed-size

Split every N units, ignore content. Deterministic, trivially parallel, and the baseline to beat.

Split by tokens, not characters. The model's limit is in tokens, and the token-to-character ratio swings hard by content type: English prose runs about four characters per token, while code, JSON, tables and non-Latin scripts run far denser. Use the model's own tokenizer and the ceiling is exact.

Fails by cutting mid-sentence, mid-word, mid-table. Starting point to tune from: 512 tokens, no overlap.

Fixed-size with overlap

Each chunk repeats the tail of the previous one. What that buys is specific: a concept straddling a boundary appears whole in at least one chunk. If a definition spans the boundary between chunks 7 and 8, a 128-token overlap puts all of it in chunk 8; without overlap neither holds it and the query matches neither.

Overlap of {p} percent inflates the index by roughly {p} percent, and one passage can take two slots in your top-{k}, so deduplicate on content hash before assembling the prompt.

Starting point: 10 to 15 percent of chunk size, enough to span the sentence or two that usually straddles a boundary. Past 25 percent you mostly buy index size.

Recursive character splitting

The pragmatic default. Supply an ordered separator list, paragraph break then line break then sentence terminator then space; the splitter tries the highest-priority separator first and recurses into any oversized piece with the next one down. Breaks land on natural boundaries when one exists, so paragraphs stay intact when they fit and a 3,000-token wall of text still gets split. For mixed or unpredictable text, start here.

Fails because it respects typography, not meaning: two unrelated paragraphs get merged if they fit, and one argument spanning four gets cut in the middle if it does not.

Starting point: those separators, 512 tokens, 64-token overlap.

Document-structure-aware

If documents have real structure, split on it: Markdown headings, HTML sections (parse them, do not regex them), PDFs with a usable outline. Chunk at section boundaries, keep each heading with its body, and recurse only on oversized sections.

Where genuine structure exists this usually beats everything else, semantic chunking included, because the author already did the segmentation: a heading is an explicit declaration that what follows is one coherent topic, and no heuristic matches it.

Carry the heading into the chunk text, since a body without its heading has lost its subject, and preserve the hierarchy as metadata for filtering and breadcrumbs.

Fails when structure is uneven: scanned PDFs have no outline, and heading density varies wildly. Merge small sections, recurse on oversized ones.

Starting point: split at the second and third heading levels, merge sections under 100 tokens, recursively split anything over 800.

Semantic chunking

Split where the topic changes, detected by embedding. Segment into sentences, embed each, compute similarity between adjacent sentences, and cut where it drops sharply, against a percentile of that document's own distribution of drops rather than a fixed constant.

The cost is that you must embed at split time: every sentence gets an embedding purely to decide where the cuts go, and those embeddings are then discarded. That is a real ingest bill and a slower pipeline, paid again on every reindex.

The benefit is inconsistent: on flowing prose with topic drift and no headings it finds boundaries nothing else finds, but on documents that already have headings it mostly rediscovers what the headings gave you, at much greater cost.

Starting point: threshold around the 90th to 95th percentile of adjacent-sentence distance, plus hard size clamps so a flat region cannot emit one enormous chunk.

Sentence-window and small-to-big

The most useful idea here, so take it slowly.

The tension exists because one piece of text does two jobs: it is the thing matched, and the thing read. Matching wants a small, sharp, single-topic unit. Reading wants surrounding context. So stop using the same text for both.

Index a small unit, a sentence or two, and attach a pointer to a larger window: the sentences either side, the paragraph, or the enclosing section. You match on the small unit and return the window. Most chunk-size agonising then evaporates, because you are no longer trading precision against usefulness; the price is one extra field per record and a fetch step.

Two shapes. Sentence window: index single sentences, return the sentence plus {n} either side; good for fact lookup, where the answer is one sentence that needs neighbours to be intelligible. Small-to-big, or parent document: index children of a few hundred tokens, return parents of a few thousand; good for explanatory content where the useful unit is a whole argument.

Deduplicate parents, or three children from one section will paste it into the prompt three times; collapse to unique parents and raise {k}, since collapsing shrinks the result set.

Starting point: 256-token children, 1,024 to 2,048-token parents, k=10, collapsed, capped at 4 parents.

Hierarchical and parent-document indexes

The generalisation: a tree of document, section, subsection and paragraph, each node pointing at its parent. Retrieval expands upward for context, or routes downward by matching a section summary and then searching only that section's children. Summary nodes help with thematic queries, since a summary embeds a section's overall topic in a way no single child does. It is also the most expensive to operate: summaries need regenerating whenever content changes.

Specialised content

Code. Split on function, method and class boundaries, never mid-function: half a function is not a partial answer but a misleading one: the reader sees an early return and never the error handling below the cut. Use a parser, not a regex, and keep the enclosing class signature and file path in metadata, since a method body alone rarely reveals what type it operates on.

Tables. Never separate a table from its header row. Body rows without column names are worse than useless: retrievable, plausible-looking and unreadable. A chunk reading "yes, 40, 2048, deprecated" says nothing about what those values describe, and a model handed it will invent the column meanings rather than decline. Repeat the header on every chunk of a long table, keep the caption, and set a ceiling high enough that most tables survive intact.

Transcripts and chat logs. Split on speaker turns and time gaps, not length, and never cut mid-turn, since half an answer attributed to a speaker is a misquote. Keep the speaker label in the chunk text, not only in metadata. Group consecutive turns up to your size limit, treating a long silence as a hard boundary; in support chats the question-and-resolution pair is the unit worth retrieving.

Comparison

Every configuration below is a starting point to tune from, not a measured optimum.

Contextual enrichment

A chunk reaches the model with no memory of where it came from. Consider one that reads, in full: "It supports up to 40 concurrent connections per instance." Fine sentence, worth nothing. What is "it"? Which product, which version? The embedding is equally lost: that vector is about concurrency limits in the abstract, so a query naming the product matches it weakly.

Prepend provenance to the chunk text before embedding it:

Product Handbook v4 / Deployment / Scaling limits / Connection pooling

It supports up to 40 concurrent connections per instance.

Now the chunk answers "40 what, in what?" by itself, and its vector sits near queries naming the product or the section. Build the prefix from document title plus the heading breadcrumb your splitter already produced.

Two cautions. The prefix spends tokens from the chunk budget, so keep it to one line. And enrich before embedding: a prefix added at prompt-assembly time helps the model read but does nothing for retrieval, which is where the failure happened.

Metadata is a separate mechanism. Attach source URI, document ID, section path, date, product version and access tags as filterable fields rather than text. They do not shape the embedding; they constrain the search space before or during the query. Filtering to the current product version removes a whole class of confidently-wrong answers drawn from deprecated docs, which no chunking cleverness achieves.

Chunk size and the embedding model

Every embedding model has a maximum input sequence length, and text beyond it is not an error, it is silently truncated. A chunk longer than that limit is partly invisible to the index: the tail still reaches the model when the chunk is retrieved, but it contributed nothing to the vector deciding whether the chunk is ever retrieved. That is the worst failure mode in the system, content that is present but unfindable, with no log line anywhere.

Look up your model's maximum sequence length and treat it as a hard ceiling, not a target. Measure with that model's tokenizer, leave headroom for the enrichment prefix, and assert the limit at ingest, so an oversized chunk is caught the day it appears.

A model accepting long inputs does not necessarily embed them usefully: semantic dilution applies regardless of what the limit permits. And the best chunk size is a joint property of your corpus and your model, not a universal constant: the 512 that works for support articles can be wrong for statutes, and wrong for the same corpus after you change embedding models.

How to actually tune this

Intuition about chunk size is usually wrong, including yours and mine. Measuring it is cheap, and almost nobody does it.

Build an evaluation set. Collect 30 to 50 real questions from support tickets, query logs, or the people who will use the system, and record the passage that actually answers each. Under about 30 questions the noise swamps the signal.

Measure retrieval, not answers. Check whether the correct source passage appears in the top {k} retrieved chunks. That is recall at k, and it isolates chunking completely: no generation, no prompt effects, no model-as-judge.

for config in configurations:
    index = build_index(corpus, config)
    hits = 0
    for question, gold_passage in eval_set:
        retrieved = index.search(question, k=10)
        if any(overlaps(chunk, gold_passage) for chunk in retrieved):
            hits += 1
    report(config.name, hits / len(eval_set))

Change one thing at a time. Same embedding model, same k, same query preprocessing; only the chunking config varies.

Sweep a real range. Do not test 500 against 512. Test 128, 256, 512 and 1024; overlap at 0, 10 and 25 percent; the two or three strategies plausible for your content.

Then read the misses by hand. The aggregate says which config is better; the individual failures say why, and that is where the truncated tables and orphaned headings turn up.

On a few thousand documents this is an afternoon and a modest embedding bill, and the only way to get a real answer.

Reindexing is the real cost

Changing chunking means re-embedding the entire corpus. There is no incremental path: every vector came from a specific piece of text, and you just changed the pieces. Cost scales with total tokens, which overlap multiplies, and wall-clock time by rate limits.

• Version your index. Record strategy, size, overlap, enrichment template and embedding model in the index name or a metadata record.
• Build into a new index and swap. Do not mutate in place: build alongside, run the eval set against both, switch the read path, keep the old index until you are confident.
• Keep the raw documents. If your only copy is the chunked form, every future strategy change becomes a re-ingest from source systems.
• Make ingest re-runnable per document. Re-indexing one document by ID should be routine.

A decision checklist

1. Real structure, meaning headings and an outline? Structure-aware splitting, recursing inside any oversized section. Usually the best answer available.
2. Code? Function and class boundaries with a language-aware parser; carry the enclosing signature and file path.
3. Tabular? Header with every fragment, plus the caption.
4. Transcript or chat log? Speaker turns and time gaps, never mid-turn, speaker labels in the text.
5. Unstructured prose with no headings? Start recursive, then try semantic chunking and compare on your eval set.
6. None of the above, or a mixture? Recursive at 512 tokens with 10-15 percent overlap.

Then, whichever you picked:

7. Apply small-to-big unless you have a specific reason not to. Index the small unit, return the larger one.
8. Prepend title and heading breadcrumb before embedding, and attach source, date, version and access tags as filterable metadata.
9. Check the embedding model's maximum sequence length and assert chunk token counts below it at ingest.
10. Build the 30-question eval set and measure recall at k across three or four configs. Expect to be surprised.

The chunk is the unit of retrieval. Everything downstream operates on the decisions you made here, and none of it can undo them.