Your Chunking Benchmark Is Measuring Chunk Size

Every chunking comparison I’ve read reports hit rate at a fixed k: retrieve the top 5 or top 10 chunks and check whether the answer is in there. Run it that way and the strategies look close. Run it against a fixed token budget, which is what your context window actually is, and the ranking changes completely.

I built the benchmark so that it scores both ways.

Setup

SQuAD is useful here for one reason: every question has an answer with a known character offset in a known article. I rebuilt the 48 articles of the dev set into whole documents, kept 2,000 random questions, and counted a retrieved chunk as a hit only if the entire answer span sits inside it. No LLM judge, no fuzzy matching.

Embeddings are bge-small-en-v1.5 via fastembed. Token counts use cl100k_base. Fourteen strategies:

• Fixed-size by token count, 128 to 1,024, two of them with 15% overlap.
• Recursive: paragraphs first, sentences inside oversized paragraphs, hard cuts as a last resort, packed up to a size limit. That’s what LangChain’s RecursiveCharacterTextSplitter approximates.
• Sentence-packed: sentences joined up to a limit, ignoring paragraphs.
• Semantic: embed every sentence, split where adjacent sentences are least similar (the 90th or 95th percentile of dissimilarity), as LangChain’s SemanticChunker does.
• Paragraph: the document’s own paragraph breaks, untouched. More on why this one is special below.

The results

Strategy	Mean tokens	hit@1	hit@10	Within 1,000 tokens
paragraph (gold boundary)	161	68.7%	93.5%	90.1%
recursive-128	104	65.0%	91.6%	90.9%
recursive-256	187	62.5%	92.0%	86.9%
recursive-512	416	51.7%	89.8%	66.3%
sentence-256	232	55.0%	89.2%	80.5%
sentence-512	476	47.7%	88.6%	58.3%
fixed-128	127	50.5%	86.7%	83.2%
fixed-256	251	49.0%	87.5%	71.9%
fixed-256, 15% overlap	253	50.6%	89.6%	74.0%
fixed-512	494	44.9%	87.8%	46.1%
fixed-512, 15% overlap	499	45.8%	89.8%	47.5%
fixed-1024	954	39.3%	85.3%	3.9%
semantic-p90	284	44.9%	83.4%	53.6%
semantic-p95	458	42.0%	82.5%	37.1%

Four things in that table.

Fixed k flattens the size question

Look at the fixed-size rows at hit@10: 86.7%, 87.5%, 87.8%, 85.3%. Chunk size barely matters. That’s the result most chunking posts report, and it’s the reason “chunk size doesn’t matter much” has become folk wisdom.

Now look at the last column. Within a 1,000-token budget, 128-token chunks retrieve the answer 83.2% of the time and 1,024-token chunks 3.9%. At top-10 you’ve handed the big chunks nearly 10,000 tokens of context to be right in; at a fixed budget, one of them fills the whole window and it had better be the right one.

Your prompt has a budget, not a k. Score against the budget.

Structure beats cleverness, and it’s free

Recursive splitting at 256 tokens beats fixed splitting at 256 tokens by 13.5 points at hit@1 (62.5% against 49.0%) and 4.5 at hit@10. The two produce similar-sized chunks. The difference is that one of them never cuts a sentence in half.

That matters most at k=1, which is the number that counts when you pass one chunk to a model, or when a reranker is deciding what to keep.

Semantic chunking lost to everything

Both semantic variants came in below plain fixed-size chunks at every size. They were also the only strategies with a real compute cost: roughly 300 seconds of wall-clock to chunk the corpus, against about a second for everything else, because every sentence has to be embedded before any boundary can be chosen.

This matches Qu, Tu and Bao’s finding that semantic chunking’s costs “are not justified by consistent performance gains.” My guess at why: a similarity dip between two sentences is weak evidence of a topic boundary, while a paragraph break is strong evidence, and the paragraph breaks are already in the file.

Overlap is a cheap, small win

Fifteen percent overlap on 256-token chunks: hit@10 up from 87.5% to 89.6%, and the number of answers split across two chunks went from 1.0% to zero. Cost: 16% more chunks to embed and store. If you must use fixed-size chunks, take the overlap.

About that “gold boundary” row

Paragraphs won hit@1 and hit@10, and I want to be clear about why that’s partly a property of the dataset. SQuAD’s questions were written against single paragraphs, so by construction every answer sits inside one. Paragraph chunking is an upper bound here, not a universal recommendation. I’m including it because it tells you what perfect boundaries are worth: about 4 points of hit@1 over the best automatic strategy.

The lesson still holds outside SQuAD. Your documents have structure: headings, paragraphs, list items, function bodies, table rows. The chunker that respects it will usually beat the one that counts tokens, and both will beat the one that computes boundaries from embeddings.

What I’d do

1. Pick the token budget before the chunk size. If retrieval feeds a 1,500-token context slot, 1,024-token chunks give you one shot. Five 256-token chunks give you five.
2. Split on structure first. Paragraphs, then sentences, then hard cuts. Code splits on functions, docs on headings. The recursive splitter you already have does this.
3. Add overlap only to fixed-size chunks. Structured chunks don’t cut mid-sentence, so there’s little for overlap to repair.
4. Skip semantic chunking unless you’ve measured a win on your own corpus. The evidence is against it and the cost is real.
5. Measure with answer spans, not vibes. You need labelled questions. Fifty from your own documents, with the answer location marked, will teach you more than any public benchmark.

Chunking decides what can be retrieved. What is retrieved depends on the index and the ranking, and I’m covering where hybrid search and reranking actually help next. For the fundamentals, start with what RAG is and what an embedding is.

Limits

One dataset of encyclopedia prose with short factoid answers; long technical documents, code and tables will behave differently. One small embedding model. The benchmark measures retrievability, not whether a model answers correctly from the chunk. The “within budget” metric fills the budget in rank order and stops, which is the simplest policy and not the only one.

Code, tests and the full results: ajinb/chunking-bench. bench.py runs on a laptop, and you can point load_documents at your own corpus.