Most retrieval-augmented generation (RAG) systems start with one tunable: chunk size. Teams try 512 tokens, then 1024, watch a benchmark move a little, and call the chunking problem solved. But chunk size stands in for three separate decisions: how much context each chunk carries, how often a thought is cut in half, and how much irrelevant text rides along for the ranker to wade through. Getting those right matters more than finding a magic number.

Fixed-size splitting optimizes the wrong thing

A fixed-size splitter has one job: cut every N tokens. It doesn't know that "Section 4.2" began three tokens ago, or that a table's header row and its data have ended up in different chunks. The result is retrieval that scores a truncated paragraph as a strong match because it contains the query terms, while the sentence that actually answers the question sits in the next chunk and never gets retrieved.

Recursive splitting is the usual cheap improvement: cut on paragraph boundaries first, fall back to sentences, then to words, only when a section still exceeds the target size. You lose uniform chunk sizes, which is a fine trade. The goal is useful embeddings, not uniform ones.

Overlap is a targeted fix, not a default

Overlapping chunks, where each repeats the tail of the previous one, exist to stop a fact from being orphaned at a boundary. That's a real failure and overlap does fix it. But overlap also inflates the index, slows retrieval, and can retrieve the same fact twice. If your documents already have strong structure (headings, list items, table rows), you often need little or none. Add it where you've observed boundary loss.

The deeper problem: a chunk without its context

Even a well-cut chunk can be unintelligible alone. Anthropic's Contextual Retrieval write-up gives a clear example: a chunk saying "The company's revenue grew by 3% over the previous quarter" doesn't say which company or which quarter. The write-up argues that traditional chunking can destroy context at the chunk level, which causes the system to fail to retrieve relevant information.

Their approach is to prepend a short, chunk-specific explanation of where the chunk sits in the document before embedding and indexing it. In their evaluation, the share of failed retrievals fell as the technique was layered in:

  • Contextual embeddings alone: from 5.7% to 3.7% (a 35% reduction).
  • Plus contextual BM25 keyword search: to 2.9% (49%).
  • Plus reranking: to 1.9% (67%).

These are Anthropic's results on its own test datasets, so treat them as an indication of what's possible, and measure on your own corpus. They also report that passing the top 20 chunks to the model worked better than 5 or 10, and that generating the context cost roughly $1.02 per million document tokens when using prompt caching, a one-time indexing cost. For how that caching works and why it matters, see our prompt-caching guide.

Test chunking like a ranking function

The only reliable way to choose a strategy is to build a small set of real questions with known correct source passages, run retrieval with each candidate strategy, and check whether the correct passage shows up in the top results. Don't judge by whether the final generated answer looks plausible. Generation can hide a chunking problem, because a model can sound confident on partial information. A retrieval-recall check can't.

A checklist

  1. Split on structure first. Use headings, paragraphs and table boundaries, and keep tables with their headers.
  2. Add overlap only where you've seen boundary loss.
  3. Attach context (document title, section, a one-line summary) to each chunk so it can be understood on its own. Prefixing metadata is a cheap start; generating a short contextual blurb per chunk is the fuller version described above.
  4. Retrieve on keywords and meaning together; see hybrid search.
  5. Measure retrieval recall on a held-out question set before touching the size knob.

The takeaway

Choose a splitter that respects document structure, give every chunk enough context to stand alone, add overlap only where it's needed, and validate with a recall test set. Chunk size is the last knob to turn, not the first.

Sources

Developer ToolsLarge Language ModelsSoftware Engineering