When a retrieval-augmented feature gives disappointing answers, the instinct is to blame the embedding model or the prompt. Often the real culprit is earlier in the pipeline: the way documents were cut into chunks. A chunk is the unit your system can retrieve, so if the answer to a question is split across two chunks, or buried inside one that is mostly about something else, no ranking improvement can fully rescue it.

Start from the document's shape

Fixed-size windows with overlap are a reasonable first pass because they are simple and uniform. Their weakness is that they ignore meaning. A window boundary can land mid-sentence, between a table header and its rows, or between a function signature and its docstring.

Most documents already tell you where the natural boundaries are:

  • Prose and docs: split on headings, then paragraphs. Keep the heading path (for example "Billing > Refunds > Partial refunds") attached to each chunk so it stays interpretable on its own.
  • Code: split on functions and classes rather than line counts, and include the file path and enclosing class name.
  • Tables and lists: keep a table whole when it fits, and repeat the header row when you have to split it.
  • Conversations and tickets: keep a question together with the reply that answers it.

Then enforce a size ceiling. Structure-aware splitting produces uneven pieces, and a three-page section still needs to be broken up. Fall back to paragraph or sentence boundaries inside oversized sections, never to an arbitrary character count.

Add context a chunk lost when it was cut

A chunk that says "It must be renewed every 90 days" is useless without knowing what "it" is. Prepending a short context line, such as the document title and section path, fixes most of these cases at the cost of a few tokens. Embed the chunk with that prefix, and show the same prefix to the model at answer time.

Tune with a measurement, not a feeling

Build a small set of real questions, ideally pulled from user queries or support tickets, each labelled with the passage that answers it. Then measure how often that passage appears in the top few results for each chunking variant. Compare:

  1. Your fixed-window baseline
  2. Structure-aware splitting with the same size ceiling
  3. Structure-aware splitting plus the context prefix

Change one thing at a time and keep the labelled set fixed so the numbers are comparable. Look at the failures by hand: if the right text is split across two chunks, you have a boundary problem; if it is present but outranked, you have a ranking problem, and chunking is not where to spend time.

The closing point

Chunking is the cheapest part of a retrieval stack to change and one of the most influential. Respect the structure your documents already have, restore the context that cutting removes, and let a labelled question set, not intuition, decide the final size.

AI IntegrationLarge Language ModelsSoftware Engineering