Any long-running conversation eventually hits the context limit, and the usual remedy is to summarize the old turns. The usual failure is a summary that reads smoothly and forgets the things that mattered: the constraint the user stated in turn three, the option they explicitly rejected, the task the assistant said it would come back to.

Summarize for the next step, not for a reader

A generic "summarize this conversation" prompt produces a narrative. A model resuming work doesn't need a narrative; it needs state. Ask for a structured record with fixed sections:

  • Goal: what the user is ultimately trying to do.
  • Decisions made: choices that are settled, with the reason when it was given.
  • Rejected options: things not to suggest again.
  • Constraints and preferences: stated requirements, in the user's words where possible.
  • Open items: commitments the assistant made and has not yet fulfilled.
  • Key facts: names, identifiers, numbers, file paths that later turns will need verbatim.

Fixed sections make omissions visible. An empty "Open items" section is a claim you can check; a paragraph that happens to skip them is not.

Keep exact things exact

Summaries paraphrase, and paraphrase corrupts identifiers, quantities, and quoted instructions. Copy those verbatim rather than describing them. Better still, keep the last several turns uncompacted, so the most recent and most detailed context is never second-hand, and compact only what lies behind that window.

Compact incrementally, and guard against drift

Each compaction should take the previous summary plus the newly aged-out turns and produce an updated summary. Do this repeatedly and errors compound: a fact that was slightly wrong in round one is treated as truth in round five. Two mitigations help. Keep the raw transcript in storage so a summary can be regenerated or audited, and periodically check the summary against the transcript with a few targeted questions ("what budget did the user give?").

Test it with a recall probe

Build a small test: take a real long conversation, plant five facts across it (two constraints, one rejection, one promise, one identifier), run compaction, then ask questions that depend on each. Score how many survive. Run it again whenever you change the compaction prompt or the model. A summary that has never been probed for recall is a guess about what your assistant remembers.

The closing point: compaction is lossy by design, so decide deliberately what must never be lost, give those things their own slots, and measure whether they survive.

AI ToolsArtificial Intelligence