Designing Agent Memory: What to Persist, What to Forget, and How to Retrieve It Later

Designing Agent Memory: What to Persist, What to Forget, and How to Retrieve It Later

The usual path to "agent memory" looks like this: someone drops every message into a vector store, wires up a similarity search before each turn, and calls it done. It works in the demo. Three months later the store is bloated with contradictory facts, retrieval pulls in irrelevant noise half the time, and nobody can explain why the agent just referenced a preference the user stated once, six weeks ago, that no longer applies. The failure isn't the retrieval algorithm. It's that three separate design decisions — what to write, when to forget it, and how to find it again — got collapsed into "store everything, search by similarity" without anyone actually deciding any of them.

What actually belongs in memory

Not everything that happens in a session is memory-worthy, and treating a raw transcript as the memory store is close to the worst option available. A transcript mixes durable facts ("the user's default output format is CSV"), transient state ("we're in the middle of a three-step task"), and noise (small talk, retries, corrected mistakes) with no way to tell them apart later. Durable facts and stated preferences are worth persisting because they should influence behavior far outside the session that produced them. Transient task state belongs somewhere else — scoped to the task, expired when the task ends — because keeping it around past its useful life is exactly how a memory store fills up with things that no longer apply but still get retrieved with full confidence. If you can't state, in one sentence, why a piece of information needs to outlive the conversation that created it, it probably shouldn't go in long-term memory.

Forgetting has to be designed, not left to running out of space

Most systems only think about forgetting once storage costs become a problem, which means the actual forgetting policy ends up being "oldest entries get evicted first" — a policy driven by storage economics, not by whether the information is still true. A user's stated preference from a month ago that has since been explicitly reversed needs to be gone, not just outranked. Two versions of the same fact should never both be retrievable at once; a new value needs to supersede the old one, not sit alongside it waiting for the model to guess which is current. Building this requires memory writes to check for conflicts against what's already stored, not just append.

Retrieval needs a query, not a vibe

Pure semantic similarity search is a reasonable fallback, but it's a poor primary retrieval mechanism for anything you actually know the shape of. If a memory is keyed to a specific entity — a user ID, a project name, a known preference category — look it up directly. Save similarity search for the genuinely fuzzy case: "has anything relevant to this come up before." And every retrieved memory should carry provenance — when it was learned, from what interaction, how confident the system is that it's still current — because the moment an agent misbehaves due to a bad memory, that metadata is the only thing that lets a human figure out why, instead of re-reading the whole history to guess.

The concrete test for whether a memory system is well-designed: can you explain, for any single fact the agent used, why it was written, why it's still considered valid, and why it was the one retrieved instead of five other candidates? If the honest answer is "the vector search returned it," the system needs more design, not more storage.