A single-shot prompt is easy to reason about: one input, one output, no history. The moment a feature becomes conversational, with a chat box, an agent session, or anywhere the user can follow up, that simplicity disappears, and in ways a three-exchange demo never reveals.

The API doesn't remember anything

The first thing to know is how the underlying APIs handle a conversation. Anthropic's Messages API documentation states it plainly: the API is stateless, which means you always send the full conversational history. Each request carries the whole list of earlier turns, and earlier turns don't even have to have come from the model. The docs note you can use synthetic assistant messages.

That gives you two things at once: complete control over what the model "remembers," and complete responsibility for it. Whatever your code puts in that list is the conversation, as far as the model is concerned.

The context you accumulate isn't free

Because the full history is resent each time, every turn you keep costs input tokens and latency on every later call. That cost compounds with conversation length, which is one reason prompt caching matters so much for chat features: the stable beginning of a conversation is exactly the kind of prefix that can be cached.

Some providers also let you add instructions partway through a conversation without disturbing that prefix. Anthropic documents a mid-conversation system message on certain models, which has the same authority as the top-level system field and, because it's appended at the end, doesn't invalidate a cached prefix before it. Check your provider and model for support.

The bigger cost, though, is semantic, not financial. An instruction in turn two doesn't expire. It keeps shaping turn nine unless something removes it. A user who said "keep responses short" and later asks a question needing a long answer is arguing with an instruction they forgot they gave. The model isn't wrong to follow it. Your system never told it the instruction was situational.

Decide what memory means (my suggestions)

Appending every message is the default because it requires no design decisions, which is exactly why it deserves some.

  • Raw history or extracted state? Carrying the full transcript preserves nuance and also every mistake and abandoned tangent. Extracting structured state (the user's goal, active constraints, decisions made) and dropping the rest takes more work and is much more predictable over a long session.
  • How should contradictions resolve? If a user changes their mind, should the newest instruction silently win, be confirmed with the user, or be blended with the old one? Pick on purpose. "The model does something plausible" is the absence of a policy.
  • Where is the turn boundary? Not every follow-up continues the same task. A feature that treats "one more question" as automatically in scope will apply constraints from an unrelated earlier request.
  • What gets dropped when you hit the limit? Decide whether to cut the oldest turns, summarize them, or fail visibly. Counting tokens at the boundary lets you decide deliberately. See Tokenization Surprises.

Hidden state inside a turn

Conversations that use extended reasoning or tool calls carry more than visible text. Anthropic's token-counting documentation notes, for instance, that thinking blocks from earlier assistant turns count toward input tokens on models that keep all prior turns, but not on models that keep only the last turn. If your history contains tool results or reasoning blocks, count and manage them too, and don't assume the model forgets them.

Test conversations, not just prompts

Single-turn tests won't find these bugs. Add multi-turn cases to your evaluation set: a constraint set early and then contradicted, a topic switch, a conversation long enough to trigger your truncation rule. Check what the model does at turn 15, not only at turn 3.

The takeaway

"What carries forward from turn to turn, and what doesn't" is a design surface with its own failure modes, like error handling or authorization. Treat it explicitly instead of as an accident of how the message list gets built.

Sources

AI IntegrationLarge Language Models