Log Enough to Replay a Bad LLM Response, Not Just to Know It Happened

When a user reports a bad answer from your AI feature, the first question is always the same: what exactly did the model see? Many teams discover, at that moment, that their logs contain a request ID, a latency, and a token count, but not the thing they need. Without it, you are guessing.

The goal of LLM logging is not observability in the abstract. It is replayability: given a log record, can an engineer re-run the same call and get a comparable result?

The minimum replay record

A record is replayable if it captures everything that shaped the output:

• The fully rendered prompt, after templating, retrieval, and truncation. Logging the template name and variables is not enough, because retrieval results and trimmed history change between runs.
• Model identifier as returned by the provider, not just the alias you requested. Aliases move; the resolved version is what actually ran.
• Sampling parameters: temperature, top-p, max tokens, stop sequences, and any seed.
• Tool definitions and tool results for agentic calls, in the order they occurred.
• The raw response, before your parsing and validation, plus the parsed result and which validation step (if any) rejected it.
• Code and prompt version: a git SHA or prompt version tag, so you know which pipeline produced it.

With those fields, you can paste a record into a script, re-run it, and see whether the failure reproduces. If it does not reproduce, that itself is information: it points at nondeterminism rather than a deterministic bug in your pipeline.

Keeping it affordable and safe

Full prompts are large and often contain user data, so two disciplines matter.

Sample by outcome, not uniformly. Store full replay records for every failed validation, every user-flagged response, and every low-confidence result, then a small random slice of successes for baseline comparison. Store only compact metadata for the rest.

Treat the replay store as sensitive. It holds the most revealing copy of your users' inputs. Redact at write time where you can, apply a short retention window, restrict access, and make deletion requests reach it. A replay store that outlives your data policy is a liability, not a debugging aid.

A test of your setup

Pick a random production failure from last week and try to reproduce it using only what you logged. If you can do it in minutes, your logging is doing its job. If you cannot, add the missing field today, because the next incident will need it, and you will not be able to go back and capture what was never recorded.