Record and Replay LLM Calls: Offline Tests That Don't Burn Tokens or Flake

Most LLM-backed codebases end up with two kinds of tests: fast unit tests that never touch the model, and a handful of slow, expensive, occasionally flaky tests that do. The middle ground is the part that gets neglected, and it is where most real bugs live: the code that builds a request, calls the model, parses the reply, and handles what comes back. Record-and-replay fixtures cover that middle ground cheaply.

The idea is borrowed from HTTP cassette libraries. The first time a test runs, the real call goes out and the full response is saved to disk. Every later run reads the saved response instead. Your parsing, retry, routing, and error-handling code all execute for real; only the network hop is faked.

What to record and how to key it

Record at the narrowest boundary you own, which is usually your own thin client wrapper rather than the raw HTTP layer. Recording at the wrapper keeps fixtures readable and insulates them from header and transport details that change for reasons unrelated to your logic.

The key matters more than the storage format. A good key is a hash of everything that should change the answer:

• the model identifier
• the rendered prompt, after templating
• sampling parameters that you set explicitly
• the tool definitions, if any

Notice what is deliberately excluded: timestamps, request IDs, and anything else that varies run to run. If those leak into the key, every run is a cache miss and the fixtures are useless. Normalise them before hashing.

Store the whole response, including stop reason, token usage, and any tool-call blocks, not only the text. Bugs about truncation and tool-call handling are invisible if you only kept the string.

Make staleness a feature, not a surprise

The danger of replay is that tests stay green while the real model has drifted. Three habits keep that honest:

1. Fail loudly on a missing recording. In CI, a cache miss should be an error, never a silent live call. Otherwise a prompt edit quietly turns your offline suite into an online one.
2. Make re-recording a deliberate command. A prompt change should produce a diff in the fixture files, and that diff should be reviewed like code. A changed recording is evidence about what changed in behaviour.
3. Keep a small live suite. Run a few real-model tests on a schedule rather than on every commit. They catch drift; the recorded suite catches regressions in your own logic.

Also record failures. Rate-limit errors, malformed JSON, and truncated outputs are the cases you most want to replay, and they are hard to reproduce on demand against a live API. Capture one when it happens in production logs or a manual run, check it in, and you have a permanent regression test.

The closing point

Replay fixtures do not test whether the model is good. They test whether your code behaves correctly given a model response, which is the part you can actually make deterministic. Separating those two questions lets you run the first on every commit for free and spend your eval budget on the second where it counts.