Mocking LLM Calls Without Lying to Your Test Suite

Mocking LLM Calls Without Lying to Your Test Suite

Once an LLM call sits in a code path, most teams reach the same conclusion fast: you cannot afford to hit the real API on every test run. It costs money, it's slow, and it's nondeterministic enough to make CI flaky for reasons that have nothing to do with the code under test. So the LLM call gets mocked. The problem is *how* it usually gets mocked — and the false confidence that results.

The common pattern is a developer hand-writing a canned response that matches whatever shape the current code expects, dropping it into a fixture, and moving on. That test will pass forever. It will keep passing after the prompt changes, after the model is upgraded, after the response format quietly shifts in a way that breaks the parsing logic in production. A test suite full of fixtures like this isn't testing your system's behavior — it's testing that your code can parse a string you invented to make the test pass. It gives you a green checkmark and no actual information.

The recording-and-replay pattern

A more honest approach is to stop hand-writing fixtures and start capturing them. Wrap the LLM call in a thin layer that can run in two modes: live, hitting the real API and recording the response to disk, and replay, reading a previously recorded response instead of calling out. Fixtures are then real model output, captured at a point in time, not a developer's guess at what the model would say. When the prompt changes meaningfully, you re-record. This is the same idea as HTTP cassette libraries that have existed for years in web development, applied to a case where "the response format might just change under you" is a much bigger risk than with a stable REST API.

Two layers of tests, not one

The fix isn't just better fixtures — it's separating two kinds of tests that get conflated:

• Logic tests, which use frozen fixtures and check your code's behavior: does it parse the response correctly, handle a malformed field, retry on a particular error shape, log the right thing. These should run on every commit, fast and deterministic.
• Contract tests, which hit the real API on a slower cadence — nightly, or before a release — and check that the *shape and rough behavior* of the response still matches what your logic tests assume. Their job is to catch drift, not to validate business logic.

Most teams have plenty of the first kind and none of the second, which means the moment a model update or prompt change breaks something, it's discovered in production rather than in CI.

A useful gut check: if your mocked LLM test suite has never once failed because the model's actual behavior changed, it probably isn't testing anything real — it's testing a fixture against itself. Budget for periodic fixture refreshes and a small number of real-API contract tests. It's cheap insurance against the exact kind of silent drift that mocking was supposed to protect you from being surprised by.