Prompts in production are rarely static strings. They are templates: a system prompt with slots for the user's name, retrieved documents, tool descriptions, and a few conditional blocks. The template is code, but the thing the model actually sees is the *rendered* output, and that is what changes behavior. Golden-file tests give you a cheap way to pin down the rendered output and notice when it moves.

What a golden-file test does here

The idea is simple. For a handful of representative inputs, render the full prompt and save the result to a file checked into the repo. The test renders again and compares against the saved file. If they differ, the test fails and shows a diff.

This catches a class of bugs that unit tests on the template logic miss:

  • A refactor that changes whitespace or ordering, so a carefully tuned instruction now sits after a long retrieved context instead of before it.
  • A conditional block that silently drops out because a variable was renamed and now evaluates falsy.
  • A helper that formats tool descriptions and starts emitting None or [object Object] for an optional field.
  • A "harmless" copy edit to a shared fragment that is included in twelve different prompts.

None of these throw an exception. All of them change what the model sees. Without a golden file, the first signal is usually a vague drop in output quality days later.

Making the fixtures useful

A few practices keep this from turning into noise:

  1. Choose inputs that exercise branches. One fixture with every optional section present, one with all of them absent, and one with a long retrieved context. Three to six fixtures is usually plenty.
  2. Make rendering deterministic. Freeze timestamps, sort anything that comes from a set or a dictionary, and use fixed IDs. A golden file that changes on every run teaches people to blindly regenerate it.
  3. Store the rendered text, not a hash. The whole point is a readable diff in code review. A reviewer should see the exact sentence that moved.
  4. Provide an explicit update command. Something like make update-goldens that rewrites the files. Regenerating should be a deliberate act, and the resulting diff should show up in the pull request.
  5. Keep secrets and real user data out. Use synthetic inputs, since these files live in version control.

What it does not do

A golden file tells you the prompt *changed*, not whether the change is good. That is deliberate. It is a tripwire that turns an invisible change into a visible one, so a human decides whether the change was intended and whether an eval run is warranted. Treat a diff in a golden file the way you treat a diff in a database migration: not necessarily wrong, but never something to merge unread.

The closing point is about cost. This technique takes an hour to set up, runs in milliseconds, needs no model call, and works with any provider. If your prompts are assembled from code, the most common way they will break is not the model changing but your own template changing underneath you. Pin the rendered output, review the diffs, and you remove a whole category of mysterious regressions.

Developer ToolsSoftware Engineering