Issue #106

Put a Stopping Rule on Prompt Tweaking Before You Start

sent to 25 subscribers

Prompt iteration has a trap that code work mostly lacks: every tweak feels like progress, and there is always one more phrase to try. Without an end condition, an afternoon of editing a prompt can leave you with a version that is slightly better on the three examples you keep looking at and worse on everything else.

A stopping rule fixes this, and it takes two minutes to write down before you begin.

Decide the target first. Write the number that would make you stop: for example, "at least 90% of the 60 cases pass, with no case that previously passed now failing". If you cannot state a target, you are exploring, not tuning, and that is fine, but call it exploring and timebox it.

Cap the attempts. Pick a budget such as eight revisions or ninety minutes. When it runs out, stop and look at what you learned rather than trying a ninth variation. Often the lesson is that the prompt is not the problem: the task is underspecified, the retrieved context is wrong, or the model is the wrong size for the job.

Change one thing at a time and keep the diff. If two edits go in together and the score moves, you do not know which one mattered. Commit each revision with the score beside it, so you can return to the best version rather than the latest.

Hold out some cases you do not look at. If you only ever read the failures you are tuning against, you are fitting the prompt to them. A handful of untouched cases checked at the end tells you whether the gain is real.

A stopping rule is not about moving fast. It is about making the decision to stop once, calmly, instead of repeatedly, tired, with a prompt that has quietly grown three paragraphs of special cases.

← Back to archive