Issue #110

Comment Your Prompts: The Line Nobody Remembers Adding Is the Line Nobody Dares Remove

sent to 25 subscribers

Open any prompt that has been in production for six months and read it as a stranger would. Somewhere in the middle is a sentence like "Never mention the word 'unfortunately'" or "Always answer in exactly two paragraphs." Nobody on the team can say why it is there.

So it stays. Removing it might break something, and nobody can tell what. Meanwhile it quietly constrains every output, and the prompt grows by accretion rather than design.

Code has solved this problem with comments, commit messages, and blame. Prompts usually have none of them, which is odd, because a prompt is closer to a specification than most code is.

Three habits help, and each takes minutes.

Record the reason beside the rule. Keep prompts in a format that allows comments the model never sees, or maintain a companion note keyed by rule. One line is enough: "Added after the March incident where summaries included customer email addresses." Now a later reader can judge whether the cause still exists.

Link each rule to a test. If a rule was added to fix a failure, the failing input belongs in your eval set, tagged with the rule it motivated. When someone proposes deleting the rule, run that case. If it still passes without the rule, the rule has earned its removal. If it fails, you have just rediscovered why it existed, without a production incident.

Review deletions as carefully as additions. Most prompt review focuses on what is new. A prompt that only ever grows gets slower, costlier, and more contradictory. Schedule an occasional pass whose only job is to remove rules that have no recorded reason and no test.

None of this requires tooling. It requires treating the prompt as something other people will have to maintain, because soon enough, the other person will be you, having forgotten everything.

← Back to archive