Issue #92

Your Prompt's Examples Are Part of Your Spec, So Review Them Like One

sent to 25 subscribers

A prompt's instructions get careful review. Its examples often don't, even though the model tends to weight them more heavily than the prose around them.

That makes examples the most influential part of a prompt, and the least audited.

Here is what to check the next time a prompt changes:

Do the examples agree with the instructions? If the rules say "keep answers under 50 words" and an example runs to 90, the model will usually follow the example. Contradictions between the two are one of the most common sources of puzzling behavior.

Do they cover the boundary, not just the middle? Examples tend to be clean, typical inputs. Add at least one that is messy, ambiguous, or should be refused, so the model sees what the edge looks like.

Do they share an accidental pattern? If every example output begins with the same phrase or has the same length, expect the model to reproduce that pattern even where it doesn't fit.

Are they still true? Examples encode the policy at the time they were written. When the product or rules change, stale examples quietly pull behavior back toward the old version.

Who owns them? An example with no owner is never removed. Give each one a reason for existing, and a note on what failure it was added to prevent.

A simple habit helps: when you add an example, write one line explaining what it is meant to teach. If you can't, it probably isn't earning its tokens.

Treating examples as specification, with review, ownership, and a reason for each one, costs very little and prevents a class of bugs that instructions alone can't fix.

← Back to archive