When an AI feature produces a bad output and you fix it, the work usually ends with the fix: a prompt tweak, a retrieval change, a validation rule. The incident gets closed, the output looks right, and everyone moves on.
The missing step is small. Before you close the ticket, add the failing input to your eval set as a permanent case, along with a clear statement of what a correct result looks like.
Why this pays off
Prompt and pipeline changes have a habit of undoing each other. A tweak that fixes one customer's problem can quietly reintroduce an older one, and without a record of the older failure nobody notices until a user reports it again. Each preserved failure is a regression test that costs almost nothing after the first time you write it.
It also changes what your eval set is made of. A set built from imagined examples tends to measure the problems you expected. A set built from real failures measures the problems you actually have, and it grows in the areas where your system is weakest.
Making it routine
- Make it part of the definition of done. A bug fix PR for an AI feature includes the new case, or an explanation of why not.
- Record the reason. A one-line note on why the case exists helps the next person decide whether it is still relevant.
- Strip sensitive data. Replace names and identifiers before an input enters a shared set.
- Keep expectations specific. "Does not invent a refund policy" is checkable. "Gives a good answer" is not.
- Prune occasionally. Cases tied to behavior you deliberately changed should be updated or removed, not left to fail noisily.
One caution
A set made only of past failures will lean pessimistic. Keep a baseline of ordinary, successful examples beside it so you can tell whether a change helps the common case while it patches the rare one.
Next time you close a bug, ask whether the same input could break the feature again in three months. If the answer is yes, it belongs in the eval set.