Somewhere in your codebase there is probably a comment like this: # the model can't reliably return valid JSON, so we parse defensively, or a line in a design doc that says long documents don't work, so we chunk to 500 tokens.
Those statements were true when someone wrote them. The problem is that nothing about them says *when*, or *on which model*, or *based on what test*. Months later they read as permanent facts, and they quietly shape architecture decisions long after the underlying limitation may have changed.
Here is a small habit that fixes this: attach a date, a model, and a pointer to evidence to every claim about model limitations.
For example:
> As of March, with model X, about 4% of outputs failed JSON parsing in our eval (see eval/json-001). Keep the repair step until re-measured.
That one sentence does three useful things:
- It tells a future reader how much weight to give the claim.
- It tells them how to re-check it, since the eval case exists.
- It turns "we've always done it this way" into a decision with an expiry.
Then add the second half of the habit: when you change models, grep for these notes and re-run the linked checks. Some workarounds will turn out to be dead weight you can delete. Others will still be earning their keep, and now you'll have fresh evidence for it.
The same applies in the opposite direction. "The new model handles this fine" deserves the same treatment, because a single good demo is not a measurement.
None of this takes more than a minute per note. What it buys is a codebase where limitations are hypotheses with owners, not folklore. Try it on the next workaround you write, and on the next one you find.