Scientists have a defense against fooling themselves: pre-registration. Before running an experiment, you write down what result would count as success — and you commit to that definition before you see the data. It exists because humans are extremely good at finding, after the fact, a metric that makes any result look like a win.
AI features have the same vulnerability, and almost nobody uses the same defense. A team ships a feature, watches the numbers for a week, and then picks whichever metric moved in the right direction to justify the launch — engagement if it's up, retention if engagement is flat, "qualitative feedback" if neither moved. Each choice feels reasonable in isolation. Collectively, it means the feature is never actually evaluated against a bar it could have failed.
The fix costs almost nothing: before shipping, write one sentence — "this feature succeeds if [specific metric] moves by [specific amount] within [specific window]" — and put it somewhere the team can't quietly edit later. It doesn't need statistical rigor or a formal experiment design. It needs to exist before the data does, so that a disappointing result stays disappointing instead of getting reframed.
This is especially worth doing for AI features, because they fail in ways that are easy to rationalize — "the model will get better," "users need time to learn the pattern," "the real win is qualitative." Some of those are true sometimes. A metric written down in advance is what stops "sometimes" from becoming "always, for every feature we ship."
If your last three AI feature launches don't have a pre-written success bar you can point to, that's worth fixing before the fourth one ships the same way.