Most teams write postmortems when something breaks. Far fewer write one when something *almost* broke, and that is a missed opportunity, because a near miss is a failure you get to learn from for free.
With AI features this matters even more. Model behavior is probabilistic, so a bad output that a human happened to catch, or a guardrail that happened to trigger, is evidence about a failure rate you cannot see directly. If a reviewer caught one wrong answer this week, the useful question is not "good catch" but "how many did they not catch?"
A near-miss review can be short. Answer four questions:
- What nearly happened? Describe the bad outcome that was avoided, in concrete terms.
- What stopped it? Name the specific control, and be honest about whether it was designed to or simply got lucky. A reviewer who happened to be paying attention is not a control.
- How likely is a recurrence without the lucky break? Even a rough guess (rare, occasional, frequent) is useful for deciding priority.
- What is the smallest change that makes the catch reliable? Often it is a validation check, an added eval case, or a log line that would have made the problem visible sooner.
Then do the part teams skip: add the case to your eval set or regression suite so it stays caught.
Keep the tone blameless and the format lightweight, or nobody will bother. A paragraph per question is enough. The goal is a habit of treating "that was close" as data rather than relief.
If you only do one thing this week, ask your team for the last time something almost went wrong with your AI feature, and write down what really prevented it.