When a team hardens an AI feature, the instinct is to add guardrails: stricter system prompts, content filters, more cautious wording. Each one reduces a risk you can name. Each one also raises the odds that the feature turns away a legitimate user, and that failure is much harder to see.

A refusal that should not have happened does not generate an error, an alert, or usually a bug report. The user rephrases, gives up, or quietly decides your product is not useful. If you only measure the refusals you wanted, you will tune yourself into a worse product and never know.

Build the other half of the safety eval

A typical safety set contains requests the system must decline. An over-refusal set is its mirror: requests that touch a sensitive area but are legitimate and should be answered. Examples for a developer tool include asking how an existing vulnerability class works in order to defend against it, asking for a script that deletes files in a user's own directory, or asking about medication interactions in a context where that is appropriate to the product.

Source these from real traffic where you can. Look at conversations where users repeated or reworded a request, or abandoned right after a refusal-shaped response. Those are your best candidates. Label each case with the behavior you want, not just "answer" or "refuse", because the right response is sometimes a partial answer with a caveat.

Track both rates together

Report two numbers side by side: the rate at which the system correctly declines the must-refuse set, and the rate at which it correctly answers the should-answer set. Looking at either alone is misleading, since a system that refuses everything scores perfectly on the first. Any change to a prompt, filter, or model should show movement on both before it ships.

A few practical details:

  • Detect refusals deliberately. Phrases like "I can't help with that" are easy to match, but soft refusals ("I'd recommend consulting a professional" with no substance) are common. Use a rubric or a judge model with a clear definition of "did not meaningfully attempt the task", and spot-check it by hand.
  • Look at the boundary, not the center. Easy cases at both extremes rarely change. The informative cases are near the line, so spend your labeling effort there.
  • Re-run after upstream changes. A model update can shift refusal behavior in either direction without any change on your side.

The takeaway

Safety work is a tradeoff, and a tradeoff needs both sides measured. Add a should-answer set next to your must-refuse set, report both rates, and treat a drop in helpfulness as a regression just like a drop in safety.

Software Engineering