Issue #133

Ask Before You Answer: Clarifying Questions Are a Design Decision

sent to 25 subscribers

Most AI features are built to answer immediately. Whatever the user types, the model produces something, and a fluent response looks like success even when the request was ambiguous and the answer rests on a guess.

Whether the system should instead ask a question is a product decision, and it deserves to be made deliberately rather than left to whatever the model happens to do.

When asking is worth it. A clarifying question is worth its cost when the possible interpretations lead to meaningfully different results, and a wrong guess is expensive: an action that is hard to undo, a long piece of generated work the user will have to discard, or an answer the user might act on without checking. It is rarely worth it for low-stakes requests where a reasonable default exists and the user can fix a miss in one edit.

When asking backfires. Every question adds a round trip and makes the tool feel less capable. A model that asks about things it could have inferred from context, or asks three questions at once, trains users to stop engaging. If a sensible assumption exists, state the assumption in the answer ("I assumed the last 30 days; tell me if you want a different range") and let the user correct it. That often delivers the benefit of asking without the delay.

Make it testable. Decide in advance which kinds of missing information should trigger a question, and write eval cases for both sides: ambiguous inputs where the system should ask, and clear inputs where asking would be a failure. Without the second group, you will tune your way into a system that interrogates everyone.

Keep the question cheap to answer. Offer two or three concrete options rather than an open-ended "can you clarify?". The user should be able to reply in a few words.

A useful habit this week: take ten real requests from your logs, mark which were genuinely ambiguous, and check what your system did with each. The gap between "should have asked" and "guessed confidently" is usually where your most frustrating bug reports are hiding.

← Back to archive