Asking a model to sort inputs into labels like billing, bug, and feature_request feels trivially easy, and on the first twenty examples it usually is. The trouble shows up on the messy middle: "I was charged twice after the app crashed." Which label is that? If your prompt does not say, the model decides, and it may decide differently on Tuesday than on Monday.

A label needs three things

Treat each label as a small specification rather than a word.

  1. A one-sentence definition in terms of what the user is trying to get done, not keywords. "The user is asking about a charge, refund, invoice, or plan change."
  2. An exclusion, naming the nearest confusable label. "If the charge is mentioned only as background to a malfunction, use bug."
  3. A tiebreak rule for when two labels both fit. "When a message contains both a defect and a billing impact, choose billing because that queue has the shorter response target."

The tiebreak is the part teams skip, and it is the part that creates consistency. It also forces a useful conversation: someone has to decide the policy, and that decision belongs to the product owner, not the model.

Add an escape hatch you actually use

Include an explicit other or unclear label with a definition of its own, and decide what the downstream system does with it, such as routing to a human. Without an exit, the model will pick the closest label with unearned confidence, and your accuracy metric will hide how often the inputs genuinely did not fit.

Test the boundaries, not the center

Build your evaluation set deliberately from boundary cases. For each pair of labels that could be confused, collect a handful of real examples that sit between them and write down the correct label *and the reason*. Then:

  • Run the classifier several times on the same boundary set. If answers flip between runs, the definition is underspecified, and no amount of rewording the instructions will fix it until the tiebreak is written down.
  • Look at a confusion matrix, not a single accuracy number. Two labels that swap frequently usually need a sharper exclusion, or should be merged.
  • When you add a label, rerun the whole set. A new label tends to steal examples from its neighbors.

Output format matters less than you think, but do constrain it

Ask for the label from a closed list, validate it in code, and treat anything outside the list as a failure to retry or escalate. Free-text labels invite near-duplicates like Billing, billing issue, and payments.

The closing point

A classifier is only as consistent as the taxonomy behind it. If two careful humans would disagree on a case, the model cannot be expected to agree with either of them. Resolve those disagreements in writing, put the resolution in the prompt, and keep the example in your eval set so it stays resolved.

AI IntegrationLarge Language Models