When a model gives you a wrong answer once, the instinct is to rewrite the prompt. But for a class of tasks, the cheaper and more reliable fix is not to change the prompt at all: ask the same question several times and take the answer most samples agree on. This is often called self-consistency or majority voting, and it is easy to misuse, so it's worth being precise about when it earns its cost.

Why voting works

If a model's errors on a given input are somewhat random, with the correct answer being the most likely single outcome but not a certain one, then independent samples will scatter their mistakes across many wrong answers while the right answer keeps recurring. Picking the most frequent answer filters out the scatter.

That logic gives you the conditions under which it helps:

  • The answer is short and comparable. A label, a number, a yes/no, a chosen option, or a normalized extracted field. You need to be able to say "these two outputs are the same answer" cheaply.
  • Errors are diverse, not systematic. If the model is wrong the *same* way every time, because the prompt is ambiguous or the model lacks the knowledge, voting just confirms the mistake with more confidence.
  • Sampling is actually varied. At temperature zero you will get near-identical outputs and nothing to vote over. You need enough temperature for the samples to differ, and varying the prompt wording or example order can add diversity further.

What to do with disagreement

The most useful by-product of voting is not the winning answer, it's the *vote split*. A 5-0 result and a 3-2 result are very different situations, and treating them identically throws away information.

Use the agreement level as a routing signal. Unanimous results go straight through. Narrow majorities can be flagged for a human, sent to a stronger model, or answered with an explicit "unsure." This turns a flat accuracy number into something you can act on: you spend extra effort only on the inputs where the model itself was unstable.

Measure whether agreement is actually informative on your data. Compare accuracy on unanimous cases versus split cases using a labeled set. If both are similar, the vote isn't telling you anything and you should not build routing around it.

Costs to count honestly

Voting multiplies cost and latency by the number of samples. Run the samples in parallel to protect latency, and remember that rate limits and retries now apply N times over. Start with three samples; gains usually flatten quickly after that, and an even number invites ties you then have to break arbitrarily.

Also avoid voting on free-form text. Majority-voting over paragraphs requires fuzzy matching or another model call to cluster them, which adds a new failure point exactly where you wanted less risk. If you must, have the model produce a short structured answer alongside the prose and vote on that.

When not to bother

If the task fails because of a bad prompt, a missing piece of context, or a retrieval miss, fix that instead. Voting is a variance-reduction tool. It cannot repair bias. Before reaching for it, run your failing cases several times and look: if the wrong answers differ from run to run, voting may help; if the same wrong answer appears every time, no number of samples will.

Large Language ModelsSoftware Engineering