When you change a prompt, the question you care about is relative: is version B better than version A on the same inputs? Asking a model judge to score each output on an absolute 1–10 scale answers a harder and noisier question than that. Pairwise comparison, where the judge sees both outputs and picks one, is usually the more sensitive instrument, but only if you control for the biases that come with it.

What the research says about LLM judges

The standard reference is the paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (Zheng et al., 2023). It reports two findings that matter here:

  • LLM judges can track human preferences well. The authors found that strong judges can match controlled and crowdsourced human preferences, with over 80% agreement, which they note is the same level of agreement as between humans. That is why model judging is a scalable way to approximate human preferences.
  • They have systematic biases. The paper identifies position, verbosity and self-enhancement biases, as well as limited reasoning ability, and proposes ways to reduce some of them.

Position bias is the one you can neutralize cheaply. If you always show the candidate prompt's output second, you may be measuring the judge's habits, not your change.

Why pairwise beats absolute scores

Absolute scores drift. A judge that gave a 7 yesterday may give an 8 today, scores cluster in a narrow band, and a small real improvement vanishes in the noise. When the judge sees A and B side by side, it has an anchor, and even weak preferences show up as a consistent win rate over many cases. (That's my practical experience of why teams prefer it. The paper's own comparison methods are worth reading if you want the evidence.)

The fix: run every pair twice, in both orders

For each test case, call the judge twice, once as (A, B) and once as (B, A), and classify the result:

  • Consistent win: the same underlying output wins in both orders. Count it.
  • Flip: the judge picks whichever came first (or second). That's a tie in disguise. Record a tie, not a coin toss.
def compare(case, out_a, out_b, judge):
    r1 = judge(case, first=out_a, second=out_b)   # returns "first" or "second"
    r2 = judge(case, first=out_b, second=out_a)
    if r1 == "first" and r2 == "second":
        return "A"
    if r1 == "second" and r2 == "first":
        return "B"
    return "tie"   # position-driven disagreement

The share of flips is itself a useful number. If 40% of your pairs flip, your judge prompt or rubric isn't separating these outputs, and the win rate on the remainder deserves less trust. Track the flip rate per run the way you track the pass rate.

Controlling the other biases

  • Verbosity. Check whether the winner is simply the longer answer. Compare lengths on the decisive pairs, or tell the judge explicitly not to reward length.
  • Self-enhancement. Anthropic's guidance for model-graded evaluations says to use a different model to grade than the one that generated the output. That's the practical defense against a judge favoring its own style.
  • Narrow criteria. Ask "which better satisfies criterion X" and name the criterion, instead of "which is better." Define it clearly in the grading prompt and ask for structured output.
  • Limited reasoning. For answers that need checking (math, code, facts), don't rely on a judge's opinion alone. Use code-based checks where you can.

Validate the judge itself

A judge can be consistent and consistently wrong. Read a sample of the decisive pairs yourself, and compare the judge's choices against your own on a small set. Ten minutes of reading catches rubric problems that no statistic will. This is the same logic as keeping a held-out set honest: the measurement needs its own checks.

The takeaway

A pairwise evaluation is only as honest as its order handling. Swap the order, count flips as ties, and report the flip rate next to the win rate. Then plug it into a minimal eval harness so every prompt change comes with a before-and-after comparison.

Sources

Large Language ModelsSoftware Engineering