Calibrate Your LLM Judge Against Human Labels Before You Trust Its Scores

An LLM judge produces a number for every output, which makes it easy to forget the number is itself a measurement that needs validating. Before you let a judge gate a release, find out how closely it agrees with the people whose opinion it is standing in for.

Build a small human-labeled anchor set

Take 60 to 100 real outputs, chosen to span easy passes, clear failures, and the awkward middle. Have two people label them independently against the same rubric you give the judge. Resist labeling together: the point is to see where humans disagree, because a judge cannot be expected to agree with people more than they agree with each other.

Compute agreement between the two humans first. Raw percent agreement flatters you when one label dominates (if 90% of outputs pass, two coin-flippers who say "pass" nine times in ten will "agree" about 82% of the time). Use a chance-corrected statistic such as Cohen's kappa, or at minimum report agreement separately for each label. If humans only reach moderate agreement, the rubric is the problem, and no judge tuning will fix that.

Compare the judge to the humans, by error type

Run the judge over the same set and build a confusion matrix against the human consensus. Two numbers matter more than overall accuracy:

• False pass rate: the judge approves outputs humans rejected. This is what lets regressions through.
• False fail rate: the judge rejects outputs humans accepted. This is what burns engineering time on phantom problems.

Which one you tolerate depends on the use. A release gate should be tuned to a low false pass rate; a triage tool that surfaces candidates for human review can live with more false fails.

Then read the disagreements, all of them. Patterns show up quickly: the judge rewards length, penalizes a valid terse answer, ignores a missing citation, or applies a criterion the humans never used. Each pattern is either a rubric ambiguity to clarify or a known blind spot to document.

Keep the calibration alive

A judge calibrated once is calibrated for one model version, one rubric version, and one distribution of outputs. Re-run the anchor set whenever you change the judge model, edit the rubric, or notice the production output distribution shifting. Store the agreement numbers next to the judge's version, the same way you would store a test coverage figure.

It also helps to hold back part of the anchor set that you never use while editing the rubric. Otherwise you will tune the rubric until it matches those exact examples, and your agreement figure will describe memorization rather than judgment.

The closing rule

Report judge scores together with the judge's measured agreement, for example "pass rate 84%, judge-human kappa 0.71 on 80 labeled items". A score without that context invites readers to treat an opinion of the judge as a fact about the product. With it, a reader can decide how much a three-point change is worth.