Your LLM-as-Judge Eval Is Only as Good as Its Rubric

Ask a team why they trust their LLM-as-judge eval and most will point to the fact that it exists at all. Having an automated judge feels like progress after months of eyeballing outputs by hand. But a judge built on a vague prompt — "rate this response from 1 to 10 for quality" — is often not much better than a coin flip with extra steps. The model grading your outputs is subject to the same failure modes as the model producing them: it's sensitive to phrasing, anchors on surface features like length and confident tone, and can drift in what it considers a "7" from one run to the next.

The fix isn't a smarter judge model. It's a rubric that constrains what the judge is allowed to consider.

Build the rubric like a test suite, not a vibe

A single holistic score collapses too many independent questions into one number, which is exactly what makes it unstable. Break the judgment into discrete, mostly binary criteria that map to things you'd actually list in a code review: did the response answer the question that was asked, did it include information not present in the source material, did it follow the required format, did it avoid the specific failure modes you already know this feature is prone to. Each criterion should be answerable from the text without inference about intent.

Anchor every criterion with one or two concrete examples of a pass and a fail, pulled from your own data rather than invented. Judge models are much more consistent when they're pattern-matching against examples than when they're applying an abstract description of "good." If you can't produce an example that would fail a given criterion, that criterion probably isn't doing useful work and should be cut.

Order matters too: put the criteria that are cheapest to check and least ambiguous first, so a run that fails early can short-circuit instead of paying for a full pass over criteria that no longer matter once the response has already missed the basics.

Calibrate against humans before you trust it

A rubric that looks reasonable on paper can still diverge badly from what a human reviewer would actually flag. Before wiring the judge into a gate that blocks releases, run it against a sample you've labeled by hand — a few dozen examples is enough to start — and measure agreement, not just correlation. Pay particular attention to disagreements where the judge passed something a human would have rejected; those are the failures that cost you later; a judge failing something a human would allow is merely annoying.

Re-run this calibration whenever you change the judge model, the judge prompt, or the underlying task, since all three can silently shift what the judge rewards. A rubric calibrated against last quarter's model version on last quarter's failure modes is quietly stale.

Use it for comparisons, not verdicts

Absolute scores from an LLM judge are noisy in ways relative comparisons are not. Rather than asking "is this response good," ask "is this response better, worse, or about the same as the previous version, on each criterion." Regression detection — did the change I just made make anything worse — is a much easier problem than quality measurement in the abstract, and it's usually the question you actually need answered before you ship.