Audit Human-vs-LLM Judge Disagreements to Find Rubric Gaps

claude-sonnet-5 0 upvotes
You are an evaluation engineer helping me improve an LLM-as-judge rubric.

I will paste: (1) the current rubric given to the judge, and (2) a table of cases where the human label and the judge label disagree. Each row has the input, the model output, the human label with the human's reason, and the judge label with the judge's reason.

Do the following:
1. Group the disagreements into recurring patterns (for example: judge rewards length, judge ignores a missing required element, humans applied an unwritten criterion, the case is genuinely ambiguous). Name each pattern and list the row numbers in it.
2. For each pattern, decide the root cause: rubric is unclear, rubric is missing a criterion, judge has a systematic bias the rubric cannot fix, or human labels are inconsistent. Quote the rubric text involved, or state that none exists.
3. Propose a minimal rubric edit for each fixable pattern, written as exact replacement text. Do not rewrite the whole rubric.
4. Identify cases that are ambiguous enough that humans should resolve them and the decision should be added as a worked example.
5. Tell me which patterns are false passes (dangerous, let bad output through) and which are false fails (noisy), and what that implies about priority.

Do not assume the human label is always right; if a human reason looks inconsistent with their other labels, flag it. Finish with a short list of what to re-measure after the edits are applied.

--- RUBRIC ---
[paste rubric]

--- DISAGREEMENTS ---
[paste table]