Ask a judge to rate a single agent response on a 1-5 rubric and you are asking them to invent an internal reference point for what a "3" means, on the spot, and then hold that reference point steady across every item they score that day. Different judges anchor differently, the same judge drifts across a long session, and an LLM-as-judge drifts across prompt phrasings you didn't even know mattered. An eval report that says "average quality 3.4" is reporting a number that almost nobody else could reproduce by re-running the same rubric, because the rubric never pinned down what 3.4 is measured against.
Pairwise comparison sidesteps the calibration problem by asking a strictly easier question: given two responses to the same prompt, which one is better? This question needs no absolute scale at all — the judge only has to notice a difference between two things placed side by side, sharing the same prompt, the same task, and the same context. Human annotators and LLM judges are both measurably more consistent on this question than on absolute scoring, which is why every serious model leaderboard (Chatbot Arena being the best-known example) is built on pairwise votes rather than average star ratings.
The previous lesson calibrated a single absolute-score judge against a human-labeled gold set so its scores could be trusted at face value. This lesson does something different: it replaces the scoring task itself. You'll still end up with a ranking of agent variants, but you'll get there by counting head-to-head wins instead of averaging scores, and that switch brings its own machinery — position bias to control for, ties to define, and a real math model (Bradley-Terry, the same family Elo approximates) to turn a pile of A-vs-B verdicts into an ordered list.