A judge model is only useful if it measures agreement with human judgment reliably, and until you've checked that, you don't actually know what its scores mean — you've just automated a number. Teams that skip calibration ship a judge that silently agrees with itself: it produces consistent, confident-looking scores that have no verified relationship to whether a human reviewing the same transcript would agree. That's worse than having no automated eval at all, because a bad number that looks authoritative gets trusted by people who never see the underlying transcripts, and it survives in dashboards and CI gates long after the humans who built it have moved on.
Lesson 06 covered designing the judge itself — writing a rubric, choosing a scoring scale, and hardening the prompt against the failure modes a judge is prone to (position bias, verbosity bias, self-preference). This lesson assumes that judge exists and asks a narrower, harder question: does it agree with a human rater closely enough, and in the right direction, to be trusted as a stand-in for one? Answering that requires a small set of examples a human has already scored, a couple of agreement statistics, and — critically — a baseline for how much two humans agree with each other before you can say the judge is falling short.
The rest of this lesson works through that pipeline end to end: build a gold set, measure the human-to-human agreement ceiling first, score the judge against that same ceiling using Cohen's kappa, then use a confusion matrix to find out which direction the judge is wrong in, because 'wrong 25% of the time' and 'always too lenient on partial credit' call for completely different fixes.
Analogy🏏Cricket
🏏 Think of it like cricket: When the BCCI and ICC introduced Ultra Edge and ball-tracking to the Decision Review System, nobody trusted the technology on day one just because the vendor said it worked. Before DRS was used to overturn an on-field umpire's call in a live Test at the MCG or Eden Gardens, the system was run for years against thousands of deliveries that experienced on-field umpires like Simon Taufel and Kumar Dharmasena had already adjudicated, comparing the machine's 'out' or 'not out' call to what the human said in real time. Only after that back-testing showed the technology agreed with expert umpires at a rate that itself matched how often two expert umpires agreed with each other — not against some flattering benchmark of accuracy in a vacuum — was it trusted to overturn a live decision in a World Cup final. Crucially, nobody expected 100% agreement, because even two Elite Panel umpires watching the same marginal caught-behind appeal from Bumrah against a nervous tail-ender at the SCG will disagree sometimes; that's the ceiling DRS was measured against, not some impossible standard of perfection. The insight: you calibrate a new measuring instrument against the agreement rate of the experts it's replacing, not against a fantasy of flawless judgment, and you never deploy it live until that calibration is done. An LLM judge scoring agent trajectories is the same story — it earns trust by matching a human rater's calls on a held-out set of labeled examples, measured against how much two human raters agree with each other, before it's allowed to grade anything nobody double-checks.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.