LLM-as-Judge: Design and Failure Modes — Agent Evaluation & Observability

Deterministic checks and fuzzy similarity scores, covered in the last two lessons, work well when there's a checkable structure to compare against — an exact string, a JSON schema, an embedding distance from a reference answer. They break down the moment an output is genuinely open-ended: a multi-paragraph summary of a technical incident, a customer support reply where tone matters as much as correctness, an agent's final answer to a task that has several valid phrasings. For that class of output, the two options are a human reviewer or a second LLM call configured to grade the first one — an LLM-as-judge. This lesson is about designing that second call so it produces a verdict you can actually trust, and about the specific, well-documented ways a careless judge design goes wrong.

A judge is not free. Every graded output costs an extra model call, adds latency to a CI run, and — if designed sloppily — introduces its own source of noise that can be worse than not measuring at all. Because of that, a judge is worth reaching for only on the subset of outputs that a deterministic assertion or a similarity score genuinely cannot resolve; using an LLM judge to check whether a function returned the string "success" is throwing away a cheap, perfectly reliable check for an expensive, imperfect one.

The next two lessons build directly on this one: calibrating a judge against a set of human-labeled examples, and running pairwise comparisons between two candidate outputs instead of grading one in isolation. Both of those techniques assume the judge itself is well designed — a structured prompt, a forced verdict format, and an awareness of the biases catalogued later in this lesson. Get the design wrong here and calibration in the next lesson will just measure how wrong, not fix it.

Analogy🏏Cricket

🏏 Think of it like cricket: In a T20 match, the on-field umpire calls easy decisions herself — a batter clean bowled, a catch taken inches above the ground and obviously safe. Nobody sends those to the third umpire; the naked eye is a perfectly good detector. But when Rohit Sharma is given out lbw and the ball's line, pitching point, and impact are all within a millimeter of the stumps, no umpire's eye is precise enough — that's exactly when the team burns a Decision Review System review, calling in Hawk-Eye and UltraEdge to adjudicate what a human genuinely cannot resolve alone. A captain who reviews every single decision, plumb or not, runs out of reviews before the match is decided, and the umpires stop trusting the team's judgment about what's actually contestable. An LLM-as-judge is the DRS of an eval pipeline: it earns its cost only on the outputs where a deterministic check (exact match, a schema, a regex) or a cheap similarity score genuinely cannot tell pass from fail — an open-ended summary, a nuanced customer-support reply, a piece of reasoning with no single correct string. Calling the judge on outputs a plain assertion would have caught is like reviewing a ball that hit off stump by a foot: expensive, and it teaches you nothing a scoreboard didn't already know. The insight is that a judge is a scarce, costly instrument for genuinely ambiguous calls, not a blanket replacement for the checks that already work.