An agent that books flights, drafts SQL queries, or answers customer questions produces an output on every single run, and someone — or something — has to decide whether that output was correct. The most powerful tool for that decision is an LLM-as-judge: another model reading the output and rendering a verdict. It's also the most expensive tool available, in three ways that compound at scale. It costs real money per call, on the order of cents to tens of cents depending on the judge model and context length. It adds seconds of latency to every evaluation run, which matters when a regression suite has to gate a deploy. And it is itself a non-deterministic system judging a non-deterministic system — ask the same judge to score the same output twice and you will not always get the same score back, a genuinely strange property to depend on when the entire point of the check is to notice when something changed.
Most of what an agent gets wrong doesn't need a judge to catch. A response that returns malformed JSON, a tool call with a misspelled argument name, a fare quoted with the wrong currency symbol, a raw SQL string leaking into a customer-facing answer — all of these are structural or categorical failures that a few lines of assertion code catch with total certainty, in microseconds, for free. The discipline this lesson is about is ordering: run every deterministic check that applies before a single output is sent to a judge, because a deterministic check that fails has already told you the output is wrong, with no ambiguity and no retry needed to confirm it. Only the outputs that pass every deterministic gate — and might still be wrong in some way only a semantic reader would notice — earn the cost of the next tier.
This lesson stays entirely in that first tier: checks that are exact, instant, and produce the identical verdict every time you run them on the identical input. The next lesson moves to fuzzy checks — similarity scores and rubrics that tolerate legitimate variation. The one after that is LLM-as-judge. The order of these three lessons is the order you should reach for them in a real pipeline, and skipping straight to the third because it feels the most 'AI-native' is the single most common way teams overspend on evaluation.