Evaluation Anti-Patterns and Benchmark Traps — Agent Evaluation & Observability

Every anti-pattern in this lesson has already cost some team a bad release, a wasted quarter, or a public benchmark score that quietly stopped meaning anything. None of them are exotic. They show up because evaluation is done under the same pressure as everything else in shipping software — a deadline, a metric someone above you cares about, a harness that was expensive to build so nobody wants to touch it — and each one has a specific, recognizable tell and a specific fix, not a vague exhortation to "be more rigorous."

Nearly every failure mode below is a variant of one idea, known as Goodhart's Law: when a measure becomes a target, it stops being a good measure. An eval score is supposed to be a proxy for a capability you actually care about — does the agent resolve the customer's issue, does it call the right tool, does it stay on task for twenty turns. The moment a team starts managing directly to the proxy instead of the underlying capability, the proxy keeps climbing while the capability it was invented to estimate quietly stops moving, or moves in the opposite direction. Everything from eval-set overfitting to the vanity dashboard is Goodhart's Law wearing a different costume.

This lesson revisits several ideas from earlier in the course under an adversarial lens: the judge design from Lesson 7, the statistical-significance arithmetic from Lesson 13, the variance-and-repeated-runs discipline from Lesson 14, and the adversarial and golden-dataset practices from Lessons 16 and 17. There, they were tools. Here, the question is what happens when a team has the tools available and still gets the evaluation wrong — because they skipped a step, trusted a convenient number, or let organizational incentives override what the tools were telling them.

Analogy🏏Cricket

🏏 Think of it like cricket: a state association tells uncapped batters that scoring 800 Ranji Trophy runs in a season puts them in the India selection conversation. The number seems like a fair proxy for batting quality, so batters start optimizing for exactly that number: choosing to bat first on flat Holkar Stadium tracks, angling for fixtures against sides missing their strike bowlers, and retiring hurt rather than risk a low score against a fired-up new-ball attack. By November, three batters cross 800 runs and their averages read like those of all-time greats. Then the selectors put them in the nets against a bowler steaming in off his full run, and two of the three cannot lay bat on ball — the runs were real, but they were scored against a version of cricket that no longer resembled what selection was meant to measure. The average did not fail to predict international-standard batting; it succeeded perfectly at predicting itself, because the moment "800 Ranji runs" became the target instead of a proxy for "can bat under pressure against high-class bowling," batters optimized the number and the number stopped correlating with the thing selectors actually wanted. Every evaluation metric in an agent pipeline runs the same risk: the instant a team manages to the score rather than to the behaviour the score was invented to estimate, the score keeps climbing and the capability it was supposed to track quietly decouples from it.