A production agent handling even a modest workload generates tens of thousands of requests a day. You cannot read every transcript, run every trajectory through an LLM judge, and have a human review every tool call — the compute and headcount cost of full coverage scales linearly with traffic, and no eval budget scales that way. So every evaluation program, whether it admits it or not, is built on a sample: some slice of production traffic that stands in for the whole.
The problem is that which slice you choose is not a neutral logistics decision — it is a statistical one, and getting it wrong silently corrupts every number that comes out the other end. If you only sample the requests that were easy to log, or only the ones a support ticket was filed about, or only the ones from the last hour before your dashboard job runs, your failure rate estimate describes that biased slice, not your production traffic. The arithmetic on the biased slice is perfectly correct; the conclusion drawn from it is not.
This lesson is about the sampling decision itself: how to draw a sample that lets you say something true about the whole population, how to deliberately oversample the failures and rare segments you actually care about without lying to yourself about the aggregate, how tracing systems decide which traces to keep, and how to budget the same sample across human reviewers and automated judges. Every one of these choices either preserves or destroys your ability to trust the number you eventually report.
Analogy🏏Cricket
🏏 Think of it like cricket: Think of it like cricket: A national selector cannot watch every ball of every domestic match in a season — with dozens of Ranji Trophy fixtures running in parallel across grounds in Nagpur, Indore, and Rajkot, the honest move is to send scouts who each watch a sample of matches and file reports. But the choice of which matches to sample is itself a decision with consequences. If the scouting budget only stretches to day matches at flat, batting-friendly tracks like the one at the Holkar Stadium, the reports will show every uncapped batter looking fluent, because nobody watched them get exposed on a turning pitch at Chepauk against a Ranji offspinner extracting sharp bounce. A selector who reads those reports and picks a squad on that basis has not picked the best players — they have picked the players who happened to bat where the scouts were standing. The batting averages coming out of that sample are not wrong arithmetic; they are the right arithmetic applied to the wrong slice of reality, and nobody flags it because the numbers look clean. The insight: which balls you choose to watch determines which conclusions you are even capable of drawing, so the sampling plan is not a logistics afterthought bolted onto evaluation — it is the first decision that decides what your eval can and cannot see.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.