Variance, Temperature and Repeated Runs — Agent Evaluation & Observability

A single eval run of an agent is a lottery ticket. Lesson 12 covered when a small eval set's aggregate score can be trusted at all — statistical significance, the question of whether a difference between two scores is real or just sampling noise. This lesson goes one level deeper: not "is the eval set big enough to trust the score," but "where does the run-to-run noise that significance test has to account for actually come from, and what do you do about it operationally, not just statistically."

That noise is not one dial. Several independent sources stack on top of each other, and they split into two very different categories: variance you deliberately built in because it serves the product (sampling temperature, top-p), and variance that sneaks in from the serving stack and the outside world whether you asked for it or not (nondeterministic kernels, live tool results, a retrieval index that changed underneath you). Treating both kinds the same way — as an undifferentiated blur of "the agent is a bit random" — makes it impossible to fix the parts you can actually control.

The practical response to all of it is the same: run more than once, and be precise about which question you're asking when you do. "Does it work at least once in k tries" and "does it work every single time in k tries" are different empirical claims, computed from the same repeated-run log, and conflating them is one of the most common mistakes in agent eval reporting — this lesson gives you the exact arithmetic to keep them separate.

Analogy🏏Cricket

🏏 Think of it like cricket: Jasprit Bumrah bowls the final over of a T20 decider at the Wankhede, needs to defend 9 runs, and does it — a dot ball, a yorker that cramps the batter, a slower ball that gets scrambled for a single, and a run-out off the last delivery. The broadcast headline the next morning says "Bumrah is unbeatable at the death." But anyone who has watched him across a full IPL season knows that single over doesn't settle the question. The same over bowled again, same batter, same asking rate, could go for 14 if the yorker drifts fractionally full and gets creamed through cover, or if dew on the ball that night changes how it grips in his fingers at release. The outcome of one over is shaped by things that have nothing to do with Bumrah's underlying skill: which batter is on strike, how much dew is on the ball, whether a marginal umpiring call goes his way. A commentator who wants to say something true about his death-bowling reliability doesn't cite one over — they cite his economy rate across dozens of spells, because that's the only way to separate "he is good at this" from "he got a good night." An agent's single eval run is that one over: impressive or damning, and almost worthless on its own for judging whether the underlying system is actually reliable.