Statistical Significance with Small Eval Sets — Agent Evaluation & Observability

Two consecutive runs of the same 50-case eval set for an agent produce 84% and 82% — a difference that shows up as a red arrow on a dashboard, a message in the team channel, and sometimes a rollback. The question this lesson answers is unglamorous but load-bearing: given only n=50 cases, is a 2-point difference evidence of anything, or is it exactly the kind of number a fair coin-flip process would produce across two batches of an identical agent? The honest answer, worked out with real code later in this lesson, is that the confidence intervals around 84% and 82% at n=50 overlap so heavily that the two runs are statistically indistinguishable — before you've changed a single line of the agent.

This matters because agent teams change something — a prompt, a tool, a retrieval index, a model version — far more often than they can afford to build a 5,000-case eval set. Fifty cases is a realistic, even generous, size for a fast-moving team's regression suite, and every ship-or-no-ship decision built on top of it inherits whatever statistical noise lives in a sample that small. Treating a 2-point swing as a real regression triggers wasted debugging; treating a real regression as noise ships a broken agent. Both mistakes are avoidable with the right arithmetic.

This lesson builds four tools for that arithmetic: a Wilson confidence interval for a single accuracy number, McNemar's test for comparing two systems that ran the identical cases, bootstrap resampling for metrics with no textbook formula, and a power table that shows how large an eval set actually needs to be before a given effect size is detectable at all. It closes with the discipline of pre-registering which metric decides the outcome before you look at the results. The next lesson picks up a second, independent source of run-to-run noise — sampling temperature and repeated runs of the same prompt — but that is a different problem from the one here, and this lesson deliberately sets it aside.

Analogy🏏Cricket

🏏 Think of it like cricket: imagine a young opener, say a domestic prospect at the Wankhede, scores 84 in an innings against Mumbai's attack, and in his very next match at the same ground manages only 81. A commentator watching just those two innings might say his form has 'dipped by three runs' — but any coach worth their salary knows two innings tell you almost nothing about a batter's true ability. Cricket has spent a century learning to distrust small samples: a player's career average over 200 innings means something, but 'this Tuesday against these bowlers' does not. Selectors don't drop a player after one low score, and they don't hand a permanent India cap after one century at the Chinnaswamy against a depleted attack either — they wait for the sample to accumulate before treating a number as signal instead of the ordinary bounce of a cricket ball. An agent eval set of 50 cases is exactly that one-match sample: 84% correct on Monday's build and 81% on Tuesday's build looks like a regression, but it is closer to a batter getting a good ball versus a bad one — the kind of run-to-run wobble you'd expect even if nothing about the underlying skill changed at all. The discipline this lesson teaches is the coach's discipline: before calling a three-point drop 'real', ask how much of a swing pure chance could produce on a sample this small — and in both cricket and evals, the honest answer at n=50 is 'quite a lot.'