Cost-Aware Evaluation at Scale — Agent Evaluation & Observability

Every eval suite you have built across this course — the deterministic checks, the LLM-as-judge rubrics, the trajectory scorers, the calibrated judge from lesson 07 — has a line item on someone's invoice. That line item is not a rounding error. An eval suite runs the agent, then runs a judge on top of the agent's output, often several times per case to control for variance, and every one of those is a billed model call. Treat the suite's cost as a design constraint from day one, the same way you treat its statistical power or its coverage, because a suite that is not affordable is not a suite — it is a demo that only ran once.

The arithmetic is multiplicative, not additive, which is exactly what makes it sneak up on a team. Suite cost scales as cases times repeats times judge calls per repeat times price per call, and each of those four factors independently creeps upward as a product matures: more cases get added as new bugs are caught and turned into regression tests, repeats increase from 1 to 3 or 5 as someone notices the pass rate is noisy at temperature 0.7, and a single judge call per case-repeat becomes two or three once pairwise comparison and a secondary rubric check get added. A suite that cost $50 a night at launch can cost $2,000 a night eight months later without anyone deciding, in one sitting, to 40x the eval budget — it happened one reasonable addition at a time.

The failure mode this produces is not a budget overrun email — it is silence. Once a suite's nightly bill becomes uncomfortable, the realistic outcomes are that someone quietly disables it, someone quietly shrinks it without telling the team what coverage was lost, or someone lets it keep running until finance asks a question nobody on the eval team can answer. All three outcomes remove the regression detection the suite existed to provide, and the removal is invisible until a regression that the suite would have caught reaches production instead. Cost-aware evaluation is not about running evals cheaply for its own sake; it is about keeping the suite affordable enough that it keeps running, because an eval suite that stops running stops being a safety net.

Analogy🏏Cricket

🏏 Think of it like cricket: an IPL franchise's nets sessions are supposed to be cheap preparation for the expensive thing — the actual match. But watch a professional setup for a week and the arithmetic inverts. Three top-order batters each face four bowling machine settings, six deliveries per setting, twice a day, with a throwdown specialist standing in for a left-arm quick the analytics team flagged as a matchup risk. That is 3 batters x 4 settings x 6 balls x 2 sessions = 144 throwdowns a day, every day, for a squad that will bowl maybe 120 overs — 720 balls — across an entire IPL match. Multiply in the extra fielding drills, the bowlers' own rhythm overs, and the specialist DRS-review simulation sessions before a big game, and a franchise can spend more balls and more specialist hours in nets during a single home stand than it does in the matches that stand exists to prepare for. No franchise abandons nets because of this — but every professional setup tiers it: light throwdowns the morning after a match, full-intensity simulation only before a must-win game, and a hard cap on how many specialist sessions the budget allows per month. The nets bill is real, it compounds the same multiplicative way — batters x variations x balls x sessions — and a franchise that never looks at that arithmetic eventually cuts nets entirely under budget pressure, which is exactly the outcome that leaves batters undercooked against a bowling attack they never practiced against.