Token, Cost and Latency Telemetry — Agent Evaluation & Observability

Cost and latency are not dashboard garnish bolted onto a working agent after the fact — they are correctness criteria in the same sense that giving the right answer is a correctness criterion. An agent that resolves every ticket correctly but costs $4 to do so, or takes ninety seconds when a human would answer in ten, has failed a requirement just as surely as one that gives the wrong answer. The eval suites and trajectory judges from earlier lessons in this course ask “did the agent do the right thing”; this lesson asks “did it do the right thing at a cost and a speed the business and the user can actually tolerate,” and both questions have to pass together before a production agent ships.

The trap is that both numbers look fine in aggregate and hide catastrophic failure in the tail. A dashboard reporting “average cost per resolution: $0.06” or “average latency: 22 seconds” can describe a fleet where the worst 1% of runs are five, ten, or twenty times that — runs that are quietly bankrupting the unit economics or making a handful of users furious, invisible behind a mean that barely moves. Averages are the wrong statistic for exactly the failure mode agents are most prone to: multi-step loops that occasionally spiral, retry, or get stuck, multiplying both cost and latency far past the typical case.

This lesson covers how to attribute cost correctly across five dimensions — per run, per user, per tenant, per tool, and per prompt version — why input and output tokens are priced asymmetrically and why cached tokens change the arithmetic again, how to decompose latency into the pieces that actually drive user perception, why a 10-step agent loop multiplies both cost and latency rather than adding to them, and the difference between stopping a runaway run before it happens and merely being told about it afterward.

Analogy🏏Cricket

🏏 Think of it like cricket: a team's chief selector can report, at the end of a bilateral series, that the side's “average required run rate achieved” across five ODIs was a perfectly healthy 6.1 an over. That number can be true and still hide the one match that actually decided the series — game four, where the required rate climbed to 11 an over in the last five overs and the middle order simply couldn't get there, and India lost by six runs. The average across five matches told the board nothing about the one match where the rate spiked, because a mean smooths exactly the spike that mattered. A required run rate isn't a garnish stat quoted for colour on commentary — it is the correctness criterion for whether the batting side is even still in the game, checked ball by ball, over by over, not once at the end of the series. The same is true of an economy rate for a bowler: a captain who only checks it after the innings is over has already lost the ability to do anything about the over that went for 24 runs in the middle of a death-overs collapse. Cost and latency for an agent are exactly this kind of criterion — not a number you glance at in a monthly report, but a live constraint that has to hold on every single run, because the runs where it doesn't hold are the ones that actually cost you the match.