Online Guardrail Metrics and Alerting — Agent Evaluation & Observability

Lesson 23 built a dashboard: panels a person reads. Lesson 24 built sampling: a defensible way to pick which live traffic gets human eyes. Neither one wakes anyone up. A dashboard that nobody is looking at when the tool-error rate doubles at 2 a.m. is a historical record, not a defense, and the natural next instinct -- put a threshold on every panel and page whenever one is crossed -- produces a pager that is either constantly wrong or constantly quiet, which are the same failure wearing two faces.

This lesson is about the small number of metrics worth paging a human for, and the alerting math that makes paging on them survivable. The metric side is a discipline of subtraction: most of what the dashboard shows should never fire an alert, because most of it is diagnostic, not a guardrail. The alerting side is a discipline from SRE practice, multi-window multi-burn-rate alerting, built to solve exactly the statistical problem agent metrics create: a fixed-size sample of a noisy binary outcome cannot support a single clean threshold that is both sensitive and quiet.

The two ideas are inseparable. A well-chosen guardrail metric paged on by a naive threshold still trains the on-call engineer to distrust the pager, and a beautifully engineered alerting pipeline pointed at fifteen metrics still produces a wall of noise no human can triage. This lesson builds both halves together, ending in code that runs the exact statistical failure and its fix side by side, on the same synthetic traffic.

Analogy🏏Cricket

🏏 Think of it like cricket: a television broadcast could put a graphic on screen for every ball -- dot ball, single, extras, review referred, boundary rope brushed -- and technically none of it would be false. But a production truck that flashes an alert to the director for every one of those events buries the moment that actually matters: Bumrah's yorker crashing into off stump versus a set batter, or a review that overturns the umpire's call and changes the match. The director's board has a handful of triggers wired to the hooter -- wicket, boundary, review outcome -- and everything else stays on the ticker where an analyst can read it at leisure. That short list is not laziness; it is the entire reason the hooter still means something in the 45th over of a tense run chase. A guardrail metric is the director's hooter list: refusal rate, tool error rate, escalation rate, latency, cost, safety triggers -- a short, deliberately incomplete set, chosen because everything on it is worth interrupting a human for, and nothing off it is.