Incident Response for Agent Failures — Agent Evaluation & Observability

It is 2:14am and the page reads 'quality guardrail below threshold for 18 consecutive runs' — not 'service unavailable,' not '5xx rate above 1%.' Every request the returns-handling agent processed tonight returned a normal response in normal time. The dashboards a traditional on-call runbook checks first — uptime, latency, error rate — are all green, and they will stay green for the entire incident, because the agent is not down. It is confidently executing the wrong decision on every request that matches a pattern its judge score has flagged, and each wrong decision is a tool call that already ran against a real order, a real account, a real dollar amount.

This is the structural difference this lesson is built around: a service incident is measured in requests that failed, and you find out about it because something stopped responding. An agent incident is measured in actions that succeeded when they should not have — refunds issued, emails sent, records changed — and you only find out about it if a quality signal is instrumented and alerting, because nothing about the transport layer ever complains. Detection, triage, and recovery all have to be rebuilt around that difference; running an outage playbook against an agent incident wastes the first, most valuable minutes checking systems that were never broken.

What follows is the on-call procedure for the 2am page: how to stop new damage before you understand the root cause, how to use the trace log to find the first bad run and bound exactly how far the damage extends, how to identify who was affected and what has to be reversed on their behalf, how the postmortem and the customer communication both have to work, and the one non-negotiable step — a new eval case — that has to exist before the incident is considered closed. The code in this lesson runs against a real run log and produces exactly those artifacts: a bounded window, a blast-radius report, and a generated regression case.

Analogy🏏Cricket

🏏 Think of it like cricket: at Eden Gardens in a day-night Test, the umpire has no DRS available on a marginal call, so he gives Cheteshwar Pujara out lbw to a delivery that pitched a shade outside leg stump. Nothing about the ground looks wrong: the floodlights are on, the sightscreen is in place, the over rate is on schedule, the scoreboard updates every ball. A spectator watching the stadium infrastructure would say the match is running perfectly. But the umpire keeps making the same category of error for the next hour — turning down clear inside edges, upholding shouts that miss leg stump by inches — because his eye-line on that particular bowler's angle has been off since the session began, and nobody has told him. The team batting doesn't get a warning; wickets just keep falling to decisions that were wrong the moment they were given, and by the time the replays are reviewed at tea, four batters are back in the pavilion who should still have been at the crease. An AI agent in production fails the same way. The service is up, requests return 200, latency looks fine — the equivalent of the floodlights and the over rate — while the agent quietly keeps making a category of wrong call on every request that fits the pattern. Nothing pages you because nothing about the infrastructure broke; only the decisions did.