Detecting Loops, Stalls and Runaway Agents — Agent Evaluation & Observability

Most agent failures are loud: an exception, a schema-validation error, a 4xx from a tool API. This lesson is about the failure mode that is quiet and expensive instead -- an agent that never actually finishes. Lesson 19 established the shape of a run as a tree of spans; lesson 21 turned that tree into token, cost and latency numbers. This lesson uses the same step history those lessons instrumented, but asks a different question of it: not "what did this run cost," but "is this run ever going to stop, and if it shouldn't, who or what says so."

A run that never converges rarely looks broken from inside any single step. Each tool call returns a valid response, each LLM completion parses cleanly, each retry is a reasonable reaction to the step before it. The defect is not in any one step -- it's in the sequence, and a sequence is exactly what a step-by-step trace viewer is bad at surfacing unless you know to look for it. That is what makes runaway agents a cost problem before they are a correctness problem: a support bot that answers wrong is a bug ticket, but a support bot that calls a tool every four seconds for six hours is a bill.

This lesson covers a taxonomy of five ways an agent run fails to end, the concrete signals that detect each one from the run's own step history, why a step-count limit is necessary but not sufficient, and the policy layer that turns a fired guard into an action -- kill, degrade or escalate -- with the reason for that action recorded as a queryable field on the run, not just a log line that scrolls away.

Analogy🏏Cricket

🏏 Think of it like cricket: An IPL match at the Wankhede has a hard resource cost per over -- ground lights, broadcast slots, player fees -- and none of it depends on whether a wicket falls. Now imagine a rain break where the umpires keep walking out to inspect the pitch, waving the covers off, calling it back on, then off again, over and over, for three hours, without the match ever resuming. Nobody's cheating and the ball is technically still "in play" in the sense that the game hasn't been abandoned -- but every inspection cycle burns broadcast minutes, floodlight hours and squad availability exactly like a completed over would, while the scoreboard doesn't move an inch. The finance director watching the broadcast-rights invoice doesn't care that no boundary was hit; the bill is metered by time elapsed and lights burned, not runs scored. This is the exact shape of the outage that costs an AI team real money: an agent that keeps "trying," burning tokens and tool calls every step, without ever converging on the goal, looks busy on a step-by-step trace and looks catastrophic on the monthly bill. The fix in cricket is a match referee empowered to call it -- abandon or resume, but stop the metered clock from running on indecision; the fix for an agent is a runtime that treats "no progress" as an event worth acting on, not just watching.