Tool Selection and Tool Argument Accuracy — Agent Evaluation & Observability

A cricket ops-desk agent that reasons correctly about which player's strike rate needs updating and then calls check_drs_status instead of update_player_stats has produced a production failure, not a partial-credit answer on a trajectory rubric. Tool use is where agents fail most measurably, because every call is checkable against ground truth: a specific tool name, a specific argument schema, and a specific expected outcome exist for each turn, unlike free-text generation where correctness is often a matter of degree.

A 62% tool-accuracy score means one of several different bugs, depending on which layer produced it: the routing model never learned the boundary between two tools, the argument schema for one tool accepts a string where the API expects a numeric id, or the agent calls the right tool with the right arguments and then loops on an identical retry once the call errors. Each of these is a different bug with a different owner and a different fix, and a single aggregate score cannot distinguish them.

This lesson decomposes tool-call quality into four separable failure modes and builds a scorer for each: precision and recall over the decision to invoke a tool at all, a confusion matrix across the full tool catalogue to find which tools get mixed up, an argument-level accuracy check with exact match on enums and ids, tolerance on numerics, and schema validity as a hard floor, and a recovery-rate classifier for what the agent does after a tool call errors.

Analogy🏏Cricket

🏏 Think of it like cricket: MS Dhoni behind the stumps for Chennai Super Kings does not call for a Decision Review System (DRS) review on instinct alone. Four separate judgments happen before the umpire raises two fingers: first, should Chennai Super Kings review at all, or is the on-field call obviously correct and a wasted review costs them their last chance in the innings; second, which specific mode of dismissal are they reviewing, leg before wicket (lbw), caught behind, or a bump ball, because filing the wrong review type wastes the fifteen-second window; third, did the fielding side communicate the right details to the third umpire, the exact ball number and the batter's name, because a review filed against the wrong delivery gets rejected on a technicality; and fourth, if the ball-tracking feed drops mid-review the way it did in a 2019 Indian Premier League (IPL) eliminator, does the team have a fallback, or does the whole review stall. Just as a wicketkeeper's instinct to review is a separate skill from correctly identifying the dismissal type, an agent's decision to call a tool at all is a separate skill from picking the right tool out of the catalogue. Just as filing a review with the wrong ball number gets it rejected on a technicality identical to a malformed API argument getting rejected by a schema validator, an agent's argument accuracy is graded independently of whether it chose the right tool. The insight is that tool use needs four separate scorers instead of one: a wicketkeeper's review success rate is high not because the reviews come often, but because each of these four judgments, whether to review, what to review, how to file it, and what to do when the technology fails, gets tracked and gets right independently, and a single team highlighted for 'great reviews' hides which of the four skills is actually doing the work.