Lesson 24 covered whether a tool works: given a fixed input, does the handler execute correctly, return the right shape, and fail the way it's supposed to fail. That question is entirely about the tool. This lesson is about the other half of the system, and it's a genuinely different question: given a fixed set of tools, does the model reach for the correct one, with the correct arguments, at the correct moment? A tool can pass every test from lesson 24 in isolation and the agent can still fail constantly in production because the model never calls it, calls a different tool instead, or calls it with a malformed date.
"Tool selection accuracy" is not one number, and treating it as one is the single most common reason teams can't debug their own agent. A model that's right 55% of the time could be failing to invoke a tool at all half the time, or invoking one but choosing the wrong one, or choosing correctly but filling the arguments wrong - three completely different engineering fixes hiding behind one score. This lesson builds the decomposition, the eval set, and the statistical discipline needed to tell those apart, and closes the loop back to the tool-description work from lesson 06.
Analogy🏏Cricket
🏏 Think of it like cricket: After India's 2020-21 tour of Australia, the series scoreline read 2-1 to India. That single number tells you almost nothing about what actually happened. Buried inside it is Adelaide, where India was bowled out for 36 - a batting collapse - and then Brisbane, where a patched-up attack of Natarajan, Siraj and Thangarasu Natarajan defended a small target with outstanding bowling. A coach who only looks at the 2-1 scoreline and says "our batting is fine" is making exactly the mistake this lesson is about: a single blended result hides which department broke. You need the batting average separately from the bowling economy separately from the catches-dropped count, because a team can win the series while one department is quietly failing and another is quietly carrying it. Tool-selection accuracy works the same way. A model that gets 70% of requests "right" might be calling a tool every time but choosing the wrong one half the time, or calling the right tool but botching the arguments, or simply not calling anything when it should. The insight is the same one that separates batting average from bowling economy: an aggregate score is a scoreline, not a diagnosis, and you cannot fix what the scoreline doesn't tell you is broken.
🏏 Showing the Cricket analogy — a Cricket version isn’t available for this concept yet.