Testing Tools Without a Model in the Loop — Tool Use, Function Calling & MCP

A tool has two separate testable surfaces, and this lesson is only about one of them. The first is whether the tool does the right thing given valid input — a `search_orders` tool called with `{"order_id": "ord-1001"}` returns the right order, a malformed payload gets rejected with a clear reason, and a downstream outage produces a labeled, recoverable error instead of a stack trace. That is ordinary software testing, and it needs nothing from a model at all. The second surface is whether, given a user's actual request, a model such as Claude Opus 5 chooses to call this tool, with these arguments, at the right moment — a reasoning and selection question that lesson 25 covers in full.

Conflating the two makes both harder to debug and slower to fix. If a `close_account` handler silently returns the wrong shape on a locked account, that defect shows up first — if it shows up at all — as a confusing failure in a model-driven eval, where it looks like a selection problem: "the model didn't handle the locked-account case correctly." It is not a selection problem. The model called the right tool with the right arguments and received a broken answer back. No amount of prompt or description tuning fixes a broken handler, and diagnosing a handler bug through eval transcripts, one model call at a time, is a slow way to find a bug that a five-line test would catch in milliseconds.

This is why testing without a model in the loop is the higher-leverage half of the work, not a lesser one. It is deterministic — the same input produces the same output every time, with no sampling variance to explain away. It is fast and free — no API key, no network round trip, no per-run cost, which means it runs on every commit rather than "when someone remembers." And it catches the majority of what actually breaks tools in production: schema bugs, unhandled error paths, results shaped in ways the agent can't parse, and handlers that quietly do something other than what they claim. A test suite that requires a model and a live API key for every single run is a test suite that gets run less, run later under deadline pressure, and eventually not run at all — the tests that only exist when it's convenient don't really exist.

Analogy🏏Cricket

🏏 Think of it like cricket: In the week before a Test at the WACA, Jasprit Bumrah spends hours in the nets with a bowling coach and a stump camera, grooving the exact release point and seam position that makes his yorker land on off stump at the death. None of that requires an actual batter, an actual match situation, or a captain's decision — it is purely "can this delivery, executed correctly, do what it is supposed to do," checked against camera footage and a target zone painted on the popping crease. That is a completely different question from the one Rohit Sharma answers three overs later: given this batter, this required run rate, this field, should Bumrah bowl this over at all, versus a part-time spinner? Bumrah's yorker can be technically flawless in the nets and still be the wrong call if the situation demanded something else — and a captain can make the right call to bring him on even if, on this particular ball, the yorker sails down leg. The nets test whether the delivery works; the middle-over decision tests whether the captain reaches for the right bowler at the right moment. A team that only ever tests deliveries in the nets, and never reviews the captain's calls, has half a program — and so does one that only reviews calls without ever checking, ball by ball, that the deliveries actually do what they claim.