100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
Tool Use, Function Calling & MCP
30 minadvanced

Cost and Latency of Tool-Heavy Loops

Every prior lesson in this course has treated a tool call as a discrete event with its own cost: a schema costs input tokens (lesson 2), a crowded catalogue costs selection accuracy (lesson 7), an oversized library costs a flat per-turn tax that tool search exists to shrink (lesson 19). What none of those lessons priced is the loop itself — the fact that a tool-using agent doesn't make one API call, it makes a sequence of them, and each call in that sequence is stateless. The API does not remember turn 3 when it processes turn 4; your code resends everything — system prompt, tool definitions, and the full message history accumulated so far — as input tokens on every single request.

That resending is where a tool-heavy loop's real cost hides. A task that takes ten tool calls to finish is not paying for ten tool calls' worth of new information; it is paying to re-transmit turns one through nine, in full, nine separate times, before turn ten ever adds anything new. The arithmetic of that compounds faster than most teams' mental model of "cost scales with turns" expects, and the same growing transcript that inflates the bill also inflates latency, because the model has to read through everything already resent before it can produce a single new token.

This lesson builds the actual arithmetic: why an N-turn loop pays close to quadratic input cost, why the schema tax from lesson 19 multiplies across every one of those turns, what prompt caching does and does not fix about that shape, how a turn's latency budget breaks into pieces that need separate diagnosis, and why the metric that should drive an agent's budget review is cost per completed task, not cost per token.

Analogy🏏Cricket
🏏 Think of it like cricket: Picture a T20 run chase where, at every strategic drinks break, the batting pair huddles and the non-striker insists on recapping absolutely everything from ball one — the toss, the fielding restrictions, every dot ball, every boundary, the over-by-over required rate — before adding the two new overs of information that actually happened since the last huddle. At the first drinks break in over 8, that recap is short: seven overs to summarize. By over 16 the recap alone eats a third of the two-minute break, because it is re-reciting fourteen overs' worth of ball-by-ball history to add value from two new ones, and by over 18 the recap is longer than the break itself allows. The pair is not just paying for the two new overs' worth of information at each stop — they are paying to re-narrate every earlier huddle in full, every single time, and that re-narration keeps growing even though the new information added at each stop stays the same modest size. A tool-use loop that resends the whole conversation on every turn works exactly the same way: turn 12 doesn't just pay for turn 12's new tool call and result, it pays to resend turns 1 through 11 in full as input tokens, and that resend gets more expensive at every single turn even when each individual turn is doing the same modest amount of new work.
Lesson 28 of 35
0% complete