100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
Tool Use, Function Calling & MCP
30 minadvanced

Rate Limits, Retries and Idempotency

Every tool call an agent makes sits between two rate limits that have nothing to do with each other. The model API — the request that produced the `tool_use` block in the first place — has its own throughput ceiling, enforced by the provider, that caps how many requests and tokens per minute your account can push through. The tool itself, when it runs, usually calls out to a second system entirely — a payment processor, an internal inventory service, a third-party search API — that enforces its own, completely separate ceiling, on its own infrastructure, with its own outage patterns. An agent that treats these as one undifferentiated "we got rate limited" signal will handle both badly.

The two limits fail differently and recover differently, which is exactly why they need different handling. A model-API 429 means your account, across every concurrent conversation you're running, has outrun the tokens-per-minute or requests-per-minute the provider allocated you — the fix lives at the client SDK layer, often shared across your whole fleet of agents. A downstream 429 from a single tool means that one specific service, called by that one specific tool, is saturated — the fix is local to that tool's call site and has zero bearing on any other tool the same turn might call next. Backing off the model API when only the downstream tool is struggling wastes a resource that was never the bottleneck; backing off a healthy downstream tool because the model API returned a 429 does the same in reverse.

This lesson works through both limits end to end: reading the signal each side actually gives you, computing a wait that doesn't make the problem worse when many callers hit it at once, telling retryable trouble apart from a request that was simply wrong, and — because a model will happily re-call a tool whose result looked ambiguous — making sure that a retried call is safe to run twice. Every example runs against `claude-opus-5`, and the same constraints apply to `claude-sonnet-5` and `claude-fable-5`.

Analogy🏏Cricket
🏏 Think of it like cricket: a touring team juggles two completely separate caps on how much bowling can happen in a day, and mixing them up costs matches. The team's own physio and workload-management plan caps how many overs Jasprit Bumrah can bowl in a day — a limit the team itself set, based on his own body, that the captain can renegotiate with the medical staff if match circumstances demand it. Completely separately, the stadium's ground-hire agreement with the host broadcaster caps how late play can run before the floodlights have to go off for a scheduled satellite window — a limit the touring team has no say over at all, set by an entirely different party for entirely different reasons. If Bumrah's workload cap is hit, the fix is internal: rotate to Mohammed Shami or Arshdeep Singh, a decision made inside the team. If the floodlight curfew is approaching, no amount of bowler rotation fixes it — the innings simply has to be declared or the game goes to a reserve day, because that ceiling belongs to the ground, not the team. A captain who responds to a floodlight curfew by resting his fast bowler, or responds to a tired bowler by complaining to the broadcast truck, is solving the wrong problem with the wrong lever. An agent facing a model-API rate limit and a downstream-tool rate limit is managing exactly these two kinds of ceiling — one it can adjust from the inside, one imposed entirely from outside — and treating them as the same problem burns exactly as much time as arguing with a broadcaster about a fast bowler's fitness.
Lesson 30 of 35
0% complete