Why Timeouts Are Non-Negotiable
Every network call a microservice makes must have an explicit timeout. Without one, a thread waiting on a slow or hung dependency can block indefinitely, and under load, enough blocked threads will exhaust the caller's thread pool or connection pool, causing it to become unresponsive even to requests that have nothing to do with the slow dependency. A good default is to set the timeout based on the dependency's observed p99 latency plus a margin, not an arbitrary round number, and to set separate connect timeouts and read timeouts since a connection that never establishes is a different failure mode than one that connects but never responds.
Cricket analogy: A captain doesn't let a bowler take unlimited time to walk back to his mark and bowl; the shot clock exists precisely because an unbounded wait for one delivery would stall the entire innings, just as an unbounded call would stall a service.
Retries: When and How to Try Again
Not every failure is worth retrying. A transient failure — a dropped packet, a momentary connection reset, a 503 during a rolling deployment — often succeeds on a second attempt. A deterministic failure — a 400 Bad Request, a validation error, a 401 Unauthorized — will fail identically every time, so retrying wastes resources and adds latency without benefit. Retries should therefore be limited to idempotent operations and specific retryable status codes or exception types, and should use exponential backoff with jitter (for example, waiting 100ms, then 200ms, then 400ms, each with a randomized offset) so that a fleet of clients doesn't all retry in lockstep and create a synchronized thundering herd against a recovering service.
Cricket analogy: A batsman reviews a plumb LBW decision only when there's genuine doubt (a transient edge case), not on a review the ball-tracking clearly shows hitting middle stump, since wasting a DRS review on a hopeless case is like retrying a call that will never succeed.
Combining Retries with Timeouts Safely
Retries and timeouts must be tuned together, or they can amplify problems instead of solving them. If a downstream service is genuinely overloaded, retrying failed calls adds more load exactly when the service needs less, potentially turning a slow degradation into a full outage — this is why retries are often paired with a circuit breaker that stops retrying once failures are widespread. It's also critical to budget the total time a caller is willing to wait: three retries at a 5-second timeout each means a caller might wait 15 seconds before giving up, which may be unacceptable for a user-facing request, so the overall retry budget should be set from the caller's tolerance backward, not the dependency's behavior forward.
Cricket analogy: A team chasing a target doesn't keep sending the same aggressive batsman back in after repeated dismissals against a bowler who's clearly got their number; continuing that pattern only accelerates the collapse, just as unchecked retries against an overloaded service accelerate its failure.
// Exponential backoff with jitter, bounded by an overall retry budget
async function callWithRetry(fn, { maxRetries = 3, baseDelayMs = 100, timeoutMs = 2000 } = {}) {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), timeoutMs);
try {
return await fn(controller.signal);
} catch (err) {
if (attempt === maxRetries || !isRetryable(err)) throw err;
const delay = baseDelayMs * 2 ** attempt;
const jitter = Math.random() * delay * 0.5;
await new Promise(r => setTimeout(r, delay + jitter));
} finally {
clearTimeout(timer);
}
}
}
function isRetryable(err) {
return [502, 503, 504].includes(err.statusCode) || err.code === 'ECONNRESET';
}Retrying non-idempotent operations (like 'charge this credit card' or 'send this email') without an idempotency key can cause duplicate side effects. Always pair retries on write operations with an idempotency key the downstream service can use to deduplicate repeated attempts.
A useful mental model: timeouts answer 'how long will I wait for one attempt?' while retries answer 'how many attempts am I willing to make, and how much total time can that consume?' Both need explicit, deliberate values — never leave either to library defaults without checking what those defaults actually are.
- Every network call needs an explicit timeout to prevent one slow dependency from exhausting the caller's thread or connection pool.
- Set connect timeouts and read timeouts separately, based on observed p99 latency plus margin, not arbitrary values.
- Only retry transient, retryable failures on idempotent operations; deterministic failures (validation errors, auth failures) should never be retried.
- Use exponential backoff with jitter to avoid synchronized thundering-herd retries against a recovering service.
- Retries without a circuit breaker can worsen an outage by adding load to an already-overloaded dependency.
- Budget total wait time from the caller's tolerance backward, factoring in retries times per-attempt timeout.
- Non-idempotent operations need an idempotency key before they can be safely retried.
Practice what you learned
1. Why must every network call in a microservices system have an explicit timeout?
2. Which of the following is generally NOT a good candidate for automatic retry?
3. What problem does exponential backoff with jitter solve?
4. Why is an idempotency key important when retrying a payment request?
5. How should the total retry budget (number of retries times per-attempt timeout) be determined?
Was this page helpful?
You May Also Like
Circuit Breaker Pattern
A fault-tolerance pattern that stops a service from repeatedly calling a failing dependency, giving it time to recover and preventing cascading failures.
Bulkhead Pattern
A resilience pattern that isolates resources per dependency or workload so that failure or saturation in one area can't sink the entire service.
Graceful Degradation
The design discipline of keeping a system partially functional and useful when some of its dependencies fail, instead of failing entirely.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 75 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics
Software EngineeringSystem Design Study Notes
Architecture · 46 topics