Load Balancing Algorithms
A load balancer's job sounds simple — spread requests across a pool of servers — but the algorithm it uses to decide which server gets the next request has real consequences for latency, fairness, and resilience under uneven load. The simplest algorithms are 'static': round robin cycles through servers in fixed order, sending each successive request to the next server in the list, which works well when servers are homogeneous and requests are roughly uniform in cost. Weighted round robin extends this by assigning each server a weight proportional to its capacity, so a server with double the CPU gets roughly double the requests. Both are static because they never look at real-time server state — they decide purely from a fixed rotation, so a server that's momentarily slow (e.g. mid-garbage-collection) still gets its normal share of new requests. Different algorithms make different assumptions about request cost, server capacity, and whether related requests need to land on the same server, and picking the wrong one can leave some servers idle while others are overloaded even though the 'load balancer' is technically doing its job.
Cricket analogy: A captain rotating bowlers strictly in a fixed order regardless of form, always giving over five to the third seamer no matter how many runs conceded last over, mirrors round robin ignoring a bowler who's momentarily struggling.
Dynamic Algorithms
Least connections routes each new request to whichever server currently has the fewest active connections, which adapts naturally to servers that are slow or handling long-lived requests — it won't keep piling work onto a server that's already backed up. Least response time goes further, factoring in both active connection count and recent response latency, giving a more accurate live picture of server health. These dynamic algorithms require the load balancer to track real-time state per backend, which adds bookkeeping overhead but produces meaningfully better load distribution when request costs vary or when servers have heterogeneous performance.
Cricket analogy: A captain who assigns the next over to whichever bowler currently has the best recent economy rate, rather than a fixed rotation, mirrors least connections adapting to real-time workload.
Hash-Based Routing
IP hash and consistent hashing route based on a hash of some request attribute (client IP, session ID, cache key) so that the same input reliably lands on the same backend — essential for session affinity ('sticky sessions') and for cache servers, where routing the same key to the same node maximizes cache hit rate. Plain modulo hashing (hash(key) % N) breaks badly when N changes, since nearly every key remaps to a different server; consistent hashing solves this by minimizing the fraction of keys that move when servers are added or removed.
Cricket analogy: Always assigning a specific fielder to cover point based on a batter's known hitting tendency, a hash of the batter's identity, lets that fielder build familiarity, like session affinity improving cache hit rate.
# Simplified least-connections selection
def pick_server(servers):
# servers: list of (server_id, active_connections)
return min(servers, key=lambda s: s[1])
# Weighted round robin (simplified, using effective weight)
def weighted_round_robin(servers, state):
# servers: {server_id: weight}
# state: {server_id: current_weight}, mutated across calls
for sid in servers:
state[sid] = state.get(sid, 0) + servers[sid]
chosen = max(state, key=state.get)
state[chosen] -= sum(servers.values())
return chosenNGINX defaults to round robin but supports least_conn and ip_hash directives, while cloud load balancers like AWS ALB use a variant of least outstanding requests by default precisely because it adapts better to uneven request costs than static round robin in typical web workloads.
A frequent mistake is using plain round robin with servers of very different capacity (e.g. mixed instance sizes after a partial upgrade). Round robin gives every server an equal share of requests regardless of its actual capacity, silently overloading the smaller instances. Weighted round robin or a dynamic algorithm is required whenever backend capacity is heterogeneous.
- Round robin and weighted round robin are static algorithms that ignore real-time server load.
- Least connections and least response time are dynamic algorithms that adapt to current backend state.
- Hash-based routing (IP hash, consistent hashing) provides session affinity and cache-friendly routing.
- Plain modulo hashing remaps almost all keys when the server count changes; consistent hashing minimizes remapping.
- The right algorithm depends on request cost uniformity, server homogeneity, and whether affinity is required.
- Dynamic algorithms cost more bookkeeping but distribute load far better under heterogeneous conditions.
Practice what you learned
1. What limitation do round robin and weighted round robin share?
2. How does the least connections algorithm decide where to route a new request?
3. Why is consistent hashing preferred over plain modulo hashing (hash(key) % N) for routing to a changing pool of servers?
4. What is a primary use case for hash-based load balancing (e.g. IP hash)?
5. What problem occurs if plain round robin is used across servers with significantly different capacity?
Was this page helpful?
You May Also Like
Vertical vs Horizontal Scaling
Explains the two fundamental ways to add capacity to a system — bigger machines versus more machines — and the architectural implications of each.
Consistent Hashing
Introduces the hashing technique that minimizes data movement when nodes are added or removed, forming the backbone of distributed caches, sharded databases, and load balancers.
Auto-Scaling Basics
Introduces how systems automatically add or remove capacity in response to demand, covering triggers, scaling policies, and common pitfalls.
Redundancy and Failover
Techniques for eliminating single points of failure by duplicating critical components and automatically routing traffic away from failed instances to healthy ones.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesMicroservices Study Notes
Software Architecture · 30 topics
Software EngineeringTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 75 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics