Load Balancing in gRPC

Analogy categoryAnalogies

Why Load Balancing in gRPC Is Different

Like a single long over where all six deliveries go to the same batsman rather than rotating strike, a single HTTP/2 connection multiplexes many gRPC calls, so an L4 load balancer that just spreads TCP connections leaves one backend batting through everything. gRPC's persistent, multiplexed connections mean traditional connection-level load balancing under-distributes traffic across a fleet of backend replicas, because once a connection is established, every RPC sent over it goes to the same server until the connection closes. This is fundamentally different from HTTP/1.1, where each request commonly opens (or reuses from a small pool) a short-lived connection that a simple TCP load balancer can freely redistribute.

🏏

Cricket analogy: Like a single long over where all six deliveries go to the same batsman rather than rotating strike, a single HTTP/2 connection multiplexes many gRPC calls, so an L4 load balancer that just spreads TCP connections leaves one backend batting through everything.

Client-Side vs Proxy-Side Load Balancing

gRPC ships with two built-in client-side balancing policies: pick_first, which connects to the first address the resolver returns and sends every call there (falling back only on failure), and round_robin, which opens connections to every resolved address and cycles calls across them evenly. Beyond these built-ins, larger deployments often adopt xDS — the same control-plane API Envoy uses — either through a service mesh sidecar or via gRPC's own 'proxyless mesh' support, letting a central control plane push load balancing configuration, endpoint membership, and health status to clients dynamically instead of hardcoding a policy in application code.

🏏

Cricket analogy: Like a captain rotating the bowling attack ball by ball between Bumrah, Shami, and Siraj rather than letting one bowler send down every over, the round_robin policy rotates each RPC across all available backend subchannels.

conn, err := grpc.Dial(
    "dns:///backend.svc.cluster.local:50051",
    grpc.WithDefaultServiceConfig(`{"loadBalancingConfig": [{"round_robin":{}}]}`),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
    log.Fatalf("failed to dial: %v", err)
}
client := pb.NewInventoryServiceClient(conn)

Name Resolution and Service Discovery

Client-side load balancing depends entirely on the name resolver returning multiple backend addresses rather than a single virtual IP; the default DNS resolver does this by resolving a hostname to its A/AAAA records, which works reasonably well against a Kubernetes headless Service (one without a clusterIP) that returns every backing pod's IP directly. The catch is that DNS results are cached for a TTL and carry no live health information, so a gRPC client can keep sending calls to a pod that Kubernetes has already terminated, or miss a newly scaled-up pod until the next scheduled re-resolution — a limitation that pushes many teams toward a custom resolver.Builder or an xDS-based control plane instead.

🏏

Cricket analogy: Like a scorer relying on a stadium announcer's periodic team-sheet updates rather than live tracking of who's on the field, DNS-based resolution only refreshes backend addresses periodically via TTL, missing pods that scale up between refreshes.

DNS-based name resolution alone is a poor fit for Kubernetes: it only refreshes on TTL expiry and returns a flat list of IPs with no per-endpoint health signal, so most production gRPC deployments pair it with either a custom resolver.Builder, an xDS control plane like Istio, or a headless Service combined with client-side round_robin.

Load Balancing Policies and Health Checking

Beyond pick_first and round_robin, more advanced policies like weighted round robin and least_request take backend load into account: weighted round robin sends proportionally more traffic to backends reporting better metrics via the ORCA (Open Request Cost Aggregation) protocol, while least_request routes each new call to whichever subchannel currently has the fewest outstanding in-flight requests. All of these policies interact with gRPC's health checking protocol, defined in grpc.health.v1.Health, which lets a balancer periodically ask each backend 'are you serving?' and exclude any backend reporting NOT_SERVING from the pick, preventing calls from being routed to a server that's still connected but overloaded or shutting down.

🏏

Cricket analogy: Like a captain giving more overs to a bowler with a better economy rate in the current match rather than splitting evenly, weighted round robin sends proportionally more RPCs to backends reporting lower load via ORCA metrics.

Placing a traditional L4 (TCP-level) load balancer in front of gRPC servers is a common mistake: because HTTP/2 connections are long-lived and multiplexed, an L4 LB balances at connection-establishment time only, so once connections are made, all subsequent RPCs stick to whichever backend was picked first, concentrating load unevenly.

gRPC's persistent, multiplexed HTTP/2 connections make simple L4 (connection-level) load balancing ineffective.
pick_first sticks to one backend; round_robin cycles calls across all resolved addresses.
Client-side load balancing requires a name resolver that returns a list of backend addresses, not a single VIP.
DNS-based resolution is TTL-cached and lacks real-time health signals, making it a weak fit for fast-scaling Kubernetes environments.
xDS-based balancing (Istio, proxyless mesh) offers dynamic, centrally-managed load balancing policies.
Weighted round robin and least_request policies distribute load based on reported backend utilization.
gRPC's health checking protocol lets a balancer avoid routing to backends that report NOT_SERVING.