/load-balancing

name: load-balancing
description: This skill should be used when the user adds a "load balancer", asks about "L4 vs L7" (transport vs application layer), picks a balancing algorithm ("round robin", "least connections", "weighted", "IP hash"), configures "health checks", needs "sticky sessions" / "session affinity", adds a "reverse proxy", does "SSL/TLS termination", spreads "traffic distribution" across an "autoscaling group", or wants a stateless web tier behind a single entry point. Use it whenever a design has more than one server behind one address, or a single box is the entry-point bottleneck or SPOF, even if the user doesn't say "load balancer".

Load Balancing

Spread incoming requests across a pool of identical backends so no single server is the bottleneck or the single point of failure. Get it wrong and the balancer becomes the SPOF it was meant to remove, routes traffic to dead servers, or *amplifies* an outage by hammering a backend that is already on its knees.

When to reach for this

Reach for a balancer when more than one backend serves the same role and traffic must be split across them; when a single entry point is a SPOF to eliminate; to add or remove servers without clients noticing (the enabler for the stateless tier and autoscaling); or to put one public address in front of a private fleet, with TLS termination, health-gating, and routing in one place.

When NOT to

One backend that comfortably handles peak load needs no balancer yet — adding one is a new component, a new failure mode, and a new thing to operate (YAGNI). If the real bottleneck is the database or a single hot shard, a balancer in front of the web tier solves nothing; diagnose the actual constraint first (→ `back-of-the-envelope`). Cross-region traffic steering is usually DNS/anycast at the edge, not an L4/L7 balancer (→ `content-delivery`). Per-client request limiting is a policy concern owned by `resilience-failure`, not the balancing algorithm.

Clarify first

• **Protocol & layer need** — raw TCP/UDP throughput (L4) or HTTP-aware routing by

path/host/header/cookie (L7)? This picks the balancer type.

• **State** — is the backend stateless, or does a session live on one server

(forcing affinity)? Moving state out is almost always the better answer.

• **Peak QPS & connection count** — one fat connection stream or many short

requests? Drives algorithm and balancer sizing (→ `back-of-the-envelope`).

• **Health signal** — what does "healthy" mean (TCP accept? a `/healthz` 200? a

deep dependency check?) and how fast must a dead node leave the pool?

• **TLS** — terminate at the balancer (offload backends, inspect L7) or pass

through end-to-end (compliance/mTLS)?

The options

**Layer of inspection**

• **L4 (transport):** route by IP/port; forward packets via NAT or DSR without

reading payload. Use when you need raw throughput, non-HTTP protocols, or the lowest added latency.

• **L7 (application):** terminate the connection, read HTTP (host, path, headers,

cookies), then route. Use when you need content-based routing, per-route pools, TLS termination, or request rewriting.

**Distribution algorithm**

• **Round robin / weighted round robin:** even rotation, weighted by capacity.

Use for uniform, stateless backends.

• **Least connections / least response time:** send to the least-busy node. Use

when request cost varies widely (long-lived connections, mixed workloads).

• **IP hash / consistent hash:** map a client (or key) to a stable backend. Use

for affinity without server-side session storage, or to keep cache locality.

• **Random (two-choices):** pick two at random, take the lighter — cheap and

surprisingly even at scale.

**Topology**

• **Active-passive:** one balancer serves, a standby takes the VIP on failure.

Simple HA.

• **Active-active:** multiple balancers share load (via DNS or anycast). Removes

the balancer's own SPOF and adds headroom.

**Reverse proxy role:** an L7 balancer is also a reverse proxy — one public face that hides backends, terminates TLS, compresses, caches, and centralizes routing. A reverse proxy is worth it even with a single backend for those benefits; load balancing is the multi-backend case of the same component.

Trade-offs

| Option | What it solves | What it worsens | Change it when | |---|---|---|---| | L4 balancing | Max throughput, low latency, any protocol | Blind to content; no path/header routing, no TLS inspection | You need content-based routing or TLS termination → L7 | | L7 balancing | Content routing, TLS offload, rewrites, observability | Higher latency/CPU; terminates connections (more state) | Raw throughput dominates and routing is trivial → L4 | | Round robin | Dead simple, even for uniform work | Ignores actual load; a slow node still gets its share | Request costs vary a lot → least-connections | | Least connections | Adapts to uneven request cost | Needs live connection state; can herd onto a just-recovered node | Backends are uniform → round robin is enough | | IP/consistent hash | Affinity & cache locality without session store | Uneven spread; rebalances on pool change | You can externalize state → round robin + shared store | | Sticky sessions | Works with stateful backends today | Breaks even spread, complicates scale-in, loses session on node death | State can move to Redis/DB → drop affinity | | Active-passive | Simple HA for the balancer | Idle standby; failover gap (seconds) | You need zero-gap headroom → active-active |

Behavior under stress

The balancer sits in the request path of *everything*, so its failure modes are the whole system's failure modes.

• **Health-check stampede (the classic foot-gun):** a backend recovers, the

balancer marks it healthy, and the full firehose hits a single cold node — empty caches, cold JIT, full connection backlog — so it fails its next health check and drops out, oscillating (flapping). Aggressive checks can effectively *DDoS a recovering servi