Load Balancing and Health Checks: The Three States Your Checks Don't Have

Balancing decides where load goes. Health checking decides where it stops going. The second one causes more outages than the first.

Load balancing looks like a solved problem. You pick round robin, or you pick least connections, and traffic gets spread around. The algorithms are twenty lines each and every proxy implements all of them.

The interesting failures are not in the algorithm. They are in the health check — the mechanism that decides which backends are eligible at all. A balancer with a mediocre algorithm and a good health check degrades gracefully. A balancer with a perfect algorithm and a naive health check can empty its own backend pool in under a minute.

L4 and L7, and the gRPC trap

An L4 balancer works with TCP or UDP. It sees addresses and ports and forwards packets or connections. It is fast, protocol-agnostic, and cheap — and it balances connections, not requests.

An L7 balancer parses the application protocol. It sees HTTP methods, paths, and headers, and it balances individual requests. That costs more per request and buys you path-based routing, header-based routing, per-request retries, and protocol-aware health checks.

For HTTP/1.1 with short-lived connections the distinction is mostly about features. For HTTP/2 and gRPC it is a correctness problem, and it catches nearly everyone once.

HTTP/2 multiplexes many concurrent requests onto a single long-lived TCP connection. An L4 balancer assigns that connection to a backend exactly once, at connect time. Every request on it for the next several hours goes to the same backend. Ten clients, ten connections, three backends: your traffic is split 4/3/3 by client, not by request, and it will stay that way. Scale the deployment to six backends and the three new ones receive nothing at all, because no new connections are being made.

Three ways out, in increasing order of correctness:

• Terminate HTTP/2 at an L7 proxy — Envoy, nginx, a service mesh sidecar — which balances each stream independently. This is what a mesh gives you and is the usual answer.
• Client-side balancing. The client resolves all endpoints and picks one per call. gRPC’s round_robin policy over a resolving name does this natively.
• Cycle connections. MAX_CONNECTION_AGE on the server forces periodic reconnection, which lets rebalancing happen eventually. This is mitigation, not a fix.

If you have ever scaled up a gRPC service and watched the new Pods sit at zero CPU, this is why.

The algorithms, and what each one assumes

Every algorithm is a bet about what you do not know.

Round robin assumes every request costs the same and every backend is equally fast. Both assumptions are usually false. Its worst property is that a backend which has become slow still receives its full share — the algorithm has no feedback from outcomes at all.

Least connections / least outstanding requests counts work in flight rather than work dispatched. This is self-correcting in exactly the way round robin is not: a slow backend accumulates outstanding requests, so it looks loaded, so it receives fewer. For variable request costs this is the right default, and it is the one to reach for if you are choosing without measuring.

Power of two choices samples two backends at random and sends to the less loaded one. The result is the interesting part: it gets close to the load distribution of a globally-optimal choice, using no global state. That property is what makes it the default in distributed data planes — Envoy and most meshes use it — because maintaining accurate global load information across every proxy is exactly the thing that does not scale.

Consistent hashing maps a key to a backend so the same key lands in the same place, and moves only a small fraction of keys when the backend set changes. You use it when locality matters more than evenness: cache affinity, session stickiness, or sharded in-memory state. The mechanics are worth understanding separately; the operational caution is that it trades evenness away deliberately, so a hot key becomes a hot backend and no amount of capacity fixes it.

Weighted variants of any of the above handle heterogeneous backends — mixed instance types, or a canary that should receive 5%. Weight is orthogonal to algorithm, not an alternative to one.

One thing worth knowing about random: with enough backends and enough requests, plain random is not much worse than round robin, and it has no shared state at all. Round robin’s advantage shows up mainly at low request counts.

Health checks: the three states

Here is where the real failures live, and it starts with a modelling error.

Most health checks are binary: up or down. Reality has three states, and the missing one is the dangerous one.

State	Meaning	Correct action
Healthy	Serving normally	Send traffic
Overloaded	Alive, but out of headroom	Send less. Do not evict.
Broken	Will not recover unaided	Evict and restart

Collapse “overloaded” into “broken” and you have built a cascade generator.

Watch it run. An instance gets busy. Its latency rises past the health check timeout — because the check shares the same thread pool as real traffic. The balancer marks it unhealthy and evicts it. Its traffic redistributes to the remaining instances, which are also busy, and are now busier. They cross the threshold. They are evicted. Their traffic redistributes to an even smaller pool.

The fleet empties itself in under a minute, and every instance was capable of serving traffic the entire time. This is the canonical cascading failure, and the health check is the loop’s amplifier.

Four things prevent it:

Make the check independent of the request path. A separate, tiny thread pool, or a separate port. The check should answer “is this process functioning” without queueing behind production traffic. A check that shares the pool measures load, not health.

Never fail everything at once. Envoy’s panic threshold is the standard answer: if fewer than a configured percentage of hosts (50% by default) are healthy, ignore health status entirely and balance across all of them. The reasoning is blunt and correct — if most of your fleet looks unhealthy, the check is more likely to be wrong than the fleet. Spreading load across degraded backends beats spreading it across zero.

Distinguish load from brokenness. Return 503 with Retry-After when overloaded — that is backpressure, and it tells the balancer to reduce rather than evict. Reserve check failure for conditions that genuinely will not clear.

Add hysteresis. Require several consecutive failures before eviction and several consecutive successes before return. Flapping instances are worse than consistently absent ones, because every transition redistributes load and every redistribution is a small shock.

The Kubernetes probe rules

Kubernetes splits this into three probes, and the distinction is not cosmetic.

Readiness — should I receive traffic right now? Failing removes the Pod from Service endpoints. No restart. This is the one you almost always want, because it is reversible and its failure mode is benign.

Liveness — am I unrecoverable? Failing restarts the container. This is a destructive action and should be used only for genuinely unrecoverable states: a deadlock, a wedged event loop.

Startup — am I still booting? It suspends the other two until it passes, which is how a slow-starting application avoids being killed by a liveness probe before it has ever served anything.

Three rules that come straight from how those semantics interact:

Do not check dependencies in a liveness probe. If liveness tests the database, a thirty-second database blip restarts every Pod in the fleet simultaneously. Now you have a cold cache, a thundering herd of reconnections, and a database that was about to recover being hit by its entire client base at once. You converted a brief dependency problem into a full outage — and you did it with a feature intended to improve reliability.

Be cautious about dependencies in readiness too. It is less destructive, but if every replica checks the same dependency, every replica goes unready at the same moment and the Service has no endpoints. Readiness should mostly reflect this instance’s ability to serve, including its ability to serve degraded responses. A service that can still return cached results is not unready.

Prefer readiness; use liveness sparingly. A good rule: if the condition would resolve on its own given time, it is readiness. If a restart is the only thing that fixes it, it is liveness. Most conditions are the first kind.

startupProbe:                 # generous — slow start is not a failure
  httpGet: { path: /health/live, port: 8081 }
  failureThreshold: 30
  periodSeconds: 5            # up to 150s to boot

readinessProbe:               # reversible, checks this instance only
  httpGet: { path: /health/ready, port: 8081 }
  periodSeconds: 5
  failureThreshold: 3         # hysteresis

livenessProbe:                # destructive — strictly local, generous timeout
  httpGet: { path: /health/live, port: 8081 }
  periodSeconds: 10
  failureThreshold: 6         # a full minute before a restart

Port 8081 in that example is deliberate: a separate port with its own server and its own small thread pool, so probes cannot queue behind production traffic.

Draining, and the 502s after every deploy

If you see a burst of 502s on every deployment, this section is the reason.

Removing an instance is not instantaneous, and it is not atomic. When a Pod is deleted, two things happen concurrently: the endpoint is removed from the Service, and SIGTERM is sent to the container. Endpoint removal has to propagate to every kube-proxy and every ingress, which takes time. During that window, traffic is still arriving at a container that has already begun shutting down.

The sequence that works:

1. Fail readiness first, then wait. A preStop sleep of 5–15 seconds does this crudely and effectively — the Pod stops being advertised, and existing routes have time to notice.
2. Stop accepting new connections, finish in-flight requests.
3. Close idle keep-alive connections with Connection: close.
4. Exit — before terminationGracePeriodSeconds expires, or you get SIGKILLed mid-request.

lifecycle:
  preStop:
    exec: { command: ["sleep", "15"] }    # let endpoint removal propagate
terminationGracePeriodSeconds: 45          # must exceed preStop + longest request

And the container must actually handle SIGTERM. As covered in what a container actually is, PID 1 does not get default signal handling — an application that never installed a handler will ignore SIGTERM entirely and sit there until the grace period expires and it is killed hard, dropping every in-flight request. That is the other common source of deploy-time 502s, and it looks identical from the outside.

Outlier detection: the check you did not write

Health checks are active probes. Outlier detection watches real traffic and ejects backends whose actual responses are bad — consecutive 5xxs, consecutive gateway failures, or a success rate significantly below the fleet’s.

This catches what active checks miss, and the gap is larger than it sounds. A backend can pass /health perfectly while failing every real request: a corrupt cache, a bad config for one code path, a partially-applied deploy. The probe returns 200 because the probe is trivial. Outlier detection notices that this instance returns errors and its peers do not.

It is also a distributed circuit breaker with the right default: eject for a short interval, then let a fraction of traffic back to test. Envoy and most meshes implement it in a handful of lines of config, and it pairs naturally with the panic threshold — one ejects individual bad hosts, the other refuses to eject when too many look bad at once.

Inference backends break the usual assumptions

Balancing model-serving traffic is a genuinely different problem, for three reasons.

Request cost varies by orders of magnitude. A 20-token completion and a 2,000-token one are not the same unit of work. Round robin is actively harmful here; least outstanding requests is close to mandatory, because it is the only common algorithm whose feedback signal tracks actual occupancy.

Prefix cache locality is worth real money. Routing a conversation’s turns to the backend that already holds its KV cache avoids recomputing the prefix — a large latency and cost saving. That is consistent hashing on the conversation ID, which means you deliberately give up even distribution for locality. The tension between the two is a genuine design decision rather than a mistake, and the usual resolution is hash-with-fallback: prefer the cached host, spill to least-loaded when it is saturated.

Health checking has an extra state. A model server can be alive, responsive, and out of KV cache memory — which means it should receive no new sequences while happily finishing its current ones. That is not readiness in the usual sense, and a binary check cannot express it. The load signal you want is queue depth and free cache blocks, not liveness, which is why serving stacks expose those metrics specifically for the balancer to read.

The rule worth remembering

Use least outstanding requests unless you have a reason not to. Make health checks cheap, local, and independent of the request path. Fail readiness, not liveness. And never let your health checking evict the whole fleet.

The algorithm is rarely what hurts you. The check that cannot tell “busy” from “broken” is what turns a hot afternoon into an empty backend pool.