Cascading Failure: Backpressure, Load Shedding, and Graceful Degradation

The loop does not need the original trigger to keep running. That is what makes it a cascade rather than an incident.

Here is the postmortem shape that keeps recurring, in organisations of every size:

A single node became unhealthy and was removed from the pool. Its traffic was redistributed to the remaining nodes, which pushed them above their sustainable capacity. Latency rose, health checks began timing out, and further nodes were removed. Within four minutes the entire fleet was unavailable. The original unhealthy node was not the cause of the outage.

That last sentence is the point. The trigger and the outage are different things. One node failing is a routine event your system should absorb without anyone noticing. What turned it into an outage is that the response to the failure produced more failure.

Cascading failure is a positive feedback loop. And the defining property of a positive feedback loop is that removing the original input does not stop it.

The loop

The general shape, with the variations that fill in the blanks:

1. Something reduces capacity. A node dies, a deploy adds latency, a dependency slows, traffic spikes.
2. The load that node was carrying moves to its peers.
3. The peers are now above the utilisation where their latency stays flat. They slow down.
4. Slowdown triggers health-check timeouts, client retries, or both.
5. More capacity is removed, or more load is added. Go to 2.

Each turn of the loop makes the next turn worse, and there are three accelerants that reliably show up:

Retries. A client that retries three times converts one failed request into four. When failures are widespread, this multiplies total load by four at exactly the moment the system has the least capacity. This is why backoff and jitter are not optional and why a retry budget — cap retries at a small fraction of total requests, fleet-wide — is one of the highest-value controls you can add.

Health checks that measure the wrong thing. A health check that shares the request path’s thread pool starts failing because the service is busy, not because it is broken. The load balancer then removes a node that would have recovered, and gives its traffic to nodes that are equally busy. Health checks should be cheap, on a separate path, and should distinguish “I am overloaded” (keep me, send less) from “I am broken” (remove me).

Work amplification. A cache that empties sends every request to the origin. Under the cascade, a service that was doing 5% of the origin’s traffic is suddenly doing 100% of it — and the origin was never sized for that. Cold caches are one of the most common reasons a system that recovers still cannot come back up.

Four controls, and why you need all four

Each one fails differently, which is the argument for layering them rather than picking one.

1. Bound every queue

A queue is a buffer for a burst: arrivals temporarily exceed service rate, then fall below it, and the queue drains. That works.

If arrivals exceed service rate persistently, no queue size is sufficient. An unbounded queue grows without limit, and queueing delay grows with it. By Little’s Law, the time in the system is the number in the system divided by the throughput, so a queue that is 10,000 deep at 500 requests/second is imposing 20 seconds of delay on everything in it. Every one of those requests will have timed out before it is served. The system is at 100% utilisation, doing full work, producing nothing.

Bound every queue. Make the bound small. Small enough that the worst-case wait — depth divided by service rate — is inside the caller’s timeout. A queue deeper than that is not a buffer; it is a place where requests go to expire.

# Reject rather than accumulate.
queue = asyncio.Queue(maxsize=100)          # 100 / 500rps = 200ms worst case

try:
    queue.put_nowait(request)
except asyncio.QueueFull:
    return Response(503, headers={"Retry-After": "1"})

The more sophisticated version drops by age rather than by arrival order: if an item has been queued longer than its deadline, discard it without processing. This is where deadline propagation pays for itself — it lets the server know, cheaply, that a piece of work is already worthless.

2. Shed load deliberately

When you are over capacity, you have exactly two options: serve everything badly, or serve some of it well. The second is strictly better, and it is not a close call — a 50% success rate with fast, clean errors is a degraded service, while 100% of requests timing out is an outage.

The rules that make shedding work:

Reject cheaply and early. If rejecting costs as much as serving, shedding does not help. Reject at the edge, before authentication, before database lookups, in constant time.

Shed by priority, not at random. Classify traffic in advance and drop from the bottom: a health check and a paying customer’s checkout must be distinguishable. If everything is priority 1, shedding is random, and random shedding breaks the important 5% of traffic at the same rate as everything else.

Say so honestly. 503 with Retry-After tells a well-behaved client to back off. A silent drop, or a 200 with an empty body, teaches clients to retry immediately — which adds load.

Make it a switch you can flip. The most valuable version of this is a documented, tested control that an on-call engineer can turn on during an incident: “shed everything below priority 3 at the edge”. Building it during the incident is not an option.

3. Apply backpressure

Shedding is unilateral: you discard work and the sender finds out afterwards. Backpressure is cooperative: you tell the sender to send less, so the work is never created.

Backpressure is less wasteful but requires the sender to listen. It is worth building on paths you control on both ends, and it is already present in more places than people realise:

• TCP flow control is backpressure, and it is why a slow reader on a socket eventually stops the writer.
• A bounded connection pool is backpressure. When the pool is exhausted, callers block or fail rather than piling more concurrent work onto the dependency. This is the single most effective and most commonly missing control in the list.
• 429 Too Many Requests with Retry-After is explicit backpressure over HTTP, and it is what rate limiting exists to express.
• Concurrency limits beat rate limits for protecting a specific resource, because “N in flight” tracks the resource’s real constraint, whereas “N per second” does not account for how long each one takes.

An adaptive concurrency limit — AIMD, the same additive-increase/multiplicative-decrease control TCP uses — finds the right number without anyone tuning it:

class AdaptiveLimit:
    def __init__(self, initial=10, floor=1, ceiling=200):
        self.limit, self.floor, self.ceiling = initial, floor, ceiling

    def on_success(self, latency_ms: float, target_ms: float) -> None:
        if latency_ms < target_ms:                    # headroom — probe upward
            self.limit = min(self.ceiling, self.limit + 1)

    def on_overload(self) -> None:                    # timeout, 503, or rejection
        self.limit = max(self.floor, int(self.limit * 0.9))

Additive increase probes for capacity; multiplicative decrease retreats fast when the system objects. The asymmetry is the whole design — being slow to add load and quick to remove it is what keeps the control loop stable.

4. Degrade gracefully

Not all functionality is equally important, and a system that knows this can lose a lot of capacity without losing its purpose.

• Recommendations time out → show popular items. Nobody notices.
• The personalisation service is down → serve the generic page. Some people notice.
• Checkout is down → that is the outage.

The engineering is straightforward: every non-critical dependency gets a defined fallback and a short timeout, and the fallback path is tested. The hard part is organisational — deciding the priority order before the incident, with the people who can say which features are actually optional. A degradation plan invented at 3am by an engineer guessing at product priorities is a guess.

Two fallbacks worth building as defaults:

Stale-if-error. Serve the last known-good cached value when the origin fails. Stale data is almost always better than an error, and often better than a slow correct answer. Say so in the response — a Warning header or a UI indicator — so the staleness is visible rather than silent.

Precomputed defaults. A static fallback response, held in memory, requiring no I/O. It cannot fail under load because it does not do anything.

Getting out of one

The recovery has its own trap, and it catches people reliably.

The instinct during a cascade is to restart everything. Do that and the entire retry backlog — every client that has been failing and backing off — arrives at once against a cold cache and an empty connection pool. The system collapses again, and now you have also lost whatever state was warm.

The sequence that works:

1. Cut the load first. Shed aggressively at the edge, or drop traffic entirely. You cannot recover a system while it is still being overwhelmed.
2. Bring capacity up against the reduced load. Let caches warm, pools fill, JITs compile.
3. Reintroduce traffic gradually. 10%, 25%, 50%. Watch latency at each step. This is a progressive rollout, applied to traffic rather than to code.

This only works if step 1 is a control that already exists. Which is the practical argument for building the load-shedding switch on a quiet day: it is the thing you reach for first, and there is no time to write it.

Agents are an unusually good cascade accelerant

Everything above predates AI. What changes is the shape of the load.

Retry behaviour is not configured. A model that sees a tool error decides, on its own, whether to try again. That decision has no backoff, no attempt cap, and no awareness that the transport layer already retried. During a partial outage, a fleet of agents can produce a retry storm that no retry budget covers, because the retries are not coming from a retry library.

Load is bursty and correlated. Agents fan out — one task becomes twelve parallel tool calls. A hundred agents reacting to the same alert produce a hundred correlated fan-outs at the same instant. This is a traffic shape most capacity planning does not anticipate, and it is the top AI failure mode for a reason.

Agents are slow, so they hold resources longer. A tool call blocked on a 20-second model call occupies a connection for 20 seconds. Your pool sizing assumed 50ms.

And an overloaded system produces text, which agents read. Errors and timeouts go into the context window, the model reasons about them, and the reasoning frequently concludes: try again, or try a different approach — which means more calls.

Three controls that specifically address this:

• Rate-limit and concurrency-limit agents at the gateway, per agent identity. Not at the tool, and not inside the agent — a limit inside the agent is a limit each agent enforces on itself, which is not a limit.
• Give every agent a wall-clock deadline and a step cap. Both are termination conditions that do not depend on the model deciding it is finished.
• Return rejections the model can act on. 429, retry after 30s, do not retry sooner in the tool result is instruction the model will generally follow. A bare error is an invitation to improvise.

The rule worth remembering

Bound every queue, shed by priority, apply backpressure through bounded pools, and decide your degradation order before you need it.

A cascading failure is not a chain of dominoes you can stop by catching one. It is a loop that sustains itself, which means the only effective interventions are the ones that were already in place when it started.