Retries, Backoff, Jitter, and Circuit Breakers: The Four Controls That Decide Whether You Recover or Melt Down

Backoff sets how much load. Jitter sets when it arrives. A circuit breaker decides whether it arrives at all.

A retry is the most reasonable-looking line of code that has ever taken down a production system. The request failed, the failure was probably transient, so you try again. Nobody has ever been talked out of that instinct, and nobody should be — most failures are transient, and most retries do succeed.

The problem is what a retry actually is when you zoom out from one client to all of them: a retry is a load multiplier that activates precisely when the system is least able to absorb load. A dependency starts failing at 20% and every client retries twice. Congratulations — you have just increased traffic to a struggling service by up to 40% at the exact moment it was asking for less. That is the shape of a cascading failure, and it has a name in the Google SRE book’s chapter on cascading failures: the system that was merely degraded is now saturated, and it cannot recover while the retries continue.

There are four controls that sit between “retry” and “meltdown.” Most teams have one or two of them. You want all four, and you want to know which problem each one solves, because they are not interchangeable.

1. Retry: what you are allowed to try again

The first control is not how to retry. It is whether.

A retry is only ever correct for two kinds of failure: the transient ones (a dropped connection, a leader election, a brief overload) and the ones the server explicitly told you to retry. Everything else is a waste of a request, and on a struggling system a wasted request is worse than no request.

Response	Retry?	Why
Connection timeout, reset, DNS blip	Yes	Classic transient; the request may never have landed
429 Too Many Requests	Yes, after Retry-After	Server is explicitly rate-limiting you and told you when to come back
503 Service Unavailable	Yes, after Retry-After if present	Server is shedding load deliberately
500, 502, 504	Usually, if the operation is idempotent	Might be transient; might be a poison request that kills every replica it touches
400, 401, 403, 404, 422	Never	Deterministic. The same request will fail identically forever
Anything non-idempotent	Only with an idempotency key	See below — this is the dangerous one

That last row is the one that quietly corrupts data rather than causing an outage. A POST /payments that times out has an unknowable outcome: the request may have been fully processed and the response lost. Retrying it blind risks charging twice. The fix is an idempotency key — a client-generated identifier the server uses to deduplicate, the way Stripe’s idempotent requests work. Idempotency is what makes a retry safe; without it, a retry is a gamble you take on every timeout. If a call is not idempotent and has no key, the correct retry count is zero.

Retry-After deserves its own line. It is defined in RFC 9110, it can accompany a 429 or a 503, and when it is present it overrides whatever your backoff formula computed. The server has information about its own recovery that your client cannot possibly have. Clients that compute a clever backoff and ignore Retry-After are the ones that keep a rate limiter pinned.

2. Backoff: how much load, over time

Fixed-interval retries are barely better than no backoff at all. Retry every second for thirty seconds and you have turned one failed request into thirty, all while the dependency is down.

Exponential backoff makes each attempt wait roughly twice as long as the last: 100 ms, 200 ms, 400 ms, 800 ms. The request count for a long outage collapses from linear to logarithmic, which is the entire point. Two parameters matter more than the base:

The cap. Uncapped doubling reaches absurd delays — attempt 15 is nine hours out. Cap the per-attempt delay at something bounded by your actual recovery expectations (often 20–30 seconds) so that a client which has been waiting through an outage comes back promptly when the service returns, rather than sleeping through the recovery.

The attempt limit. Backoff shapes the retries; it does not end them. Something has to. Bound retries by total elapsed time rather than attempt count where you can — “give up after 10 seconds” composes far better across a call chain than “try 5 times,” because the time budget is the thing your caller actually cares about.

And there is a multiplication problem that catches almost everyone. Retries at multiple layers multiply, not add. Three retries in your HTTP client, inside a service that itself retries three times, inside a gateway that retries three times, is 27 requests hitting the bottom of the stack for one user action. The standard discipline is to retry at exactly one layer — usually the one closest to the failure, where you know most about whether the failure is transient — and to have every other layer fail fast. If you cannot enforce that, enforce a retry budget instead: Envoy’s retry budgets cap concurrent retries as a percentage of active requests, so retries can never become more than a fixed fraction of your traffic no matter how many layers ask for them.

3. Jitter: when the load arrives

This is the control that gets skipped, and it is the one that turns a working backoff into a useless one.

Consider a dependency that fails for every client at the same moment — a deploy, a leader election, a network partition healing. Every client starts its backoff clock at the same instant. With pure exponential backoff, every client retries at t+100ms, then every client retries at t+200ms, then t+400ms. The retries are fewer, but they are perfectly synchronised: the service sees a series of sharp, full-fleet spikes separated by total silence. The first spike knocks it back down, everyone backs off in lockstep, and the pattern repeats. The system oscillates instead of recovering.

Jitter fixes this by randomising the delay so clients decorrelate. The canonical treatment is still AWS’s “Exponential Backoff And Jitter”, which compared the variants directly. The one worth defaulting to is full jitter:

import random, time

BASE = 0.1          # seconds
CAP  = 20.0         # seconds
MAX_ELAPSED = 30.0  # total time budget

def call_with_retry(fn, *, base=BASE, cap=CAP, max_elapsed=MAX_ELAPSED):
    started = time.monotonic()
    attempt = 0
    while True:
        try:
            return fn()
        except Retryable as exc:
            elapsed = time.monotonic() - started
            if elapsed >= max_elapsed:
                raise

            # Server instruction always wins over our own arithmetic.
            if exc.retry_after is not None:
                delay = exc.retry_after
            else:
                # Full jitter: uniform over the whole backoff window, not
                # a small wobble around its edge. This is what decorrelates
                # clients that all failed at the same instant.
                window = min(cap, base * (2 ** attempt))
                delay = random.uniform(0, window)

            # Never sleep past the budget.
            delay = min(delay, max_elapsed - elapsed)
            time.sleep(delay)
            attempt += 1

Note what full jitter is not. It is not window * random.uniform(0.9, 1.1) — a small wobble around a synchronised instant is still a synchronised instant. Full jitter samples uniformly across the entire window, which is what actually flattens the spike into a plateau. The counter-intuitive result from the AWS work is that this both reduces total load and does not meaningfully increase completion time, because the clients that draw a short delay get through early and drain the queue for everyone else.

4. The circuit breaker: whether the load arrives at all

Backoff and jitter are per-client, per-request controls. They shape one caller’s behaviour. They do nothing about the fact that a hard-down dependency is still receiving — and slowly timing out — every single request your service makes to it.

That slow timeout is the real danger. Each in-flight call to a dead dependency holds a connection, a thread or task, and a slice of your memory for the full timeout duration. Enough of them and your service exhausts its own resources waiting on someone else’s outage, and now you are down too. The failure has propagated one layer up.

A circuit breaker, as popularised in Michael Nygard’s Release It!, is shared state in front of a dependency with three positions:

• Closed — normal. Calls pass through; failures are counted.
• Open — the failure rate crossed a threshold. Every call fails immediately and locally, with no network I/O and no timeout wait. This is the load being removed from the struggling dependency, and simultaneously the resources being returned to you.
• Half-open — after a cooldown, let a small number of trial calls through. If they succeed, close. If they fail, open again and restart the cooldown.

The half-open state is the part people get wrong. Its job is to probe with the smallest possible amount of traffic, because the dependency is by assumption fragile. Slamming it with full traffic the instant the cooldown expires just re-opens the breaker and wastes the recovery. Let a handful of requests through, and ramp.

A note on thresholds: trip on a rate over a window with a minimum-volume floor, not on a raw count. “Five consecutive failures” trips spuriously on a low-traffic endpoint that happened to see a blip. “More than 50% failures over the last 20 requests, minimum 20 requests” is the shape you want. Envoy calls this outlier detection and applies it per-host, ejecting individual bad backends rather than the whole cluster — usually the better granularity, since partial failures are far more common than total ones.

Two things a breaker needs that are easy to forget. It needs a fallback — a cached value, a degraded response, a clear error — because “open” means something is returned to your caller and you should decide what. And it needs to be observable: breaker state transitions are among the highest-signal events in a distributed system, and a breaker that opens silently is an outage you find out about from a user.

How the four compose

The mental model that makes these stop feeling like a grab-bag:

Retry decides if. Backoff decides how much. Jitter decides when. The circuit breaker decides whether at all.

They operate at different scopes, which is why you need all four. Retry and backoff are per-call. Jitter is about the fleet. The breaker is about the dependency. Remove any one and you have a gap: backoff without jitter gives you synchronised spikes; jitter without a breaker still hammers a hard-down service forever; a breaker without backoff means every closed-circuit period starts with a stampede.

And one thing that sits above all four: load shedding on the server side. Every client-side control is voluntary, and you do not control your clients — not the one written by another team, not the one with an old SDK, not the one with a retry loop somebody added during an incident. The server has to be able to say no cheaply. A fast 429 with a Retry-After costs almost nothing to produce and is strictly better than a slow timeout, which costs you a held connection and tells the client nothing.

The agent loop breaks all of this

Here is why this fundamentals post is running on a blog about AI platforms.

Every control above assumes the retry decision is made by code you configured. In an agentic system it is not. The stack now looks like this:

1. Your HTTP client retries the tool’s API call — 3 attempts, backoff, jitter. Correct.
2. The agent SDK retries the tool invocation — another 3 attempts. Already 9.
3. The model sees a tool error in its context and decides to call the tool again. No backoff. No cap. No shared budget. No awareness that layers 1 and 2 exist.

That third layer is new, and it is qualitatively different from the first two: it is a discretionary retry made by a non-deterministic component that is optimising for task completion, not for the health of your dependency. It will cheerfully retry a 403, because from inside the context window a permission error looks like something worth trying differently. And it multiplies with everything beneath it.

Worse, agent retries are correlated across the fleet in exactly the way jitter exists to prevent. A thousand agent sessions running the same workflow against the same newly-degraded MCP server will all reach the same failing step and all decide to retry, because they are running the same model on near-identical context. This is fleet-wide synchronisation with none of the decorrelation, and the capacity failure mode it produces is the one I keep seeing first in AI platforms.

Four things that actually help:

• Retry inside the tool, not above it. The tool implementation owns the retry policy — backoff, jitter, cap, the lot. It is ordinary code and you can reason about it.
• Return terminal errors to the model. When the tool has exhausted its own retries, the error string handed back should be unambiguous and final: "Rate limited by the upstream API. Retried 3 times over 12s and gave up. Do not retry this call." Models respond to explicit instruction in tool output far more reliably than they infer policy from an error code.
• Cap total tool calls per task at the harness. A hard structural ceiling that the model cannot reason its way past. This is the one control that is actually guaranteed.
• Put the breaker in the MCP gateway, not in the agent. Breaker state must be shared across sessions to be meaningful. A breaker inside a single agent session protects nothing, because the other 999 sessions have their own.

The general principle, and the one worth carrying out of here: in an agentic system, a retry policy that depends on the model’s judgement is not a policy. It is a suggestion. Make it structural — in the tool, in the gateway, in the harness — where it is enforced by code rather than inferred from context. That is the same shift as bounded autonomy applied to one very specific, very expensive decision.

The four controls are forty years old and they still work. They just need to be installed somewhere the model cannot argue with them.