Idempotency Keys: The Thing That Makes a Retry Safe

The timeout is the problem. The client cannot tell “never happened” from “happened, response lost” — and those need opposite responses.

Here is the single most under-appreciated fact in distributed systems: a timeout tells you nothing about whether the work was done.

Your client sends POST /payments. Twenty seconds pass. The connection drops. What happened? Maybe the request never arrived. Maybe it arrived, was fully processed, the card was charged, and the response was lost on the way back. Maybe it is still executing right now. From the client’s position these are indistinguishable, and they demand opposite actions: retry in the first case, absolutely do not retry in the second.

This is not a corner case. It is the normal, expected behaviour of every network call that has ever been made, and it is the reason retry logic is dangerous in a way that retry logic does not look dangerous. Idempotency keys are how you resolve the ambiguity, and they are simple enough to explain in a sentence and subtle enough that most implementations have at least one of the bugs below.

Idempotent, and what it actually means

An operation is idempotent if performing it multiple times has the same effect as performing it once.

GET /users/42 is idempotent — reading twice changes nothing. PUT /users/42 {"name": "Ajin"} is idempotent: apply it a hundred times and the name is still “Ajin”, because it sets rather than modifies. DELETE /users/42 is idempotent in effect, even though the second call returns 404 — the resource is gone either way.

POST /payments {"amount": 5000} is not, and neither is PATCH /account {"balance": "+100"}. Both describe a change relative to current state, so applying them twice produces a different world than applying them once.

RFC 9110 defines which HTTP methods are idempotent, and it is worth being precise about what that definition gives you: it is a statement about the semantics the method promises, not a guarantee your handler honours it. A PUT implemented as “append to a list” is not idempotent no matter what the spec says about PUT. The method is a contract with your callers. Whether you keep it is up to your code.

The important consequence: the non-idempotent operations are exactly the ones that matter. Charging a card, sending an email, creating an order, provisioning a server, posting to a channel. Nobody has ever been paged because a GET ran twice.

The mechanism

The client generates a unique key before its first attempt and sends it with every attempt of that same logical operation:

POST /v1/payments HTTP/1.1
Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7
Content-Type: application/json

{"amount": 5000, "currency": "usd", "customer": "cus_abc"}

The server, on receiving a request with a key:

1. Look up the key. If a completed record exists, return the stored response. Do not execute.
2. If it is new, claim it atomically, execute the operation, and store the response against the key before replying.
3. If a record exists but is still in flight, return 409 Conflict — a duplicate is executing right now.

That is the whole idea. Stripe’s idempotent requests are the reference implementation most APIs copy, and AWS does the same thing under a different name — the ClientToken parameter on operations like RunInstances exists for exactly this reason, because “accidentally launched 200 instances instead of 100” is a retry bug with a bill attached.

The key must come from the client. This is the part that gets reinvented wrong. If the server minted the identifier, it would have been in the response — the response that got lost, which is the whole reason we are retrying. The retry would arrive with no key and look like new work. The key has to exist before the first attempt and survive the failure, which means the client owns it.

And it must be reused unchanged across retries of the same logical operation, and be different for genuinely different operations. A client that generates a fresh UUID inside its retry loop has implemented nothing.

Four ways this goes wrong

The mechanism is three steps. The bugs are all in the details of those three steps.

1. The concurrency race

This looks correct and is not:

record = db.query("SELECT * FROM idempotency WHERE key = %s", key)
if record:
    return record.response
result = charge_card(amount)        # two concurrent requests both reach here
db.insert("INSERT INTO idempotency ...", key, result)
return result

Two requests carrying the same key arrive within milliseconds of each other — which is precisely what happens when a client’s timeout fires just as the original is completing. Both read no record. Both charge the card. The check-then-act is not atomic.

The fix is to let the database enforce it. A unique constraint on the key column, and an insert before doing the work:

try:
    # Claim the key first. The unique index is the lock.
    db.execute(
        "INSERT INTO idempotency (key, request_fingerprint, state) VALUES (%s, %s, 'in_progress')",
        key, fingerprint,
    )
except UniqueViolation:
    existing = db.query("SELECT * FROM idempotency WHERE key = %s", key)
    if existing.state == 'in_progress':
        # A duplicate is executing right now. Tell the client to come back.
        raise Conflict("A request with this Idempotency-Key is in progress")
    if existing.request_fingerprint != fingerprint:
        raise UnprocessableEntity("Idempotency-Key reused with a different request body")
    return existing.response          # completed — replay it

result = charge_card(amount)
db.execute(
    "UPDATE idempotency SET state = 'completed', response = %s WHERE key = %s",
    serialize(result), key,
)
return result

Claim first, then work. The unique index does the mutual exclusion for you, correctly, without a distributed lock.

This also exposes the crash window: if the process dies after charge_card and before the UPDATE, the key is stuck in_progress forever and every retry gets a 409. You need a reaper — a job that finds stale in_progress records past a timeout and either reconciles them against the downstream system or releases them. It is unglamorous and it is the difference between a mechanism that works in a demo and one that works in production.

2. Not storing the response

Storing only “this key was used” is half an implementation. The retry then gets a bare 200 with no body — but the client needed the payment ID, and now it has no way to get it. It will likely resubmit with a fresh key, which is the duplicate charge you were preventing, reached by a different path.

Store the full response: status code, body, and the headers the client needs. A replay should be byte-identical to what the original attempt would have returned. Include a marker — Idempotent-Replay: true is a common one — so the client can tell the difference if it cares, without changing the payload.

Store errors too, but be careful which ones. A 422 Validation Failed is deterministic and should be replayed. A 503 from a downstream outage is transient, and caching it means a client that retries after the dependency recovers gets the old failure forever. The rule: cache the outcome when the outcome is a property of the request; release the key when it is a property of the moment.

3. Not fingerprinting the request

What should happen if a client sends the same key with a different body?

Idempotency-Key: abc-123    {"amount": 5000}     → charges $50
Idempotency-Key: abc-123    {"amount": 500000}   → returns the $50 result

Silently returning the first result is wrong — the client asked for something different and got no indication it did not happen. It is almost always a client bug (a key reused across a loop iteration), and silence lets it ship.

Hash the canonicalised request body on first use, store the hash, and compare on replay. Mismatch gets a 422 with a clear message. This one check catches a whole class of client bugs before they become support tickets.

4. Getting the scope wrong

Keys must be scoped to the caller. If two tenants can collide on the same key, tenant B can receive tenant A’s response — a data leak dressed as a caching bug. Scope by (api_key_or_tenant, endpoint, key), not by key alone. Clients generate UUIDs, and UUIDs do not collide; that is not the point. The point is that a scoped key cannot leak across a boundary even when something upstream goes wrong.

And set a TTL. Twenty-four hours is the common choice; Stripe’s documentation notes that keys can be cleared automatically once they are at least 24 hours old. It has to outlive any retry that could plausibly arrive — slow client loops, queued jobs replayed after an outage, a human clicking submit again after lunch — without letting the table grow forever. Whatever you pick, document it, because the client’s maximum retry window has to fit inside your retention window or the guarantee silently evaporates at the edge.

Exactly-once does not exist. Effectively-once does.

Worth stating plainly, because “exactly-once delivery” is still marketed.

Over an unreliable network, a sender cannot distinguish a lost message from a lost acknowledgement. It must either not resend (at-most-once — messages can be lost) or resend (at-least-once — messages can be duplicated). There is no third option at the delivery layer. Any system claiming exactly-once delivery is doing deduplication somewhere and calling the combination by a shorter name.

What you can build is:

at-least-once delivery + idempotent processing = effectively-once side effects

The duplicates still arrive. You just make the second one a no-op. That is what an idempotency key is: the deduplication half of effectively-once, pushed to the boundary where the side effect happens.

This is why the pattern shows up everywhere once you know the shape. Kafka consumers commit offsets after processing and dedupe on a message key. Payment processors use client tokens. Webhook receivers dedupe on event ID — and any webhook you consume will be delivered twice eventually, because the sender’s ack got lost, so treating webhook handlers as idempotent is not optional. Terraform-style provisioning is idempotent by design: declare the desired state and converge, rather than issue a create.

Agent tool calls need this more than anything else

Everything above is forty years old. Here is what changed.

In a conventional system, the set of things that retry is finite and configured: your HTTP client, your SDK, your queue consumer. You can enumerate them. In an agentic system there is an additional retry layer that is discretionary and non-deterministic — the model sees a tool error in its context and decides, on its own, to call the tool again.

That decision is not governed by your retry config. It has no backoff, no attempt cap, and no knowledge that your HTTP client already retried three times. And critically, it often retries after an ambiguous timeout — the exact situation where retrying a non-idempotent operation is unsafe. An agent that times out calling create_incident and sees an error will very reasonably try again, and you get two incidents, two pages, two Slack threads.

It gets worse in a way that is specific to models: the retry may not be identical. A model that got an error often adjusts the arguments before trying again — a slightly different description, a reworded summary — reasoning that the first phrasing may have been the problem. So even naive body-hash deduplication on the server misses it. Two calls, different payloads, same intended effect, two side effects.

So: every agent tool with a side effect needs an idempotency key, generated by the harness, not the model.

def call_tool(tool_name: str, args: dict, *, task_id: str, step: int) -> dict:
    # Derived from the task and the step, NOT from args and NOT from the model.
    # A model that retries with reworded arguments still lands on the same key.
    idem_key = f"{task_id}:{step}:{tool_name}"

    return http.post(
        TOOL_ENDPOINTS[tool_name],
        json=args,
        headers={"Idempotency-Key": idem_key},
    )

Three properties follow from deriving the key that way, and all three matter:

• The model cannot influence it. It is not in the context window, so it cannot be reasoned about, changed, or hallucinated.
• Reworded retries still collapse. The key depends on the task and the step, so a semantically-equivalent retry with different phrasing maps to the same key. This is the behaviour body-hashing alone will not give you.
• It survives a session restart. An agent resumed from a checkpoint replays the same step with the same key and does not duplicate the work it already did — which makes checkpointing safe rather than merely possible.

The natural place for this is the MCP gateway, alongside the circuit breakers and rate limits. Keys minted there apply uniformly across every agent, every session, and every tool server, and they cannot be skipped by a tool implementation that forgot. Put it in the agent and you have as many policies as you have agents.

One design note that changes how you write tools: prefer declarative tool surfaces over imperative ones. ensure_incident(dedupe_key=...) is idempotent by construction; create_incident() needs a key bolted on to become safe. When you control the tool schema, the version that describes a desired end state is almost always the more robust thing to hand a model — for the same reason declarative infrastructure beat imperative provisioning scripts. A model calling ensure_* twice is harmless. A model calling create_* twice is an incident.

The rule worth remembering

If an operation has a side effect and can be retried, it needs an idempotency key. In an agentic system, assume everything can be retried.

The mechanism is three steps. The correctness is in four details: claim the key atomically before doing the work, store the complete response, fingerprint the request, and scope the key to the caller with a documented TTL. Get those right and a retry becomes what everyone always assumed it was — free.