· 6 min read APIFundamentalsPlatform Engineering

Webhooks: Verify, Store, Say 200, Then Think

A webhook is a message queue whose producer belongs to someone else. The receiver pattern that survives retries, duplicates, replays and out-of-order events.

Webhook receiver flow: verify the signature and timestamp over the raw body, insert into an inbox table keyed by event ID, return 200, then a worker fetches current state and applies an idempotent effect; a periodic reconciliation sweep catches anything the sender gave up on

A webhook looks like an ordinary HTTP request, and that’s the trap. Operationally it’s a message queue where somebody else owns the producer. They pick the retry schedule and the timeout, and they have no idea your database is having a bad afternoon.

The usual webhook bug comes from treating the request as a function call: parse it, do the work, return 200. That works in the demo, and then the first slow query turns into a duplicate refund.

Here’s the shape that holds up. Four steps, in this order, and the order matters more than any of the individual steps.

What the sender promises (less than you think)

Read the delivery section of any serious provider’s docs and the same three properties keep coming up.

At least once, not exactly once. Stripe says endpoints “might occasionally receive the same event more than once.” A retry can land even when your first attempt worked, because the sender only knows what it saw: a timeout, a reset connection, a 502 from your load balancer.

No ordering. Stripe again: it “doesn’t guarantee the delivery of events in the order that they’re generated.” Create a subscription and you can get invoice.paid before customer.subscription.created.

Retries on their schedule, not yours. In live mode Stripe retries for up to three days with exponential backoff. The Standard Webhooks spec recommends a multi-day schedule with jitter too. GitHub sits at the other end: it wants a 2xx within 10 seconds, and its guidance is that you redeliver missed webhooks yourself once your server is back.

So you’re building a consumer for a queue that duplicates, reorders, and occasionally gives up. Design for exactly that.

The receiver, in four moves

  1. Verify the signature and timestamp against the raw body.
  2. Store the raw event, keyed by the sender’s event ID.
  3. Say 200.
  4. Then think. A worker processes the stored event asynchronously.

Here’s a complete receiver using only the Python standard library. It follows the Standard Webhooks scheme: headers webhook-id, webhook-timestamp and webhook-signature, with the signature computed over id.timestamp.body.

import base64, hashlib, hmac, json, sqlite3, time

TOLERANCE = 300  # seconds; Stripe's libraries default to 5 minutes


def sign(secret: bytes, msg_id: str, ts: int, body: bytes) -> str:
    signed = f"{msg_id}.{ts}.".encode() + body
    mac = hmac.new(secret, signed, hashlib.sha256).digest()
    return "v1," + base64.b64encode(mac).decode()


class Rejected(Exception):
    pass


def verify(secret: bytes, headers: dict, body: bytes, now: int) -> str:
    msg_id = headers["webhook-id"]
    ts = int(headers["webhook-timestamp"])
    if abs(now - ts) > TOLERANCE:
        raise Rejected(f"timestamp outside tolerance ({now - ts}s old)")
    expected = sign(secret, msg_id, ts, body)
    # The header can carry several space-separated signatures during a
    # secret rotation; accept if any of them matches.
    for candidate in headers["webhook-signature"].split():
        if hmac.compare_digest(candidate, expected):
            return msg_id
    raise Rejected("signature mismatch")


db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE inbox (id TEXT PRIMARY KEY, body BLOB, received INTEGER)")


def receive(secret: bytes, headers: dict, body: bytes, now: int) -> tuple[int, str]:
    try:
        msg_id = verify(secret, headers, body, now)
    except Rejected as e:
        return 400, str(e)
    # Persist first, process later. The primary key does the dedupe.
    cur = db.execute("INSERT OR IGNORE INTO inbox VALUES (?, ?, ?)", (msg_id, body, now))
    db.commit()
    return 200, "accepted" if cur.rowcount else "duplicate, ignored"


if __name__ == "__main__":
    secret = b"test-secret-not-a-real-one"
    now = int(time.time())
    body = json.dumps({"type": "invoice.paid", "invoice": "in_123"}).encode()
    h = {"webhook-id": "msg_2b7f", "webhook-timestamp": str(now)}
    h["webhook-signature"] = sign(secret, h["webhook-id"], now, body)

    print("1 first delivery  ", receive(secret, h, body, now))
    print("2 retry, same id  ", receive(secret, h, body, now + 40))
    print("3 tampered body   ", receive(secret, h, body.replace(b"in_123", b"in_999"), now))
    print("4 replay, 2h later", receive(secret, h, body, now + 7200))
    # During a secret rotation the sender signs with old and new secrets.
    h2 = {"webhook-id": "msg_2b80", "webhook-timestamp": str(now)}
    old_sig = sign(b"previous-secret", h2["webhook-id"], now, body)
    h2["webhook-signature"] = old_sig + " " + sign(secret, h2["webhook-id"], now, body)
    print("5 rotation, 2 sigs", receive(secret, h2, body, now))

Running it:

1 first delivery   (200, 'accepted')
2 retry, same id   (200, 'duplicate, ignored')
3 tampered body    (400, 'signature mismatch')
4 replay, 2h later (400, 'timestamp outside tolerance (7200s old)')
5 rotation, 2 sigs (200, 'accepted')

Case 2 is the one that matters most and gets the least attention. The duplicate gets a 200, not an error. If you return 409 for a duplicate, a lot of senders will treat that as a failure and keep retrying something you’ve already got.

Verification: the details that break it

Verify the raw bytes. This is the classic. Your framework parses the JSON, you re-serialize it to verify, and the signature fails because key order or whitespace changed. Stripe’s docs warn about it in a callout of its own. Grab the body before anything touches it.

Compare in constant time. hmac.compare_digest, not ==. A plain string comparison returns early at the first differing byte, and that timing difference leaks how much of a forged signature was right.

Sign the timestamp, check the window. A signature over the body alone proves who sent it, not when. Anyone who captures one valid delivery can replay it forever. Put the timestamp inside the signed content and reject anything outside a few minutes. Stripe’s libraries default to five, and its docs say outright that a tolerance of zero turns the check off.

GitHub’s X-Hub-Signature-256 is an HMAC over the body only, with no timestamp. Its docs suggest using the X-GitHub-Delivery ID to make sure each delivery is processed once, which covers you here as long as you actually store it.

Accept more than one signature. Rotating a signing secret means a window where the sender signs with both the old and new secrets. If your verifier only checks the first signature in the header, rotation is an outage. Case 5 above is that path.

Pin the scheme. Stripe tells you to ignore every scheme that isn’t v1, to block downgrade attacks. Generalize that: decide which algorithm you accept and refuse everything else, even if it’s well-formed.

Ordering: treat the payload as a hint

The payload tells you what the object looked like when the event was created. By the time you process it, that may be stale. If you apply payloads in arrival order, an old subscription.updated arriving late will overwrite the newer state.

Two fixes, and I’d reach for the first one by default.

Fetch current state. Treat the webhook as “something changed on sub_123,” then call the API for the current object. Stripe’s newer thin events are built for this: the payload carries a reference rather than a snapshot, and the SDK gives you fetch_related_object(). You pay one API call per event and in exchange you never apply stale state.

Compare versions. If fetching is too expensive, store a version, sequence or updated field and only apply an event if it’s newer than what you have. Stripe notes its snapshot events record created in seconds, so two events can share a timestamp. Don’t use it as a tiebreaker.

Deduping on event ID isn’t the end of idempotency either. Stripe mentions cases where two separate event objects are generated for the same change; for those it suggests deduping on the object ID plus the event type. The real fix is the same one from idempotency keys: make the effect itself safe to apply twice, so that “mark invoice paid” on an already-paid invoice is a no-op rather than a second ledger entry.

Why “say 200” comes before “think”

If the handler does the real work inline, latency goes straight into the sender’s timeout. A slow query means a timeout, the sender retries, and now two copies of the same event are being processed concurrently. Dedupe can’t save you if both copies got past the check before either one committed.

The inbox table breaks that loop. The insert is fast, the primary key serializes duplicates, and the 200 goes out in milliseconds. The worker reading the inbox can take as long as it needs, retry with backoff, and park poison messages without the sender ever knowing.

It also helps with the recovery nobody plans for. When your endpoint comes back after a long outage, every sender drains its retry backlog at once. Rate-limit the worker, not the receiver. Let the inbox absorb the burst, which is what a queue is for.

Webhooks are a hint, so reconcile

Senders give up eventually. Stripe stops after three days, and some providers disable an endpoint after enough consecutive failures. GitHub, by its own docs, leaves redelivery to you. If you were down past the retry window, those events are gone.

So run a sweep. Every so often, list recently changed objects through the provider’s API and compare them with what you have. Anything missing gets enqueued exactly as if a webhook had arrived. That turns webhooks from the source of truth into a latency optimization on top of a reconciliation loop, which is the property you want.

If you’re the one sending them

Everything above flips into obligations.

  • Give every event a stable ID that stays the same across retries, and sign the ID, the timestamp and the body.
  • Use a queue per endpoint. One customer’s endpoint timing out for an hour shouldn’t delay anyone else’s deliveries.
  • Back off exponentially with jitter, and tell the customer before you disable their endpoint.
  • Support two active secrets during rotation.
  • Treat customer-supplied URLs as hostile. A webhook URL is a request your servers make to an address someone else chose. Resolve the hostname and refuse private, loopback and link-local ranges (yes, including 169.254.169.254) at connection time, not only when the URL is saved, or DNS rebinding walks around your validation. It’s the same problem as giving an agent an HTTP tool.

The short version

Verify against raw bytes with a timestamp in the signature. Insert into an inbox keyed by event ID and return 200, duplicates included. Process from the inbox, fetch current state instead of trusting the payload, and make the effect idempotent. Then run a reconciliation sweep, because the sender will eventually give up on you.

None of this is exotic. It’s just treating the thing as the distributed system it is.

Related: Idempotency Keys: The Thing That Makes a Retry Safe · Queues and Message Brokers · Every HTTP Tool You Give an Agent Is an SSRF Primitive

Frequently asked questions

Are webhooks delivered exactly once?

No. Webhook senders deliver at least once: if your endpoint times out or returns a non-2xx status, the sender retries, and a retry can arrive even when your first attempt actually succeeded. Stripe states that endpoints might occasionally receive the same event more than once and recommends logging processed event IDs. Deduplicate on the event ID the sender provides, and make the business effect idempotent as well.

How do you verify a webhook signature?

Recompute an HMAC-SHA256 over the exact raw request body (plus the timestamp and message ID, if the sender's scheme includes them) using the shared signing secret, then compare it to the signature header with a constant-time comparison. Reject the request if the timestamp is outside a tolerance window, which blocks replays. Never verify against a re-serialized JSON object, because any change in whitespace or key order breaks the signature.

Why should a webhook handler return 200 before processing?

Senders enforce short timeouts. GitHub expects a 2xx response within 10 seconds and counts slower responses as failures. If your handler does real work before responding, a slow database call turns into a timeout, the sender retries, and you end up processing the same event twice, often concurrently. Verify, persist the raw event, acknowledge, and process it asynchronously from your own queue.

Do webhooks arrive in order?

Not reliably. Stripe explicitly does not guarantee delivery in the order events were generated. Treat a webhook as a notification that an object changed, then fetch the object's current state from the API, or compare a version or updated-at field before applying the payload, so an older event can never overwrite newer state.

Comments