Webhooks, Idempotency & Retries

Concept#

The beginner framing: a webhook is the REVERSE of a normal API call, instead of your app asking a service "did anything happen?", the service proactively POSTs to a URL YOU provide the moment something happens (a payment succeeded, a file finished processing). This inverts the usual client-server relationship, and inverts its trust assumptions right along with it.

HMAC signature verification, confirmed with a real sign/tamper/verify run#

const crypto = require("crypto");
 
function signPayload(payload, secret) {
  return crypto.createHmac("sha256", secret).update(payload).digest("hex");
}
 
const secret = "whsec_test123";
const payload = JSON.stringify({ event: "payment.succeeded", amount: 5000 });
const signature = signPayload(payload, secret);

Confirmed by actually running this this session: signing the same payload with the same secret twice produces the SAME signature every time (deterministic); tampering with even one field (amount: 5000 → amount: 999999) and recomputing produces a completely different signature, confirmed directly, crypto.timingSafeEqual correctly returns false when comparing the original signature against what the tampered payload's signature should have been. This is the entire trust mechanism: a webhook RECEIVER has no other way to know a POST actually came from the claimed sender rather than an attacker who simply learned or guessed the webhook URL (URLs aren't secret, they can leak via logs, browser history, referrer headers, or just be enumerable) and is sending fabricated event payloads. Verification requires the receiver to recompute the signature over the RAW received payload using the shared secret, and compare it (with a timing-safe comparison, per XSS's and CSRF's established discipline on this) against the signature the sender included in a header.

At-least-once delivery, the guarantee that makes idempotency non-negotiable#

Webhook sender's guarantee: "I will deliver this event AT LEAST ONCE"
NOT: "I will deliver this event EXACTLY once"

Nearly every production webhook system (Stripe, GitHub, Twilio) explicitly documents at-least-once delivery, not exactly-once, if the receiver's endpoint times out, returns a 5xx, or the sender simply can't confirm receipt for any reason (a dropped connection, an ambiguous network failure), the sender WILL retry, and the receiver may genuinely process the same event twice. This isn't a bug or an edge case to handle defensively "just in case", it's the documented, expected, designed behavior of the system. A webhook handler that isn't idempotent, say, one that credits a user's account balance every time it receives a payment.succeeded event, with no deduplication, WILL eventually double-credit a real payment, not as a rare failure mode but as a predictable consequence of the delivery model itself.

Idempotency keys, the receiver-side mechanism that makes retries safe#

async function handlePaymentWebhook(event) {
  const eventId = event.id; // the sender's own unique event identifier
 
  const alreadyProcessed = await db.processedEvents.findOne({ eventId });
  if (alreadyProcessed) {
    return { status: 200 }; // acknowledge again, but do NOT reprocess
  }
 
  await db.processedEvents.insert({ eventId, processedAt: new Date() });
  await creditUserBalance(event.data.userId, event.data.amount); // the ACTUAL side effect
}

The receiver tracks which event IDs it has already fully processed, and on a duplicate delivery of the same eventId, acknowledges success (still returning 200, this is important, since NOT doing so would make the sender think delivery is still failing and keep retrying forever) WITHOUT re-running the actual side effect. This is what genuinely closes the gap the at-least-once guarantee opens: the sender's retry behavior stays unpredictable and out of the receiver's control, but the receiver's OWN processing becomes safe under duplication regardless.

Try It#

Predict the outcome before checking the solution.

// A webhook handler that DOES check for duplicate events, but in the WRONG order:
async function handlePaymentWebhook(event) {
  await creditUserBalance(event.data.userId, event.data.amount); // side effect FIRST
 
  const alreadyProcessed = await db.processedEvents.findOne({ eventId: event.id });
  if (!alreadyProcessed) {
    await db.processedEvents.insert({ eventId: event.id }); // dedup record written AFTER
  }
}

Is this handler actually safe against duplicate deliveries, despite having deduplication logic present?