Webhooks Explained: Receiving, Verifying, and Not Paying Twice

Every git-push deploy on PandaStack starts life as a GitHub webhook. Our billing runs on Stripe webhooks. I've debugged enough failed deliveries, signature mismatches, and double-processed events to have opinions about this — so here's the guide I wish I'd had: what webhooks actually are, how to receive them without getting burned, and how to make your processing safe against the retries that *will* happen.

What a webhook actually is

A webhook is just an HTTP POST that someone else's server makes to yours when something happens. Instead of you polling GET /events every 30 seconds, the provider calls you: Stripe POSTs when a payment succeeds, GitHub POSTs when someone pushes, your monitoring tool POSTs when a check fails.

That inversion is the whole trick, and it comes with three consequences people underestimate:

1. 1Your endpoint is public. Anyone who finds the URL can POST garbage — or forged events — at it.
2. 2Delivery is at-least-once, not exactly-once. Providers retry on timeouts and non-2xx responses. You *will* receive duplicates.
3. 3Order is not guaranteed. A retried invoice.created can arrive after invoice.paid. Never assume sequence.

Everything below follows from those three facts.

Receiving: acknowledge fast, work later

The provider's timeout is short — often single-digit seconds. If your handler does real work inline (call another API, run a build, send an email) and takes 12 seconds, the provider marks the delivery failed and retries, and now you're doing the work twice while your endpoint looks flaky in their dashboard.

The right shape is:

1. 1Verify the signature.
2. 2Persist or enqueue the event.
3. 3Return 2xx immediately.
4. 4Process asynchronously.

On our side, the GitHub push webhook does almost nothing synchronously — verify, record, hand off to the build pipeline, respond. The build takes minutes; the webhook response takes milliseconds. Those are different jobs and should never share a request lifecycle.

Verifying signatures (and the raw-body trap)

Since your endpoint is public, you need proof that a request actually came from the provider. The standard mechanism is an HMAC: the provider signs the request body with a shared secret and sends the signature in a header — X-Hub-Signature-256 for GitHub, Stripe-Signature for Stripe. You recompute the HMAC over the body you received and compare.

Here's the part that bites everyone at least once: the signature is computed over the raw bytes of the body. If your framework parses JSON before you verify — which express.json() as a global middleware happily does — you no longer have the raw bytes. Re-serializing the parsed object (JSON.stringify(req.body)) is not guaranteed to reproduce the original byte sequence: key order, whitespace, and unicode escaping can all differ. Your verification will fail intermittently and you will lose an afternoon to it.

The fix is to give webhook routes the raw body. In Express:

const express = require('express');
const crypto = require('crypto');

const app = express();

// Webhook route gets the RAW body — registered before any global json parser
app.post(
  '/webhooks/github',
  express.raw({ type: 'application/json' }),
  (req, res) => {
    const signature = req.get('X-Hub-Signature-256') || '';
    const expected =
      'sha256=' +
      crypto
        .createHmac('sha256', process.env.GITHUB_WEBHOOK_SECRET)
        .update(req.body) // req.body is a Buffer here
        .digest('hex');

    const valid =
      signature.length === expected.length &&
      crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));

    if (!valid) return res.status(401).send('invalid signature');

    const event = JSON.parse(req.body.toString('utf8'));
    enqueue(event); // persist for async processing
    res.status(202).send('accepted');
  }
);

// Normal routes can parse JSON as usual
app.use(express.json());

Two details worth calling out:

• crypto.timingSafeEqual, not ===. String comparison short-circuits on the first differing byte, which leaks timing information an attacker can exploit to forge signatures byte by byte. It's a niche attack, but the fix is one line.
• Replay protection. A valid signed request can be captured and replayed later. Stripe addresses this by including a timestamp in the signed payload and recommending you reject events outside a tolerance window — their [webhook docs](https://docs.stripe.com/webhooks) cover the scheme. GitHub's approach is documented in [their delivery validation guide](https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries). If your provider signs a timestamp, check it.

When we ported our Stripe and GitHub webhook handlers to a new backend, the raw-body mounting order was the single thing we tested most carefully — it's the classic silent breakage, because everything works until the first signature check fails in production.

Idempotency: surviving at-least-once delivery

Retries mean duplicates, so processing an event twice must be harmless. There are two complementary strategies; use both.

1. Deduplicate on event ID

Most providers include a unique ID per event (event.id in Stripe, X-GitHub-Delivery header for GitHub). Record it, and skip anything you've seen:

CREATE TABLE processed_events (
  event_id     text PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

const result = await db.query(
  'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING',
  [event.id]
);
if (result.rowCount === 0) {
  return; // already handled — duplicate delivery
}
await handle(event);

The subtle question is *when* to insert. Claim-first (as above) means a crash between claiming and handling drops the event. Handle-first means a crash between handling and recording double-processes it. If your handler's side effects live in the same database, do both in one transaction and the problem disappears. If the side effects are external (charging a card, triggering a build), claim-first plus a reconciliation job that re-checks unfinished work is the pragmatic answer.

2. Make the handler naturally idempotent

Dedup tables are a gate, not a guarantee — so write handlers where running twice produces the same state as running once:

• Upsert instead of insert. INSERT ... ON CONFLICT (id) DO UPDATE beats "insert and hope."
• Set state, don't increment it. UPDATE subscriptions SET status = 'active' is idempotent; UPDATE credits SET balance = balance + 100 is not. If you must increment, key the increment to the event ID.
• Treat the payload as a hint, not the truth. Because events arrive out of order, the safest pattern — Stripe explicitly recommends it — is to use the webhook as a signal to fetch the current object state from the API, then reconcile against that. The event tells you *something changed*; the API tells you *what's true now*.

Debugging deliveries

Three habits that save time:

• Log every delivery before you judge it — event ID, type, signature result — so "did the webhook even arrive?" is answerable from your logs, not the provider's dashboard.
• Use the provider's redelivery button. GitHub and Stripe both keep delivery history with request/response bodies and let you redeliver. It's the fastest reproduction tool you have — and it's also why idempotency matters, because *you* will be the source of many duplicates.
• For local development, tunnel. Tools like the Stripe CLI's stripe listen forward real events to localhost, which beats deploying to test a one-line handler change.

The checklist

Verify the HMAC over the raw body with a timing-safe compare. Reject stale timestamps if the provider signs them. Ack fast, process async. Dedupe on event ID. Write handlers that set state instead of mutating it. Fetch truth from the API when order matters.

Do those six things and webhooks become what they should be: boring plumbing. And if you'd like the receiving end handled for you — git-push deploys on PandaStack are exactly this machinery, wired up and running — you can try it at https://pandastack.io.