Idempotency Keys: Making Retries Safe in APIs and Jobs

A client sends a payment request. The server processes it, charges the card, writes the row — and the connection drops before the response makes it back. The client sees a timeout. Did the request fail before the work, or after?

The client cannot know. That's the whole problem. And because it cannot know, any sane client retries — which means your API will receive the same logical request twice, and if you process it twice, someone gets charged twice. We learned this the expensive way when a retried billing job on our own platform double-invoiced usage. Nothing teaches you idempotency faster than refunding people.

An operation is idempotent when running it N times has the same effect as running it once. Retries become safe instead of dangerous. Here's how to build that.

Some operations are naturally idempotent — design toward those

Before reaching for infrastructure, check whether the operation can be idempotent by construction:

• PUT /users/42 {"email": "a@b.com"} — absolute state. Run it ten times, same result.
• DELETE /sessions/abc — deleting something already deleted is a no-op (return 404 or 204, just be consistent).
• UPDATE accounts SET balance = 500 — absolute write, idempotent.
• UPDATE accounts SET balance = balance - 50 — relative write, not idempotent. Run it twice and you've deducted 100.

The pattern: prefer absolute state over increments, and "ensure X exists" over "create X". A surprising amount of API surface can be redesigned this way — INSERT ... ON CONFLICT DO NOTHING against a natural unique key is the cheapest idempotency you'll ever get.

But some operations are irreducibly non-idempotent: charging a card, sending an email, creating an order where two identical orders are legitimately different things. For those you need idempotency keys.

Idempotency keys: the contract

The contract, popularized by Stripe's API, is simple:

1. 1The client generates a unique key (a UUIDv4 is fine) per *logical* operation and sends it in a header: Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324.
2. 2If the client retries — timeout, 500, network blip — it reuses the same key.
3. 3The server executes the operation once, records the result under that key, and replays the recorded response for any subsequent request with the same key.

The critical detail is on the client side: a retry reuses the key; a *new* operation gets a new key. If the user clicks "Pay" twice on purpose, that's two keys; if your HTTP library retries a timeout, that's one key. Generate it when the user initiates the action, not per HTTP attempt.

A correct server implementation on Postgres

Most broken implementations fail in the same place: they check for the key and then insert it, as two separate steps. Two concurrent requests both pass the check, both execute the charge. The fix is to make claiming the key a single atomic write and let the database's unique constraint arbitrate.

The table:

CREATE TABLE idempotency_keys (
  key            text NOT NULL,
  scope          text NOT NULL,   -- endpoint + caller, see below
  request_hash   text NOT NULL,   -- sha256 of the request body
  status         text NOT NULL DEFAULT 'in_progress',
  response_code  int,
  response_body  jsonb,
  created_at     timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (key, scope)
);

The handler wrapper (Node + pg, but the shape is language-agnostic):

async function withIdempotency(req, res, handler) {
  const key = req.header('Idempotency-Key');
  if (!key) {
    return res.status(400).json({ error: 'Idempotency-Key header required' });
  }

  const scope = `${req.method} ${req.route.path}:${req.user.id}`;
  const bodyHash = sha256(JSON.stringify(req.body));

  // Atomically claim the key. Exactly one concurrent request wins.
  const claimed = await db.query(
    `INSERT INTO idempotency_keys (key, scope, request_hash)
     VALUES ($1, $2, $3)
     ON CONFLICT (key, scope) DO NOTHING
     RETURNING key`,
    [key, scope, bodyHash]
  );

  if (claimed.rowCount === 0) {
    // Someone already claimed it — replay or reject.
    const { rows: [prior] } = await db.query(
      `SELECT status, request_hash, response_code, response_body
       FROM idempotency_keys WHERE key = $1 AND scope = $2`,
      [key, scope]
    );
    if (prior.request_hash !== bodyHash) {
      return res.status(422).json({
        error: 'Idempotency-Key reused with a different request body',
      });
    }
    if (prior.status === 'in_progress') {
      // Original request still running — tell the client to back off.
      return res.status(409).json({ error: 'Request already in progress' });
    }
    return res.status(prior.response_code).json(prior.response_body);
  }

  // We won the claim: do the real work, then record the outcome.
  const { code, body } = await handler(req);
  await db.query(
    `UPDATE idempotency_keys
     SET status = 'completed', response_code = $3, response_body = $4
     WHERE key = $1 AND scope = $2`,
    [key, scope, code, body]
  );
  return res.status(code).json(body);
}

Details that matter:

Scope the key. A raw UUID as the sole primary key means user A's key can collide with user B's, and a key used on /payments blocks the same key on /refunds. Scope to endpoint plus caller.

Hash the request body. The same key with a different payload is a client bug, not a retry. Reject it with a 422 instead of silently replaying a response for a request the client didn't make. Stripe does exactly this.

Decide what to do with in_progress. The 409 above is the simple answer. If the original request crashed mid-flight, that row is stuck — so add a recovery path: rows older than your request timeout plus margin get reclaimed or expired. Without this, one crash poisons the key forever.

Decide whether errors are replayable. Replaying a recorded 400 is right — the request was bad and will stay bad. Replaying a 500 is debatable; if the failure was transient, the client can never succeed with that key. A reasonable policy: persist 2xx and 4xx outcomes, delete the key row on 5xx so a retry re-executes.

Expire old keys. Keys only need to live as long as a client might plausibly retry — 24 hours covers almost everything. A nightly DELETE FROM idempotency_keys WHERE created_at < now() - interval '24 hours' keeps the table small. And don't put this table in Redis without persistence: lose the keyspace, lose your dedup guarantee at the worst possible moment.

Background jobs: the same problem, delivered differently

Every mainstream queue — RabbitMQ, Pub/Sub, SQS — gives you *at-least-once* delivery. Not exactly-once. Redeliveries happen on consumer crashes, ack timeouts, and network partitions, and they happen most under load — exactly when duplicates hurt most. The queue will not save you; the consumer has to be idempotent.

The consumer-side pattern is the same claim trick, keyed on message ID:

INSERT INTO processed_messages (message_id)
VALUES ($1)
ON CONFLICT DO NOTHING;

If that insert affects zero rows, you've seen this message — ack it and move on. Crucially, do the insert in the same database transaction as the job's side effects. If you mark the message processed and then crash before doing the work, the work never happens; if you do the work and crash before marking, it happens twice. One transaction, both writes, no gap.

Side effects that live outside your database — sending an email, calling a third-party API — can't join that transaction. For those, pass the idempotency downstream: most serious providers accept an idempotency key of their own, so derive one from your message ID and let them dedupe.

Cronjobs retry in disguise: a scheduler reruns a missed window, a deploy restarts a job mid-run, a slow run overlaps the next tick. Write every scheduled task assuming it will sometimes run twice for the same period — key the work on the period (billing:2026-07) with a unique constraint, and the second run becomes a no-op instead of a double charge.

The checklist

• Prefer naturally idempotent designs: absolute writes, upserts, unique constraints.
• For everything else: client-generated key, reused across retries only.
• Claim keys atomically with INSERT ... ON CONFLICT DO NOTHING — never check-then-insert.
• Scope keys to endpoint + caller; hash the body and reject mismatches.
• Handle stuck in_progress rows and expire old keys.
• Consumers dedupe by message ID, transactionally with their side effects.
• Scheduled work is keyed on the period it covers.

None of this needs exotic infrastructure — a Postgres unique constraint does the heavy lifting in every pattern above. On PandaStack, the managed Postgres auto-wired into your app via DATABASE_URL is the same database you'd build this on, with cronjobs running alongside your services, so the dedup table lives one transaction away from the work it protects. If you're setting up a stack like this, it's easy to try at https://pandastack.io.