Idempotency Keys: What Stripe and Shopify Actually Do

I charged a customer $392.00 for a $49.00 course. Three times in a row, deliberately, against a real PostgreSQL 17.6 instance.

The handler was the one you find in most tutorials: check whether you've seen this idempotency key, and if you haven't, do the work. Eight concurrent workers, one shared key, each on its own connection, all released from a barrier at the same instant.

--- naive check-then-insert, NO unique index, 8 workers, same key, READ COMMITTED ---
  trial 1: charges=8 ($392.00) outcomes={"ok:charged":8}
  trial 2: charges=8 ($392.00) outcomes={"ok:charged":8}
  trial 3: charges=8 ($392.00) outcomes={"ok:charged":8}

Not a flaky one-in-a-thousand interleaving. Eight out of eight, every time. Under READ COMMITTED every worker's SELECT runs against a snapshot taken before anyone's INSERT committed, so all eight see nothing and all eight proceed. The window isn't narrow — it's the entire duration of your handler.

That's the thing worth internalising before any of the design discussion: check-then-act on a table with no constraint behind it doesn't fail rarely. Under real concurrency it fails reliably.

FOR UPDATE does nothing here, and this is the part that bites

The usual reflex is row locking. Add FOR UPDATE to the SELECT, take the lock, race solved.

I ran exactly that — same eight workers, same key, SELECT ... FOR UPDATE:

SELECT ... FOR UPDATE on a row that does not exist yet:
  outcomes {"ok:charged":8}  charges=8  total=$392.00
 
Same code, but the row already exists:
  outcomes {"ok:replayed":8}  charges=1  total=$49.00

FOR UPDATE locks rows the query returned. The first request for a given key returns zero rows, so it locks nothing, and eight workers acquire eight empty lock sets and cheerfully carry on. The lock works perfectly on the second request — and the second request was never the problem.

Isolation level: only one of the three helps

Same naive handler, same eight workers, only the isolation level changed:

REPEATABLE READ is the surprise for most people. It gives you a stable snapshot, and Postgres aborts write-write conflicts on the same row — but eight INSERTs create eight different rows, so there is no conflict to detect. A stable snapshot of "no such key" is exactly the wrong answer, held more firmly.

SERIALIZABLE does work. Postgres takes predicate locks over the index range your SELECT scanned and raises 40001 (serialization_failure) when a concurrent insert falsifies it. That's a genuine fix, and it costs you a retry loop on every caller plus SSI's tracking overhead on every transaction in the database, not just this one. It's the right tool if the invariant spans several tables. For "one key, one row" there's something cheaper that fails less often.

Let the unique index be the referee

Stop asking whether the key exists. Insert it and let the database tell you.

Here is the ledger of keys, plus the table every headline number in this post has been counting rows in, so that the rest of this is runnable rather than illustrative:

CREATE TABLE idempotency_keys (
  key         text PRIMARY KEY,
  endpoint    text        NOT NULL,
  request_fp  text        NOT NULL,
  status      text        NOT NULL CHECK (status IN ('in_progress','completed')),
  response    jsonb,
  created_at  timestamptz NOT NULL DEFAULT now()
);
 
CREATE TABLE charges (
  id              bigserial   PRIMARY KEY,
  user_id         text        NOT NULL,
  amount_cents    integer     NOT NULL,
  idempotency_key text        NOT NULL,
  created_at      timestamptz NOT NULL DEFAULT now()
);

Note what charges does not have: a unique index on idempotency_key. That is deliberate. Every count in this post is measuring what the handler does, not what a second constraint would have rescued it from, and adding one here would hide the naive version's eight charges behind a 23505. In production, put one on. Belt and braces beats belt.

INSERT ... ON CONFLICT (key) DO NOTHING RETURNING key collapses the check and the claim into one statement. One row back means you own this request. Zero rows back means someone else does — and because the conflicting row is uncommitted, your statement blocks until the winner commits or rolls back, which is precisely the wait you wanted from a lock.

import { createHash } from "node:crypto";
import type { Pool, PoolClient } from "pg";
 
const fingerprint = (endpoint: string, body: unknown) =>
  createHash("sha256").update(`${endpoint} ${JSON.stringify(body)}`).digest("hex");
 
async function withTransaction<T>(
  pool: Pool,
  fn: (tx: PoolClient) => Promise<T>
): Promise<T> {
  const tx = await pool.connect();
  try {
    await tx.query("BEGIN");
    const out = await fn(tx);
    await tx.query("COMMIT");
    return out;
  } catch (err) {
    await tx.query("ROLLBACK").catch(() => {});
    throw err;
  } finally {
    tx.release();
  }
}
 
async function purchase(
  pool: Pool,
  req: { key: string; endpoint: string; body: { userId: string; amountCents: number } }
) {
  const fp = fingerprint(req.endpoint, req.body);
 
  return withTransaction(pool, async (tx) => {
    // Three attempts, because losing the claim and then finding no row is a
    // real state, not an impossible one. See "The cleanup path that never runs".
    for (let attempt = 0; attempt < 3; attempt++) {
      const claim = await tx.query(
        `INSERT INTO idempotency_keys (key, endpoint, request_fp, status)
         VALUES ($1, $2, $3, 'in_progress')
         ON CONFLICT (key) DO NOTHING
         RETURNING key`,
        [req.key, req.endpoint, fp]
      );
 
      if (claim.rowCount === 0) {
        const prior = (await tx.query(
          `SELECT endpoint, request_fp, status, response
             FROM idempotency_keys WHERE key = $1`,
          [req.key]
        )).rows[0];
 
        if (prior === undefined) continue;   // swept between the INSERT and here
        if (prior.endpoint !== req.endpoint || prior.request_fp !== fp) {
          return { status: 422, body: { error: "idempotency_key_reuse" } };
        }
        if (prior.status === "completed") {
          // node-postgres already parsed the jsonb column. Do not JSON.parse it.
          return { status: 200, body: prior.response, replayed: true };
        }
        return { status: 409, body: { error: "request_in_progress" } };
      }
 
      const charged = await tx.query(
        `INSERT INTO charges (user_id, amount_cents, idempotency_key)
         VALUES ($1, $2, $3) RETURNING id`,
        [req.body.userId, req.body.amountCents, req.key]
      );
      // `id` is a string here, not a number: see below.
      const response = { chargeId: charged.rows[0].id };
 
      await tx.query(
        `UPDATE idempotency_keys SET status = 'completed', response = $1 WHERE key = $2`,
        [JSON.stringify(response), req.key]
      );
      return { status: 200, body: response, replayed: false };
    }
    return { status: 409, body: { error: "claim_contended" } };
  });
}

That loop is not a serialization-failure retry — there is no 40001 to catch at READ COMMITTED. It exists because the losing branch reads a row it did not write, and a row you did not write can be gone by the time you read it. Further down this post there is a cleanup recommendation that makes exactly that happen; without the prior === undefined line, the handler dies on prior.endpoint with a TypeError. I ran both versions against a claim row deleted between the failed INSERT and the SELECT:

unguarded : THREW TypeError: Cannot read properties of undefined (reading 'endpoint')
with retry: claimed on attempt 2

Falling through all three attempts returns 409, which is the fail-closed answer: the caller retries, nobody is charged twice, and nothing crashes.

Eight concurrent workers through that exact function, READ COMMITTED:

outcomes: { '200': 1, '200 replayed': 7 }
charges=1  total=$49.00

One small thing about the response body you just stored. RETURNING id on a bigserial column comes back from node-postgres as the string "1", not the number 1 — int8 is 64 bits and a JS number is not, so the driver refuses to lose precision on your behalf and hands you text. The first request writes {"chargeId":"1"} into the response column; every replay hands the client back that same string. It only bites you when the winner and the replays disagree, which happens the moment someone "helpfully" coerces the fresh path to a number and leaves the replay path alone. Then the same key returns 1 once and "1" seven times, and a strict client comparison starts failing on the retries only.

Compare that with what the naive-plus-primary-key version does. If you keep the SELECT-first shape but do have a PRIMARY KEY on the table, the constraint saves your money — and seven of eight callers get a raw 23505 unique-violation exception, which almost always becomes a 500. The database prevented the double charge; your handler still told seven customers the purchase failed. Catching the conflict and replaying the stored response is the step that turns a constraint violation into a correct answer, and it is the step that gets skipped.

Same key, different body

This is where implementations quietly diverge, and where a table of just (key, response, status) becomes dangerous.

A client bug reuses one key across two different requests. With no request fingerprint stored, the second one hits the completed branch and replays. I ran it:

req1 $49.00   -> charged
req2 $9000.00 same key -> replayed:1
charges=1 total=$49.00  -- the $9000 order silently became the $49 one

A $9,000 request returned 200 OK and a confirmation pointing at a $49 charge. Nobody errored. Nobody logged anything. Reverse the order and you have a customer holding a $49 receipt for a $9,000 charge.

Stripe's docs are unambiguous about what should happen instead:

The idempotency layer compares incoming parameters to those of the original request and errors if they're not the same to prevent accidental misuse.

And their error taxonomy names it:

Idempotency errors occur when an Idempotency-Key is re-used on a request that does not match the first request's API endpoint and parameters.

Note endpoint and parameters — the scope is both. Storing sha256(endpoint + body) alongside the key and returning 422 on mismatch is about six lines, and it converts a silent wrong answer into a loud one. In my run it caught both the changed amount and the same key sent to POST /v1/refund.

Stripe's documentation, read closely

Most of what gets written about Stripe's implementation is a paraphrase of a paraphrase. Here is what the documentation says, and where it disagrees with the folklore.

Failures are cached too. Not just successes:

Stripe's idempotency works by saving the resulting status code and body of the first request made for any given idempotency key, regardless of whether it succeeds or fails. Subsequent requests with the same key return the same result, including 500 errors.

So retrying a failed call with the same key gets you the same failure back, forever. Their guidance for 4xx is to generate a fresh key when you change the request.

Keys are accepted, not required. The API reference says "All POST requests accept idempotency keys" and recommends using them on every POST. There's no endpoint that rejects you for omitting one.

Retention is 24 hours, phrased two different ways. The API reference: "You can remove keys from the system automatically after they're at least 24 hours old. We generate a new request if a key is reused after the original is pruned." The low-level error page — docs.stripe.com/error-low-level, not the general error-handling guide — is blunter: "keys expire out of the system after 24 hours". Either way, a key older than a day is not a duplicate any more — it's a new charge. If your client retries from a durable queue that can lag more than 24 hours, Stripe's idempotency will not save you.

One detail worth wiring into your own API: Stripe marks replays with a response header, Idempotent-Replayed: true. Costs nothing and makes client-side debugging enormously easier. Their status table also reserves 409 Conflict for "the request conflicts with another request (perhaps due to using the same idempotent key)" — the concurrent case, distinct from the mismatched-parameters case.

What Shopify built

Shopify's Payment Service post from 2019 is the best public write-up of this, and the most-repeated claim about it is wrong. There is no client-supplied X-Request-Id header. They deliberately avoided headers entirely:

The idempotency key is a 'first class citizen' of the API, we're not using an HTTP header for middleware. This allows us to require the presence of the idempotency key using the same GraphQL parameter validation as the rest of the API.

Their key is a GraphQL mutation input, which means it validates like everything else. There is more in that post worth stealing.

Their uniqueness scope is wider than the key alone — an IncomingRequest model where "each model instance is uniquely identified by the client and idempotency key." Two tenants can't collide, and a stolen key can't be used to read back another client's response.

And they solved the hard half — the side effect that isn't a database write. Each mutation is split into recovery points, classified by side-effect type: no side effects, local side effects (wrapped in a transaction), and remote side effects (calls to payment providers). The completed step name is stored on the row, so a retry resumes from where it died instead of guessing. For simultaneous duplicates they take a lock and "reject the request with an HTTP code of 409, meaning that the client may try again shortly."

They're also honest about the bill: "Storing the progress of a request requires extra database writes, this will add overhead to every API call."

The bug that survives all of the above

Every implementation above assumes BEGIN and COMMIT land on the same connection. In Node with pg, that is not automatic, and the failure is ugly.

pool.query("BEGIN") acquires a connection, runs BEGIN, and releases the connection back to the pool while the transaction is still open. Any other request can now be handed that dirty connection. I ran two concurrent handlers — A doing the article-shaped pool.query("BEGIN") ... pool.query("ROLLBACK"), and B an unrelated insert:

pool max=1  statement order: A:BEGIN B:INSERT A:INSERT A:ROLLBACK
  surviving rows: []          -> B's row was destroyed by A's ROLLBACK
 
pool max=4  statement order: A:BEGIN B:INSERT A:INSERT A:ROLLBACK
  surviving rows: [B,A]       -> A's row survived its own ROLLBACK

Both outcomes from the same code, decided by pool size and timing. An unrelated request's write got rolled back by someone else's error path; and in the wider pool, the "transaction" didn't hold at all, because the INSERT ran in autocommit on a different backend from the BEGIN. Any FOR UPDATE in that handler is also meaningless — the lock is released the moment the statement's connection goes back to the pool.

Always check out a client (pool.connect()), run the whole transaction on it, and release() in a finally. That's what withTransaction above exists for.

The cleanup path that never runs

One more, because it turns a transient failure into a permanently stuck key. The common error handler looks like this:

// Broken. Do not copy.
async function brokenErrorPath(tx: PoolClient, key: string, work: () => Promise<void>) {
  try {
    await work();
  } catch (err) {
    await tx.query("UPDATE idempotency_keys SET status = 'failed' WHERE key = $1", [key]);
    await tx.query("ROLLBACK");
    throw err;
  }
}

What that error path actually does, measured:

side effect threw:               SQLSTATE 23505
the cleanup UPDATE ->            SQLSTATE 25P02: current transaction is aborted,
                                 commands ignored until end of transaction block
status after the "cleanup":      in_progress

Postgres poisons the whole transaction after any error, so the UPDATE never executes (25P02). On a healthy transaction it would not have executed either — failed is not one of the two values the CHECK constraint on that table allows, so you would get 23514 instead. And even if the write had landed, the ROLLBACK on the next line would have undone it. Three separate reasons the same line does nothing. The row stays in_progress forever, every subsequent retry hits the "someone else is working on this" branch, and that key is dead — the customer can never complete that purchase.

Mark failure outside the failed transaction, on a fresh one. Better still, don't store failed at all: delete the claim row so a retry can take it cleanly, and let a created_at sweeper reclaim rows whose process died without deleting anything.

That recommendation is where the prior === undefined line in purchase came from. A sweeper is, by construction, a thing that makes rows vanish between one of your statements and the next, and the loser's branch is reading a row it did not write. You do not get to adopt the cleanup and skip the guard. The two live in different files and land in different pull requests, which is exactly why the pairing is easy to miss.

Webhooks are a different ledger

Client-facing idempotency keys and inbound webhooks are different problems and want different keys.

For webhooks, the provider's event id is the key, not the payment id — a single payment arrives as several distinct events (checkout.session.completed, then invoice.paid), and keying on the payment either drops legitimate follow-ups or lets a redelivery double-fulfil. Claim the event id in a ledger row, then do the work in one transaction.

Underneath that, put a unique index on the provider's payment identifier. Belt and braces: the ledger short-circuits the common redelivery cheaply, and the unique index is what actually holds when two deliveries interleave inside the same millisecond. If the ledger claim is the only thing standing between you and a duplicate charge, you're relying on a SELECT, and you already know how that ends.

The subtlety people get wrong is what to return when you lose the claim to an in-flight delivery. Answering 200 tells the provider the event was handled — and if the holder of that claim died mid-request, nobody will ever process it. Money in, nothing delivered, no retry coming. Return 409 and let the provider retry into a claim that has since gone stale.

$392.00

Go back to the number at the top. Eight workers, one key, $392.00 charged for a $49.00 course, three trials out of three, no flakiness anywhere in it. Everything else in this post is a footnote to how that happened.

It happened because a SELECT was asked a question it cannot answer. "Has anyone claimed this key?" has no true answer at the moment you ask it, only an answer that was true a moment ago, and under concurrency those are different things. FOR UPDATE does not help, because there is no row to lock. REPEATABLE READ does not help, because it makes the stale answer more stable rather than less. SERIALIZABLE does help, by making the database re-litigate the question at commit time and throw 40001 at seven of your eight callers. And a unique index helps by never letting the question get asked: INSERT ... ON CONFLICT (key) DO NOTHING RETURNING key decides and claims in one statement, and the row count is the answer.

The rest is bookkeeping, and the bookkeeping is where the money goes missing. Leave out the sha256(endpoint + body) beside the key and a client bug replays a $9,000 order as a $49 one, silently, with a 200. Leave out the stored response body and seven of your eight callers get a raw 23505 and tell the customer the purchase failed. Run the transaction on the pool instead of a checked-out client and your open transaction goes to somebody else's request. Mark a failure inside the transaction that already failed and the row sits at in_progress forever. Inherit Stripe's 24 hours by accident and keys expire on a clock nobody in your building chose. Omit Idempotent-Replayed: true and you debug replays blind. Not one of those raises an error rate.