Idempotency

"What happens if the same operation happens twice?"

Applies To

payments, webhooks, queues, mutating POSTs, external APIs, enrollments

Why It Happens

A first-principles walkthrough of why the network makes every request potentially twice — timeout, retry, redelivery — what “the same operation” means as business policy (same key vs same key+amount), and how a persisted idempotency key with a UNIQUE constraint turns a non-idempotent POST into a safe retry. The key must be written before the side effect, not after.

How It Works Underneath

The network has amnesia: a client that times out after 5 seconds cannot know whether the server received the bytes, processed them, or never saw them. TCP hid the loss, HTTP hides the handler state. The only way to make “send again” safe is to give the *logical* operation a name that survives retries. That name is the idempotency key — a UUID generated by the client for one checkout attempt, not per HTTP attempt. The server stores key → result durably (database table with UNIQUE(key), not an in-memory map) before doing work, and on a repeat key returns the stored result without re-executing. The key turns a non-idempotent POST into “at most once.”

Analogy: a postal office that stamps every letter with a tracking number. If the same tracking number arrives twice, the office does not deliver two parcels — it returns the delivery receipt of the first.

Common Misconceptions

“The provider’s idempotency is enough.” Stripe’s key protects Stripe, not your local INSERT enrollment that follows it. You need a local key that guards the whole operation. Another: “A random UUID per retry is fine.” A new UUID per retry is a new logical operation — it guarantees duplication.

How to Detect It

Any mutating POST or queue consumer that can be retried and lacks a SELECT ... WHERE key = ? before the side effect is suspect. Ask: “If the client retries with the same key, do we have a UNIQUE to catch it and a cached response to return?”

Cataloged Failure Modes

• duplicate_side_effect
• duplicate_record
• double_charge

Code Comparison

# NAIVE: Retrying this endpoint charges the card multiple times
@app.post("/api/enroll")
async def enroll_student(payload: EnrollRequest):
    res = await payment_gateway.charge(payload.amount)
    await db.execute("INSERT INTO enrollments (student_id, course_id) VALUES (%s, %s)", payload.student_id, payload.course_id)
    return {"status": "enrolled"}

# IMPROVED: Idempotency table + unique constraint
@app.post("/api/enroll")
async def enroll_student(payload: EnrollRequest, idem_key: str = Header(None)):
    if not idem_key:
        raise HTTPException(400, "Idempotency-Key header required")
    existing = await db.fetch_one("SELECT response, status FROM idempotency_keys WHERE key = %s", idem_key)
    if existing:
        if existing['status'] == 'completed':
            return json.loads(existing['response'])
        elif existing['status'] == 'pending':
            raise HTTPException(409, "Operation currently processing in flight")
    await db.execute("INSERT INTO idempotency_keys (key, status) VALUES (%s, 'pending')", idem_key)
    try:
        charge = await payment_gateway.charge(payload.amount, idempotency_key=idem_key)
        await db.execute("INSERT INTO enrollments (student_id, course_id) VALUES (%s, %s) ON CONFLICT DO NOTHING", payload.student_id, payload.course_id)
        resp = {"status": "enrolled", "charge_id": charge.id}
        await db.execute("UPDATE idempotency_keys SET status = 'completed', response = %s WHERE key = %s", json.dumps(resp), idem_key)
        return resp
    except Exception as e:
        await db.execute("DELETE FROM idempotency_keys WHERE key = %s", idem_key)
        raise

// NAIVE: Retrying this endpoint charges the card multiple times
app.post('/api/enroll', async (req, res) => {
  const charge = await paymentGateway.charge(req.body.amount);
  await db.query('INSERT INTO enrollments (student_id, course_id) VALUES ($1,$2)', [req.body.studentId, req.body.courseId]);
  res.json({ status: 'enrolled' });
});

// IMPROVED: Idempotency table + unique constraint
app.post('/api/enroll', async (req, res) => {
  const key = req.headers['idempotency-key'];
  if (!key) return res.status(400).json({ error: 'Idempotency-Key required' });
  const existing = await db.query('SELECT response, status FROM idempotency_keys WHERE key = $1', [key]);
  if (existing.rows[0]) {
    if (existing.rows[0].status === 'completed') return res.json(JSON.parse(existing.rows[0].response));
    if (existing.rows[0].status === 'pending') return res.status(409).json({ error: 'Operation in flight' });
  }
  await db.query('INSERT INTO idempotency_keys (key, status) VALUES ($1, 'pending')', [key]);
  try {
    const charge = await paymentGateway.charge(req.body.amount, { idempotencyKey: key });
    await db.query('INSERT INTO enrollments (student_id, course_id) VALUES ($1,$2) ON CONFLICT DO NOTHING', [req.body.studentId, req.body.courseId]);
    const resp = { status: 'enrolled', chargeId: charge.id };
    await db.query('UPDATE idempotency_keys SET status = 'completed', response = $1 WHERE key = $2', [JSON.stringify(resp), key]);
    res.json(resp);
  } catch (e) {
    await db.query('DELETE FROM idempotency_keys WHERE key = $1', [key]);
    throw e;
  }
});

Mitigation Patterns

• idempotency-key