HIGH

Retry Safety

"Is it safe to retry? What retries are allowed?"

Invariant: Retries must be safe, bounded, with backoff and idempotency.

Applies To

external calls, webhooks, queue consumers

Why It Happens

A first-principles walkthrough of why retries are inevitable in any distributed system and why an unbounded retry loop is a distributed amplifier. How a tight loop on a 500 turns a single slow dependency into a thundering herd, and how exponential backoff with jitter, a retry budget, and the rule “only retry idempotent operations” make retries safe, bounded, and backpressure-aware.

How It Works Underneath

Retries are inevitable: the network drops, the provider 503s, the user double-clicks. An unbounded retry loop is a distributed amplifier. One slow dependency that returns 500 for 2 seconds, retried immediately by 100 clients, becomes 100× load at the exact moment the dependency is weakest. The fix is three parts: only retry what is idempotent, wait with exponential backoff and jitter so retries spread, and bound the total with a retry budget and a circuit breaker that trips to open after N failures and fails fast without hammering.

attempt 1 → 429 → wait 1s + jitter → attempt 2 → 429 → wait 2s → attempt 3 → 200
Without jitter: 100 clients retry at 1.0s, 2.0s — thundering herd. With jitter: retries scatter.

Common Misconceptions

“More retries = more reliability.” More retries without backoff is a self-inflicted DDoS. Another: “Retry on 400.” 400 is the client’s fault — retrying never fixes a bad request.

Cataloged Failure Modes

retry_storm
thundering_herd
duplicate_on_retry

Code Comparison

Language:

Fragile (AI Happy Path) — Python retry-safety_fragile.py

# NAIVE: Tight retry loop
async def send_sms(user_id: str, msg: str):
    for _ in range(10):
        try: return await sms.send(user_id, msg)
        except: time.sleep(0.1)
    raise FailedException()

Resilient (Failures Verified) — Python

retry-safety_safe.py

# IMPROVED: Exponential backoff with full jitter
async def send_sms(user_id: str, msg: str, key: str):
    for attempt in range(1, 5):
        try: return await sms.send(user_id, msg, idempotency_key=f"{key}_{attempt}")
        except RetriableException:
            delay = random.uniform(0, min(8.0, 0.5 * (2 ** (attempt - 1))))
            await asyncio.sleep(delay)
    raise FailedException()

Fragile (AI Happy Path) — TypeScript retry-safety_fragile.ts

// NAIVE: Tight retry loop
async function sendSms(userId, msg) {
  for (let i = 0; i < 10; i++) {
    try { return await sms.send(userId, msg); }
    catch (e) { await new Promise(r => setTimeout(r, 100)); }
  }
  throw new Error('Failed');
}

Resilient (Failures Verified) — TypeScript

retry-safety_safe.ts

// IMPROVED: Exponential backoff with full jitter
async function sendSms(userId, msg, key) {
  for (let attempt = 1; attempt <= 4; attempt++) {
    try { return await sms.send(userId, msg, { idempotencyKey: `${key}_${attempt}` }); }
    catch (e) {
      if (!isRetriable(e)) throw e;
      const delay = Math.random() * Math.min(8000, 500 * (2 ** (attempt - 1)));
      await new Promise(r => setTimeout(r, delay));
    }
  }
  throw new Error('Failed');
}

Mitigation Patterns

circuit-breaker