Timeout / Ambiguous Outcome

"Timeout does not equal failure. Did it succeed?"

Applies To

external APIs, payment providers, databases, queues

Why It Happens

A first-principles walkthrough of why a timeout tells you nothing about success. TCP hid whether the bytes arrived, HTTP hid whether the handler ran. Every timeout is ambiguous by design, so we never treat it as failure: persist pending before calling, then reconcile with the provider as the source of truth, never blindly retry a non-idempotent charge.

How It Works Underneath

Timeout is a client-side decision to stop waiting. It tells you nothing about the server: the server may have succeeded a millisecond after you gave up, may be still running, or may never have received the request. This is the ambiguous outcome — the most dangerous failure class. TCP already hid whether the bytes arrived; HTTP hides whether the handler committed. Treating timeout as failure and blindly retrying a non-idempotent charge is how double debits happen.

The correct machinery is: never retry without an idempotency key, persist a pending state before calling, and on timeout do not guess — reconcile. The reconciler asks the provider “what is the truth for reference X?” and drives the local row to completed or failed. The provider, not the timeout, is the source of truth.

Common Misconceptions

“Timeout means it failed.” No — it means “I don’t know.” Another: “A longer timeout fixes it.” A longer wait just moves the ambiguity window, it does not remove it.

How to Detect It

Every outgoing HTTP call must have an explicit timeout=. Search for requests.post or httpx without timeout — that worker will hang forever and its pool will exhaust.

Cataloged Failure Modes

• ambiguous_success
• success_then_timeout
• retry_after_ambiguous

Code Comparison

# NAIVE: Treating timeout as failure
async def pay_invoice(invoice_id: str, amount: float):
    try: return await http_client.post("/charge", json={"amount": amount}, timeout=10.0)
    except TimeoutException: return await http_client.post("/charge", json={"amount": amount})

# IMPROVED: Handling timeout as AMBIGUOUS
async def pay_invoice(invoice_id: str, amount: float):
    op_id = f"inv_{invoice_id}_{uuid4()}"
    try:
        res = await http_client.post("/charge", json={"amount": amount}, headers={"Idempotency-Key": op_id}, timeout=10.0)
        return res.json()
    except (TimeoutException, httpx.NetworkError):
        status = await reconcile_with_provider(op_id)
        if status.is_confirmed: return status.data
        raise OperationPendingException("Payment pending confirmation")

// NAIVE: Treating timeout as failure
async function payInvoice(invoiceId, amount) {
  try { return await http.post('/charge', { amount }, { timeout: 10000 }); }
  catch (e) { if (e.code === 'ETIMEDOUT') return await http.post('/charge', { amount }); throw e; }
}

// IMPROVED: Handling timeout as AMBIGUOUS
async function payInvoice(invoiceId, amount) {
  const opId = `inv_${invoiceId}_${crypto.randomUUID()}`;
  try {
    const res = await http.post('/charge', { amount }, { headers: { 'Idempotency-Key': opId }, timeout: 10000 });
    return res.data;
  } catch (e) {
    if (e.code === 'ETIMEDOUT' || e.code === 'ECONNRESET') {
      const status = await reconcileWithProvider(opId);
      if (status.isConfirmed) return status.data;
      throw new OperationPendingError('Payment pending confirmation');
    }
    throw e;
  }
}

Mitigation Patterns

• idempotency-key
• reconciliation