Ordering

"What if events arrive out of order?"

Applies To

queues, webhooks, event streams, enrollments

Why It Happens

A first-principles walkthrough of why queues and webhooks cannot promise order. A delayed “created” delivery can arrive after “cancelled” and wrongly re-activate a subscription. Why sequence numbers and monotonic timestamp guards (WHERE last_ts < incoming) make stale deliveries harmless without global ordering.

How It Works Underneath

Queues and webhooks promise at-least-once, not in-order. A “subscription cancelled” event that is retried can arrive after a “renewed” event and wrongly re-activate a user. The fix is not to force global order — that does not scale — but to make the consumer monotonic: store last_ts per entity and only apply an event if incoming.ts > last_ts. Stale deliveries become no-ops. Sequence numbers work the same way.

Cataloged Failure Modes

• out_of_order
• stale_overwrites_fresh

Code Comparison

# NAIVE: Overwriting status blindly on webhook delivery
@app.post("/webhooks/sub")
async def handle_sub(event: Event):
    await db.execute("UPDATE subs SET status = %s WHERE id = %s", event.status, event.sub_id)

# IMPROVED: Monotonic timestamp sequence check
@app.post("/webhooks/sub")
async def handle_sub(event: Event):
    await db.execute("UPDATE subs SET status = %s, last_ts = %s WHERE id = %s AND last_ts < %s", event.status, event.ts, event.sub_id, event.ts)

// NAIVE: Overwriting status blindly on webhook delivery
app.post('/webhooks/sub', async (req, res) => {
  await db.query('UPDATE subs SET status = $1 WHERE id = $2', [req.body.status, req.body.sub_id]);
  res.sendStatus(200);
});

// IMPROVED: Monotonic timestamp sequence check
app.post('/webhooks/sub', async (req, res) => {
  await db.query('UPDATE subs SET status = $1, last_ts = $2 WHERE id = $3 AND last_ts < $2', [req.body.status, req.body.ts, req.body.sub_id]);
  res.sendStatus(200);
});

Mitigation Patterns

• sequence-numbering