Task State Machine & Retry Budget Accounting Tasks transition from pending or scheduled to running, and settle into completed, failed, cancelled, or expired while separating retry_count from infrastructure interruptions. pending scheduled_at == None Ready for immediate claim scheduled scheduled_at > now Delayed or backoff retry running (Fenced Lease) attempts = retry_count - interruptions Active heartbeat_at lease refresh Exception & attempts < max_retries -> scheduled Stale lease reclaim -> retry_count++, interruptions++ completed Result stored • completed_at set failed Retries exhausted • error recorded cancelled cancel_task() • cooperative abort expired expires_at passed before claim
Independent Task Clocks Comparison of expires_in/expires_at (queued deadline), timeout (per-attempt execution ceiling), and terminal_ttl/terminal_retention (post-settlement storage cleanup). Queue Expiration Clock expires_in / expires_at Ticks while pending or scheduled Stops the moment a worker claims task Never interrupts a running task Outcome: status = "expired" Execution Timeout Clock timeout=30.0 | timedelta(...) Ticks per execution attempt (`started_at`) Enforced via asyncio deadline scope Consumes retry budget if max_retries > 0 Outcome: retry or status = "failed" Terminal Retention Clock terminal_ttl / terminal_retention Ticks after completed/failed/cancelled/expired Native Redis/Valkey PEXPIRE (`terminal_ttl`) Maintenance Phase (`cleanup_terminal`) Preserves unique_until="forever"
Uniqueness Windows & Permanent TaskReservation Comparison of unique_until='terminal', duplicate policies ('return_existing' vs 'raise'), and unique_until='forever' backed by TaskReservation. In-Flight Deduplication unique_until="terminal" Active while pending, scheduled, or running Duplicate enqueue returns existing task_id Releases lock as soon as task settles Ideal for coalescing webhook bursts Duplicate Conflict Policies on_duplicate="return_existing" | "raise" "return_existing" returns active TaskResult "raise" raises DuplicateTaskError Effective key hashed from args or unique_key Inspect via get_task_identity() Permanent Reservation unique_until="forever" Backed by compact TaskReservation record Survives terminal_ttl / cleanup_terminal Failed/cancelled/expired releases lock to retry Reset via reset_task_identity()