Usage-Refresh Deadlock & Auto-Swap Fixes

Problem

Stuck server. A usage refresh could wedge the entire server until restart: account rows froze at "checking", the single-account refresh endpoints never returned, and the background token-refresh and heal sweeps silently died. The dashboard's refresh buttons locked out forever because the frontend fetch had no timeout either.

Auto-swap shortfalls. Independently, the swap algorithm was stranding capacity (an active T0 with a long runway blocked an earlier-expiring T0; the fixed T1 target of 90 stranded the buffer when expiry landed outside working hours; the burn-floor check stranded the final slice of every account) and churning (id-keyed emergence streaks reset each other; asymmetric hysteresis fired premature drains; failed swaps retried every tick; manual switches were silently reverted within minutes). The decision log recorded every blocked swap as a generic tick, making any of this invisible.

Root Cause & Evidence Chain

Verified against the code, reproduced in tests/unit/test_refresh_deadlock.py:

1. _refresh_token_flow(PRIMARY) in jacked/web/auth.py ran the post-refresh fetch_profile while still holding the per-account primary refresh lock.
2. fetch_profile's 401 recovery path re-enters _refresh_token_flow on the same non-reentrant asyncio.Lock — the coroutine awaited its own lock: self-deadlock, lock never released.
3. Compounding: refresh_all_expiring_tokens and heal_invalid_accounts wrapped refresh_account_token in async with lock: — the flow acquires that same lock internally, so those paths self-deadlocked even without the profile-fetch trigger.
4. The lock acquire was unbounded, so every later refresh / 401-recovery / validate path for that account suspended forever behind the wedged holder.
5. main.py's _token_refresh_loop and _heal_sweep_loop awaited the wedged pass with no timeout — both loops dead until server restart.
6. HTTP routes (/refresh-usage, /refresh-token, /validate) awaited the same wedged coroutines unbounded — endless "checking" spinner; the frontend fetch() had no timeout, so its in-flight flags (_singleRefreshInFlight, _usageRefreshInProgress) stuck until page reload.

flowchart TD
    A[refresh_account_token] --> B[_refresh_token_flow PRIMARY
acquires per-account lock]
    B --> C[token exchange OK]
    C --> D[fetch_profile
WHILE HOLDING LOCK]
    D -->|401| E[recovery: re-enter
_refresh_token_flow]
    E -->|acquire same
non-reentrant lock| B
    E -. wedged forever .-> F[every later refresh /
heal / validate for this
account blocks unbounded]
    F --> G[main.py sweep loops +
HTTP routes await forever]
  

flowchart TD
    A[refresh_account_token] --> B[_refresh_token_flow PRIMARY
acquires per-account lock]
    B --> C[token exchange OK]
    C --> D[fetch_profile
WHILE HOLDING LOCK]
    D -->|401| E[recovery: re-enter
_refresh_token_flow]
    E -->|acquire same
non-reentrant lock| B
    E -. wedged forever .-> F[every later refresh /
heal / validate for this
account blocks unbounded]
    F --> G[main.py sweep loops +
HTTP routes await forever]
  

Fix Inventory

Bundle A — Deadlock / stuck server

Bundle B — Churn

Bundle C — Utilization

Bundle D — Observability

Test Strategy

New test files:

• tests/unit/test_refresh_deadlock.py — sweep and heal complete without deadlock (timeout-bounded), phantom-heal regression, bounded lock acquire returns lock_timeout, profile-fetch 401 re-entry does not deadlock, lock released after flow.
• tests/unit/test_swap_log.py — record/update status, residency, legacy-table migration, trailing-24h committed count, endpoint payload shape.
• tests/unit/api/test_routes_auth_timeouts.py — 504s with codes, status reset on usage timeout, 429 envelope, acquired_at reset.

Extended: test_auto_swap.py (target_for_tier, deadline-aware T1, damped-deficit admission, chronological sort, intra-T0 preemption, reset-suppress invariant, achievable_burn/stranding_estimate), test_usage_monitor.py (residency gate, failure backoff, taxonomy branches, fast-path emergence, tier-keyed persistence, same-tier advisory, execute-swap atomicity/residency, priming, active fetch timeout), test_use_account.py (manual switch arms pause + residency), test_token_refresh.py.

uv run python -m pytest tests/unit/test_refresh_deadlock.py tests/unit/test_swap_log.py \
  tests/unit/api/test_routes_auth_timeouts.py tests/unit/test_auto_swap.py \
  tests/unit/test_usage_monitor.py tests/unit/test_token_refresh.py \
  tests/unit/test_use_account.py -q

Rollout Notes

• Restart the server after install. The deadlock fix only takes effect in a fresh process — a currently-wedged server stays wedged until restarted. The swap_log column migration runs automatically at startup.
• Watch the swaps_last_24h counter (in GET /api/settings/swap-log). Expect a modest increase from 24/7 execution + intra-T0 preemption; min-residency, cooldown, and failure backoff should keep it in the single digits. A sustained high count means ping-pong — read the decision log triggers.
• In steady state the logs should NOT contain "could not acquire ... refresh lock" or "pass exceeded ... cancelled" — those lines now mean a new wedge is being contained rather than freezing the server.

Generated with the jacked HTML artifact template. Edit freely.