# 429 Recovery: Token Refresh + Escalating Backoff + Tier Override

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Break the 429 cycle by refreshing the access token (which clears the per-token rate limit), escalating backoff on consecutive failures, and forcing idle tier when fetches keep failing.

**Architecture:** Add cross-process locking to `credential_helpers.py` (compatible with Claude Code's `proper-lockfile` protocol). Add `_try_refresh_on_429` to `auth.py` that acquires the lock, exchanges the refresh token, saves all tokens, and retries. Track consecutive 429 count in coordinator state for escalating backoff and tier override.

**Tech Stack:** Python 3.12+ (httpx, os.mkdir for atomic locking)

**Design spec:** `docs/superpowers/specs/2026-04-03-429-token-refresh-and-lockfile-design.md`

---

## File Structure

| File | Role | Change |
|------|------|--------|
| `jacked/api/credential_helpers.py` | Credential management | Add `acquire_claude_lock()` / `release_claude_lock()` context manager |
| `jacked/web/auth.py` | Usage coordinator | Add `_try_refresh_on_429()`, escalating backoff, consecutive 429 tracking |
| `jacked/api/usage_monitor.py` | Poll loop | `_compute_poll_interval` checks consecutive 429s for tier override |
| `tests/unit/test_credential_helpers.py` | Lock tests | Test lock acquire/release/stale detection |
| `tests/unit/test_usage_refresh.py` | 429 tests | Test escalating backoff, consecutive tracking |

---

### Task 1: Cross-process lock compatible with proper-lockfile

**Files:**
- Modify: `jacked/api/credential_helpers.py`

`proper-lockfile` creates a `{path}.lock` directory as its lock mechanism. We implement the same protocol in Python.

- [ ] **Step 1: Add the lock context manager**

Add to `jacked/api/credential_helpers.py` after the imports:

```python
import contextlib
import signal


_LOCK_DIR = str(Path.home() / ".claude.lock")
_LOCK_MAX_RETRIES = 5
_LOCK_RETRY_BASE_MS = 1000


@contextlib.contextmanager
def acquire_claude_lock(timeout_retries: int = _LOCK_MAX_RETRIES):
    """Acquire a cross-process lock compatible with Claude Code's proper-lockfile.

    proper-lockfile creates a {path}.lock directory as its lock. We use
    os.mkdir (atomic on POSIX) to implement the same protocol.

    Includes stale lock detection: if the lock directory exists but the
    PID file inside is from a dead process, the lock is removed and retried.

    Usage:
        with acquire_claude_lock():
            # ... exchange tokens, write credentials ...
    """
    lock_path = _LOCK_DIR
    pid_file = os.path.join(lock_path, "pid")
    acquired = False

    for attempt in range(timeout_retries):
        try:
            os.mkdir(lock_path)
            # Write our PID for stale detection
            try:
                with open(pid_file, "w") as f:
                    f.write(str(os.getpid()))
            except OSError:
                pass
            acquired = True
            break
        except FileExistsError:
            # Lock exists — check if it's stale
            try:
                with open(pid_file, "r") as f:
                    holder_pid = int(f.read().strip())
                # Check if the holder process is alive
                os.kill(holder_pid, 0)  # signal 0 = existence check
            except (FileNotFoundError, ValueError, ProcessLookupError, PermissionError):
                # PID file missing, unreadable, or process dead — stale lock
                try:
                    import shutil
                    shutil.rmtree(lock_path, ignore_errors=True)
                    logger.info("Removed stale Claude lock (holder PID gone)")
                    continue  # Retry immediately
                except OSError:
                    pass
            except OSError:
                pass  # Process exists — lock is valid

            # Lock is held by a live process — wait and retry
            if attempt < timeout_retries - 1:
                wait = (_LOCK_RETRY_BASE_MS + random.randint(0, 1000)) / 1000.0
                logger.debug("Claude lock held, retrying in %.1fs (attempt %d)", wait, attempt + 1)
                time.sleep(wait)

    if not acquired:
        logger.warning("Failed to acquire Claude lock after %d retries", timeout_retries)

    try:
        yield acquired
    finally:
        if acquired:
            try:
                import shutil
                shutil.rmtree(lock_path, ignore_errors=True)
            except OSError:
                pass
```

Note: add `import random` and `import time` to the imports if not already present.

- [ ] **Step 2: Run tests**

Run: `uv run python -m pytest tests/ --tb=short -q 2>&1 | tail -5`

- [ ] **Step 3: Commit**

```bash
git add jacked/api/credential_helpers.py
git commit -m "feat: cross-process lock compatible with Claude Code's proper-lockfile"
```

---

### Task 2: Add consecutive 429 tracking and escalating backoff

**Files:**
- Modify: `jacked/web/auth.py`
- Modify: `tests/unit/test_usage_refresh.py`

- [ ] **Step 1: Add consecutive_429s to coordinator state**

In `jacked/web/auth.py`, update `_get_usage_state` to include `consecutive_429s`:

```python
def _get_usage_state(account_id: int) -> dict:
    if account_id not in _account_usage_state:
        _account_usage_state[account_id] = {
            "last_fetched_at": 0.0,
            "backoff_until": 0.0,
            "tier": "idle",
            "interval": _TIER_INTERVALS["idle"],
            "consecutive_429s": 0,
        }
    # Migration: add field for existing state dicts
    state = _account_usage_state[account_id]
    if "consecutive_429s" not in state:
        state["consecutive_429s"] = 0
    return state
```

- [ ] **Step 2: Update the 429 handler for escalating backoff**

In the 429 handler block (around line 513-530), replace:

```python
            if resp.status_code == 429:
                retry_after = resp.headers.get("retry-after", str(_USAGE_RATE_LIMIT_CEILING))
                try:
                    backoff_seconds = min(max(int(retry_after), _USAGE_RATE_LIMIT_CEILING), 900)
                except (ValueError, TypeError):
                    backoff_seconds = _USAGE_RATE_LIMIT_CEILING
                state["backoff_until"] = time.time() + backoff_seconds
                state["last_fetched_at"] = time.time()
```

With:

```python
            if resp.status_code == 429:
                state["consecutive_429s"] = state.get("consecutive_429s", 0) + 1
                n = state["consecutive_429s"]
                # Escalating backoff: 65 -> 130 -> 260 -> 520 -> cap 900
                base_backoff = min(_USAGE_RATE_LIMIT_CEILING * (2 ** (n - 1)), 900)
                retry_after = resp.headers.get("retry-after", str(base_backoff))
                try:
                    backoff_seconds = min(max(int(retry_after), base_backoff), 900)
                except (ValueError, TypeError):
                    backoff_seconds = base_backoff
                state["backoff_until"] = time.time() + backoff_seconds
                state["last_fetched_at"] = time.time()
```

- [ ] **Step 3: Reset consecutive count on success**

In the success block (around line 493, after `state["last_fetched_at"] = time.time()`), add:

```python
                state["consecutive_429s"] = 0
```

- [ ] **Step 4: Write failing test for escalating backoff**

Add to `tests/unit/test_usage_refresh.py`:

```python
class TestEscalatingBackoff:
    def test_consecutive_429s_escalate(self):
        """Each consecutive 429 should double the backoff."""
        import jacked.web.auth as mod
        mod._account_usage_state.clear()

        db = _mock_db({"usage_cached_at": int(time.time()) - 200})
        client = _mock_client(429, {}, headers={"retry-after": "0"})

        # First 429: backoff should be ~65s
        state = mod._get_usage_state(1)
        state["last_fetched_at"] = time.time() - 200
        with patch("jacked.web.auth.httpx.AsyncClient", return_value=client):
            asyncio.run(fetch_usage(1, db))
        backoff1 = mod._get_usage_state(1)["backoff_until"] - time.time()
        assert 60 <= backoff1 <= 70

        # Second 429: backoff should be ~130s
        state["last_fetched_at"] = time.time() - 200
        state["backoff_until"] = 0
        with patch("jacked.web.auth.httpx.AsyncClient", return_value=client):
            asyncio.run(fetch_usage(1, db))
        backoff2 = mod._get_usage_state(1)["backoff_until"] - time.time()
        assert 125 <= backoff2 <= 135

    def test_success_resets_consecutive_count(self):
        """Successful fetch should reset consecutive_429s to 0."""
        import jacked.web.auth as mod
        mod._account_usage_state.clear()
        state = mod._get_usage_state(1)
        state["consecutive_429s"] = 5
        state["last_fetched_at"] = time.time() - 200

        db = _mock_db({"usage_cached_at": int(time.time()) - 200})
        client = _mock_client(200, {"five_hour": {"utilization": 10.0}, "seven_day": {"utilization": 20.0}})
        with patch("jacked.web.auth.httpx.AsyncClient", return_value=client):
            asyncio.run(fetch_usage(1, db))
        assert mod._get_usage_state(1)["consecutive_429s"] == 0
```

- [ ] **Step 5: Run tests**

Run: `uv run python -m pytest tests/unit/test_usage_refresh.py -v --tb=short`

- [ ] **Step 6: Commit**

```bash
git add jacked/web/auth.py tests/unit/test_usage_refresh.py
git commit -m "feat: escalating backoff on consecutive 429s (65->130->260->cap 900)"
```

---

### Task 3: Force idle tier after consecutive 429s

**Files:**
- Modify: `jacked/api/usage_monitor.py`

- [ ] **Step 1: Check consecutive 429s in _compute_poll_interval**

In `_compute_poll_interval`, after computing the tier, check if the account is in a 429 cycle. Find the block where `state["tier"] = tier` is set (around line 103) and add an override:

```python
        state["tier"] = tier
        state["interval"] = base

        # Override: force idle if stuck in 429 cycle (stale data makes
        # urgency tiers unreliable — polling faster is pointless).
        if state.get("consecutive_429s", 0) >= 3:
            tier = "idle"
            base = _TIER_INTERVALS["idle"]
            state["tier"] = tier
            state["interval"] = base
```

Note: `_TIER_INTERVALS` needs to be imported. Add it to the existing late-import block or import at the top of the function:

```python
        from jacked.web.auth import compute_urgency_tier, _get_usage_state, _TIER_INTERVALS
```

- [ ] **Step 2: Run tests**

Run: `uv run python -m pytest tests/unit/test_usage_monitor.py --tb=short -q`

- [ ] **Step 3: Commit**

```bash
git add jacked/api/usage_monitor.py
git commit -m "feat: force idle tier after 3+ consecutive 429s"
```

---

### Task 4: Token refresh on 429 to clear rate limit

**Files:**
- Modify: `jacked/web/auth.py`

This is the core fix. When we get a 429 and the account has a refresh token, try to exchange it for a fresh access token (which clears the per-token rate limit), using the cross-process lock.

- [ ] **Step 1: Add the refresh-on-429 function**

Add to `jacked/web/auth.py`, before `fetch_usage`:

```python
async def _try_refresh_on_429(
    account_id: int, db: Database, state: dict,
) -> str | None:
    """Attempt to get a fresh access token to clear a per-token rate limit.

    Acquires the Claude Code cross-process lock, exchanges the refresh
    token for a fresh access token, and saves all tokens to DB + credential
    stores. Returns the fresh access token on success, None on failure.

    MUST be called within the 429 handler — assumes the current token
    is rate-limited.
    """
    from jacked.api.credential_helpers import (
        acquire_claude_lock,
        read_platform_credentials,
        sync_credential_to_all_stores,
    )

    account = db.get_account(account_id)
    if not account:
        return None

    # Prefer CC refresh token, fall back to primary
    refresh_token = account.get("cc_refresh_token") or account.get("refresh_token")
    if not refresh_token:
        logger.debug("Account %d: no refresh token for 429 recovery", account_id)
        return None

    with acquire_claude_lock() as locked:
        if not locked:
            logger.warning("Account %d: could not acquire lock for 429 recovery", account_id)
            return None

        # Re-read from live store — another process may have refreshed
        live = read_platform_credentials()
        if not live:
            cred_path = Path.home() / ".claude" / ".credentials.json"
            if cred_path.exists() and not cred_path.is_symlink():
                try:
                    live = json.loads(cred_path.read_text(encoding="utf-8"))
                except (json.JSONDecodeError, OSError):
                    live = None

        if live and live.get("_jackedAccountId") == account_id:
            live_token = live.get("claudeAiOauth", {}).get("accessToken")
            db_token = account.get("cc_access_token") or account.get("access_token")
            if live_token and live_token != db_token:
                # Another process already refreshed — use their token
                logger.info("Account %d: 429 recovery — using token from live store", account_id)
                db.update_account(account_id, cc_access_token=live_token)
                return live_token

        # Exchange refresh token for fresh access token
        try:
            async with httpx.AsyncClient(timeout=15.0) as client:
                resp = await client.post(
                    TOKEN_URL,
                    json={
                        "grant_type": "refresh_token",
                        "refresh_token": refresh_token,
                        "client_id": CLIENT_ID,
                    },
                    headers={
                        "Content-Type": "application/json",
                        "anthropic-beta": OAUTH_BETA_HEADER,
                    },
                )

            if resp.status_code != 200:
                logger.warning(
                    "Account %d: 429 recovery token exchange failed (HTTP %d)",
                    account_id, resp.status_code,
                )
                return None

            tokens = resp.json()
            fresh_access = tokens["access_token"]
            new_refresh = tokens.get("refresh_token", refresh_token)
            new_expires = int(time.time()) + tokens.get("expires_in", 28800)

            # Determine which token set was used
            used_cc = bool(account.get("cc_refresh_token"))
            if used_cc:
                db.update_account(
                    account_id,
                    cc_access_token=fresh_access,
                    cc_refresh_token=new_refresh,
                    cc_expires_at=new_expires,
                )
            else:
                db.update_account(
                    account_id,
                    access_token=fresh_access,
                    refresh_token=new_refresh,
                    expires_at=new_expires,
                )

            # Write to credential stores so Claude Code picks up the new token
            updated_account = db.get_account(account_id)
            if updated_account:
                sync_credential_to_all_stores(
                    account_id, updated_account,
                    email=updated_account.get("email"),
                )

            logger.info(
                "Account %d: 429 recovery — fresh token obtained%s",
                account_id,
                " (refresh rotated)" if new_refresh != refresh_token else "",
            )
            return fresh_access

        except Exception as exc:
            logger.warning("Account %d: 429 recovery failed: %s", account_id, exc)
            return None
```

- [ ] **Step 2: Call _try_refresh_on_429 from the 429 handler**

In the 429 handler block, BEFORE the escalating backoff, add the refresh attempt. Change the 429 block to:

```python
            if resp.status_code == 429:
                # Try to clear the per-token rate limit by getting a fresh token
                fresh_token = await _try_refresh_on_429(account_id, db, state)
                if fresh_token:
                    # Retry with the fresh token
                    state["consecutive_429s"] = 0
                    state["last_fetched_at"] = 0  # Allow immediate retry
                    logger.info("Account %d: retrying usage fetch with fresh token", account_id)
                    # Recursive call with the fresh token
                    return await fetch_usage(account_id, db, access_token=fresh_token)

                # No refresh available — escalating backoff
                state["consecutive_429s"] = state.get("consecutive_429s", 0) + 1
                n = state["consecutive_429s"]
                base_backoff = min(_USAGE_RATE_LIMIT_CEILING * (2 ** (n - 1)), 900)
                retry_after = resp.headers.get("retry-after", str(base_backoff))
                try:
                    backoff_seconds = min(max(int(retry_after), base_backoff), 900)
                except (ValueError, TypeError):
                    backoff_seconds = base_backoff
                state["backoff_until"] = time.time() + backoff_seconds
                state["last_fetched_at"] = time.time()
                db.record_account_error(
                    account_id,
                    f"Usage fetch rate limited (429) — backing off {backoff_seconds}s "
                    f"(consecutive: {n})",
                    increment_failures=False,
                )
                logger.warning(
                    f"Usage fetch rate limited for account {account_id}, "
                    f"backing off {backoff_seconds}s (consecutive: {n})"
                )
                return None
```

- [ ] **Step 3: Run tests**

Run: `uv run python -m pytest tests/ --tb=short -q 2>&1 | tail -5`

- [ ] **Step 4: Commit**

```bash
git add jacked/web/auth.py
git commit -m "feat: refresh access token on 429 to clear per-token rate limit"
```

---

### Task 5: Run full test suite

- [ ] **Step 1: Run full test suite**

Run: `uv run python -m pytest tests/ --tb=short -q`

Expected: All tests pass.
