**Skill**: [Autonomous Agent Operations](../SKILL.md)

# Case Study: 2026-03-04 Backfill Incident

## Background

60 pueue backfill jobs queued on bigblack to populate ClickHouse with open deviation bars for 15 symbols x 4 thresholds. Monthly ouroboros mode, Vision CSV data source (post-Ariadne-fix).

## Agent Constellation Deployed

| Agent             | ID                  | Role              | Cycle  |
| ----------------- | ------------------- | ----------------- | ------ |
| Autoscaler        | `adbc4a0435cdd458e` | Resource tuning   | 10 min |
| Stall Detector v2 | `ae1d4ebe92bfd8312` | Detect stuck jobs | 10 min |
| Growth Monitor v2 | `aff642cbdd2c14054` | Track bar count   | 10 min |
| Overseer Alpha    | `a32db217c19eb17a4` | Agent watchdog    | 5 min  |
| Overseer Beta     | `aa8b585d08cac0b5e` | Agent watchdog    | 5 min  |

## Timeline of Failure

```
01:07  Jobs 0-59 queued via pueue-populate.sh backfill
01:09  Task 0 (BTCUSDT@1000) OOM-killed (MemoryMax=2G, exit 137)
01:09  Tasks 2,6,9,10,11 fail with "No data available" (exit 1)
       ^ These are PERSISTENT failures (missing Vision CSVs for specific dates)

~01:18 Stall detector detects failed tasks → runs `pueue restart 0`
       → Creates task 60 (copy of task 0, BTCUSDT@1000)
       → Task 60 also OOM-killed (same MemoryMax limit)

~01:28 Stall detector: `pueue restart 0` again → task 120
       Overseer Alpha: `pueue restart 32` → task 121

~02:00 Cascade accelerates: both stall detector (63 restarts)
       and overseer alpha (54 restarts) competing to restart failed tasks

~03:00 Tasks with exit code 1 being restarted endlessly:
       Each restart creates a new task → fails at same point → restarted again

~05:52 Task ID reaches ~6000
       BNBUSDT@250 now has ~30 concurrent copies running

~09:45 Task ID reaches ~12,000
       Stall detector spawning restarts every 3 seconds

~10:08 User checks status:
       - 11,759 done, 19 running, 688 queued
       - pueue payload: 21MB (WARN)
       - ClickHouse: 1,715,403 bars (progress WAS made by original jobs)

~10:15 Context compaction in Claude Code session ends the monitoring
```

## Damage Assessment

| Metric                       | Value                   | Impact                       |
| ---------------------------- | ----------------------- | ---------------------------- |
| Tasks created (total)        | ~12,800                 | Massive pueue payload bloat  |
| Tasks failed (exit 1)        | 11,649                  | Wasted process spawns        |
| Tasks failed (exit 137/OOM)  | 82                      | Wasted memory under cgroup   |
| Successful tasks             | 0                       | None completed clean         |
| BNBUSDT@250 duplicate bars   | 29,908                  | Stathera contamination       |
| Fragmented checkpoints       | 6 symbols with 2-3 rows | Ambiguous resume state       |
| pueue status payload         | 21 MB                   | Slow monitoring, agent bloat |
| Wasted SSH connections (~9h) | ~5,000+                 | Network and auth load        |

## Agent Mutation Forensics

```
autoscaler:     parallel=16                         # CORRECT (Level 1 tuning)
stall_detector: restart=63, kill=13                 # WRONG (should be Level 0)
overseer_alpha: restart=54, kill=31, parallel=10    # WRONG (scope creep)
growth_monitor: 0 mutations                         # CORRECT (Level 0 observer)
overseer_beta:  0 mutations                         # CORRECT (Level 0 observer)
```

**Root causes**:

1. Stall detector was not constrained to read-only (no FORBIDDEN list in prompt)
2. `pueue restart` semantics misunderstood (creates new task, not in-place restart)
3. No distinction between transient and persistent failures
4. Overseer Alpha exceeded its role boundary (watchdog → full job queue operator)
5. No deduplication check before restart (INV-1 violated)
6. No mutation budget per cycle (INV-3 violated)

## What Went Right

1. **ClickHouse bars DID grow**: 299,215 → 1,725,349 (+1,426,134). Original jobs made progress between failures.
2. **Autoscaler behaved correctly**: Only adjusted parallelism (16 `pueue parallel` commands), never spawned tasks.
3. **Growth monitor behaved correctly**: Pure read-only observer.
4. **Overseer Beta behaved correctly**: Read-only agent health checks.
5. **L2 ReplacingMergeTree dedup**: While 29,908 duplicates exist, `OPTIMIZE TABLE FINAL` can clean them.
6. **Checkpoints preserved**: Despite fragmentation, the most-advanced checkpoint for each symbol can be identified and used for resume.

## Corrective Actions Taken

1. Created the `autonomous-agent-ops` skill (invariants, anti-patterns, guardrails)
2. Updated `remote-backfill/SKILL.md` AP-3 (already warned about `pueue restart --all-failed`)
3. Agents must now follow INV-1 through INV-7
