---
name: autonomous-agent-ops
description: >-
  Multi-agent autonomous operations for long-running distributed jobs via Claude Code.
  Covers pueue job queue management, SSH-remote monitoring, mutual-watchdog patterns,
  and hard-won guardrails from the 2026-03-04 backfill incident where unconstrained
  agents created ~12,800 runaway tasks, 29,908 duplicate database rows, and 21MB
  pueue payloads.
  TRIGGERS - autonomous monitoring, multi-agent, background agents, pueue monitoring,
  watchdog, autoscaler, stall detector, long-running jobs, backfill monitoring,
  agent constellation, 12-hour monitoring, dynamic parallelism.
allowed-tools: Read, Bash, Grep, Glob, AskUserQuestion, Agent
---

# Autonomous Agent Operations

Guide Claude Code background agents operating autonomously over long-running distributed job queues (pueue, systemd, etc.) accessed via SSH.

**Born from**: [The 2026-03-04 backfill incident](./references/case-study-2026-03-04.md) — 5 agents, 9 hours unsupervised, ~12,800 runaway tasks.

**Pueue command reference**: [Pueue Semantics](./references/pueue-semantics.md)

---

## Task Templates

### Deploy Monitoring Constellation

```
[Preflight]
1. Run Pre-Flight Checklist (section below)
2. Verify all jobs are queued and groups are running
3. Decide autonomy level per agent (RP-2)

[Execute]
4. Write agent prompts with explicit ALLOWED/FORBIDDEN lists (INV-4)
5. Launch workers (Level 0-1) as background agents
6. Launch overseers (Level 0) as background agents, staggered
7. Record agent IDs in shared state file

[Verify]
8. Check all agents are running (TaskOutput block=false)
9. Verify first cycle output is structured (RP-3)
10. Confirm no mutations occurred beyond allowed scope
```

### Investigate Agent Misbehavior

```
[Preflight]
1. Stop all agents (TaskStop or let them timeout)
2. Count current pueue tasks: ssh HOST 'pueue status | wc -l'

[Execute]
3. Parse agent transcripts for mutation commands (grep pueue restart/add/kill)
4. Count tasks per label to identify duplicates
5. Check ClickHouse for Stathera overlaps (Recovery Scenario 2)
6. Check for checkpoint fragmentation (Recovery Scenario 3)

[Verify]
7. Document findings in revision log
8. Update anti-patterns if new failure mode discovered
```

### Recover from Runaway Tasks

```
[Preflight]
1. Assess damage: task count, failure distribution, data integrity

[Execute]
2. Follow Recovery Procedure Scenario 1 (pause, kill, clean, deduplicate)
3. If Stathera overlaps: OPTIMIZE TABLE FINAL
4. If checkpoint fragmentation: resolve to single canonical checkpoint

[Verify]
5. Verify clean pueue state (no duplicates)
6. Verify ClickHouse data integrity (overlaps = 0)
7. Re-queue jobs from clean state
```

---

## Architecture: Agent Roles

```
         ┌──────────────────────────────────────────┐
Layer 3  │  Overseers (mutual watchdog, 2 agents)   │  Revive dead workers
         └──────────────┬───────────────────────────┘
         ┌──────────────┴───────────────────────────┐
Layer 2  │  Workers (2-3 specialized agents)        │  Monitor and tune
         └──────────────┬───────────────────────────┘
         ┌──────────────┴───────────────────────────┐
Layer 1  │  Job Queue (pueue/systemd on remote)     │  SSH access
         └─────────────────────────────────────────────┘
```

| Role                | Reads                            | Writes (ALLOWED)             | Writes (FORBIDDEN)                         |
| ------------------- | -------------------------------- | ---------------------------- | ------------------------------------------ |
| Resource Autoscaler | `free`, `uptime`, `pueue status` | `pueue parallel N --group G` | `pueue add`, `pueue restart`, `pueue kill` |
| Stall Detector      | `pueue status`, `pueue log`      | **Report only**              | `pueue restart`, `pueue add`, `pueue kill` |
| Growth Monitor      | DB queries, `pueue status`       | **Report only**              | Any pueue mutation                         |
| Overseer            | Agent status (TaskOutput)        | Relaunch dead agents         | Direct pueue mutation                      |

**Critical**: Stall detectors and growth monitors are **READ-ONLY observers**. They report — they do NOT remediate.

---

## Invariants

### INV-1: No Task Spawning Without Deduplication

`pueue restart` creates a **NEW task** by default (NOT in-place). Use `pueue restart --in-place` if restart is truly needed. See [Pueue Semantics](./references/pueue-semantics.md).

### INV-2: Distinguish Transient vs Persistent Failures

| Exit Code | Meaning            | Transient? | Action                         |
| --------- | ------------------ | ---------- | ------------------------------ |
| 137       | OOM (cgroup kill)  | Maybe      | Check if memory limit is wrong |
| 1         | Application error  | **Check**  | Read `pueue log` first         |
| 130       | SIGINT (Ctrl-C)    | Yes        | Safe to restart                |
| 143       | SIGTERM (graceful) | Yes        | Safe to restart                |

**Persistent failures** (missing data, bad arguments) will **never succeed on retry**.

### INV-3: Mutation Budget Per Cycle

| Mutation Type    | Max Per Cycle | Rationale                                  |
| ---------------- | ------------- | ------------------------------------------ |
| `pueue parallel` | 1             | Parallelism changes need time to stabilize |
| `pueue restart`  | 0 (default)   | Should be human-authorized                 |
| `pueue kill`     | 3             | Emergency only                             |
| `pueue add`      | 0             | Agents should never spawn new work         |

### INV-4: Explicit Authorization Boundary

Agent prompts MUST include an **allowlist** AND a **FORBIDDEN** list of commands. Template:

```
ALLOWED: pueue status, pueue log, free -h, uptime
FORBIDDEN: pueue add, pueue restart, pueue kill, pueue clean, pueue reset
```

### INV-5: Payload Size Awareness

Use `pueue status --group G | grep` instead of `pueue status --json` for routine checks. Full JSON serializes ALL tasks (10-50+ MB with thousands of tasks).

### INV-6: No Concurrent Writers to Same Resource

Never allow two jobs to write to the same (symbol, threshold) simultaneously. Verify before any restart:

```bash
pueue status | grep "Running" | grep "SYMBOL@THRESHOLD"
```

### INV-7: Sleep Duration Must Match Monitoring Purpose

| Agent Purpose       | Cycle Time | Rationale                      |
| ------------------- | ---------- | ------------------------------ |
| Resource autoscaler | 10-15 min  | RAM/CPU changes slowly         |
| Stall detector      | 10-15 min  | Jobs process days, not seconds |
| Growth monitor      | 15-30 min  | DB growth visible at 30-min    |
| Overseer            | 5 min      | Agent liveness must be prompt  |

---

## Anti-Patterns

### AP-1: `pueue restart` Without `--in-place` (CRITICAL)

Creates NEW tasks. Cascade: restart → fail → restart → exponential task growth. Evidence: 63 restarts → ~12,800 total tasks. **Fix**: Agents must NEVER use `pueue restart`.

### AP-2: Restarting Persistent Failures

`RuntimeError: No data available` will never succeed on retry. 11,649 tasks failed with exit code 1 — all restarts wasted. **Fix**: Read `pueue log` before any restart.

### AP-3: Agent Scope Creep

Overseer Alpha ran 54 restarts, 31 kills, 10 parallelism changes — far beyond its watchdog role. Overseer Beta ran 0 mutations (correct). **Fix**: Overseers must ONLY monitor agent health.

### AP-4: Multiple Agents Restarting Same Task

Both stall detector AND overseer restarted task 17 (10+ times), creating 54 concurrent copies of the same job. **Fix**: If remediation is needed, coordinate via shared lock file.

### AP-5: Concurrent Writers Causing Data Corruption

54 copies of BNBUSDT@250 produced 29,908 duplicate bars and 2 divergent checkpoints. **Fix**: Enforce INV-6.

### AP-6: `pueue status --json` for Routine Monitoring

21MB payload per call consumed agent context and caused 2-3s latency. **Fix**: Use text-mode grep queries.

---

## Recommended Patterns

### RP-1: Read-Only Observer Pattern

```
"You are a READ-ONLY monitor. You MUST NOT run commands that modify job state.
 ALLOWED: pueue status, pueue log, free -h, uptime, clickhouse-client --query
 FORBIDDEN: pueue restart, pueue add, pueue kill, pueue remove, pueue clean
 When you detect issues, REPORT them — do NOT fix them."
```

### RP-2: Graduated Autonomy Levels

| Level | Name         | May Do                                 | Must NOT                          |
| ----- | ------------ | -------------------------------------- | --------------------------------- |
| 0     | Observer     | Read status, logs, query DB            | Any mutation                      |
| 1     | Tuner        | Level 0 + `pueue parallel N`           | restart, add, kill                |
| 2     | Remediator   | Level 1 + `restart --in-place` (INV-2) | add, kill without approval        |
| 3     | Orchestrator | Level 2 + `pueue add` (INV-1 dedup)    | `pueue reset`, unguarded restarts |

**Default**: Level 0 for detectors/monitors. Level 1 for autoscalers. Level 2+ requires explicit user authorization.

### RP-3: Structured Status Reporting

```
[CYCLE 14/72] 2026-03-04T05:30:00Z
  RAM: 38GB/61GB (62%) | Load: 23.8 (32 cores)
  Jobs: 19 running, 491 queued, 352 done (346 failed, 6 OOM)
  DB: 1,725,349 bars (+12,451 since last, +37/min)
  Issues: None | Action: None
```

### RP-4: Failure Classification

```python
def classify_failure(task_id):
    log = pueue_log(task_id, lines=20)
    if "No data available" in log:    return "PERSISTENT:missing_data"
    if exit_code == 137:              return "PERSISTENT:oom_limit"
    if "Connection refused" in log:   return "TRANSIENT:connection"
    if "RuntimeError" in log:         return "PERSISTENT:app_error"
    return "UNKNOWN"  # Report to human
```

### RP-5: Heartbeat File for Coordination

```bash
ssh HOST "echo '{\"agent\":\"autoscaler\",\"cycle\":14}' >> /tmp/agent-heartbeat.jsonl"
ssh HOST "tail -5 /tmp/agent-heartbeat.jsonl"  # Overseer reads
```

### RP-6: Graceful SSH Failure Degradation

1st failure: retry once after 30s. 2nd: skip cycle, warn. 3rd consecutive: report critical. NEVER retry in tight loop.

### RP-7: Consider Temporal for Structured Workloads

For repeatable, well-defined job pipelines (e.g., backfill), [Temporal](https://temporal.io/) eliminates the need for monitoring agents entirely:

| Pueue + Agents Problem                    | Temporal Built-in Solution                    |
| ----------------------------------------- | --------------------------------------------- |
| `pueue restart` creates duplicates        | Workflow ID uniqueness — same ID = rejected   |
| Blind retry of persistent failures        | `non_retryable_error_types` stops immediately |
| Agents competing to restart same task     | Temporal owns retries — no external agents    |
| No parallelism control without autoscaler | `max_concurrent_activities` on worker         |
| 21MB `pueue status --json` payload        | Web UI with pagination and search             |

**When to use Temporal vs pueue**: Pueue for ad-hoc shell commands; Temporal for structured, repeatable workflows needing durability and dedup. See `Skill(devops-tools:distributed-job-safety)` for the full comparison.

---

## Pre-Flight Checklist

Before launching autonomous monitoring agents:

1. **Job Queue Health**: `ssh HOST 'pueue status | grep -c "Failed"'` (must be 0 or understood)
2. **No Duplicates**: `ssh HOST 'pueue status | grep "Running" | awk "{print \$4}" | sort | uniq -d'` (must be empty)
3. **Resource Baseline**: `ssh HOST 'free -h | head -2 && uptime'`
4. **Task Count Baseline**: Record initial task count for drift detection
5. **Agent Prompts**: Each has explicit ALLOWED + FORBIDDEN lists, cycle time, duration, autonomy level

---

## Recovery Procedures

### Scenario 1: Runaway Task Accumulation

```bash
ssh HOST 'pueue pause --all'         # Stop new starts
ssh HOST 'pueue kill --all'          # Kill running duplicates
ssh HOST 'pueue clean'              # Remove done/failed
# Manually remove queued duplicates, keep one per label
ssh HOST 'pueue start --all'         # Resume clean
```

### Scenario 2: Database Overlap Contamination

```bash
clickhouse-client --query "OPTIMIZE TABLE opendeviationbar_cache.open_deviation_bars FINAL"
# Verify: SELECT count() - uniq(first_agg_trade_id) should be 0
```

### Scenario 3: Checkpoint Fragmentation

Query `population_checkpoints` for `count() > 1` per (symbol, threshold). Keep the most-advanced checkpoint, delete others.

### Scenario 4: Agent Death

Overseer launches fresh agent with same prompt. Do NOT `resume` dead agents (context may be corrupted). Do NOT modify job queue directly.

---

## Post-Change Checklist

After modifying this skill:

- [ ] Invariants INV-1 through INV-7 remain consistent with anti-patterns
- [ ] Recovery procedures reference correct invariants
- [ ] Task templates use `[Preflight]`/`[Execute]`/`[Verify]` labels
- [ ] References in `references/` updated if anti-patterns changed
- [ ] Line count under 500 (move new case studies to `references/`)
- [ ] Revision Log updated below

---

## References

- [Case Study: 2026-03-04 Backfill Incident](./references/case-study-2026-03-04.md) — full timeline, damage, forensics
- [Pueue Command Semantics](./references/pueue-semantics.md) — restart, kill, status, exit codes
- `Skill(remote-backfill)` — canonical backfill recipe
- `Skill(opendeviationbar-job-safety)` — memory profiles, cgroup limits
- `Skill(devops-tools:distributed-job-safety)` — universal job invariants

---

## Revision Log

| Date       | Rev | Change                                                                                          | Author |
| ---------- | --- | ----------------------------------------------------------------------------------------------- | ------ |
| 2026-03-04 | 1.0 | Initial skill from backfill incident forensics                                                  | Claude |
| 2026-03-04 | 1.1 | Refactored: case study + pueue ref to references/, added task templates + post-change checklist | Claude |
| 2026-03-04 | 1.2 | Added RP-7 (Temporal as SOTA alternative for structured workloads)                              | Claude |

<!-- Future revisions go here as new incidents teach new lessons -->
