---
name: ops-monitoring
description: >-
  On-demand operational monitoring scripts for bigblack infrastructure.
  Covers pueue backfill progress watching with Telegram reports, and future
  ad-hoc monitoring tools. Scripts use stdlib only (no pip dependencies)
  so they work in any venv. Invoke when you need to monitor long-running
  jobs, check backfill progress, or deploy a temporary watcher.
  TRIGGERS - pueue watcher, backfill progress, monitoring script,
  job watcher, ops monitoring, progress report, backfill status.
allowed-tools: Read, Bash, Grep, Glob, Agent
---

# Ops Monitoring

On-demand operational monitoring scripts for bigblack. These are NOT always-running services — they are invoked when needed (e.g., during multi-day backfills) and exit when the job is done.

All scripts are **stdlib-only** (use `urllib.request` instead of `requests`) so they run in any Python 3.13 venv without extra dependencies.

---

## Available Scripts

### `pueue-watcher.py` — Backfill Progress Reporter

Polls pueue every 5 minutes and sends rich Telegram progress reports with per-job day progress, failure reasons, and ETA estimates.

**When to use**: During multi-threshold historical backfills (e.g., BTCUSDT @500/@750/@1000 across years).

**What it does**:

- Groups jobs by pueue group (btc500, btc750, btc1000, etc.)
- Shows per-job day progress by parsing tqdm output from pueue logs
- Extracts failure reasons from task logs (OOM, SIGKILL, timeout, etc.)
- Maps exit codes: 134=SIGKILL/OOM, 137=OOM killer, 139=SIGSEGV
- Estimates ETA based on average completed job duration
- On completion: removes kintsugi systemd override and restarts kintsugi

**Deploy to bigblack**:

```bash
# Copy script
scp .claude/skills/ops-monitoring/scripts/pueue-watcher.py bigblack:/tmp/pueue_watcher.py

# Source env vars and run in tmux
ssh bigblack
tmux new -s pw
source ~/opendeviationbar-py/deploy/opendeviationbar.local.env
export OPENDEVIATIONBAR_TELEGRAM_TOKEN OPENDEVIATIONBAR_TELEGRAM_CHAT_ID
/home/tca/opendeviationbar-py/.venv/bin/python /tmp/pueue_watcher.py
# Ctrl-b d to detach
```

**Environment variables** (from `deploy/opendeviationbar.local.env`):

- `OPENDEVIATIONBAR_TELEGRAM_TOKEN` — Bot token for @obdpy_bot
- `OPENDEVIATIONBAR_TELEGRAM_CHAT_ID` — Telegram chat ID

**Auto-cleanup**: When all pueue jobs finish, the watcher removes the kintsugi systemd override (if any) and restarts kintsugi to resume normal healing across all thresholds.

---

## Adding New Scripts

Place new monitoring scripts in `scripts/` under this skill. Requirements:

1. **Stdlib only** — use `urllib.request` for HTTP, `json` for parsing. No `requests`, no `httpx`.
2. **Self-contained** — single file, no imports from the opendeviationbar package.
3. **Env-var configured** — read tokens/config from environment, not hardcoded.
4. **Exit cleanly** — return 0 on success, non-zero on error.
5. **Document in this SKILL.md** — add a section above with usage instructions.
