## <!-- SSoT-OK: threshold values in dbps (decimal basis points), not software versions -->

name: opendeviationbar-job-safety
description: >-
Project-specific job safety configuration for opendeviationbar-py. Extends devops-tools:distributed-job-safety
with concrete function names, env vars, memory profiles, and ClickHouse queries.
TRIGGERS - opendeviationbar pueue jobs, deploy to bigblack, opendeviationbar cache population,
SHIBUSDT overflow, checkpoint race, opendeviationbar threshold.
allowed-tools: Read, Bash, Write

---

# opendeviationbar-job-safety

This skill extends `devops-tools:distributed-job-safety` with opendeviationbar-py project-specific details. The universal skill provides the invariants and anti-patterns; this skill provides concrete enforcement.

## Prerequisite Skills

- `devops-tools:distributed-job-safety` (universal invariants)
- `devops-tools:pueue-job-orchestration` (pueue commands)

## opendeviationbar-py Invariant Enforcement

| Invariant           | Universal (INV-N) | opendeviationbar-py Enforcement                                                  |
| ------------------- | ----------------- | -------------------------------------------------------------------------------- |
| Filename Uniqueness | INV-1             | `_get_checkpoint_path()` includes `threshold_decimal_bps` in filename            |
| Idempotent Cleanup  | INV-2             | All `unlink()` calls in `checkpoint.py` use `missing_ok=True`                    |
| Atomic Writes       | INV-3             | `checkpoint.save()` uses tempfile + fsync + os.replace                           |
| Config SSoT         | INV-4             | `.mise.toml [env]` owns `OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD`                  |
| Post-Write Dedup    | INV-5             | `OpenDeviationBarCache.deduplicate_bars()` after population                      |
| Gate Before Compute | INV-6             | `validate_symbol_registered()` at top of every entry point                       |
| Memory Isolation    | INV-7             | `systemd-run` in `pueue-populate.sh`, bypass with `OPENDEVIATIONBAR_NO_CGROUP=1` |

## Key Environment Variables

| Variable                                | SSoT         | Default  | Purpose                              |
| --------------------------------------- | ------------ | -------- | ------------------------------------ |
| `OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD` | `.mise.toml` | 1000     | Minimum threshold (dev safety guard) |
| `OPENDEVIATIONBAR_CH_HOSTS`             | `.mise.toml` | bigblack | ClickHouse host                      |
| `OPENDEVIATIONBAR_NO_CGROUP`            | per-job      | unset    | Bypass systemd-run memory caps       |

## Memory Profiles by Threshold Tier

Empirically measured on bigblack (32 cores, 61 GB):

| Threshold | Peak RSS | MemoryMax | Headroom |
| --------- | -------- | --------- | -------- |
| 250 dbps  | ~5 GB    | 8G        | 60%      |
| 500 dbps  | ~1.5 GB  | 4G        | 167%     |
| 750 dbps  | ~1.5 GB  | 4G        | 167%     |
| 1000 dbps | ~1 GB    | 2G        | 100%     |

## Real-World Scaling Example

bigblack (32 cores, 61 GB):

```
Initial:   p2=2, p3=3  ->  5 concurrent, load 5,  mem 23 GB
Bump 1:    p2=4, p3=5  ->  9 concurrent, load 9,  mem 24 GB
Bump 2:    p2=6, p3=8  -> 14 concurrent, load 12, mem 25 GB
(Still within 80% margin: 25.6 cores, 49 GB)
```

## Per-Year Parallelization (Ouroboros)

Day-mode ouroboros makes each day independently reprocessable. For large backfills, split by year-sized date ranges into per-symbol pueue groups — each job processes one year of days sequentially.

**Concrete groups used on bigblack:**

| Group            | Parallel | Symbol+Threshold | Memory/Job |
| ---------------- | -------- | ---------------- | ---------- |
| `shib250-yearly` | 5        | SHIBUSDT@250     | ~5 GB      |
| `doge250-yearly` | 4        | DOGEUSDT@250     | ~3 GB      |
| `btc250-yearly`  | 4        | BTCUSDT@250      | ~5 GB      |
| `eth250-yearly`  | 4        | ETHUSDT@250      | ~5 GB      |
| `bnb250-yearly`  | 4        | BNBUSDT@250      | ~3 GB      |

**Bar production rates** (determines compute time, not input trade count):

| Symbol   | Bars/Day @250 dbps                       | Why                                                 |
| -------- | ---------------------------------------- | --------------------------------------------------- |
| SHIBUSDT | ~1,287                                   | Tiny price moves, needs huge volume to breach 0.25% |
| DOGEUSDT | ~1,000 (avg), 219K (pump day 2021-01-28) | Meme coin volatility bursts                         |
| BTCUSDT  | ~135                                     | Large price moves, breaches threshold quickly       |
| ETHUSDT  | ~200                                     | Moderate price moves                                |
| BNBUSDT  | ~180                                     | Moderate price moves                                |

**Compute bottleneck**: Intra-bar features (Hurst DFA O(n log n), Permutation Entropy O(n!)) run per bar. 10x more bars = 10x more compute time. Input trade count is NOT the bottleneck.

**Critical rules**: See `devops-tools:pueue-job-orchestration` Per-Year Parallelization section.

## Pipeline Monitoring

`scripts/pipeline-monitor.sh` — background monitor polling bigblack every 5 min via SSH:

- Detects phase transitions by **group completion** (not hardcoded job IDs — see `devops-tools:distributed-job-safety` AP-8)
- Runs 4 integrity checks at each boundary: negative volumes, lookback coverage, OFI bounds, duplicate bars
- Reports failures immediately with actionable commands
- Exits automatically when all jobs complete

Launch as Claude Code background task: `bash scripts/pipeline-monitor.sh`

## ClickHouse Forensic Audit

```bash
ssh bigblack "clickhouse-client --query \"
  SELECT symbol, threshold_decimal_bps,
    count() as bars,
    countIf(volume < 0) as neg_vol,
    countIf(buy_volume < 0) as neg_buy,
    countIf(sell_volume < 0) as neg_sell,
    round(countIf(volume < 0) * 100.0 / count(), 2) as pct_corrupt
  FROM opendeviationbar_cache.open_deviation_bars
  GROUP BY symbol, threshold_decimal_bps
  HAVING neg_vol > 0 OR neg_buy > 0 OR neg_sell > 0
  ORDER BY symbol, threshold_decimal_bps
  FORMAT PrettyCompact\""
```

## Per-Symbol Minimum Threshold

Some symbols produce too many bars at low thresholds, making computation infeasible.

| Symbol   | min_threshold (dbps) | Reason                                |
| -------- | -------------------- | ------------------------------------- |
| SHIBUSDT | 500                  | 1,287 bars/day @250 — days of compute |
| DOGEUSDT | 500                  | 219K bars on pump day @250            |

**SSoT**: `.mise.toml [env]` — the ONLY source of truth for per-symbol thresholds:

```toml
OPENDEVIATIONBAR_MIN_THRESHOLD_SHIBUSDT = "500"
OPENDEVIATIONBAR_MIN_THRESHOLD_DOGEUSDT = "500"
```

**Resolution chain** (`threshold.py:get_min_threshold_for_symbol()`):

1. `OPENDEVIATIONBAR_MIN_THRESHOLD_<SYMBOL>` (per-symbol env var, highest priority)
2. `OPENDEVIATIONBAR_<ASSET_CLASS>_MIN_THRESHOLD` (asset-class env var)
3. Fallback default (with deprecation warning)

Violations raise `ThresholdError`. No code changes needed — mise sets the env var, `threshold.py` reads it.

## Provenance

All issue numbers reference [github.com/terrylica/opendeviationbar-py/issues](https://github.com/terrylica/opendeviationbar-py/issues).

| Issue                                                             | Lesson                                              |
| ----------------------------------------------------------------- | --------------------------------------------------- |
| [#78](https://github.com/terrylica/opendeviationbar-py/issues/78) | Parquet deduplication, silent 209M duplicates       |
| [#79](https://github.com/terrylica/opendeviationbar-py/issues/79) | Symbol registry gate prevents ghost trades          |
| [#84](https://github.com/terrylica/opendeviationbar-py/issues/84) | Checkpoint race condition, threshold in filename    |
| [#85](https://github.com/terrylica/opendeviationbar-py/issues/85) | Pandas 3.0 datetime resolution on bigblack          |
| [#88](https://github.com/terrylica/opendeviationbar-py/issues/88) | FixedPoint i64 overflow, manual post-processing     |
| [#89](https://github.com/terrylica/opendeviationbar-py/issues/89) | Per-symbol min threshold for meme coins (SHIB/DOGE) |

## References

- [opendeviationbar-specific gotchas](./references/opendeviationbar-gotchas.md)
- Universal skill: `devops-tools:distributed-job-safety`
