# Scripts & Operations

**Parent**: [/CLAUDE.md](/CLAUDE.md) | **Skills**: `Skill(devops-tools:pueue-job-orchestration)` | `Skill(opendeviationbar-job-safety)`

Operational scripts for cache population, validation, and distributed job management on remote GPU hosts.

---

## Quick Reference

| Task                           | Command                                           | Details                                   |
| ------------------------------ | ------------------------------------------------- | ----------------------------------------- |
| Kintsugi catch-up (historical) | `mise run kintsugi:catchup`                       | [Kintsugi](#kintsugi)                     |
| Check cache status             | `mise run cache:status`                           | ClickHouse bar counts                     |
| Detect volume overflow         | `mise run cache:detect-overflow`                  | Issue #88                                 |
| Validate dedup hardening       | `mise run cache:validate-dedup`                   | Issue #90                                 |
| Validate microstructure        | `uv run python scripts/validate_microstructure_*` | [Validation Scripts](#validation-scripts) |
| Run streaming sidecar          | `mise run streaming:start`                        | v12.20+, real-time open deviation bars    |
| Kintsugi (single pass)         | `mise run kintsugi`                               | [Kintsugi](#kintsugi)                     |
| Kintsugi (daemon)              | `mise run kintsugi:daemon`                        | Long-running self-healer                  |
| Kintsugi (dry-run)             | `mise run kintsugi:check`                         | Discover shards without repairing         |

---

## Kintsugi: Sole Gap-Filling Mechanism

**All gap filling flows through kintsugi.** Direct `populate_cache_resumable()` is guarded to max 366 days. Historical backfill uses kintsugi catch-up mode, which enforces recency-first ordering at every level.

### Reverse Chronological Invariant (3 Levels)

| Level                      | What                              | How                                                                                        |
| -------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------ |
| **L1: Shard selection**    | Which gap to repair first         | `_shard_sort_key()` sorts by `-gap_end_ms`                                                 |
| **L2: Segment processing** | Within a gap, which month first   | `reversed(segments)` — newest Ouroboros segment first                                      |
| **L3: Day processing**     | Within a segment, which day first | Chronological (required for processor state), but L1+L2 ensure newest _segment_ runs first |

### Polling Intervals

The CLI (`scripts/kintsugi.py`) defaults to aggressive intervals for all deployments:

| Parameter           | CLI Default | Python API Default |
| ------------------- | ----------- | ------------------ |
| `--interval-active` | 60s         | 900s               |
| `--interval-clean`  | 300s        | 1800s              |
| `--max-repairs`     | 10          | 3                  |

Both `kintsugi` and `kintsugi-catchup` systemd services use the CLI defaults (identical `ExecStart`). The two services exist for operational control (`Conflicts=` prevents concurrent runs) — not for different modes. **Reverse-chronological processing is the only mode.**

### Usage

```bash
# Normal steady-state healing
mise run kintsugi:daemon

# Aggressive historical backfill (start manually, stop when caught up)
mise run kintsugi:catchup

# Single-job bounded population (max 366 days)
uv run python scripts/populate_full_cache.py \
    --symbol BTCUSDT --threshold 250 \
    --start-date 2025-01-01 --end-date 2025-12-31
```

---

## Monitoring

```bash
# Cache status
mise run cache:status                 # ClickHouse bar counts
python scripts/cache_status.py        # Direct

# System resources (bigblack)
ssh bigblack 'uptime && free -h'
```

---

## Gap Detection & Monitoring

Automated gap detection to prevent silent data loss in ClickHouse. Born from the Feb 2026 BTCUSDT@500 incident (5.5-day gap caused by `try_cache_write()` silently swallowing SSH tunnel failures).

### Scripts

| Script                           | Purpose                                            | Exit Code              |
| -------------------------------- | -------------------------------------------------- | ---------------------- |
| `detect_gaps.py`                 | Find temporal/price gaps + freshness in ClickHouse | 0=clean, 1=gaps, 2=err |
| `verify_checkpoint_integrity.py` | Cross-validate checkpoint files vs ClickHouse      | 0=clean, 1=mismatch    |
| `setup_daily_gap_check.sh`       | Install daily pueue monitoring job on bigblack     | —                      |

### Usage

```bash
# Direct script usage (no mise task exists)
python scripts/detect_gaps.py --help
python scripts/detect_gaps.py --symbol BTCUSDT --threshold 500
python scripts/detect_gaps.py --recent-days 7 --json

# Freshness check (flag stale data >72 hours since last bar)
python scripts/detect_gaps.py --recent-days 7 --max-stale-hours 72

# Checkpoint integrity
python scripts/verify_checkpoint_integrity.py --quiet
```

### Freshness Detection

The `--max-stale-hours N` flag computes `now - latest_bar` for each (symbol, threshold) pair. If any pair hasn't produced a bar in N hours, it's flagged as stale and triggers exit code 1. The heartbeat uses `--max-stale-hours 72` (3 days).

This catches scenarios where ALL thresholds for a symbol stop simultaneously (e.g., backfill_watcher not running) — something inter-bar gap detection alone misses because it only checks gaps between consecutive bars within existing data.

### Telegram Alerts (@obdpy_bot)

Gap detection integrates with `opendeviationbar.notify.telegram`:

- **Gaps found** → persistent alert (stays forever, LOUD notification)
- **All clear** → no message (or ephemeral heartbeat via launchd)
- **`--no-notify`** flag suppresses Telegram (for dry runs)

### Heartbeat (systemd timer — bigblack only)

`scripts/health_heartbeat.py` runs every 5 minutes via `opendeviationbar-heartbeat.timer`. Self-contained (no package imports for Telegram API — avoids version mismatch with installed PyPI package).

| Check           | Method                              | Threshold   |
| --------------- | ----------------------------------- | ----------- |
| ClickHouse      | `SELECT 1` via `clickhouse_connect` | Reachable   |
| sidecar         | `systemctl --user is-active`        | Active      |
| kintsugi        | `systemctl --user is-active`        | Active      |
| Stathera audit  | `lagInFrame()` gap + overlap query  | 0 anomalies |
| Disk            | `shutil.disk_usage("/")`            | >10 GB free |
| Circuit breaker | `get_fatal_write_breaker().state`   | Not OPEN    |

**Healthy**: Ephemeral message, tracked in `~/.cache/opendeviationbar/heartbeat/pending_messages.json`, auto-deleted after 15 min (~2-3 visible at any time).

**Unhealthy**: LOUD persistent message, never auto-deleted.

**Systemd files**: `scripts/systemd/opendeviationbar-heartbeat.{service,timer}` | [systemd CLAUDE.md](/scripts/systemd/CLAUDE.md)

### Gap Detection Query (SSoT)

Uses ClickHouse `lagInFrame()` window function to compute trade-ID deltas within each `(symbol, threshold)` partition. `tid_delta > 0` = gap, `tid_delta < 0` = overlap.

---

## Kintsugi

Unified gap reconciler + trailing-edge fill (Issues #115, #238). Discovers interior gaps ("shards") in the ClickHouse cache and repairs them, plus fills the trailing edge (latest bar → now). All gap filling consolidated in kintsugi daemon since #238 Phase 2.

| Script/File        | Purpose                                |
| ------------------ | -------------------------------------- |
| `kintsugi.py`      | CLI entry point + daemon mode          |
| `kintsugi.toml`    | mise task definitions (`.mise/tasks/`) |
| `kintsugi.service` | systemd unit (`scripts/systemd/`)      |

### Core engine: `python/opendeviationbar/kintsugi.py`

- `discover_shards()` — Stathera trade-ID audit via direct ClickHouse query (Issue #158)
- `repair_shard()` — All repairs via `backfill_gap()` (Vision-first + REST fallback)
- `verify_repair()` — Stathera verification via ClickHouse (checks trade-ID gap is healed)
- `kintsugi_pass()` — single discover-repair-verify cycle
- `kintsugi_daemon()` — long-running adaptive polling (30 min clean, 15 min active)
- `kintsugi_daemon()` — long-running adaptive polling (always reverse-chronological)

### Safeguards

- Max 3 repair attempts per shard per 24h (rate-limited via `kintsugi_log`)
- Repairs capped by `--max-repairs` (default 3)
- Sidecar dead-letter replay on daemon startup
- Telegram alerts on repair failures

---

## Stathera (σταθερά)

Trade-ID alignment paradigm (Issue #158). Replaces all time-based gap detection with aggregate trade-ID continuity checking. The invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for all consecutive bars within the same (symbol, threshold) pair.

### Core Concepts

| Concept                | Definition                                                                                              |
| ---------------------- | ------------------------------------------------------------------------------------------------------- |
| **Stathera Invariant** | Trade-ID continuity: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for consecutive bars |
| **tid_delta**          | `first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) - 1` per bar. >0 = gap, =0 = clean, <0 = overlap |
| **Stathera Gap**       | A violation where `tid_delta > 0` (missing aggTrade records between consecutive bars)                   |
| **Stathera Overlap**   | A violation where `tid_delta < 0` (bars from different sources share trade-ID ranges)                   |

### Scripts

| Script                  | Purpose                                                                              |
| ----------------------- | ------------------------------------------------------------------------------------ |
| `detect_gaps.py`        | Trade-ID audit for all (symbol, threshold) pairs. Exit code 1 if gaps/overlaps found |
| `detect_gaps.py --json` | Machine-readable Stathera audit output                                               |

### How Stathera Audit Works

1. **Audit Query** (`detect_gaps.py`):
   - Window function ordered by `(first_agg_trade_id, close_time_ms)` (critical: fixes overlap detection)
   - Computes `tid_delta = first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) - 1`
   - Classifies: gap (>0), clean (=0), overlap (<0), legacy (first_agg_trade_id=0)

2. **Overlap Repair** (kintsugi day-recompute):
   - Both gaps (`tid_delta > 0`) and overlaps (`tid_delta < 0`) trigger day-recompute
   - DELETE entire affected Ouroboros day(s), reprocess from scratch via Ariadne
   - Absorbed from legacy `stathera_cleanup.py` in #238 Phase 1

3. **Source Tagging** (Phase 2):
   - Each bar's `opendeviationbar_version` encoded as `{version}+{source}`:
     - `+sidecar` — single writer (cannot overlap itself)
     - `+backfill` — REST backfill (skip if sidecar already wrote)
     - `+kintsugi` — repair processing (authoritative, can replace)
     - `+populate` — cache population (warn on overlaps, investigate)

### Integration with Kintsugi

- `_discover_shards_stathera()` — direct ClickHouse Stathera audit, no subprocess
- Per-gap trade-ID anchors (`anchor_last_tid`, `resume_first_tid`) replace global high-water-mark
- All repairs via `backfill_gap()` (Vision-first + REST fallback)

### Verification

```bash
# Local
python scripts/detect_gaps.py --json               # Audit all pairs

# Bigblack
python scripts/detect_gaps.py --symbol BTCUSDT --threshold 250
```

---

## Streaming Reliability Concepts

The sidecar uses a layered self-healing architecture with named concepts from Greek mythology:

| Concept      | Layer      | Purpose                                                                                      |
| ------------ | ---------- | -------------------------------------------------------------------------------------------- |
| **Antaeus**  | Rust (WS)  | Resilient reconnection: `backon` exponential backoff with jitter, resets after stable (≥60s) |
| **Lethe**    | Python     | Ring buffer overflow alerting: detects `dropped_bars` metric increases, Telegram alerts      |
| **Argus**    | Python     | Per-symbol liveness: tracks last bar time per symbol, alerts if silent ≥30 min               |
| **Charon**   | Python     | Dead-letter ferry: auto-replays failed Parquet writes on startup + every 5 min               |
| **Ariadne**  | Python     | `fromId` REST pagination for gap-fill: trade-ID anchored, zero-gap by construction           |
| **Kintsugi** | Python     | Self-healing gap reconciliation via Stathera trade-ID audit + Ariadne repair                 |
| **Stathera** | ClickHouse | Trade-ID alignment invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id`  |

### Configuration (Environment Variables)

| Variable                                 | Default | Purpose                         |
| ---------------------------------------- | ------- | ------------------------------- |
| `OPENDEVIATIONBAR_LETHE_ALERT_THRESHOLD` | `1`     | Alert on first ring buffer drop |
| `OPENDEVIATIONBAR_ARGUS_TIMEOUT_S`       | `1800`  | Per-symbol silence timeout (s)  |
| `OPENDEVIATIONBAR_CHARON_INTERVAL_S`     | `300`   | Dead-letter replay interval (s) |

---

## Validation Scripts

Portable scripts for GPU workstations without full dev environment:

| Script                                 | Purpose                             |
| -------------------------------------- | ----------------------------------- |
| `validate_n_open_deviation_bars.py`    | Count-bounded API validation        |
| `validate_microstructure_features.py`  | v7.0 feature validation             |
| `validate_microstructure_roundtrip.py` | Compute → store → read → verify     |
| `validate_backfill_preflight.py`       | Pre-population checks               |
| `validate_clickhouse.py`               | ClickHouse connectivity             |
| `validate_memory_efficiency.py`        | Memory usage during processing      |
| `deduplicate_parquet_cache.py`         | Fix pre-v12.8 duplicate ticks (#78) |
| `detect_volume_overflow.py`            | Find negative volumes (#88)         |
| `validate_dedup_hardening.py`          | Issue #90 dedup layer validation    |

---

## Related

- [/CLAUDE.md](/CLAUDE.md) - Project hub
- [/crates/CLAUDE.md](/crates/CLAUDE.md) - Rust crate details, microstructure features
- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) - Python API, caching, validation
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache layer
- mise task definitions (cache status, kintsugi)
