# odb-validation: Runtime Health Checks for opendeviationbar on bigblack

Comprehensive validation suite that runs direct-query checks against ClickHouse, kintsugi.ndjson, and Redis to surface known failure modes before they become production incidents. Every check name encodes the mythological guardian's identity AND the exact failure mode it detects.

## When to Use

- After any kintsugi restart or deploy
- When kintsugi throughput appears low
- When heartbeat reports UNHEALTHY
- After a `TRUNCATE TABLE kintsugi_log` or `ALTER TABLE DELETE` operation
- Routine monitoring (run weekly or after any parallelism tuning)

## Quick Start

```bash
# Full suite (run from your Mac — executes on bigblack)
ssh bigblack 'bash ~/opendeviationbar-py/.claude/skills/odb-validation/scripts/validate.sh'

# Specific checks only
ssh bigblack 'bash ~/opendeviationbar-py/.claude/skills/odb-validation/scripts/validate.sh aletheia-concurrent-worker-cascade-heal-race-artifact-detector nemesis-kintsugi-repair-throughput-regression-and-near-stall-detector'

# From bigblack directly (if already SSH'd in)
bash ~/opendeviationbar-py/.claude/skills/odb-validation/scripts/validate.sh
```

Exit codes: `0` = all pass, `1` = warnings only, `2` = at least one failure

---

## The Six Checks

Each check lives in `scripts/checks/` and can be run standalone. The filename IS the description.

### 1. `aletheia-concurrent-worker-cascade-heal-race-artifact-detector.sh`

**What it detects**: When N concurrent kintsugi workers process shards for the same (symbol, threshold), one worker's Vision ZIP download fills adjacent shards. The remaining workers log `clean+0-bar` — race artifacts that trigger the 30-day dismissal window and block thousands of valid historical shards from re-queuing.

**Heuristic**: `(symbol, threshold)` pairs that have both `healed` and `clean+0-bar` entries within the same 2-hour window. A genuine sub-threshold day cannot coincide with active healing of the same pair.

**Known limitation at high throughput**: At ≥370 repairs/hr (Vision-mode) or ≥5000+/hr (Parquet-cached mode at MAX_CONCURRENT_REPAIRS=24), healing is near-continuous. The 2h coincidence window produces false positives (any clean+0-bar entry will coincide with some healing). At this throughput, WARN on 3-10 artifacts should be treated as informational.

**Escalation**: WARN >3 artifacts, FAIL >10 artifacts, WARN >200 unique dismissed shards

**Fix**: Set `Environment=OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REPAIRS=1` in focus-mode drop-in. `_aletheia_purge()` in `kintsugi.py` auto-clears artifacts before each dismissed-set query.

---

### 2. `tantalus-redis-write-lock-serialization-timeout-contention-auditor.sh`

**What it detects**: If `OPENDEVIATIONBAR_REDIS_LOCK_BLOCKING_TIMEOUT` is too high (≥30s), concurrent kintsugi workers queue deterministically behind the same Redis write lock — defeating parallelism entirely. The incident value that caused throughput collapse was 60s.

**Also checks**: Live Redis `blocked_clients` count as a runtime proxy for active lock contention.

**Escalation**: WARN ≥10s, FAIL ≥30s, WARN >2 blocked clients, FAIL >5 blocked clients

**Safe value**: 2s (current deploy/opendeviationbar.env setting)

---

### 3. `charybdis-kintsugi-repair-total-duration-and-vision-download-saturation-monitor.sh`

**What it detects**: Total per-repair duration (`duration_s` from `repair_complete` events in kintsugi.ndjson). Under Vision-mode with single outer worker, typical repair duration is 10-60s for a full day's shard. Spikes above 120s (median) indicate Vision download saturation from USB ethernet or network congestion.

**Note on field**: Uses `duration_s` (total repair time: download + Rust processing + ClickHouse insert combined). No separate download-only field exists in current kintsugi NDJSON schema.

**Baselines**: Vision-mode: 10-60s/repair, ~83/hr | Parquet-cached: 0.5-2s/repair, ~5000+/hr (empirical: 44 repairs/30s at MAX_CONCURRENT_REPAIRS=24, bigblack 2026-03-13)

**Escalation**: WARN median >60s, FAIL median >120s

---

### 4. `nemesis-kintsugi-repair-throughput-regression-and-near-stall-detector.sh`

**What it detects**: Throughput regressions by comparing repairs/hr over 1h and 6h windows against known baselines. Catches three failure modes:

- **Blocked stall**: 0 repairs + high dismissed-shard count (dismissals blocking all work)
- **Recent stop**: Repairs ran in 6h window but stopped in last hour (crash, queue exhaustion, or stall)
- **Severe degradation**: Rate <10/hr while repairs are running (Redis lock contention or dismissal buildup)

**Baselines**: Vision-mode ~83/hr, Parquet-cached ~5000+/hr (empirical: 44 repairs/30s at MAX_CONCURRENT_REPAIRS=24)

---

### 5. `mnemosyne-kintsugi-log-stale-shard-dismissal-accumulation-auditor.sh`

**What it detects**: Accumulation of `clean+0-bar` entries in `kintsugi_log` that suppress shard re-repair for 30 days. Under normal operation with Aletheia running, this count should stay near zero. High accumulation means Aletheia is not clearing artifacts fast enough, or genuinely sub-threshold shards are present.

**Also reports**: Total table size and oldest entry date (useful for deciding whether to `TRUNCATE TABLE kintsugi_log`).

**Escalation**: WARN >50 unique dismissed shards, FAIL >200

---

### 6. `clickhouse-mergetree-active-parts-and-merge-queue-pressure-monitor.sh`

**What it detects**: Active parts count and pending merge queue depth in the `opendeviationbar_cache` database. Under heavy kintsugi throughput (370+/hr), each INSERT creates parts that must be merged in the background. Approaching `parts_to_delay_insert` (1000) causes write slowdown; approaching `parts_to_throw_insert` (3000) causes write failures.

**Note on 700-range active_parts during heavy backfill**: At 370 repairs/hr, 700-800 active parts is expected and not immediately dangerous (threshold is 1000). WARN fires at >500 per the perf-analysis skill capacity model as a conservative early warning.

**Escalation**: WARN active_parts >500 or pending_merges >5, FAIL active_parts >800 or pending_merges >10

---

## Verdict Interpretation

| All PASS        | System healthy. Continue monitoring.                                                   |
| --------------- | -------------------------------------------------------------------------------------- |
| Aletheia WARN   | Check throughput rate first. At ≥200/hr, small WARN counts are likely false positives. |
| Tantalus FAIL   | Urgent — reduce `BLOCKING_TIMEOUT` to 2s immediately, restart kintsugi.                |
| Charybdis WARN  | Reduce `MAX_CONCURRENT_REPAIRS` by 2, wait 15 min, re-run.                             |
| Nemesis FAIL    | Check Aletheia first (dismissal stall), then Tantalus (lock contention).               |
| Mnemosyne WARN  | Run Aletheia check; if no artifacts, genuine sub-threshold data — expected.            |
| ClickHouse WARN | Hold parallelism. Normal during heavy backfill if pending_merges = 0.                  |
| ClickHouse FAIL | Reduce `MAX_CONCURRENT_REPAIRS` by 50% immediately. Wait for parts to merge.           |

---

## Directory Structure

```
.claude/skills/odb-validation/
├── SKILL.md                          ← this file
├── scripts/
│   ├── validate.sh                   ← main orchestrator
│   └── checks/
│       ├── aletheia-concurrent-worker-cascade-heal-race-artifact-detector.sh
│       ├── tantalus-redis-write-lock-serialization-timeout-contention-auditor.sh
│       ├── charybdis-kintsugi-repair-total-duration-and-vision-download-saturation-monitor.sh
│       ├── nemesis-kintsugi-repair-throughput-regression-and-near-stall-detector.sh
│       ├── mnemosyne-kintsugi-log-stale-shard-dismissal-accumulation-auditor.sh
│       └── clickhouse-mergetree-active-parts-and-merge-queue-pressure-monitor.sh
└── references/
    └── thresholds.md                 ← escalation threshold reference
```

---

## Related Mechanisms in kintsugi.py

| Python Function        | Description                                                                  |
| ---------------------- | ---------------------------------------------------------------------------- |
| `_aletheia_purge()`    | Auto-clears race artifacts before building dismissed-shard set               |
| `_get_dismissed_set()` | Queries kintsugi_log for 30-day dismissal window (calls `_aletheia_purge()`) |
| `_repair_under_lock()` | Crash-safe per-chunk replace with Stathera verification                      |

## Related Skills

| Skill              | Use When                                                         |
| ------------------ | ---------------------------------------------------------------- |
| `/odb-telemetry`   | Reading production logs, heartbeat interpretation, full CH audit |
| `/perf-analysis`   | Calculating safe MAX_CONCURRENT_REPAIRS for current hardware     |
| `/odb-ship verify` | Post-deploy 73-point production health check                     |
