# Verification Checklist (73 Points)

**Skill**: [odb-ship](../SKILL.md)

Run after every release, deploy, or package update. Group SSH commands to minimize round-trips.

---

## Section 0: Network & Connectivity (5 checks)

### 0.1 GitHub SSH

```bash
ssh -o ConnectTimeout=5 -T git@github.com 2>&1 | head -3
```

### 0.2 Bigblack SSH

```bash
ssh -o ConnectTimeout=5 bigblack 'echo OK'
```

### 0.3 GitHub HTTPS Fallback

```bash
if ! ssh -o ConnectTimeout=5 -T git@github.com 2>&1 | grep -q "successfully authenticated"; then
  git remote set-url origin "https://github.com/terrylica/opendeviationbar-py.git"
  echo "Swapped to HTTPS. Restore after: git remote set-url origin git@github.com-terrylica:terrylica/opendeviationbar-py.git"
fi
```

### 0.4 GitHub API (HTTPS)

```bash
gh api repos/terrylica/opendeviationbar-py --jq '.pushed_at'
```

### 0.5 semantic-release SSH dependency

semantic-release runs `git ls-remote --heads <remote>` — swap to HTTPS if SSH is blocked (Section 0.3) before running `semantic-release --no-ci`.

---

## Section 1: Package & Version (9 checks)

### 1.1 Python package version

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)"'
```

### 1.2 Cargo.toml workspace version

```bash
grep '^version' Cargo.toml | head -1
```

### 1.3 PyPI published version

```bash
uv pip index versions opendeviationbar 2>/dev/null | head -5
```

### 1.4 Crates.io published versions

```bash
for crate in opendeviationbar-core opendeviationbar-providers opendeviationbar-streaming opendeviationbar-client opendeviationbar-hurst; do
  echo "$crate: $(cargo search $crate --limit 1 2>/dev/null | head -1)"
done
```

### 1.5 Git state (local + remote)

```bash
git log -1 --oneline && git describe --tags --abbrev=0 && git status --porcelain
ssh bigblack 'cd ~/opendeviationbar-py && git log -1 --oneline && git describe --tags --abbrev=0'
```

### 1.6 Local vs remote git SHA

```bash
LOCAL_SHA=$(git rev-parse --short HEAD)
REMOTE_SHA=$(ssh bigblack 'cd ~/opendeviationbar-py && git rev-parse --short HEAD')
echo "Local=$LOCAL_SHA Remote=$REMOTE_SHA"
[ "$LOCAL_SHA" = "$REMOTE_SHA" ] && echo "MATCH" || echo "DIVERGED"
```

### 1.7 Wheel architecture

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import opendeviationbar._core as c; import pathlib
so = pathlib.Path(c.__file__)
print(f\"Extension: {so.name}, Size: {so.stat().st_size / 1024 / 1024:.1f} MB\")
"'
```

### 1.8 Full version alignment (cross-check all sources)

```bash
ssh bigblack '
VER=$(~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)")
cd ~/opendeviationbar-py
TAG=$(git describe --tags --abbrev=0 2>/dev/null | sed "s/^v//")
CARGO=$(grep "^version" Cargo.toml | head -1 | sed "s/.*\"\(.*\)\"/\1/")
echo "Package=$VER Git=$TAG Cargo=$CARGO"

SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
for proc in "streaming_sidecar" "kintsugi"; do
  PID=$(pgrep -f "$proc" | head -1)
  if [ -n "$PID" ]; then
    START_EPOCH=$(date -d "$(ps -o lstart= -p $PID)" +%s 2>/dev/null)
    if [ -n "$START_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$START_EPOCH" -lt "$SO_EPOCH" ]; then
      echo "STALE .so: $proc (PID $PID) started before package install"
    fi
  fi
done

svc_state="ALIGNED"
[ "$VER" != "$CARGO" ] && svc_state="MISALIGNED"
[ -n "$TAG" ] && [ "$VER" != "$TAG" ] && svc_state="MISALIGNED"
echo "$svc_state"
'
```

### 1.9 Install mode (wheel vs editable)

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import subprocess, sys
r = subprocess.run([\"uv\", \"pip\", \"show\", \"--python\", sys.executable, \"opendeviationbar\"], capture_output=True, text=True)
print(\"editable\" if \"Editable project location\" in r.stdout else \"wheel\")
"'
```

---

## Section 2: Native Extension (6 checks)

### 2.1 Native .so import

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar._core import PyOpenDeviationBarProcessor
p = PyOpenDeviationBarProcessor(250)
print(f\"Processor OK: threshold={p.threshold_decimal_bps}\")
"'
```

### 2.2 Stale .so detection

See Section 1.8 (automatic timestamp comparison with STALE warnings).

### 2.3 Circular import test

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar import get_open_deviation_bars, process_trades_to_dataframe
from opendeviationbar.clickhouse import OpenDeviationBarCache
from opendeviationbar.clickhouse.constants import CLICKHOUSE_TRANSIENT_ERRORS
print(\"All imports OK\")
"'
```

### 2.4 DRY constants

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.constants import DEFAULT_THRESHOLD
from opendeviationbar.symbol_registry import get_thresholds_for_symbol
assert DEFAULT_THRESHOLD == 250
assert get_thresholds_for_symbol(\"BTCUSDT\") == (100, 250, 500, 750)
print(\"Constants verified\")
"'
```

### 2.5 Config defaults from symbol registry

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.config.sidecar import SidecarConfig
from opendeviationbar.symbol_registry import get_all_symbol_threshold_pairs
cfg = SidecarConfig()
expected = sorted({t for _, t in get_all_symbol_threshold_pairs(enabled_only=True)})
assert cfg.thresholds == expected, f\"Expected {expected}, got {cfg.thresholds}\"
print(f\"SidecarConfig.thresholds: {cfg.thresholds}\")
"'
```

### 2.6 Code verification (key entry points exist)

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
import importlib
sidecar = importlib.import_module(\"opendeviationbar.sidecar\")
assert hasattr(sidecar, \"run_sidecar\")
api = importlib.import_module(\"opendeviationbar.sidecar_api\")
print(\"Entry points verified\")
"'
```

---

## Section 3: Process Management (6 checks)

### 3.1 Service status

Note: On bigblack (zsh), `status` is read-only. Use `svc_state`.

```bash
ssh bigblack '
for svc in opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer opendeviationbar-seeder.timer; do
  svc_state=$(systemctl --user is-active "$svc" 2>/dev/null || echo "unknown")
  echo "$svc: $svc_state"
done
'
```

### 3.2 Memory vs MemoryMax

| Service   | MemoryMax |
| --------- | --------- |
| sidecar   | 4G        |
| kintsugi  | 20G       |
| heartbeat | 256M      |

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  mem=$(systemctl --user show $svc --property=MemoryCurrent --value 2>/dev/null)
  max=$(systemctl --user show $svc --property=MemoryMax --value 2>/dev/null)
  echo "$svc: current=$mem max=$max"
done'
```

### 3.3 Restart count

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  restarts=$(systemctl --user show $svc --property=NRestarts --value 2>/dev/null)
  echo "$svc restarts: ${restarts:-0}"
done'
```

### 3.4 Uptime

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  echo "$svc: $(systemctl --user show $svc --property=ActiveEnterTimestamp --value)"
done'
```

### 3.5 Journal error scan (last 1h)

```bash
ssh bigblack 'journalctl --user -u "opendeviationbar-*" --since "1 hour ago" --priority=err --no-pager -q 2>/dev/null | tail -20'
```

### 3.6 PID alive check

```bash
ssh bigblack 'for svc in opendeviationbar-sidecar opendeviationbar-kintsugi; do
  pid=$(systemctl --user show $svc --property=MainPID --value)
  if [ "$pid" != "0" ] && [ -d "/proc/$pid" ]; then echo "$svc: PID $pid alive"
  else echo "$svc: PID $pid NOT FOUND"; fi
done'
```

---

## Section 4: ClickHouse Health (8 checks)

### 4.1 Ping

```bash
ssh bigblack 'curl -s "http://localhost:8123/ping"'
```

Expected: `Ok.`

### 4.2 Version

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+version()"'
```

### 4.3 Database exists

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.tables+WHERE+database%3D%27opendeviationbar_cache%27+AND+name%3D%27open_deviation_bars%27"'
```

Expected: `1`

### 4.4 Total bar count

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+opendeviationbar_cache.open_deviation_bars"'
```

Expected: >40M

### 4.5 Per-symbol bar counts

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT symbol, threshold_decimal_bps, count() as bars
FROM opendeviationbar_cache.open_deviation_bars
GROUP BY symbol, threshold_decimal_bps
ORDER BY symbol, threshold_decimal_bps
FORMAT PrettyCompact
"'
```

### 4.6 Schema columns

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT name, type FROM system.columns
WHERE database = '\''opendeviationbar_cache'\'' AND table = '\''open_deviation_bars'\''
ORDER BY position FORMAT PrettyCompact
"'
```

### 4.7 Pending mutations

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.mutations+WHERE+database%3D%27opendeviationbar_cache%27+AND+is_done%3D0"'
```

Expected: `0`

### 4.8 Active merges

```bash
ssh bigblack 'curl -s "http://localhost:8123/?query=SELECT+count()+FROM+system.merges+WHERE+database%3D%27opendeviationbar_cache%27"'
```

---

## Section 5: Stathera Audit (4 checks)

### 5.1 Gap scan (intra-day)

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH bars AS (
    SELECT symbol, threshold_decimal_bps, first_agg_trade_id, last_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) AS prev_last_tid,
           close_time_ms, lagInFrame(close_time_ms, 1) OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) AS prev_close_ms
    FROM opendeviationbar_cache.open_deviation_bars FINAL
)
SELECT symbol, threshold_decimal_bps, count() AS gap_count
FROM bars WHERE prev_last_tid > 0 AND first_agg_trade_id - prev_last_tid > 1
  AND toDate(fromUnixTimestamp64Milli(close_time_ms)) - toDate(fromUnixTimestamp64Milli(prev_close_ms)) <= 0
GROUP BY symbol, threshold_decimal_bps ORDER BY gap_count DESC
FORMAT PrettyCompact
" 2>/dev/null'
```

### 5.2 Overlap scan

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH bars AS (
    SELECT symbol, threshold_decimal_bps, first_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) AS prev_last_tid
    FROM opendeviationbar_cache.open_deviation_bars FINAL
)
SELECT symbol, threshold_decimal_bps, count() AS overlap_count
FROM bars WHERE prev_last_tid > 0 AND first_agg_trade_id <= prev_last_tid
GROUP BY symbol, threshold_decimal_bps HAVING overlap_count > 0
ORDER BY overlap_count DESC FORMAT PrettyCompact
" 2>/dev/null'
```

### 5.3 Kintsugi healing progress (24h)

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT result, count() AS cnt, max(timestamp) AS last_repair
FROM opendeviationbar_cache.kintsugi_log
WHERE timestamp > now() - INTERVAL 24 HOUR
GROUP BY result FORMAT PrettyCompact
" 2>/dev/null'
```

### 5.4 Kintsugi recent entries

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT timestamp, symbol, threshold_decimal_bps, result, bars_written
FROM opendeviationbar_cache.kintsugi_log ORDER BY timestamp DESC LIMIT 10
FORMAT PrettyCompact
" 2>/dev/null'
```

---

## Section 6: Sidecar Endpoints (4 checks)

### 6.1 Health

```bash
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -m json.tool'
```

### 6.2 Ariadne lookup

```bash
ssh bigblack 'curl -sf "http://localhost:8081/ariadne/BTCUSDT/250" | python3 -c "
import sys, json; d=json.load(sys.stdin)
print(f\"Source: {d.get(\"source\",\"?\")}, Bars: {len(d.get(\"bars\",[]))}\")
"'
```

### 6.3 Catchup endpoint

```bash
ssh bigblack 'curl -sf "http://localhost:8081/catchup/BTCUSDT/250?max_trades=1000" | python3 -c "
import sys, json; d=json.load(sys.stdin)
print(f\"Count: {d.get(\"count\",0)}, Partial: {d.get(\"partial\",False)}\")
"'
```

### 6.4 Forming bar status

```bash
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -c "
import sys, json; d=json.load(sys.stdin)
for sym, info in d.get(\"engines\",{}).items():
    print(f\"{sym}: forming={info.get(\"has_forming_bar\",\"?\")}\")
" 2>/dev/null'
```

---

## Section 7: Infrastructure (6 checks)

### 7.1 Disk

```bash
ssh bigblack 'df -h / /home 2>/dev/null | tail -n +2'
```

Root <80%, data <90%.

### 7.2 Memory and swap

```bash
ssh bigblack 'free -h'
```

Swap <25%.

### 7.3 Load average

```bash
ssh bigblack 'uptime'
```

Load <8.

### 7.4 Redis

```bash
ssh bigblack 'redis-cli ping 2>/dev/null || echo "Redis not available (graceful degradation OK)"'
```

### 7.5 Monit

```bash
ssh bigblack 'sudo monit summary 2>/dev/null || echo "Monit not available"'
```

### 7.6 Dead letters

```bash
ssh bigblack 'ls -la /tmp/opendeviationbar-dead-letter/ 2>/dev/null && echo "Dead letters found" || echo "Clean"'
```

---

## Section 8: Monitoring (3 checks)

### 8.1 Heartbeat timer

```bash
ssh bigblack 'systemctl --user status opendeviationbar-heartbeat.timer --no-pager 2>/dev/null | grep -E "Active:|Trigger:"'
```

### 8.2 Circuit breaker

```bash
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -c "
import sys, json; d=json.load(sys.stdin)
print(f\"Circuit breaker: {d.get(\"circuit_breaker\", d.get(\"circuit_breaker_state\", \"not reported\"))}\")" 2>/dev/null'
```

Expected: `CLOSED`.

### 8.3 Monit heartbeat last run

```bash
ssh bigblack 'sudo monit status monit-heartbeat 2>/dev/null | grep -E "last|status"'
```

---

## Section 9: Configuration (5 checks)

### 9.1 Connection mode

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep OPENDEVIATIONBAR_MODE .mise.toml 2>/dev/null'
```

### 9.2 Ouroboros mode

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep OUROBOROS .mise.toml 2>/dev/null'
```

Expected: `day`.

### 9.3 Flush threshold

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep FLUSH_THRESHOLD .mise.toml 2>/dev/null'
```

### 9.4 Symbol registry

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar.symbol_registry import get_registered_symbols
syms = get_registered_symbols()
print(f\"Registered: {len(syms)} symbols\")
"'
```

### 9.5 Threshold configuration

```bash
ssh bigblack 'cd ~/opendeviationbar-py && grep -E "CRYPTO_MIN_THRESHOLD|thresholds" .mise.toml 2>/dev/null'
```

### 9.6 Kintsugi parallelism (bigblack sweet spot: 24)

```bash
ssh bigblack 'grep -E "KINTSUGI_MAX_CONCURRENT_REPAIRS|KINTSUGI_MAX_CONCURRENT_PARQUET|KINTSUGI_BATCH_WORKERS|KINTSUGI_BATCH_DAYS" ~/opendeviationbar-py/deploy/opendeviationbar.env'
```

Expected: `MAX_CONCURRENT_REPAIRS=24`, `MAX_CONCURRENT_PARQUET=64`, `BATCH_WORKERS=4`, `BATCH_DAYS=365`. Do NOT increase MAX_CONCURRENT_REPAIRS beyond 24 — empirically verified cliff at 32 (ClickHouse query scheduler contention).

### 9.7 Redis lock timeout (Issue #266: must be 2, not 60)

```bash
ssh bigblack 'grep REDIS_LOCK_BLOCKING_TIMEOUT ~/opendeviationbar-py/deploy/opendeviationbar.env'
```

Expected: `OPENDEVIATIONBAR_REDIS_LOCK_BLOCKING_TIMEOUT=2`. Value of 60 serializes all workers to effective parallelism=1.

---

## Section 10: Release Pipeline Pitfalls (7 checks)

### 10.1 Version file consistency

After semantic-release, all sources must agree: `Cargo.toml`, `git tag`, Python `__version__`.
Local Python `__version__` may lag — run `maturin develop` to sync.

### 10.2 Crates.io "already exists"

If `release:crates` races with version bump, old version tries to publish. Re-run after Cargo.toml has new version.

### 10.3 semantic-release SSH dependency

Uses `git ls-remote --heads` — needs SSH or HTTPS. Swap remote if SSH blocked.

### 10.4 semantic-release dry-run

`--no-ci` required when running locally. Without it, no tags/versions are created.

### 10.5 PyPI CDN propagation

30-60s delay after upload. Check `https://pypi.org/project/opendeviationbar/X.Y.Z/` directly.

### 10.6 Git push HTTPS fallback

```bash
GH_TOKEN=$(gh auth token) git \
  -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
  push origin main
```

### 10.7 Local .so version lags

After semantic-release bumps Cargo.toml, local `__version__` still shows old version. Run `maturin develop` to sync. Bigblack is always correct after `uv pip install`.
