---
name: odb-telemetry
description: >-
  Retrieve and interpret opendeviationbar production telemetry from bigblack.
  Covers heartbeat messages, kintsugi repair logs, Stathera integrity, sidecar
  health, ClickHouse stats, cross-midnight oracle, symbol registry compliance,
  GPU utilization, monit watchdog state, and system metrics. Use whenever the
  user asks about production health, recent alerts, repair status, gap counts,
  what the Telegram bot is reporting, what's happening on bigblack, monit
  status, GPU usage, or ODB status. Also use when the user pastes a Telegram
  heartbeat message and wants to understand it or check the source data. Do NOT
  use for deploying, releasing, or modifying production services.
allowed-tools: Read, Bash, Grep, Glob, Agent
---

# ODB Telemetry

Retrieve and interpret opendeviationbar production telemetry from bigblack (el02). All data reachable via `ssh bigblack`.

**Progressive disclosure**: Start with Tier 1 (single commands). Escalate to Tier 2 if anomalies appear. Tier 3 for deep investigation.

---

## Tier 1 — Single-Command Snapshot

### Full system snapshot (run this first)

```bash
ssh bigblack '
echo "=== HEARTBEAT (last 15 min) ===" && journalctl --user -u opendeviationbar-heartbeat --since "15 min ago" --no-pager 2>/dev/null | grep -v "^$" | tail -20
echo "" && echo "=== MONIT ===" && sudo monit summary 2>/dev/null
echo "" && echo "=== SIDECAR HEALTH ===" && curl -s http://localhost:8081/health | python3 -m json.tool 2>/dev/null
echo "" && echo "=== SYSTEM ===" && echo "Disk: $(df -h / | tail -1 | awk "{print \$3\"/\"\$2\" (\"\$5\")\"}") | Mem: $(free -h | awk "/^Mem/{print \$3\"/\"\$2}") | Load: $(uptime | awk -F\"load average:\" \"{print \$2}\")"
echo "" && echo "=== GPU ===" && nvidia-smi --query-gpu=name,utilization.gpu,utilization.memory,memory.used,temperature.gpu --format=csv,noheader 2>/dev/null || echo "no nvidia-smi"
'
```

### "Is everything healthy?"

```bash
ssh bigblack 'journalctl --user -u opendeviationbar-heartbeat --since "10 min ago" --no-pager | grep -E "HEALTHY|UNHEALTHY|WARNING|ERROR|cross-midnight|kintsugi"'
```

### "What did the last heartbeat say?"

```bash
ssh bigblack 'journalctl --user -u opendeviationbar-heartbeat --since "6 min ago" --no-pager'
```

Heartbeat runs every 5 minutes. If no output, widen to `--since "15 min ago"`.

### "How many cross-midnight bars?"

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT count() AS cross_midnight_bars
FROM opendeviationbar_cache.open_deviation_bars FINAL
WHERE toDate(open_time_ms / 1000, '"'"'UTC'"'"') <> toDate(close_time_ms / 1000, '"'"'UTC'"'"')
FORMAT PrettyCompact
"'
```

### "What's kintsugi doing?"

```bash
ssh bigblack 'echo "=== Recent NDJSON events ===" && tail -20 ~/opendeviationbar-py/logs/kintsugi.ndjson 2>/dev/null | python3 -c "
import sys, json
for line in sys.stdin:
    try:
        e = json.loads(line)
        ts = e.get(\"timestamp\",\"?\")[:19]; ev = e.get(\"event\",\"?\")
        sym = e.get(\"symbol\",\"\"); res = e.get(\"result\",\"\")
        bars = e.get(\"bars_written\",\"\"); shards = e.get(\"total_shards\",\"\")
        healed = e.get(\"healed\",\"\"); failed = e.get(\"failed\",\"\")
        print(f\"{ts} [{ev}] {sym} res={res} bars={bars} shards={shards} healed={healed} failed={failed}\")
    except: print(line.strip())
" && echo "" && echo "=== Journal (last 20) ===" && journalctl --user -u opendeviationbar-kintsugi -n 20 --no-pager'
```

### "Monit status?"

```bash
ssh bigblack 'sudo monit summary 2>/dev/null'
```

### "System resources?"

```bash
ssh bigblack 'echo "Disk:"; df -h / | tail -1; echo "Memory:"; free -h | head -2; echo "Load:"; uptime; echo "GPU:"; nvidia-smi --query-gpu=name,utilization.gpu,utilization.memory,memory.used,memory.free,temperature.gpu --format=csv,noheader 2>/dev/null || echo "no GPU data"'
```

### "Are we using bigblack's hardware?"

Quick utilization check against the capacity envelope (32 cores, 61GB RAM, NVMe, 1Gbps):

```bash
ssh bigblack '
echo "=== CPU ===" && nproc && uptime
echo "" && echo "=== MEMORY ===" && free -h | head -2
echo "" && echo "=== FD USAGE (top 5 processes) ===" && for pid in $(pgrep -f "kintsugi|sidecar|seeder|repair" 2>/dev/null | head -5); do echo "PID $pid ($(cat /proc/$pid/cmdline 2>/dev/null | tr "\0" " " | cut -c-60)): $(ls /proc/$pid/fd 2>/dev/null | wc -l) fds"; done
echo "" && echo "=== CH ACTIVE QUERIES ===" && curl -s "http://localhost:8123/" --data-binary "SELECT count() AS active FROM system.processes" 2>/dev/null
echo "" && echo "=== KINTSUGI WORKERS ===" && journalctl --user -u opendeviationbar-kintsugi -n 10 --no-pager 2>/dev/null | grep -oP "parallel batch of \d+ days with \d+ workers" | tail -1
'
```

**Expected healthy ranges** (see [scripts/CLAUDE.md Bigblack Capacity Envelope](/scripts/CLAUDE.md#bigblack-capacity-envelope)):

- CPU: 20-25% at 24 kintsugi workers (ClickHouse is the bottleneck, not CPU)
- Memory: <10 GB used by ODB services combined
- CH active queries: ~143 at peak (soft cap; >200 = contention)
- fds per process: <200 (thread-local CH cache keeps it at ~4N+50)

---

## Tier 2 — Per-Subsystem Deep Dives

### ClickHouse — Per-Pair Freshness (full, all symbols)

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT symbol, threshold_decimal_bps, count() AS total_bars,
       round(dateDiff('"'"'second'"'"', fromUnixTimestamp64Milli(toInt64(max(close_time_ms))), now64(3, '"'"'UTC'"'"')) / 60, 1) AS minutes_stale
FROM opendeviationbar_cache.open_deviation_bars FINAL
GROUP BY symbol, threshold_decimal_bps
ORDER BY symbol, threshold_decimal_bps
FORMAT PrettyCompact
"'
```

### ClickHouse — 24h Growth Rate

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT symbol, threshold_decimal_bps, count() AS bars_last_24h
FROM opendeviationbar_cache.open_deviation_bars
WHERE close_time_ms > (toUnixTimestamp(now()) - 86400) * 1000
GROUP BY symbol, threshold_decimal_bps ORDER BY bars_last_24h DESC
FORMAT PrettyCompact
"'
```

### ClickHouse — Stalest Pairs

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT symbol, threshold_decimal_bps,
       round(dateDiff('"'"'second'"'"', fromUnixTimestamp64Milli(toInt64(max(close_time_ms))), now64(3, '"'"'UTC'"'"')) / 3600, 1) AS hours_stale
FROM opendeviationbar_cache.open_deviation_bars FINAL
GROUP BY symbol, threshold_decimal_bps ORDER BY hours_stale DESC LIMIT 10
FORMAT PrettyCompact
"'
```

### Kintsugi — Repair History (last 3 days, per symbol+threshold)

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT toDate(timestamp) AS day, symbol, threshold_decimal_bps,
       count() AS repairs, sum(bars_written) AS bars_written
FROM opendeviationbar_cache.kintsugi_log
WHERE timestamp >= now() - INTERVAL 3 DAY
GROUP BY day, symbol, threshold_decimal_bps ORDER BY day DESC, bars_written DESC
FORMAT PrettyCompact
"'
```

### Kintsugi — NDJSON filter by current PID

```bash
ssh bigblack 'PID=$(systemctl --user show opendeviationbar-kintsugi --property=MainPID --value); grep "\"pid\":$PID" ~/opendeviationbar-py/logs/kintsugi.ndjson | tail -10'
```

### Stathera Integrity — Full Analysis

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH inner AS (
    SELECT symbol, threshold_decimal_bps, first_agg_trade_id,
           first_agg_trade_id - lagInFrame(last_agg_trade_id, 1)
               OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) - 1 AS tid_delta,
           row_number() OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) AS rn
    FROM opendeviationbar_cache.open_deviation_bars FINAL
    SETTINGS do_not_merge_across_partitions_select_final = 1
)
SELECT
    countIf(tid_delta < 0) AS overlaps,
    countIf(tid_delta > 0) AS gaps
FROM inner WHERE rn > 1
FORMAT PrettyCompact
"'
```

### Symbol Registry — Compliance Audit

Verify every actively-streaming symbol is `enabled = true` in the registry:

```bash
# Check registry vs what's actually in ClickHouse
ssh bigblack 'python3 -c "
from opendeviationbar.symbol_registry import get_registered_symbols
enabled = set(get_registered_symbols(enabled_only=True))
all_syms = set(get_registered_symbols(enabled_only=False))
disabled = all_syms - enabled
print(\"Enabled:\", sorted(enabled))
print(\"Disabled:\", sorted(disabled))
"'
```

```bash
# Check what symbols are in STREAMING_SYMBOLS env vs enabled in registry
ssh bigblack 'python3 -c "
import os
from opendeviationbar.symbol_registry import get_registered_symbols, filter_pairs_by_registry
streaming = os.environ.get(\"OPENDEVIATIONBAR_STREAMING_SYMBOLS\", \"\").split(\",\")
thresholds = [int(t) for t in os.environ.get(\"OPENDEVIATIONBAR_STREAMING_THRESHOLDS\", \"250,500,750,1000\").split(\",\")]
all_pairs = [(s.strip(), t) for s in streaming for t in thresholds if s.strip()]
allowed = set(filter_pairs_by_registry(all_pairs))
blocked = set(all_pairs) - allowed
if blocked:
    print(\"BLOCKED (disabled in registry):\", sorted(blocked))
else:
    print(\"All streaming symbols are enabled in registry.\")
"'
```

### GPU Utilization

```bash
ssh bigblack 'nvidia-smi --query-gpu=name,utilization.gpu,utilization.memory,memory.used,memory.free,temperature.gpu,power.draw --format=csv,noheader 2>/dev/null && echo "" && nvidia-smi 2>/dev/null | head -20'
```

**Expected**: RTX 4090. ODB pipeline does NOT use GPU (0% is normal). GPU should be available for ML workloads.

---

## Tier 3 — Deep Investigation

### Dead Letter Queue

```bash
ssh bigblack 'ls -la /tmp/opendeviationbar-dead-letter/ 2>/dev/null && echo "Count: $(ls /tmp/opendeviationbar-dead-letter/*.parquet 2>/dev/null | wc -l)"'
```

Dead letters are bars that failed to write to ClickHouse and were saved locally. Charon replays them every 5 min. Permanently unplayable files (e.g., cross-midnight bars) should be deleted manually.

### ClickHouse Query Load

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT query_kind, count() AS queries, avg(query_duration_ms) AS avg_ms
FROM system.query_log
WHERE event_time >= now() - INTERVAL 10 MINUTE
GROUP BY query_kind ORDER BY queries DESC
FORMAT PrettyCompact
"'
```

### Sidecar Full Log (last hour)

```bash
ssh bigblack 'journalctl --user -u opendeviationbar-sidecar --since "1 hour ago" --no-pager | grep -E "ERROR|WARNING|gap detected|HEALED|cache_write|reconnect"'
```

### Kintsugi Backlog Assessment

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
WITH continuity AS (
    SELECT symbol, threshold_decimal_bps, first_agg_trade_id,
           first_agg_trade_id - lagInFrame(last_agg_trade_id, 1)
               OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) - 1 AS tid_delta,
           toDate(open_time_ms / 1000, '"'"'UTC'"'"') - toDate(
               lagInFrame(close_time_ms, 1) OVER (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id) / 1000,
               '"'"'UTC'"'"') AS day_diff
    FROM opendeviationbar_cache.open_deviation_bars FINAL
    SETTINGS do_not_merge_across_partitions_select_final = 1
)
SELECT symbol, threshold_decimal_bps, countIf(tid_delta > 0 AND day_diff <> 1) AS real_gaps
FROM continuity
GROUP BY symbol, threshold_decimal_bps
HAVING real_gaps > 0
ORDER BY real_gaps DESC
FORMAT PrettyCompact
"'
```

### Kintsugi Workers Active

```bash
ssh bigblack 'journalctl --user -u opendeviationbar-kintsugi -n 50 --no-pager | grep -E "parallel batch|worker-parallel|day-recompute"'
```

Expect: `parallel batch of N days with N workers`. The outer concurrency is `MAX_CONCURRENT_REPAIRS` (default code: 4, bigblack: **24** — empirically validated sweet spot). **Do NOT increase beyond 24** — 32 causes ClickHouse internal scheduler saturation (271 active queries, 0 repairs/30s). The bottleneck at 24 is ClickHouse soft query cap (~143-145 concurrent queries), not CPU/RAM/NVMe.

### Seeder State

```bash
ssh bigblack 'cat /var/tmp/opendeviationbar-seeder-state.txt'
```

Shows `ok` + timestamp when seeder last completed. If mtime > 15 min, monit fires `seeder-state` alert. Note: monit needs `PrivateTmp=no` in its systemd override to see `/var/tmp/` (diagnosed 2026-03-13).

---

## Telemetry Source Map

| Data                        | Where             | Command / Path                                                                          |
| --------------------------- | ----------------- | --------------------------------------------------------------------------------------- |
| Heartbeat message (full)    | systemd journal   | `journalctl --user -u opendeviationbar-heartbeat --since "6 min ago" --no-pager`        |
| Heartbeat healthy/unhealthy | systemd journal   | grep `HEALTHY\|UNHEALTHY\|WARNING`                                                      |
| Cross-midnight bar count    | ClickHouse (live) | Tier 1 query above                                                                      |
| Kintsugi daemon log         | systemd journal   | `journalctl --user -u opendeviationbar-kintsugi -n 50 --no-pager`                       |
| Kintsugi NDJSON events      | File on bigblack  | `tail -30 ~/opendeviationbar-py/logs/kintsugi.ndjson`                                   |
| Kintsugi repair history     | ClickHouse table  | `opendeviationbar_cache.kintsugi_log`                                                   |
| Sidecar log                 | systemd journal   | `journalctl --user -u opendeviationbar-sidecar -n 50 --no-pager`                        |
| Sidecar health              | HTTP endpoint     | `curl -s http://localhost:8081/health`                                                  |
| Stathera integrity          | ClickHouse (live) | Tier 2 Stathera query above                                                             |
| Monit watchdog              | monit CLI         | `sudo monit summary`                                                                    |
| ClickHouse stats            | ClickHouse (live) | Tier 2 queries above                                                                    |
| Circuit breaker state       | Sidecar /health   | `.checks.circuit_breaker` field                                                         |
| Symbol registry             | Python module     | `python3 -c "from opendeviationbar.symbol_registry import get_registered_symbols; ..."` |
| GPU utilization             | nvidia-smi        | `nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv,noheader`              |
| Disk / memory / load        | OS commands       | `df -h /`, `free -h`, `uptime`                                                          |
| Dead letter queue           | Filesystem        | `/tmp/opendeviationbar-dead-letter/*.parquet`                                           |

### State Files (on bigblack)

| File                                                        | Purpose                                           |
| ----------------------------------------------------------- | ------------------------------------------------- |
| `~/.cache/opendeviationbar/heartbeat/pending_messages.json` | Telegram message IDs for auto-delete (15 min TTL) |
| `~/.cache/opendeviationbar/heartbeat/stathera_counts.json`  | Previous Stathera counts for delta tracking       |
| `~/opendeviationbar-py/logs/kintsugi.ndjson`                | NDJSON event log (10 MB rolling, 1 backup)        |

### Configuration Files

| File                                | Content                                         | Git-Tracked |
| ----------------------------------- | ----------------------------------------------- | ----------- |
| `deploy/opendeviationbar.env`       | Non-secret env vars (chat ID, mode, thresholds) | Yes         |
| `deploy/opendeviationbar.local.env` | Secrets (Telegram token, Redis)                 | No          |
| `.mise.toml`                        | Mise tasks, Python version, hosts               | Yes         |

---

## Systemd Units (all `--user` scope)

| Unit                                        | Type    | Cycle       | Purpose                                                                         |
| ------------------------------------------- | ------- | ----------- | ------------------------------------------------------------------------------- |
| `opendeviationbar-heartbeat.timer`          | Timer   | Every 5 min | Triggers heartbeat service                                                      |
| `opendeviationbar-heartbeat.service`        | Oneshot | On timer    | Runs `health_heartbeat.py`                                                      |
| `opendeviationbar-sidecar.service`          | Simple  | Always-on   | Real-time streaming                                                             |
| `opendeviationbar-kintsugi.service`         | Notify  | Always-on   | Gap reconciliation daemon                                                       |
| `opendeviationbar-kintsugi-catchup.service` | Notify  | On-demand   | Historical backfill (Conflicts= with kintsugi)                                  |
| `opendeviationbar-seeder.timer`             | Timer   | Every 5 min | Triggers seeder service                                                         |
| `opendeviationbar-seeder.service`           | Oneshot | On timer    | Parquet tick cache seeder — writes `/var/tmp/opendeviationbar-seeder-state.txt` |

Common journal flags:

- `-n 50` — last 50 lines
- `--since "N min ago"` — time window
- `--no-pager` — no less/more
- `--grep="pattern"` — filter

---

## Monit Watchdog (10 monitors)

### Expected healthy state

```
sidecar                          OK     Process
kintsugi                         OK     Process
clickhouse                       OK     Remote Host
sidecar-health                   OK     Remote Host
redis                            OK     Remote Host
heartbeat-timer                  OK     Program
monit-heartbeat                  OK     Program
rootfs                           OK     Filesystem
el02                             OK     System
seeder-state                     OK     File
```

### Monit Operations

```bash
# Status
sudo monit summary

# Re-enable a service that shows "Not monitored"
sudo monit monitor kintsugi      # ← fix for "Not monitored" state

# Reload after config change
sudo monit reload

# Force check immediately (don't wait for 120s cycle)
sudo monit validate
```

### Config location

- Source: `deploy/monit/opendeviationbar.conf` (git-tracked)
- Deployed: `/etc/monit/conf.d/opendeviationbar.conf`
- Daemon cycle: 120s
- Process match patterns: `"streaming_sidecar|sidecar.py"` (sidecar), `"kintsugi"` (kintsugi)

**After any deploy**, verify monit re-picked up the new PID:

```bash
ssh bigblack 'sudo monit summary && sudo monit validate'
```

---

## Symbol Registry

### Invariant

Every symbol that is actively streamed or backfilled **must** have `enabled = true` in `python/opendeviationbar/data/symbols.toml`.

| Symbol gate mode   | Behavior                                    |
| ------------------ | ------------------------------------------- |
| `strict` (default) | `SymbolNotRegisteredError` for unregistered |
| `warn`             | `UserWarning` + continue                    |
| `off`              | No gating (dev/testing only)                |

### Currently enabled symbols (v8.0+)

16 crypto symbols enabled (forex disabled since v8.0): `AAVEUSDT ADAUSDT AVAXUSDT BCHUSDT BNBUSDT BTCUSDT DOGEUSDT ETHUSDT FILUSDT LINKUSDT LTCUSDT NEARUSDT SOLUSDT SUIUSDT UNIUSDT XRPUSDT`

> **Note**: WIFUSDT and WLDUSDT were removed. Forex symbols exist in `symbols.toml` with `enabled=false`.

Registry gate is enforced at:

- **Sidecar startup** (`run_sidecar()` line ~1540) — drops disabled pairs before streaming
- **Sidecar `_gap_fill()`** — pre-filter before backfill_gap
- **Kintsugi** — pre-filter pairs before repair

### Adding a new symbol

1. Edit `python/opendeviationbar/data/symbols.toml` — add entry with `enabled = true`
2. Run `maturin develop` locally (rebuilds Rust bindings)
3. On bigblack: `mise run deploy:bigblack` or `pip install --upgrade opendeviationbar`
4. Add symbol to `OPENDEVIATIONBAR_STREAMING_SYMBOLS` in `.mise.toml`
5. Restart sidecar: `systemctl --user restart opendeviationbar-sidecar`

---

## Kintsugi NDJSON Events

| Event                | Key Fields                              | Meaning                               |
| -------------------- | --------------------------------------- | ------------------------------------- |
| `discovery_complete` | `total_shards`                          | Queue depth after shard scan          |
| `repair_start`       | `symbol`, `threshold`, `gap_start_ms`   | Repair initiated                      |
| `repair_complete`    | `result`, `bars_written`                | Repair outcome                        |
| `pass_summary`       | `healed`, `clean`, `deferred`, `failed` | End-of-pass totals                    |
| `watchdog_ping`      | —                                       | Keepalive, confirms daemon is running |

Repair outcomes: `healed` (gap closed) | `clean` (no gap found) | `deferred` (today, sidecar active) | `failed` (error)

**4-state pass summary**: healed / clean / deferred / failed. `deferred` is not an error — it's today's shards where the sidecar is still writing.

---

## Interpreting the Heartbeat Message

| Section        | Healthy                                            | Unhealthy                                        |
| -------------- | -------------------------------------------------- | ------------------------------------------------ |
| **Header**     | `❤️ Heartbeat OK` (silent, auto-deletes in 15 min) | `🔴 UNHEALTHY` (loud, persistent) + likely cause |
| **Services**   | All ✅ with uptime + memory                        | ❌ for down services                             |
| **Version**    | ✅ matches GitHub latest                           | ⚠️ drift detected                                |
| **Kintsugi**   | telemetry fresh, no stuck, low repair rate         | ❌ stale telemetry or stuck repair               |
| **Freshness**  | All symbols <5 min                                 | ⚠️ symbols >30 min stale                         |
| **ClickHouse** | Stats + growth + freshest/stalest                  | ❌ if unreachable                                |
| **Stathera**   | ✅ clean or 🔧 being healed                        | 🔴 anomalies growing and kintsugi not healing    |
| **System**     | ✅ disk/breaker                                    | ⚠️ disk >80% or breaker OPEN                     |

**Stathera health logic**: `anomalies_growing AND NOT kintsugi_healing` = UNHEALTHY. Growing counts are OK if kintsugi is actively repairing.

**Cross-midnight health logic** (v13.11.0+): HEALTHY if `cross_midnight_count == 0 OR kintsugi_healing`. Historical cross-midnight bars are expected during backfill — kintsugi cleans them as it processes each day.

**False alarms during restarts**: Heartbeat catching kintsugi at 0m uptime reports UNHEALTHY. Wait for next 5-min cycle. Confirm with `sudo monit summary`.

---

## Patterns and Anti-Patterns

### Pattern: Comprehensive Health Assessment

**Do this** when asked "is everything OK?" — run ALL of Tier 1 in parallel, then escalate only if anomalies found.

```
Tier 1 checklist:
[ ] Heartbeat: HEALTHY or UNHEALTHY?
[ ] Monit: all services OK? (kintsugi "Not monitored" is a known failure mode)
[ ] Sidecar: status=healthy, circuit_breaker=closed, dropped_bars=0?
[ ] System: disk <80%, RAM available >10Gi, load <8?
[ ] GPU: utilization (informational — ODB doesn't use GPU, 0% is normal)
```

If any item is anomalous, run the matching Tier 2 query.

---

### Pattern: Interpreting Staleness

**High-threshold pairs (750, 1000 dbps) are naturally infrequent.** A 5–7 hour staleness on `XRPUSDT@1000` is expected — that pair requires a 1% price move per bar.

**Stale pairs that signal real problems**:

- Any `@250` pair stale >30 min (liquid pairs should emit bars continuously)
- `BTCUSDT@250` stale >5 min (highest-volume pair)
- Entire symbol completely stale across all thresholds (stream disconnected)

**Stale pairs that are probably fine**:

- Low-volume symbols `@1000` stale for hours
- `PAXGUSDT`, `XLMUSDT` with multi-day staleness (these are in ClickHouse from historical backfill but not in current `OPENDEVIATIONBAR_STREAMING_SYMBOLS`)

---

### Pattern: Monit "Not monitored" Recovery

**Cause**: `sudo monit unmonitor <service>` was run explicitly (e.g., during debugging), or monit failed to match the process at startup and auto-unmonitored after N cycles.

**Recovery**:

```bash
ssh bigblack 'sudo monit monitor kintsugi && sleep 5 && sudo monit summary | grep kintsugi'
```

**Prevention**: After any manual intervention that stops/starts a service, always run `sudo monit validate` to confirm monit re-acquired the PID.

**When to NOT re-enable**: If you intentionally stopped a service for maintenance, leave it unmonitored until maintenance is complete to prevent monit from auto-restarting it.

---

### Pattern: Kintsugi Backlog Interpretation

A `discovery_complete` event with `total_shards=2948` means 2948 gaps to repair — this is a large historical backlog. Interpret progress by:

1. **Rate**: How many `repair_complete` events per hour? At ~122 bars per repair and ~765s/repair, that's ~4–5 repairs/hour/worker.
2. **Workers**: `parallel batch of N days with N workers` — current default is 2. Bump to 4–6 if load avg < 4 and RAM > 20Gi free.
3. **Gap persistence**: `gap persists after repair (tid_delta=N)` is NOT an error. Kintsugi writes one day at a time and the next pass picks up the remainder. A `tid_delta` in the billions means multiple years of history remain — this is expected for a fresh deployment.
4. **Today guard**: `deferred_today_guard` is correct behavior — kintsugi won't touch today's shards while the sidecar is writing.

---

### Pattern: Reading Sidecar Health JSON

```json
{
  "status": "healthy",
  "checks": {
    "clickhouse": true,
    "memory": true,
    "disk_space": true,
    "websocket": true,
    "circuit_breaker": "closed"   ← "open" means ClickHouse is down
  },
  "engine": {
    "bars_emitted": 23,            ← low is fine for high-threshold pairs
    "gap_fills": 6,                ← Puck auto-filling WS gaps
    "continuity_gaps_detected": 9, ← gaps Puck found
    "continuity_verified": 7,      ← gaps confirmed closed
    "rate_limiter_utilization": 0.0, ← 0% = no pressure; >60% = shadow probes paused
    "dropped_bars": 0,             ← MUST be 0; any >0 = immediate investigate
    "reconnections": 6,            ← normal; WS reconnects on network hiccups
    "shadow_divergences": 0        ← REST vs WS trade-ID mismatch count
  }
}
```

**Red flags in sidecar health**:

- `circuit_breaker: "open"` — ClickHouse down, writes failing
- `dropped_bars > 0` — backpressure causing data loss (Lethe alert)
- `shadow_divergences > 10` — WS and REST diverging (data quality issue)
- `status: "unhealthy"` with any `checks` field `false`

---

### Pattern: Symbol Registry Compliance Check

After any change to `OPENDEVIATIONBAR_STREAMING_SYMBOLS` or `symbols.toml`, verify consistency:

```bash
ssh bigblack 'python3 -c "
import os
from opendeviationbar.symbol_registry import filter_pairs_by_registry, get_registered_symbols
streaming_str = os.environ.get(\"OPENDEVIATIONBAR_STREAMING_SYMBOLS\", \"\")
thresholds_str = os.environ.get(\"OPENDEVIATIONBAR_STREAMING_THRESHOLDS\", \"250,500,750,1000\")
streaming = [s.strip() for s in streaming_str.split(\",\") if s.strip()]
thresholds = [int(t) for t in thresholds_str.split(\",\")]
all_pairs = [(s, t) for s in streaming for t in thresholds]
allowed = set(filter_pairs_by_registry(all_pairs))
blocked = [(s, t) for s, t in all_pairs if (s, t) not in allowed]
if blocked:
    print(f\"BLOCKED (disabled/unregistered): {blocked}\")
else:
    print(f\"OK: all {len(allowed)} pairs enabled in registry\")
"'
```

**Invariant**: `blocked` must be empty in production. Any blocked pair means the sidecar will silently skip it at startup.

---

### Pattern: GPU Monitoring

bigblack has an RTX 4090. ODB pipeline does NOT use the GPU — 0% utilization is correct/expected.

```bash
ssh bigblack 'nvidia-smi --query-gpu=name,utilization.gpu,utilization.memory,memory.used,memory.free,temperature.gpu,power.draw --format=csv,noheader'
```

**When GPU usage IS expected**: ML training jobs, PyTorch/JAX inference, or explicit GPU-accelerated ClickHouse queries. Check with `nvidia-smi` if a GPU task is running and competing with ODB CPU/RAM.

**GPU memory pressure**: RTX 4090 has 24GB VRAM. If `memory.used` approaches 23GB, GPU workloads are constrained. This does NOT affect ODB (CPU-only).

---

### Anti-Pattern: Treating `tid_delta > 500` as a real gap

**Wrong**: Hardcoded threshold `tid_delta > 500` misclassifies 97.7% of gaps — BTCUSDT@1000 naturally has `tid_delta=300,000` across a day boundary.

**Right**: ANY `tid_delta > 0` is a real gap. Since kintsugi always repairs using day-start fills (Vision ZIP or REST `startTime=midnight_ms`), properly repaired day boundaries have `tid_delta = 0`. No special-casing needed.

---

### Anti-Pattern: `structural_day_boundary` as a third Stathera category

**Wrong**: Classifying `tid_delta > 0 AND day_diff = 1` as "structural day boundary" (ignore). This was a workaround for partial fills — if you filled from the middle of a day, day boundaries might have `tid_delta > 0`. But it masks real data gaps and creates confusion.

**Right**: Kintsugi fills entire days from UTC midnight. After repair, `tid_delta = 0` at every day boundary. There is no such thing as a "structural gap" — only real gaps (requiring repair) and clean boundaries (`tid_delta = 0`). Stathera has exactly two violation types: gaps (`tid_delta > 0`) and overlaps (`tid_delta < 0`).

---

### Anti-Pattern: `ariadne_used=True` in kintsugi logs implies fromId pagination

**Wrong**: Interpreting `(day-start-fill, anchored=True)` in repair logs as "kintsugi used Ariadne `fromId` pagination". The `anchored` flag only means the shard was discovered between two known bars (interior gap) vs a trailing-edge shard.

**Right**: Kintsugi NEVER uses Ariadne `fromId` pagination for repairs. The repair path is always:

1. Vision ZIP (entire UTC day, rate-limit-free, deterministic)
2. REST `startTime=midnight_ms` fallback (entire UTC day from midnight)

Ariadne `fromId` is only used by Puck (live WebSocket sub-minute gap fills) and the sidecar consumer API — not kintsugi.

---

### Anti-Pattern: `monit unmonitor kintsugi` without `monit monitor kintsugi`

If kintsugi is manually stopped for debugging and `monit unmonitor kintsugi` is run, monit will NOT auto-restart it when you restart systemd. The service shows as "Not monitored" in perpetuity until `sudo monit monitor kintsugi` is run.

**Recovery**: `ssh bigblack 'sudo monit monitor kintsugi && sleep 5 && sudo monit summary'`

---

### Anti-Pattern: Ignoring `enabled = false` in symbols.toml

Symbols with `enabled = false` are gated out by `filter_pairs_by_registry()`. If `OPENDEVIATIONBAR_STREAMING_SYMBOLS` lists a symbol that has `enabled = false`, the sidecar will silently skip it at startup (no error, just an INFO log line). The symbol will NOT be streamed, but no alert fires.

**Detection**: Run the Symbol Registry Compliance Check above.

**Fix**: Set `enabled = true` in `symbols.toml`, rebuild (`maturin develop`), deploy (`mise run deploy:bigblack`), restart sidecar.

---

### Anti-Pattern: Silent write failure swallowed by `try_cache_write()`

**Wrong** (pre-incident pattern): `try_cache_write()` was fire-and-forget — ClickHouse connection failure was logged as WARNING but the checkpoint still advanced.

**Right** (v13.0+): Populate path uses `_fatal_cache_write()` which raises on any ClickHouse failure. The circuit breaker catches cascades (3 failures → OPEN, 60s recovery). Heartbeat checks `.circuit_breaker` every 5 min.

**Detection**: `circuit_breaker: "open"` in `/health`, or `breaker OPEN` in heartbeat.

---

### Anti-Pattern: Interpreting "kintsugi: ❌" in heartbeat as a crash

**False positive scenario**: Heartbeat runs every 5 min. If kintsugi restarts (monit respawn, deploy, watchdog), heartbeat may catch the 0m-uptime window and report UNHEALTHY.

**Check**: `sudo monit summary | grep kintsugi` — if OK, kintsugi is running. The next heartbeat cycle (5 min) will report healthy.

**Real crash**: kintsugi shows as "Not monitored" AND NDJSON file mtime > 30 min ago.

---

### Anti-Pattern: Cross-midnight bars as a terminal error

**Wrong**: Raising `ValueError` and dead-lettering cross-midnight bars causes a retry storm → dead-letter cascade → Charon loops forever.

**Right** (v13.10.0+):

- **L1** (sidecar): Filter cross-midnight bars before batch accumulation
- **L2** (bulk_operations): `filter + warn` instead of `raise ValueError`
- **L3** (kintsugi): Delete permanently-unplayable dead letters instead of breaking replay loop

**Data safety**: Discarded bars create Stathera gaps. Kintsugi re-fetches trades from Binance using proper `split_trades_by_day()` + `reset_at_ouroboros()`.

---

## Troubleshooting Common Alerts

| Alert                                                         | Where to Look             | Resolution                                                                                                         |
| ------------------------------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- |
| `stathera_gaps: Real gaps and kintsugi not healing`           | Kintsugi journal + NDJSON | Check `monit summary`. If "Not monitored": `sudo monit monitor kintsugi`. If running, check for stuck repair.      |
| `cross_midnight_bars=N` (stable count)                        | heartbeat journal         | Expected during kintsugi backfill. Monitor count trending toward 0.                                                |
| `cross_midnight_bars=N` (growing count)                       | sidecar journal           | L1 filter may have regressed. Check sidecar startup logs for cross-midnight filter output.                         |
| `version drift`                                               | heartbeat journal         | Package on bigblack doesn't match GitHub. Run `mise run deploy:bigblack`.                                          |
| `sidecar down`                                                | sidecar journal           | `journalctl --user -u opendeviationbar-sidecar -n 50`. Monit should auto-restart within 2 min.                     |
| `kintsugi telemetry stale`                                    | NDJSON file mtime         | Kintsugi may be stuck or crashed. Check `monit summary` and journal.                                               |
| `disk >80%`                                                   | `df -h /`                 | Check ClickHouse parts, dead letters, log rotation. `ls -lah /tmp/opendeviationbar-dead-letter/`                   |
| `breaker OPEN`                                                | sidecar /health           | ClickHouse likely down or SSH tunnel dropped. Fix CH first. Breaker auto-recovers in 60s.                          |
| `kintsugi: ❌` but monit shows OK                             | heartbeat timing artifact | Heartbeat caught 0m-uptime window. Wait 5 min. Confirm next heartbeat shows HEALTHY.                               |
| `kintsugi Not monitored` in monit                             | monit state               | Run `sudo monit monitor kintsugi && sleep 5 && sudo monit summary`. Was likely manually unmonitored for debugging. |
| Symbol streaming fewer pairs than expected                    | sidecar startup logs      | Check `journalctl --user -u opendeviationbar-sidecar --since "restart time"                                        | grep "skipping"`. Run symbol registry compliance check. |
| `gap persists after repair (tid_delta=N)` in kintsugi journal | Normal                    | Kintsugi fills one day at a time. Next pass picks up remainder. Not an error.                                      |
| `dropped_bars > 0` in sidecar health                          | sidecar journal           | Backpressure — ClickHouse writes are slower than bar production. Check ClickHouse load and disk.                   |

---

## Sidecar Log Level Strategy (v13.11.2+)

`sidecar_api.py` 5-source ariadne cascade and gap-fill fallbacks:

| Level       | When                                              | Examples                                                    |
| ----------- | ------------------------------------------------- | ----------------------------------------------------------- |
| **DEBUG**   | Benign/expected fallbacks, normal operation       | PyO3 borrow conflict, missing checkpoint file, Parquet skip |
| **INFO**    | Normal fallback indicating infrastructure state   | Parquet cache unavailable, falling through to REST          |
| **WARNING** | Real infrastructure issue, but not fatal          | ClickHouse unreachable, Binance REST failed                 |
| **ERROR**   | All sources exhausted, degraded response returned | All 5 ariadne sources failed                                |

No `exc_info=True` tracebacks — inline `%s` exception message only. This keeps normal fallback cascades from looking like crashes in production logs.
