# Health Heartbeat

**Parent**: [/scripts/CLAUDE.md](/scripts/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Health heartbeat system running every 5 minutes via systemd timer (`opendeviationbar-heartbeat.timer`). Monitors sidecar, kintsugi, ClickHouse connectivity, Stathera integrity, disk space, circuit breaker, and resource utilization, sending results as ephemeral or persistent Telegram messages based on severity.

---

## Files

| File             | Purpose                                                                                                                    |
| ---------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `__init__.py`    | Package marker for the heartbeat module                                                                                    |
| `__main__.py`    | Entry point: runs all checks via `run_checks()`, determines health level, formats and sends Telegram msg                   |
| `config.py`      | Constants: TTL (15 min), state paths, alert dedup TTL (1h), 4-tier health keys (CRITICAL/WARNING/DEGRADED/HEALTHY)         |
| `formatter.py`   | `format_message()`: builds sectioned Telegram HTML with per-check icons, remediation hints, and resource summaries         |
| `alert_dedup.py` | Redis-based alert deduplication: `should_alert()` suppresses duplicate alerts within TTL (fail-open if Redis down)         |
| `state.py`       | JSON state persistence: pending message tracking for ephemeral deletion, Stathera anomaly count deltas                     |
| `telegram.py`    | Telegram Bot API functions: `send_telegram()`, `delete_expired()`, `push_deadman_switch()` for ephemeral message lifecycle |

## Entry Point

`__main__.py` orchestrates the full heartbeat cycle:

1. Run all health checks via `run_checks()` (from `checks/__init__.py`)
2. Classify health level: CRITICAL > WARNING > DEGRADED > HEALTHY (based on key sets in `config.py`)
3. Format the Telegram message via `format_message()` with per-check icons and remediation hints
4. Alert dedup via Redis (`should_alert()`) to prevent notification spam
5. Send via Telegram Bot API; track message ID for ephemeral deletion
6. Delete expired healthy messages (auto-cleanup after 15 min TTL)
7. Push deadman switch (lets external monitors detect if heartbeat itself stops)

Emergency crash handler sends a LOUD alert if the heartbeat process itself throws an unhandled exception.

## 4-Tier Health Levels

| Level    | Icon | Telegram Behavior          | Keys (from `config.py`)                                                    |
| -------- | ---- | -------------------------- | -------------------------------------------------------------------------- |
| CRITICAL | RED  | Persistent, LOUD           | clickhouse, sidecar, kintsugi, disk, stathera\_\*, circuit_breaker_healthy |
| WARNING  | ORG  | Persistent, silent         | kintsugi_stuck/failures/memory, version_drift, seeder_ok                   |
| DEGRADED | YEL  | Ephemeral (15 min), silent | kintsugi_telemetry, yesterday_completion                                   |
| HEALTHY  | GRN  | Ephemeral (15 min), silent | All checks pass                                                            |

## Alert Deduplication

`alert_dedup.py` uses Redis SETEX with 1-hour TTL keyed by MD5 hash of the alert condition string. Prevents repeated LOUD notifications for the same issue. Fail-open: if Redis is unavailable, alerts are always sent (never silenced).

## Dynamic Symbol Coverage (v5.0)

`get_heartbeat_symbols()` in `config.py` dynamically queries the symbol registry for all enabled Binance spot symbols, replacing the previous hardcoded 2-symbol list.

**Implementation:**

- Uses `lru_cache(maxsize=1)` -- heartbeat is a 5-min systemd oneshot, so the cache never goes stale within a single run
- Filters registry entries by `exchange="binance"` and `market="spot"` (Exness forex symbols excluded -- they have no `agg_trade_id` and cannot do Stathera checks)
- Returns a sorted tuple of symbol names (e.g., `("BTCUSDT", "ETHUSDT", "SOLUSDT", ...)`)

**Graceful fallback:** On any exception (ImportError, registry corruption, missing file), falls back to `("BTCUSDT", "ETHUSDT")` -- heartbeat must never crash.

**Backward compatibility:** `HEARTBEAT_PRIORITY_SYMBOLS` constant is kept (deprecated) for `freshness.py` display ordering, but `yesterday.py` now uses `get_heartbeat_symbols()` for its completion check. The yesterday completion check now runs for all 17+ streaming symbols, not just BTC/ETH.

## Children

- [checks/CLAUDE.md](/scripts/heartbeat/checks/CLAUDE.md) -- Individual health check modules

## v7.2 Memory Efficiency Impact

v7.2 introduced Arrow FFI drain (`drain_bars_arrow()`) which eliminates ~200K dict allocations per sync_flush cycle. Impact on heartbeat checks:

- **Freshness**: Aion symbols use rolling-window check (`AION_FRESHNESS_WINDOW_MIN=30`). If all symbols show `aion:stale`, verify sidecar is actively streaming (check `/health` endpoint)
- **Stathera**: ClickHouse-backed checks are unaffected -- they query committed bars regardless of Arrow vs dict drain path
- **Memory**: Sidecar RSS should be ~0.9G peak (down from ~2.6G). Monit/systemd MemoryMax=4G remains as safety guard
- **Arrow fallback**: If sidecar falls back to dict path (older binary), memory returns to ~2.6G. Check journal for `sync_flush[dict]` vs `sync_flush[arrow]` prefix

## Related

- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) -- Parent: scripts and operations
- [/CLAUDE.md](/CLAUDE.md) -- Hub: project overview
- [/scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md) -- systemd timer and service configuration
- [/deploy/CLAUDE.md](/deploy/CLAUDE.md) -- Monit watchdog and deployment
