# Heartbeat Check Modules

**Parent**: [/scripts/heartbeat/CLAUDE.md](/scripts/heartbeat/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Individual health check modules, each collecting specific metrics and populating a shared `results` dict consumed by the heartbeat formatter. The orchestrator (`__init__.py`) calls each check in sequence, aggregating boolean pass/fail flags and string metadata into a single results dict.

---

## Checks

| Module             | Purpose                                                                                                                                                                                                                                           |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `__init__.py`      | Orchestrator: `run_checks()` calls all check modules, returns unified `dict[str, bool \| str]`                                                                                                                                                    |
| `stathera.py`      | Stathera integrity audit: detects gaps and overlaps in aggTrade IDs (TELEM-01, TELEM-04, TELEM-05). LOUD alert on today's new gaps                                                                                                                |
| `freshness.py`     | Per-symbol freshness: `collect_symbol_freshness()` queries latest bar time per (symbol, threshold) pair. Aion freshness uses rolling window (default 30m, configurable via AION_FRESHNESS_WINDOW_MIN). All symbols stale = sidecar not streaming. |
| `clickhouse.py`    | ClickHouse database stats: total bars, pair count, active symbols, pending mutations, active parts. Delegates to `freshness.py`                                                                                                                   |
| `services.py`      | Systemd service status: `check_oom_kills()`, `check_systemd_overrides()`, `collect_service_stats()`, `is_kintsugi_catchup_active()`                                                                                                               |
| `sidecar.py`       | Sidecar health: `check_dead_letters()` counts dead-letter files, `check_ws_errors()` classifies 429/ban vs transient disconnects                                                                                                                  |
| `kintsugi.py`      | Kintsugi NDJSON telemetry parsing: daemon age, stuck repair detection, active repair visibility, shard queue depth                                                                                                                                |
| `resources.py`     | System resource monitoring via Prometheus: CPU, memory, load average peaks over 15-min window                                                                                                                                                     |
| `junction.py`      | Junction telemetry (TELEM-03): REST-to-WS junction diagnostics via sidecar `/health` and `/status` endpoints. Informational only, not health-gating                                                                                               |
| `ratelimit.py`     | REST API rate limit utilization: aggregate budget check across all consumers, sustained high-utilization LOUD alert (D-22, D-24)                                                                                                                  |
| `seeder.py`        | Parquet tick cache seeder state: timer freshness check, per-symbol seeder status (Issue #270)                                                                                                                                                     |
| `version.py`       | Version drift detection: compares installed wheel version vs git HEAD, detects hotfixed `.py` files scp'd post-install                                                                                                                            |
| `vision_oracle.py` | Weekly Vision vs REST trade count oracle (D-10): samples random days/symbols, alerts on >0.1% mismatch                                                                                                                                            |
| `yesterday.py`     | Aion freshness (MON-01): rolling-window check that last committed bar is within configured freshness window per symbol                                                                                                                            |

## Check Interface

Every check module follows the same pattern:

```python
def check_or_collect_*(
    client: object | None,   # clickhouse_connect Client (or omitted for non-CH checks)
    results: dict[str, bool | str],  # Shared results dict, mutated in place
) -> None:
    """Populate results dict with check-specific keys."""
```

**Convention:**

- Boolean keys (e.g., `"clickhouse"`, `"sidecar"`, `"stathera_gaps"`) participate in health gating -- `False` triggers CRITICAL/WARNING based on `config.py` key sets
- String keys (e.g., `"disk_free_gb"`, `"kintsugi_repairs"`, `"kintsugi_mode"`) are informational metadata displayed in the Telegram message but do not gate health

## Key Checks

**`stathera.py`** is the primary data integrity check. It runs the Stathera audit query (trade-ID continuity via `lagInFrame()` window function) across all enabled (symbol, threshold) pairs. Detects both gaps (`tid_delta > 0`) and overlaps (`tid_delta < 0`). Features:

- Unified per-symbol results (TELEM-01)
- Today-gap LOUD Telegram alert (TELEM-04) -- immediate notification for gaps in today's bars (sidecar issue, kintsugi cannot fix)
- Kintsugi queue position per anomalous pair (TELEM-05)
- Delta tracking via `state.py` Stathera counts -- detects growing vs stable anomalies
- Known gap fingerprint suppression (from `config.py`)

## Related

- [/scripts/heartbeat/CLAUDE.md](/scripts/heartbeat/CLAUDE.md) -- Parent: heartbeat package
- [/CLAUDE.md](/CLAUDE.md) -- Hub: project overview
