---
phase: 12-telemetry-heartbeat
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - scripts/heartbeat/checks/checksum.py
  - scripts/heartbeat/checks/__init__.py
  - scripts/heartbeat/formatter.py
  - scripts/heartbeat/config.py
  - python/opendeviationbar/checksum_telemetry.py
autonomous: true
requirements: [TELEM-01, TELEM-02, TELEM-04]

must_haves:
  truths:
    - "Heartbeat message includes a Checksum Health section showing per-symbol last verified date and failure count"
    - "When all symbols have clean checksums, the section shows a single summary line (exception-only pattern)"
    - "Any checksum failure triggers a LOUD Telegram alert with symbol, date, expected hash, actual hash"
    - "All checksum operations (verify, skip, fail, retry) are appended to a local JSONL telemetry log"
  artifacts:
    - path: "scripts/heartbeat/checks/checksum.py"
      provides: "Heartbeat check module reading ChecksumRegistry"
      exports: ["check_checksum"]
    - path: "python/opendeviationbar/checksum_telemetry.py"
      provides: "Append-only JSONL telemetry log for checksum operations"
      exports: ["log_checksum_event"]
  key_links:
    - from: "scripts/heartbeat/checks/checksum.py"
      to: "python/opendeviationbar/checksum_registry.py"
      via: "ChecksumRegistry import"
      pattern: "from opendeviationbar.checksum_registry import ChecksumRegistry"
    - from: "scripts/heartbeat/checks/__init__.py"
      to: "scripts/heartbeat/checks/checksum.py"
      via: "check_checksum call in run_checks()"
      pattern: "check_checksum"
    - from: "scripts/heartbeat/formatter.py"
      to: "checksum results keys"
      via: "results dict keys prefixed checksum_"
      pattern: "checksum_"
---

<objective>
Add checksum verification health monitoring to the 5-minute heartbeat and wire a local JSONL telemetry log for all checksum operations.

Purpose: Complete the v6.0 Data Integrity milestone by surfacing checksum verification status in production monitoring. Operators see per-symbol checksum health at a glance, get LOUD alerts on failures, and have a local log file for post-mortem analysis.

Output: New heartbeat check module, updated formatter with checksum section, local telemetry log module.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@scripts/heartbeat/checks/CLAUDE.md
@scripts/heartbeat/CLAUDE.md

<interfaces>
<!-- ChecksumRegistry API (Phase 9) — executor uses these directly -->

From python/opendeviationbar/checksum_registry.py:

```python
class ChecksumRegistry:
    def __init__(self, path: Path | None = None) -> None: ...
    def has_verified(self, symbol: str, date: str) -> bool: ...
    def query_by_symbol(self, symbol: str) -> list[VerificationRecord]: ...
    def summary(self) -> dict:
        """Returns {"total_verified": int, "by_symbol": {sym: count}, "latest_verified_at": str|None}"""
        ...

@dataclass
class VerificationRecord:
    symbol: str
    date: str       # ISO YYYY-MM-DD
    sha256: str
    verified_at: str  # ISO 8601 with UTC
    source: str       # "seeder" or "sweep"
```

From scripts/heartbeat/checks/**init**.py:

```python
def run_checks() -> dict[str, bool | str]:
    """Calls each check module, returns shared results dict."""
```

From scripts/heartbeat/config.py:

```python
CRITICAL_KEYS = frozenset({...})    # LOUD persistent
WARNING_KEYS = frozenset({...})     # silent persistent
DEGRADED_KEYS = frozenset({...})    # ephemeral silent
```

From scripts/heartbeat/formatter.py:

```python
def format_message(results: dict[str, bool | str], healthy: bool) -> str:
    """Builds sectioned Telegram HTML message."""
```

Check module interface pattern (from checks/CLAUDE.md):

```python
def check_*(results: dict[str, bool | str]) -> None:
    """Populate results dict with check-specific keys."""
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Checksum telemetry log module + heartbeat check module</name>
  <read_first>
    - python/opendeviationbar/checksum_registry.py (full ChecksumRegistry API)
    - scripts/heartbeat/checks/seeder.py (pattern for check module)
    - scripts/heartbeat/checks/__init__.py (wiring pattern)
    - scripts/heartbeat/config.py (health tier keys)
  </read_first>
  <files>
    python/opendeviationbar/checksum_telemetry.py
    scripts/heartbeat/checks/checksum.py
    scripts/heartbeat/checks/__init__.py
    scripts/heartbeat/config.py
  </files>
  <action>
    **A. Create `python/opendeviationbar/checksum_telemetry.py`** (per TELEM-04)

    Append-only JSONL telemetry log for all checksum operations. Path: `~/.cache/opendeviationbar/checksum-telemetry.jsonl` (matches CONTEXT.md suggestion).

    Provide a single function:
    ```python
    def log_checksum_event(
        event: str,          # "verify_pass", "verify_fail", "verify_skip", "verify_retry", "sweep_pass", "sweep_fail", "sweep_skip"
        symbol: str,
        date: str,           # ISO YYYY-MM-DD
        *,
        expected_hash: str | None = None,
        actual_hash: str | None = None,
        source: str | None = None,    # "seeder" or "sweep"
        reason: str | None = None,    # skip/fail reason
    ) -> None:
    ```

    Implementation:
    - Build a dict with `timestamp` (ISO 8601 UTC), `event`, `symbol`, `date`, plus any non-None kwargs.
    - `mkdir(parents=True, exist_ok=True)` on parent dir.
    - Open file in append mode (`"a"`), write `json.dumps(entry) + "\n"`, close.
    - Wrap entire function body in `try/except Exception` with `logging.getLogger(__name__).debug(...)` — telemetry must NEVER crash the caller.
    - No file rotation (JSONL grows; operator can logrotate externally).

    **B. Create `scripts/heartbeat/checks/checksum.py`** (per TELEM-01, TELEM-02)

    Follow the check module pattern from seeder.py. Provide:
    ```python
    def check_checksum(results: dict[str, bool | str]) -> None:
    ```

    Implementation:
    1. Import `ChecksumRegistry` from `opendeviationbar.checksum_registry` inside the function (wrapped in try/except ImportError for graceful degradation).
    2. Import `get_heartbeat_symbols` from `heartbeat.config` for the symbol list.
    3. Load registry: `registry = ChecksumRegistry()`.
    4. Get `summary = registry.summary()` — provides `total_verified`, `by_symbol`, `latest_verified_at`.
    5. For each symbol from `get_heartbeat_symbols()`:
       - Get records: `records = registry.query_by_symbol(symbol)`
       - Compute `last_verified_date`: `records[-1].date if records else None`
       - Compute `last_verified_at`: `records[-1].verified_at if records else None`
       - Store per-symbol data in results dict.
    6. Scan the telemetry JSONL log (`~/.cache/opendeviationbar/checksum-telemetry.jsonl`) for recent failures (last 24h). Read file line-by-line, parse JSON, filter for `event` containing "fail" and `timestamp` within 24h of now. Collect failure records.
    7. Populate results dict keys:
       - `checksum_total_verified` (str): total verified count from registry summary
       - `checksum_latest` (str): latest_verified_at from summary (human-readable)
       - `checksum_failures_24h` (str): count of failures in last 24h from JSONL
       - `checksum_failure_details` (list[dict]): up to 5 most recent failures with symbol, date, expected_hash, actual_hash — used by formatter for LOUD alert detail
       - `checksum_per_symbol` (dict[str, dict]): `{symbol: {"verified": count, "last_date": str|None}}` for formatter
       - `checksum_ok` (bool): `True` if zero failures in last 24h, `False` if any failures exist
    8. On any exception, log warning and set `checksum_ok = True` (fail-open — don't block heartbeat on registry errors).

    **C. Wire into `scripts/heartbeat/checks/__init__.py`**

    Add import at top: `from heartbeat.checks.checksum import check_checksum`
    Add call at end of `run_checks()` (before the junction telemetry block): `check_checksum(results)`

    **D. Update `scripts/heartbeat/config.py`**

    Add `"checksum_ok"` to `WARNING_KEYS` frozenset (not CRITICAL — checksum failure doesn't mean data loss, it means unverified data). A checksum failure means the download might be corrupted, which warrants a WARNING-level persistent alert.

    Add `"checksum_ok"` to the `_remediation` dict in formatter.py — handled in Task 2.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "
from opendeviationbar.checksum_telemetry import log_checksum_event
log_checksum_event('verify_pass', 'BTCUSDT', '2026-01-01', source='test')
print('telemetry module OK')
" && python -c "
import sys; sys.path.insert(0, 'scripts')
from heartbeat.checks.checksum import check_checksum
r = {}
check_checksum(r)
print('checksum_ok' in r)
print('check module OK')
" && grep -q 'check_checksum' scripts/heartbeat/checks/__init__.py && grep -q 'checksum_ok' scripts/heartbeat/config.py && echo "ALL CHECKS PASSED"</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "def log_checksum_event" python/opendeviationbar/checksum_telemetry.py
    - grep -q "def check_checksum" scripts/heartbeat/checks/checksum.py
    - grep -q "check_checksum" scripts/heartbeat/checks/__init__.py
    - grep -q "checksum_ok" scripts/heartbeat/config.py
    - grep -q "checksum-telemetry.jsonl" python/opendeviationbar/checksum_telemetry.py
    - grep -q "ChecksumRegistry" scripts/heartbeat/checks/checksum.py
    - grep -q "checksum_failures_24h" scripts/heartbeat/checks/checksum.py
    - grep -q "checksum_per_symbol" scripts/heartbeat/checks/checksum.py
  </acceptance_criteria>
  <done>
    Telemetry log module writes JSONL entries without crashing callers.
    Heartbeat check module reads ChecksumRegistry and JSONL log, populates results dict with checksum_ok boolean + per-symbol data + failure details.
    check_checksum is wired into run_checks(). checksum_ok is in WARNING_KEYS.
  </done>
</task>

<task type="auto">
  <name>Task 2: Formatter checksum section + LOUD alert wiring</name>
  <read_first>
    - scripts/heartbeat/formatter.py (full file — understand section ordering and HTML patterns)
    - scripts/heartbeat/checks/checksum.py (from Task 1 — keys it populates)
  </read_first>
  <files>
    scripts/heartbeat/formatter.py
  </files>
  <action>
    **A. Add checksum remediation hint** to the `_remediation` dict in `format_message()`:

    ```python
    "checksum_ok": "Checksum verification failure — check: cat ~/.cache/opendeviationbar/checksum-telemetry.jsonl | tail -20",
    ```

    **B. Add Checksum Health section** to the formatter, placed AFTER the Per-Symbol Health section and BEFORE the System section.

    Follow the Phase 7 exception-only pattern:
    - If `checksum_per_symbol` is in results:
      ```
      lines.append("")
      lines.append("<b>Checksum Health</b>")
      ```
    - Read `checksum_per_symbol` dict, `checksum_total_verified`, `checksum_failures_24h`, `checksum_latest`.
    - **Clean state** (zero failures in 24h): Single summary line:
      ```
        [checkmark] {total_verified} files verified, latest {latest} — all clean
      ```
    - **Failures present** (`checksum_failures_24h` > "0"):
      ```
        [X] {failures_24h} checksum failure(s) in 24h
      ```
      Then expand `checksum_failure_details` (list of dicts from check module):
      ```
          [X] {symbol} {date}: expected {expected[:16]}... got {actual[:16]}...
      ```
      (Truncate hashes to 16 chars for readability. HTML-escape all values.)
    - **Per-symbol breakdown** (only when issues exist): For each symbol in `checksum_per_symbol` where `verified == 0` or where that symbol appears in failure details:
      ```
          {symbol}: {verified} verified, last {last_date or "never"}
      ```
    - If no `checksum_per_symbol` in results (check module failed to load), skip the entire section silently.

    **C. LOUD alert detail for TELEM-02**

    The LOUD alert behavior is already handled by the health level system: `checksum_ok = False` is in `WARNING_KEYS`, which makes the message persistent with silent notification. However, TELEM-02 specifies LOUD (disable_notification=False). To achieve LOUD on checksum failure specifically:

    In `scripts/heartbeat/checks/checksum.py` (update from Task 1 if needed), when failures are detected, also set:
    ```python
    results["checksum_loud_alert"] = True
    ```

    In `scripts/heartbeat/__main__.py`, no change needed — the existing 4-tier system handles this. WARNING level sends persistent, silent. But the requirement says LOUD. Two options:
    - Move `checksum_ok` to `CRITICAL_KEYS` instead of `WARNING_KEYS` (makes it LOUD)
    - OR keep in WARNING_KEYS but add a special case in __main__.py

    Decision: Move `checksum_ok` to `CRITICAL_KEYS` in config.py. A checksum failure means potentially corrupted data in the cache — this IS critical for data integrity. This makes the alert LOUD (disable_notification=False) per TELEM-02. Update Task 1's config.py change: add to CRITICAL_KEYS instead of WARNING_KEYS.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "
import sys; sys.path.insert(0, 'scripts')
from heartbeat.formatter import format_message
# Test clean state
r = {
    'clickhouse': True, 'sidecar': True, 'kintsugi': True, 'disk': True,
    'checksum_ok': True, 'checksum_total_verified': '1500',
    'checksum_failures_24h': '0', 'checksum_latest': '2026-03-25T12:00:00+00:00',
    'checksum_per_symbol': {'BTCUSDT': {'verified': 500, 'last_date': '2026-03-25'}},
}
msg = format_message(r, True)
assert 'Checksum Health' in msg, 'Missing Checksum Health section'
assert 'all clean' in msg, 'Missing clean summary'
print('Clean state OK')

# Test failure state

r2 = dict(r)
r2['checksum_ok'] = False
r2['checksum_failures_24h'] = '2'
r2['checksum_failure_details'] = [
{'symbol': 'BTCUSDT', 'date': '2026-03-24', 'expected_hash': 'abc123def456', 'actual_hash': 'xyz789000111'}
]
msg2 = format_message(r2, False)
assert 'failure' in msg2.lower(), 'Missing failure info'
print('Failure state OK')
print('ALL FORMATTER CHECKS PASSED')
"</automated>
</verify>
<acceptance_criteria> - grep -q "Checksum Health" scripts/heartbeat/formatter.py - grep -q "checksum_ok" scripts/heartbeat/formatter.py - grep -q "checksum_per_symbol" scripts/heartbeat/formatter.py - grep -q "checksum_failure_details" scripts/heartbeat/formatter.py - grep -q "all clean" scripts/heartbeat/formatter.py - grep -q "checksum_ok" scripts/heartbeat/config.py (in CRITICAL_KEYS)
</acceptance_criteria>
<done>
Heartbeat message includes Checksum Health section after Per-Symbol Health.
Clean state shows single summary line. Failure state shows per-failure detail with truncated hashes.
checksum_ok in CRITICAL_KEYS triggers LOUD persistent alert per TELEM-02.
</done>
</task>

<task type="auto">
  <name>Task 3: Wire telemetry log into seeder and sweep callers</name>
  <read_first>
    - scripts/tick_cache_seeder.py (find _download_and_verify_vision_zip and checksum pass/fail/skip logging)
    - scripts/integrity_sweep.py (find checksum verification calls)
    - python/opendeviationbar/checksum_telemetry.py (from Task 1)
  </read_first>
  <files>
    scripts/tick_cache_seeder.py
    scripts/integrity_sweep.py
  </files>
  <action>
    Wire `log_checksum_event()` calls into the existing seeder and sweep checksum paths so that every verify/skip/fail/retry operation is captured in the JSONL telemetry log (per TELEM-04 and feedback-local-telemetry-log.md).

    **A. In `scripts/tick_cache_seeder.py` (`_download_and_verify_vision_zip` function):**

    Add import at top of file: `from opendeviationbar.checksum_telemetry import log_checksum_event`

    Insert `log_checksum_event()` calls at each outcome:

    1. After successful checksum verification (the `logger.info("%s %s: checksum verified")` line ~line 479):
       ```python
       log_checksum_event("verify_pass", symbol, date_str, source="seeder", expected_hash=sha256[:16])
       ```

    2. After checksum mismatch on retry (the `logger.warning("%s %s: checksum mismatch (attempt %d/%d)")` line ~line 449):
       ```python
       log_checksum_event("verify_retry", symbol, date_str, source="seeder", reason=f"attempt {attempt+1}/{max_retries}")
       ```

    3. After final checksum failure (the `logger.warning("%s %s: checksum mismatch after %d retries")` line ~line 455):
       ```python
       log_checksum_event("verify_fail", symbol, date_str, source="seeder", expected_hash=expected[:16] if expected else None, actual_hash=actual[:16] if actual else None, reason="max_retries_exhausted")
       ```
       Note: extract expected and actual hashes from the RuntimeError that was caught.

    4. After checksum soft-skip (the `logger.debug/warning` lines for 404, timeout, format error ~line 485):
       ```python
       log_checksum_event("verify_skip", symbol, date_str, source="seeder", reason=result.get("reason", "unknown"))
       ```

    **B. In `scripts/integrity_sweep.py`:**

    Add import: `from opendeviationbar.checksum_telemetry import log_checksum_event`

    Wire at each outcome in the sweep's verification logic:
    - Pass: `log_checksum_event("sweep_pass", symbol, date_str, source="sweep")`
    - Fail: `log_checksum_event("sweep_fail", symbol, date_str, source="sweep", reason=...)`
    - Skip (already verified): `log_checksum_event("sweep_skip", symbol, date_str, source="sweep", reason="already_verified")`

    **Key constraint:** These are append calls to a fire-and-forget function. They MUST NOT affect the seeder/sweep control flow. The `log_checksum_event` function already wraps in try/except, so no additional error handling needed at call sites.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -c "log_checksum_event" scripts/tick_cache_seeder.py && grep -c "log_checksum_event" scripts/integrity_sweep.py && echo "WIRING CHECK PASSED"</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "from opendeviationbar.checksum_telemetry import log_checksum_event" scripts/tick_cache_seeder.py
    - grep -q "from opendeviationbar.checksum_telemetry import log_checksum_event" scripts/integrity_sweep.py
    - grep -q "log_checksum_event.*verify_pass" scripts/tick_cache_seeder.py
    - grep -q "log_checksum_event.*verify_fail" scripts/tick_cache_seeder.py
    - grep -q "log_checksum_event.*verify_skip" scripts/tick_cache_seeder.py
    - grep -q "log_checksum_event.*sweep_pass" scripts/integrity_sweep.py
    - grep -q "log_checksum_event.*sweep_fail" scripts/integrity_sweep.py
  </acceptance_criteria>
  <done>
    Every checksum operation in seeder (pass, fail, skip, retry) and sweep (pass, fail, skip) calls log_checksum_event().
    JSONL telemetry log at ~/.cache/opendeviationbar/checksum-telemetry.jsonl captures all operations for post-mortem.
  </done>
</task>

</tasks>

<verification>
1. `python -c "from opendeviationbar.checksum_telemetry import log_checksum_event; print('OK')"` — telemetry module importable
2. `cd scripts && python -c "import sys; sys.path.insert(0,'.'); from heartbeat.checks.checksum import check_checksum; r={}; check_checksum(r); print(r.get('checksum_ok'))"` — check module runs without crash
3. `grep -q "Checksum Health" scripts/heartbeat/formatter.py` — formatter has new section
4. `grep -c "log_checksum_event" scripts/tick_cache_seeder.py` should be >= 3 (pass, fail, skip)
5. `grep -c "log_checksum_event" scripts/integrity_sweep.py` should be >= 2 (pass, fail or skip)
</verification>

<success_criteria>

- Heartbeat includes Checksum Health section with per-symbol verified count and last date
- Clean checksums show single summary line (exception-only pattern)
- checksum_ok=False triggers CRITICAL-tier LOUD persistent Telegram alert
- All seeder checksum operations logged to ~/.cache/opendeviationbar/checksum-telemetry.jsonl
- All sweep checksum operations logged to the same JSONL file
- Heartbeat check reads both registry and JSONL log for 24h failure detection
  </success_criteria>

<output>
After completion, create `.planning/phases/12-telemetry-heartbeat/12-01-SUMMARY.md`
</output>
