---
phase: 03-heartbeat-telemetry-overhaul
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - scripts/heartbeat/checks/yesterday.py
  - scripts/heartbeat/checks/junction.py
  - scripts/heartbeat/checks/__init__.py
  - scripts/heartbeat/formatter.py
  - scripts/heartbeat/config.py
  - tests/test_heartbeat_yesterday_junction.py
autonomous: true
requirements: [TELEM-02, TELEM-03]

must_haves:
  truths:
    - "Heartbeat reports yesterday completion status per symbol -- whether yesterday's orphan bar matches the midnight crossover trade ID"
    - "After sidecar restart, heartbeat reports junction telemetry: REST last trade ID, WS first trade ID, forming bar state, committed_floors value"
    - "Yesterday completion and junction telemetry are visible in the Telegram heartbeat message"
  artifacts:
    - path: "scripts/heartbeat/checks/yesterday.py"
      provides: "Yesterday orphan completion check per symbol via crossover TID"
      contains: "collect_yesterday_completion"
    - path: "scripts/heartbeat/checks/junction.py"
      provides: "Junction telemetry from sidecar health endpoint"
      contains: "collect_junction_telemetry"
    - path: "tests/test_heartbeat_yesterday_junction.py"
      provides: "Tests for yesterday completion and junction telemetry"
  key_links:
    - from: "scripts/heartbeat/checks/__init__.py"
      to: "scripts/heartbeat/checks/yesterday.py"
      via: "collect_yesterday_completion call in run_checks()"
      pattern: "collect_yesterday_completion"
    - from: "scripts/heartbeat/checks/__init__.py"
      to: "scripts/heartbeat/checks/junction.py"
      via: "collect_junction_telemetry call in run_checks()"
      pattern: "collect_junction_telemetry"
    - from: "scripts/heartbeat/checks/yesterday.py"
      to: "Binance REST API"
      via: "fetch_aggtrades_rest for midnight crossover TID"
      pattern: "fetch_aggtrades_rest"
---

<objective>
Add yesterday completion status (TELEM-02) and junction telemetry (TELEM-03) checks to heartbeat monitoring.

Purpose: Yesterday completion validates that the midnight boundary is clean for each streaming symbol -- the orphan bar's last trade ID matches crossover_tid - 1. Junction telemetry provides enough diagnostic data after a sidecar restart to diagnose any junction issue without SSH (REST last TID, WS first TID, forming bar state, committed_floors).

Output: Two new check modules (yesterday.py, junction.py), wired into run_checks() and formatter, with test coverage.
</objective>

<execution_context>
@~/.claude/get-shit-done/workflows/execute-plan.md
@~/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/REQUIREMENTS.md
@.planning/phases/03-heartbeat-telemetry-overhaul/03-CONTEXT.md

<interfaces>
<!-- Key types and contracts the executor needs -->

From python/opendeviationbar/sidecar.py (Phase 2):

```python
def _get_midnight_crossover_tid(symbol: str) -> int | None:
    """Query Binance REST for the first aggregated trade at or after UTC midnight.
    Returns the agg_trade_id of the first trade of today, or None on failure."""
```

From python/opendeviationbar/clickhouse/stathera_query.py:

```python
FULL_TABLE = "opendeviationbar_cache.open_deviation_bars"
def enabled_symbols_sql() -> str: ...  # AND symbol IN ('BTCUSDT', ...)
```

From scripts/heartbeat/checks/**init**.py:

- `run_checks() -> dict[str, bool | str]` -- orchestrator, calls all check modules
- `client` = clickhouse_connect.get_client(host="localhost", port=8123)

From scripts/heartbeat/config.py:

- `HEALTH_GATE_KEYS` -- frozenset controlling healthy/unhealthy decision
- `HEARTBEAT_PRIORITY_SYMBOLS = ("BTCUSDT", "ETHUSDT")` -- streaming symbols shown first

From scripts/heartbeat/formatter.py:

- `format_message(results, healthy) -> str` -- builds HTML Telegram message with sections
- Pattern: `lines.append("")` then `lines.append("<b>Section</b>")` then detail lines

From scripts/heartbeat/checks/sidecar.py:

- Sidecar health endpoint: `http://localhost:8081/health` (existing pattern)
  </interfaces>

</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add yesterday completion and junction telemetry check modules</name>
  <read_first>
    - python/opendeviationbar/sidecar.py (lines 523-570 -- _get_midnight_crossover_tid implementation)
    - scripts/heartbeat/checks/__init__.py (full file -- run_checks orchestrator)
    - scripts/heartbeat/checks/freshness.py (example check module pattern)
    - scripts/heartbeat/checks/sidecar.py (sidecar health check pattern)
    - scripts/heartbeat/config.py (HEALTH_GATE_KEYS, HEARTBEAT_PRIORITY_SYMBOLS)
    - python/opendeviationbar/clickhouse/stathera_query.py (FULL_TABLE, enabled_symbols_sql)
  </read_first>
  <files>
    scripts/heartbeat/checks/yesterday.py,
    scripts/heartbeat/checks/junction.py,
    scripts/heartbeat/checks/__init__.py,
    scripts/heartbeat/config.py,
    tests/test_heartbeat_yesterday_junction.py
  </files>
  <behavior>
    - Test 1: collect_yesterday_completion with matching orphan TID sets `yesterday_BTCUSDT` to True with detail "clean"
    - Test 2: collect_yesterday_completion with mismatching orphan TID sets `yesterday_BTCUSDT` to False with expected vs actual detail
    - Test 3: collect_yesterday_completion when no orphan bar exists sets `yesterday_BTCUSDT` to "no_orphan" (not False -- not an error, just fresh deploy)
    - Test 4: collect_yesterday_completion when crossover query fails returns gracefully (None -> skip symbol)
    - Test 5: collect_junction_telemetry with sidecar health response populates junction results
    - Test 6: collect_junction_telemetry with sidecar down sets junction keys to "unavailable"
  </behavior>
  <action>
    **yesterday.py** -- New check module `scripts/heartbeat/checks/yesterday.py`:

    ```python
    def collect_yesterday_completion(
        client: object,  # clickhouse_connect Client
        results: dict[str, bool | str],
    ) -> None:
    ```

    For each streaming symbol (read from symbol registry via `get_registered_symbols(enabled_only=True)` filtered to those in `HEARTBEAT_PRIORITY_SYMBOLS` plus any symbols the sidecar is streaming -- for now use HEARTBEAT_PRIORITY_SYMBOLS as the streaming set):

    1. **Get midnight crossover TID**: Reuse the logic from `sidecar.py:_get_midnight_crossover_tid()` but extract it into a standalone function here. Use `fetch_aggtrades_rest(symbol, midnight_ms, midnight_ms + 1000)` from `opendeviationbar._core`. Apply stamina retry (3 attempts, 30s timeout). If fails, set `results[f"yesterday_{symbol}"] = "query_failed"` and continue.

    2. **Query yesterday's orphan bar from ClickHouse**:
       ```sql
       SELECT last_agg_trade_id
       FROM {FULL_TABLE} FINAL
       WHERE symbol = '{symbol}'
         AND toDate(open_time_ms / 1000, 'UTC') = yesterday('UTC')
         AND is_orphan = 1
       ORDER BY last_agg_trade_id DESC
       LIMIT 1
       ```

    3. **Compare**: If orphan `last_agg_trade_id == crossover_tid - 1`: set `results[f"yesterday_{symbol}"] = True` and `results[f"yesterday_{symbol}_detail"] = "clean"`. If mismatch: set `results[f"yesterday_{symbol}"] = False` and `results[f"yesterday_{symbol}_detail"] = f"expected {crossover_tid - 1}, got {orphan_tid}"`. If no orphan: set `results[f"yesterday_{symbol}"] = "no_orphan"`.

    4. **Health gating**: Add `"yesterday_completion"` to `HEALTH_GATE_KEYS` in config.py. Set `results["yesterday_completion"] = all(results.get(f"yesterday_{s}") is not False for s in checked_symbols)`. This means: True if all symbols are clean or no_orphan or query_failed (graceful), False only if a real mismatch is detected.

    **junction.py** -- New check module `scripts/heartbeat/checks/junction.py`:

    ```python
    def collect_junction_telemetry(
        results: dict[str, bool | str],
    ) -> None:
    ```

    1. **Query sidecar health endpoint**: `GET http://localhost:8081/health` (same pattern as sidecar.py check). Parse JSON response. The sidecar health endpoint returns engine state including per-symbol info.

    2. **If sidecar is running**, also query `GET http://localhost:8081/status` which returns engine state. Extract per-symbol:
       - `rest_last_tid`: last trade ID from fill_from_rest (if available in engine state)
       - `ws_first_tid`: first WS trade ID received
       - `forming_bar`: current forming bar state (open_time, trade count)
       - `committed_floors`: current floor values

       NOTE: The sidecar may not expose all these fields via HTTP. If the `/status` endpoint doesn't exist or doesn't have this data, fall back to reading the most recent sidecar journal logs:
       ```python
       subprocess.run(
           ["journalctl", "--user", "-u", "opendeviationbar-sidecar",
            "--since", "30 min ago", "--no-pager", "-o", "short-precise"],
           capture_output=True, text=True, timeout=10
       )
       ```
       Parse log lines for:
       - `fill_from_rest` completion messages (contain last TID)
       - `WS trade received` or first trade messages
       - `committed_floors` log entries
       - `forming bar` state entries

    3. **Store results**:
       - `results["junction_telemetry"] = True` if data was collected, `False` if sidecar unavailable
       - `results["junction_rest_last_tid"] = str(rest_last_tid)` or "unavailable"
       - `results["junction_ws_first_tid"] = str(ws_first_tid)` or "unavailable"
       - `results["junction_forming_bar"] = str(forming_bar_summary)` or "unavailable"
       - `results["junction_committed_floors"] = str(floors_summary)` or "unavailable"

       Junction telemetry is informational only -- do NOT add to HEALTH_GATE_KEYS. Sidecar being down is already gated by the "sidecar" key.

    **Wire into run_checks()** in `checks/__init__.py`:
    - Import `collect_yesterday_completion` from `heartbeat.checks.yesterday`
    - Import `collect_junction_telemetry` from `heartbeat.checks.junction`
    - Call `collect_yesterday_completion(client, results)` after the Stathera audit block (needs client)
    - Call `collect_junction_telemetry(results)` after the sidecar dead letters check (no client needed)

    **Update config.py**:
    - Add `"yesterday_completion"` to `HEALTH_GATE_KEYS`

    **Tests**: Create `tests/test_heartbeat_yesterday_junction.py`:
    - Mock `fetch_aggtrades_rest` and ClickHouse client for yesterday tests
    - Mock `requests.get` / `subprocess.run` for junction tests
    - Test clean orphan match, mismatch, no orphan, query failure
    - Test junction with sidecar up and down

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_heartbeat_yesterday_junction.py -xvs 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - scripts/heartbeat/checks/yesterday.py exists and contains collect_yesterday_completion function
    - scripts/heartbeat/checks/junction.py exists and contains collect_junction_telemetry function
    - grep -c "collect_yesterday_completion" scripts/heartbeat/checks/__init__.py returns >= 1
    - grep -c "collect_junction_telemetry" scripts/heartbeat/checks/__init__.py returns >= 1
    - grep -c "yesterday_completion" scripts/heartbeat/config.py returns >= 1 (in HEALTH_GATE_KEYS)
    - tests/test_heartbeat_yesterday_junction.py exists with >= 5 test functions
    - pytest tests/test_heartbeat_yesterday_junction.py passes
  </acceptance_criteria>
  <done>
    - Yesterday completion check queries orphan bar and compares with midnight crossover TID per symbol
    - Junction telemetry collects REST/WS trade IDs, forming bar state, committed_floors from sidecar
    - Both checks wired into run_checks() orchestrator
    - yesterday_completion added to HEALTH_GATE_KEYS (mismatch = unhealthy)
    - Junction telemetry is informational only (not health-gating)
    - Tests cover match, mismatch, no orphan, query failure, sidecar up/down
  </done>
</task>

<task type="auto">
  <name>Task 2: Add yesterday completion and junction telemetry sections to formatter</name>
  <read_first>
    - scripts/heartbeat/formatter.py (full file -- to understand section ordering)
    - scripts/heartbeat/checks/yesterday.py (after Task 1 -- result key names)
    - scripts/heartbeat/checks/junction.py (after Task 1 -- result key names)
  </read_first>
  <files>
    scripts/heartbeat/formatter.py
  </files>
  <action>
    Add two new sections to `format_message()` in `scripts/heartbeat/formatter.py`:

    1. **Yesterday Completion section** -- Place AFTER the Stathera section, BEFORE the System section. Format:
       ```
       <b>Yesterday</b>
         {icon} BTCUSDT: clean (crossover tid {crossover_tid})
         {icon} ETHUSDT: mismatch -- expected {expected}, got {actual}
       ```
       - Read `results[f"yesterday_{symbol}"]` for each symbol in HEARTBEAT_PRIORITY_SYMBOLS
       - True -> checkmark + "clean"
       - False -> X + detail from `results[f"yesterday_{symbol}_detail"]`
       - "no_orphan" -> info icon + "no orphan bar (fresh deploy or gap)"
       - "query_failed" -> warning icon + "crossover query failed"
       - Skip section entirely if no `yesterday_*` keys exist in results

    2. **Junction Telemetry section** -- Place AFTER Yesterday, BEFORE System. Only show if `junction_telemetry` key exists in results. Format:
       ```
       <b>Junction</b>
         REST last TID: {value}
         WS first TID: {value}
         Forming bar: {value}
         Floors: {value}
       ```
       - Read `results["junction_rest_last_tid"]`, `results["junction_ws_first_tid"]`, `results["junction_forming_bar"]`, `results["junction_committed_floors"]`
       - If all are "unavailable", show single line: "info sidecar telemetry unavailable"
       - This section is purely informational -- no icons based on pass/fail

    Add `"yesterday_completion"` to the `_remediation` dict:
    ```python
    "yesterday_completion": "Yesterday's orphan bar doesn't match crossover TID — kintsugi will repair tomorrow",
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python3 -c "
from heartbeat.formatter import format_message
# Test with yesterday and junction results
results = {
    'clickhouse': True, 'sidecar': True, 'kintsugi': True, 'disk': True,
    'yesterday_BTCUSDT': True, 'yesterday_BTCUSDT_detail': 'clean',
    'yesterday_completion': True,
    'junction_telemetry': True,
    'junction_rest_last_tid': '12345678',
    'junction_ws_first_tid': '12345679',
    'junction_forming_bar': 'BTCUSDT open=42000',
    'junction_committed_floors': 'BTCUSDT@250=12345677',
}
msg = format_message(results, True)
assert 'Yesterday' in msg, 'Missing Yesterday section'
assert 'Junction' in msg, 'Missing Junction section'
assert 'clean' in msg
assert '12345678' in msg
print('formatter OK')
" 2>&1</automated>
  </verify>
  <acceptance_criteria>
    - grep -c "Yesterday" scripts/heartbeat/formatter.py returns >= 2 (section header + logic)
    - grep -c "Junction" scripts/heartbeat/formatter.py returns >= 2 (section header + logic)
    - grep -c "yesterday_completion" scripts/heartbeat/formatter.py returns >= 1 (in _remediation dict)
    - grep -c "junction_rest_last_tid\|junction_ws_first_tid" scripts/heartbeat/formatter.py returns >= 2
    - The inline verification script above prints "formatter OK"
  </acceptance_criteria>
  <done>
    - Yesterday completion section shows per-symbol orphan vs crossover TID status
    - Junction telemetry section shows REST/WS trade IDs, forming bar, committed_floors
    - Yesterday section only appears when yesterday_* keys exist
    - Junction section only appears when junction_telemetry key exists
    - Remediation hint added for yesterday_completion failures
  </done>
</task>

</tasks>

<verification>
1. `uv run --python 3.13 pytest tests/test_heartbeat_yesterday_junction.py -xvs` -- all tests pass
2. `grep -c "collect_yesterday_completion\|collect_junction_telemetry" scripts/heartbeat/checks/__init__.py` -- returns 2 (both wired)
3. `grep -c "yesterday_completion" scripts/heartbeat/config.py` -- returns >= 1 (in HEALTH_GATE_KEYS)
4. `python3 -c "from heartbeat.checks.yesterday import collect_yesterday_completion; from heartbeat.checks.junction import collect_junction_telemetry; print('OK')"` -- imports work
</verification>

<success_criteria>

- Yesterday completion check per symbol validates orphan vs crossover TID (TELEM-02)
- Junction telemetry collects REST/WS TIDs, forming bar, committed_floors (TELEM-03)
- Both visible in Telegram heartbeat message
- Yesterday completion is health-gating (mismatch = unhealthy)
- Junction telemetry is informational only
- Test coverage for both check modules
  </success_criteria>

<output>
After completion, create `.planning/phases/03-heartbeat-telemetry-overhaul/03-02-SUMMARY.md`
</output>
