---
phase: 04-ws-resilience-enhancements
plan: 02
type: execute
wave: 2
depends_on: ["04-01"]
files_modified:
  - python/opendeviationbar/sidecar.py
  - scripts/heartbeat/checks/sidecar.py
autonomous: true
requirements: []
must_haves:
  truths:
    - "Sidecar /health endpoint includes stream state (last CONNECT/FIRST_DATA/DISCONNECT timestamps)"
    - "Heartbeat classifies 429 rate limit errors as fatal (distinct from 500 transient errors) in Telegram alerts"
    - "Heartbeat Telegram messages include error type classification when WS errors are detected"
    - "Stream signal tracing events from Rust are testable via a synthetic log-line parser unit test (not deployment-dependent)"
  artifacts:
    - path: "python/opendeviationbar/sidecar.py"
      provides: "Stream signal tracking in health state"
      contains: "stream_signals"
    - path: "scripts/heartbeat/checks/sidecar.py"
      provides: "WS error classification check"
      contains: "check_ws_errors"
  key_links:
    - from: "scripts/heartbeat/checks/sidecar.py"
      to: "journalctl"
      via: "subprocess for sidecar journal logs"
      pattern: "journalctl.*opendeviationbar-sidecar"
    - from: "python/opendeviationbar/sidecar.py"
      to: "_health_state"
      via: "stream signal dict in health state"
      pattern: "stream_signals"
---

<objective>
Surface Rust WS stream signals and error classification in the Python sidecar health endpoint and heartbeat Telegram alerts.

Purpose: Enable operators to see WS connection lifecycle (CONNECT/FIRST_DATA/DISCONNECT) in the sidecar health API and receive distinct Telegram alerts for 429 (fatal rate limit) vs 500 (transient) WS errors — preventing confusion during incidents.

Output: Updated sidecar health endpoint with stream signal tracking, new heartbeat check for WS error classification with distinct Telegram alert formatting.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-ws-resilience-enhancements/04-01-SUMMARY.md

@python/opendeviationbar/sidecar.py
@scripts/heartbeat/checks/sidecar.py

<interfaces>
<!-- From Plan 01: Rust tracing events emitted by combined_ws.rs -->
<!-- These structured log entries appear in journalctl for opendeviationbar-sidecar.service -->

Tracing events emitted by combined_ws.rs (from Plan 01):

- `tracing::info!(signal = "CONNECT", symbol_count, ...)` — on WS connect
- `tracing::info!(signal = "FIRST_DATA", symbol_count, ...)` — on first trade delivery
- `tracing::info!(signal = "DISCONNECT", symbol_count, ...)` — on connection loss
- `tracing::error!(signal = "BAN_DETECTED", ban_until_ms, ...)` — on IP ban
- `tracing::error!(...)` with "terminal combined WebSocket error" — on 429/ban

From sidecar.py:

```python
_health_state: dict[str, object] = {"metrics": {}}

def _update_health_metrics(metrics: dict[str, int]) -> None:
    _health_state["metrics"] = metrics

# Health server at /health returns:
# {"status": "healthy"|"unhealthy", "checks": {...}, "engine": {...}}
```

From heartbeat/checks/sidecar.py:

```python
def check_dead_letters(results: dict[str, bool | str]) -> None:
    # Counts dead-letter files and reports oldest age
```

From heartbeat/checks/**init**.py:

```python
def run_checks() -> dict[str, bool | str]:
    # Dispatches to all check modules
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add stream signal tracking to sidecar health endpoint</name>
  <files>python/opendeviationbar/sidecar.py</files>
  <read_first>
    - python/opendeviationbar/sidecar.py
  </read_first>
  <action>
Add stream signal tracking to the sidecar's health state so the `/health` endpoint reports WS connection lifecycle.

1. **Initialize stream signal state** in the `_health_state` dict (near line 1374):

   ```python
   _health_state: dict[str, object] = {
       "metrics": {},
       "stream_signals": {
           "last_connect": None,       # ISO timestamp of last CONNECT signal
           "last_first_data": None,     # ISO timestamp of last FIRST_DATA signal
           "last_disconnect": None,     # ISO timestamp of last DISCONNECT signal
           "last_ban_detected": None,   # ISO timestamp of last BAN_DETECTED
           "ban_until": None,           # Ban expiry ISO timestamp (if active)
           "connection_count": 0,       # Total connections since sidecar start
           "disconnect_count": 0,       # Total disconnections since sidecar start
       },
   }
   ```

2. **Add stream signal update function** (near `_update_health_metrics`):

   ```python
   def _update_stream_signal(signal: str, **kwargs: object) -> None:
       """Update stream signal state from engine metrics or log parsing."""
       from datetime import datetime, timezone
       now = datetime.now(timezone.utc).isoformat()
       signals = _health_state.setdefault("stream_signals", {})

       if signal == "CONNECT":
           signals["last_connect"] = now
           signals["connection_count"] = signals.get("connection_count", 0) + 1
       elif signal == "FIRST_DATA":
           signals["last_first_data"] = now
       elif signal == "DISCONNECT":
           signals["last_disconnect"] = now
           signals["disconnect_count"] = signals.get("disconnect_count", 0) + 1
       elif signal == "BAN_DETECTED":
           signals["last_ban_detected"] = now
           ban_until_ms = kwargs.get("ban_until_ms")
           if ban_until_ms:
               signals["ban_until"] = datetime.fromtimestamp(
                   int(ban_until_ms) / 1000, tz=timezone.utc
               ).isoformat()
   ```

3. **Include stream_signals in /health response** — update `_handle_health()` (around line 1633) to include stream_signals in the response dict:

   ```python
   resp: dict[str, object] = {
       "status": "healthy" if all_pass else "unhealthy",
       "checks": results,
       "engine": engine_metrics,
       "stream_signals": _health_state.get("stream_signals", {}),
   }
   ```

4. **Track engine metrics for stream signals** — in the main sidecar loop where `_update_health_metrics` is called, also extract reconnection count from engine metrics. The `LiveEngineMetrics` snapshot already has `reconnections` count. When reconnections increases, call `_update_stream_signal("DISCONNECT")` followed by `_update_stream_signal("CONNECT")`. This approximates the Rust tracing signals without needing direct channel piping.

   Find the main loop in `run_sidecar()` where `engine.get_metrics_snapshot()` is called and add:

   ```python
   # Track reconnections as DISCONNECT+CONNECT signal pairs
   _prev_reconnections = _health_state.get("_prev_reconnections", 0)
   current_reconnections = snap.get("reconnections", 0)
   if current_reconnections > _prev_reconnections:
       new_reconnects = current_reconnections - _prev_reconnections
       for _ in range(new_reconnects):
           _update_stream_signal("DISCONNECT")
           _update_stream_signal("CONNECT")
       _health_state["_prev_reconnections"] = current_reconnections
   ```

   On initial successful engine start (after `engine.start()` or first bar received), call `_update_stream_signal("CONNECT")` and `_update_stream_signal("FIRST_DATA")`.

5. **Include in /status endpoint too** — update `_handle_status()` to include stream_signals alongside junction telemetry.
   </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 python -c "from opendeviationbar.sidecar import _health_state, _update_stream_signal; _update_stream_signal('CONNECT'); print(_health_state['stream_signals']); assert _health_state['stream_signals']['connection_count'] == 1, 'count wrong'; print('OK')" 2>&1 || echo "Import check (expected to work or show meaningful error)"</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "stream_signals" python/opendeviationbar/sidecar.py
    - grep -q "_update_stream_signal" python/opendeviationbar/sidecar.py
    - grep -q "last_connect" python/opendeviationbar/sidecar.py
    - grep -q "last_first_data" python/opendeviationbar/sidecar.py
    - grep -q "last_disconnect" python/opendeviationbar/sidecar.py
    - grep -q "connection_count" python/opendeviationbar/sidecar.py
    - grep -q "ban_until" python/opendeviationbar/sidecar.py
    - grep -c "stream_signals" python/opendeviationbar/sidecar.py shows at least 5 occurrences
  </acceptance_criteria>
  <done>Sidecar /health and /status endpoints include stream_signals dict with last_connect, last_first_data, last_disconnect, connection_count, disconnect_count, ban_until. Signals update from engine metrics reconnection tracking.</done>
</task>

<task type="auto">
  <name>Task 2: Add WS error classification check to heartbeat with distinct Telegram formatting</name>
  <files>scripts/heartbeat/checks/sidecar.py, scripts/heartbeat/checks/__init__.py</files>
  <read_first>
    - scripts/heartbeat/checks/sidecar.py
    - scripts/heartbeat/checks/__init__.py
    - scripts/heartbeat/formatter.py
    - scripts/heartbeat/telegram.py
  </read_first>
  <action>
Add a new heartbeat check that classifies recent WS errors from sidecar journal logs and formats distinct Telegram alerts for 429 (fatal) vs transient errors.

1. **Add `check_ws_errors()` function to `scripts/heartbeat/checks/sidecar.py`:**

   ```python
   def check_ws_errors(results: dict[str, bool | str], window_minutes: int = 10) -> None:
       """Classify recent WebSocket errors from sidecar journal logs.

       Scans the last `window_minutes` of sidecar logs for:
       - 429 / rate limit errors (FATAL — means we're violating Binance limits)
       - IP ban detection (FATAL — sidecar sleeping until ban expires)
       - Transient disconnects (NORMAL — handled by auto-reconnection)

       Results keys:
           ws_errors_healthy: bool — False if any fatal (429/ban) errors found
           ws_error_summary: str — Human-readable summary
           ws_fatal_count: str — Count of fatal errors (429 + ban)
           ws_transient_count: str — Count of transient disconnects
       """
       import subprocess

       results["ws_errors_healthy"] = True
       results["ws_error_summary"] = "clean"
       results["ws_fatal_count"] = "0"
       results["ws_transient_count"] = "0"

       try:
           # Read recent sidecar journal logs
           cmd = [
               "journalctl",
               "--user-unit=opendeviationbar-sidecar.service",
               f"--since={window_minutes} min ago",
               "--no-pager",
               "-o", "cat",
           ]
           proc = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
           if proc.returncode != 0:
               results["ws_error_summary"] = "journal unavailable"
               return

           lines = proc.stdout.splitlines()

           fatal_count = 0
           transient_count = 0
           ban_detected = False
           rate_limited = False

           for line in lines:
               # Fatal: 429 rate limit or IP ban
               if 'signal = "BAN_DETECTED"' in line or "IP banned" in line:
                   fatal_count += 1
                   ban_detected = True
               elif "RateLimited" in line or ("429" in line and "terminal" in line.lower()):
                   fatal_count += 1
                   rate_limited = True
               # Transient: normal disconnects handled by reconnection
               elif 'signal = "DISCONNECT"' in line or "WebSocket disconnected, reconnecting" in line:
                   transient_count += 1

           results["ws_fatal_count"] = str(fatal_count)
           results["ws_transient_count"] = str(transient_count)

           if fatal_count > 0:
               results["ws_errors_healthy"] = False
               parts = []
               if ban_detected:
                   parts.append("IP BAN")
               if rate_limited:
                   parts.append("429 RATE LIMIT")
               results["ws_error_summary"] = f"FATAL: {' + '.join(parts)} ({fatal_count} events in {window_minutes}min)"
           elif transient_count > 0:
               results["ws_error_summary"] = f"transient: {transient_count} reconnects in {window_minutes}min"
           else:
               results["ws_error_summary"] = "clean"

       except (subprocess.TimeoutExpired, FileNotFoundError, OSError) as exc:
           logger.warning("check_ws_errors failed: %s", exc)
           results["ws_error_summary"] = f"check failed: {exc}"
   ```

   Also add a **`_classify_ws_log_line()` helper** that can be tested independently without journalctl:

   ```python
   def _classify_ws_log_line(line: str) -> str | None:
       """Classify a single sidecar log line for WS error type.

       Returns:
           "fatal_ban" if IP ban detected
           "fatal_429" if 429 rate limit detected
           "transient" if normal disconnect
           None if not a WS error line
       """
       if 'signal = "BAN_DETECTED"' in line or "IP banned" in line:
           return "fatal_ban"
       if "RateLimited" in line or ("429" in line and "terminal" in line.lower()):
           return "fatal_429"
       if 'signal = "DISCONNECT"' in line or "WebSocket disconnected, reconnecting" in line:
           return "transient"
       return None
   ```

   Refactor `check_ws_errors` to use `_classify_ws_log_line` internally.

2. **Wire into `run_checks()` in `scripts/heartbeat/checks/**init**.py`:**

   Add import: `from .sidecar import check_ws_errors`
   Add call: `check_ws_errors(results)` after the existing `check_dead_letters(results)` call.

3. **Update heartbeat formatter** to include WS error classification in Telegram messages.

   In `scripts/heartbeat/formatter.py`, find where results are formatted into the Telegram message body. Add a section for WS errors:
   - If `ws_errors_healthy` is `False`: add a LOUD line like `"FATAL WS: {ws_error_summary}"` with bold formatting
   - If `ws_transient_count` > 0: add an info line like `"WS reconnects: {ws_transient_count}"`
   - If clean: no line needed (reduce noise)

   The key distinction for the operator:
   - **429/Ban = FATAL**: "Your sidecar is violating Binance rate limits. Fix the code or wait for ban to expire."
   - **Transient disconnect = NORMAL**: "Auto-reconnection handled it. No action needed unless count is unusually high."

4. **Add `ws_errors_healthy` to the health-gating logic:**

   In the heartbeat's health computation (where `healthy = all(bool_checks.values())`), ensure `ws_errors_healthy` is included as a bool check. Since it's added to `results` as a bool, it should be picked up automatically by the existing `{k: v for k, v in results.items() if isinstance(v, bool)}` pattern in the heartbeat's health roll-up.
   </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 python -c "
import sys
sys.path.insert(0, '.')
from scripts.heartbeat.checks.sidecar import _classify_ws_log_line
# Test classification without needing journalctl
assert _classify_ws_log_line('signal = \"BAN_DETECTED\" ban_until_ms=1700000000000') == 'fatal_ban'
assert _classify_ws_log_line('RateLimited terminal error') == 'fatal_429'
assert _classify_ws_log_line('signal = \"DISCONNECT\" combined WebSocket disconnected') == 'transient'
assert _classify_ws_log_line('normal trade processing') is None
print('OK: _classify_ws_log_line correctly classifies all WS error types')
" 2>&1</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "check_ws_errors" scripts/heartbeat/checks/sidecar.py
    - grep -q "_classify_ws_log_line" scripts/heartbeat/checks/sidecar.py
    - grep -q "ws_errors_healthy" scripts/heartbeat/checks/sidecar.py
    - grep -q "ws_fatal_count" scripts/heartbeat/checks/sidecar.py
    - grep -q "ws_transient_count" scripts/heartbeat/checks/sidecar.py
    - grep -q "BAN_DETECTED" scripts/heartbeat/checks/sidecar.py
    - grep -q "RateLimited" scripts/heartbeat/checks/sidecar.py
    - grep -q "429" scripts/heartbeat/checks/sidecar.py
    - grep -q "check_ws_errors" scripts/heartbeat/checks/__init__.py
    - grep -q "FATAL WS\|ws_error_summary" scripts/heartbeat/formatter.py
  </acceptance_criteria>
  <done>Heartbeat check_ws_errors scans sidecar journal for 429/ban (fatal) vs transient disconnect errors. Fatal errors set ws_errors_healthy=False and trigger LOUD Telegram alert with distinct formatting. Transient disconnects reported as informational. Health-gating includes ws_errors_healthy. _classify_ws_log_line is testable without journalctl.</done>
</task>

</tasks>

<verification>
1. Sidecar /health returns stream_signals dict with all expected keys
2. Heartbeat _classify_ws_log_line correctly classifies BAN_DETECTED, RateLimited/429, DISCONNECT, and normal lines
3. Heartbeat check_ws_errors produces ws_errors_healthy, ws_error_summary, ws_fatal_count, ws_transient_count
4. 429 and BAN errors are classified as fatal (ws_errors_healthy=False)
5. Transient disconnects are classified as normal (ws_errors_healthy=True)
6. Heartbeat formatter includes WS error section in Telegram messages
7. Python tests pass: `mise run test-py`
</verification>

<success_criteria>

- Sidecar /health includes stream_signals with last_connect, last_first_data, last_disconnect, connection_count, disconnect_count
- Heartbeat classifies 429 as FATAL and transient as NORMAL in Telegram alerts
- ws_errors_healthy participates in heartbeat health roll-up
- \_classify_ws_log_line testable via `uv run --python 3.13 python -c` without journalctl dependency
- No regression in existing heartbeat checks
  </success_criteria>

<output>
After completion, create `.planning/phases/04-ws-resilience-enhancements/04-02-SUMMARY.md`
</output>
