---
phase: 04-ws-resilience-enhancements
plan: 02
subsystem: streaming
tags:
  [
    websocket,
    health-endpoint,
    heartbeat,
    telegram,
    error-classification,
    stream-signals,
  ]

# Dependency graph
requires:
  - "04-01: Rust WS error variants and StreamSignal enum for journal log parsing"
provides:
  - Stream signal tracking in sidecar /health and /status endpoints (CONNECT/FIRST_DATA/DISCONNECT/BAN_DETECTED)
  - WS error classification heartbeat check with distinct Telegram formatting for 429/ban vs transient
  - _classify_ws_log_line() testable helper for log line classification
affects: [sidecar, heartbeat, deploy]

# Tech tracking
tech-stack:
  added: []
  patterns:
    [
      stream-signal-health-tracking,
      ws-error-classification,
      journalctl-log-parsing,
    ]

key-files:
  created: []
  modified:
    - python/opendeviationbar/sidecar.py
    - scripts/heartbeat/checks/sidecar.py
    - scripts/heartbeat/checks/__init__.py
    - scripts/heartbeat/formatter.py

key-decisions:
  - "Stream signals derived from engine metrics reconnection counter rather than direct Rust channel -- simpler integration, no PyO3 channel piping needed"
  - "WS error classification uses _classify_ws_log_line() helper extractable for unit testing without journalctl dependency"
  - "429/ban errors classified as FATAL (ws_errors_healthy=False), transient disconnects as NORMAL (ws_errors_healthy=True)"
  - "ws_errors_healthy participates in health roll-up via existing bool-check pattern"

patterns-established:
  - "Stream signal tracking: _update_stream_signal() maintains timestamp + counter state in _health_state dict"
  - "Log line classification: testable pure function (_classify_ws_log_line) separated from I/O (journalctl subprocess)"

requirements-completed: []

# Metrics
duration: 4min
completed: 2026-03-22
---

# Phase 04 Plan 02: Stream Signals & WS Error Classification Summary

**Sidecar /health endpoint with WS connection lifecycle tracking (CONNECT/FIRST_DATA/DISCONNECT) and heartbeat Telegram alerts with 429-fatal vs transient-disconnect classification**

## Performance

- **Duration:** 4 min
- **Started:** 2026-03-22T20:13:58Z
- **Completed:** 2026-03-22T20:17:59Z
- **Tasks:** 2
- **Files modified:** 4

## Accomplishments

- Sidecar /health and /status endpoints now include stream_signals dict with connection lifecycle timestamps and counters
- \_update_stream_signal() function tracks CONNECT/FIRST_DATA/DISCONNECT/BAN_DETECTED signals from engine reconnection metrics
- Heartbeat check_ws_errors() scans sidecar journal for 429/ban (fatal) vs transient (normal) WS errors
- \_classify_ws_log_line() testable helper classifies log lines without journalctl dependency
- Telegram formatter shows FATAL WS alert with bold formatting for 429/ban, info line for transient reconnects
- ws_errors_healthy participates in health roll-up and remediation hints

## Task Commits

Each task was committed atomically:

1. **Task 1: Add stream signal tracking to sidecar health endpoint** - `bf638b8` (feat)
2. **Task 2: Add WS error classification check to heartbeat with distinct Telegram formatting** - `ad6b154` (feat)

## Files Created/Modified

- `python/opendeviationbar/sidecar.py` - stream_signals dict in_health_state,\_update_stream_signal() function, stream_signals in /health and /status responses, reconnection tracking in main loop, initial CONNECT/FIRST_DATA on engine start
- `scripts/heartbeat/checks/sidecar.py` - \_classify_ws_log_line() helper, check_ws_errors() function scanning journal for fatal vs transient WS errors
- `scripts/heartbeat/checks/__init__.py` - Wire check_ws_errors into run_checks() orchestrator
- `scripts/heartbeat/formatter.py` - FATAL WS bold alert for 429/ban, info line for transient reconnects, ws_errors_healthy remediation hint

## Decisions Made

- Stream signals are derived from the engine metrics `reconnections` counter rather than piping Rust tracing events through a channel -- simpler Python-side integration with no PyO3 overhead
- \_classify_ws_log_line() is a pure function returning string classification, separated from subprocess I/O, enabling unit testing without journalctl
- 429 and BAN errors are FATAL (set ws_errors_healthy=False, trigger LOUD Telegram alert) because they indicate Binance rate limit violation
- Transient disconnects are NORMAL (ws_errors_healthy remains True) because auto-reconnection handles them

## Deviations from Plan

None - plan executed exactly as written.

## Issues Encountered

None

## Known Stubs

None - all implementations are complete and functional.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- WS resilience enhancements complete (both Plan 01 and Plan 02)
- Stream signals available in /health for operational monitoring
- Heartbeat WS error classification ready for production deployment

---

_Phase: 04-ws-resilience-enhancements_
_Completed: 2026-03-22_
