---
phase: 04-ws-resilience-enhancements
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - crates/opendeviationbar-providers/src/binance/websocket.rs
  - crates/opendeviationbar-providers/src/binance/combined_ws.rs
autonomous: true
requirements: []
must_haves:
  truths:
    - "Client-side ping frames are sent every 5 seconds on combined WS connections"
    - "Pong timeout of 10 seconds triggers reconnection via a per-ping deadline timer — if no Pong frame arrives within 10s of sending a Ping, connection is treated as dead and returns PongTimeout error (which is recoverable, triggering reconnection)"
    - "WebSocket subscribe/unsubscribe operations are paced at max 3 effective messages/sec (5 total minus 2 reserved for pings)"
    - "429 rate limit errors are classified as terminal (no retry storm)"
    - "IP ban duration is extracted from Binance error responses and sidecar sleeps until expiry"
    - "Stream state signals (Connect, FirstData, Disconnect) are emitted as tracing events"
    - "Per-symbol mpsc trade channels are already bounded at 50,000 capacity (live_engine.rs:324) and completed bar output uses ConcurrentRingBuffer with configurable OPENDEVIATIONBAR_MAX_PENDING_BARS (default 10K) — no new backpressure work needed for SC-6"
  artifacts:
    - path: "crates/opendeviationbar-providers/src/binance/websocket.rs"
      provides: "New error variants, keepalive config, stream signal enum"
      contains: "RateLimited"
    - path: "crates/opendeviationbar-providers/src/binance/combined_ws.rs"
      provides: "Client-side ping sender with pong deadline, subscribe rate limiter, ban handling"
      contains: "ping_interval"
  key_links:
    - from: "combined_ws.rs"
      to: "websocket.rs"
      via: "use super::websocket::{WebSocketError, ReconnectionPolicy, StreamSignal}"
      pattern: "WebSocketError::RateLimited|WebSocketError::BanDetected|WebSocketError::PongTimeout"
---

<objective>
Harden the Rust WebSocket layer with client-side keepalive pings (with real 10s pong deadline), subscribe rate limiting, 429/ban error handling, and stream state signal emission.

Purpose: Prevent silent connection death in NAT/proxy environments, avoid Binance rate limit violations on control messages, handle IP bans gracefully instead of retry-storming, and provide observability signals for downstream consumers.

Output: Updated `websocket.rs` with new types/config and `combined_ws.rs` with ping sender (including pong deadline tracking), subscribe pacer, ban handler, and signal emission.

Note on SC-6 (ring buffer OOM prevention): Investigation confirms this is ALREADY COVERED. Per-symbol `mpsc::channel::<AggTrade>(50_000)` in `live_engine.rs:324` provides bounded trade input channels. Completed bar output uses `ConcurrentRingBuffer` (capacity from `OPENDEVIATIONBAR_MAX_PENDING_BARS`, default 10K) with drop-on-full semantics and `dropped_bars`/`backpressure_events` metrics. No new work required.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@crates/opendeviationbar-providers/src/binance/websocket.rs
@crates/opendeviationbar-providers/src/binance/combined_ws.rs
@crates/opendeviationbar-providers/src/binance/rate_limiter.rs

<interfaces>
<!-- Key types the executor needs from existing code -->

From websocket.rs:

```rust
#[derive(Error, Debug)]
pub enum WebSocketError {
    ConnectionFailed(Box<tokio_tungstenite::tungstenite::Error>),
    JsonParsingFailed(#[from] serde_json::Error),
    ChannelSendFailed,
    InvalidSymbol(String),
    ConnectionClosed,
    PongSendFailed(String),
}

impl WebSocketError {
    pub const fn is_terminal(&self) -> bool { ... }
}

#[derive(Debug, Clone)]
pub struct ReconnectionPolicy {
    pub backoff_ms_initial: u64,
    pub backoff_multiplier: u64,
    pub backoff_ms_max: u64,
    pub data_timeout_secs: u64,
    pub stable_connection_ms: u64,
}
```

From combined_ws.rs:

```rust
pub struct CombinedWebSocketStream { symbols: Vec<String> }

impl CombinedWebSocketStream {
    pub async fn connect_and_demux(
        symbols: &[String],
        senders: &HashMap<String, mpsc::Sender<AggTrade>>,
        shutdown: &CancellationToken,
        data_timeout: Duration,
        first_trade_signal: Option<oneshot::Sender<()>>,
    ) -> Result<(), WebSocketError>;

    pub async fn run_with_reconnect(
        symbols: Vec<String>,
        senders: HashMap<String, mpsc::Sender<AggTrade>>,
        policy: ReconnectionPolicy,
        shutdown: CancellationToken,
    );
}
```

From live_engine.rs (existing backpressure — NO changes needed):

```rust
// line 324: Per-symbol trade channels are BOUNDED at 50K
let (trade_tx, trade_rx) = mpsc::channel::<AggTrade>(50_000);

// line 236: Bar output uses ConcurrentRingBuffer (configurable via OPENDEVIATIONBAR_MAX_PENDING_BARS)
let bar_buffer = ConcurrentRingBuffer::new(config.bar_channel_capacity);

// Metrics track drops: backpressure_events, dropped_bars, max_queue_depth
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add WS error variants, keepalive config, and stream signal enum to websocket.rs</name>
  <files>crates/opendeviationbar-providers/src/binance/websocket.rs</files>
  <read_first>
    - crates/opendeviationbar-providers/src/binance/websocket.rs
  </read_first>
  <action>
Modify `websocket.rs` to add three things:

1. **New error variants to `WebSocketError`**:
   - `RateLimited { retry_after_ms: Option<u64> }` with message "Rate limited by Binance WebSocket (429): retry after {retry_after_ms:?}ms". This is TERMINAL (`is_terminal() -> true`) — 429 on WS means we are violating rate limits and must stop, not retry-storm.
   - `BanDetected { ban_until_ms: u64, reason: String }` with message "IP banned by Binance until {ban_until_ms}: {reason}". This is TERMINAL.
   - `PongTimeout` with message "Pong timeout -- server did not respond to ping within deadline". This is NOT terminal (recoverable -- triggers reconnection).

   Update `is_terminal()` match arms:
   - `RateLimited { .. }` => `true`
   - `BanDetected { .. }` => `true`
   - `PongTimeout` => `false`

2. **Keepalive fields to `ReconnectionPolicy`**:
   - `ping_interval_secs: u64` — How often to send client-side ping frames. Default: `5`.
   - `pong_timeout_secs: u64` — Max wait for pong response before treating connection as dead. Default: `10`.
   - `max_subscribe_msgs_per_sec: u32` — Max WS control messages per second (Binance limit is 5). Default: `5`.
   - `ping_reserve_msgs_per_sec: u32` — Messages reserved for ping frames per second. Default: `2`.

   Update `Default` impl to include these fields with the specified defaults.

3. **Stream signal enum** (public, used by combined_ws.rs and live_engine.rs):

   ```rust
   /// Stream lifecycle signals for consumer observability.
   /// Emitted as tracing events and optionally via signal channel.
   #[derive(Debug, Clone, Copy, PartialEq, Eq)]
   pub enum StreamSignal {
       /// WebSocket TCP connection established
       Connect,
       /// First trade data received on this connection
       FirstData,
       /// Connection lost (intentional or error)
       Disconnect,
   }
   ```

   Export `StreamSignal` from the module's public API.

4. **Update tests**:
   - Add test `test_rate_limited_is_terminal` asserting `WebSocketError::RateLimited { retry_after_ms: Some(60000) }.is_terminal()` is `true`.
   - Add test `test_ban_detected_is_terminal` asserting `WebSocketError::BanDetected { ban_until_ms: 1700000000000, reason: "test".into() }.is_terminal()` is `true`.
   - Add test `test_pong_timeout_is_not_terminal` asserting `WebSocketError::PongTimeout.is_terminal()` is `false`.
   - Add test `test_keepalive_defaults` asserting `ReconnectionPolicy::default().ping_interval_secs == 5` and `pong_timeout_secs == 10` and `max_subscribe_msgs_per_sec == 5` and `ping_reserve_msgs_per_sec == 2`.
   - Update existing `test_reconnection_policy_default` to also assert the new fields.

5. **Update `BinanceWebSocketStream::connect_and_stream()`** to add client-side ping with real pong deadline (for legacy per-symbol path):

   Add a `tokio::time::interval` for ping_interval_secs and a `Option<tokio::time::Instant>` to track when last ping was sent (pong deadline). In the `tokio::select!` loop:
   - **Ping branch**: On tick, send `Message::Ping(vec![])` and record `last_ping_sent = Some(Instant::now())`.
   - **Pong tracking in message branch**: When a `Message::Pong(_)` is received, clear `last_ping_sent = None`.
   - **Pong deadline check**: After each select iteration (or as an additional select branch using `tokio::time::sleep_until`), if `last_ping_sent` is `Some(t)` and `t.elapsed() > pong_timeout`, return `Err(WebSocketError::PongTimeout)`.

   Implementation approach for the pong deadline select branch:

   ```rust
   // Compute pong deadline future
   let pong_deadline = match last_ping_sent {
       Some(sent_at) => {
           let deadline = sent_at + pong_timeout;
           tokio::time::sleep_until(tokio::time::Instant::from_std(deadline))
       }
       None => tokio::time::sleep(Duration::from_secs(86400)), // effectively never
   };

   // In select!:
   _ = pong_deadline => {
       tracing::warn!(%symbol, pong_timeout_secs = policy.pong_timeout_secs, "pong timeout — no response within deadline");
       return Err(WebSocketError::PongTimeout);
   }
   ```

   Note: The `pin!` macro may be needed to pin the future for use in select. Use `tokio::pin!` or restructure as needed.
   </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-providers --no-fail-fast 2>&1 | tail -20</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "RateLimited" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "BanDetected" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "PongTimeout" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "ping_interval_secs" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "pong_timeout_secs" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "max_subscribe_msgs_per_sec" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "StreamSignal" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "test_rate_limited_is_terminal" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "test_ban_detected_is_terminal" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "test_pong_timeout_is_not_terminal" crates/opendeviationbar-providers/src/binance/websocket.rs
    - grep -q "ping_interval_secs: 5" crates/opendeviationbar-providers/src/binance/websocket.rs
    - cargo nextest run -p opendeviationbar-providers exits 0
  </acceptance_criteria>
  <done>WebSocketError has RateLimited (terminal), BanDetected (terminal), PongTimeout (recoverable) variants. ReconnectionPolicy has ping_interval_secs=5, pong_timeout_secs=10, max_subscribe_msgs_per_sec=5, ping_reserve_msgs_per_sec=2. StreamSignal enum exported. Legacy per-symbol WS has real pong deadline tracking. All tests pass.</done>
</task>

<task type="auto">
  <name>Task 2: Implement ping sender with pong deadline, subscribe rate limiter, ban handling, and stream signals in combined_ws.rs</name>
  <files>crates/opendeviationbar-providers/src/binance/combined_ws.rs</files>
  <read_first>
    - crates/opendeviationbar-providers/src/binance/combined_ws.rs
    - crates/opendeviationbar-providers/src/binance/websocket.rs
  </read_first>
  <action>
Modify `combined_ws.rs` to implement the four WS hardening features:

**1. Client-side ping sender with real pong deadline in `connect_and_demux()`:**

Add `ping_interval: Duration` and `pong_timeout: Duration` as parameters (derived from `ReconnectionPolicy` by the caller `run_with_reconnect`). Inside the `tokio::select!` loop, implement a proper ping/pong deadline tracker:

```rust
let mut ping_ticker = tokio::time::interval(ping_interval);
ping_ticker.tick().await; // consume first immediate tick
let mut last_ping_sent: Option<std::time::Instant> = None;
```

Add to the select loop:

```rust
// Ping branch: send ping and start pong deadline
_ = ping_ticker.tick() => {
    if let Err(e) = ws_sender.send(Message::Ping(vec![])).await {
        tracing::warn!(%e, "combined stream client-side ping failed");
        return Err(WebSocketError::PongSendFailed(e.to_string()));
    }
    last_ping_sent = Some(std::time::Instant::now());
    tracing::trace!("combined stream client-side ping sent");
}
```

In the message-receive branch, when `Message::Pong(_)` is received, clear the deadline:

```rust
Some(Ok(Message::Pong(_))) => {
    last_ping_sent = None;
    tracing::trace!("combined stream pong received");
}
```

Add a pong deadline check. After each select iteration or as an additional select branch, check:

```rust
// At the start of each loop iteration, before select!:
if let Some(sent_at) = last_ping_sent {
    if sent_at.elapsed() > pong_timeout {
        tracing::warn!(
            pong_timeout_ms = pong_timeout.as_millis(),
            "pong timeout -- no response to ping within deadline, reconnecting"
        );
        return Err(WebSocketError::PongTimeout);
    }
}
```

Alternatively, integrate the deadline as a `tokio::time::sleep_until` branch in the select. The key requirement is: if a Ping is sent and no Pong arrives within `pong_timeout` (10s), the function returns `PongTimeout`. This is distinct from `data_timeout` (90s) which covers overall silence. The 10s pong timeout catches the specific case where the server receives data but does not respond to our pings (proxy/NAT issue, half-open TCP where keepalive hasn't fired yet).

**2. Update `connect_and_demux()` signature:**

Add parameters:

- `ping_interval: Duration`
- `pong_timeout: Duration`

Update the function to accept these. The caller `run_with_reconnect` computes them from `ReconnectionPolicy`:

```rust
let ping_interval = Duration::from_secs(policy.ping_interval_secs);
let pong_timeout = Duration::from_secs(policy.pong_timeout_secs);
```

**3. Stream signal emission via tracing:**

At key lifecycle points, emit structured tracing events:

- After `connect_async` succeeds: `tracing::info!(signal = "CONNECT", symbol_count, "combined WebSocket connected -- stream signal");`
- After first trade is sent to a channel (where `first_trade_signal.take()` fires): `tracing::info!(signal = "FIRST_DATA", symbol_count, "combined WebSocket first trade delivered -- stream signal");`
- On `ConnectionClosed` / close frame / stream end: `tracing::info!(signal = "DISCONNECT", symbol_count, "combined WebSocket disconnected -- stream signal");`

These structured log events can be parsed by Python heartbeat via journalctl.

**4. Ban detection in `run_with_reconnect()`:**

After a `ConnectionFailed` error in the reconnect loop, check if the error message contains Binance ban indicators. Binance close frames may include JSON like `{"code":-1015,"msg":"Too many requests; please use the websocket for live updates. IP banned until 1700000000000."}`.

Add a helper function:

```rust
/// Parse Binance error responses for ban duration.
/// Returns ban expiry timestamp (ms) if detected.
fn parse_ban_duration(error_msg: &str) -> Option<u64> {
    // Look for "IP banned until <timestamp>" pattern
    if let Some(idx) = error_msg.find("banned until ") {
        let after = &error_msg[idx + "banned until ".len()..];
        // Extract digits
        let digits: String = after.chars().take_while(|c| c.is_ascii_digit()).collect();
        digits.parse::<u64>().ok()
    } else {
        None
    }
}
```

In `run_with_reconnect`, when handling close frames in `connect_and_demux`, check for 429/ban patterns. When the close frame has code 1008 (Policy Violation) or the text contains "429" or "banned":

- If ban detected: emit `tracing::error!(signal = "BAN_DETECTED", ban_until_ms, ...)`, sleep until `ban_until_ms` instead of normal backoff.
- If 429 detected without ban duration: treat as terminal, return (do not retry).

Modify the `Close` frame handler in `connect_and_demux` to inspect close reason:

```rust
Some(Ok(Message::Close(frame))) => {
    let reason = frame.as_ref().map(|f| f.reason.to_string()).unwrap_or_default();
    let code = frame.as_ref().map(|f| f.code);

    tracing::info!(signal = "DISCONNECT", ?frame, "combined WebSocket closed by server");

    // Check for rate limit / ban
    if reason.contains("429") || reason.contains("Too many requests") {
        if let Some(ban_until) = parse_ban_duration(&reason) {
            return Err(WebSocketError::BanDetected {
                ban_until_ms: ban_until,
                reason: reason.clone(),
            });
        }
        return Err(WebSocketError::RateLimited {
            retry_after_ms: None,
        });
    }

    return Err(WebSocketError::ConnectionClosed);
}
```

In `run_with_reconnect`, add special handling for `BanDetected`:

```rust
Err(WebSocketError::BanDetected { ban_until_ms, ref reason }) => {
    let now_ms = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap()
        .as_millis() as u64;
    let sleep_ms = ban_until_ms.saturating_sub(now_ms);
    tracing::error!(
        ban_until_ms,
        sleep_secs = sleep_ms / 1000,
        %reason,
        "IP banned by Binance -- sleeping until ban expires"
    );
    if sleep_ms > 0 {
        tokio::select! {
            () = tokio::time::sleep(Duration::from_millis(sleep_ms)) => {
                tracing::info!("ban sleep complete, resuming");
                // Reset backoff after ban sleep
                backoff = Self::build_backoff(&policy);
                continue; // Do NOT break -- resume after ban
            }
            () = shutdown.cancelled() => {
                tracing::info!("shutdown during ban sleep");
                break;
            }
        }
    }
    // If ban already expired, just continue
    backoff = Self::build_backoff(&policy);
    continue;
}
```

Note: `BanDetected` is terminal in `is_terminal()`, but in `run_with_reconnect` we handle it specially BEFORE the terminal check -- we sleep until ban expires then resume. The terminal classification prevents lower-level code from blindly retrying.

Move the `BanDetected` match arm BEFORE the `e.is_terminal()` check in the error handling block.

And add special handling for `PongTimeout` -- it is recoverable (not terminal), so the existing reconnect logic will handle it automatically. Just log distinctly:

```rust
Err(WebSocketError::PongTimeout) => {
    tracing::warn!("pong timeout -- reconnecting");
    // Falls through to normal reconnect (PongTimeout.is_terminal() == false)
}
```

**5. Subscribe rate limiter helper (for future subscribe/unsubscribe operations):**

Add a simple token-bucket style rate limiter for WS control messages:

```rust
/// Rate limiter for WebSocket control messages (subscribe/unsubscribe).
/// Binance enforces max 5 control messages/sec including pings.
pub(crate) struct WsControlRateLimiter {
    /// Minimum interval between control messages
    min_interval: Duration,
    /// Last message send time
    last_send: Option<std::time::Instant>,
}

impl WsControlRateLimiter {
    pub fn new(max_msgs_per_sec: u32, reserved_for_pings: u32) -> Self {
        let effective = max_msgs_per_sec.saturating_sub(reserved_for_pings).max(1);
        Self {
            min_interval: Duration::from_millis(1000 / effective as u64),
            last_send: None,
        }
    }

    /// Wait until the next control message can be sent.
    pub async fn acquire(&mut self) {
        if let Some(last) = self.last_send {
            let elapsed = last.elapsed();
            if elapsed < self.min_interval {
                tokio::time::sleep(self.min_interval - elapsed).await;
            }
        }
        self.last_send = Some(std::time::Instant::now());
    }
}
```

This is NOT used in the current connect flow (we don't dynamically subscribe/unsubscribe -- symbols are in the URL). It's available for future use when dynamic subscription management is added. Add a test for it.

**6. Update tests:**

- Add test `test_parse_ban_duration` with input `"Too many requests; IP banned until 1700000000000."` asserting result is `Some(1700000000000)`.
- Add test `test_parse_ban_duration_no_ban` with input `"Connection closed normally"` asserting result is `None`.
- Add test `test_ws_control_rate_limiter_spacing` creating a `WsControlRateLimiter::new(5, 2)` and verifying `min_interval` is `333ms` (1000 / 3).
- Update `test_proactive_reconnect_constant` to still pass.

**7. Update `connect_and_demux` and `run_with_reconnect` call sites:**

In `run_with_reconnect`, compute `ping_interval` and `pong_timeout` from `ReconnectionPolicy` and pass to `connect_and_demux`. Update both the normal call and the proactive reconnection spawn call.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-providers --no-fail-fast 2>&1 | tail -20</automated>
</verify>
<acceptance_criteria> - grep -q "ping_ticker" crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q "last_ping_sent" crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q "pong_timeout\|PongTimeout" crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q "parse_ban_duration" crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q "WsControlRateLimiter" crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q 'signal = "CONNECT"' crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q 'signal = "FIRST_DATA"' crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q 'signal = "DISCONNECT"' crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q "BanDetected" crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q "test_parse_ban_duration" crates/opendeviationbar-providers/src/binance/combined_ws.rs - grep -q "test_ws_control_rate_limiter" crates/opendeviationbar-providers/src/binance/combined_ws.rs - cargo nextest run -p opendeviationbar-providers exits 0 - cargo build --workspace exits 0 (no compile errors in downstream crates)
</acceptance_criteria>
<done>Combined WS sends client-side pings every 5s with real 10s pong deadline (PongTimeout on no response), emits CONNECT/FIRST_DATA/DISCONNECT tracing signals, parses ban duration from Binance close frames and sleeps until expiry, has WsControlRateLimiter for subscribe pacing, all tests pass, workspace builds clean.</done>
</task>

</tasks>

<verification>
1. `cargo nextest run -p opendeviationbar-providers` — all provider tests pass
2. `cargo build --workspace` — no downstream compilation errors from new types
3. `cargo nextest run --workspace --exclude opendeviationbar-py` — full workspace regression
4. New error variants are correctly classified: RateLimited (terminal), BanDetected (terminal), PongTimeout (recoverable)
5. Keepalive defaults: ping_interval_secs=5, pong_timeout_secs=10
6. Subscribe rate limiter: 5 total - 2 ping reserve = 3 effective msg/sec
7. Pong deadline: if Ping sent and no Pong within 10s, PongTimeout returned (not waiting 90s data_timeout)
</verification>

<success_criteria>

- All Rust tests pass (`cargo nextest run --workspace --exclude opendeviationbar-py`)
- WebSocketError has 3 new variants with correct terminality
- ReconnectionPolicy has 4 new keepalive/rate-limit fields
- StreamSignal enum is exported and usable by downstream crates
- Client-side ping fires every 5s in both combined and per-symbol WS
- Pong deadline of 10s is tracked per-ping — PongTimeout fires if no Pong received within 10s (not relying on 90s data_timeout)
- Ban duration parser extracts timestamps from Binance error messages
- WsControlRateLimiter enforces 3 effective subscribe msgs/sec
  </success_criteria>

<output>
After completion, create `.planning/phases/04-ws-resilience-enhancements/04-01-SUMMARY.md`
</output>
