# Phase 4: Combined WebSocket Streams - Research

**Researched:** 2026-03-24
**Domain:** Binance WebSocket streaming infrastructure, Rust async networking, sidecar architecture
**Confidence:** HIGH

## Summary

Phase 4 migrates from per-symbol WebSocket connections to Binance combined streams with a 2x9 hybrid architecture for all 26 enabled symbols. The critical discovery is that **combined WS is already fully implemented in Rust** (`combined_ws.rs`, ~730 LOC) and **already wired into the live engine's `start_ws()` method**. The current code spawns a single `CombinedWebSocketStream::run_with_reconnect()` with all symbols on one connection.

The remaining work is: (1) add a feature flag for rollback to per-symbol mode (D-01), (2) split into 2 connections (2x13 given 26 symbols, not 2x9 as originally scoped for 18 symbols), (3) scale ring buffer and validate capacity for 26 symbols x 4 thresholds = 104 processors, and (4) A/B validation with Stathera continuity checks (D-03). The Rust infrastructure (combined envelope parsing, proactive 23h reconnection, Antaeus backoff, ban handling) is production-ready with 19 unit tests.

**Primary recommendation:** Add a `WS_MODE` env var toggling between `combined` (default, current behavior) and `per_symbol` (rollback path). Split the single combined connection into 2 connections by liquidity grouping. Validate with 24h+ Stathera continuity A/B comparison.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

- D-01: Rollback-first design — feature flag to switch between per-symbol and combined modes without code changes
- D-02: Follow-up phase for monitoring integration (monit, Telegram, heartbeat) -- OUT OF SCOPE for Phase 4
- D-03: Thorough validation with A/B comparison (per-symbol vs combined Stathera continuity, ring buffer overflow, reconnection behavior)

### Claude's Discretion

- Exact stream grouping strategy for the 2x9 split (by liquidity, alphabetical, etc.)
- Whether to use Rust-side or Python-side combined stream management
- Antaeus backoff parameters (base delay, max delay, jitter range)
- Feature flag mechanism (env var name, default behavior)

### Deferred Ideas (OUT OF SCOPE)

- Phase 4.1: Combined WS Monitoring Integration -- monit, Telegram, heartbeat checks for combined stream architecture
  </user_constraints>

<phase_requirements>

## Phase Requirements

| ID    | Description                                                                                          | Research Support                                                                                                                                                                       |
| ----- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| WS-01 | Sidecar uses Binance combined streams with 2x9 hybrid architecture instead of per-symbol connections | Combined WS already implemented in `combined_ws.rs`. Need to split single connection into 2 connections and add feature flag for rollback.                                             |
| WS-02 | Combined stream reconnection uses Antaeus exponential backoff with jitter                            | Already implemented in `CombinedWebSocketStream::run_with_reconnect()` with `backon` crate, stable connection reset, ban handling, and proactive 23h reconnection. No new work needed. |
| WS-03 | All 26 symbols maintain Stathera continuity after migration to combined streams                      | Requires validation: 24h+ production run with Stathera gap checks. A/B comparison per D-03.                                                                                            |

</phase_requirements>

## Standard Stack

### Core (Already in Workspace)

| Library             | Version   | Purpose                          | Why Standard                                        |
| ------------------- | --------- | -------------------------------- | --------------------------------------------------- |
| `tokio-tungstenite` | workspace | WebSocket client (async)         | Already used for per-symbol and combined WS         |
| `backon`            | workspace | Exponential backoff with jitter  | Already used for Antaeus reconnection               |
| `tokio`             | workspace | Async runtime, channels, timers  | Already used throughout streaming crate             |
| `serde_json`        | workspace | Combined envelope parsing        | Already used for `CombinedEnvelope` deserialization |
| `tokio_util`        | workspace | `CancellationToken` for shutdown | Already used for graceful shutdown                  |

### Supporting

| Library             | Version  | Purpose                         | When to Use                                       |
| ------------------- | -------- | ------------------------------- | ------------------------------------------------- |
| `pydantic-settings` | existing | `SidecarConfig` env var parsing | Feature flag env var (`OPENDEVIATIONBAR_WS_MODE`) |

**No new dependencies required.** All Rust and Python libraries are already in the workspace.

## Architecture Patterns

### Current State (Already Implemented)

```
crates/opendeviationbar-providers/src/binance/
├── websocket.rs          # Per-symbol WS (BinanceWebSocketStream) -- FALLBACK
├── combined_ws.rs        # Combined WS (CombinedWebSocketStream) -- CURRENT
└── mod.rs                # Exports both

crates/opendeviationbar-streaming/src/live_engine/
├── mod.rs                # start_ws() spawns CombinedWebSocketStream
├── symbol_task.rs        # Per-symbol processing (unchanged by WS mode)
└── ...
```

### What start_ws() Currently Does (mod.rs:179-211)

```rust
// Creates per-symbol mpsc channels (50K capacity each)
// Spawns ONE CombinedWebSocketStream::run_with_reconnect()
// with ALL symbols on a SINGLE connection
tokio::spawn(async move {
    CombinedWebSocketStream::run_with_reconnect(
        symbols, senders, policy, ws_shutdown,
    ).await;
});
```

### Target Architecture: 2-Connection Split

```
start_ws()
  ├── Connection 1: CombinedWebSocketStream (13 symbols, high-liquidity)
  └── Connection 2: CombinedWebSocketStream (13 symbols, lower-liquidity)

Each connection:
  - Independent CancellationToken (child of main shutdown)
  - Independent Antaeus backoff state
  - Independent 23h proactive reconnection timer
  - Demuxes to shared per-symbol mpsc::Sender<AggTrade> channels
```

### Feature Flag Architecture

```
OPENDEVIATIONBAR_WS_MODE=combined   (default, new behavior)
OPENDEVIATIONBAR_WS_MODE=per_symbol (rollback to legacy)

                 ┌──────────────────────────────┐
                 │  start_ws() in mod.rs         │
                 │  reads ws_mode from config     │
                 ├──────────┬───────────────────┤
                 │ combined │ per_symbol         │
                 ├──────────┼───────────────────┤
                 │ 2 x      │ N x               │
                 │ Combined │ BinanceWebSocket   │
                 │ WS tasks │ Stream tasks       │
                 └──────────┴───────────────────┘
                 Both modes produce identical per-symbol
                 mpsc::Receiver<AggTrade> channels
```

**Key insight:** `symbol_task()` is mode-agnostic. It receives `pre_buffered_rx: mpsc::Receiver<AggTrade>` regardless of whether the trade came from a combined or per-symbol WS connection. The feature flag only affects `start_ws()`.

### Recommended Connection Grouping (2x13)

With 26 enabled symbols (not 18 as in the original spike), use 2x13 split by volume:

**Connection 1 (13 high-volume):** BTCUSDT, ETHUSDT, SOLUSDT, XRPUSDT, BNBUSDT, DOGEUSDT, ADAUSDT, AVAXUSDT, LINKUSDT, TRXUSDT, SUIUSDT, LTCUSDT, NEARUSDT

**Connection 2 (13 lower-volume):** UNIUSDT, BCHUSDT, FILUSDT, WLDUSDT, WIFUSDT, AAVEUSDT, XLMUSDT, DOTUSDT, PAXGUSDT, MATICUSDT, ATOMUSDT, SHIBUSDT, APTUSDT

**Rationale:** Distributing high-volume symbols across connections balances load. If one connection drops, only 13 symbols briefly need Puck gap-fill, not all 26.

**Note:** The exact grouping should be configurable (env var or registry metadata) so it can be tuned without code changes.

### Anti-Patterns to Avoid

- **1x26 single connection:** No fault isolation -- one drop affects all 26 symbols simultaneously
- **26 individual connections:** Causes recv timeouts on low-volume symbols (proven empirically), rate limit risk (300 connections/5min/IP)
- **Removing per-symbol WS code:** Must keep for rollback (D-01)
- **Changing symbol_task interface:** It's mode-agnostic by design; don't couple it to WS mode

## Don't Hand-Roll

| Problem                   | Don't Build                  | Use Instead                                                                 | Why                                           |
| ------------------------- | ---------------------------- | --------------------------------------------------------------------------- | --------------------------------------------- |
| Exponential backoff       | Custom retry loop            | `backon` crate (already used)                                               | Jitter, factor, max delay all handled         |
| Combined envelope parsing | Manual JSON field extraction | `serde::Deserialize` for `CombinedEnvelope` (already implemented)           | Type-safe, zero-copy price/qty via FixedPoint |
| 24h connection lifetime   | Manual timer                 | `PROACTIVE_RECONNECT_SECS` + overlapping reconnection (already implemented) | Overlap ensures zero-gap handoff              |
| Feature flag parsing      | Custom env parsing           | `pydantic-settings` SidecarConfig (already used)                            | Validation, defaults, type coercion           |

**Key insight:** Almost all infrastructure already exists. Phase 4 is primarily a configuration and validation phase, not a greenfield implementation.

## Common Pitfalls

### Pitfall 1: Ring Buffer Capacity

**What goes wrong:** With 26 symbols x 4 thresholds = 104 processors, the ring buffer may overflow if capacity is sized for 2 symbols.
**Why it happens:** Ring buffer capacity is set by `OPENDEVIATIONBAR_MAX_PENDING_BARS` env var (default 10,000). With 13x more symbols producing bars, the buffer fills faster.
**How to avoid:** Verify current capacity is sufficient. 26 symbols with ~50 bars/flush cycle x 4 thresholds = ~5,200 bars/cycle, well within 10K default. Monitor Lethe overflow detection in production.
**Warning signs:** Lethe alerts in sidecar logs indicating ring buffer drops.

### Pitfall 2: Puck REST Rate Budget Competition

**What goes wrong:** 26 symbols competing for REST gap-fill via Puck could exhaust the 3000 weight/min budget.
**Why it happens:** Each Puck gap-fill costs ~20 API weight per chunk. With 26 symbols potentially needing gap-fill simultaneously (e.g., after a connection drop), budget burns fast.
**How to avoid:** The `AdaptiveRateLimiter` already paces requests. During `fill_from_rest`, ceiling is raised to 4800 (mod.rs:236). The risk is mainly at reconnection when all 13 symbols on a dropped connection need simultaneous gap-fill.
**Warning signs:** Rate limit 429 errors in sidecar logs, Puck fill taking >10s.

### Pitfall 3: Feature Flag Default Direction

**What goes wrong:** Setting combined as default before validation risks production regression.
**Why it happens:** Combined mode is already the default (start_ws uses CombinedWebSocketStream). Adding per-symbol fallback is the new code path.
**How to avoid:** The feature flag default should be `combined` (matching current production behavior). The `per_symbol` mode is the NEW code path that needs to be added/restored. This is the opposite of what the CONTEXT.md suggests ("default to per-symbol until validated") -- combined is already deployed.
**Warning signs:** Unexpected behavior after restart if flag defaults are wrong.

### Pitfall 4: 24h Connection Lifetime Overlap

**What goes wrong:** Proactive reconnection at 23h creates a brief overlap window where both old and new connections send trades.
**Why it happens:** By design (COMB-05 in combined_ws.rs). The processor's trade-ID monotonicity guard drops duplicates.
**How to avoid:** Already handled. The `first_trade_signal` oneshot confirms the new connection is live before cancelling the old one. No action needed.
**Warning signs:** None expected -- this is working correctly.

### Pitfall 5: Per-Symbol Fallback Requires WS Code Resurrection

**What goes wrong:** The per-symbol WS fallback path in `start_ws()` was removed. `symbol_task` requires `pre_buffered_rx` and errors if it's None.
**Why it happens:** `symbol_task.rs:60-62` shows: the per-symbol fallback was explicitly removed ("Per-symbol fallback removed -- all symbols use combined WS (COMB-04)").
**How to avoid:** The feature flag for per-symbol mode must re-add the per-symbol WS spawning in `start_ws()`. This is the main new code needed.
**Warning signs:** Symbol tasks exiting immediately with "combined stream requires pre-buffered receiver" error.

## Code Examples

### Existing: CombinedWebSocketStream (combined_ws.rs)

```rust
// Source: crates/opendeviationbar-providers/src/binance/combined_ws.rs
// Already fully implemented with:
// - CombinedEnvelope parsing
// - Per-symbol demux via HashMap<String, mpsc::Sender<AggTrade>>
// - Antaeus backoff with stable connection reset
// - Proactive 23h reconnection with overlapping handoff
// - Ban detection and sleep-until-expiry
// - Client-side ping/pong with deadline tracking
```

### Existing: start_ws() Using Combined (live_engine/mod.rs:179-211)

```rust
// Source: crates/opendeviationbar-streaming/src/live_engine/mod.rs
pub fn start_ws(&mut self) {
    // Creates per-symbol channels
    let mut senders = HashMap::new();
    for symbol in &self.config.symbols {
        let (trade_tx, trade_rx) = mpsc::channel::<AggTrade>(50_000);
        senders.insert(symbol.clone(), trade_tx);
        receivers.insert(symbol, trade_rx);
    }
    // Single combined WS connection for all symbols
    tokio::spawn(async move {
        CombinedWebSocketStream::run_with_reconnect(
            symbols, senders, policy, ws_shutdown,
        ).await;
    });
}
```

### Needed: 2-Connection Split in start_ws()

```rust
// PSEUDOCODE -- the key change needed
pub fn start_ws(&mut self) {
    match self.config.ws_mode {
        WsMode::Combined => {
            let (group_a, group_b) = split_symbols(&symbols);
            // Connection 1
            let senders_a = senders filtered to group_a;
            tokio::spawn(CombinedWebSocketStream::run_with_reconnect(
                group_a, senders_a, policy.clone(), shutdown.child_token(),
            ));
            // Connection 2
            let senders_b = senders filtered to group_b;
            tokio::spawn(CombinedWebSocketStream::run_with_reconnect(
                group_b, senders_b, policy.clone(), shutdown.child_token(),
            ));
        }
        WsMode::PerSymbol => {
            for symbol in &symbols {
                let tx = senders.get(&symbol).unwrap().clone();
                tokio::spawn(BinanceWebSocketStream::run_with_reconnect(
                    symbol.clone(), tx, policy.clone(), shutdown.child_token(),
                ));
            }
        }
    }
}
```

### Needed: Feature Flag in SidecarConfig

```python
# python/opendeviationbar/config/sidecar.py
class SidecarConfig(BaseSettings):
    # ... existing fields ...
    ws_mode: Literal["combined", "per_symbol"] = "combined"

    model_config = SettingsConfigDict(
        env_prefix="OPENDEVIATIONBAR_",
    )
```

## State of the Art

| Old Approach                       | Current Approach                        | When Changed                   | Impact                            |
| ---------------------------------- | --------------------------------------- | ------------------------------ | --------------------------------- |
| Per-symbol WS (18 connections)     | Combined WS (1 connection, all symbols) | Already shipped (current code) | 4x throughput, zero recv timeouts |
| 18 symbols                         | 26 symbols (symbols.toml enabled=true)  | Registry grew                  | 2x13 split needed (not 2x9)       |
| Per-symbol fallback in symbol_task | Removed (COMB-04)                       | Current code                   | Must re-add for D-01 rollback     |

**Key finding:** The project is further ahead than the spike document suggests. Combined WS is already the production code path. The spike's "Phase 2: Implement combined WS in Rust" is DONE. What remains is the split into 2 connections and the rollback feature flag.

## Open Questions

1. **Connection grouping source of truth**
   - What we know: 26 symbols need to be split into 2 groups by volume
   - What's unclear: Should grouping be hardcoded, derived from symbols.toml metadata, or configurable via env var?
   - Recommendation: Add a `ws_connection_groups` field to LiveEngineConfig with a sensible default split. Configurable via `OPENDEVIATIONBAR_WS_GROUPS` env var (comma-separated group 1, pipe separator, group 2).

2. **Ring buffer sizing with 26 symbols**
   - What we know: Current default is 10,000. 26 symbols x 4 thresholds x ~50 bars/flush = ~5,200 bars/cycle.
   - What's unclear: Whether burst scenarios (26-symbol simultaneous gap-fill after reconnection) could exceed 10K.
   - Recommendation: Keep 10K default but add a validation test. If Lethe overflow triggers in production, increase to 20K.

3. **Actual symbol count on bigblack**
   - What we know: symbols.toml has 26 enabled=true symbols. sidecar.env does NOT override STREAMING_SYMBOLS (defaults to all enabled).
   - What's unclear: Whether bigblack actually streams all 26 or only a subset.
   - Recommendation: Verify with `ssh bigblack 'systemctl --user show opendeviationbar-sidecar | grep STREAMING'` during implementation.

## Validation Architecture

### Test Framework

| Property           | Value                                                                |
| ------------------ | -------------------------------------------------------------------- |
| Framework          | Rust: `cargo nextest` / Python: `pytest`                             |
| Config file        | `[tool.pytest.ini_options]` in pyproject.toml / `.cargo/config.toml` |
| Quick run command  | `cargo nextest run -p opendeviationbar-providers --test combined_ws` |
| Full suite command | `mise run check-full`                                                |

### Phase Requirements -> Test Map

| Req ID | Behavior                                 | Test Type | Automated Command                                                           | File Exists?                                                                     |
| ------ | ---------------------------------------- | --------- | --------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| WS-01  | Combined streams with 2-connection split | unit      | `cargo nextest run -p opendeviationbar-providers -E 'test(combined_ws)' -x` | Partially (19 tests exist for combined_ws, need split logic tests)               |
| WS-01  | Feature flag toggles WS mode             | unit      | `cargo nextest run -p opendeviationbar-streaming -E 'test(ws_mode)' -x`     | No -- Wave 0                                                                     |
| WS-01  | Per-symbol fallback path works           | unit      | `cargo nextest run -p opendeviationbar-streaming -E 'test(per_symbol)' -x`  | No -- Wave 0                                                                     |
| WS-02  | Antaeus backoff with jitter              | unit      | `cargo nextest run -p opendeviationbar-providers -E 'test(backoff)' -x`     | Yes (test_antaeus_backoff_reset, test_backoff_iterator_produces_delays)          |
| WS-02  | Proactive 23h reconnection               | unit      | `cargo nextest run -p opendeviationbar-providers -E 'test(proactive)' -x`   | Yes (test_proactive_reconnect_constant, test_proactive_reconnect_oneshot_signal) |
| WS-03  | Stathera continuity after migration      | manual    | ClickHouse query on bigblack after 24h+ run                                 | No -- production validation                                                      |

### Sampling Rate

- **Per task commit:** `cargo nextest run -p opendeviationbar-providers -p opendeviationbar-streaming -x`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green + 24h+ Stathera continuity check on bigblack

### Wave 0 Gaps

- [ ] `WsMode` enum and config plumbing tests in `live_engine/mod.rs`
- [ ] Per-symbol fallback spawn test in `start_ws()`
- [ ] 2-connection split logic test (symbol grouping)
- [ ] Python `SidecarConfig.ws_mode` field test

## Sources

### Primary (HIGH confidence)

- `crates/opendeviationbar-providers/src/binance/combined_ws.rs` -- Full combined WS implementation (730 LOC, 19 tests)
- `crates/opendeviationbar-providers/src/binance/websocket.rs` -- Per-symbol WS implementation with Antaeus
- `crates/opendeviationbar-streaming/src/live_engine/mod.rs` -- `start_ws()` already uses combined WS
- `crates/opendeviationbar-streaming/src/live_engine/symbol_task.rs` -- Mode-agnostic trade processing
- [Binance WebSocket Streams docs](https://developers.binance.com/docs/binance-spot-api-docs/web-socket-streams) -- 1024 streams/conn, 300 conn/5min, 24h lifetime
- `.claude/skills/full-symbol-streaming-spike/SKILL.md` -- Prior investigation (partially superseded by current code)

### Secondary (MEDIUM confidence)

- `python/opendeviationbar/data/symbols.toml` -- 26 enabled symbols (verified via grep count)
- `deploy/sidecar.env` -- No STREAMING_SYMBOLS override (defaults to all enabled)

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - All libraries already in workspace, no new dependencies
- Architecture: HIGH - Combined WS already implemented, only splitting and feature flag needed
- Pitfalls: HIGH - Based on actual codebase analysis, not hypothetical
- Validation: MEDIUM - Production validation (24h Stathera) requires deployment

**Research date:** 2026-03-24
**Valid until:** 2026-04-24 (stable -- Binance WS API has not changed since 2017)

## Project Constraints (from CLAUDE.md)

- **Python 3.13 ONLY** -- never use 3.14 or other versions
- **Leverage Rust** -- heavy lifting in Rust, Python handles I/O only
- **Local-first CI/CD** -- all quality gates run locally via `mise run check-full`
- **Spot-only data source** -- all WS connections to `stream.binance.com` (spot), never `fstream.binance.com` (futures)
- **ClickHouse COMMENT = SSoT** -- any new schema columns need COMMENTs
- **Writer ownership** -- sidecar owns today, kintsugi owns yesterday+older
- **Stathera continuity** -- trade-ID alignment invariant must be maintained
