# Roadmap: REST→WS Junction Fix + Full-Symbol Streaming

**Created:** 2026-03-19
**Phases:** 10
**Requirements:** 30/30 mapped
**Granularity:** Fine (config.json)

---

## Phase 1: Junction Fix (Root Cause)

**Goal:** Eliminate forming-bar abandonment at the REST→WS transition.

**Requirements:**

- JUNC-01: Preserve forming bar across REST→WS transition
- JUNC-02: Single-processor continuity invariant
- JUNC-03: Re-seed gap detector past junction before main WS loop
- JUNC-04: Monotonicity guard for overlapping WS buffer trades

**Location:** `crates/opendeviationbar-streaming/src/live_engine.rs`, `symbol_task()`, between processor reuse and main `loop { select! {} }`. ~15 lines.

**Success Criteria:**

1. Unit test: synthetic REST→WS transition with known trade-ID gap produces zero Stathera gaps
2. Puck does NOT fire on the expected junction gap (verified via tracing output)
3. Forming bar trade count is preserved (>0 trades carried into WS phase)
4. WS buffer trades with trade-IDs <= processor's last_agg_trade_id are silently dropped (monotonicity)

**Parallelism:** Can be done in same session as Phases 2-3.

---

## Phase 2: Junction Telemetry + Audit Items

**Goal:** Instrument the junction to validate Phase 1 and provide tuning data for later phases.

**Requirements:**

- TELE-01: Log junction gap size (trade-ID delta) per symbol at startup
- TELE-02: Log forming bar trade count at junction transition per symbol
- TELE-03: Log WS buffer depth per symbol when engine starts
- TELE-04: Log Puck suppressed-at-junction vs fired-during-streaming events
- TELE-05: Log per-symbol fill_from_rest duration, trade count, bars produced
- AUDT-01: Verify ring buffer capacity (10K) sufficient for 18 symbols
- AUDT-02: Per-symbol REST fill timing with per-bar-per-threshold breakdown
- AUDT-03: WS buffer depth logging per symbol (channel pending count)
- AUDT-04: Puck junction vs normal-streaming event differentiation
- AUDT-05: Forming bar preservation metric (trade count logged if >0)

**Location:** Same `symbol_task()` area + new tracing spans. ~20 lines for telemetry, audit items overlap with TELE-\* instrumentation.

**Success Criteria:**

1. Journal output contains structured fields: `junction_gap_tid_delta`, `forming_bar_trades`, `ws_buffer_depth`, `puck_event_type` for every symbol at startup
2. `fill_from_rest` emits per-symbol summary: `{symbol, duration_ms, trade_count, bars_produced}`
3. Ring buffer capacity analysis documented (10K vs worst-case 18-symbol fill)
4. Puck events carry `event_type: "junction_suppressed"` vs `event_type: "normal_gap_fill"` field

**Parallelism:** Depends on Phase 1. Can be done in same session as Phase 1.

---

## Phase 3: Rate Limiter Dynamic Ceiling

**Goal:** Enable dynamic rate limit adjustment for parallel fill without new crates.

**Requirements:**

- PARA-03: Dynamic ceiling — 4800 weight/min during fill, 3000 after start

**Location:** `AdaptiveRateLimiter` — `ceiling: u32` → `ceiling: AtomicU32`, add `set_ceiling()` method. ~10 lines.

**Success Criteria:**

1. `set_ceiling(4800)` before fill_from_rest, `set_ceiling(3000)` after, verified in unit test
2. Existing `X-MBX-USED-WEIGHT-1m` server feedback integration unchanged
3. `ceiling` is `AtomicU32` — safe for concurrent access from JoinSet tasks

**Parallelism:** No dependencies. Can be done in same session as Phases 1-2.

---

## Phase 4: Parallel fill_from_rest

**Goal:** Reduce startup time from ~5 min to <90s, prevent WS buffer overflow.

**Requirements:**

- PARA-01: fill_from_rest processes all 18 symbols concurrently (Semaphore-bounded)
- PARA-02: Total fill time <90s with 4 concurrent workers
- PARA-04: WS channel capacity increased from 1,000 to 50,000

**Location:** `fill_from_rest()` in `live_engine.rs` — replace sequential loop with `JoinSet` + `Semaphore(4)`. Channel capacity constant change. ~40 lines.

**Success Criteria:**

1. 18 symbols fill concurrently with max 4 active at any time (Semaphore(4))
2. Total fill_from_rest wall time <90s (measured via TELE-05 from Phase 2)
3. WS channel capacity = 50,000 (verified in code, logged at startup)
4. No channel overflow errors in 10+ restart cycles
5. Rate limiter never exceeds 4800 weight/min during fill (verified via `X-MBX-USED-WEIGHT-1m`)

**Parallelism:** Depends on Phase 3 (dynamic ceiling).

---

## Phase 5: Combined WS Spike Test

**Goal:** Data-driven selection of optimal combined stream configuration.

**Requirements:**

- COMB-01: Spike-test 1x18, 2x9 even, and 2x9 volume-tiered — measure throughput, reconnect latency, gap count

**Location:** Standalone test script (can be a Rust binary in `crates/opendeviationbar-streaming/examples/`).

**Success Criteria:**

1. Each configuration tested for minimum 30 minutes with real Binance data
2. Metrics collected: messages/sec, max inter-message latency per symbol, reconnect time, gap count
3. Written comparison table with recommendation and rationale
4. Winner selected and documented in PROJECT.md key decisions

**Parallelism:** Independent of Phases 1-4. Can run in parallel.

---

## Phase 6: Combined WS Implementation

**Goal:** Implement winning combined stream configuration, eliminate per-symbol WS connections.

**Requirements:**

- COMB-02: `CombinedWebSocketStream` struct with per-symbol demux
- COMB-03: Combined stream envelope parsing (`{"stream":"...","data":{...}}`)
- COMB-04: All 18 symbols streamed via combined connections
- COMB-06: `symbol_task` interface unchanged — combined streams output to same per-symbol channels

**Location:** New `combined_ws.rs` in `crates/opendeviationbar-streaming/src/`. ~250 lines.

**Success Criteria:**

1. `CombinedWebSocketStream` connects to `wss://stream.binance.com/stream?streams=...` with correct URL encoding
2. Envelope `{"stream":"btcusdt@aggTrade","data":{...}}` correctly parsed and routed to per-symbol `mpsc::Sender<AggTrade>`
3. All 18 symbols receive trades via combined connections (zero per-symbol WS connections)
4. `symbol_task()` consumes from the same `mpsc::Receiver<AggTrade>` channel — no interface change
5. Existing Antaeus reconnection logic works with combined streams

**Parallelism:** Depends on Phase 5 results (configuration selection).

---

## Phase 7: Overlapping Reconnection

**Goal:** Eliminate the 24h forced-disconnect gap via overlapping connection pattern.

**Requirements:**

- COMB-05: Overlapping reconnection at 23h — open new, stabilize, close old. Stagger 2 connections by 30 min.

**Success Criteria:**

1. At 23h mark, new connection opens and receives trades while old connection still active
2. Trade-ID dedup at processor level handles the overlap window (zero duplicates emitted)
3. Two connections staggered by 30 min — never reconnect simultaneously
4. Zero Stathera gaps across the reconnection window (verified in test)

**Parallelism:** Depends on Phase 6 (combined streams must exist).

---

## Phase 8: Integration Testing

**Goal:** Validate all components working together before production deploy.

**Success Criteria:**

1. Full startup sequence tested: fill_from_rest (parallel, 4 workers) → junction fix → combined WS → Puck operational
2. Simulated 23h reconnection produces zero gaps across all 18 symbols
3. Rate limiter transitions: 4800 during fill → 3000 after start, verified end-to-end
4. Sidecar restart cycle (stop → start) produces zero intra-day Stathera gaps in local test
5. All telemetry fields (TELE-01 through TELE-05) present in journal output

**Parallelism:** Depends on Phases 1-7.

---

## Phase 9: Build + Deploy Prep

**Goal:** Produce deployable artifacts and prepare rollback safety net.

**Requirements:**

- DEPL-01: Linux x86_64 wheel via Zig cross-compile
- DEPL-02: Backup today's bars to `_backup` table before clean-slate delete
- DEPL-03: Clean-slate deploy playbook: Stop → Delete today → Clear checkpoints → Start
- DEPL-04: Deploy during UTC 00:00-02:00 low-volume window
- DEPL-05: Rollback SQL script prepared and tested

**Success Criteria:**

1. `mise run release:linux` produces wheel successfully
2. Rollback SQL tested on staging data: restores bars from `_backup` + restores checkpoints
3. Deploy playbook documented as executable checklist (not prose)
4. Backup table `open_deviation_bars_backup` created and verified on bigblack

**Parallelism:** Can start wheel build (DEPL-01) as soon as all code is merged. Rollback prep (DEPL-05) independent.

---

## Phase 10: Clean-Slate Deploy + Verification

**Goal:** Zero-tolerance production verification across all 18 symbols for 24h+.

**Requirements:**

- VERF-01: 0 intra-day Stathera gaps across all 18 symbols for 24h+
- VERF-02: 0 Stathera overlaps across all 18 symbols for 24h+
- VERF-03: All 18 symbols producing bars via combined WS streams
- VERF-04: Junction telemetry confirms forming bar preservation at every restart
- VERF-05: Rate limiter headroom maintained (never exceeds 5000 weight/min during normal operation)

**Success Criteria:**

1. ClickHouse query: `SELECT count(*) FROM ... WHERE tid_delta > 0 AND NOT is_day_boundary` returns 0 for all 18 symbols over 24h window
2. ClickHouse query: `SELECT count(*) FROM ... WHERE tid_delta < 0` returns 0 (no overlaps)
3. Journal logs show all 18 symbols with `source: "combined_ws"` (not per-symbol connections)
4. At least 1 sidecar restart during 24h window with junction telemetry confirming forming bar preservation
5. `X-MBX-USED-WEIGHT-1m` peak during normal operation < 5000

---

## Dependency Graph

```
Phase 1 (Junction Fix) ──────┐
Phase 2 (Telemetry) ─────────┤
Phase 3 (Rate Limiter) ──┐   ├── Phase 8 (Integration) ── Phase 9 (Build) ── Phase 10 (Deploy)
Phase 4 (Parallel Fill) ←┘   │
Phase 5 (WS Spike) ──────────┤
Phase 6 (Combined WS) ←──────┘
Phase 7 (Overlapping Reconnect) ← Phase 6
```

**Fast path:** Phases 1+2+3 in one session (~45 lines). Phase 5 in parallel. Phase 4 immediately after 3.

---

## Coverage Verification

| Category      | Requirements                                         | Count                                   | Phases  |
| ------------- | ---------------------------------------------------- | --------------------------------------- | ------- |
| Junction Fix  | JUNC-01, JUNC-02, JUNC-03, JUNC-04                   | 4                                       | 1/1 | Complete   | 2026-03-19 | TELE-01, TELE-02, TELE-03, TELE-04, TELE-05          | 5                                       | 2       |
| Audit Items   | AUDT-01, AUDT-02, AUDT-03, AUDT-04, AUDT-05          | 5                                       | 2       |
| Parallel Fill | PARA-01, PARA-02, PARA-03, PARA-04                   | 4                                       | 3, 4    |
| Combined WS   | COMB-01, COMB-02, COMB-03, COMB-04, COMB-05, COMB-06 | 6                                       | 5, 6, 7 |
| Deployment    | DEPL-01, DEPL-02, DEPL-03, DEPL-04, DEPL-05          | 5                                       | 9       |
| Verification  | VERF-01, VERF-02, VERF-03, VERF-04, VERF-05          | 5                                       | 10      |
| **Total**     |                                                      | **34** (30 v1 + 4 implicit integration) | 1-10    |

**v1 requirements mapped: 30/30. Zero unmapped.**

---

_Roadmap created: 2026-03-19_
