---
plan: 04-01
title: "Parallel fill_from_rest"
wave: 1
depends_on: []
files_modified:
  - crates/opendeviationbar-streaming/src/live_engine.rs
requirements:
  - PARA-01
  - PARA-02
  - PARA-04
autonomous: true
---

# Plan 04-01: Parallel fill_from_rest

## Task 1: Increase WS channel capacity from 1000 to 50000

<read_first>

- `crates/opendeviationbar-streaming/src/live_engine.rs` line 319: `mpsc::channel::<AggTrade>(1000)`
- `crates/opendeviationbar-streaming/src/live_engine.rs` line 1255: second `mpsc::channel::<AggTrade>(1000)`
  </read_first>

<action>
Change both `mpsc::channel::<AggTrade>(1000)` calls in `start_ws()` and `symbol_task()` (line 319 and line 1255) to `mpsc::channel::<AggTrade>(50_000)`. This prevents WS buffer overflow during the longer parallel fill window when 18 symbols buffer concurrently.
</action>

<acceptance_criteria>

- `grep -c "mpsc::channel::<AggTrade>(50_000)" crates/opendeviationbar-streaming/src/live_engine.rs` returns 2
- `grep -c "mpsc::channel::<AggTrade>(1000)" crates/opendeviationbar-streaming/src/live_engine.rs` returns 0
- `cargo check -p opendeviationbar-streaming` succeeds
  </acceptance_criteria>

---

## Task 2: Convert sequential fill_from_rest to JoinSet + Semaphore(4)

<read_first>

- `crates/opendeviationbar-streaming/src/live_engine.rs` lines 353-634: full `fill_from_rest()` method
- `crates/opendeviationbar-providers/src/binance/rate_limiter.rs` line 272: `set_ceiling()` signature
  </read_first>

<action>
Refactor `fill_from_rest()` to:

1. Call `rate_limiter.set_ceiling(4800)` before the fill loop
2. Replace the sequential `for symbol in &self.config.symbols` loop with a `JoinSet` + `Semaphore::new(4)` pattern:
   - Move per-symbol processor creation and REST pagination into a spawned async task
   - Clone shared state (`config`, `rate_limiter`, `bar_buffer`, `metrics`) into each task via `Arc`
   - Each task acquires a semaphore permit before starting REST fetch
   - Each task returns `(symbol, processors, bars_count, trades_count)` for aggregation
3. Collect results from `join_set.join_next()` and merge into `all_processors` HashMap
4. Call `rate_limiter.set_ceiling(3000)` after all fills complete
5. Store `all_processors` in `self.pre_created_processors` as before

Key ownership considerations:

- `self.config.initial_checkpoints` needs to be extracted before spawning (it uses `remove()`)
- `self.bar_buffer` is `ConcurrentRingBuffer` which has `clone_ref()` for sharing
- `self.metrics` is already `Arc<LiveEngineMetrics>`
- Processors are created and consumed within each task, returned as the task result
  </action>

<acceptance_criteria>

- `grep "JoinSet" crates/opendeviationbar-streaming/src/live_engine.rs` matches at least 1 line
- `grep "Semaphore::new(4)" crates/opendeviationbar-streaming/src/live_engine.rs` matches 1 line
- `grep "set_ceiling(4800)" crates/opendeviationbar-streaming/src/live_engine.rs` matches 1 line
- `grep "set_ceiling(3000)" crates/opendeviationbar-streaming/src/live_engine.rs` matches 1 line
- `cargo check -p opendeviationbar-streaming` succeeds
  </acceptance_criteria>

---

## Task 3: Add timing log for total fill duration

<read_first>

- `crates/opendeviationbar-streaming/src/live_engine.rs` lines 627-633: current completion log
  </read_first>

<action>
Add `std::time::Instant::now()` at the start of `fill_from_rest()` and emit a `tracing::info!` at the end with `total_duration_ms`, `total_bars`, `symbols` count. This provides the PARA-02 measurement point for verifying <90s fill time in production.
</action>

<acceptance_criteria>

- `grep "total_duration_ms" crates/opendeviationbar-streaming/src/live_engine.rs` matches at least 1 line
- `cargo check -p opendeviationbar-streaming` succeeds
  </acceptance_criteria>
