---
phase: 03-rust-hot-path
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - crates/opendeviationbar-core/src/fixed_point.rs
  - crates/opendeviationbar-providers/src/binance/historical.rs
autonomous: true
requirements: [RUST-01, RUST-02]

must_haves:
  truths:
    - "CSV f64 price/volume converts to FixedPoint without heap allocation (no .to_string())"
    - "IntraDayChunkIterator does not re-parse already-consumed CSV records on each chunk"
  artifacts:
    - path: "crates/opendeviationbar-core/src/fixed_point.rs"
      provides: "FixedPoint::from_f64() method"
      contains: "pub fn from_f64"
    - path: "crates/opendeviationbar-providers/src/binance/historical.rs"
      provides: "Zero-alloc CSV conversion + O(N) chunk iteration"
  key_links:
    - from: "CsvAggTrade conversion (historical.rs)"
      to: "FixedPoint::from_f64 (fixed_point.rs)"
      via: "direct call replacing from_str(&.to_string())"
      pattern: "FixedPoint::from_f64"
    - from: "parse_chunk_from_zip (historical.rs)"
      to: "csv::Reader"
      via: "byte offset tracking instead of skip loop"
      pattern: "records_consumed|byte_offset"
---

<objective>
Eliminate two hot-path allocation sources in Vision CSV processing: the f64-to-String-to-FixedPoint double conversion (RUST-01) and the O(N^2) chunk re-parsing in IntraDayChunkIterator (RUST-02).

Purpose: Vision CSV is the primary backfill path — 20M+ trades/day for BTCUSDT. Each unnecessary allocation and re-scan compounds across billions of historical trades.
Output: Zero-allocation FixedPoint::from_f64(), O(N) chunk iteration via byte offset tracking.
</objective>

<execution_context>
@/Users/terryli/.claude/get-shit-done/workflows/execute-plan.md
@/Users/terryli/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md

@crates/opendeviationbar-core/src/fixed_point.rs
@crates/opendeviationbar-providers/src/binance/historical.rs

<interfaces>
<!-- Key types and contracts the executor needs -->

From crates/opendeviationbar-core/src/fixed_point.rs:

```rust
pub const SCALE: i64 = 100_000_000; // 8 decimal places
pub struct FixedPoint(pub i64);
impl FixedPoint {
    pub fn from_str(s: &str) -> Result<Self, FixedPointError>;
    pub fn to_f64(&self) -> f64 { self.0 as f64 / SCALE as f64 }
}
```

From crates/opendeviationbar-providers/src/binance/historical.rs:

```rust
pub struct CsvAggTrade(
    pub u64,  // agg_trade_id
    pub f64,  // price         <-- these two use f64→String→FixedPoint
    pub f64,  // quantity      <-- these two use f64→String→FixedPoint
    pub u64,  // first_trade_id
    pub u64,  // last_trade_id
    pub u64,  // timestamp
    pub bool, // is_buyer_maker
);

// Current wasteful conversion (lines 211-212, 245-246):
// price: FixedPoint::from_str(&csv_trade.1.to_string()).unwrap_or(FixedPoint(0)),

// IntraDayChunkIterator (line 1129 parse_chunk_from_zip):
// Re-creates ZipArchive + csv::Reader per chunk, skips `records_consumed` records
```

From crates/opendeviationbar-providers/src/serde_helpers.rs:

```rust
// Already exists for REST/WS JSON deserialization:
pub fn deserialize_fixedpoint_from_str<'de, D>(deserializer: D) -> Result<FixedPoint, D::Error>
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add FixedPoint::from_f64() and replace CSV double-conversion</name>
  <files>crates/opendeviationbar-core/src/fixed_point.rs, crates/opendeviationbar-providers/src/binance/historical.rs</files>
  <read_first>
    - crates/opendeviationbar-core/src/fixed_point.rs (full file — understand SCALE, from_str, existing tests)
    - crates/opendeviationbar-providers/src/binance/historical.rs lines 140-255 (CsvAggTrade struct + both conversion impls)
    - crates/opendeviationbar-providers/src/binance/historical.rs lines 1296-1460 (existing CsvAggTrade tests)
  </read_first>
  <behavior>
    - Test: FixedPoint::from_f64(50000.12345678).to_f64() round-trips within 1e-8 tolerance
    - Test: FixedPoint::from_f64(0.0) == FixedPoint(0)
    - Test: FixedPoint::from_f64(1.5).0 == 150_000_000
    - Test: FixedPoint::from_f64(-1.5).0 == -150_000_000
    - Oracle test: for each of ["50000.12345678", "0.01328000", "112070.01000000"], FixedPoint::from_f64(s.parse::<f64>().unwrap()) must produce bit-identical .0 values to FixedPoint::from_str(s) (matching the oracle pattern in serde_helpers.rs)
    - Test: CsvAggTrade conversion via From<CsvAggTrade> produces same AggTrade as before (regression)
    - Test: CsvAggTrade.to_agg_trade() produces same AggTrade as before (regression)
  </behavior>
  <action>
    1. In `crates/opendeviationbar-core/src/fixed_point.rs`, add `from_f64()` to the `impl FixedPoint` block:
       ```rust
       /// Convert f64 to FixedPoint without intermediate String allocation.
       /// Uses direct multiplication by SCALE, matching the precision of from_str()
       /// for values with <= 8 decimal places (all Binance prices/quantities).
       #[inline]
       pub fn from_f64(value: f64) -> Self {
           FixedPoint((value * SCALE as f64).round() as i64)
       }
       ```
       The `.round()` is critical — without it, `50000.12345678 * 1e8` could produce 5000012345677 due to IEEE 754.

    2. In `crates/opendeviationbar-providers/src/binance/historical.rs`, update `impl From<CsvAggTrade> for AggTrade` (lines 211-212):
       Replace:
       ```rust
       price: FixedPoint::from_str(&csv_trade.1.to_string()).unwrap_or(FixedPoint(0)),
       volume: FixedPoint::from_str(&csv_trade.2.to_string()).unwrap_or(FixedPoint(0)),
       ```
       With:
       ```rust
       price: FixedPoint::from_f64(csv_trade.1),
       volume: FixedPoint::from_f64(csv_trade.2),
       ```

    3. Same replacement in `CsvAggTrade::to_agg_trade()` (lines 245-246):
       Replace:
       ```rust
       price: FixedPoint::from_str(&self.1.to_string()).unwrap_or(FixedPoint(0)),
       volume: FixedPoint::from_str(&self.2.to_string()).unwrap_or(FixedPoint(0)),
       ```
       With:
       ```rust
       price: FixedPoint::from_f64(self.1),
       volume: FixedPoint::from_f64(self.2),
       ```

    4. Add tests in `fixed_point.rs` test module for `from_f64()` covering the behaviors above.

    5. Update existing test `test_csv_agg_trade_to_agg_trade` (line 1316) if the expected values need adjustment due to f64 rounding differences (unlikely for clean values like 50000.12345678).

    IMPORTANT: The oracle test is the correctness gate. Binance CSV values always have <= 8 decimal places, so `(f64 * 1e8).round() as i64` must match `from_str()` for all production values.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-core --lib -- fixed_point && cargo nextest run -p opendeviationbar-providers --lib -- csv_agg_trade</automated>
  </verify>
  <acceptance_criteria>
    - `grep -n "from_f64" crates/opendeviationbar-core/src/fixed_point.rs` shows the new method
    - `grep -n "to_string()" crates/opendeviationbar-providers/src/binance/historical.rs` returns NO matches in CsvAggTrade conversion blocks (lines 207-254)
    - `grep -n "from_f64" crates/opendeviationbar-providers/src/binance/historical.rs` shows 4 call sites (2 in From impl, 2 in to_agg_trade)
    - All tests pass: `cargo nextest run -p opendeviationbar-core --lib` and `cargo nextest run -p opendeviationbar-providers --lib`
  </acceptance_criteria>
  <done>FixedPoint::from_f64() exists with oracle-validated correctness. All 4 CsvAggTrade→FixedPoint conversions use from_f64() with zero String allocation. All existing tests pass.</done>
</task>

<task type="auto">
  <name>Task 2: Optimize IntraDayChunkIterator to O(N) via byte offset tracking</name>
  <files>crates/opendeviationbar-providers/src/binance/historical.rs</files>
  <read_first>
    - crates/opendeviationbar-providers/src/binance/historical.rs lines 1000-1200 (IntraDayChunkIterator struct, parse_chunk_from_zip, Iterator impl)
    - crates/opendeviationbar-providers/src/binance/historical.rs lines 1060-1105 (constructor with_checksum)
  </read_first>
  <action>
    The current `parse_chunk_from_zip()` re-opens the ZipArchive and re-creates a csv::Reader each call, then skips `records_consumed` records before reading the next chunk. For a 20M-trade day split into 4 chunks (500K each), this means: chunk 1 reads 500K, chunk 2 skips 500K + reads 500K, chunk 3 skips 1M + reads 500K, chunk 4 skips 1.5M + reads 500K. Total records touched: 500K + 1M + 1.5M + 2M = 5M, but only 2M are actually used. This is O(N * chunks) = O(N^2/chunk_size).

    **Fix**: Track the byte offset after each chunk's CSV parsing, then seek directly to that offset on the next call.

    1. Add a new field to `IntraDayChunkIterator`:
       ```rust
       /// Byte offset into the CSV file within the ZIP, for O(N) chunk resumption.
       /// After parsing a chunk, this stores the position where the csv::Reader stopped.
       /// Next chunk seeks directly here instead of re-scanning from the start.
       csv_byte_offset: u64,
       ```
       Initialize to `0` in both constructors (`new` via `with_checksum`, and `with_checksum` directly).

    2. Rewrite `parse_chunk_from_zip()` to use the byte offset:
       ```rust
       fn parse_chunk_from_zip(&mut self) -> Result<(Vec<AggTrade>, bool), HistoricalError> {
           let zip_bytes = self.day_zip_bytes.as_ref().expect("ZIP bytes must be cached");
           let cursor = Cursor::new(zip_bytes.as_slice());
           let mut archive = ZipArchive::new(cursor)?;
           let csv_file = archive.by_name(&self.day_csv_filename)?;

           // Seek past already-consumed bytes
           let buf_reader = BufReader::with_capacity(64 * 1024, csv_file);
           let mut reader = ReaderBuilder::new()
               .has_headers(self.day_has_headers && self.csv_byte_offset == 0)
               .from_reader(buf_reader);

           // Skip to our byte offset by reading raw bytes
           if self.csv_byte_offset > 0 {
               let inner = reader.into_inner();
               let mut limited = inner.take(0); // we need to skip bytes
               // Actually: use the csv reader's position tracking
           }
       ```

       WAIT — the `csv` crate's Reader does not support seeking. And `ZipFile` doesn't support `Seek`. The correct approach is different:

       **Revised approach**: Instead of byte offsets (which ZipFile doesn't support for Seek), decompress the CSV file ONCE into a `Vec<u8>` on the first chunk call, then use a `Cursor` with seek for subsequent chunks.

       Actually, re-reading the code: the ZIP bytes are already cached (`day_zip_bytes`). The problem is that `ZipArchive::new()` + `archive.by_name()` re-decompresses each time. The real fix:

       **Approach A — Cache decompressed CSV bytes**: On first chunk for a day, decompress the CSV from the ZIP into a new `day_csv_bytes: Option<Vec<u8>>`. Subsequent chunks create a `Cursor` over these bytes, seek to `csv_byte_offset`, and resume parsing.

       Implementation:

       a. Add two fields to `IntraDayChunkIterator`:
          ```rust
          /// Decompressed CSV bytes for current day (decompressed once from ZIP)
          day_csv_bytes: Option<Vec<u8>>,
          /// Byte position in decompressed CSV where last chunk ended
          csv_byte_offset: u64,
          ```
          Initialize both: `day_csv_bytes: None, csv_byte_offset: 0`

       b. At the start of `parse_chunk_from_zip()`, if `day_csv_bytes` is None, decompress:
          ```rust
          if self.day_csv_bytes.is_none() {
              let zip_bytes = self.day_zip_bytes.as_ref().expect("ZIP bytes must be cached");
              let cursor = Cursor::new(zip_bytes.as_slice());
              let mut archive = ZipArchive::new(cursor)?;
              let mut csv_file = archive.by_name(&self.day_csv_filename)?;
              let mut csv_bytes = Vec::with_capacity(csv_file.size() as usize);
              std::io::Read::read_to_end(&mut csv_file, &mut csv_bytes)
                  .map_err(|e| HistoricalError::ParseError { message: e.to_string() })?;
              self.day_csv_bytes = Some(csv_bytes);
          }
          ```

       c. Create a `Cursor` over `day_csv_bytes`, seek to `csv_byte_offset`:
          ```rust
          let csv_bytes = self.day_csv_bytes.as_ref().unwrap();
          let mut cursor = Cursor::new(csv_bytes.as_slice());
          cursor.set_position(self.csv_byte_offset);

          let buf_reader = BufReader::with_capacity(64 * 1024, cursor);
          let mut reader = ReaderBuilder::new()
              .has_headers(false) // headers only on first read; offset > 0 means we're past them
              .from_reader(buf_reader);
          ```
          For the first chunk (offset == 0), set `has_headers(self.day_has_headers)`.

       d. After reading the chunk, save the cursor position:
          ```rust
          // Get position from the reader's underlying cursor
          let position = reader.into_inner().into_inner().position();
          // BufReader may have read ahead, so use: total_csv_len - remaining_in_buf
          // Actually, simpler: track records_consumed and on NEXT call we know offset.
          ```

          Hmm, getting exact byte position from a csv::Reader wrapped in BufReader is tricky because of buffering. Let me reconsider.

       **Approach B (simpler, still O(N))**: Decompress once into `day_csv_bytes`. For each chunk, use the existing `records_consumed` skip approach BUT over an in-memory `Cursor` instead of re-decompressing from ZIP each time. This eliminates the ZIP re-decompression (the expensive part) while keeping the skip logic (cheap over memory).

       The current O(N^2) cost comes from two sources:
       1. ZIP decompression per chunk (expensive I/O) — fixed by caching decompressed CSV
       2. CSV record skipping per chunk (cheap iteration) — acceptable for in-memory data

       For a 20M-trade day, ZIP decompression is ~200-400ms per chunk. Skipping CSV records in memory is ~50ms per 1M records. So caching the decompressed CSV eliminates the dominant cost.

       **Final implementation (Approach B)**:

       a. Add field `day_csv_bytes: Option<Vec<u8>>` to `IntraDayChunkIterator`. Initialize to `None`.

       b. In `parse_chunk_from_zip()`, replace the ZIP re-open with:
          ```rust
          fn parse_chunk_from_zip(&mut self) -> Result<(Vec<AggTrade>, bool), HistoricalError> {
              // Decompress CSV from ZIP once, cache for subsequent chunks
              if self.day_csv_bytes.is_none() {
                  let zip_bytes = self.day_zip_bytes.as_ref().expect("ZIP bytes must be cached");
                  let cursor = Cursor::new(zip_bytes.as_slice());
                  let mut archive = ZipArchive::new(cursor)?;
                  let mut csv_file = archive.by_name(&self.day_csv_filename)?;
                  let mut csv_bytes = Vec::with_capacity(csv_file.size() as usize);
                  std::io::Read::read_to_end(&mut csv_file, &mut csv_bytes)
                      .map_err(|e| HistoricalError::ParseError { message: e.to_string() })?;
                  self.day_csv_bytes = Some(csv_bytes);
              }

              let csv_bytes = self.day_csv_bytes.as_ref().unwrap();
              let cursor = Cursor::new(csv_bytes.as_slice());
              let buf_reader = BufReader::with_capacity(64 * 1024, cursor);
              let mut reader = ReaderBuilder::new()
                  .has_headers(self.day_has_headers)
                  .from_reader(buf_reader);

              // Rest of the existing skip + take logic unchanged
              let skip = self.records_consumed;
              let take = self.max_trades_per_chunk;
              // ... (keep existing code)
          }
          ```

       c. When advancing to next day (in the Iterator `next()` impl), clear the cache:
          When `day_fully_consumed` triggers day advancement, set `self.day_csv_bytes = None` to free memory for the previous day.

       d. Update the `HistoricalError` enum if `ParseError` variant doesn't exist — check first, use existing appropriate variant.

    NOTE: `&mut self` — the current signature is `&self`. Change it to `&mut self` since we now mutate `day_csv_bytes`. Check all call sites. The `Iterator::next()` already has `&mut self` so calling `self.parse_chunk_from_zip()` with `&mut self` will work.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-providers --lib</automated>
  </verify>
  <acceptance_criteria>
    - `grep -n "day_csv_bytes" crates/opendeviationbar-providers/src/binance/historical.rs` shows the new cached field
    - `grep -c "ZipArchive::new" crates/opendeviationbar-providers/src/binance/historical.rs` inside `parse_chunk_from_zip` shows exactly 1 call (inside the `if self.day_csv_bytes.is_none()` branch)
    - Method signature changed from `&self` to `&mut self` for `parse_chunk_from_zip`
    - All tests pass: `cargo nextest run -p opendeviationbar-providers --lib`
    - `cargo build -p opendeviationbar-providers` compiles without warnings
  </acceptance_criteria>
  <done>IntraDayChunkIterator decompresses ZIP once per day (cached in day_csv_bytes), eliminating O(N^2) re-decompression. Cache cleared on day advancement to bound memory. All existing tests pass.</done>
</task>

</tasks>

<verification>
```bash
# All provider tests pass
cargo nextest run -p opendeviationbar-providers --lib

# All core tests pass (new from_f64 tests)

cargo nextest run -p opendeviationbar-core --lib

# Full workspace builds

cargo build --workspace

```
</verification>

<success_criteria>
1. Zero `to_string()` calls in CsvAggTrade→AggTrade conversion paths
2. FixedPoint::from_f64() produces oracle-identical values to from_str() for all Binance price formats
3. IntraDayChunkIterator decompresses ZIP exactly once per day, not once per chunk
4. All existing tests pass across both crates
</success_criteria>

<output>
After completion, create `.planning/phases/03-rust-hot-path/03-01-SUMMARY.md`
</output>
```
