---
phase: 06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests
plan: 02
type: execute
wave: 2
depends_on: [06-01]
files_modified:
  - crates/opendeviationbar-core/src/trade.rs
  - crates/opendeviationbar-core/src/arrow_export.rs
  - src/core_bindings.rs
  - crates/opendeviationbar-streaming/src/live_engine.rs
autonomous: true
requirements: [MEM-01, MEM-02, MEM-08]

must_haves:
  truths:
    - "AggTrade derives Copy — push(*trade) replaces push(trade.clone()) in processor hot path"
    - "Arrow input path constructs stack-local AggTrade per iteration — no heap Vec<AggTrade> materialization"
    - "FormingBar watch updates throttled to 1 Hz — 99% fewer OpenDeviationBar clones in streaming"
    - "Oracle regression tests still pass — bit-exact output preserved"
  artifacts:
    - path: "crates/opendeviationbar-core/src/trade.rs"
      provides: "AggTrade with Copy derive"
      contains: "#[derive(Debug, Clone, Copy,"
    - path: "crates/opendeviationbar-core/src/arrow_export.rs"
      provides: "Zero-copy Arrow iterator adapter"
      contains: "fn process_from_arrow_columns"
    - path: "crates/opendeviationbar-streaming/src/live_engine.rs"
      provides: "FormingBar 1Hz throttle"
      contains: "last_forming_update_us"
  key_links:
    - from: "crates/opendeviationbar-core/src/arrow_export.rs"
      to: "crates/opendeviationbar-core/src/processor.rs"
      via: "process_single_trade(&trade) call with stack-local AggTrade"
      pattern: "process_single_trade"
    - from: "src/core_bindings.rs"
      to: "crates/opendeviationbar-core/src/arrow_export.rs"
      via: "process_from_arrow_columns call replacing record_batch_to_aggtrades"
      pattern: "process_from_arrow_columns"
---

<objective>
Implement three Rust-side P0 memory optimizations: (1) derive Copy on AggTrade to eliminate clone overhead, (2) replace Vec<AggTrade> materialization with stack-local Arrow iteration, (3) throttle FormingBar watch updates to 1 Hz.

Purpose: Eliminates the largest Rust-side memory waste — 14-56 MB per Arrow chunk for Vec<AggTrade>, ~1-3ns clone overhead per trade, and ~291 KB/sec of unnecessary FormingBar clones.

Output: Modified Rust crates with zero behavior change, verified by oracle regression tests.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests/06-CONTEXT.md

@crates/opendeviationbar-core/src/trade.rs
@crates/opendeviationbar-core/src/arrow_export.rs
@crates/opendeviationbar-core/src/processor.rs
@src/core_bindings.rs
@crates/opendeviationbar-streaming/src/live_engine.rs
@.scratch/mem-audit/01-rust-copies.md
@.scratch/mem-audit/02-arrow-zerocopy.md
@.scratch/mem-audit/07-sidecar-streaming.md

<interfaces>
<!-- Key types and contracts the executor needs -->

From crates/opendeviationbar-core/src/trade.rs:

```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct AggTrade {
    pub agg_trade_id: i64,
    pub price: FixedPoint,      // FixedPoint is i64 wrapper
    pub volume: FixedPoint,     // FixedPoint is i64 wrapper
    pub first_trade_id: i64,
    pub last_trade_id: i64,
    pub timestamp: i64,
    pub is_buyer_maker: bool,
    pub is_best_match: Option<bool>,
}
```

From crates/opendeviationbar-core/src/processor.rs:

```rust
pub fn process_single_trade(&mut self, trade: &AggTrade) -> Result<Option<OpenDeviationBar>, ProcessingError>;
pub fn process_agg_trade_records(&mut self, trades: &[AggTrade]) -> Result<Vec<OpenDeviationBar>, ProcessingError>;
```

From crates/opendeviationbar-core/src/arrow_export.rs:

```rust
pub fn record_batch_to_aggtrades(batch: &RecordBatch, timestamp_is_microseconds: bool) -> Result<Vec<AggTrade>, ArrowImportError>;
```

From crates/opendeviationbar-streaming/src/live_engine.rs (line 1180-1188):

```rust
if let Some(tx) = forming_tx_map.get(threshold)
    && let Some(incomplete) = processor.get_incomplete_bar()
{
    let _ = tx.send(Some(FormingBar {
        symbol: symbol_arc.clone(),
        threshold_decimal_bps: *threshold,
        bar: incomplete,              // clones ~520B OpenDeviationBar
        last_trade_timestamp_us: trade.timestamp,
    }));
}
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Derive Copy on AggTrade + Arrow zero-copy iterator adapter</name>
  <files>
    crates/opendeviationbar-core/src/trade.rs
    crates/opendeviationbar-core/src/arrow_export.rs
    src/core_bindings.rs
  </files>
  <read_first>
    crates/opendeviationbar-core/src/trade.rs
    crates/opendeviationbar-core/src/arrow_export.rs
    crates/opendeviationbar-core/src/processor.rs
    src/core_bindings.rs
    crates/opendeviationbar-core/src/fixed_point.rs
  </read_first>
  <behavior>
    - Test 1: AggTrade is Copy — `let a = AggTrade{...}; let b = a; let c = a;` compiles (use-after-move is valid for Copy types)
    - Test 2: `process_from_arrow_columns()` produces identical Vec<OpenDeviationBar> as `record_batch_to_aggtrades() + process_agg_trade_records()` for the same input RecordBatch
    - Test 3: Oracle golden snapshot test passes (bit-exact output)
  </behavior>
  <action>
**Part A: Derive Copy on AggTrade (MEM-01)**

In `crates/opendeviationbar-core/src/trade.rs`, change:

```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
```

to:

```rust
#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
```

This is safe because AggTrade contains only: 5x i64, 2x FixedPoint(i64), 1x bool, 1x Option<bool> — all Copy types. FixedPoint must also derive Copy — check `fixed_point.rs` and add `Copy` if not already derived.

In `processor.rs`, change `trade.clone()` at lines ~1233 and ~1250 to `*trade` (dereference instead of clone). The `.clone()` calls still work with Copy (Copy is a subtrait of Clone), but `*trade` is more idiomatic and signals the intent.

**Part B: Arrow zero-copy iterator adapter (MEM-02)**

In `crates/opendeviationbar-core/src/arrow_export.rs`, add a new public function:

```rust
/// Process trades directly from Arrow column arrays without materializing Vec<AggTrade>.
/// Each AggTrade is constructed on the stack per iteration and passed to process_single_trade().
/// Saves 14-56 MB per 250K-trade chunk by avoiding the heap Vec allocation.
pub fn process_from_arrow_columns(
    processor: &mut OpenDeviationBarProcessor,
    batch: &RecordBatch,
    timestamp_is_microseconds: bool,
) -> Result<Vec<OpenDeviationBar>, ArrowImportError> {
    let num_rows = batch.num_rows();
    if num_rows == 0 {
        return Ok(Vec::new());
    }

    // Extract column slices (zero-copy references into Arrow buffers)
    let timestamp_col = get_int64_column(batch, "timestamp")?;
    let price_col = get_float64_column(batch, "price")?;
    // ... (same column extraction as record_batch_to_aggtrades)

    let ts_multiplier: i64 = if timestamp_is_microseconds { 1 } else { 1000 };

    let mut bars = Vec::with_capacity(num_rows / 100); // ~1 bar per 100 trades heuristic

    for i in 0..num_rows {
        let trade = AggTrade {
            timestamp: timestamp_col.value(i) * ts_multiplier,
            price: f64_to_fixed_point(price_col.value(i)),
            volume: f64_to_fixed_point(volume_col.value(i)),
            agg_trade_id: agg_id_col.map_or(i as i64, |c| c.value(i)),
            first_trade_id: first_id_col.map_or(agg_id_val, |c| c.value(i)),
            last_trade_id: last_id_col.map_or(agg_id_val, |c| c.value(i)),
            is_buyer_maker: buyer_maker_col.map_or(false, |c| c.value(i)),
            is_best_match: best_match_col.map(|c| if c.is_null(i) { None } else { Some(c.value(i)) }).unwrap_or(None),
        };
        match processor.process_single_trade(&trade) {
            Ok(Some(bar)) => bars.push(bar),
            Ok(None) => {}
            Err(e) => return Err(ArrowImportError::ProcessingError(e.to_string())),
        }
    }
    Ok(bars)
}
```

Add a `ProcessingError` variant to `ArrowImportError`:

```rust
#[error("Processing error: {0}")]
ProcessingError(String),
```

Handle optional columns the same way as `record_batch_to_aggtrades()` — extract them once before the loop as `Option<&Int64Array>` / `Option<&BooleanArray>`.

**Part C: Wire into PyO3 bindings**

In `src/core_bindings.rs`, update `process_trades_arrow()` and `process_trades_arrow_native()` to use `process_from_arrow_columns()` instead of `record_batch_to_aggtrades()` + `process_agg_trade_records()`:

```rust
// OLD:
let agg_trades = record_batch_to_aggtrades(record_batch, false)?;
let bars = self.processor.process_agg_trade_records(&agg_trades)?;

// NEW:
let bars = process_from_arrow_columns(&mut self.processor, record_batch, false)
    .map_err(|e| PyRuntimeError::new_err(format!("Arrow processing failed: {e}")))?;
```

Keep `record_batch_to_aggtrades()` as a public function (other callers may use it) — do NOT delete it.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-core --features test-utils test_golden_snapshot_bit_exact 2>&1 | tail -5 && cargo nextest run --workspace --exclude opendeviationbar-py 2>&1 | tail -5</automated>
</verify>
<acceptance_criteria> - `trade.rs` contains `#[derive(Debug, Clone, Copy,` - `arrow_export.rs` contains `pub fn process_from_arrow_columns` - `core_bindings.rs` contains `process_from_arrow_columns` call in `process_trades_arrow` - `record_batch_to_aggtrades` still exists as a public function (not deleted) - `cargo nextest run -p opendeviationbar-core --features test-utils test_golden_snapshot_bit_exact` exits 0 - `cargo nextest run --workspace --exclude opendeviationbar-py` exits 0
</acceptance_criteria>
<done>AggTrade derives Copy, Arrow iterator adapter eliminates Vec<AggTrade> heap allocation, PyO3 bindings use new zero-copy path, oracle regression passes.</done>
</task>

<task type="auto">
  <name>Task 2: FormingBar watch throttle to 1 Hz</name>
  <files>
    crates/opendeviationbar-streaming/src/live_engine.rs
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/live_engine.rs
  </read_first>
  <action>
In `crates/opendeviationbar-streaming/src/live_engine.rs`, throttle forming bar watch channel updates to 1 Hz (MEM-08).

**In `process_trade_through_processors()` (the function containing lines ~1180-1188):**

Add a `last_forming_update_us: &mut i64` parameter to the function signature (or use a per-processor HashMap passed from the caller).

Before the forming bar send block (line ~1177-1189), add a throttle check:

```rust
Ok(None) => {
    // Issue #214 + MEM-08: Update forming bar snapshot for consumers.
    // Throttled to 1 Hz — consumers poll at 1s intervals, so 99.3% of
    // updates are immediately overwritten by watch::send().
    let now_us = trade.timestamp;
    if let Some(tx) = forming_tx_map.get(threshold) {
        if now_us - *last_forming_update_us >= 1_000_000 {  // 1 second in microseconds
            if let Some(incomplete) = processor.get_incomplete_bar() {
                let _ = tx.send(Some(FormingBar {
                    symbol: symbol_arc.clone(),
                    threshold_decimal_bps: *threshold,
                    bar: incomplete,
                    last_trade_timestamp_us: trade.timestamp,
                }));
                *last_forming_update_us = now_us;
            }
        }
    }
}
```

**Tracking state**: Add a `last_forming_update_us: HashMap<u32, i64>` (keyed by threshold) initialized to 0 at the call site in `symbol_task()` / `symbol_task_from_checkpoint()`. Pass a mutable reference to the per-threshold timestamp into `process_trade_through_processors()`.

Alternatively, if the function signature change is too invasive, use a simpler approach: add a `last_forming_update_us: i64` field directly into the loop state in `symbol_task()`. Since `process_trade_through_processors` iterates over all processors for a single trade, and the forming bar update is per-processor, track it as a `HashMap<u32, i64>` in the outer loop:

```rust
let mut last_forming_update: HashMap<u32, i64> = HashMap::new();
// ... in the trade processing loop:
for (idx, (threshold, processor)) in processors.iter_mut().enumerate() {
    match processor.process_single_trade(&trade) {
        // ... bar completed case unchanged ...
        Ok(None) => {
            let last_us = last_forming_update.entry(*threshold).or_insert(0);
            if trade.timestamp - *last_us >= 1_000_000 {
                if let Some(tx) = forming_tx_map.get(threshold) {
                    if let Some(incomplete) = processor.get_incomplete_bar() {
                        let _ = tx.send(Some(FormingBar { ... }));
                        *last_us = trade.timestamp;
                    }
                }
            }
        }
    }
}
```

IMPORTANT: When a bar COMPLETES (Ok(Some(bar))), always reset the throttle timer for that threshold so the new forming bar is sent immediately:

```rust
Ok(Some(bar)) => {
    // ... existing bar completion logic ...
    // Reset forming bar throttle — new bar started, send forming state immediately
    last_forming_update.insert(*threshold, 0);
}
```

This applies to BOTH `process_trade_through_processors()` functions (there are two — one for normal mode, one for checkpoint-restored mode at ~line 1239).
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-streaming 2>&1 | tail -5</automated>
</verify>
<acceptance_criteria> - `live_engine.rs` contains `last_forming_update` variable - `live_engine.rs` contains `1_000_000` (1 second throttle in microseconds) - `live_engine.rs` contains throttle reset on bar completion (`last_forming_update.insert`) - Both `process_trade_through_processors` functions (normal and checkpoint-restored) have the throttle - `cargo nextest run -p opendeviationbar-streaming` exits 0
</acceptance_criteria>
<done>FormingBar watch updates throttled to 1 Hz. New forming bar sent immediately after bar completion (throttle reset). Both process_trade_through_processors variants updated.</done>
</task>

</tasks>

<verification>
Oracle regression tests prove bit-exact output preservation:
- `cargo nextest run -p opendeviationbar-core --features test-utils test_golden_snapshot_bit_exact` exits 0
- `cargo nextest run --workspace --exclude opendeviationbar-py` exits 0
- `maturin develop && pytest tests/test_oracle_regression.py -v -x` exits 0 (full Python pipeline)
</verification>

<success_criteria>

- AggTrade derives Copy (trade.rs line 25 contains `Copy`)
- Arrow iterator adapter exists and is wired into PyO3 bindings
- FormingBar updates throttled to 1 Hz with reset on bar completion
- All existing tests pass (cargo nextest, pytest oracle)
- No behavior change — identical output for identical input
  </success_criteria>

<output>
After completion, create `.planning/phases/06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests/06-02-SUMMARY.md`
</output>
