---
phase: 06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests
plan: 03
type: execute
wave: 2
depends_on: [06-01]
files_modified:
  - src/core_bindings.rs
autonomous: true
requirements: [MEM-03]

must_haves:
  truths:
    - "GIL released during pure Rust computation in process_trades_arrow() and process_trades_arrow_native()"
    - "Other Python threads can execute during Rust bar processing"
    - "Oracle regression tests still pass — bit-exact output preserved"
  artifacts:
    - path: "src/core_bindings.rs"
      provides: "GIL-released Arrow processing methods"
      contains: "allow_threads"
  key_links:
    - from: "src/core_bindings.rs"
      to: "crates/opendeviationbar-core/src/arrow_export.rs"
      via: "py.allow_threads closure around process_from_arrow_columns"
      pattern: "allow_threads"
---

<objective>
Release the GIL during pure Rust computation in batch process_trades_arrow() and process_trades_arrow_native() methods.

Purpose: When processing 100K+ trades (~50-200ms of pure Rust computation), the GIL blocks all Python threads — heartbeat, Telegram alerting, HTTP health endpoint. Releasing the GIL allows concurrent Python execution during Rust processing.

Output: Modified PyO3 bindings with py.allow_threads() around pure Rust computation.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests/06-CONTEXT.md

@src/core_bindings.rs
@.scratch/mem-audit/03-pyo3-boundary.md

<interfaces>
<!-- Key contracts from PyO3 boundary audit -->

From src/core_bindings.rs:

```rust
// process_trades_arrow (line ~455-479):
fn process_trades_arrow(&mut self, py: Python, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    let record_batch = batch.as_ref();
    let agg_trades = record_batch_to_aggtrades(record_batch, false)?;
    let bars = self.processor.process_agg_trade_records(&agg_trades)?;
    // ... convert bars to RecordBatch ...
}

// process_trades_arrow_native (line ~498-522):
fn process_trades_arrow_native(&mut self, py: Python, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    let record_batch = batch.as_ref();
    let agg_trades = record_batch_to_aggtrades(record_batch, true)?;  // timestamp_is_microseconds=true
    let bars = self.processor.process_agg_trade_records(&agg_trades)?;
    // ...
}
```

PyO3 GIL release pattern:

```rust
// py.allow_threads requires the closure to NOT reference any Python objects
// The processor (&mut self) is a #[pyclass] but its inner data is pure Rust
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: GIL release in process_trades_arrow and process_trades_arrow_native</name>
  <files>
    src/core_bindings.rs
  </files>
  <read_first>
    src/core_bindings.rs
    src/lib.rs
    crates/opendeviationbar-core/src/arrow_export.rs
  </read_first>
  <action>
In `src/core_bindings.rs`, modify `process_trades_arrow()` and `process_trades_arrow_native()` to release the GIL during pure Rust computation (MEM-03).

**Strategy**: The `py.allow_threads()` closure requires `Send` bounds. The `&mut self.processor` (`OpenDeviationBarProcessor`) is pure Rust data wrapped in a `#[pyclass]`. We can temporarily take ownership of the processor's inner data for the GIL-free section.

**Implementation pattern** (using `std::mem::take` or a wrapper):

```rust
fn process_trades_arrow(&mut self, py: Python, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    // Phase 1: Extract Arrow data (needs Python reference for PyRecordBatch)
    let record_batch = batch.0.clone();  // Clone the Arc<RecordBatch> - cheap refcount bump

    // Phase 2: Process (release GIL)
    // Take processor out temporarily - it's pure Rust, no Python references
    let mut processor = std::mem::take(&mut self.processor);
    let result = py.allow_threads(move || {
        process_from_arrow_columns(&mut processor, &record_batch, false)
    });
    self.processor = processor;  // Put processor back

    let bars = result.map_err(|e| PyRuntimeError::new_err(format!("Arrow processing failed: {e}")))?;

    // Phase 3: Convert output (needs GIL for PyRecordBatch construction)
    let output_batch = opendeviationbar_vec_to_record_batch(&bars);
    Ok(PyRecordBatch(output_batch))
}
```

**CRITICAL**: For `std::mem::take` to work, `OpenDeviationBarProcessor` must implement `Default`. Check if it does. If not, use an `Option<OpenDeviationBarProcessor>` wrapper in the `#[pyclass]`:

Alternative approach if Default is not available — use `unsafe` to temporarily move the processor:

```rust
fn process_trades_arrow(&mut self, py: Python, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    let record_batch = batch.0.clone();

    // SAFETY: We move processor out, use it in the GIL-free closure, then move it back.
    // No other code can access self.processor between the take and put-back because
    // we hold &mut self for the entire method duration.
    let mut processor = unsafe {
        std::ptr::read(&self.processor)
    };
    let result = py.allow_threads(move || {
        let r = process_from_arrow_columns(&mut processor, &record_batch, false);
        (processor, r)
    });
    let (processor, bars_result) = result;
    unsafe {
        std::ptr::write(&mut self.processor, processor);
    }

    let bars = bars_result.map_err(|e| PyRuntimeError::new_err(format!("Arrow processing failed: {e}")))?;
    let output_batch = opendeviationbar_vec_to_record_batch(&bars);
    Ok(PyRecordBatch(output_batch))
}
```

**Preferred approach**: Check if `PyRecordBatch` wraps `RecordBatch` directly or via `Arc`. If it's `Arc<RecordBatch>`, cloning is just a refcount bump. Read the PyRecordBatch struct definition in the `arrow` PyO3 interop.

**Simpler alternative if Send bounds are problematic**: Use `py.allow_threads` with a standalone function that takes owned data:

```rust
fn process_trades_arrow(&mut self, py: Python, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    let record_batch = batch.0.clone();
    // Extract processor state needed for processing
    let bars = {
        let processor = &mut self.processor;
        // Since processor is &mut reference to pure Rust data,
        // and allow_threads just needs the closure to not hold Py<> references:
        py.allow_threads(|| {
            process_from_arrow_columns(processor, &record_batch, false)
        })
    };
    let bars = bars.map_err(|e| PyRuntimeError::new_err(format!("Arrow processing failed: {e}")))?;
    let output_batch = opendeviationbar_vec_to_record_batch(&bars);
    Ok(PyRecordBatch(output_batch))
}
```

NOTE: `py.allow_threads` in PyO3 takes `FnOnce() -> T` where the closure must be `Send`. The `&mut self.processor` borrow may or may not satisfy Send depending on the processor's internals. If the processor contains only standard Rust types (no Rc, no raw pointers, no thread-local refs), it should be Send.

Try the simplest approach first. If compiler rejects due to Send bounds, escalate to the `std::mem::take` approach (requires Default) or the `unsafe ptr::read/write` approach.

Apply the same pattern to `process_trades_arrow_native()` (identical structure, `timestamp_is_microseconds=true`).

Do NOT modify `process_trades()` or `process_trades_streaming()` — those use the dict path which requires the GIL for dict extraction. GIL release there would require splitting into extract-then-process phases, which is more complex and lower priority.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && maturin develop 2>&1 | tail -3 && uv run --python 3.13 pytest tests/test_oracle_regression.py -v -x 2>&1 | tail -10</automated>
</verify>
<acceptance_criteria> - `core_bindings.rs` contains `allow_threads` in the `process_trades_arrow` method - `core_bindings.rs` contains `allow_threads` in the `process_trades_arrow_native` method - `maturin develop` compiles without errors - `pytest tests/test_oracle_regression.py -v -x` exits 0 (oracle regression passes through full PyO3 boundary) - No `allow_threads` in `process_trades()` or `process_trades_streaming()` (dict path unchanged)
</acceptance_criteria>
<done>GIL released during pure Rust computation in both Arrow processing methods. Other Python threads can run during 50-200ms Rust processing window. Oracle regression passes.</done>
</task>

</tasks>

<verification>
Oracle regression test proves bit-exact output through the full Python pipeline with GIL release:
- `maturin develop` compiles
- `pytest tests/test_oracle_regression.py -v -x` exits 0
- `cargo nextest run --workspace --exclude opendeviationbar-py` exits 0
</verification>

<success_criteria>

- `py.allow_threads` wraps the Rust computation in both Arrow processing methods
- Oracle regression test passes (bit-exact through PyO3 boundary)
- No changes to non-Arrow processing methods
  </success_criteria>

<output>
After completion, create `.planning/phases/06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests/06-03-SUMMARY.md`
</output>
