# Phase 6: Memory Efficiency Refactoring — P0 Findings from Issue #300 — Context

**Gathered:** 2026-03-23
**Status:** Ready for planning
**Mode:** Infrastructure phase — discuss skipped (pure internal refactoring, no user-facing behavior changes)

<domain>
## Phase Boundary

Implement the 8 P0 findings from the 9-agent memory efficiency audit (Issue #300, `.scratch/mem-audit/01-09*.md`). Each refactoring MUST preserve bit-exact Oracle output — identical OHLCV, trade IDs, timestamps, and microstructure features for the same input trades. The 866K-bar ClickHouse oracle proof methodology from `docs/OPEN-DEVIATION-BAR.md` is the gold standard.

### P0 Findings to Implement

1. **Arrow iterator adapter** — Replace `Vec<AggTrade>` materialization in `record_batch_to_aggtrades()` with column-slice iteration (`arrow_export.rs:79-184`). Saves 14-56 MB per call.
2. **`#[derive(Copy)]` on AggTrade** — 58-byte scalar struct currently cloned on every trade with intra-bar features (`processor.rs:1233,1250`).
3. **GIL release in batch `process_trades*`** — `py.allow_threads()` around pure Rust computation (`core_bindings.rs`).
4. **Remove Polars→Pandas→Polars roundtrip** — `_flush_accumulated` in `checkpoint.py:619` converts for `enrich_bars()` then back. Plugin enrichment should accept Polars.
5. **Footer-only row counting** — `get_cache_stats()`/`get_stats()` materializes 12-42 GB to count rows (`storage/parquet.py:660`). Use `pyarrow.parquet.read_metadata().num_rows`.
6. **REST fallback → Arrow streaming** — REST path in `backfill.py:546` materializes all trades as `list[dict]` (~400-600 MB/day). Parquet/Vision paths already use Arrow.
7. **Merge `with_columns()` calls** — 4+ separate calls in `store_bars_batch` (`bulk_operations.py:815-891`) each create intermediate DataFrames.
8. **FormingBar watch throttle** — `OpenDeviationBar` clone on every trade for watch channels polled at 1 Hz (`live_engine.rs:1180`). Throttle to 1 Hz.

### Oracle Bit-Exact Regression Tests

Every refactoring must prove output identity. Test strategy:

- **Golden snapshot**: Capture output for a known set of trades (BTCUSDT 2024-01-15, ~760K trades) at all 4 thresholds using current code
- **Assert**: Refactored code produces byte-identical OHLCV + trade IDs + timestamps + microstructure features
- **Scope**: Rust-side tests (cargo nextest) for findings 1-3, 8. Python-side tests (pytest) for findings 4-7.
- **Orphan bars**: Include Ouroboros reset boundary in golden data

</domain>

<decisions>
## Implementation Decisions

### Claude's Discretion

All implementation choices are at Claude's discretion — pure infrastructure phase. Use the 9-agent audit reports (`.scratch/mem-audit/01-09*.md`) and codebase conventions to guide decisions. Key constraints:

- **No API changes** — all refactoring is internal-only. Public Python API signatures unchanged.
- **Batch/streaming parity** — `process_agg_trade_records()` and `process_single_trade()` must continue to produce identical output.
- **Checkpoint continuity** — `create_checkpoint()` / `from_checkpoint()` format unchanged.
- **No feature flags** — these are drop-in replacements, not feature-gated.

</decisions>

<code_context>

## Existing Code Insights

### Reusable Assets

- `process_trades_arrow()` — already the zero-copy Arrow path, proven in repair_direct_parquet.py (379 days/min)
- `_get_worker_cache()` — thread-local CH cache pattern (scripts/repair_direct_parquet.py)
- `_force_allocator_purge()` — mimalloc/glibc page return utility
- `docs/OPEN-DEVIATION-BAR.md` — 866K-bar oracle proof methodology

### Established Patterns

- Rust: `with_capacity()` pre-allocation, `take()` instead of clone for completed bars
- Python: Polars lazy scan with predicate pushdown, Arrow zero-copy boundaries
- Testing: `cargo nextest` for Rust, `pytest` for Python, property-based with `proptest`

### Integration Points

- `crates/opendeviationbar-core/src/arrow_export.rs` — Arrow↔Rust conversion (findings 1, 2)
- `src/core_bindings.rs` — PyO3 boundary (finding 3)
- `python/opendeviationbar/checkpoint.py` — populate pipeline (finding 4)
- `python/opendeviationbar/storage/parquet.py` — tick storage (finding 5)
- `python/opendeviationbar/backfill.py` — REST fallback (finding 6)
- `python/opendeviationbar/clickhouse/bulk_operations.py` — CH write path (finding 7)
- `crates/opendeviationbar-streaming/src/live_engine.rs` — streaming (finding 8)

</code_context>

<specifics>
## Specific Ideas

- Use the BTCUSDT 2024-01-15 daily Parquet file as the golden test fixture (~760K trades, covers flash crash wide-open bars)
- For Rust iterator adapter: benchmark with Criterion before/after on 100K, 500K, 1M trade batches
- For GIL release: verify with Python `threading` test that other threads can run during Rust computation
- For `enrich_bars()`: check if any production plugins actually exist (REQUIREMENTS.md says "No plugins in production")

</specifics>

<deferred>
## Deferred Ideas

- P1 findings (25 items) — tracked in Issue #300, separate phase
- P2 findings (61 items) — tracked in Issue #300, backlog
- `store_bars_bulk` pandas path migration to Polars — requires caller audit
- Polars-native public API return path — API change, separate milestone

</deferred>
