---
phase: 06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests
plan: 04
type: execute
wave: 3
depends_on: [06-02, 06-03]
files_modified:
  - python/opendeviationbar/checkpoint.py
  - python/opendeviationbar/storage/parquet.py
  - python/opendeviationbar/backfill.py
  - python/opendeviationbar/clickhouse/bulk_operations.py
autonomous: true
requirements: [MEM-04, MEM-05, MEM-06, MEM-07]

must_haves:
  truths:
    - "Plugin enrichment skips Polars-to-Pandas conversion when no plugins installed"
    - "get_stats() counts rows from Parquet footer metadata — never materializes file data"
    - "REST fallback path in backfill.py uses Arrow streaming — no list[dict] materialization"
    - "store_bars_batch() uses a single with_columns() call — no intermediate DataFrames"
    - "Oracle regression test passes — bit-exact output preserved"
  artifacts:
    - path: "python/opendeviationbar/checkpoint.py"
      provides: "Skip plugin roundtrip when no plugins"
      contains: "discover_providers"
    - path: "python/opendeviationbar/storage/parquet.py"
      provides: "Footer-only row counting"
      contains: "read_metadata"
    - path: "python/opendeviationbar/backfill.py"
      provides: "Arrow streaming REST fallback"
      contains: "process_trades_arrow"
    - path: "python/opendeviationbar/clickhouse/bulk_operations.py"
      provides: "Single with_columns call"
      contains: "new_cols"
  key_links:
    - from: "python/opendeviationbar/checkpoint.py"
      to: "python/opendeviationbar/plugins/loader.py"
      via: "discover_providers() check before to_pandas()"
      pattern: "discover_providers"
    - from: "python/opendeviationbar/backfill.py"
      to: "src/core_bindings.rs"
      via: "process_trades_arrow() replacing process_trades()"
      pattern: "process_trades_arrow"
---

<objective>
Implement four Python-side P0 memory optimizations: (1) skip Polars-Pandas roundtrip when no plugins, (2) footer-only Parquet row counting, (3) Arrow streaming REST fallback, (4) merge with_columns calls.

Purpose: Eliminates the largest Python-side memory waste — 2x flush peak from Polars-Pandas roundtrip, 12-42 GB materialization for row counting, 400-600 MB/day for REST dict path, and 5+ intermediate DataFrames per CH write.

Output: Modified Python files with zero behavior change, verified by oracle regression tests.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/phases/06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests/06-CONTEXT.md

@python/opendeviationbar/checkpoint.py
@python/opendeviationbar/storage/parquet.py
@python/opendeviationbar/backfill.py
@python/opendeviationbar/clickhouse/bulk_operations.py
@python/opendeviationbar/plugins/loader.py
@.scratch/mem-audit/04-python-alloc.md
@.scratch/mem-audit/05-polars-lazy.md
@.scratch/mem-audit/06-kintsugi-pipeline.md
@.scratch/mem-audit/08-ch-write-path.md
@.scratch/mem-audit/09-storage-seeder.md
</context>

<tasks>

<task type="auto">
  <name>Task 1: Skip plugin roundtrip + footer-only row counting</name>
  <files>
    python/opendeviationbar/checkpoint.py
    python/opendeviationbar/storage/parquet.py
  </files>
  <read_first>
    python/opendeviationbar/checkpoint.py
    python/opendeviationbar/storage/parquet.py
    python/opendeviationbar/plugins/loader.py
  </read_first>
  <action>
**Part A: Skip Polars-to-Pandas roundtrip when no plugins installed (MEM-04)**

In `python/opendeviationbar/checkpoint.py`, find the `_flush_accumulated()` inner function (appears at TWO locations: ~line 605 and ~line 749).

At BOTH locations, change the plugin enrichment block from:

```python
# CURRENT (line ~616-620):
if ctx.include_microstructure and not batch_df.is_empty():
    from opendeviationbar.plugins.loader import enrich_bars
    bars_pd = batch_df.to_pandas()
    batch_df = enrich_bars(bars_pd, ctx.symbol, ctx.threshold_decimal_bps)
```

to:

```python
# NEW: Skip Polars->Pandas roundtrip when no plugins installed (MEM-04)
if ctx.include_microstructure and not batch_df.is_empty():
    from opendeviationbar.plugins.loader import discover_providers, enrich_bars
    if discover_providers():  # Non-empty list = plugins installed
        bars_pd = batch_df.to_pandas()
        batch_df = enrich_bars(bars_pd, ctx.symbol, ctx.threshold_decimal_bps)
```

This works because `discover_providers()` caches its result after the first call (`_providers_cache` in `loader.py`). The common case (no plugins in production, per REQUIREMENTS.md "No plugins in production") skips the entire `to_pandas()` + `enrich_bars()` path.

NOTE: `enrich_bars()` already calls `discover_providers()` internally and returns the input unchanged when empty. But the `to_pandas()` conversion happens BEFORE `enrich_bars()` is called — that's the waste. By checking first, we skip the conversion entirely.

**Part B: Footer-only row counting in get_stats() (MEM-05)**

In `python/opendeviationbar/storage/parquet.py`, find `get_stats()` method (~line 630).

Change the row counting loop from:

```python
# CURRENT (line ~659-662):
for f in parquet_files:
    file_data = pl.read_parquet(f)
    total_rows += len(file_data)
    days.append(f.stem)
```

to:

```python
# NEW: Use Parquet footer metadata for row counting (MEM-05)
# Reads only the ~few KB footer per file instead of materializing 12-42 GB of data
import pyarrow.parquet as pq
for f in parquet_files:
    metadata = pq.read_metadata(f)
    total_rows += metadata.num_rows
    days.append(f.stem)
```

Add `import pyarrow.parquet as pq` at the top of the function (lazy import to avoid import cost when get_stats is not called). `pyarrow` is already a dependency (used by Polars and Arrow bindings).

This change reduces I/O from reading entire files (~35 MB each for BTCUSDT) to reading only the Parquet footer (~few KB each). For 531+ daily files, this reduces peak memory from ~42 GB to ~negligible.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_oracle_regression.py -v -x 2>&1 | tail -10</automated>
</verify>
<acceptance_criteria> - `checkpoint.py` contains `discover_providers()` check BEFORE `to_pandas()` (at both_flush_accumulated locations) - `checkpoint.py` does NOT contain bare `bars_pd = batch_df.to_pandas()` without the discover_providers guard - `parquet.py` `get_stats()` method contains `pq.read_metadata(f)` instead of `pl.read_parquet(f)` - `parquet.py` `get_stats()` method does NOT contain `pl.read_parquet` for row counting - `pytest tests/test_oracle_regression.py -v -x` exits 0
</acceptance_criteria>
<done>Plugin enrichment skips Polars-to-Pandas conversion when no plugins installed. Row counting uses Parquet footer metadata with zero data materialization.</done>
</task>

<task type="auto">
  <name>Task 2: REST Arrow streaming + merge with_columns calls</name>
  <files>
    python/opendeviationbar/backfill.py
    python/opendeviationbar/clickhouse/bulk_operations.py
  </files>
  <read_first>
    python/opendeviationbar/backfill.py
    python/opendeviationbar/clickhouse/bulk_operations.py
    python/opendeviationbar/orchestration/helpers.py
    python/opendeviationbar/clickhouse/constants.py
  </read_first>
  <action>
**Part A: REST fallback Arrow streaming in backfill.py (MEM-06)**

In `python/opendeviationbar/backfill.py`, find `_process_gap_fallback_starttime()` (the REST fallback path, ~line 546).

The current flow is:

1. `_fetch_aggtrades_rest()` returns `list[dict]` (~400-600 MB/day)
2. `split_trades_by_day(trades)` splits into sub-lists
3. `processor.process_trades(day_trades)` returns `list[dict]`
4. `pl.DataFrame(all_bars)` creates DataFrame from dicts

Replace with Arrow-based processing:

```python
# NEW: Use Arrow path to avoid list[dict] materialization (MEM-06)
# Instead of: trades = _fetch_aggtrades_rest(...)  # list[dict], 400-600 MB
# Use the processor's Arrow path with the REST data

# Fetch trades as list[dict] (still needed for REST)
trades = _fetch_aggtrades_rest(
    symbol, start_ms=start_ms, end_ms=end_ms, ...
)
if not trades:
    return 0

# Convert trades to Polars DataFrame then Arrow, avoiding the dict-based process_trades()
import polars as pl
trades_df = pl.DataFrame(trades)
del trades  # Free the list[dict] immediately

# Split by day using Polars operations (not list iteration)
from opendeviationbar.validation.day_mode.guards import DAY_MS
trades_df = trades_df.with_columns(
    (pl.col("timestamp") // DAY_MS).alias("_day_key")
)

all_bar_frames = []
for day_key, day_trades in trades_df.group_by("_day_key", maintain_order=True):
    day_trades = day_trades.drop("_day_key")
    # Convert to Arrow and process via zero-copy path
    chunk_arrow = day_trades.to_arrow()
    bars_arrow = processor.process_trades_arrow(chunk_arrow)
    if bars_arrow.num_rows > 0:
        bars_df = pl.from_arrow(bars_arrow)
        all_bar_frames.append(bars_df)
    # Reset processor at day boundary
    processor.reset_at_ouroboros()

if not all_bar_frames:
    return 0

bars_df = pl.concat(all_bar_frames, rechunk=False)
```

IMPORTANT: Read the actual `_process_gap_fallback_starttime()` function carefully before modifying. The function may have additional logic (error handling, logging, verification). Preserve ALL existing logic — only change the data path from dict-based to Arrow-based.

The key insight: `_fetch_aggtrades_rest()` returns `list[dict]` because the Rust binding `fetch_aggtrades_rest` outputs Python dicts. We cannot change the Rust binding in this plan (that would be a separate MEM finding). Instead, we convert the dicts to Polars DataFrame immediately (one conversion) and then use the Arrow path for processing (avoiding the double-dict path of `processor.process_trades()` which creates ANOTHER list of dicts for output).

The column mapping for the trades DataFrame from REST: the REST binding returns dicts with keys `a` (agg_trade_id), `p` (price), `q` (quantity), `T` (timestamp), `m` (is_buyer_maker), `f` (first_trade_id), `l` (last_trade_id). These need to be renamed to match the Arrow schema expected by `process_trades_arrow`: `agg_trade_id`, `price`, `volume`/`quantity`, `timestamp`, `is_buyer_maker`, `first_trade_id`, `last_trade_id`.

Check the actual REST binding output format by reading `src/binance_bindings.rs` or the existing code in `_process_gap_fallback_starttime()`. The REST fallback may use `_fetch_aggtrades_by_id()` (which uses Rust bindings with full column names) rather than raw REST single-letter keys. Read the code to confirm before implementing.

Per Phase 05 decision: "REST fallback uses Rust bindings (full column names) not raw REST single-letter keys". So the dict keys should already be full names.

**Part B: Merge with_columns calls in store_bars_batch() (MEM-07)**

In `python/opendeviationbar/clickhouse/bulk_operations.py`, find `store_bars_batch()` (~line 687).

Merge the three separate `with_columns()` calls (lines ~829-834, ~849-853, ~891-894) into a single call:

Replace:

```python
# Current: 3 separate with_columns + 1 per-column loop
df = df.with_columns(
    pl.lit(symbol).alias("symbol"),
    pl.lit(threshold_decimal_bps).alias("threshold_decimal_bps"),
    pl.lit(ouroboros_mode).alias("ouroboros_mode"),
    pl.lit(effective_version).alias("opendeviationbar_version"),
)
# ... compute start_ts, end_ts, cache_key ...
df = df.with_columns(
    pl.lit(start_ts).alias("source_start_ts"),
    pl.lit(end_ts).alias("source_end_ts"),
    pl.lit(cache_key).alias("cache_key"),
)
# ... later ...
for col in EXCHANGE_SESSION_COLUMNS:
    if col in df.columns:
        df = df.with_columns(pl.col(col).cast(pl.UInt8))
        columns.append(col)
```

With a single merged call:

```python
# NEW: Single with_columns call for all metadata + session casts (MEM-07)
new_cols = [
    pl.lit(symbol).alias("symbol"),
    pl.lit(threshold_decimal_bps).alias("threshold_decimal_bps"),
    pl.lit(ouroboros_mode).alias("ouroboros_mode"),
    pl.lit(effective_version).alias("opendeviationbar_version"),
    pl.lit(start_ts).alias("source_start_ts"),
    pl.lit(end_ts).alias("source_end_ts"),
    pl.lit(cache_key).alias("cache_key"),
]

# Add exchange session casts to the same batch
for col in EXCHANGE_SESSION_COLUMNS:
    if col in df.columns:
        new_cols.append(pl.col(col).cast(pl.UInt8))
        columns.append(col)

df = df.with_columns(new_cols)
```

IMPORTANT: The `start_ts`, `end_ts`, and `cache_key` computation (lines ~837-847) must happen BEFORE the merged `with_columns()` call. Keep the computation logic in place, just move the column additions into the single batch.

Handle the `else` branch (when `close_time_ms` not present or empty df) — the merged call should include the appropriate fallback literals (0, 0, "").

Read the full function carefully to understand the flow. The column list building and optional column detection (lines ~862-894) interleaves with the `with_columns` calls — restructure so ALL column expressions are collected first, then applied in one `df.with_columns(new_cols)`.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_oracle_regression.py tests/test_rust_bindings.py -v -x 2>&1 | tail -10</automated>
</verify>
<acceptance_criteria> - `backfill.py` `_process_gap_fallback_starttime()` contains `process_trades_arrow` (Arrow path) instead of `process_trades` (dict path) - `backfill.py` contains `del trades` to free list[dict] after Polars conversion - `bulk_operations.py` `store_bars_batch()` has a single `df = df.with_columns(new_cols)` call for metadata + session casts - `bulk_operations.py` does NOT have separate `with_columns` calls for metadata columns (lines ~829, ~849 merged) - `bulk_operations.py` does NOT have a `for col in EXCHANGE_SESSION_COLUMNS: df = df.with_columns(...)` loop (merged into single call) - `pytest tests/test_oracle_regression.py -v -x` exits 0
</acceptance_criteria>
<done>REST fallback uses Arrow streaming (400-600 MB/day savings). store_bars_batch uses single with_columns call (5+ fewer intermediate DataFrames per write). Oracle regression passes.</done>
</task>

</tasks>

<verification>
Oracle regression test proves bit-exact output through the full pipeline:
- `pytest tests/test_oracle_regression.py -v -x` exits 0
- `pytest tests/ -v -x --ignore=tests/benchmark` exits 0 (full test suite)
</verification>

<success_criteria>

- Plugin enrichment is no-op when no plugins installed (no Polars-to-Pandas conversion)
- get_stats() reads only Parquet footer (~KB per file, not ~35 MB)
- REST fallback processes via Arrow path, freeing dict list immediately after conversion
- store_bars_batch() applies all column mutations in a single with_columns() call
- All existing tests pass
  </success_criteria>

<output>
After completion, create `.planning/phases/06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests/06-04-SUMMARY.md`
</output>
