---
phase: 06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests
plan: 04
subsystem: python-memory
tags: [polars, pyarrow, memory-optimization, arrow, parquet]

requires:
  - phase: 06-02
    provides: Rust core memory optimizations
  - phase: 06-03
    provides: GIL release in Arrow processing
provides:
  - Skip plugin Polars-to-Pandas roundtrip when no plugins installed (MEM-04)
  - Footer-only Parquet row counting in get_stats() (MEM-05)
  - Arrow streaming REST fallback path (MEM-06)
  - Single with_columns call in store_bars_batch (MEM-07)
affects: [backfill, checkpoint, bulk-operations, parquet-storage]

tech-stack:
  added: [pyarrow.parquet.read_metadata]
  patterns:
    [discover_providers guard before to_pandas, single with_columns batch]

key-files:
  created: []
  modified:
    - python/opendeviationbar/checkpoint.py
    - python/opendeviationbar/storage/parquet.py
    - python/opendeviationbar/backfill.py
    - python/opendeviationbar/clickhouse/bulk_operations.py

key-decisions:
  - "discover_providers() check gates to_pandas() — cached after first call, zero overhead"
  - "REST fallback converts list[dict] to Polars immediately then uses Arrow path — orphan bars handled via dict-to-DataFrame fallback"
  - "diagonal_relaxed concat used for mixing Arrow-output and dict-constructed DataFrames"

patterns-established:
  - "Guard plugin enrichment with discover_providers() before to_pandas() conversion"
  - "Use pyarrow.parquet.read_metadata() for row counting instead of materializing files"
  - "Collect all with_columns expressions into a list before applying in single call"

requirements-completed: [MEM-04, MEM-05, MEM-06, MEM-07]

duration: 4min
completed: 2026-03-23
---

# Phase 06 Plan 04: Python-Side P0 Memory Optimizations Summary

**Skip plugin Polars-Pandas roundtrip, footer-only Parquet row counting, Arrow REST fallback, and merged with_columns calls**

## Performance

- **Duration:** 4 min
- **Started:** 2026-03-23T03:44:39Z
- **Completed:** 2026-03-23T03:49:09Z
- **Tasks:** 2
- **Files modified:** 4

## Accomplishments

- Plugin enrichment skips Polars-to-Pandas conversion when no plugins installed (eliminates 2x flush peak)
- get_stats() reads only Parquet footer metadata (~few KB) instead of materializing entire files (eliminates 12-42 GB for 531+ daily files)
- REST fallback converts list[dict] to Polars immediately, frees dicts, processes via process_trades_arrow (saves 400-600 MB/day)
- store_bars_batch() applies all metadata columns + session casts in a single with_columns call (eliminates 5+ intermediate DataFrames)

## Task Commits

1. **Task 1: Skip plugin roundtrip + footer-only row counting** - `7937476` (feat)
2. **Task 2: REST Arrow streaming + merge with_columns calls** - `79810fb` (feat)

## Files Created/Modified

- `python/opendeviationbar/checkpoint.py` - Added discover_providers() guard before to_pandas() at both \_flush_accumulated locations
- `python/opendeviationbar/storage/parquet.py` - Replaced pl.read_parquet(f) with pq.read_metadata(f) for row counting
- `python/opendeviationbar/backfill.py` - REST fallback uses Polars+Arrow path instead of list[dict] + process_trades
- `python/opendeviationbar/clickhouse/bulk_operations.py` - Merged three with_columns calls + session cast loop into single call

## Decisions Made

- Used `discover_providers()` (cached after first call) as the guard before `to_pandas()` — the common case (no plugins in production) skips the entire conversion
- REST fallback: kept `_fetch_aggtrades_rest()` returning list[dict] (Rust binding unchanged) but converts to Polars immediately and deletes the dict list; orphan bars from `reset_at_ouroboros()` (dict) handled via `pl.DataFrame([orphan])` and `concat(how="diagonal_relaxed")`
- Used `diagonal_relaxed` concat to handle schema differences between Arrow-output bars and dict-constructed orphan bars

## Deviations from Plan

None - plan executed exactly as written.

## Issues Encountered

None.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- All four Python-side P0 memory optimizations implemented
- Zero behavior change — all optimizations are internal data path improvements
- Ready for oracle regression verification

---

_Phase: 06-memory-efficiency-refactoring-p0-findings-from-issue-300-with-oracle-bit-exact-regression-tests_
_Completed: 2026-03-23_
