---
phase: 23-arrow-ffi-drain-bars
plan: "02"
subsystem: streaming
tags: [arrow, polars, zero-copy, memory-efficiency, sidecar]

requires:
  - phase: 23-arrow-ffi-drain-bars-01
    provides: drain_bars_arrow() Rust method, sync_flush_arrow in arrow_flush.py

provides:
  - _sync_flush_rest_bars_arrow() with df.filter() zero-copy per-pair slicing
  - _sync_flush_rest_bars_dict() fallback for older builds
  - Polars column-level junction telemetry and Stathera diagnostics

affects: [sidecar-startup, memory-efficiency, hot-path-allocation]

tech-stack:
  added: []
  patterns:
    [
      df.filter() zero-copy slicing instead of partition_by,
      pl.col() expressions for cross-midnight filter,
    ]

key-files:
  created: []
  modified:
    - python/opendeviationbar/sidecar/arrow_flush.py
    - python/opendeviationbar/sidecar/midnight.py

key-decisions:
  - "df.filter() for zero-copy per-pair slicing instead of partition_by (which copies data)"
  - "Kept Arrow implementation in arrow_flush.py (Plan 01 decision) to keep midnight.py under file size threshold"
  - "Import alias in midnight.py makes _sync_flush_rest_bars_arrow importable from both modules"

patterns-established:
  - "Zero-copy pair iteration: unique().iter_rows() + df.filter() instead of partition_by()"
  - "Polars column aggregation for telemetry: df['col'].max() instead of max(b.get() for b in bars)"

requirements-completed: [ARROW-02, ARROW-03]

duration: 4min
completed: 2026-03-27
---

# Phase 23 Plan 02: Arrow sync_flush consumer with Polars zero-copy filter Summary

**Replaced partition_by with df.filter() zero-copy slicing in sync_flush Arrow path, split dispatch into \_sync_flush_rest_bars_arrow/\_dict functions**

## Performance

- **Duration:** 4 min
- **Started:** 2026-03-27T21:09:57Z
- **Completed:** 2026-03-27T21:14:26Z
- **Tasks:** 2
- **Files modified:** 2

## Accomplishments

- Replaced all `partition_by()` calls with `df.filter()` for true zero-copy per-pair slicing on Arrow-backed data
- Split `_sync_flush_rest_bars` into dispatch + `_sync_flush_rest_bars_arrow` + `_sync_flush_rest_bars_dict`
- Cross-midnight filter uses `pl.col()` expressions, junction telemetry uses Polars `.max()`/`.min()` aggregation
- All 9 existing sync_flush tests pass, empty drain via Arrow path verified

## Task Commits

1. **Task 1: Rewrite \_sync_flush_rest_bars with Arrow path + dict fallback** - `b70cc5a` (feat)
2. **Task 2: End-to-end verification** - verification only, no code changes

## Files Created/Modified

- `python/opendeviationbar/sidecar/arrow_flush.py` - Replaced partition_by with df.filter() zero-copy, renamed to_sync_flush_rest_bars_arrow
- `python/opendeviationbar/sidecar/midnight.py` - Dispatch function + \_sync_flush_rest_bars_dict fallback

## Decisions Made

- Used `df.filter()` for zero-copy per-pair slicing instead of `partition_by()` which creates copies -- key memory optimization for 200K-bar drains
- Kept Arrow implementation in `arrow_flush.py` (Plan 01 decision to avoid midnight.py exceeding file size limit) with import alias in midnight.py for importability
- Stathera diagnostic also converted to df.filter() for consistency

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None - all data paths are fully wired.

## Issues Encountered

None.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Arrow FFI drain path complete with zero-copy Polars consumer
- ARROW-03 (peak RSS measurement) requires production deployment with memray
- Phase 24 (Hot-Path Allocation Reduction) can proceed independently

## Self-Check: PASSED

- FOUND: python/opendeviationbar/sidecar/arrow_flush.py
- FOUND: python/opendeviationbar/sidecar/midnight.py
- FOUND: .planning/phases/23-arrow-ffi-drain-bars/23-02-SUMMARY.md
- FOUND: commit b70cc5a

---

_Phase: 23-arrow-ffi-drain-bars_
_Completed: 2026-03-27_
