---
phase: 23-arrow-ffi-drain-bars
plan: "01"
subsystem: streaming
tags: [arrow, ffi, polars, zero-copy, memory-efficiency, pyo3]

requires:
  - phase: none
    provides: existing drain_bars dict path and opendeviationbar_vec_to_record_batch

provides:
  - drain_bars_arrow() Rust method on StreamManager returning Arrow RecordBatch
  - PyStreamManager.drain_bars_arrow() PyO3 binding with GIL release
  - sync_flush Arrow path with Polars zero-copy grouping
  - completed_bars_to_record_batch() in opendeviationbar-streaming

affects: [sidecar-startup, hot-path-allocation, memory-efficiency]

tech-stack:
  added: [arrow feature on opendeviationbar-streaming]
  patterns: [Arrow FFI for bulk bar transfer, Polars partition_by for grouping]

key-files:
  created:
    - crates/opendeviationbar-streaming/src/arrow_export.rs
    - python/opendeviationbar/sidecar/arrow_flush.py
  modified:
    - crates/opendeviationbar-streaming/Cargo.toml
    - crates/opendeviationbar-streaming/src/lib.rs
    - crates/opendeviationbar-streaming/src/stream_manager.rs
    - src/stream_manager_bindings.rs
    - Cargo.toml
    - python/opendeviationbar/sidecar/midnight.py

key-decisions:
  - "Arrow feature on streaming crate enables core/arrow + arrow-array + arrow-schema"
  - "Weak dep syntax (streaming?/arrow) wires arrow-export to streaming when both enabled"
  - "Separate arrow_flush.py module to keep midnight.py under file size threshold"
  - "Legacy dict path preserved as fallback for OdbEngine or older builds"

patterns-established:
  - "Arrow FFI: Rust CompletedBar vec -> RecordBatch with metadata columns -> Polars DataFrame"
  - "Feature dispatch: hasattr(engine, 'drain_bars_arrow') gates Arrow vs dict path"

requirements-completed: [ARROW-01, ARROW-02]

duration: 19min
completed: 2026-03-27
---

# Phase 23 Plan 01: Arrow FFI drain_bars Summary

**Arrow RecordBatch drain with \_symbol/\_threshold metadata columns, Polars zero-copy grouping in sync_flush, GIL-released PyO3 binding**

## Performance

- **Duration:** 19 min
- **Started:** 2026-03-27T20:47:54Z
- **Completed:** 2026-03-27T21:06:54Z
- **Tasks:** 3
- **Files modified:** 8

## Accomplishments

- Rust `completed_bars_to_record_batch()` extends core Arrow schema with `_symbol` (Utf8) and `_threshold_decimal_bps` (UInt32) metadata columns
- `StreamManager::drain_bars_arrow()` drains ring buffer directly to Arrow RecordBatch (no per-bar dict allocation)
- PyO3 `drain_bars_arrow()` binding returns PyRecordBatch with GIL released during Rust work
- `sync_flush_arrow()` uses Polars `partition_by` for zero-copy grouping, `semi` join for valid_pairs, vectorized cross-midnight filter

## Task Commits

1. **Task 1: Rust Arrow export for CompletedBar vec** - `41fc02e` (feat)
2. **Task 2: PyO3 drain_bars_arrow binding** - `811d4e1` (feat)
3. **Task 3: Python sync_flush uses Arrow path** - `75d115e` (feat)

## Files Created/Modified

- `crates/opendeviationbar-streaming/src/arrow_export.rs` - CompletedBar to RecordBatch with metadata columns
- `crates/opendeviationbar-streaming/Cargo.toml` - Arrow feature + deps
- `crates/opendeviationbar-streaming/src/lib.rs` - arrow_export module registration
- `crates/opendeviationbar-streaming/src/stream_manager.rs` - drain_bars_arrow() method
- `src/stream_manager_bindings.rs` - PyO3 drain_bars_arrow() binding
- `Cargo.toml` - Weak dep syntax for streaming arrow feature
- `python/opendeviationbar/sidecar/arrow_flush.py` - Arrow-based sync_flush with Polars grouping
- `python/opendeviationbar/sidecar/midnight.py` - Arrow path dispatch

## Decisions Made

- Used weak dependency syntax (`opendeviationbar-streaming?/arrow`) to conditionally enable Arrow on the streaming crate only when both `arrow-export` and `streaming` features are active
- Extracted Arrow flush logic into `arrow_flush.py` to avoid midnight.py exceeding file size limits
- Preserved legacy dict path as fallback -- `hasattr(engine, 'drain_bars_arrow')` gates the dispatch
- Stathera diagnostic reimplemented in Polars (shift/slice) instead of Python iteration

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None - all data paths are fully wired.

## Issues Encountered

None.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Arrow FFI drain path is complete and ready for production deployment
- Phase 24 (Hot-Path Allocation Reduction) can proceed independently
- Phase 25 (Lazy Evaluation) can proceed independently
- ARROW-03 (peak RSS measurement) requires production deployment with memray

---

_Phase: 23-arrow-ffi-drain-bars_
_Completed: 2026-03-27_
