---
phase: 23-arrow-ffi-drain-bars
verified: 2026-03-27T21:18:05Z
status: human_needed
score: 5/6 must-haves verified
human_verification:
  - test: "Deploy sidecar on bigblack and monitor RSS during fill_from_rest with 16 symbols active (full 200K-bar drain scenario)"
    expected: "Peak sidecar RSS stays below 2.5G during drain (ARROW-03 threshold)"
    why_human: "Cannot measure production RSS without a live ClickHouse + WebSocket environment. memray/psutil instrumentation requires a running sidecar with real data volume."
---

# Phase 23: Arrow FFI drain_bars Verification Report

**Phase Goal:** `drain_bars` returns Arrow RecordBatch instead of `list[dict]`, reducing peak RSS during sync_flush from ~2.6G to ~0.9G
**Verified:** 2026-03-27T21:18:05Z
**Status:** human_needed
**Re-verification:** No — initial verification

## Goal Achievement

### Observable Truths

| #   | Truth                                                                                        | Status    | Evidence                                                                                                                                             |
| --- | -------------------------------------------------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1   | `StreamManager.drain_bars_arrow()` returns Arrow RecordBatch with all bar fields + metadata  | VERIFIED  | `stream_manager.rs:390` — `pub fn drain_bars_arrow()` returns `arrow_array::RecordBatch`, calls `completed_bars_to_record_batch()`                   |
| 2   | RecordBatch includes `_symbol` (Utf8) and `_threshold_decimal_bps` (UInt32) columns          | VERIFIED  | `arrow_export.rs:72-77` — appends both fields; tests at lines 121-126 and 145-153 assert column presence and values                                  |
| 3   | `_sync_flush_rest_bars` dispatches to Arrow path when `drain_bars_arrow` is available        | VERIFIED  | `midnight.py:499-501` — `hasattr(engine, "drain_bars_arrow")` gates dispatch; dict fallback at line 500 preserved                                    |
| 4   | Polars zero-copy filter (`df.filter()`) replaces `defaultdict(list)` grouping in Arrow path  | VERIFIED  | `arrow_flush.py:147-150` — per-pair slices via `df.filter()`, no `partition_by()` or `defaultdict` in Arrow path                                     |
| 5   | Cross-midnight filter, Stathera diagnostic, and junction telemetry operate on Polars columns | VERIFIED  | `arrow_flush.py:86-105` (cross-midnight on `pl.col()`), lines 212-258 (Stathera via `shift`/`slice`), lines 182-194 (junction via `.max()`/`.min()`) |
| 6   | Peak RSS during 200K-bar drain stays below 2.5G                                              | UNCERTAIN | Code path eliminates ~200K dict allocations; actual RSS measurement requires production deployment with memray — not verifiable programmatically     |

**Score:** 5/6 truths verified (1 deferred to human)

### Required Artifacts

| Artifact                                                                 | Expected                                              | Status   | Details                                                                                          |
| ------------------------------------------------------------------------ | ----------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------ |
| `crates/opendeviationbar-streaming/src/arrow_export.rs`                  | CompletedBar-to-RecordBatch with metadata columns     | VERIFIED | 189 lines; `completed_bars_to_record_batch()` + `append_metadata_columns()` + 3 unit tests       |
| `crates/opendeviationbar-streaming/src/stream_manager.rs` (drain method) | `drain_bars_arrow()` calling arrow_export             | VERIFIED | Lines 382-393; gated by `#[cfg(feature = "arrow")]`, drains via `engine.drain_bars()` + converts |
| `src/stream_manager_bindings.rs` (PyO3 binding)                          | `PyStreamManager.drain_bars_arrow()` with GIL release | VERIFIED | Lines 490-500; `#[cfg(feature = "arrow-export")]`; `py.detach()` releases GIL during Rust work   |
| `python/opendeviationbar/sidecar/arrow_flush.py`                         | Arrow-based `_sync_flush_rest_bars_arrow`             | VERIFIED | 259 lines; `pl.from_arrow()`, `df.filter()`, no `partition_by`, no `defaultdict`                 |
| `python/opendeviationbar/sidecar/midnight.py` (dispatch)                 | Dispatch + `_sync_flush_rest_bars_dict` fallback      | VERIFIED | Lines 492-509; dispatch at line 499; dict path preserved as `_sync_flush_rest_bars_dict` at 504  |

### Key Link Verification

| From                                           | To                                      | Via                                                    | Status | Details                                                                                                  |
| ---------------------------------------------- | --------------------------------------- | ------------------------------------------------------ | ------ | -------------------------------------------------------------------------------------------------------- |
| `midnight.py`                                  | `StreamManager.drain_bars_arrow()`      | `hasattr(engine, 'drain_bars_arrow')` check            | WIRED  | `midnight.py:499` dispatches to `arrow_flush._sync_flush_rest_bars_arrow(engine, ...)` when attr present |
| `midnight.py`                                  | `arrow_flush.py`                        | `from .arrow_flush import _sync_flush_rest_bars_arrow` | WIRED  | `midnight.py:11` — import confirmed, function importable (smoke test passed)                             |
| `arrow_flush.py`                               | `polars`                                | `pl.from_arrow()` zero-copy conversion                 | WIRED  | `arrow_flush.py:51` — `df: pl.DataFrame = pl.from_arrow(arrow_batch)` executes on every drain            |
| `arrow_export.rs`                              | `opendeviationbar-core/arrow_export.rs` | `opendeviationbar_vec_to_record_batch()`               | WIRED  | `arrow_export.rs:13-14` — imports core function; `streaming/Cargo.toml:49` wires feature                 |
| Root `Cargo.toml` `arrow-export` feature       | `opendeviationbar-streaming?/arrow`     | Weak dep syntax                                        | WIRED  | `Cargo.toml:182` — `opendeviationbar-streaming?/arrow` activates streaming arrow when both features on   |
| `crates/opendeviationbar-streaming/src/lib.rs` | `arrow_export` module                   | `pub mod arrow_export`                                 | WIRED  | `lib.rs:28` — module registered; `Cargo.toml:49` `arrow` feature enables it                              |

### Data-Flow Trace (Level 4)

| Artifact         | Data Variable | Source                                                | Produces Real Data                                                                                                                 | Status  |
| ---------------- | ------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `arrow_flush.py` | `df`          | `engine.drain_bars_arrow()` -> Rust ring buffer drain | Yes — drains live `CompletedBar` vec from `StreamManager`; returns empty RecordBatch when buffer is empty (verified by smoke test) | FLOWING |

**Empty-drain check:** Running `_sync_flush_rest_bars(sm, None)` on a fresh `StreamManager` returns 0 without error (smoke test confirmed). The `if raw_count == 0: return 0` early-exit path at `arrow_flush.py:54` is correct.

### Behavioral Spot-Checks

| Behavior                            | Command                                                                                                                                                                      | Result             | Status |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------ |
| Both flush paths importable         | `uv run --python 3.13 python -c "from opendeviationbar.sidecar.midnight import _sync_flush_rest_bars, _sync_flush_rest_bars_arrow, _sync_flush_rest_bars_dict; print('OK')"` | OK                 | PASS   |
| Existing sync_flush tests (9 tests) | `uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py -x -q`                                                                                                  | 9 passed, 0 failed | PASS   |
| RSS measurement (200K-bar drain)    | Requires live sidecar + bigblack                                                                                                                                             | Not runnable       | SKIP   |

### Requirements Coverage

| Requirement | Source Plan  | Description                                                                               | Status      | Evidence                                                                                                |
| ----------- | ------------ | ----------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------- |
| ARROW-01    | 23-01        | `StreamManager.drain_bars_arrow()` returns Arrow RecordBatch instead of `list[dict]`      | SATISFIED   | `stream_manager.rs:390-393`; `arrow_export.rs:30-49`; PyO3 binding `stream_manager_bindings.rs:490-499` |
| ARROW-02    | 23-01, 23-02 | `_sync_flush_rest_bars` consumes Arrow RecordBatch with zero-copy Polars slicing per pair | SATISFIED   | `midnight.py:499-501`; `arrow_flush.py:51,147-150`; `df.filter()` confirmed; no `partition_by`          |
| ARROW-03    | 23-02        | Peak sidecar RSS during drain stays below 2.5G for 200K-bar drain                         | NEEDS HUMAN | Code path verified correct (no dict allocations in Arrow path); production RSS measurement deferred     |

**Notes:**

- REQUIREMENTS.md marks ARROW-03 as "Complete" in the traceability table, but both Plan 01 SUMMARY and Plan 02 SUMMARY explicitly note "ARROW-03 (peak RSS measurement) requires production deployment with memray." This is a documentation inconsistency — ARROW-03's implementation prerequisite is complete, but its measurement gate has not been run.
- No orphaned requirements: all 3 ARROW-\* IDs claimed by plans and mapped to Phase 23 in REQUIREMENTS.md.

### Anti-Patterns Found

| File              | Line    | Pattern           | Severity | Impact                                                                   |
| ----------------- | ------- | ----------------- | -------- | ------------------------------------------------------------------------ | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `arrow_export.rs` | 102-104 | `bars.iter().map( | b        | (\*b).clone()).collect()` — clones all bar structs to call core function | Info | One allocation per drain for `opendeviationbar_vec_to_record_batch_borrowed`; comment documents this as negligible vs Arrow builder cost. Not a stub — future optimization noted in comment. |

No blockers or warnings found. The clone in `opendeviationbar_vec_to_record_batch_borrowed` is acknowledged in a comment and does not affect the Python-side zero-copy invariant.

### Human Verification Required

#### 1. ARROW-03 Peak RSS Measurement

**Test:** Deploy sidecar on bigblack with all 16+ symbols active. Allow `fill_from_rest` to process a full day's REST backfill (produces ~200K bars across all symbols). Monitor peak RSS via `psutil` log or `systemd` `MemoryCurrent` during `_sync_flush_rest_bars` execution. Alternatively, run memray on the sidecar startup sequence in a staging environment.

**Expected:** Peak sidecar RSS stays below 2.5G during the Arrow-path sync_flush (was ~2.6G on dict path). The phase goal claims reduction to ~0.9G.

**Why human:** Cannot measure production RSS without a live ClickHouse + Binance WebSocket environment. The code change eliminates ~200K `opendeviationbar_to_dict()` calls and the associated Python dict allocations, which is the correct mechanism — but the actual memory delta requires runtime measurement.

### Gaps Summary

No blocking gaps. All code artifacts exist, are substantive, and are fully wired. The Arrow FFI path is implemented end-to-end:

1. Rust `completed_bars_to_record_batch()` builds a RecordBatch with `_symbol`/`_threshold_decimal_bps` metadata columns
2. `StreamManager::drain_bars_arrow()` drains the ring buffer and converts in one step
3. `PyStreamManager.drain_bars_arrow()` returns `PyRecordBatch` with GIL released during Rust work
4. `midnight.py` dispatches to the Arrow path when the attribute is present, falling back to the dict path
5. `arrow_flush._sync_flush_rest_bars_arrow()` uses `pl.from_arrow()` (zero-copy), `df.filter()` (zero-copy per-pair slicing), Polars column aggregation for junction telemetry and Stathera diagnostics

The only open item is ARROW-03 runtime RSS measurement, which by design requires a production deployment. The code changes necessary to achieve the RSS reduction are confirmed in place.

---

_Verified: 2026-03-27T21:18:05Z_
_Verifier: Claude (gsd-verifier)_
