# Phase 23: Arrow FFI drain_bars - Research

**Researched:** 2026-03-27
**Domain:** PyO3 Arrow FFI, columnar data transfer, memory efficiency
**Confidence:** HIGH

## Summary

Phase 23 replaces the `list[dict]` return path from `StreamManager.drain_bars()` with an Arrow RecordBatch that includes `_symbol` and `_threshold` metadata columns. This eliminates ~200K Python dict allocations during the sidecar's `sync_flush` path, reducing peak RSS from ~2.6G to ~0.9G. The downstream `_sync_flush_rest_bars()` then uses Polars zero-copy `pl.from_arrow()` to filter/slice by `(symbol, threshold)` pairs instead of building `defaultdict(list)` of dicts.

All infrastructure already exists: `opendeviationbar-core` has `opendeviationbar_vec_to_record_batch()` converting `Vec<OpenDeviationBar>` to Arrow RecordBatch, `pyo3-arrow` 0.17 provides `PyRecordBatch` for zero-copy FFI, and `_sync_flush_rest_bars()` already converts `list[dict]` to `pl.DataFrame` per-pair before writing to ClickHouse. The new path skips the dict materialization entirely.

**Primary recommendation:** Add a `drain_bars_arrow()` method to `PyStreamManager` in Rust that converts `Vec<CompletedBar>` to a single Arrow RecordBatch with two extra columns (`_symbol: Utf8`, `_threshold: UInt32`), then rewrite `_sync_flush_rest_bars()` to accept this RecordBatch and use Polars `filter()` for per-pair slicing.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all implementation choices at Claude's discretion (infrastructure phase).

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

### Deferred Ideas (OUT OF SCOPE)

None.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID       | Description                                                                                                      | Research Support                                                                                                                         |
| -------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| ARROW-01 | `StreamManager.drain_bars_arrow()` returns Arrow RecordBatch with all bar fields + \_symbol/\_threshold metadata | Extend `opendeviationbar_vec_to_record_batch()` with two extra columns; use existing `pyo3-arrow` 0.17 `PyRecordBatch` for zero-copy FFI |
| ARROW-02 | `_sync_flush_rest_bars` groups RecordBatch by (symbol, threshold) using Polars zero-copy filter/slice            | `pl.from_arrow()` zero-copy from PyRecordBatch; `df.filter()` on string/int columns for O(1) slicing                                     |
| ARROW-03 | Peak sidecar RSS during 200K-bar drain stays below 2.5G                                                          | Eliminates ~200K dict allocations (~1.7G); Arrow columnar uses ~0.1G for same data                                                       |

</phase_requirements>

## Standard Stack

### Core

| Library      | Version           | Purpose                                                     | Why Standard                                                          |
| ------------ | ----------------- | ----------------------------------------------------------- | --------------------------------------------------------------------- |
| arrow-array  | 58                | Arrow columnar arrays (builders, typed arrays)              | Already in workspace, used by `opendeviationbar-core/arrow_export.rs` |
| arrow-schema | 58                | Arrow schema/field/datatype definitions                     | Already in workspace                                                  |
| pyo3-arrow   | 0.17              | `PyRecordBatch` for zero-copy Rust-to-Python Arrow transfer | Already in workspace, used by `arrow_bindings.rs`                     |
| polars       | (project version) | Zero-copy `from_arrow()`, filter/slice operations           | Already used in `_sync_flush_rest_bars` for DataFrame creation        |

### Alternatives Considered

| Instead of        | Could Use             | Tradeoff                                                                                       |
| ----------------- | --------------------- | ---------------------------------------------------------------------------------------------- |
| Arrow RecordBatch | PyArrow Table         | RecordBatch is simpler, single-batch (no chunking needed), already used everywhere in codebase |
| Polars filter     | Arrow compute kernels | Polars already imported; filter is zero-copy on Arrow-backed data                              |

## Architecture Patterns

### Current Flow (list[dict] -- high memory)

```
Rust drain_bars() -> Vec<CompletedBar>
  -> PyO3 loop: opendeviationbar_to_dict() per bar (48 PyString allocs each)
  -> Python list[dict] (~200K dicts, ~1.7G)
  -> _sync_flush_rest_bars: iterate dicts, pop _symbol/_threshold
  -> defaultdict(list) grouping
  -> pl.DataFrame(bar_dicts) per (symbol, threshold)
  -> store_bars_batch()
```

### New Flow (Arrow RecordBatch -- low memory)

```
Rust drain_bars() -> Vec<CompletedBar>
  -> completed_bars_to_record_batch() (single-pass columnar build, ~0.1G)
  -> PyRecordBatch (zero-copy FFI)
  -> _sync_flush_rest_bars: pl.from_arrow() (zero-copy)
  -> df.filter() per (symbol, threshold) pair (zero-copy slice)
  -> store_bars_batch() (already accepts pl.DataFrame)
```

### Pattern 1: Extended Arrow Schema with Metadata Columns

**What:** Add `_symbol` (Utf8) and `_threshold` (UInt32) columns to the existing `opendeviationbar_schema()`.
**When to use:** When `CompletedBar` metadata needs to travel with bar data across FFI.
**Implementation approach:**

Create a new function `completed_bars_to_record_batch(bars: &[CompletedBar]) -> RecordBatch` in `opendeviationbar-core/src/arrow_export.rs` (or a new function in `src/stream_manager_bindings.rs` at the PyO3 layer). This function:

1. Reuses existing builder logic from `opendeviationbar_vec_to_record_batch()`
2. Adds two extra builders: `StringBuilder` for `_symbol`, `UInt32Builder` for `_threshold`
3. Iterates `CompletedBar` instead of `OpenDeviationBar`

**Design choice -- where to put the conversion:**

- **Option A (recommended):** New function in `src/stream_manager_bindings.rs` at PyO3 layer. Calls existing `opendeviationbar_vec_to_record_batch()` then appends columns, OR builds from scratch. Keeps `opendeviationbar-core` free of streaming concerns.
- **Option B:** New function in `opendeviationbar-core/src/arrow_export.rs`. Requires `opendeviationbar-core` to know about `CompletedBar`, creating a circular dependency (CompletedBar is in `opendeviationbar-streaming`).

Option A is clearly correct -- the conversion belongs at the PyO3 binding layer where `CompletedBar` metadata is consumed.

```rust
// In src/stream_manager_bindings.rs (new method on PyStreamManager)
pub fn drain_bars_arrow(&self) -> PyResult<PyRecordBatch> {
    let manager = self.manager.as_ref()
        .ok_or_else(|| PyRuntimeError::new_err("Manager not available"))?;
    let bars = manager.drain_bars();
    if bars.is_empty() {
        // Return empty RecordBatch with extended schema
        return Ok(PyRecordBatch::new(empty_completed_bar_batch()));
    }
    let batch = completed_bars_to_record_batch(&bars);
    Ok(PyRecordBatch::new(batch))
}
```

### Pattern 2: Polars Zero-Copy Filter in Python

**What:** Replace dict-based grouping with Polars column-based filtering.
**When to use:** After receiving Arrow RecordBatch with \_symbol/\_threshold columns.

```python
# Zero-copy from Arrow
df = pl.from_arrow(engine.drain_bars_arrow())

# Get unique (symbol, threshold) pairs
pairs = df.select("_symbol", "_threshold").unique()

# Filter per pair (zero-copy slice, no data copy)
for row in pairs.iter_rows():
    sym, thr = row
    pair_df = df.filter(
        (pl.col("_symbol") == sym) & (pl.col("_threshold") == thr)
    ).drop("_symbol", "_threshold")
    # pair_df is already a pl.DataFrame -- pass directly to store_bars_batch()
```

### Anti-Patterns to Avoid

- **Building RecordBatch column-by-column via Arrow compute after initial build:** Use single-pass builders instead -- one iteration over `Vec<CompletedBar>` populating all columns.
- **Converting RecordBatch back to list[dict] in Python:** Defeats the purpose; always stay in columnar form.
- **Using `df.partition_by()` for grouping:** Creates new DataFrames (copies). `df.filter()` on Arrow-backed data is zero-copy (returns view).

## Don't Hand-Roll

| Problem                   | Don't Build                          | Use Instead                                                               | Why                                                           |
| ------------------------- | ------------------------------------ | ------------------------------------------------------------------------- | ------------------------------------------------------------- |
| Arrow schema for ODB bars | Custom schema definition             | Extend existing `opendeviationbar_schema()`                               | Schema already has 34 fields; just add 2 more                 |
| RecordBatch construction  | Manual column stitching              | Single-pass builder pattern from `opendeviationbar_vec_to_record_batch()` | Proven pattern, already handles FixedPoint, i128 volume, etc. |
| Polars groupby            | `defaultdict(list)` + dict iteration | `df.filter()` on \_symbol/\_threshold columns                             | Zero-copy, no Python object creation                          |
| Arrow FFI transfer        | Custom serialization                 | `pyo3-arrow` `PyRecordBatch`                                              | Already used in 10+ places in codebase                        |

## Common Pitfalls

### Pitfall 1: Arrow Schema Mismatch Between Empty and Non-Empty Batches

**What goes wrong:** Empty RecordBatch has different schema than populated one (missing \_symbol/\_threshold columns).
**Why it happens:** Different code paths for empty vs non-empty returns.
**How to avoid:** Use a single `completed_bar_schema()` function for both empty and populated batches.
**Warning signs:** `SchemaError` when calling `pl.from_arrow()` on empty result.

### Pitfall 2: Polars `partition_by` vs `filter` Memory Semantics

**What goes wrong:** Using `partition_by()` creates N separate DataFrames with copied data, losing zero-copy benefit.
**Why it happens:** `partition_by` is designed for independent DataFrames, not views.
**How to avoid:** Use `df.filter()` which returns a view over the same Arrow buffers.
**Warning signs:** RSS still high after switching to Arrow path.

### Pitfall 3: FixedPoint Volume Conversion Inconsistency

**What goes wrong:** Volume columns use i128 FixedPoint internally but Arrow uses f64. The scale factor (100_000_000.0) must match between `opendeviationbar_vec_to_record_batch()` and the new function.
**Why it happens:** Copy-paste errors when replicating builder logic.
**How to avoid:** Extract the conversion into the same function or call the existing one.
**Warning signs:** Volume values off by factor of 100M.

### Pitfall 4: Cross-Midnight Filter Needs open_time_us/close_time_us Column Names

**What goes wrong:** The existing Arrow schema uses `open_time_us` and `close_time_us` column names, but `_sync_flush_rest_bars()` accesses `bar.get("open_time_us")` and `bar.get("close_time_us")`. The Arrow schema already uses these names, so this is compatible.
**Why it happens:** Historical naming differences between dict path and Arrow path.
**How to avoid:** Verify column names match between Arrow schema and Python access patterns.
**Warning signs:** Cross-midnight filter always passes (columns not found, defaults to 0).

### Pitfall 5: Stathera Diagnostics Need Sorted Access

**What goes wrong:** The Stathera continuity check in `_sync_flush_rest_bars` sorts bars by `first_agg_trade_id`. With Polars this needs `df.sort("first_agg_trade_id")`.
**Why it happens:** Dict-based sorting uses `sorted(batch_bars, key=lambda b: b.get(...))`.
**How to avoid:** Use `pair_df.sort("first_agg_trade_id")` for Polars equivalent.
**Warning signs:** Stathera gap warnings that don't match reality.

### Pitfall 6: \_junction_state Telemetry Uses max/min on bar dicts

**What goes wrong:** After the main write loop, `_sync_flush_rest_bars()` populates `_junction_state` with `max()` and `min()` over bar dicts' trade IDs. With Polars, this needs column aggregation.
**Why it happens:** Junction telemetry was added for the dict path.
**How to avoid:** Use `pair_df["last_agg_trade_id"].max()` and `pair_df["first_agg_trade_id"].min()`.
**Warning signs:** Junction telemetry missing or wrong values on /status endpoint.

## Code Examples

### Rust: drain_bars_arrow() Implementation Pattern

```rust
// Source: Derived from existing opendeviationbar_vec_to_record_batch() in arrow_export.rs
// New method on PyStreamManager in src/stream_manager_bindings.rs

pub fn drain_bars_arrow(&self) -> PyResult<PyRecordBatch> {
    let manager = self.manager.as_ref()
        .ok_or_else(|| PyRuntimeError::new_err("Manager not available"))?;
    let bars = manager.drain_bars();

    // Build extended schema: all ODB fields + _symbol + _threshold
    let base_bars: Vec<&OpenDeviationBar> = bars.iter().map(|b| &b.bar).collect();
    // ... build RecordBatch with extra columns

    Ok(PyRecordBatch::new(batch))
}
```

### Python: Polars Zero-Copy Grouping Pattern

```python
# Source: Existing pattern in processors/api.py line 189
import polars as pl

# Zero-copy from Arrow RecordBatch
arrow_batch = engine.drain_bars_arrow()
df = pl.from_arrow(arrow_batch)

# Get unique pairs
pairs = df.select("_symbol", "_threshold").unique().iter_rows()

for sym, thr in pairs:
    pair_df = df.filter(
        (pl.col("_symbol") == sym) & (pl.col("_threshold") == thr)
    ).drop("_symbol", "_threshold")

    # pair_df is already pl.DataFrame -- direct to store_bars_batch
    cache.store_bars_batch(
        symbol=sym,
        threshold_decimal_bps=thr,
        bars=pair_df,
        version=source_version,
    )
```

## Key Implementation Details

### RecordBatch Construction Strategy

Two approaches for building the extended RecordBatch:

**Approach A (Recommended): Single-pass with all builders.**
Build all 36 columns (34 existing + `_symbol` + `_threshold`) in one iteration over `Vec<CompletedBar>`. Copy the builder pattern from `opendeviationbar_vec_to_record_batch()` and add two extra builders. This duplicates ~80 lines of builder code but is the most cache-friendly approach.

**Approach B: Build base batch, then append columns.**
Call `opendeviationbar_vec_to_record_batch()` for the 34 base columns, then construct `_symbol` and `_threshold` arrays separately, and create a new RecordBatch by appending columns to the schema. This avoids code duplication but requires a second pass and potentially a second allocation.

Approach A is preferred for this performance-critical path.

### Memory Math

Current path (200K bars as `list[dict]`):

- Each dict: ~48 keys x (PyString key + PyObject value) = ~3-4 KB per bar
- 200K bars x 4KB = ~800 MB just for dicts
- Plus Python list overhead, plus intermediate copies = ~1.7G peak

Arrow path (200K bars as RecordBatch):

- 36 columns x 200K rows x 8 bytes avg = ~57 MB columnar data
- Plus string column for \_symbol: 200K x ~8 bytes = ~1.6 MB
- Total: ~60 MB
- After `del bars` in Rust (Vec<CompletedBar> freed): ~60 MB in Python

Savings: ~1.6G peak RSS reduction.

### Backward Compatibility

The existing `drain_bars()` method (returning `list[dict]`) MUST be preserved. `drain_bars_arrow()` is a new method. The `_sync_flush_rest_bars()` function should use `drain_bars_arrow()` when available, falling back to `drain_bars()` for older builds (same pattern as the existing `hasattr(engine, "drain_bars")` fallback).

### Test Strategy

1. **Unit test (Rust):** Verify `completed_bars_to_record_batch()` produces correct schema with \_symbol/\_threshold columns.
2. **Unit test (Python):** Mock engine returning Arrow RecordBatch, verify `_sync_flush_rest_bars()` correctly groups and writes.
3. **Integration test:** Create StreamManager, produce bars via fill_from_rest, drain_bars_arrow(), verify RecordBatch shape and column types.
4. **Memory test:** Use `tracemalloc` or `/proc/self/status` to verify RSS stays under 2.5G during 200K-bar drain (may need production-like data volume).

## Validation Architecture

### Test Framework

| Property           | Value                                               |
| ------------------ | --------------------------------------------------- |
| Framework          | pytest + cargo nextest                              |
| Config file        | `pyproject.toml` (pytest section) + `Cargo.toml`    |
| Quick run command  | `pytest tests/test_sync_flush_atomic_replace.py -x` |
| Full suite command | `mise run check-full`                               |

### Phase Requirements -> Test Map

| Req ID   | Behavior                                                       | Test Type   | Automated Command                                                                  | File Exists? |
| -------- | -------------------------------------------------------------- | ----------- | ---------------------------------------------------------------------------------- | ------------ |
| ARROW-01 | drain_bars_arrow returns RecordBatch with \_symbol/\_threshold | unit        | `pytest tests/test_drain_bars_arrow.py::test_returns_recordbatch_with_metadata -x` | Wave 0       |
| ARROW-01 | Empty drain returns empty RecordBatch with correct schema      | unit        | `pytest tests/test_drain_bars_arrow.py::test_empty_drain_arrow -x`                 | Wave 0       |
| ARROW-02 | \_sync_flush_rest_bars groups by Polars filter                 | unit        | `pytest tests/test_sync_flush_atomic_replace.py::TestSyncFlushArrow -x`            | Wave 0       |
| ARROW-03 | RSS stays under 2.5G                                           | manual-only | Requires production-scale data (200K bars from 16-symbol fill_from_rest)           | N/A          |

### Sampling Rate

- **Per task commit:** `pytest tests/test_drain_bars_arrow.py tests/test_sync_flush_atomic_replace.py -x`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_drain_bars_arrow.py` -- covers ARROW-01 (RecordBatch schema, empty case, metadata columns)
- [ ] Extend `tests/test_sync_flush_atomic_replace.py` with `TestSyncFlushArrow` class -- covers ARROW-02

## Sources

### Primary (HIGH confidence)

- **Codebase inspection** -- `src/stream_manager_bindings.rs` (drain_bars PyO3 binding, lines 459-479)
- **Codebase inspection** -- `crates/opendeviationbar-core/src/arrow_export.rs` (opendeviationbar_vec_to_record_batch, schema definition)
- **Codebase inspection** -- `python/opendeviationbar/sidecar/midnight.py` (\_sync_flush_rest_bars, lines 491-726)
- **Codebase inspection** -- `crates/opendeviationbar-streaming/src/live_engine/types.rs` (CompletedBar struct)
- **Cargo.toml** -- arrow-array 58, arrow-schema 58, pyo3-arrow 0.17 (workspace versions)

### Secondary (MEDIUM confidence)

- pyo3-arrow PyRecordBatch usage pattern verified across 10+ call sites in `src/core_bindings.rs` and `src/arrow_bindings.rs`

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- all libraries already in workspace, versions verified from Cargo.toml
- Architecture: HIGH -- pattern directly extends existing codebase patterns (opendeviationbar_vec_to_record_batch + pyo3-arrow)
- Pitfalls: HIGH -- identified from direct code reading of \_sync_flush_rest_bars and its dict-based access patterns

**Research date:** 2026-03-27
**Valid until:** 2026-04-27 (stable -- internal infrastructure, no external API changes)
