---
phase: 23-arrow-ffi-drain-bars
plan: 02
type: execute
wave: 2
depends_on: ["23-01"]
files_modified:
  - python/opendeviationbar/sidecar/midnight.py
autonomous: true
requirements:
  - ARROW-02
  - ARROW-03

must_haves:
  truths:
    - "_sync_flush_rest_bars uses drain_bars_arrow() when available, falling back to drain_bars()"
    - "Polars zero-copy filter replaces defaultdict(list) grouping for per-pair slicing"
    - "Junction telemetry (_junction_state) is populated from Polars column aggregation"
    - "Stathera diagnostic sort uses Polars sort, not Python sorted()"
    - "Cross-midnight filter works on Polars columns, not dict .get()"
    - "Peak RSS during 200K-bar drain is significantly lower than dict path"
  artifacts:
    - path: "python/opendeviationbar/sidecar/midnight.py"
      provides: "Arrow-based _sync_flush_rest_bars implementation"
      contains: "drain_bars_arrow"
  key_links:
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "StreamManager.drain_bars_arrow()"
      via: "hasattr(engine, 'drain_bars_arrow') check"
      pattern: "drain_bars_arrow"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "polars"
      via: "pl.from_arrow() zero-copy conversion"
      pattern: "pl\\.from_arrow"
---

<objective>
Rewrite `_sync_flush_rest_bars()` to consume Arrow RecordBatch from `drain_bars_arrow()` using Polars zero-copy filter/slice, eliminating dict-based grouping and reducing peak RSS from ~2.6G to ~0.9G.

Purpose: The Python-side consumer of the new Arrow FFI path — replaces `defaultdict(list)` grouping with `pl.filter()` zero-copy slicing.
Output: Updated `_sync_flush_rest_bars()` that uses Arrow when available, falls back to dict path for older builds.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/23-arrow-ffi-drain-bars/23-RESEARCH.md
@.planning/phases/23-arrow-ffi-drain-bars/23-01-SUMMARY.md

@python/opendeviationbar/sidecar/midnight.py
@src/stream_manager_bindings.rs
</context>

<interfaces>
<!-- Key types and contracts the executor needs -->

From src/stream_manager_bindings.rs (Plan 01 output):

```rust
// Returns PyRecordBatch with 36 columns:
// 34 ODB fields (open_time_us, close_time_us, open, high, low, close, volume, ...)
// + _symbol (Utf8)
// + _threshold (UInt32)
pub fn drain_bars_arrow(&self) -> PyResult<PyRecordBatch>
```

From python/opendeviationbar/clickhouse/bulk_operations.py:

```python
def store_bars_batch(
    self,
    symbol: str,
    threshold_decimal_bps: int,
    bars: pl.DataFrame,  # Already accepts Polars DataFrame
    version: str | None = None,
    ouroboros_mode: str | None = None,
    *,
    skip_dedup: bool = False,
    overlap_strategy: OverlapStrategy = "warn",
    _skip_lock: bool = False,
) -> int:
```

From python/opendeviationbar/sidecar/clickhouse_write.py:

```python
def _write_batch_with_retry(
    symbol: str,
    threshold: int,
    bar_dicts: list[dict[str, Any]],  # NOTE: expects list[dict], not DataFrame
    *,
    cache: object | None = None,
) -> bool:
```

From python/opendeviationbar/sidecar/http_server.py:

```python
_junction_state: dict[str, dict] = {}
# Used as: _junction_state[symbol]["rest_last_tid"] = int
#           _junction_state[symbol]["rest_first_tid"] = int
```

Arrow schema column names (from opendeviationbar_schema + metadata):

- Timestamps: "open_time_us", "close_time_us" (Int64, microseconds)
- Trade IDs: "first_agg_trade_id", "last_agg_trade_id" (Int64)
- Metadata: "\_symbol" (Utf8), "\_threshold" (UInt32)
  </interfaces>

<tasks>

<task type="auto">
  <name>Task 1: Rewrite _sync_flush_rest_bars to use Arrow path with dict fallback</name>
  <files>python/opendeviationbar/sidecar/midnight.py</files>
  <read_first>
    - python/opendeviationbar/sidecar/midnight.py (full _sync_flush_rest_bars function, lines 491-726)
    - python/opendeviationbar/sidecar/clickhouse_write.py (_write_batch_with_retry signature — expects list[dict])
    - python/opendeviationbar/sidecar/http_server.py (_junction_state dict structure)
    - .planning/phases/23-arrow-ffi-drain-bars/23-01-SUMMARY.md (drain_bars_arrow schema details)
  </read_first>
  <action>
Rewrite `_sync_flush_rest_bars()` in `python/opendeviationbar/sidecar/midnight.py` to use Arrow RecordBatch when `drain_bars_arrow` is available, with full fallback to the existing dict path.

**Structure:** Replace the current function body with a dispatch:

```python
def _sync_flush_rest_bars(engine, valid_pairs, *, symbol_modes=None):
    if hasattr(engine, "drain_bars_arrow"):
        return _sync_flush_rest_bars_arrow(engine, valid_pairs, symbol_modes=symbol_modes)
    return _sync_flush_rest_bars_dict(engine, valid_pairs, symbol_modes=symbol_modes)
```

**Keep the existing dict-based implementation** as `_sync_flush_rest_bars_dict()` (rename, no logic changes). This is the fallback for older builds.

**Create `_sync_flush_rest_bars_arrow()`** with the following logic:

1. **Drain:** `arrow_batch = engine.drain_bars_arrow()` — returns PyArrow RecordBatch.
2. **Convert to Polars (zero-copy):** `df = pl.from_arrow(arrow_batch)` — no data copy.
3. **Early exit:** If `df.is_empty()`, log and return 0.
4. **Log raw count:** `raw_count = len(df)` (same diagnostic as dict path).

5. **Valid-pairs filter:**
   If `valid_pairs is not None`:
   - Build a Polars DataFrame from valid_pairs: `vp_df = pl.DataFrame({"_symbol": [s for s,t in valid_pairs], "_threshold": [t for s,t in valid_pairs]})`.
   - Use `df = df.join(vp_df, on=["_symbol", "_threshold"], how="semi")` to keep only valid pairs (semi-join = filter, zero-copy friendly).
   - Log rejected count: `filtered_invalid_pairs = raw_count - len(df) - cross_midnight_count`.

6. **Cross-midnight filter (Day-mode only):**

   ```python
   day_us = 86_400_000_000
   # Add day-number columns for comparison
   df = df.with_columns([
       (pl.col("open_time_us") // day_us).alias("_open_day"),
       (pl.col("close_time_us") // day_us).alias("_close_day"),
   ])
   # Determine per-row mode
   if symbol_modes:
       mode_df = pl.DataFrame({
           "_symbol": list(symbol_modes.keys()),
           "_mode": list(symbol_modes.values()),
       })
       df = df.join(mode_df, on="_symbol", how="left").with_columns(
           pl.col("_mode").fill_null("day")
       )
   else:
       df = df.with_columns(pl.lit("day").alias("_mode"))

   # Filter: keep Aion bars OR bars that don't cross midnight
   cross_midnight_mask = (
       (pl.col("_mode") == "day")
       & (pl.col("open_time_us") > 0)
       & (pl.col("close_time_us") > 0)
       & (pl.col("_open_day") != pl.col("_close_day"))
   )
   cross_midnight_count = df.filter(cross_midnight_mask).height
   if cross_midnight_count > 0:
       logger.warning("sync_flush: %d cross-midnight bars discarded (Arrow path)", cross_midnight_count)
   df = df.filter(~cross_midnight_mask)

   # Drop temp columns
   df = df.drop(["_open_day", "_close_day", "_mode"])
   ```

7. **Log drained count:** `drained = len(df)`.

8. **Stathera continuity diagnostic (per-pair):**
   Get unique pairs: `pairs = df.select("_symbol", "_threshold").unique().iter_rows()`.
   For each `(sym, thr)`:

   ```python
   pair_df = df.filter(
       (pl.col("_symbol") == sym) & (pl.col("_threshold") == thr)
   ).sort("first_agg_trade_id")
   if pair_df.height >= 2:
       prev_last = pair_df["last_agg_trade_id"][:-1].to_list()
       curr_first = pair_df["first_agg_trade_id"][1:].to_list()
       for i, (pl_val, cf_val) in enumerate(zip(prev_last, curr_first)):
           if cf_val <= pl_val:
               logger.warning("sync_flush: stathera non-monotonic ...")
           elif cf_val != pl_val + 1:
               logger.warning("sync_flush: stathera gap ...")
   ```

9. **Write per-pair to ClickHouse:**

   ```python
   from opendeviationbar import __version__
   from opendeviationbar.clickhouse.cache import OpenDeviationBarCache
   from opendeviationbar.symbol_registry import get_symbol_entries as _get_entries

   source_version = f"{__version__}+sidecar"
   written = 0
   pairs = df.select("_symbol", "_threshold").unique().iter_rows()

   for sym, thr in pairs:
       pair_df = df.filter(
           (pl.col("_symbol") == sym) & (pl.col("_threshold") == thr)
       ).drop("_symbol", "_threshold")

       try:
           # Per-symbol ouroboros_mode from registry
           try:
               _entries = _get_entries()
               _e = _entries.get(sym)
               _mode = _e.ouroboros_mode.value if _e else "day"
           except Exception:
               _mode = "day"

           with OpenDeviationBarCache() as cache:
               inserted = cache.store_bars_batch(
                   symbol=sym,
                   threshold_decimal_bps=thr,
                   bars=pair_df,
                   version=source_version,
                   overlap_strategy="off",
                   ouroboros_mode=_mode,
                   _skip_lock=False,
               )
           written += inserted
       except Exception:
           logger.info(
               "sync_flush: store_bars_batch failed for %s@%d (Arrow path), "
               "falling back to _write_batch_with_retry (non-fatal)",
               sym, thr, exc_info=True,
           )
           # Fallback: convert pair_df to list[dict] for retry path
           bar_dicts = pair_df.to_dicts()
           if _write_batch_with_retry(sym, thr, bar_dicts):
               written += len(bar_dicts)
   ```

10. **Junction telemetry (Polars aggregation):**

    ```python
    for sym in df["_symbol"].unique().to_list():
        sym_df = df.filter(pl.col("_symbol") == sym)
        if sym not in _junction_state:
            _junction_state[sym] = {}
        last_tid = sym_df["last_agg_trade_id"].max()
        if last_tid and last_tid > 0:
            _junction_state[sym]["rest_last_tid"] = last_tid
        min_tid = sym_df["first_agg_trade_id"].min()
        if min_tid and min_tid > 0:
            _junction_state[sym]["rest_first_tid"] = min(
                _junction_state[sym].get("rest_first_tid", min_tid),
                min_tid,
            )
    ```

11. **Log final diagnostics** (same format as dict path):

    ```python
    pairs_with_bars = df.group_by("_symbol", "_threshold").len().sort("len", descending=True)
    logger.info("sync_flush: %d/%d REST bars written to CH (Arrow path, %d pairs)", written, drained, pairs_with_bars.height)
    ```

12. Return `written`.

**Important implementation notes:**

- `pl.from_arrow()` does zero-copy conversion from PyArrow RecordBatch — this is the key memory win.
- `df.filter()` returns a view (zero-copy slice) on Arrow-backed data — no data duplication per pair.
- The fallback to `_write_batch_with_retry` converts `pair_df.to_dicts()` only for the specific failing pair, not all bars.
- The `_threshold` column is `UInt32` in Arrow but `int` in Python — Polars handles this transparently.
- Do NOT use `df.partition_by()` — it creates copies. Use `df.filter()` for zero-copy.
  </action>
  <verify>
  <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 python -c "from opendeviationbar.sidecar.midnight import \_sync_flush_rest_bars, \_sync_flush_rest_bars_arrow,\_sync_flush_rest_bars_dict; print('Both paths importable: OK')"</automated>
  </verify>
  <acceptance_criteria>
  - `midnight.py` contains `def _sync_flush_rest_bars_arrow(`
  - `midnight.py` contains `def _sync_flush_rest_bars_dict(`
  - `midnight.py` contains `hasattr(engine, "drain_bars_arrow")`
  - `midnight.py` contains `pl.from_arrow(` (zero-copy conversion)
  - `midnight.py` contains `df.filter(` (zero-copy per-pair slicing)
  - `midnight.py` does NOT contain `df.partition_by(` (anti-pattern)
  - `midnight.py` does NOT contain `defaultdict(list)` in the Arrow path
  - Junction telemetry uses `["last_agg_trade_id"].max()` (Polars aggregation), not `max(b.get(...) for b in bars)`
  - Cross-midnight filter uses `pl.col("open_time_us")`, not `bar.get("open_time_us")`
  - Fallback `_write_batch_with_retry` call uses `pair_df.to_dicts()` for conversion
  - Both `_sync_flush_rest_bars_arrow` and `_sync_flush_rest_bars_dict` are importable
    </acceptance_criteria>
    <done>\_sync_flush_rest_bars uses Arrow path when drain_bars_arrow() is available, with Polars zero-copy filter for per-pair slicing, dict fallback preserved, junction telemetry and Stathera diagnostic updated to Polars column operations</done>
    </task>

<task type="auto">
  <name>Task 2: End-to-end verification — build, import, and validate both paths</name>
  <files>python/opendeviationbar/sidecar/midnight.py</files>
  <read_first>
    - python/opendeviationbar/sidecar/midnight.py (the rewritten function)
  </read_first>
  <action>
Run `maturin develop --features "streaming,arrow-export,data-providers"` to rebuild the extension.

Then run the existing sync_flush tests to confirm no regressions:

```bash
uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py -x -v
```

Also run a quick smoke test verifying the Arrow path dispatch works:

```python
uv run --python 3.13 python -c "
from opendeviationbar.sidecar.midnight import _sync_flush_rest_bars
from opendeviationbar._core import StreamManager

# StreamManager has drain_bars_arrow — should dispatch to Arrow path
sm = StreamManager(['BTCUSDT'], [250])
result = _sync_flush_rest_bars(sm, None)
print(f'Empty drain result: {result}')
assert result == 0, f'Expected 0 bars from empty drain, got {result}'
print('Arrow path dispatch: OK')
"
```

If any test fails, fix the issue in midnight.py and re-verify.

Run `mise run check-full` to ensure full suite passes (Rust + Python lint + tests).
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py -x -v 2>&1 | tail -10</automated>
</verify>
<acceptance_criteria> - `maturin develop` succeeds - `pytest tests/test_sync_flush_atomic_replace.py` passes (all existing tests green) - Empty drain via Arrow path returns 0 without error - No import errors when loading the sidecar midnight module
</acceptance_criteria>
<done>Both Arrow and dict paths work correctly, existing tests pass, empty drain handled, no regressions</done>
</task>

</tasks>

<verification>
1. `_sync_flush_rest_bars` dispatches to Arrow path when `drain_bars_arrow` is available
2. Arrow path uses `pl.from_arrow()` for zero-copy conversion
3. Per-pair slicing uses `df.filter()` (zero-copy), NOT `partition_by()` (copies)
4. Junction telemetry populated via Polars column aggregation
5. Stathera diagnostic uses Polars sort, not Python `sorted()`
6. Cross-midnight filter operates on Polars columns
7. Existing dict fallback path preserved as `_sync_flush_rest_bars_dict`
8. `_write_batch_with_retry` fallback converts only the failing pair's DataFrame to dicts
9. All existing tests pass
</verification>

<success_criteria>

- \_sync_flush_rest_bars uses drain_bars_arrow() when available (Arrow path)
- Polars zero-copy filter replaces defaultdict(list) grouping
- All dict-based operations (junction telemetry, Stathera, cross-midnight) converted to Polars column ops
- Existing sync_flush tests pass without modification
- Dict fallback path preserved for backward compatibility
- Peak RSS during 200K-bar drain expected to drop from ~2.6G to ~0.9G (ARROW-03 — full verification requires production-scale data)
  </success_criteria>

<output>
After completion, create `.planning/phases/23-arrow-ffi-drain-bars/23-02-SUMMARY.md`
</output>
