---
phase: 24-hot-path-allocation-reduction
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar/__init__.py
  - python/opendeviationbar/sidecar/clickhouse_write.py
  - python/opendeviationbar/sidecar/arrow_flush.py
  - python/opendeviationbar/sidecar/recovery.py
  - tests/test_sidecar_alloc.py
autonomous: true
requirements: [ALLOC-01, ALLOC-02, ALLOC-03]

must_haves:
  truths:
    - "Single OpenDeviationBarCache() instance created in _create_engine, reused by run_sidecar main loop"
    - "_write_bars_batch_to_clickhouse accepts pl.DataFrame directly without converting to dicts first"
    - "Per-flush ouroboros_mode lookup eliminated — mode resolved from pre-computed symbol_modes dict"
    - "arrow_flush.py uses shared cache and pre-computed symbol_modes instead of per-pair cache + registry lookup"
  artifacts:
    - path: "python/opendeviationbar/sidecar/__init__.py"
      provides: "Single cache instance in _create_engine, 5-tuple return, symbol_modes threaded to _flush_all_batches"
      contains: "return engine, loaded, min_tid_guard, symbol_modes, ch_cache"
    - path: "python/opendeviationbar/sidecar/clickhouse_write.py"
      provides: "Dual-type bars parameter, ouroboros_mode parameter, cache parameter"
      contains: "bars: pl.DataFrame | list[dict]"
    - path: "python/opendeviationbar/sidecar/arrow_flush.py"
      provides: "Shared cache and symbol_modes parameters, no per-pair OpenDeviationBarCache"
      contains: "cache: object | None = None"
    - path: "python/opendeviationbar/sidecar/recovery.py"
      provides: "5-tuple destructuring from _create_engine"
      contains: "new_engine, _, new_min_tid_guard, new_symbol_modes, new_ch_cache"
    - path: "tests/test_sidecar_alloc.py"
      provides: "Unit tests for ALLOC-01, ALLOC-02, ALLOC-03"
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/clickhouse_write.py"
      via: "_flush_all_batches passes ch_cache and ouroboros_mode to _write_batch_with_retry"
      pattern: "_write_batch_with_retry.*cache=ch_cache.*ouroboros_mode="
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/arrow_flush.py"
      via: "_sync_flush_rest_bars_arrow receives cache and symbol_modes"
      pattern: "_sync_flush_rest_bars_arrow.*cache="
---

<objective>
Eliminate redundant allocations in sidecar startup and per-flush write paths: single CH cache instance, dual-type write function, and pre-computed ouroboros_mode threading.

Purpose: Reduce sidecar memory churn and connection overhead — 3 separate OpenDeviationBarCache instantiations in \_create_engine become 1, per-flush dict→DataFrame conversion is skippable, and per-flush registry lookups are eliminated.

Output: Modified sidecar files with shared cache instance, dual-type write function, and pre-computed symbol_modes threading. Unit tests covering all three optimizations.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/24-hot-path-allocation-reduction/24-RESEARCH.md

<interfaces>
<!-- Key types and contracts the executor needs. -->

From python/opendeviationbar/sidecar/**init**.py:

```python
def _create_engine(
    config: SidecarConfig,
    trace_id: str,
    gap_fill_results: dict | None,
    valid_pairs: set[tuple[str, int]] | None = None,
) -> tuple[object, int, dict[tuple[str, int], int], dict[str, str]]:
    # Returns (engine, checkpoints_loaded, min_tid_guard, symbol_modes)
    # Currently creates 3 OpenDeviationBarCache() context managers at lines 296, 361, 512
    # run_sidecar() creates a 4th at line 723
```

Call sites for \_create_engine (BOTH must be updated for 5-tuple):

1. `__init__.py:783` — `engine, checkpoints_loaded, min_tid_guard, symbol_modes = _create_engine(...)`
2. `recovery.py:54` — `new_engine, _, new_min_tid_guard, new_symbol_modes = _create_engine(...)`

From python/opendeviationbar/sidecar/clickhouse_write.py:

```python
def _write_bars_batch_to_clickhouse(
    symbol: str,
    threshold: int,
    bar_dicts: list[dict],  # ← rename to 'bars', accept pl.DataFrame | list[dict]
    *,
    cache: object | None = None,
) -> None:

def _write_batch_with_retry(
    symbol: str,
    threshold: int,
    bar_dicts: list[dict[str, Any]],  # ← same rename
    *,
    cache: object | None = None,
) -> bool:
```

From python/opendeviationbar/sidecar/arrow_flush.py:

```python
def _sync_flush_rest_bars_arrow(
    engine: object,
    valid_pairs: set[tuple[str, int]] | None,
    *,
    symbol_modes: dict[str, str] | None = None,
    # Currently creates OpenDeviationBarCache() per (sym, thr) pair at line 157
    # Currently calls get_symbol_entries() at line 134
) -> int:
```

From python/opendeviationbar/sidecar/midnight.py:

```python
def _sync_flush_rest_bars(
    engine: object,
    valid_pairs: set[tuple[str, int]] | None,
    *,
    symbol_modes: dict[str, str] | None = None,
) -> int:
    # Dispatches to _sync_flush_rest_bars_arrow or _sync_flush_rest_bars_dict
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Single cache instance + 5-tuple return + symbol_modes threading</name>
  <files>
    python/opendeviationbar/sidecar/__init__.py
    python/opendeviationbar/sidecar/recovery.py
    tests/test_sidecar_alloc.py
  </files>
  <read_first>
    python/opendeviationbar/sidecar/__init__.py
    python/opendeviationbar/sidecar/recovery.py
    python/opendeviationbar/sidecar/clickhouse_write.py
    python/opendeviationbar/clickhouse/cache.py
    .planning/phases/24-hot-path-allocation-reduction/24-RESEARCH.md
  </read_first>
  <behavior>
    - Test 1: _create_engine return tuple has 5 elements (engine, loaded, min_tid_guard, symbol_modes, ch_cache)
    - Test 2: ch_cache returned from _create_engine is an OpenDeviationBarCache instance
    - Test 3: _flush_all_batches passes ouroboros_mode from symbol_modes to _write_batch_with_retry
  </behavior>
  <action>
    **ALLOC-01: Single cache instance in _create_engine:**

    1. In `_create_engine()`, create `OpenDeviationBarCache()` ONCE at the top (after symbol_modes computation, ~line 214):
       ```python
       from opendeviationbar.clickhouse.cache import OpenDeviationBarCache
       ch_cache = OpenDeviationBarCache()
       ```

    2. Replace the 3 `with OpenDeviationBarCache() as cache:` blocks with direct `ch_cache` usage:
       - **Line ~296 (Aion DELETE):** Remove `with OpenDeviationBarCache() as cache:`, use `ch_cache.client.command(...)` directly
       - **Line ~361 (Day kill+OPTIMIZE+DELETE):** Remove `with OpenDeviationBarCache() as cache:`, use `ch_cache.client.command(...)` and `ch_cache.client.query(...)` directly
       - **Line ~512 (min-TID guard):** Remove `with OpenDeviationBarCache() as cache:`, use `ch_cache.get_latest_bar_with_trade_id(...)` directly

       CRITICAL: Do NOT use `with ch_cache:` — the context manager's `__exit__` calls `close()` which kills the connection for all subsequent uses.

    3. Update the return statement (line ~532):
       ```python
       return engine, loaded, min_tid_guard, symbol_modes, ch_cache
       ```

    4. Update return type annotation to 5-tuple.

    5. In `run_sidecar()` (line ~783), update destructuring:
       ```python
       engine, checkpoints_loaded, min_tid_guard, symbol_modes, ch_cache = _create_engine(...)
       ```
       Remove the separate `ch_cache = OpenDeviationBarCache()` at line ~723.

    6. In `recovery.py` (line ~54), update destructuring:
       ```python
       new_engine, _, new_min_tid_guard, new_symbol_modes, new_ch_cache = _create_engine(...)
       ```
       Return `new_ch_cache` as well — update the return statement and the caller in `__init__.py` that destructures the recovery result (line ~1028).

    **ALLOC-03: Thread symbol_modes to _flush_all_batches:**

    7. In `_flush_all_batches()` (line ~903), use the already-available `symbol_modes` (nonlocal from `run_sidecar`):
       ```python
       mode = symbol_modes.get(symbol, "day")
       if _write_batch_with_retry(
           symbol, threshold, bars,
           cache=ch_cache,
           ouroboros_mode=mode,
       ):
       ```
       This requires `_write_batch_with_retry` to accept `ouroboros_mode` — that's done in Task 2.

    8. Create `tests/test_sidecar_alloc.py` with unit tests for ALLOC-01 and ALLOC-03 behaviors.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_sidecar_alloc.py -x -q 2>&1 | tail -20</automated>
  </verify>
  <acceptance_criteria>
    - grep -n "ch_cache = OpenDeviationBarCache()" python/opendeviationbar/sidecar/__init__.py shows exactly ONE creation in _create_engine (not 3+1)
    - grep -c "with OpenDeviationBarCache() as cache:" python/opendeviationbar/sidecar/__init__.py returns 0 (all context managers eliminated)
    - grep "return engine, loaded, min_tid_guard, symbol_modes, ch_cache" python/opendeviationbar/sidecar/__init__.py returns a match
    - grep "new_symbol_modes, new_ch_cache" python/opendeviationbar/sidecar/recovery.py returns a match
    - grep "ouroboros_mode=" python/opendeviationbar/sidecar/__init__.py shows mode threading in _flush_all_batches
  </acceptance_criteria>
  <done>
    - _create_engine creates exactly 1 OpenDeviationBarCache and returns it as 5th tuple element
    - run_sidecar reuses that cache instance instead of creating its own
    - recovery.py destructures and returns the 5-tuple correctly
    - _flush_all_batches threads ouroboros_mode from symbol_modes
    - Tests pass
  </done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Dual-type write function + arrow_flush shared cache</name>
  <files>
    python/opendeviationbar/sidecar/clickhouse_write.py
    python/opendeviationbar/sidecar/arrow_flush.py
    python/opendeviationbar/sidecar/midnight.py
    tests/test_sidecar_alloc.py
  </files>
  <read_first>
    python/opendeviationbar/sidecar/clickhouse_write.py
    python/opendeviationbar/sidecar/arrow_flush.py
    python/opendeviationbar/sidecar/midnight.py
    python/opendeviationbar/sidecar/__init__.py
    .planning/phases/24-hot-path-allocation-reduction/24-RESEARCH.md
  </read_first>
  <behavior>
    - Test 1: _write_bars_batch_to_clickhouse accepts pl.DataFrame and skips dict→DataFrame conversion
    - Test 2: _write_bars_batch_to_clickhouse accepts list[dict] and converts as before (backward compat)
    - Test 3: _write_bars_batch_to_clickhouse uses ouroboros_mode parameter instead of calling get_symbol_entries()
    - Test 4: _write_batch_with_retry passes ouroboros_mode through to _write_bars_batch_to_clickhouse
  </behavior>
  <action>
    **ALLOC-02: Dual-type write function:**

    1. In `clickhouse_write.py`, update `_write_bars_batch_to_clickhouse` signature:
       ```python
       def _write_bars_batch_to_clickhouse(
           symbol: str,
           threshold: int,
           bars: pl.DataFrame | list[dict],
           *,
           cache: object | None = None,
           ouroboros_mode: str = "day",
       ) -> None:
       ```

    2. Replace body: if `bars` is `list`, convert to DataFrame. If already DataFrame, use directly. Remove the `get_symbol_entries()` call — use the `ouroboros_mode` parameter instead:
       ```python
       if isinstance(bars, list):
           if not bars:
               return
           bars_df = pl.DataFrame(bars)
       else:
           bars_df = bars
           if bars_df.is_empty():
               return
       ```
       Remove lines 44-52 (the `get_symbol_entries` lookup + try/except). Use `ouroboros_mode` parameter directly in `store_bars_batch(ouroboros_mode=ouroboros_mode)`.

    3. Update `_write_batch_with_retry` signature to accept and pass `ouroboros_mode`:
       ```python
       def _write_batch_with_retry(
           symbol: str,
           threshold: int,
           bars: pl.DataFrame | list[dict],
           *,
           cache: object | None = None,
           ouroboros_mode: str = "day",
       ) -> bool:
       ```
       Pass `ouroboros_mode=ouroboros_mode` to `_write_bars_batch_to_clickhouse`.

    **ALLOC-02 + ALLOC-03 in arrow_flush.py:**

    4. Update `_sync_flush_rest_bars_arrow` to accept `cache` parameter:
       ```python
       def _sync_flush_rest_bars_arrow(
           engine: object,
           valid_pairs: set[tuple[str, int]] | None,
           *,
           symbol_modes: dict[str, str] | None = None,
           cache: object | None = None,
       ) -> int:
       ```

    5. Remove the `_get_entries()` call (line ~134) — use `symbol_modes` directly for mode lookup.

    6. Replace the per-pair `with OpenDeviationBarCache() as cache:` (line ~157) with the shared `cache` parameter. If `cache` is None, fall back to creating one:
       ```python
       _cache = cache
       if _cache is None:
           from opendeviationbar.clickhouse.cache import OpenDeviationBarCache
           _cache = OpenDeviationBarCache()
       ```
       Use `_cache` for all `store_bars_batch` calls in the loop — ONE cache instance for all pairs, not one per pair.

    7. In the per-pair loop, resolve mode from `symbol_modes`:
       ```python
       _mode = (_modes.get(sym, "day")) if _modes else "day"
       ```
       Remove the `_entries.get(sym)` + `.ouroboros_mode.value` lookup.

    8. In the fallback path (line ~177), pass the DataFrame directly instead of converting to dicts:
       ```python
       if _write_batch_with_retry(sym, threshold, pair_df, cache=_cache, ouroboros_mode=_mode):
           written += pair_df.height
       ```

    9. Update `_sync_flush_rest_bars` in `midnight.py` to pass `cache` through:
       ```python
       def _sync_flush_rest_bars(
           engine, valid_pairs, *, symbol_modes=None, cache=None,
       ) -> int:
           if hasattr(engine, "drain_bars_arrow"):
               return _sync_flush_rest_bars_arrow(engine, valid_pairs, symbol_modes=symbol_modes, cache=cache)
           return _sync_flush_rest_bars_dict(engine, valid_pairs, symbol_modes=symbol_modes)
       ```

    10. Update the call site in `__init__.py` (line ~484) to pass `ch_cache`:
        ```python
        flushed = _sync_flush_rest_bars(
            engine, valid_pairs, symbol_modes=symbol_modes, cache=ch_cache
        )
        ```

    11. Add tests to `tests/test_sidecar_alloc.py` for ALLOC-02 behaviors (DataFrame acceptance, ouroboros_mode threading).

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_sidecar_alloc.py -x -q 2>&1 | tail -20</automated>
  </verify>
  <acceptance_criteria>
    - grep "bars: pl.DataFrame | list\[dict\]" python/opendeviationbar/sidecar/clickhouse_write.py returns matches for both functions
    - grep "ouroboros_mode: str" python/opendeviationbar/sidecar/clickhouse_write.py returns matches for both functions
    - grep -c "get_symbol_entries" python/opendeviationbar/sidecar/clickhouse_write.py returns 0 (registry lookup eliminated)
    - grep -c "get_symbol_entries" python/opendeviationbar/sidecar/arrow_flush.py returns 0 (registry lookup eliminated)
    - grep -c "with OpenDeviationBarCache" python/opendeviationbar/sidecar/arrow_flush.py returns 0 (per-pair cache eliminated)
    - grep "cache: object | None" python/opendeviationbar/sidecar/arrow_flush.py returns a match
    - grep "cache=ch_cache" python/opendeviationbar/sidecar/__init__.py shows cache threading to _sync_flush_rest_bars
  </acceptance_criteria>
  <done>
    - _write_bars_batch_to_clickhouse accepts pl.DataFrame directly (skips conversion) or list[dict] (converts as before)
    - ouroboros_mode parameter replaces per-flush get_symbol_entries() in both clickhouse_write.py and arrow_flush.py
    - arrow_flush.py uses shared cache instance instead of creating one per (sym, thr) pair
    - Fallback path in arrow_flush.py passes DataFrame directly to _write_batch_with_retry (no .to_dicts())
    - All tests pass
  </done>
</task>

</tasks>

<verification>
1. `uv run --python 3.13 pytest tests/test_sidecar_alloc.py -x -q` — all allocation tests pass
2. `uv run --python 3.13 pytest tests/test_sidecar.py -x -q` — existing sidecar tests still pass
3. `grep -c "with OpenDeviationBarCache" python/opendeviationbar/sidecar/__init__.py` — returns 0
4. `grep -c "with OpenDeviationBarCache" python/opendeviationbar/sidecar/arrow_flush.py` — returns 0
5. `grep -c "get_symbol_entries" python/opendeviationbar/sidecar/clickhouse_write.py` — returns 0
6. `grep -c "get_symbol_entries" python/opendeviationbar/sidecar/arrow_flush.py` — returns 0
</verification>

<success_criteria>

1. Single `OpenDeviationBarCache()` instance created in `_create_engine`, returned as 5th tuple element, reused by `run_sidecar` main loop — zero redundant connections
2. `_write_bars_batch_to_clickhouse` accepts `pl.DataFrame` directly — zero dict→DataFrame conversion when caller already has a DataFrame
3. Per-flush `get_symbol_entries()` calls eliminated from both `clickhouse_write.py` and `arrow_flush.py` — ouroboros_mode resolved from pre-computed `symbol_modes` dict
4. `arrow_flush.py` uses shared cache and pre-computed modes — no per-pair `OpenDeviationBarCache()` or registry lookup
5. All existing sidecar tests continue to pass (backward compat)
   </success_criteria>

<output>
After completion, create `.planning/phases/24-hot-path-allocation-reduction/24-01-SUMMARY.md`
</output>
