# Phase 24: Hot-Path Allocation Reduction - Research

**Researched:** 2026-03-27
**Domain:** Python sidecar memory optimization, ClickHouse connection reuse, data conversion elimination
**Confidence:** HIGH

## Summary

Phase 24 eliminates three sources of redundant allocations in the sidecar: (1) multiple `OpenDeviationBarCache()` instantiations during `_create_engine` startup, (2) per-flush `list[dict]` to `pl.DataFrame` conversion in `_write_bars_batch_to_clickhouse`, and (3) per-flush `get_symbol_entries()` calls for ouroboros mode lookup.

The changes are purely Python-side refactoring. The `_create_engine()` function creates at least 3 separate `OpenDeviationBarCache()` context managers during startup (Aion DELETE, Day kill+OPTIMIZE+DELETE, min-TID guard), each establishing a new ClickHouse connection. The main loop in `run_sidecar()` already creates a long-lived `ch_cache` instance (line 723) but does not pass it into `_create_engine()`. For per-flush, `_write_bars_batch_to_clickhouse` accepts `list[dict]` and converts to `pl.DataFrame` on every call (line 37), though `store_bars_batch()` already accepts `pl.DataFrame`. Finally, `get_symbol_entries()` is `@lru_cache(maxsize=1)` so the TOML is parsed once, but the function is called inside the per-flush path unnecessarily -- the entries never change during a sidecar session.

**Primary recommendation:** Create a single `OpenDeviationBarCache()` instance early in `_create_engine`, pass it through all startup phases and return it for use in the main loop. Make `_write_bars_batch_to_clickhouse` accept `pl.DataFrame | list[dict]`, skipping conversion when already a DataFrame. Cache `get_symbol_entries()` result at startup and thread it through write functions.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all implementation choices at Claude's discretion (infrastructure phase).

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

### Deferred Ideas (OUT OF SCOPE)

None -- discuss phase skipped.

</user_constraints>

<phase_requirements>

## Phase Requirements

| ID       | Description                                                                                    | Research Support                                                                                                                                            |
| -------- | ---------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ALLOC-01 | Single `OpenDeviationBarCache()` instance reused across `_create_engine` (not 3 separate ones) | 3 instantiation sites identified in `_create_engine`: lines 296, 361, 512. One more at line 723 in `run_sidecar`. Consolidate to one created early.         |
| ALLOC-02 | `_write_bars_batch_to_clickhouse` accepts Arrow/Polars directly (no dict->DataFrame per flush) | `clickhouse_write.py` line 37: `bars_df = pl.DataFrame(bar_dicts)` on every call. `store_bars_batch()` already accepts `pl.DataFrame`. Skip conversion.     |
| ALLOC-03 | `get_symbol_entries()` cached at sidecar startup, not re-called per flush                      | Called in `clickhouse_write.py:48`, `arrow_flush.py:134`, and 7+ times in `midnight.py`. Already `@lru_cache` but call overhead + import per flush remains. |

</phase_requirements>

## Standard Stack

### Core

No new libraries. This phase uses only existing project dependencies.

| Library            | Version       | Purpose                                       | Why Standard                       |
| ------------------ | ------------- | --------------------------------------------- | ---------------------------------- |
| polars             | (project ver) | DataFrame creation, zero-copy filter          | Already used throughout sidecar    |
| clickhouse-connect | (project ver) | ClickHouse client via `OpenDeviationBarCache` | Already the sole CH client library |

## Architecture Patterns

### ALLOC-01: Single Cache Instance in \_create_engine

**Current state:** `_create_engine()` creates 3 separate `OpenDeviationBarCache()` context managers:

1. **Line 296** (`__init__.py`): Aion DELETE loop -- `with OpenDeviationBarCache() as cache:`
2. **Line 361** (`__init__.py`): Day kill mutations + OPTIMIZE + DELETE -- `with OpenDeviationBarCache() as cache:`
3. **Line 512** (`__init__.py`): min-TID guard -- `with OpenDeviationBarCache() as cache:`

Additionally, `run_sidecar()` creates a 4th instance at **line 723**: `ch_cache = OpenDeviationBarCache()`.

**Pattern:** Create `OpenDeviationBarCache()` once at the start of `_create_engine`, pass it to all internal uses, and return it as part of the tuple so `run_sidecar()` can reuse it.

```python
# Current signature:
def _create_engine(config, trace_id, gap_fill_results, valid_pairs=None):
    # returns: (engine, checkpoints_loaded, min_tid_guard, symbol_modes)

# New signature:
def _create_engine(config, trace_id, gap_fill_results, valid_pairs=None):
    # returns: (engine, checkpoints_loaded, min_tid_guard, symbol_modes, ch_cache)
```

**Key constraint:** `OpenDeviationBarCache` supports context manager (`__enter__`/`__exit__`) but also works as a plain object. The `__exit__` calls `close()` which sets `_client = None`. The main loop's `ch_cache` at line 723 is used as a plain object (never enters a context manager), confirming long-lived usage is safe.

**Downstream impact in midnight.py:** Several functions (`_run_midnight_preflight`, `_check_yesterday_integrity`, `_heal_day_boundary_gaps`, `fill_from_rest` via `_sync_flush_rest_bars_arrow`) also create their own `OpenDeviationBarCache()` instances. These are called FROM `_create_engine`, so they can accept a `cache` parameter. However, midnight.py functions are also called independently (e.g., `_heal_day_boundary_gaps` from `run_sidecar` line 800), so they should accept `cache=None` with a fallback to creating their own.

### ALLOC-02: Skip dict-to-DataFrame Conversion

**Current state:** `_write_bars_batch_to_clickhouse()` in `clickhouse_write.py`:

- Signature: `bar_dicts: list[dict]`
- Line 37: `bars_df = pl.DataFrame(bar_dicts)` -- creates DataFrame from dicts every call

**The main loop hot path:**

1. `engine.next_bar()` returns a `dict` (single bar)
2. Bars accumulate in `bar_batch_buffer[batch_key]` as `list[dict]` (max 10)
3. `_write_batch_with_retry()` -> `_write_bars_batch_to_clickhouse()` -> `pl.DataFrame(bar_dicts)`

**The Arrow flush path (already optimized):** `arrow_flush.py` calls `cache.store_bars_batch()` directly with `pair_df` (a `pl.DataFrame`), bypassing `_write_bars_batch_to_clickhouse` entirely. This path is already optimized by Phase 23.

**Pattern:** Make `_write_bars_batch_to_clickhouse` accept `pl.DataFrame | list[dict]`:

```python
def _write_bars_batch_to_clickhouse(
    symbol: str,
    threshold: int,
    bars: pl.DataFrame | list[dict],
    *,
    cache: object | None = None,
    symbol_entries: dict | None = None,  # ALLOC-03
) -> None:
    if isinstance(bars, list):
        bars_df = pl.DataFrame(bars)
    else:
        bars_df = bars
```

**Note on main loop:** The main loop accumulates bars as `list[dict]` because `engine.next_bar()` returns individual dicts. The conversion to DataFrame still happens, but now it can be skipped when the caller already has a DataFrame (e.g., if future code pre-batches). The main win is that `arrow_flush.py`'s fallback path (line 177) currently converts pair_df back to dicts just to call `_write_batch_with_retry`, which then reconverts to DataFrame. With this change, it can pass the DataFrame directly.

### ALLOC-03: Cache symbol_entries at Startup

**Current state:** `get_symbol_entries()` is called in:

- `clickhouse_write.py:48` -- per WS flush (10-bar batch, ~1-6 times/sec)
- `arrow_flush.py:134` -- per sync_flush (startup only)
- `midnight.py` -- 7+ call sites (startup only)
- `__init__.py:204` -- startup
- `__init__.py:863` -- Argus check (periodic)

The function is `@lru_cache(maxsize=1)` so the TOML parse happens once. But the per-flush call in `clickhouse_write.py` adds: function call overhead + `try/except` wrapper + `.get()` lookup + `.ouroboros_mode.value` attribute chain, all executed on every 10-bar flush.

**Pattern:** Resolve ouroboros mode once at startup, store in a `dict[str, str]` (same as `symbol_modes` already computed in `_create_engine` line 205-211), and pass it through:

```python
# Already computed at line 205-211 of __init__.py:
symbol_modes: dict[str, str] = {}
for sym in config.symbols:
    entry = entries.get(sym)
    if entry and entry.ouroboros_mode == OuroborosMode.AION:
        symbol_modes[sym] = "aion"
    else:
        symbol_modes[sym] = "day"
```

Thread `symbol_modes` to `_write_bars_batch_to_clickhouse` via a new `ouroboros_mode` parameter, eliminating the per-flush registry lookup entirely.

### Anti-Patterns to Avoid

- **Breaking `_write_batch_with_retry` signature compatibility:** The retry function wraps `_write_bars_batch_to_clickhouse`. Both must be updated in lockstep.
- **Removing the `cache=None` fallback:** Functions called from non-sidecar contexts (e.g., tests, scripts) must still work without a pre-created cache.
- **Closing the shared cache prematurely:** Do not use `with cache:` in internal functions if the cache is shared -- the `__exit__` will close the connection for everyone.
- **Storing mutable state on symbol_entries:** `get_symbol_entries()` returns `MappingProxyType` (immutable). The `symbol_modes` dict pattern is the right way to cache the derived values.

## Don't Hand-Roll

| Problem                  | Don't Build               | Use Instead                                   | Why                                           |
| ------------------------ | ------------------------- | --------------------------------------------- | --------------------------------------------- |
| Connection pooling       | Custom connection pool    | Single `OpenDeviationBarCache` instance       | CH client already handles TCP keepalive       |
| Thread-safe cache access | Lock wrapper around cache | clickhouse-connect is thread-safe for queries | Driver handles concurrency internally         |
| Symbol mode caching      | Custom cache invalidation | `symbol_modes` dict from startup              | Symbols never change during a sidecar session |

## Common Pitfalls

### Pitfall 1: Context Manager Closing Shared Connection

**What goes wrong:** Passing the shared `ch_cache` into a `with cache:` block causes `__exit__` to call `cache.close()`, killing the connection for all subsequent uses.
**Why it happens:** Existing code uses `with OpenDeviationBarCache() as cache:` pattern everywhere, assuming each context manager owns its own instance.
**How to avoid:** When refactoring to use a shared cache, replace `with OpenDeviationBarCache() as cache:` with direct usage of the passed-in cache parameter. Only use context manager syntax when the cache is locally created.
**Warning signs:** `AttributeError: NoneType has no attribute 'command'` after first `with` block exits.

### Pitfall 2: Return Tuple Size Change

**What goes wrong:** `_create_engine` currently returns a 4-tuple `(engine, loaded, min_tid_guard, symbol_modes)`. Adding `ch_cache` makes it a 5-tuple. All callers must be updated.
**Why it happens:** Python destructuring assignment fails silently with wrong tuple size.
**How to avoid:** Grep for all `_create_engine(` call sites. There are exactly 2: `run_sidecar()` (line 783) and `_restart_engine_with_recovery()` in `recovery.py`.
**Warning signs:** `ValueError: not enough values to unpack`.

### Pitfall 3: Arrow Flush Fallback Still Using Dicts

**What goes wrong:** `arrow_flush.py` line 177 converts `pair_df.to_dicts()` before calling `_write_batch_with_retry`. This creates unnecessary dict allocations in the fallback path.
**Why it happens:** `_write_batch_with_retry` and `_write_bars_batch_to_clickhouse` currently only accept `list[dict]`.
**How to avoid:** Update both functions to accept `pl.DataFrame | list[dict]`, and update the fallback in `arrow_flush.py` to pass the DataFrame directly.
**Warning signs:** N/A -- this is a latent inefficiency, not a bug.

## Code Examples

### Pattern: Shared Cache Through \_create_engine

```python
# In _create_engine (early):
from opendeviationbar.clickhouse.cache import OpenDeviationBarCache
ch_cache = OpenDeviationBarCache()

# Replace:  with OpenDeviationBarCache() as cache:
# With:     cache = ch_cache
# (no context manager -- shared instance)

# Aion DELETE:
for sym, thr in aion_pairs:
    ch_cache.client.command(...)

# Day DELETE:
ch_cache.client.command("KILL MUTATION ...")
for sym, thr in pairs:
    ch_cache.client.command(...)

# min-TID guard:
for symbol, threshold in unloaded_pairs:
    _, ch_last_tid = ch_cache.get_latest_bar_with_trade_id(...)

# Return:
return engine, loaded, min_tid_guard, symbol_modes, ch_cache
```

### Pattern: Dual-Type Write Function

```python
def _write_bars_batch_to_clickhouse(
    symbol: str,
    threshold: int,
    bars: pl.DataFrame | list[dict],
    *,
    cache: object | None = None,
    ouroboros_mode: str = "day",
) -> None:
    from opendeviationbar.clickhouse.cache import OpenDeviationBarCache

    if isinstance(bars, list):
        if not bars:
            return
        bars_df = pl.DataFrame(bars)
    else:
        bars_df = bars
        if bars_df.is_empty():
            return

    from opendeviationbar import __version__
    source_version = f"{__version__}+sidecar"

    if cache is not None:
        cache.store_bars_batch(
            symbol=symbol,
            threshold_decimal_bps=threshold,
            bars=bars_df,
            version=source_version,
            overlap_strategy="off",
            ouroboros_mode=ouroboros_mode,
        )
    else:
        with OpenDeviationBarCache() as ch_cache:
            ch_cache.store_bars_batch(
                symbol=symbol,
                threshold_decimal_bps=threshold,
                bars=bars_df,
                version=source_version,
                overlap_strategy="off",
                ouroboros_mode=ouroboros_mode,
            )
```

### Pattern: Threading ouroboros_mode Through Main Loop

```python
# In main loop (run_sidecar), the batch flush already has symbol_modes:
for burst_bar in burst_bars:
    symbol = burst_bar.pop("_symbol", "UNKNOWN")
    threshold = burst_bar.pop("_threshold", 0)
    # ...
    bar_batch_buffer[batch_key].append(burst_bar)

# In _flush_all_batches:
def _flush_all_batches() -> int:
    written = 0
    for (symbol, threshold), bars in bar_batch_buffer.items():
        if not bars:
            continue
        mode = symbol_modes.get(symbol, "day")  # <-- use pre-computed
        if _write_batch_with_retry(
            symbol, threshold, bars,
            cache=ch_cache,
            ouroboros_mode=mode,
        ):
            written += len(bars)
    bar_batch_buffer.clear()
    return written
```

## Validation Architecture

### Test Framework

| Property           | Value                                |
| ------------------ | ------------------------------------ |
| Framework          | pytest                               |
| Config file        | `pyproject.toml` [tool.pytest]       |
| Quick run command  | `pytest tests/test_sidecar.py -x -q` |
| Full suite command | `mise run test-py`                   |

### Phase Requirements -> Test Map

| Req ID   | Behavior                                                | Test Type | Automated Command                                                     | File Exists? |
| -------- | ------------------------------------------------------- | --------- | --------------------------------------------------------------------- | ------------ |
| ALLOC-01 | Single cache instance created, not 3 in \_create_engine | unit      | `pytest tests/test_sidecar_alloc.py::test_single_cache_instance -x`   | Wave 0       |
| ALLOC-02 | \_write_bars_batch accepts DataFrame directly           | unit      | `pytest tests/test_sidecar_alloc.py::test_write_accepts_dataframe -x` | Wave 0       |
| ALLOC-03 | ouroboros_mode threaded from startup, no per-flush call | unit      | `pytest tests/test_sidecar_alloc.py::test_ouroboros_mode_threaded -x` | Wave 0       |

### Sampling Rate

- **Per task commit:** `pytest tests/test_sidecar_alloc.py -x -q`
- **Per wave merge:** `mise run test-py`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_sidecar_alloc.py` -- covers ALLOC-01, ALLOC-02, ALLOC-03

## Open Questions

1. **Should `arrow_flush.py` also accept a shared cache?**
   - What we know: `_sync_flush_rest_bars_arrow` creates `OpenDeviationBarCache()` per (symbol, threshold) pair in a loop (line 157). This is called only at startup (sync_flush path).
   - What's unclear: Whether startup overhead justifies the refactoring.
   - Recommendation: Yes, include it since the shared cache is already available from `_create_engine`. Pass through `_sync_flush_rest_bars` -> `_sync_flush_rest_bars_arrow`.

2. **Should `midnight.py` functions also use the shared cache?**
   - What we know: Functions like `_run_midnight_preflight`, `_check_yesterday_integrity`, `fill_from_rest` create their own `OpenDeviationBarCache()`.
   - What's unclear: These are called only during startup, so the savings are small.
   - Recommendation: Thread `cache` parameter through as optional (`cache=None` fallback) to be consistent, but this is lower priority than the hot-path optimizations.

## Sources

### Primary (HIGH confidence)

- Direct codebase inspection: `python/opendeviationbar/sidecar/__init__.py`, `clickhouse_write.py`, `arrow_flush.py`, `midnight.py`
- Direct codebase inspection: `python/opendeviationbar/clickhouse/cache.py` (class definition), `mixin.py` (context manager)
- Direct codebase inspection: `python/opendeviationbar/symbol_registry.py` (`@lru_cache` on `get_symbol_entries`)

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- no new libraries, all changes in existing Python code
- Architecture: HIGH -- direct codebase reading, all patterns verified against source
- Pitfalls: HIGH -- context manager behavior verified against `mixin.py` source

**Research date:** 2026-03-27
**Valid until:** 2026-04-27 (stable -- internal refactoring, no external dependencies)
