---
phase: 02-sidecar-fill-pipeline
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar/__init__.py
  - python/opendeviationbar/sidecar/midnight.py
  - tests/test_sync_flush_atomic_replace.py
autonomous: true
requirements: [STREAM-01, STREAM-02]

must_haves:
  truths:
    - "fill_from_rest returns a bar count (u64), not raw trades — no Python trade DataFrame exists in the fill path"
    - "dropped_bars metric is checked after fill_from_rest and logged as a WARNING if > 0"
    - "Stathera junction telemetry records both rest_last_tid and ws_first_tid per symbol for post-startup verification"
    - "sync_flush validates Stathera ordering within each (symbol, threshold) batch before CH write"
  artifacts:
    - path: "python/opendeviationbar/sidecar/__init__.py"
      provides: "dropped_bars check after fill_from_rest, junction telemetry enrichment"
      contains: "dropped_bars"
    - path: "python/opendeviationbar/sidecar/midnight.py"
      provides: "Stathera continuity assertion in _sync_flush_rest_bars"
      contains: "stathera"
    - path: "tests/test_sync_flush_atomic_replace.py"
      provides: "Tests for Stathera ordering validation and dropped_bars diagnostic"
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "engine.get_metrics()"
      via: "dropped_bars check after fill_from_rest"
      pattern: "dropped_bars.*after.*fill_from_rest"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "python/opendeviationbar/sidecar/http_server.py"
      via: "_junction_state telemetry"
      pattern: "_junction_state.*ws_first_tid"
---

<objective>
Harden the sidecar fill pipeline with runtime assertions for STREAM-01 (no trade DataFrame in Python, dropped_bars diagnostic) and STREAM-02 (Stathera continuity validation at REST-WS junction).

Purpose: The fill pipeline already works (Rust fill_from_rest -> drain_bars -> sync_flush -> CH), but lacks runtime assertions that would catch regressions or edge cases (ring buffer overflow, Stathera gaps in REST bars). This plan adds defensive hardening without changing the data flow.

Output: Hardened \_create_engine() with dropped_bars check, Stathera ordering validation in_sync_flush_rest_bars, enriched junction telemetry, and tests.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/02-sidecar-fill-pipeline/02-RESEARCH.md

@python/opendeviationbar/sidecar/**init**.py
@python/opendeviationbar/sidecar/midnight.py
@python/opendeviationbar/sidecar/http_server.py
@tests/test_sync_flush_atomic_replace.py

<interfaces>
<!-- Key types and contracts the executor needs -->

From python/opendeviationbar/sidecar/**init**.py:

```python
# engine.fill_from_rest() returns u64 bar count (NOT trades)
rest_bars = engine.fill_from_rest()  # line 365

# engine.get_metrics() returns dict with dropped_bars, trades_received, etc.
metrics = engine.get_metrics()  # returns {"dropped_bars": int, "trades_received": int, ...}

# _sync_flush_rest_bars drains ring buffer and writes to CH
flushed = _sync_flush_rest_bars(engine, valid_pairs)  # returns int (written count)
```

From python/opendeviationbar/sidecar/midnight.py:

```python
def _sync_flush_rest_bars(engine: object, valid_pairs: set[tuple[str, int]] | None) -> int:
    # drain_bars() returns list[dict] with keys including:
    #   _symbol, _threshold, first_agg_trade_id, last_agg_trade_id,
    #   open_time_ms (actually microseconds), close_time_ms (actually microseconds),
    #   Open, High, Low, Close, Volume
```

From python/opendeviationbar/sidecar/http_server.py:

```python
_junction_state: dict[str, dict] = {}
# Currently populated with: _junction_state[symbol]["rest_last_tid"] = last_tid
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add dropped_bars diagnostic and Stathera ordering validation</name>
  <files>
    python/opendeviationbar/sidecar/__init__.py,
    python/opendeviationbar/sidecar/midnight.py,
    tests/test_sync_flush_atomic_replace.py
  </files>
  <read_first>
    python/opendeviationbar/sidecar/__init__.py,
    python/opendeviationbar/sidecar/midnight.py,
    python/opendeviationbar/sidecar/http_server.py,
    tests/test_sync_flush_atomic_replace.py
  </read_first>
  <behavior>
    - Test 1: _sync_flush_rest_bars logs WARNING when bars within a (symbol, threshold) batch have non-monotonic last_agg_trade_id ordering (Stathera violation within REST fill)
    - Test 2: _sync_flush_rest_bars passes cleanly when bars within each batch have monotonically increasing trade IDs
    - Test 3: Bars with first_agg_trade_id > previous bar's last_agg_trade_id + 1 (Stathera gap within REST batch) trigger a WARNING log but still write to CH (non-fatal diagnostic)
  </behavior>
  <action>
    **In tests/test_sync_flush_atomic_replace.py**, add 3 new test functions:

    1. `test_stathera_ordering_valid_batch()` — Create 3 bars for same (symbol, threshold) with sequential trade IDs (bar1: tid 100-200, bar2: tid 201-300, bar3: tid 301-400). Mock engine.drain_bars() to return them. Assert _sync_flush_rest_bars completes without WARNING log.

    2. `test_stathera_gap_warning()` — Create 3 bars with a Stathera gap (bar1: tid 100-200, bar2: tid 250-300 — gap of 49 trades). Assert _sync_flush_rest_bars logs a WARNING containing "stathera" (case-insensitive) AND still writes all bars to CH (non-fatal).

    3. `test_stathera_non_monotonic_warning()` — Create bars with out-of-order trade IDs (bar1: tid 300-400, bar2: tid 100-200). Assert WARNING is logged.

    **In python/opendeviationbar/sidecar/midnight.py (_sync_flush_rest_bars)**, after grouping bars into `batches` dict but BEFORE the CH write loop (around line 745):

    Add a Stathera continuity check within each (symbol, threshold) batch:
    - Sort bars by first_agg_trade_id within each batch
    - For consecutive bars in same batch: check `bars[i]["first_agg_trade_id"] == bars[i-1]["last_agg_trade_id"] + 1`
    - If gap found: `logger.warning("sync_flush: stathera gap in REST batch %s@%d: bar[%d].last_tid=%d -> bar[%d].first_tid=%d (delta=%d)", symbol, threshold, i-1, prev_last, i, curr_first, curr_first - prev_last - 1)`
    - If non-monotonic (curr_first <= prev_last): `logger.warning("sync_flush: stathera non-monotonic in REST batch %s@%d: bar[%d].last_tid=%d >= bar[%d].first_tid=%d", ...)`
    - This is DIAGNOSTIC ONLY — all bars still get written to CH regardless.

    **In python/opendeviationbar/sidecar/__init__.py (_create_engine)**, after `rest_bars = engine.fill_from_rest()` succeeds (around line 367), before the sync_flush block:

    Add dropped_bars check:
    ```python
    # Check ring buffer overflow — if bars were dropped during REST fill, log LOUD warning
    if hasattr(engine, "get_metrics"):
        post_fill_metrics = engine.get_metrics()
        post_fill_dropped = post_fill_metrics.get("dropped_bars", 0)
        if post_fill_dropped > 0:
            logger.warning(
                "fill_from_rest: %d bars DROPPED due to ring buffer overflow "
                "(capacity may need increase via OPENDEVIATIONBAR_MAX_PENDING_BARS)",
                post_fill_dropped,
            )
    ```

    Also enrich `_junction_state` in `_sync_flush_rest_bars` to track per-symbol min first_agg_trade_id (rest_first_tid) alongside existing rest_last_tid. After the existing junction telemetry block (~line 771-779), add:
    ```python
    min_tid = min(
        (b.get("first_agg_trade_id", 0) for b in bars),
        default=0,
    )
    if min_tid > 0:
        _junction_state[symbol]["rest_first_tid"] = min(
            _junction_state[symbol].get("rest_first_tid", min_tid),
            min_tid,
        )
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py -x -v 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - grep -n "stathera.*gap\|stathera.*non.monotonic" python/opendeviationbar/sidecar/midnight.py returns at least 2 matches (gap + non-monotonic warning paths)
    - grep -n "dropped_bars.*ring buffer\|DROPPED.*ring buffer" python/opendeviationbar/sidecar/__init__.py returns at least 1 match
    - grep -n "rest_first_tid" python/opendeviationbar/sidecar/midnight.py returns at least 1 match
    - grep -n "test_stathera" tests/test_sync_flush_atomic_replace.py returns at least 3 matches
    - uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py -x passes
  </acceptance_criteria>
  <done>
    - Stathera continuity validation runs on every REST batch before CH write (diagnostic, non-fatal)
    - dropped_bars checked after fill_from_rest with WARNING if > 0
    - Junction telemetry includes both rest_first_tid and rest_last_tid per symbol
    - 3+ new tests pass covering gap, non-monotonic, and valid ordering cases
  </done>
</task>

<task type="auto">
  <name>Task 2: Code audit — verify no trade DataFrame in Python fill path</name>
  <files>
    python/opendeviationbar/sidecar/__init__.py
  </files>
  <read_first>
    python/opendeviationbar/sidecar/__init__.py,
    python/opendeviationbar/sidecar/midnight.py
  </read_first>
  <action>
    Run a grep audit to verify STREAM-01 success criteria #3: "No intermediate trade DataFrame is materialized in Python between REST fetch and bar computation."

    Verify these facts by code reading (no code changes needed if clean):

    1. `engine.fill_from_rest()` at __init__.py:365 returns `u64` (bar count), NOT a DataFrame or list of trades
    2. `engine.drain_bars()` at midnight.py:665 returns `list[dict]` of BARS (not trades)
    3. No `pd.DataFrame` or `pl.DataFrame` construction of TRADES exists anywhere in the fill path (sidecar/__init__.py or sidecar/midnight.py)
    4. The only DataFrame construction in midnight.py:748 (`pl.DataFrame(bars)`) is for BARS going to CH write, not trades

    If any trade DataFrame is found in the fill path, remove it and route through Rust instead.

    If audit is clean (expected based on research), add a brief code comment at __init__.py:365 documenting the contract:
    ```python
    # STREAM-01: fill_from_rest processes trades entirely in Rust.
    # Returns bar count (u64). Bars retrieved via drain_bars() in _sync_flush_rest_bars().
    # No trade DataFrame is materialized in Python during this path.
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && echo "=== Trade DataFrame in sidecar fill path ===" && grep -n "pd\.DataFrame\|pl\.DataFrame" python/opendeviationbar/sidecar/__init__.py python/opendeviationbar/sidecar/midnight.py | grep -iv "bar" ; echo "Exit: $? (1 = clean, no trade DataFrames found)"</automated>
  </verify>
  <acceptance_criteria>
    - grep -n "STREAM-01" python/opendeviationbar/sidecar/__init__.py returns at least 1 match (contract comment added)
    - grep -n "pd.DataFrame\|pl.DataFrame" python/opendeviationbar/sidecar/__init__.py python/opendeviationbar/sidecar/midnight.py | grep -iv "bar" returns NO matches (no trade DataFrames in fill path)
    - uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py tests/test_midnight_preflight.py -x passes (no regressions)
  </acceptance_criteria>
  <done>
    - Code audit confirms no trade DataFrame exists in the Python fill path
    - STREAM-01 contract documented in source comment
    - Existing tests still pass
  </done>
</task>

</tasks>

<verification>
After both tasks complete:

1. `uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py tests/test_midnight_preflight.py -x -v` — all tests pass
2. `grep -rn "stathera.*gap\|dropped_bars.*ring\|STREAM-01" python/opendeviationbar/sidecar/` — hardening assertions present
3. `mise run check-full` — full suite green (no regressions)
   </verification>

<success_criteria>

- STREAM-01: fill_from_rest returns bars (not trades), no trade DataFrame in Python, dropped_bars diagnostic logged on overflow
- STREAM-02: Stathera continuity validated within REST batches before CH write, junction telemetry records both first and last TIDs
- All existing tests pass, 3+ new tests added
  </success_criteria>

<output>
After completion, create `.planning/phases/02-sidecar-fill-pipeline/02-01-SUMMARY.md`
</output>
