---
phase: 02-midnight-preflight-verification
plan: 02
type: execute
wave: 2
depends_on: [02-01]
files_modified:
  - python/opendeviationbar/sidecar.py
  - tests/test_midnight_preflight.py
autonomous: true
requirements: [PREFLIGHT-03]

must_haves:
  truths:
    - "On checkpoint-cleared restart, fill_from_rest starts from the verified crossover trade ID, not from ClickHouse max"
    - "yesterday_orphan.last_agg_trade_id + 1 == today_first_bar.first_agg_trade_id when preflight succeeds"
    - "When preflight returns None (API error, no orphan), sidecar falls back to current CH-seeded behavior with no crash"
  artifacts:
    - path: "python/opendeviationbar/sidecar.py"
      provides: "_run_midnight_preflight() orchestrator and seed override in _create_engine()"
      contains: "def _run_midnight_preflight"
    - path: "tests/test_midnight_preflight.py"
      provides: "Integration tests for preflight wiring into _create_engine"
  key_links:
    - from: "python/opendeviationbar/sidecar.py (_run_midnight_preflight)"
      to: "python/opendeviationbar/sidecar.py (_get_midnight_crossover_tid)"
      via: "per-symbol crossover query loop"
      pattern: "_get_midnight_crossover_tid"
    - from: "python/opendeviationbar/sidecar.py (_create_engine)"
      to: "engine.seed_trade_id(symbol, threshold, crossover_tid - 1)"
      via: "override CH seed with verified crossover tid"
      pattern: "seed_trade_id.*crossover"
---

<objective>
Wire midnight preflight verification into the sidecar _create_engine() startup flow, overriding fill_from_rest seeds with verified crossover trade IDs.

Purpose: This completes the midnight preflight feature by inserting the crossover query and verification step between \_delete_today_bars() and fill_from_rest() in_create_engine(). The verified crossover trade ID becomes the authoritative seed, ensuring fill_from_rest starts from exactly the right trade -- closing the multi-processor seam between kintsugi (yesterday) and sidecar (today).

Output: \_run_midnight_preflight() orchestrator function wired into_create_engine(), with integration tests proving seed override works.
</objective>

<execution_context>
@~/.claude/get-shit-done/workflows/execute-plan.md
@~/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/02-midnight-preflight-verification/02-CONTEXT.md
@.planning/phases/02-midnight-preflight-verification/02-01-SUMMARY.md

<interfaces>
<!-- From Plan 01 (must exist before this plan executes) -->

From python/opendeviationbar/sidecar.py (created by Plan 01):

```python
def _get_midnight_crossover_tid(symbol: str) -> int | None:
    """Returns agg_trade_id of first trade at/after UTC midnight, or None."""

def _verify_yesterday_orphan(symbol: str, crossover_tid: int) -> bool:
    """Returns True if orphan.last_agg_trade_id == crossover_tid - 1."""
```

From python/opendeviationbar/sidecar.py (existing):

```python
def _create_engine(
    config: SidecarConfig,
    trace_id: str,
    gap_fill_results: dict | None,
    valid_pairs: set[tuple[str, int]] | None = None,
) -> tuple[object, int]:
    # Line 846: _delete_today_bars(valid_pairs)
    # Line 849: rest_bars = engine.fill_from_rest()
    # engine.seed_trade_id(symbol, threshold, last_tid) — per-(symbol,threshold) seed
```

From python/opendeviationbar/sidecar.py (existing seeding pattern):

```python
# _seed_trade_ids_from_clickhouse() at line 815:
#   For each (symbol, threshold): engine.seed_trade_id(symbol, threshold, ch_last_tid)
# This is the CH-seeded baseline. Preflight OVERRIDES these seeds with crossover_tid - 1.
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add _run_midnight_preflight and wire into _create_engine</name>
  <files>python/opendeviationbar/sidecar.py</files>
  <read_first>
    python/opendeviationbar/sidecar.py
    python/opendeviationbar/clickhouse/query_operations.py
  </read_first>
  <action>
**Step 1: Add `_run_midnight_preflight()` function** to sidecar.py, placed near `_get_midnight_crossover_tid` and `_verify_yesterday_orphan` (added by Plan 01):

```python
def _run_midnight_preflight(
    engine: object,
    symbols: list[str],
    thresholds: list[int],
) -> dict[str, int]:
    """Run midnight preflight verification for all streaming symbols.

    PREFLIGHT-03: For each symbol, queries the crossover trade ID (first trade
    after UTC midnight) and verifies yesterday's orphan bar. If verified,
    overrides the engine's trade-ID seed for ALL thresholds of that symbol
    with crossover_tid - 1, so fill_from_rest starts from the exact right point.

    Returns dict mapping symbol -> crossover_tid for symbols that passed preflight.
    Symbols that fail preflight keep their existing CH-seeded values (graceful fallback).
    """
```

Implementation:

- Initialize `verified: dict[str, int] = {}`
- For each symbol in `symbols`:
  - `crossover_tid = _get_midnight_crossover_tid(symbol)`
  - If `crossover_tid is None`: log WARNING "preflight skipped for {symbol}: could not determine crossover trade ID", continue
  - `_verify_yesterday_orphan(symbol, crossover_tid)` (log-only, does not gate behavior)
  - Override seed: for each threshold in `thresholds`, call `engine.seed_trade_id(symbol, threshold, crossover_tid - 1)`
  - `verified[symbol] = crossover_tid`
- Log INFO summary: "midnight preflight: verified {len(verified)}/{len(symbols)} symbols, overrode seeds for {len(verified) \* len(thresholds)} pairs"
- Return `verified`

**Step 2: Wire into `_create_engine()`**

Insert the preflight call in `_create_engine()` AFTER `_seed_trade_ids_from_clickhouse()` (line ~815) and AFTER `_delete_today_bars()` (line ~846), but BEFORE `engine.fill_from_rest()` (line ~849).

The exact insertion point is right after `_delete_today_bars(valid_pairs)` and before the `try: rest_bars = engine.fill_from_rest()` block. Add:

```python
        # (v1.4 PREFLIGHT-03) Override CH seeds with verified midnight crossover TIDs.
        # Must run AFTER _seed_trade_ids_from_clickhouse (baseline) and AFTER
        # _delete_today_bars (clean slate), but BEFORE fill_from_rest (uses seeds).
        preflight_verified = _run_midnight_preflight(
            engine, config.symbols, config.thresholds
        )
        if preflight_verified:
            logger.info(
                "preflight: overrode seeds for %d symbols: %s",
                len(preflight_verified),
                ", ".join(f"{s}@{tid}" for s, tid in sorted(preflight_verified.items())),
            )
```

Important: The preflight OVERRIDES the CH seeds set by `_seed_trade_ids_from_clickhouse`. This is correct because `seed_trade_id` is a simple assignment (last write wins). The CH seed is the fallback; the crossover TID is authoritative.

The condition `if hasattr(engine, "fill_from_rest") and loaded < n_expected:` already gates this code path to checkpoint-cleared restarts only — exactly when preflight is needed.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && grep -n "\_run_midnight_preflight" python/opendeviationbar/sidecar.py</automated>
</verify>
<acceptance_criteria> - grep "def \_run_midnight_preflight" python/opendeviationbar/sidecar.py returns a match - grep "\_run_midnight_preflight" python/opendeviationbar/sidecar.py shows both the function definition AND the call site in \_create_engine - grep "seed_trade_id.\*crossover" python/opendeviationbar/sidecar.py returns a match (seed override) - grep "PREFLIGHT-03" python/opendeviationbar/sidecar.py returns a match (traceability) - The preflight call is between \_delete_today_bars and fill_from_rest in_create_engine - Fallback: when_get_midnight_crossover_tid returns None, no seed override happens (CH seed preserved)
</acceptance_criteria>
<done>\_run_midnight_preflight() orchestrates per-symbol crossover query, verification, and seed override. Wired into_create_engine() between_delete_today_bars and fill_from_rest. Symbols that fail preflight gracefully keep CH-seeded values.</done>
</task>

<task type="auto">
  <name>Task 2: Add integration tests for preflight wiring</name>
  <files>tests/test_midnight_preflight.py</files>
  <read_first>
    tests/test_midnight_preflight.py
    python/opendeviationbar/sidecar.py
  </read_first>
  <action>
Extend `tests/test_midnight_preflight.py` (created in Plan 01) with integration tests for `_run_midnight_preflight`:

1. `test_run_midnight_preflight_overrides_seeds` — mock `_get_midnight_crossover_tid` to return a known TID for one symbol, mock `_verify_yesterday_orphan` to return True, create a mock engine with `seed_trade_id` method. Verify `engine.seed_trade_id` was called with `(symbol, threshold, crossover_tid - 1)` for each threshold. Verify returned dict contains the symbol.

2. `test_run_midnight_preflight_partial_failure` — mock `_get_midnight_crossover_tid` to return TID for symbol A but None for symbol B. Verify seeds overridden only for A, not B. Verify returned dict has only symbol A.

3. `test_run_midnight_preflight_all_fail` — mock `_get_midnight_crossover_tid` to return None for all symbols. Verify no `seed_trade_id` calls. Verify returned dict is empty.

4. `test_run_midnight_preflight_verification_mismatch_still_overrides` — mock `_get_midnight_crossover_tid` returns TID, mock `_verify_yesterday_orphan` returns False (mismatch). Verify seeds are STILL overridden (verification is log-only, does not gate behavior). This ensures we always anchor from crossover TID when available, even if yesterday's orphan doesn't match (kintsugi will fix yesterday).

Run all tests:

```bash
uv run --python 3.13 pytest tests/test_midnight_preflight.py -v
```

All tests (6 from Plan 01 + 4 new = 10 total) must pass.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_midnight_preflight.py -v --no-header 2>&1 | tail -20</automated>
</verify>
<acceptance_criteria> - tests/test_midnight_preflight.py contains at least 10 test functions - grep "test_run_midnight_preflight" tests/test_midnight_preflight.py returns 4+ matches - All 10 tests pass - test_run_midnight_preflight_verification_mismatch_still_overrides verifies seeds ARE overridden even on mismatch
</acceptance_criteria>
<done>10 tests total in test_midnight_preflight.py covering: crossover query (3), orphan verification (3), preflight orchestration (4). All pass.</done>
</task>

</tasks>

<verification>
- `grep -c "def test_" tests/test_midnight_preflight.py` shows >= 10
- `uv run --python 3.13 pytest tests/test_midnight_preflight.py -v` — all 10 tests pass
- `grep -n "_run_midnight_preflight" python/opendeviationbar/sidecar.py` — shows both definition and call site
- `grep -A2 "_delete_today_bars" python/opendeviationbar/sidecar.py` — shows preflight call immediately after
</verification>

<success_criteria>

- \_run_midnight_preflight wired into_create_engine between_delete_today_bars and fill_from_rest
- Verified crossover TIDs override CH seeds via engine.seed_trade_id
- Graceful fallback when crossover query fails (CH seed preserved)
- All 10 tests pass (6 unit + 4 integration)
- No regression in existing sidecar tests
  </success_criteria>

<output>
After completion, create `.planning/phases/02-midnight-preflight-verification/02-02-SUMMARY.md`
</output>
