---
phase: 01-yesterday-integrity-check-signal-file
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar/signal.py
  - python/opendeviationbar/sidecar/midnight.py
  - python/opendeviationbar/sidecar/__init__.py
  - tests/test_yesterday_integrity.py
autonomous: true
requirements: [STARTUP-01, STARTUP-02]

must_haves:
  truths:
    - "Clean yesterday detected in < 5s (2 CH queries per symbol, no REST repair calls)"
    - "Broken symbols flagged with specific reason (orphan_misaligned, stathera_gap, stathera_overlap)"
    - "Signal file written atomically to /var/tmp/opendeviationbar-yesterday-repair.json"
    - "_repair_yesterday_if_needed() no longer called during sidecar startup"
    - "CH unavailable gracefully defers all symbols to kintsugi via signal file"
  artifacts:
    - path: "python/opendeviationbar/sidecar/signal.py"
      provides: "Atomic signal file writer + reader for yesterday-repair handoff"
      exports:
        ["write_yesterday_signal", "read_yesterday_signal", "SIGNAL_FILE_PATH"]
    - path: "python/opendeviationbar/sidecar/midnight.py"
      provides: "New _check_yesterday_integrity() function replacing _repair_yesterday_if_needed()"
      exports: ["_check_yesterday_integrity"]
    - path: "python/opendeviationbar/sidecar/__init__.py"
      provides: "Updated _create_engine() calling check+signal instead of repair"
    - path: "tests/test_yesterday_integrity.py"
      provides: "Unit tests for integrity check + signal file"
      min_lines: 80
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/midnight.py"
      via: "_check_yesterday_integrity() call at line ~265"
      pattern: "_check_yesterday_integrity"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "python/opendeviationbar/sidecar/signal.py"
      via: "write_yesterday_signal() for broken symbols"
      pattern: "write_yesterday_signal"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "python/opendeviationbar/clickhouse/stathera_query.py"
      via: "Stathera audit query using shared SQL builder"
      pattern: "STATHERA_CORE_COLUMNS|per_pair_filter|per_pair_window"
---

<objective>
Replace the blocking `_repair_yesterday_if_needed()` (4h sequential repair) with a fast integrity check (~3s) that writes a signal file for kintsugi to pick up.

Purpose: Remove yesterday repair from sidecar startup critical path, enabling sub-minute startup (STARTUP-01 + STARTUP-02).
Output: New signal.py module, rewritten integrity check in midnight.py, updated \_create_engine() call site, tests.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/REQUIREMENTS.md
@.planning/phases/01-yesterday-integrity-check-signal-file/01-CONTEXT.md

@python/opendeviationbar/sidecar/midnight.py
@python/opendeviationbar/sidecar/**init**.py
@python/opendeviationbar/clickhouse/stathera_query.py
@python/opendeviationbar/kintsugi/discovery.py

<interfaces>
<!-- Key types and contracts the executor needs. Extracted from codebase. -->

From python/opendeviationbar/clickhouse/stathera_query.py:

```python
DATABASE = "opendeviationbar_cache"
TABLE = "open_deviation_bars"
FULL_TABLE = f"{DATABASE}.{TABLE}"

STATHERA_CORE_COLUMNS = """\
    first_agg_trade_id,
    last_agg_trade_id,
    close_time_ms,
    lagInFrame(close_time_ms, 1) OVER w AS prev_close_time_ms,
    lagInFrame(last_agg_trade_id, 1) OVER w AS prev_last_tid,
    first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta,
    row_number() OVER w AS rn"""

VALID_TID_FILTER = "first_agg_trade_id > 0 AND last_agg_trade_id > 0"

def per_pair_filter() -> str: ...   # WHERE symbol={symbol:String} AND threshold_decimal_bps={threshold:UInt32} ...
def per_pair_window() -> str: ...   # WINDOW w AS (ORDER BY first_agg_trade_id, close_time_ms)
```

From python/opendeviationbar/sidecar/midnight.py (existing functions to reuse):

```python
def _get_midnight_crossover_tid(symbol: str) -> int | None: ...
def _verify_yesterday_orphan(symbol: str, crossover_tid: int) -> bool: ...
def _ch_yesterday_last_tid(symbol: str) -> int | None: ...
def _run_midnight_preflight(engine, symbols, thresholds) -> dict[str, int]: ...
```

From python/opendeviationbar/clickhouse/query_operations.py:

```python
def get_yesterday_orphan_tid(self, symbol: str, ouroboros_mode: str | None = None) -> int | None: ...
```

Call site in sidecar/**init**.py line 265 (to be replaced):

```python
# Step 2: Verify yesterday is clean.
_repair_yesterday_if_needed(
    config.symbols, config.thresholds, preflight_verified, valid_pairs
)
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Create signal.py module and _check_yesterday_integrity() function</name>
  <files>python/opendeviationbar/sidecar/signal.py, python/opendeviationbar/sidecar/midnight.py, tests/test_yesterday_integrity.py</files>
  <behavior>
    - Test: write_yesterday_signal writes valid JSON atomically (tempfile + rename) to SIGNAL_FILE_PATH
    - Test: write_yesterday_signal with empty dict writes file with empty symbols dict
    - Test: read_yesterday_signal returns None when file does not exist
    - Test: read_yesterday_signal returns parsed dict when file exists
    - Test: _check_yesterday_integrity returns empty dict when all symbols pass orphan check + Stathera audit
    - Test: _check_yesterday_integrity returns {"AVAXUSDT": "orphan_misaligned"} when orphan TID mismatches
    - Test: _check_yesterday_integrity returns {"BTCUSDT": "stathera_gap"} when Stathera audit finds gap on yesterday
    - Test: _check_yesterday_integrity returns {"ETHUSDT": "stathera_overlap"} when Stathera audit finds overlap
    - Test: _check_yesterday_integrity gracefully handles CH unavailable (returns dict with all symbols flagged as "ch_unavailable")
    - Test: _check_yesterday_integrity skips symbols not in preflight_verified
  </behavior>
  <action>
    **Create `python/opendeviationbar/sidecar/signal.py`** (per D: STARTUP-02 signal file):

    ```python
    SIGNAL_FILE_PATH = "/var/tmp/opendeviationbar-yesterday-repair.json"
    ```

    Implement `write_yesterday_signal(flagged: dict[str, str], sidecar_pid: int | None = None) -> None`:
    - Write JSON: `{"symbols": flagged, "timestamp": datetime.now(UTC).isoformat(), "sidecar_pid": os.getpid()}`
    - Atomic write: `tempfile.NamedTemporaryFile(dir="/var/tmp", delete=False)` then `os.rename(tmp, SIGNAL_FILE_PATH)`
    - If flagged is empty, still write the file (signals "sidecar checked, all clean")
    - Log at INFO: "yesterday-repair signal: wrote N flagged symbols to {path}"

    Implement `read_yesterday_signal() -> dict | None`:
    - Returns parsed JSON dict or None if file doesn't exist
    - Does NOT delete the file (Phase 2 kintsugi handles deletion)

    **Create `_check_yesterday_integrity()` in `midnight.py`** (per D: STARTUP-01):

    New function signature:
    ```python
    def _check_yesterday_integrity(
        symbols: list[str],
        thresholds: list[int],
        preflight_verified: dict[str, int],
    ) -> dict[str, str]:
    ```

    Returns `{symbol: reason}` where reason is one of: "orphan_misaligned", "orphan_missing", "stathera_gap", "stathera_overlap", "ch_unavailable".

    Implementation:
    1. For each symbol in `preflight_verified`:
       a. **Orphan check** (reuse existing `_verify_yesterday_orphan(symbol, crossover_tid)`):
          - If returns False, flag as "orphan_misaligned" (or "orphan_missing" based on whether orphan_tid was None)
          - Note: `_verify_yesterday_orphan` already queries CH and logs, but currently returns bool.
          - Enhance it to return a richer result OR add a new wrapper that distinguishes missing vs misaligned.
          - Simplest approach: call `cache.get_yesterday_orphan_tid(symbol)` directly and compare to `crossover_tid - 1`.

       b. **Stathera audit for yesterday** (new CH query, one per symbol across ALL thresholds):
          Use the shared stathera_query.py builder. Query pattern (batch all thresholds for one symbol):
          ```sql
          SELECT threshold_decimal_bps, tid_delta
          FROM (
              SELECT threshold_decimal_bps, {STATHERA_CORE_COLUMNS}
              FROM {FULL_TABLE} FINAL
              WHERE symbol = {symbol:String}
                AND ouroboros_mode = {ouroboros:String}
                AND toDate(open_time_ms / 1000, 'UTC') = yesterday()
                AND {VALID_TID_FILTER}
              WINDOW w AS (PARTITION BY threshold_decimal_bps ORDER BY first_agg_trade_id, close_time_ms)
          )
          WHERE rn > 1 AND tid_delta != 0
          LIMIT 1
          ```
          - If any row returned with tid_delta > 0: flag as "stathera_gap"
          - If any row returned with tid_delta < 0: flag as "stathera_overlap"
          - Use LIMIT 1 for speed — we only need to know IF broken, not enumerate all gaps.

    2. Wrap entire function in try/except for CH connection errors:
       - On connection failure: return `{sym: "ch_unavailable" for sym in preflight_verified}` and log WARNING.

    3. Log summary at end:
       - All clean: INFO "yesterday integrity: all {N} symbols clean"
       - Some broken: WARNING "yesterday integrity: {N}/{M} symbols broken: {details}"

    **IMPORTANT**: Do NOT import or call `_repair_under_lock`, `cache_write_lock`, or any kintsugi repair functions. This function ONLY checks and returns results. The signal file write happens in the caller (Task 2).

    **Write tests** in `tests/test_yesterday_integrity.py`:
    - Mock `OpenDeviationBarCache` and its `client.query()` for Stathera audit
    - Mock `get_yesterday_orphan_tid` for orphan checks
    - Use `tmp_path` fixture for signal file tests (monkeypatch `SIGNAL_FILE_PATH`)
    - Test the atomic write (file should exist after write, contain valid JSON)
    - Test CH unavailable path (mock cache constructor to raise ConnectionError)

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_yesterday_integrity.py -x -v 2>&1 | tail -30</automated>
  </verify>
  <done>
    - signal.py exists with write_yesterday_signal() and read_yesterday_signal()
    - midnight.py has _check_yesterday_integrity() that returns {symbol: reason} dict
    - Stathera audit uses shared stathera_query.py builder (not hand-rolled SQL)
    - All tests pass covering: clean path, orphan mismatch, stathera gap, stathera overlap, CH unavailable
  </done>
</task>

<task type="auto">
  <name>Task 2: Wire _check_yesterday_integrity into sidecar startup and remove repair path</name>
  <files>python/opendeviationbar/sidecar/__init__.py, python/opendeviationbar/sidecar/midnight.py</files>
  <action>
    **Update `sidecar/__init__.py`** (per D: STARTUP-02 — replace repair with check+signal):

    1. **Update imports** (around line 134-142):
       - Remove: `_repair_yesterday_if_needed` from the midnight import
       - Add: `_check_yesterday_integrity` from midnight import
       - Add: `from .signal import write_yesterday_signal`

    2. **Replace call site** at line 264-267 in `_create_engine()`:

       Replace:
       ```python
       # Step 2: Verify yesterday is clean.
       _repair_yesterday_if_needed(
           config.symbols, config.thresholds, preflight_verified, valid_pairs
       )
       ```

       With:
       ```python
       # Step 2: Check yesterday integrity (fast, < 5s) — signal kintsugi if broken.
       flagged_symbols = _check_yesterday_integrity(
           config.symbols, config.thresholds, preflight_verified
       )
       if flagged_symbols:
           write_yesterday_signal(flagged_symbols)
           logger.warning(
               "yesterday integrity: %d symbols flagged for kintsugi repair: %s",
               len(flagged_symbols),
               ", ".join(f"{s}={r}" for s, r in sorted(flagged_symbols.items())),
           )
       else:
           logger.info("yesterday integrity: all symbols clean, no signal file needed")
       ```

    3. **Do NOT delete `_repair_yesterday_if_needed()`** from midnight.py yet — it may still be referenced by recovery.py or other callers. Just remove it from the `__init__.py` import and call site. Per CONTEXT.md decision: "the function body changes from repair to check+signal" — but since we created a NEW function `_check_yesterday_integrity`, the old function stays as dead code until Phase 3 cleanup.

    4. **Verify no other callers** of `_repair_yesterday_if_needed`:
       - `sidecar/__init__.py` line 265 (the one we're replacing) — ONLY caller
       - The function is also exported from `sidecar/__init__.py` line 139 — remove that export too

    5. **Update `sidecar/recovery.py`** if it imports `_repair_yesterday_if_needed` (grep shows it does NOT — it only imports `_create_engine`). No changes needed.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_yesterday_integrity.py tests/test_sidecar_watchdog.py -x -v 2>&1 | tail -30</automated>
  </verify>
  <done>
    - _create_engine() calls _check_yesterday_integrity() + write_yesterday_signal() instead of _repair_yesterday_if_needed()
    - _repair_yesterday_if_needed removed from sidecar/__init__.py imports/exports (function body kept in midnight.py as dead code for Phase 3)
    - No kintsugi import or repair call in the startup critical path
    - Existing sidecar watchdog tests still pass (they mock _create_engine internals)
    - Signal file written only when flagged_symbols is non-empty
  </done>
</task>

</tasks>

<verification>
1. `uv run --python 3.13 pytest tests/test_yesterday_integrity.py -x -v` — all new tests pass
2. `uv run --python 3.13 pytest tests/test_sidecar_watchdog.py -x -v` — existing sidecar tests still pass
3. `grep -rn "_repair_yesterday_if_needed" python/opendeviationbar/sidecar/__init__.py` — should return zero matches (removed from imports and call site)
4. `grep -rn "_check_yesterday_integrity" python/opendeviationbar/sidecar/__init__.py` — should match at import and call site
5. `grep -rn "write_yesterday_signal" python/opendeviationbar/sidecar/__init__.py` — should match at import and call site
6. `python -c "from opendeviationbar.sidecar.signal import write_yesterday_signal, read_yesterday_signal, SIGNAL_FILE_PATH; print('OK')"` — imports work
</verification>

<success_criteria>

- \_repair_yesterday_if_needed() is no longer called during sidecar startup
- \_check_yesterday_integrity() performs 2 CH queries per symbol (orphan + Stathera), no REST repair calls
- Signal file written atomically to /var/tmp/opendeviationbar-yesterday-repair.json with {symbol: reason} format
- CH unavailable gracefully defers (no crash, all symbols flagged)
- All existing sidecar tests continue to pass
- New test file covers clean path, orphan mismatch, stathera gap/overlap, CH unavailable
  </success_criteria>

<output>
After completion, create `.planning/phases/01-yesterday-integrity-check-signal-file/01-01-SUMMARY.md`
</output>
