---
phase: 37-python-integration-staging-validation
plan: "02"
type: execute
wave: 2
depends_on: ["37-01"]
files_modified:
  - scripts/validate_shadow_mode.py
autonomous: false
requirements: [VALID-03, VALID-04]

must_haves:
  truths:
    - "Shadow mode runs for 24+ hours on bigblack with zero SHADOW MISMATCH warnings"
    - "Stathera audit shows zero new gaps or overlaps introduced during shadow period"
    - "ch_writer_bars_flushed > 0 throughout staging (Rust writer is active)"
    - "ch_writer_dead_letter_count == 0 throughout staging (no Rust-side failures)"
    - "Bar counts increased normally during shadow period (no dropped bars)"
  artifacts:
    - path: "scripts/validate_shadow_mode.py"
      provides: "Pre-configured validation script for 24h staging checkpoint"
      contains: "validate_shadow_mode"
  key_links:
    - from: "scripts/validate_shadow_mode.py"
      to: "scripts/detect_gaps.py"
      via: "subprocess call for Stathera audit"
      pattern: "detect_gaps"
---

<objective>
Create a validation script and run 24h shadow mode staging on bigblack to prove the Rust ClickHouse writer produces identical results to the Python writer with zero data integrity regressions.

Purpose: Gate the live-mode cutover behind empirical proof of correctness over a sustained period.
Output: Validation script, human-verified 24h staging results.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/37-python-integration-staging-validation/37-RESEARCH.md
@.planning/phases/37-python-integration-staging-validation/37-01-SUMMARY.md

@scripts/detect_gaps.py
@deploy/sidecar.env
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create shadow mode validation script</name>
  <files>scripts/validate_shadow_mode.py</files>
  <read_first>
    scripts/detect_gaps.py
    deploy/sidecar.env
  </read_first>
  <action>
    Create `scripts/validate_shadow_mode.py` -- a self-contained validation script for the 24h staging checkpoint. The script has three subcommands:

    1. `pre` -- Capture baseline before enabling shadow mode:
       - Run `detect_gaps.py --json` and save to `/tmp/pre-shadow-stathera.json`
       - Query CH for current bar counts per (symbol, threshold): `SELECT symbol, threshold_decimal_bps, count() FROM opendeviationbar_cache.open_deviation_bars GROUP BY symbol, threshold_decimal_bps ORDER BY symbol, threshold_decimal_bps` and save to `/tmp/pre-shadow-counts.json`
       - Print instructions: "Now set OPENDEVIATIONBAR_USE_RUST_CH_WRITER=shadow in deploy/sidecar.env and run: mise run deploy:bigblack"

    2. `check` -- Run during the 24h shadow period (can be run multiple times):
       - Query sidecar journal for SHADOW MISMATCH: `journalctl --user-unit=opendeviationbar-sidecar.service --since '24 hours ago' --no-pager -o cat | grep -i 'shadow mismatch'`
       - Count matches -- expect 0
       - Query CH for current `ch_writer_bars_flushed` via sidecar /health endpoint
       - Print summary: mismatch count, bars flushed, dead letter count, elapsed time since pre-shadow baseline

    3. `post` -- Run after 24+ hours to validate results:
       - Run `detect_gaps.py --json` and save to `/tmp/post-shadow-stathera.json`
       - Compare with pre-shadow baseline: count new gaps, new overlaps
       - Query current bar counts and compare with baseline (should have increased)
       - Print PASS/FAIL verdict with details
       - If PASS: print instructions for enabling live mode

    The script uses `clickhouse_connect` for CH queries and `urllib.request` for sidecar health. It should be runnable on bigblack with `uv run --python 3.13 python scripts/validate_shadow_mode.py {pre|check|post}`.

    Include `argparse` with the three subcommands. Use `sys.exit(0)` for pass, `sys.exit(1)` for fail.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "import ast; ast.parse(open('scripts/validate_shadow_mode.py').read()); print('syntax OK')" && grep -q "def main" scripts/validate_shadow_mode.py</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "pre-shadow-stathera" scripts/validate_shadow_mode.py
    - grep -q "SHADOW MISMATCH" scripts/validate_shadow_mode.py
    - grep -q "detect_gaps" scripts/validate_shadow_mode.py
    - grep -q "ch_writer_bars_flushed" scripts/validate_shadow_mode.py
    - grep -q "argparse" scripts/validate_shadow_mode.py
  </acceptance_criteria>
  <done>
    - validate_shadow_mode.py has pre/check/post subcommands
    - pre captures Stathera baseline + bar counts
    - check reports shadow mismatches and writer health
    - post compares against baseline and prints PASS/FAIL
    - Script is syntactically valid Python
  </done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 2: 24h shadow mode staging on bigblack</name>
  <files>deploy/sidecar.env</files>
  <action>
    Human runs the 24h staging validation on bigblack using the validation script from Task 1. Claude cannot automate this -- it requires 24+ hours of wall-clock time on a remote host. The steps below are for the human to follow.
  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && echo "HUMAN CHECKPOINT: Verify 24h shadow mode results. Run: uv run --python 3.13 python scripts/validate_shadow_mode.py post"</automated>
  </verify>
  <done>
    - 24h shadow mode completed on bigblack with zero SHADOW MISMATCH warnings
    - Stathera audit shows zero new gaps or overlaps
    - ch_writer_dead_letter_count == 0 throughout staging
    - Human typed "approved"
  </done>
  <what-built>
    Feature-flagged Rust ClickHouse writer (Plan 01) with shadow mode dual-write, heartbeat monitoring, and validation script.
  </what-built>
  <how-to-verify>
    **Step 1: Deploy and capture baseline**
    ```bash
    # On bigblack
    mise run deploy:bigblack
    uv run --python 3.13 python scripts/validate_shadow_mode.py pre
    ```

    **Step 2: Enable shadow mode**
    - Edit `deploy/sidecar.env`: set `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=shadow`
    - `mise run deploy:bigblack` (restarts sidecar)

    **Step 3: Monitor for 24 hours**
    - Heartbeat Telegram should show "Rust CH Writer" section with bars_flushed > 0
    - Run periodic checks: `uv run --python 3.13 python scripts/validate_shadow_mode.py check`
    - Watch for any SHADOW MISMATCH warnings in heartbeat/journal

    **Step 4: After 24+ hours, validate**
    ```bash
    uv run --python 3.13 python scripts/validate_shadow_mode.py post
    ```

    **Expected results (VALID-03, VALID-04):**
    - Zero SHADOW MISMATCH warnings in journal
    - ch_writer_bars_flushed > 0 (Rust writer active)
    - ch_writer_dead_letter_count == 0 (no Rust failures)
    - Stathera audit: zero NEW gaps or overlaps (same integrity as Python-only)
    - Bar counts increased normally (no dropped bars)

  </how-to-verify>
  <resume-signal>Type "approved" if 24h shadow mode passed, or describe issues found</resume-signal>
</task>

</tasks>

<verification>
1. Validation script exists: `test -f scripts/validate_shadow_mode.py`
2. Script has all subcommands: `grep -c "def cmd_" scripts/validate_shadow_mode.py` >= 3
3. Human verified: 24h shadow mode on bigblack passed with zero mismatches
</verification>

<success_criteria>

- validate_shadow_mode.py created with pre/check/post subcommands
- 24h shadow mode on bigblack completed with zero SHADOW MISMATCH warnings
- Stathera audit shows zero new gaps or overlaps during shadow period
- ch_writer_dead_letter_count == 0 throughout staging
- Human approved staging results
  </success_criteria>

<output>
After completion, create `.planning/phases/37-python-integration-staging-validation/37-02-SUMMARY.md`
</output>
