---
phase: 37-python-integration-staging-validation
plan: "01"
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar/__init__.py
  - python/opendeviationbar/sidecar/recovery.py
  - python/opendeviationbar/sidecar/sync_flush.py
  - python/opendeviationbar/sidecar/arrow_flush.py
  - scripts/heartbeat/checks/sidecar.py
  - scripts/heartbeat/checks/__init__.py
  - scripts/heartbeat/config.py
  - scripts/heartbeat/formatter.py
  - deploy/sidecar.env
  - tests/test_sidecar_ch_writer_flag.py
autonomous: true
requirements: [INTEG-03, VALID-01, VALID-02, VALID-04, VALID-05]

must_haves:
  truths:
    - "Setting OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off causes Python-only CH writes (current behavior)"
    - "Setting OPENDEVIATIONBAR_USE_RUST_CH_WRITER=shadow causes dual-write with shadow comparison"
    - "Setting OPENDEVIATIONBAR_USE_RUST_CH_WRITER=live causes Python to skip _write_batch_with_retry() entirely"
    - "Feature flag is evaluated once at _create_engine() startup -- no mid-session switching"
    - "Heartbeat reports ch_writer_dead_letters and triggers LOUD alert if > 0"
    - "Dedup token parity test (Phase 34) still passes"
    - "Dead-letter round-trip test (Phase 35) still passes"
  artifacts:
    - path: "python/opendeviationbar/sidecar/__init__.py"
      provides: "Feature flag read in _create_engine(), gated writes in _flush_all_batches() and inline batch path"
      contains: "OPENDEVIATIONBAR_USE_RUST_CH_WRITER"
    - path: "python/opendeviationbar/sidecar/recovery.py"
      provides: "6-tuple return from _restart_engine_with_recovery()"
      contains: "rust_ch_writer_mode"
    - path: "python/opendeviationbar/sidecar/sync_flush.py"
      provides: "Feature flag gating for REST bar flush"
      contains: "rust_ch_writer_mode"
    - path: "scripts/heartbeat/checks/sidecar.py"
      provides: "check_ch_writer_metrics() function"
      contains: "ch_writer_dead_letters_ok"
    - path: "scripts/heartbeat/config.py"
      provides: "ch_writer_dead_letters_ok in CRITICAL_KEYS"
      contains: "ch_writer_dead_letters_ok"
    - path: "deploy/sidecar.env"
      provides: "Feature flag env var"
      contains: "OPENDEVIATIONBAR_USE_RUST_CH_WRITER"
    - path: "tests/test_sidecar_ch_writer_flag.py"
      provides: "Unit tests for feature flag routing"
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/recovery.py"
      via: "6-tuple return from _create_engine → _restart_engine_with_recovery"
      pattern: "rust_ch_writer_mode"
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/sync_flush.py"
      via: "rust_ch_writer_mode parameter to _sync_flush_rest_bars"
      pattern: "rust_ch_writer_mode"
    - from: "scripts/heartbeat/checks/sidecar.py"
      to: "scripts/heartbeat/config.py"
      via: "ch_writer_dead_letters_ok key in CRITICAL_KEYS"
      pattern: "ch_writer_dead_letters_ok"
---

<objective>
Wire the Rust ClickHouse writer into the Python sidecar via a 3-mode feature flag, add shadow comparison for dual-write validation, extend heartbeat monitoring for Rust writer metrics, and verify existing Phase 34/35 tests still pass.

Purpose: Enable shadow mode deployment on bigblack where both Rust and Python write bars, allowing validation before full cutover.
Output: Feature-flagged sidecar, heartbeat monitoring, unit tests, deploy env var.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/37-python-integration-staging-validation/37-RESEARCH.md

@python/opendeviationbar/sidecar/**init**.py
@python/opendeviationbar/sidecar/recovery.py
@python/opendeviationbar/sidecar/sync_flush.py
@python/opendeviationbar/sidecar/arrow_flush.py
@python/opendeviationbar/sidecar/clickhouse_write.py
@scripts/heartbeat/checks/sidecar.py
@scripts/heartbeat/checks/**init**.py
@scripts/heartbeat/config.py
@scripts/heartbeat/formatter.py
@deploy/sidecar.env

<interfaces>
<!-- Key contracts from existing code -->

From python/opendeviationbar/sidecar/**init**.py:

```python
def _create_engine(
    config: SidecarConfig,
    trace_id: str,
    gap_fill_results: dict | None,
    valid_pairs: set[tuple[str, int]] | None = None,
) -> tuple[object, int, dict[tuple[str, int], int], dict[str, str], object]:
    """Returns (engine, checkpoints_loaded, min_tid_guard, symbol_modes, ch_cache)."""
    # Currently 5-tuple — must become 6-tuple with rust_ch_writer_mode

# Main unpack at line 767:
engine, checkpoints_loaded, min_tid_guard, symbol_modes, ch_cache = _create_engine(...)

# Watchdog restart unpack at line 1012:
new_engine, success, new_min_tid, new_symbol_modes, new_ch_cache = _restart_engine_with_recovery(...)
```

From python/opendeviationbar/sidecar/recovery.py:

```python
def _restart_engine_with_recovery(
    engine, config, trace_id, bars_written, restart_attempt, valid_pairs=None,
) -> tuple[object | None, bool, dict[tuple[str, int], int], dict[str, str], object | None]:
    """Returns (new_engine, success_bool, min_tid_guard, symbol_modes, ch_cache)."""
    # Line 54: new_engine, _, new_min_tid_guard, new_symbol_modes, new_ch_cache = _create_engine(...)
    # Must update to 6-tuple unpack
```

From python/opendeviationbar/sidecar/sync_flush.py:

```python
def _sync_flush_rest_bars(
    engine, valid_pairs, *, symbol_modes=None, cache=None,
) -> int:
    # Dispatches to arrow or dict path — both call _write_batch_with_retry
```

From scripts/heartbeat/config.py:

```python
CRITICAL_KEYS = frozenset({
    "clickhouse", "sidecar", "kintsugi", "disk",
    "stathera_overlaps", "stathera_gaps",
    "stathera_today_gaps", "stathera_today_overlaps",
    "circuit_breaker_healthy", "checksum_ok",
})
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Feature flag in sidecar + shadow compare + sync_flush gating</name>
  <files>
    python/opendeviationbar/sidecar/__init__.py
    python/opendeviationbar/sidecar/recovery.py
    python/opendeviationbar/sidecar/sync_flush.py
    python/opendeviationbar/sidecar/arrow_flush.py
    tests/test_sidecar_ch_writer_flag.py
  </files>
  <read_first>
    python/opendeviationbar/sidecar/__init__.py
    python/opendeviationbar/sidecar/recovery.py
    python/opendeviationbar/sidecar/sync_flush.py
    python/opendeviationbar/sidecar/arrow_flush.py
  </read_first>
  <behavior>
    - Test 1: _should_python_write("off") returns True, _should_python_write("shadow") returns True, _should_python_write("live") returns False
    - Test 2: _should_shadow_compare("shadow") returns True, others return False
    - Test 3: Invalid mode string defaults to "off" (via validation logic)
    - Test 4: _create_engine return type annotation includes rust_ch_writer_mode as 6th element
  </behavior>
  <action>
    1. In `sidecar/__init__.py`, add two module-level helper functions (per INTEG-03):
       ```python
       def _should_python_write(rust_ch_writer_mode: str) -> bool:
           return rust_ch_writer_mode != "live"

       def _should_shadow_compare(rust_ch_writer_mode: str) -> bool:
           return rust_ch_writer_mode == "shadow"
       ```

    2. In `_create_engine()` (line ~180):
       - Read env var at top of function, after engine_mode: `rust_ch_writer_mode = os.environ.get("OPENDEVIATIONBAR_USE_RUST_CH_WRITER", "off").lower()`
       - Validate: if not in ("off", "shadow", "live"), warn and default to "off"
       - Log: `logger.info("Rust CH writer mode: %s", rust_ch_writer_mode)`
       - Update return type annotation to 6-tuple adding `str` for rust_ch_writer_mode
       - Update return statement (line ~520): add `rust_ch_writer_mode` as 6th element
       - Update docstring to document 6th tuple element

    3. In `run_sidecar()` (line ~767):
       - Update 5-tuple unpack to 6-tuple: add `rust_ch_writer_mode`

    4. In `_flush_all_batches()` closure (line ~884):
       - Gate Python write: `if _should_python_write(rust_ch_writer_mode):`  around the _write_batch_with_retry call
       - In live mode: still count `written += len(bars)` and call `_update_data_freshness()`
       - After write (in both shadow and off modes that wrote): if `_should_shadow_compare(rust_ch_writer_mode)`: call `_shadow_compare(symbol, threshold, bars, ch_cache)`

    5. In inline batch write path (line ~1253-1274):
       - Same gating: `if _should_python_write(rust_ch_writer_mode):` around _write_batch_with_retry
       - In live mode: still call `_update_data_freshness()`
       - If shadow: call `_shadow_compare()`

    6. Add `_shadow_compare()` function per the RESEARCH.md pattern:
       - Query CH for `count(), countDistinct(dedup_token)` in the TID range
       - Do NOT use FINAL (expensive)
       - Log warning on mismatch, debug on match
       - Wrap in try/except for connection errors

    7. In watchdog restart unpack (line ~1012):
       - Update 5-tuple unpack to 6-tuple: extract new `rust_ch_writer_mode` from recovery result
       - Assign to the outer `rust_ch_writer_mode` variable

    8. In `recovery.py`:
       - Update `_restart_engine_with_recovery` return type to 6-tuple (add `str`)
       - Update line 54 unpack to 6-tuple
       - Update line 63 failure return to include `"off"` as 6th element
       - Update line 65 success return to include `new_rust_ch_writer_mode`

    9. In `sync_flush.py`:
       - Add `rust_ch_writer_mode: str = "off"` parameter to `_sync_flush_rest_bars()`
       - Pass through to `_sync_flush_rest_bars_arrow()` and `_sync_flush_rest_bars_dict()`
       - In `_sync_flush_rest_bars_dict()`: gate `_write_batch_with_retry()` calls with `_should_python_write(rust_ch_writer_mode)` — import the helper from `__init__`
       - In live mode: still count bars but skip Python CH write (Rust already wrote via fan-out)

    10. In `arrow_flush.py`:
        - Add `rust_ch_writer_mode: str = "off"` parameter to `_sync_flush_rest_bars_arrow()`
        - Gate the `_write_batch_with_retry()` call — in live mode, skip Python write
        - Import `_should_python_write` from sidecar package

    11. In `run_sidecar()`, update the `_sync_flush_rest_bars()` call (during startup, around the fill_from_rest section) to pass `rust_ch_writer_mode` parameter.

    12. Create `tests/test_sidecar_ch_writer_flag.py` with tests for helper functions and mode validation.

    IMPORTANT: The feature flag must be read exactly ONCE in `_create_engine()` — no mid-session env var re-reads. The value flows through the return tuple to all consumers. This is per INTEG-03: "evaluated exactly once at _create_engine() startup".

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_sidecar_ch_writer_flag.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "_should_python_write" python/opendeviationbar/sidecar/__init__.py
    - grep -q "_should_shadow_compare" python/opendeviationbar/sidecar/__init__.py
    - grep -q "OPENDEVIATIONBAR_USE_RUST_CH_WRITER" python/opendeviationbar/sidecar/__init__.py
    - grep -q "rust_ch_writer_mode" python/opendeviationbar/sidecar/__init__.py
    - grep -q "rust_ch_writer_mode" python/opendeviationbar/sidecar/recovery.py
    - grep -q "rust_ch_writer_mode" python/opendeviationbar/sidecar/sync_flush.py
    - grep -q "rust_ch_writer_mode" python/opendeviationbar/sidecar/arrow_flush.py
    - grep -q "_shadow_compare" python/opendeviationbar/sidecar/__init__.py
    - test -f tests/test_sidecar_ch_writer_flag.py
  </acceptance_criteria>
  <done>
    - Feature flag read once in _create_engine(), returned as 6th tuple element
    - _flush_all_batches() and inline batch path skip Python CH write in live mode
    - Shadow comparison runs in shadow mode after each batch write
    - recovery.py returns 6-tuple matching _create_engine
    - sync_flush.py and arrow_flush.py accept and honor rust_ch_writer_mode
    - All write paths gated: _flush_all_batches, inline batch, _sync_flush_rest_bars (both arrow and dict)
    - Unit tests pass for helper functions and mode validation
  </done>
</task>

<task type="auto">
  <name>Task 2: Heartbeat ch_writer monitoring + deploy env + verify existing tests</name>
  <files>
    scripts/heartbeat/checks/sidecar.py
    scripts/heartbeat/checks/__init__.py
    scripts/heartbeat/config.py
    scripts/heartbeat/formatter.py
    deploy/sidecar.env
  </files>
  <read_first>
    scripts/heartbeat/checks/sidecar.py
    scripts/heartbeat/checks/__init__.py
    scripts/heartbeat/config.py
    scripts/heartbeat/formatter.py
    deploy/sidecar.env
  </read_first>
  <action>
    1. In `scripts/heartbeat/checks/sidecar.py`, add `check_ch_writer_metrics()`:
       ```python
       def check_ch_writer_metrics(results: dict[str, bool | str]) -> None:
           """Check Rust CH writer metrics from sidecar /health endpoint."""
           import json
           import urllib.request

           try:
               with urllib.request.urlopen("http://localhost:8081/health", timeout=5) as resp:
                   data = json.loads(resp.read())
           except (OSError, json.JSONDecodeError, urllib.error.URLError):
               return  # Sidecar not running or health endpoint unavailable

           metrics = data.get("metrics", {})
           bars_flushed = metrics.get("ch_writer_bars_flushed", 0)
           dead_letters = metrics.get("ch_writer_dead_letter_count", 0)

           if bars_flushed == 0 and dead_letters == 0:
               return  # Writer not active

           results["ch_writer_bars_flushed"] = str(bars_flushed)
           results["ch_writer_bars_failed"] = str(metrics.get("ch_writer_bars_failed", 0))
           results["ch_writer_flush_count"] = str(metrics.get("ch_writer_flush_count", 0))
           results["ch_writer_dead_letter_count"] = str(dead_letters)
           latency_us = metrics.get("ch_writer_last_flush_latency_us", 0)
           results["ch_writer_latency_ms"] = f"{latency_us / 1000:.1f}"
           results["ch_writer_dead_letters_ok"] = dead_letters == 0
       ```

    2. In `scripts/heartbeat/checks/__init__.py`:
       - Add import: `from heartbeat.checks.sidecar import check_ch_writer_metrics`
       - Add call after the existing `check_dead_letters(results)` line: `check_ch_writer_metrics(results)`

    3. In `scripts/heartbeat/config.py`:
       - Add `"ch_writer_dead_letters_ok"` to `CRITICAL_KEYS` frozenset — dead letters from Rust writer = bars lost, same severity as Python dead letters

    4. In `scripts/heartbeat/formatter.py`:
       - Add a "Rust CH Writer" section in `format_message()` after the dead_letter/ws_error section (around line 499-510)
       - Only display if `ch_writer_bars_flushed` key exists in results
       - Format: show mode (if available), bars flushed, bars failed, latency, dead letter count
       - If `ch_writer_dead_letters_ok` is False, show with cross mark emoji and bold

    5. In `deploy/sidecar.env`:
       - Add at the end:
         ```
         # Phase 37: Rust ClickHouse writer feature flag
         # Modes: off (Python only), shadow (dual-write + compare), live (Rust only)
         OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off
         ```

    6. Verify existing Phase 34/35 tests still pass (VALID-02, VALID-05):
       - Run: `cargo nextest run -p opendeviationbar-streaming test_dedup_token_parity_with_python`
       - Run: `uv run --python 3.13 pytest tests/test_dead_letter_roundtrip.py -x`
       - These tests are ALREADY SATISFIED by prior phases — we only verify they still pass

    7. Verify full Python test suite: `uv run --python 3.13 pytest tests/ -x --timeout=60`

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -q "ch_writer_dead_letters_ok" scripts/heartbeat/config.py && grep -q "check_ch_writer_metrics" scripts/heartbeat/checks/__init__.py && grep -q "OPENDEVIATIONBAR_USE_RUST_CH_WRITER" deploy/sidecar.env && uv run --python 3.13 pytest tests/test_sidecar_ch_writer_flag.py tests/test_dead_letter_roundtrip.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "check_ch_writer_metrics" scripts/heartbeat/checks/sidecar.py
    - grep -q "check_ch_writer_metrics" scripts/heartbeat/checks/__init__.py
    - grep -q "ch_writer_dead_letters_ok" scripts/heartbeat/config.py
    - grep -q "ch_writer" scripts/heartbeat/formatter.py
    - grep -q "OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off" deploy/sidecar.env
  </acceptance_criteria>
  <done>
    - check_ch_writer_metrics() queries sidecar /health endpoint for Rust writer metrics
    - ch_writer_dead_letters_ok added to CRITICAL_KEYS (LOUD alert on dead letters)
    - Formatter shows Rust CH Writer section when active
    - deploy/sidecar.env has OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off
    - Phase 34 dedup token parity test passes (VALID-02 already covered)
    - Phase 35 dead-letter round-trip test passes (VALID-05 already covered)
    - Full Python test suite passes
  </done>
</task>

</tasks>

<verification>
1. Feature flag routing: `grep -c "rust_ch_writer_mode" python/opendeviationbar/sidecar/__init__.py` should be >= 5 (read, return, unpack, two write paths)
2. Recovery 6-tuple: `grep "rust_ch_writer_mode" python/opendeviationbar/sidecar/recovery.py` shows updated return
3. Sync flush gated: `grep "rust_ch_writer_mode" python/opendeviationbar/sidecar/sync_flush.py` shows parameter
4. Heartbeat monitoring: `grep "ch_writer_dead_letters_ok" scripts/heartbeat/config.py` shows CRITICAL key
5. Deploy env: `grep "OPENDEVIATIONBAR_USE_RUST_CH_WRITER" deploy/sidecar.env` shows flag
6. Tests pass: `uv run --python 3.13 pytest tests/test_sidecar_ch_writer_flag.py tests/test_dead_letter_roundtrip.py -x`
</verification>

<success_criteria>

- Feature flag read once in \_create_engine(), flows through 6-tuple to all write paths
- All four Python CH write paths gated: \_flush_all_batches, inline batch, sync_flush dict, sync_flush arrow
- Shadow comparison queries CH after dual-write (no FINAL, count distinct dedup_tokens)
- Heartbeat triggers LOUD alert on ch_writer_dead_letters > 0
- Phase 34 dedup parity test and Phase 35 dead-letter round-trip test both pass
- Deploy sidecar.env starts with flag=off (safe default)
  </success_criteria>

<output>
After completion, create `.planning/phases/37-python-integration-staging-validation/37-01-SUMMARY.md`
</output>
