# Phase 37: Python Integration + Staging Validation - Research

**Researched:** 2026-03-28
**Domain:** Python sidecar integration, feature flag routing, heartbeat monitoring, staging validation
**Confidence:** HIGH

## Summary

Phase 37 wires the Rust ClickHouse writer (built in Phases 34-36) into the Python sidecar orchestrator via a 3-mode feature flag (`off`/`shadow`/`live`), adds dead-letter monitoring to the heartbeat, and defines the 24h staging validation checkpoint on bigblack.

The integration surface is well-defined: (1) `_create_engine()` reads the env var once and passes the mode through, (2) the main loop conditionally skips `_write_batch_with_retry()` in live mode, (3) heartbeat reads `ch_writer_*` metrics already exposed by StreamManager's `get_metrics()` (Phase 36), and (4) the deploy env file gets one new variable. Shadow mode comparison queries CH after both writers flush to diff row counts and dedup tokens. The 24h staging checkpoint is a human gate -- the planner needs a validation script checklist, not automation.

**Primary recommendation:** Implement feature flag as a simple string read in `_create_engine()`, gate Python CH writes in exactly two locations (main loop `_flush_all_batches` and startup `_sync_flush_rest_bars`), and add a `check_ch_writer()` function to `heartbeat/checks/sidecar.py` that reads metrics from the sidecar `/health` endpoint.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None explicitly locked -- all implementation choices are at Claude's discretion per CONTEXT.md.

### Claude's Discretion

All implementation choices are at Claude's discretion.

Key constraints:

- Feature flag `OPENDEVIATIONBAR_USE_RUST_CH_WRITER` with 3 modes: off/shadow/live
- Flag evaluated once at `_create_engine()` startup -- no mid-session switching
- Shadow mode: both Rust and Python write, Python compares results
- Live mode: Python skips `_write_batch_with_retry()` entirely
- 24h staging on bigblack is a human-action checkpoint (cannot be automated)
- Heartbeat must report ch_dead_letters and trigger LOUD alert if > 0
- Stathera audit verifies zero new gaps/overlaps after cutover
- Dedup token parity test (VALID-02) is a Rust unit test (already covered by Phase 34)
- Dead-letter round-trip test (VALID-05) is already covered by Phase 35

### Deferred Ideas (OUT OF SCOPE)

None.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID       | Description                                                                                                                                                               | Research Support                                                                                       |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| INTEG-03 | Feature flag `OPENDEVIATIONBAR_USE_RUST_CH_WRITER` with three modes: `off` (Python only), `shadow` (Rust writes + Python verifies), `live` (Rust writes, Python skips CH) | Feature flag read in `_create_engine()`, gates in `_flush_all_batches()` and `_sync_flush_rest_bars()` |
| VALID-01 | Shadow mode dual-write -- Rust and Python both write bars, Python queries CH and compares Rust-written vs Python-written bars (zero diff)                                 | Shadow comparison via dedup_token query after each batch flush                                         |
| VALID-02 | Dedup token parity test in Rust test suite -- fixed inputs produce identical MD5 hashes to Python `hashlib.md5().hexdigest()`                                             | Already covered by Phase 34 (Rust unit test) -- Phase 37 only needs to verify it passes                |
| VALID-03 | Staging validation on bigblack with shadow mode for minimum 24 hours before enabling `live` mode                                                                          | Human checkpoint with validation script checklist                                                      |
| VALID-04 | Stathera audit after cutover shows zero new gaps or overlaps -- same data integrity guarantees as Python writer                                                           | Existing `collect_stathera()` in heartbeat + manual `detect_gaps.py` run                               |
| VALID-05 | Dead-letter round-trip test -- Rust serializes to Parquet, Python Charon replays successfully, bars appear in CH with correct values                                      | Already covered by Phase 35 -- Phase 37 only needs to verify it passes                                 |

</phase_requirements>

## Architecture Patterns

### Feature Flag Flow

```
Environment:
  OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off|shadow|live
  (default: off)

Startup (_create_engine):
  1. Read env var once → store as rust_ch_writer_mode
  2. Pass mode to StreamManager constructor (Rust already has the sink)
  3. Return mode as part of _create_engine 5-tuple → 6-tuple
  4. Log: "Rust CH writer mode: {mode}"

Main loop (run_sidecar):
  rust_ch_writer_mode stored in local variable after _create_engine returns

_flush_all_batches():
  if rust_ch_writer_mode == "live":
      skip _write_batch_with_retry() entirely
      (Rust ClickHouseWriterSink already wrote via fan-out in LiveBarEngine)
  elif rust_ch_writer_mode == "shadow":
      still call _write_batch_with_retry()
      then run shadow comparison query
  else:  # "off"
      normal Python write (current behavior)

_sync_flush_rest_bars():
  Same gating: live mode skips Python CH write for REST bars
```

### Integration Points (Exact Locations)

**1. `_create_engine()` in `sidecar/__init__.py` (line ~195-520)**

Read the env var at the top of the function, after `engine_mode` is read:

```python
rust_ch_writer_mode = os.environ.get(
    "OPENDEVIATIONBAR_USE_RUST_CH_WRITER", "off"
).lower()
if rust_ch_writer_mode not in ("off", "shadow", "live"):
    logger.warning(
        "invalid OPENDEVIATIONBAR_USE_RUST_CH_WRITER=%r, defaulting to off",
        rust_ch_writer_mode,
    )
    rust_ch_writer_mode = "off"
logger.info("Rust CH writer mode: %s", rust_ch_writer_mode)
```

Return it as part of the tuple (currently 5-tuple, becomes 6-tuple):

```python
return engine, loaded, min_tid_guard, symbol_modes, ch_cache, rust_ch_writer_mode
```

**2. `run_sidecar()` in `sidecar/__init__.py` (line ~767)**

Unpack the 6-tuple:

```python
engine, checkpoints_loaded, min_tid_guard, symbol_modes, ch_cache, rust_ch_writer_mode = _create_engine(...)
```

**3. `_flush_all_batches()` closure (line ~884-903)**

Gate the Python write:

```python
def _flush_all_batches() -> int:
    nonlocal last_batch_flush_time
    written = 0
    for (symbol, threshold), bars in bar_batch_buffer.items():
        if not bars:
            continue
        mode = symbol_modes.get(symbol, "aion")
        if rust_ch_writer_mode != "live":
            # Python writes in "off" and "shadow" modes
            if _write_batch_with_retry(
                symbol, threshold, bars, cache=ch_cache, ouroboros_mode=mode,
            ):
                written += len(bars)
                _update_data_freshness(symbol, threshold, bars)
        else:
            # "live" mode: Rust already wrote, just update freshness
            written += len(bars)
            _update_data_freshness(symbol, threshold, bars)
        if rust_ch_writer_mode == "shadow":
            _shadow_compare(symbol, threshold, bars, ch_cache)
    bar_batch_buffer.clear()
    last_batch_flush_time = time.monotonic()
    return written
```

**4. Inline batch write in main loop (line ~1253-1274)**

Same gate for the per-batch write path:

```python
if len(bar_batch_buffer[batch_key]) >= batch_max_size:
    batch = bar_batch_buffer[batch_key]
    _batch_mode = symbol_modes.get(symbol, "aion")
    if rust_ch_writer_mode != "live":
        wrote = _write_batch_with_retry(
            symbol, threshold, batch, cache=ch_cache, ouroboros_mode=_batch_mode,
        )
        if wrote:
            _update_data_freshness(symbol, threshold, batch)
    else:
        _update_data_freshness(symbol, threshold, batch)
    if rust_ch_writer_mode == "shadow":
        _shadow_compare(symbol, threshold, batch, ch_cache)
    bar_batch_buffer[batch_key] = []
```

**5. `_sync_flush_rest_bars()` in `sync_flush.py` and `arrow_flush.py`**

Both paths call `_write_batch_with_retry()` or `store_bars_batch()`. Need to accept `rust_ch_writer_mode` parameter and skip Python CH write in live mode. The Rust writer handles REST bars via fan-out in LiveBarEngine -- `drain_bars_arrow()` returns bars that the Rust sink has already flushed.

### Shadow Mode Comparison

Shadow comparison queries CH after both Rust and Python write to verify row parity. The Rust writer uses the same dedup token (MD5 of `{symbol}_{threshold}_{start_ts}_{end_ts}_{ouroboros_mode}`), so ReplacingMergeTree should deduplicate identical bars. The comparison checks:

```python
def _shadow_compare(
    symbol: str,
    threshold: int,
    bars: list[dict],
    ch_cache: object,
) -> None:
    """Compare Rust-written vs Python-written bars after dual-write."""
    if not bars:
        return
    first_tid = bars[0].get("first_agg_trade_id", 0)
    last_tid = bars[-1].get("last_agg_trade_id", 0)
    if first_tid <= 0 or last_tid <= 0:
        return
    try:
        result = ch_cache.client.query(
            f"SELECT count(), countDistinct(dedup_token) "
            f"FROM opendeviationbar_cache.open_deviation_bars "
            f"WHERE symbol = '{symbol}' AND threshold_decimal_bps = {threshold} "
            f"AND first_agg_trade_id >= {first_tid} "
            f"AND last_agg_trade_id <= {last_tid}"
        )
        total_rows, distinct_tokens = result.result_rows[0]
        expected = len(bars)
        if distinct_tokens != expected:
            logger.warning(
                "SHADOW MISMATCH %s@%d: expected %d distinct tokens, got %d (total rows %d)",
                symbol, threshold, expected, distinct_tokens, total_rows,
            )
        else:
            logger.debug(
                "shadow compare %s@%d: %d bars match", symbol, threshold, expected
            )
    except (OSError, RuntimeError, ConnectionError):
        logger.warning("shadow compare query failed for %s@%d", symbol, threshold, exc_info=True)
```

**Important**: Shadow comparison should NOT use `FINAL` (expensive). Query raw rows and check distinct dedup_tokens matches expected count. Duplicates from dual-write are expected before ReplacingMergeTree merge -- the check verifies both writers produced bars for the same trade-ID ranges.

### Heartbeat: ch_writer Metrics Display

**Where**: `scripts/heartbeat/checks/sidecar.py` -- add `check_ch_writer_metrics()` alongside existing `check_dead_letters()`.

**How**: Heartbeat already queries sidecar `/health` endpoint (in `junction.py`). The `ch_writer_*` metrics are in `StreamManager.get_metrics()` and exposed via the health endpoint. The heartbeat should:

1. Query `/health` or `/status` for `ch_writer_dead_letter_count`
2. If > 0, set `results["ch_writer_dead_letters_ok"] = False`
3. Display `ch_writer_bars_flushed`, `ch_writer_flush_count`, `ch_writer_last_flush_latency_us` in the formatter

**CRITICAL_KEYS addition**: Add `"ch_writer_dead_letters_ok"` to `CRITICAL_KEYS` in `config.py` -- dead letters from the Rust writer mean bars failed to reach CH, same severity as Python dead letters.

**Formatter addition** in `formatter.py`: Add a "Rust CH Writer" section between ClickHouse and Stathera:

```
Rust CH Writer
  mode: shadow | 4,231 bars flushed | 0 failed
  latency: 23ms | 0 dead letters
```

### Deploy Env Var

Add to `deploy/sidecar.env` (sidecar-specific, not shared):

```ini
# Phase 37: Rust ClickHouse writer feature flag
# Modes: off (Python only), shadow (dual-write + compare), live (Rust only)
OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off
```

Start as `off`, switch to `shadow` for the 24h validation, then `live` after validation passes.

### 24h Staging Validation Checkpoint

This is a human-action gate. The validation procedure:

**Pre-shadow (before enabling shadow mode):**

1. `python scripts/detect_gaps.py --json > /tmp/pre-shadow-stathera.json` -- baseline Stathera state
2. Note current bar counts per (symbol, threshold) pair

**Enable shadow mode:** 3. Set `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=shadow` in `deploy/sidecar.env` 4. `mise run deploy:bigblack` (restarts sidecar)

**During shadow (monitor for 24h):** 5. Watch heartbeat Telegram for any SHADOW MISMATCH warnings 6. `journalctl --user -u opendeviationbar-sidecar --since '24 hours ago' | grep -i shadow` 7. Check: zero "SHADOW MISMATCH" lines 8. Check: `ch_writer_bars_flushed > 0` in heartbeat (Rust writer is active) 9. Check: `ch_writer_dead_letter_count == 0` (no Rust-side failures)

**Post-shadow validation:** 10. `python scripts/detect_gaps.py --json > /tmp/post-shadow-stathera.json` 11. Compare: `diff /tmp/pre-shadow-stathera.json /tmp/post-shadow-stathera.json` 12. Verify: no NEW gaps or overlaps introduced during shadow period 13. Verify: bar counts increased normally (no dropped bars)

**Enable live mode:** 14. Set `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=live` in `deploy/sidecar.env` 15. `mise run deploy:bigblack`

**Post-live validation (first hour):** 16. Watch heartbeat for stathera_gaps, stathera_overlaps 17. Verify bars still being written: `ch_writer_bars_flushed` increasing 18. Verify Python dead letters at 0 (Python no longer writing)

## Don't Hand-Roll

| Problem                       | Don't Build                  | Use Instead                                                 | Why                                                                      |
| ----------------------------- | ---------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------ |
| Shadow comparison query       | Custom SQL diffing framework | Simple dedup_token count query                              | ReplacingMergeTree handles dedup; just verify both writers produced bars |
| Metrics bridge                | Custom PyO3 metrics struct   | `get_metrics()` HashMap -> PyDict (Phase 36)                | Already implemented, auto-exposes ch*writer*\* keys                      |
| Dead-letter monitoring        | Separate dead-letter scanner | Existing `check_dead_letters()` pattern + `/health` metrics | Same pattern, just add Rust-side metric check                            |
| Staging validation automation | CI/CD pipeline for 24h test  | Manual checklist + existing scripts                         | 24h wall-clock; human judgment needed                                    |

## Common Pitfalls

### Pitfall 1: Forgetting to gate \_sync_flush_rest_bars

**What goes wrong:** REST bars at startup still written by Python even in live mode, causing duplicates before ReplacingMergeTree merges.
**Why it happens:** `_sync_flush_rest_bars()` is called once during startup and uses separate write paths from the main loop.
**How to avoid:** Pass `rust_ch_writer_mode` to `_sync_flush_rest_bars()` via parameter. Gate both `arrow_flush.py` and `sync_flush.py` paths.
**Warning signs:** Double bar counts after sidecar restart in live mode.

### Pitfall 2: Shadow comparison using FINAL

**What goes wrong:** `SELECT ... FROM table FINAL` is expensive (full merge). Doing it per-batch causes massive CH load.
**Why it happens:** Developer instinct to get deduplicated view.
**How to avoid:** Query without FINAL, count distinct dedup_tokens. Expect 2x rows (one from each writer) before merge.
**Warning signs:** CH active queries spike during shadow mode.

### Pitfall 3: Not gating the inline batch write path

**What goes wrong:** Main loop has TWO write paths -- the batch flush and the per-threshold inline write (line ~1256). Missing one means Python still writes in live mode.
**Why it happens:** `_flush_all_batches()` is the timeout path; inline write triggers on batch_max_size.
**How to avoid:** Gate both paths. Extract the gating logic into a helper.
**Warning signs:** Python dead letters still appearing in live mode (Python write failures).

### Pitfall 4: Heartbeat querying get_metrics() directly

**What goes wrong:** Heartbeat runs as a separate process -- it cannot call `engine.get_metrics()` on the sidecar's engine.
**Why it happens:** Confusion between in-process access (sidecar) and out-of-process access (heartbeat).
**How to avoid:** Heartbeat must query the sidecar `/health` or `/status` HTTP endpoint, which already serves metrics.
**Warning signs:** `ImportError` or `RuntimeError` in heartbeat when trying to access engine.

### Pitfall 5: Returning 6-tuple from \_create_engine breaks watchdog restart

**What goes wrong:** `_restart_engine_with_recovery()` also returns the same tuple. Both must be updated to 6-tuple.
**Why it happens:** `_restart_engine_with_recovery` in `recovery.py` calls `_create_engine` and returns its result.
**How to avoid:** Update `recovery.py` to unpack and return the 6-tuple. Update the unpacking in the watchdog restart block (line ~1012).
**Warning signs:** `ValueError: not enough values to unpack` on watchdog restart.

## Code Examples

### Feature flag gating helper

```python
def _should_python_write(rust_ch_writer_mode: str) -> bool:
    """Whether Python should write to ClickHouse."""
    return rust_ch_writer_mode != "live"

def _should_shadow_compare(rust_ch_writer_mode: str) -> bool:
    """Whether to run shadow comparison after write."""
    return rust_ch_writer_mode == "shadow"
```

### Heartbeat ch_writer check

```python
def check_ch_writer_metrics(results: dict[str, bool | str]) -> None:
    """Check Rust CH writer metrics from sidecar /health endpoint."""
    import json
    import urllib.request

    try:
        with urllib.request.urlopen("http://localhost:8081/health", timeout=5) as resp:
            data = json.loads(resp.read())
    except (OSError, json.JSONDecodeError):
        # Sidecar not running or health endpoint unavailable
        return

    metrics = data.get("metrics", {})

    # Only report if Rust writer is active (bars_flushed > 0 or mode != off)
    bars_flushed = metrics.get("ch_writer_bars_flushed", 0)
    dead_letters = metrics.get("ch_writer_dead_letter_count", 0)

    if bars_flushed == 0 and dead_letters == 0:
        # Writer not active, skip display
        return

    results["ch_writer_bars_flushed"] = str(bars_flushed)
    results["ch_writer_bars_failed"] = str(metrics.get("ch_writer_bars_failed", 0))
    results["ch_writer_flush_count"] = str(metrics.get("ch_writer_flush_count", 0))
    results["ch_writer_dead_letter_count"] = str(dead_letters)

    latency_us = metrics.get("ch_writer_last_flush_latency_us", 0)
    results["ch_writer_latency_ms"] = f"{latency_us / 1000:.1f}"

    # LOUD alert if dead letters > 0
    results["ch_writer_dead_letters_ok"] = dead_letters == 0
```

## Validation Architecture

### Test Framework

| Property           | Value                                                                   |
| ------------------ | ----------------------------------------------------------------------- |
| Framework          | pytest + cargo test                                                     |
| Config file        | `pyproject.toml` [tool.pytest]                                          |
| Quick run command  | `pytest tests/test_sidecar.py tests/test_heartbeat_health_levels.py -x` |
| Full suite command | `mise run test-py`                                                      |

### Phase Requirements to Test Map

| Req ID   | Behavior                               | Test Type   | Automated Command                                                           | File Exists?      |
| -------- | -------------------------------------- | ----------- | --------------------------------------------------------------------------- | ----------------- |
| INTEG-03 | Feature flag routing (off/shadow/live) | unit        | `pytest tests/test_sidecar_ch_writer_flag.py -x`                            | Wave 0            |
| VALID-01 | Shadow mode comparison query           | unit        | `pytest tests/test_sidecar_ch_writer_flag.py::test_shadow_compare -x`       | Wave 0            |
| VALID-02 | Dedup token parity (Rust unit test)    | unit (Rust) | `cargo nextest run -p opendeviationbar-streaming --test dedup_token_parity` | Phase 34          |
| VALID-03 | 24h staging validation                 | manual-only | N/A (human checkpoint)                                                      | N/A               |
| VALID-04 | Stathera audit after cutover           | smoke       | `python scripts/detect_gaps.py --json`                                      | Exists            |
| VALID-05 | Dead-letter round-trip                 | unit        | `pytest tests/test_dead_letter_roundtrip.py -x`                             | Exists (Phase 35) |

### Sampling Rate

- **Per task commit:** `pytest tests/test_sidecar_ch_writer_flag.py tests/test_heartbeat_health_levels.py -x`
- **Per wave merge:** `mise run test-py`
- **Phase gate:** Full suite green + 24h staging validation (human checkpoint)

### Wave 0 Gaps

- [ ] `tests/test_sidecar_ch_writer_flag.py` -- covers INTEG-03, VALID-01 (feature flag routing and shadow compare)
- [ ] Update `tests/test_heartbeat_health_levels.py` -- add ch_writer_dead_letters_ok to CRITICAL_KEYS test

## Environment Availability

| Dependency                   | Required By                 | Available          | Version | Fallback            |
| ---------------------------- | --------------------------- | ------------------ | ------- | ------------------- |
| Python 3.13                  | All Python code             | Local only         | 3.13    | --                  |
| ClickHouse                   | Shadow comparison, Stathera | bigblack only      | --      | Tests mock CH       |
| Sidecar health endpoint      | Heartbeat ch_writer check   | bigblack only      | :8081   | Skip if unavailable |
| Rust clickhouse-sink feature | Rust CH writer              | Built with feature | --      | Feature flag = off  |

**Missing dependencies with no fallback:** None -- all code is testable locally with mocks.

**Missing dependencies with fallback:** Staging validation (VALID-03) requires bigblack -- cannot run locally, but is a human checkpoint by design.

## Open Questions

1. **Health endpoint metrics format**
   - What we know: `get_metrics()` returns a flat HashMap with `ch_writer_*` keys. The `/health` endpoint serializes this.
   - What's unclear: Exact JSON structure of `/health` response -- whether metrics are nested or flat.
   - Recommendation: Check the health endpoint handler in `http_server.py` to confirm structure. If metrics are nested under a `"metrics"` key, adjust heartbeat query accordingly. LOW risk -- the handler is visible in code.

## Sources

### Primary (HIGH confidence)

- `python/opendeviationbar/sidecar/__init__.py` -- full sidecar orchestrator, write paths, tuple returns
- `python/opendeviationbar/sidecar/clickhouse_write.py` -- Python write path implementation
- `python/opendeviationbar/sidecar/sync_flush.py` -- REST bar flush dispatch
- `python/opendeviationbar/sidecar/arrow_flush.py` -- Arrow drain path
- `crates/opendeviationbar-streaming/src/stream_manager.rs:571-598` -- ch*writer*\* metrics exposed
- `scripts/heartbeat/checks/sidecar.py` -- existing sidecar check pattern
- `scripts/heartbeat/checks/__init__.py` -- check orchestration
- `scripts/heartbeat/config.py` -- CRITICAL_KEYS, health level tiers
- `scripts/heartbeat/formatter.py` -- Telegram message format
- `deploy/sidecar.env` -- sidecar-specific env vars

### Secondary (MEDIUM confidence)

- `src/stream_manager_bindings.rs:549-565` -- PyO3 get_metrics() dict conversion

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- no new libraries, all existing patterns
- Architecture: HIGH -- integration points precisely identified with line numbers
- Pitfalls: HIGH -- based on direct code reading of write paths and tuple returns

**Research date:** 2026-03-28
**Valid until:** 2026-04-28 (stable domain, no external dependencies changing)
