---
phase: 37-python-integration-staging-validation
plan: "01"
subsystem: sidecar
tags: [feature-flag, clickhouse-writer, heartbeat, dual-write, shadow-compare]

requires:
  - phase: 34-clickhouse-sink-core
    provides: "ClickHouseWriter with BarSink trait, dedup token generation"
  - phase: 35-dead-letter-niffler
    provides: "Niffler dead-letter fallback in Rust"
  - phase: 36-fan-out-metrics-bridge
    provides: "dispatch_to_sinks() fan-out, ch_writer_* metrics in health endpoint"
provides:
  - "3-mode feature flag (off/shadow/live) for Rust CH writer in sidecar"
  - "Shadow comparison function for dual-write validation"
  - "Heartbeat monitoring for Rust CH writer metrics (CRITICAL on dead letters)"
  - "Deploy env var OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off (safe default)"
affects: [37-02-PLAN, deploy, heartbeat]

tech-stack:
  added: []
  patterns:
    - "Feature flag read once at _create_engine(), flows through 6-tuple return"
    - "ch_writer_flag.py module for flag helpers (not in __init__.py)"
    - "Shadow compare queries count+countDistinct(dedup_token) without FINAL"

key-files:
  created:
    - python/opendeviationbar/sidecar/ch_writer_flag.py
    - tests/test_sidecar_ch_writer_flag.py
  modified:
    - python/opendeviationbar/sidecar/__init__.py
    - python/opendeviationbar/sidecar/recovery.py
    - python/opendeviationbar/sidecar/sync_flush.py
    - python/opendeviationbar/sidecar/arrow_flush.py
    - scripts/heartbeat/checks/sidecar.py
    - scripts/heartbeat/checks/__init__.py
    - scripts/heartbeat/config.py
    - scripts/heartbeat/formatter.py
    - deploy/sidecar.env

key-decisions:
  - "Feature flag helpers in dedicated ch_writer_flag.py module (not __init__.py) per INIT-MONOLITH guard"
  - "Shadow compare uses count+countDistinct(dedup_token) without FINAL to avoid expensive queries"
  - "ch_writer_dead_letters_ok in CRITICAL_KEYS -- same severity as Python dead letters"

patterns-established:
  - "6-tuple return from _create_engine() for extensible engine config flow"
  - "Feature flag gating pattern: _should_python_write() / _should_shadow_compare()"

requirements-completed: [INTEG-03, VALID-01, VALID-02, VALID-04, VALID-05]

duration: 14min
completed: 2026-03-29
---

# Phase 37 Plan 01: Python Integration + Staging Validation Summary

**3-mode feature flag (off/shadow/live) gating all 4 sidecar CH write paths, with shadow comparison, heartbeat LOUD alerts on Rust writer dead letters, and verified Phase 34/35 test parity**

## Performance

- **Duration:** 14 min
- **Started:** 2026-03-29T11:22:09Z
- **Completed:** 2026-03-29T11:36:08Z
- **Tasks:** 2
- **Files modified:** 11

## Accomplishments

- Feature flag OPENDEVIATIONBAR_USE_RUST_CH_WRITER read once at \_create_engine(), returned as 6th tuple element, flowing to all write paths
- All 4 Python CH write paths gated: \_flush_all_batches, inline batch, sync_flush dict, sync_flush arrow
- Shadow comparison queries CH for count+countDistinct(dedup_token) in TID range after dual-write
- Heartbeat check_ch_writer_metrics() queries sidecar /health endpoint, ch_writer_dead_letters_ok in CRITICAL_KEYS
- Phase 34 dedup token parity test passes (clickhouse-sink feature), Phase 35 dead-letter round-trip test passes

## Task Commits

Each task was committed atomically:

1. **Task 1: Feature flag in sidecar + shadow compare + sync_flush gating (TDD)**
   - `427d823` (test: add failing tests for flag routing - RED)
   - `711462f` (feat: wire Rust CH writer feature flag into sidecar write paths - GREEN)
2. **Task 2: Heartbeat ch_writer monitoring + deploy env + verify existing tests** - `27a40f3` (feat)

## Files Created/Modified

- `python/opendeviationbar/sidecar/ch_writer_flag.py` - Feature flag helpers: \_should_python_write,\_should_shadow_compare, \_validate_rust_ch_writer_mode,\_shadow_compare
- `python/opendeviationbar/sidecar/__init__.py` - 6-tuple return from \_create_engine(), gated \_flush_all_batches and inline batch writes
- `python/opendeviationbar/sidecar/recovery.py` - 6-tuple return matching \_create_engine
- `python/opendeviationbar/sidecar/sync_flush.py` - rust_ch_writer_mode parameter, gated store_bars_batch and_write_batch_with_retry
- `python/opendeviationbar/sidecar/arrow_flush.py` - rust_ch_writer_mode parameter, gated Arrow write path
- `scripts/heartbeat/checks/sidecar.py` - check_ch_writer_metrics() querying /health for Rust writer metrics
- `scripts/heartbeat/checks/__init__.py` - Wire check_ch_writer_metrics into run_checks()
- `scripts/heartbeat/config.py` - ch_writer_dead_letters_ok in CRITICAL_KEYS
- `scripts/heartbeat/formatter.py` - Rust CH Writer section in Telegram messages
- `deploy/sidecar.env` - OPENDEVIATIONBAR_USE_RUST_CH_WRITER=off (safe default)
- `tests/test_sidecar_ch_writer_flag.py` - 9 unit tests for feature flag routing

## Decisions Made

- Feature flag helpers placed in dedicated `ch_writer_flag.py` module rather than `__init__.py` to comply with INIT-MONOLITH guard
- Shadow compare uses `count()` and `countDistinct(dedup_token)` without FINAL to avoid expensive full-table scans
- `ch_writer_dead_letters_ok` added to CRITICAL_KEYS (same blast radius as Python dead letters -- bars lost)

## Deviations from Plan

None - plan executed exactly as written.

## Issues Encountered

- Pre-existing Rust compilation errors in opendeviationbar-streaming test targets (gap_edge_cases, reconnect_gap_fill, command_channel_tests) unrelated to Phase 37. The dedup_token_parity test requires `--features clickhouse-sink` to compile (CompletedBar behind `binance-integration` feature gate).
- Pre-existing Python import errors in kintsugi and ouroboros test files due to uncommitted working tree changes. All Phase 37-specific tests pass cleanly.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Feature flag is wired and defaults to "off" (safe) -- ready for shadow mode testing on bigblack
- Phase 37-02 can proceed with staging deployment verification
- All write paths gated -- switching to "shadow" or "live" only requires changing the env var

---

_Phase: 37-python-integration-staging-validation_
_Completed: 2026-03-29_
