---
phase: 36-fan-out-pyo3-bridge
plan: "02"
type: execute
wave: 2
depends_on: ["36-01"]
files_modified:
  - crates/opendeviationbar-streaming/src/stream_manager.rs
  - src/stream_manager_bindings.rs
  - Cargo.toml
autonomous: true
requirements: [INTEG-04]

must_haves:
  truths:
    - "Python calling stream_manager.get_metrics() sees ch_writer_* keys updating in real time"
    - "When clickhouse-sink feature is disabled, get_metrics() works unchanged (no ch_writer_* keys)"
    - "Root Cargo.toml streaming feature propagates clickhouse-sink to opendeviationbar-streaming"
  artifacts:
    - path: "crates/opendeviationbar-streaming/src/stream_manager.rs"
      provides: "get_combined_metrics() includes ch_writer_* counters from FlushThreadMetrics"
      contains: "ch_writer_bars_flushed"
    - path: "Cargo.toml"
      provides: "streaming feature includes clickhouse-sink"
      contains: "clickhouse-sink"
  key_links:
    - from: "stream_manager.rs get_combined_metrics()"
      to: "FlushThreadMetrics"
      via: "LiveBarEngine.ch_writer_metrics() Arc reference"
      pattern: "ch_writer_metrics"
    - from: "stream_manager_bindings.rs get_metrics()"
      to: "stream_manager.rs get_combined_metrics()"
      via: "existing PyDict conversion (no change needed)"
      pattern: "get_combined_metrics"
---

<objective>
Expose ClickHouse writer metrics through the StreamManager -> PyO3 bridge so Python heartbeat can monitor Rust writer health.

Purpose: Python heartbeat reads `stream_manager.get_metrics()` dict. Adding `ch_writer_*` keys lets heartbeat monitor flush latency, failure counts, and dead-letter status without any Python code changes (heartbeat already reads all dict keys).

Output: StreamManager.get_combined_metrics() extended with 6 CH writer metrics, root Cargo.toml wiring clickhouse-sink into the streaming feature.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/36-fan-out-pyo3-bridge/36-RESEARCH.md
@.planning/phases/36-fan-out-pyo3-bridge/36-01-PLAN.md

@crates/opendeviationbar-streaming/src/stream_manager.rs
@src/stream_manager_bindings.rs

<interfaces>
<!-- From Plan 01 outputs -->

From live_engine/mod.rs (created in Plan 01):

```rust
impl LiveBarEngine {
    #[cfg(feature = "clickhouse-sink")]
    pub fn ch_writer_metrics(&self) -> Option<&Arc<FlushThreadMetrics>> { ... }
}
```

From flush_thread.rs:

```rust
pub struct FlushThreadMetrics {
    pub bars_flushed: AtomicU64,
    pub bars_failed: AtomicU64,
    pub flush_count: AtomicU64,
    pub last_flush_latency_us: AtomicU64,
    pub dead_letter_count: AtomicU64,
    pub dead_letter_replayed: AtomicU64,
}
```

From stream_manager.rs existing get_combined_metrics():

```rust
pub fn get_combined_metrics(&self) -> HashMap<String, u64> {
    // Returns: trades_received, bars_emitted, reconnections, dropped_bars,
    //          gap_fills, gap_trades_recovered, continuity_verified,
    //          continuity_gaps_detected, shadow_probes, shadow_divergences,
    //          rate_limiter_remaining
}
```

From stream_manager_bindings.rs existing get_metrics():

```rust
// Already converts get_combined_metrics() HashMap to Python dict
// Adding keys to get_combined_metrics() automatically flows to Python
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Metrics bridge in StreamManager + Cargo.toml wiring</name>
  <files>crates/opendeviationbar-streaming/src/stream_manager.rs, Cargo.toml</files>
  <read_first>
    - crates/opendeviationbar-streaming/src/stream_manager.rs (full file -- especially get_combined_metrics() and new())
    - crates/opendeviationbar-streaming/src/live_engine/mod.rs (ch_writer_metrics() method added in Plan 01)
    - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs (FlushThreadMetrics fields)
    - Cargo.toml (root -- features section, streaming feature definition)
    - src/stream_manager_bindings.rs (get_metrics() -- verify it already proxies get_combined_metrics())
  </read_first>
  <behavior>
    - get_combined_metrics() includes "ch_writer_bars_flushed" key when clickhouse-sink feature enabled
    - get_combined_metrics() includes "ch_writer_bars_failed" key when clickhouse-sink feature enabled
    - get_combined_metrics() includes "ch_writer_flush_count" key when clickhouse-sink feature enabled
    - get_combined_metrics() includes "ch_writer_last_flush_latency_us" key when clickhouse-sink feature enabled
    - get_combined_metrics() includes "ch_writer_dead_letter_count" key when clickhouse-sink feature enabled
    - get_combined_metrics() includes "ch_writer_dead_letter_replayed" key when clickhouse-sink feature enabled
    - get_combined_metrics() still works unchanged when clickhouse-sink feature disabled (no ch_writer_* keys)
  </behavior>
  <action>
    **1. StreamManager struct (stream_manager.rs) -- store CH writer metrics reference:**

    Add a field to `StreamManager`:
    ```rust
    #[cfg(feature = "clickhouse-sink")]
    ch_writer_metrics: Option<Arc<crate::clickhouse_writer::flush_thread::FlushThreadMetrics>>,
    ```

    In `StreamManager::new()`, after creating the engine, extract the CH writer metrics:
    ```rust
    #[cfg(feature = "clickhouse-sink")]
    let ch_writer_metrics = engine.ch_writer_metrics().cloned();
    ```

    Store it in the struct constructor.

    **2. get_combined_metrics() -- extend with CH writer counters:**

    At the end of `get_combined_metrics()`, add:
    ```rust
    #[cfg(feature = "clickhouse-sink")]
    if let Some(ref ch_metrics) = self.ch_writer_metrics {
        m.insert("ch_writer_bars_flushed".into(),
            ch_metrics.bars_flushed.load(Ordering::Relaxed));
        m.insert("ch_writer_bars_failed".into(),
            ch_metrics.bars_failed.load(Ordering::Relaxed));
        m.insert("ch_writer_flush_count".into(),
            ch_metrics.flush_count.load(Ordering::Relaxed));
        m.insert("ch_writer_last_flush_latency_us".into(),
            ch_metrics.last_flush_latency_us.load(Ordering::Relaxed));
        m.insert("ch_writer_dead_letter_count".into(),
            ch_metrics.dead_letter_count.load(Ordering::Relaxed));
        m.insert("ch_writer_dead_letter_replayed".into(),
            ch_metrics.dead_letter_replayed.load(Ordering::Relaxed));
    }
    ```

    The metric key names use `ch_writer_` prefix (per RESEARCH.md pitfall 4) to avoid collision with existing engine metrics like `dropped_bars`.

    **3. PyO3 get_metrics() -- NO CHANGES NEEDED:**

    `PyStreamManager::get_metrics()` in `stream_manager_bindings.rs` already calls `manager.get_combined_metrics()` and converts the entire HashMap to a Python dict. Adding keys to the Rust side automatically flows to Python. Verify this by reading the file -- do not modify it unless the pattern has changed.

    **4. Root Cargo.toml -- wire clickhouse-sink into streaming feature:**

    In the root `Cargo.toml` features section, update the `streaming` feature to include `clickhouse-sink`:
    ```toml
    streaming = ["opendeviationbar-streaming", "opendeviationbar-streaming/binance-integration", "opendeviationbar-streaming/clickhouse-sink", "opendeviationbar-providers", "tokio", "tokio-util"]
    ```

    This ensures that when `maturin develop` builds the PyO3 extension (which enables `streaming` by default), the `clickhouse-sink` feature is automatically enabled in the streaming crate.

    **5. Tests:**

    Add a test in `stream_manager.rs` tests section:
    ```rust
    #[cfg(feature = "clickhouse-sink")]
    #[test]
    fn test_combined_metrics_includes_ch_writer_keys() {
        let engine_config = LiveEngineConfig::new(vec!["BTCUSDT".into()], vec![250]);
        let limiter = shared_binance_limiter();
        let config = StreamManagerConfig::new(engine_config, limiter);
        let sm = StreamManager::new(config);

        let metrics = sm.get_combined_metrics();
        // CH writer keys should be present (possibly all zeros if no CH hosts configured)
        // When OPENDEVIATIONBAR_CH_HOSTS is not set, ch_writer_metrics is None,
        // so these keys may or may not be present depending on env.
        // Test the existing keys are still there:
        assert!(metrics.contains_key("trades_received"));
        assert!(metrics.contains_key("bars_emitted"));
        assert!(metrics.contains_key("continuity_verified"));
    }
    ```

    The CH writer keys only appear when `OPENDEVIATIONBAR_CH_HOSTS` is set (which it won't be in CI). The test verifies the metrics function doesn't crash and existing keys are preserved. A more targeted test: create a StreamManager, manually set the ch_writer_metrics field to a fresh FlushThreadMetrics, and assert the keys appear.

  </action>
  <verify>
    <automated>cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(combined_metrics)' && cargo build --features streaming</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "ch_writer_bars_flushed" crates/opendeviationbar-streaming/src/stream_manager.rs
    - grep -q "ch_writer_bars_failed" crates/opendeviationbar-streaming/src/stream_manager.rs
    - grep -q "ch_writer_flush_count" crates/opendeviationbar-streaming/src/stream_manager.rs
    - grep -q "ch_writer_last_flush_latency_us" crates/opendeviationbar-streaming/src/stream_manager.rs
    - grep -q "ch_writer_dead_letter_count" crates/opendeviationbar-streaming/src/stream_manager.rs
    - grep -q "ch_writer_dead_letter_replayed" crates/opendeviationbar-streaming/src/stream_manager.rs
    - grep -q "clickhouse-sink" Cargo.toml
  </acceptance_criteria>
  <done>StreamManager.get_combined_metrics() includes 6 ch_writer_* keys from FlushThreadMetrics. Root Cargo.toml wires clickhouse-sink into streaming feature. PyO3 get_metrics() automatically exposes these to Python via existing dict conversion.</done>
</task>

</tasks>

<verification>
- `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink` -- all tests pass (including combined_metrics test)
- `cargo build --features streaming` -- root crate builds with clickhouse-sink propagated
- `cargo build -p opendeviationbar-streaming --features binance-integration --no-default-features` -- still compiles without clickhouse-sink
</verification>

<success_criteria>

- Python `stream_manager.get_metrics()` dict contains `ch_writer_bars_flushed`, `ch_writer_bars_failed`, `ch_writer_flush_count`, `ch_writer_last_flush_latency_us`, `ch_writer_dead_letter_count`, `ch_writer_dead_letter_replayed` keys (INTEG-04)
- Keys only appear when clickhouse-sink feature is enabled AND OPENDEVIATIONBAR_CH_HOSTS is set
- No PyO3 code changes needed -- metrics flow automatically through existing get_combined_metrics() -> PyDict path
- Root Cargo.toml streaming feature includes clickhouse-sink
  </success_criteria>

<output>
After completion, create `.planning/phases/36-fan-out-pyo3-bridge/36-02-SUMMARY.md`
</output>
