# Phase 36: Fan-Out + PyO3 Bridge - Research

**Researched:** 2026-03-29
**Domain:** Rust streaming engine fan-out architecture + PyO3 metrics bridge
**Confidence:** HIGH

## Summary

Phase 36 integrates the ClickHouseWriterSink (built in Phase 34) into the live streaming pipeline alongside the existing ChannelSink (Python consumer). The key challenge is a **dual-sink fan-out** where each sink is fault-isolated -- a slow or failing CH write must never block Python from receiving bars.

The codebase already has the exact pattern needed: `OdbEngine.fan_out()` (Issue #178) iterates over a `Vec<Box<dyn BarSink>>` with per-sink error isolation. The `LiveBarEngine`, however, currently pushes bars to a `ConcurrentRingBuffer` directly from `trade_dispatch.rs`, NOT through the `BarSink` trait. The fan-out must be added at the `trade_dispatch` level (where `CompletedBar` is created) or by wrapping the ring buffer push in a `BarSink`-like pattern that also dispatches to `ClickHouseWriterSink`.

Pre-write guards exist in Python (`_run_write_guards()` in `bulk_operations.py`) and must be ported to Rust to run before `ClickHouseBarRow::from_completed_bar()`. These guards are simple range checks (timestamp microsecond scale, volume non-negative) that translate directly to Rust.

**Primary recommendation:** Add an optional `Vec<Box<dyn BarSink>>` to `LiveBarEngine` that receives bars alongside the ring buffer push. The `ClickHouseWriterSink` is instantiated conditionally via `#[cfg(feature = "clickhouse-sink")]` using `ClickHouseWriterConfig::from_env()`. Metrics flow through the existing `get_combined_metrics()` pattern in `StreamManager`.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None explicitly locked -- all implementation choices at Claude's discretion (infrastructure phase).

### Claude's Discretion

All implementation choices are at Claude's discretion. Key constraints:

- Fan-out must NOT block either sink -- if CH sink is slow, Python sink must still receive bars
- ChannelSink (Python) MUST stay active -- Python needs bars for checkpoints, SSE, Argus, Lethe, min-TID guard
- Pre-write guards run BEFORE BarSink.on_bar() to prevent invalid bars from being serialized
- Config from env vars only (OPENDEVIATIONBAR_CH_HOSTS, OPENDEVIATIONBAR_MODE, etc.) -- no PyO3 config bridge
- Ouroboros mode hardcoded to "aion" in ClickHouseBarRow (already done in row.rs)
- Metrics exposed via StreamManager.get_metrics() dict -- Python heartbeat already reads this

### Deferred Ideas (OUT OF SCOPE)

None.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID       | Description                                                                                                             | Research Support                                                                                               |
| -------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| INTEG-01 | Fan-out in LiveBarEngine -- ClickHouseWriterSink runs alongside ChannelSink                                             | `trade_dispatch.rs` bar emission point identified; `OdbEngine.fan_out()` pattern available as reference        |
| INTEG-02 | Configuration via environment variables (OPENDEVIATIONBAR_CH_HOSTS, OPENDEVIATIONBAR_MODE)                              | `ClickHouseWriterConfig::from_env()` already implements this; OPENDEVIATIONBAR_MODE read via `std::env::var()` |
| INTEG-04 | Metrics bridge to Python -- Rust writer exposes write latency, batch sizes, error counts via StreamManager metrics dict | `FlushThreadMetrics` fields mapped; `get_combined_metrics()` extension pattern documented                      |
| INTEG-05 | Pre-write guards ported to Rust: close_time_us scale validation, volume non-negative, threshold registry check          | Python guards in `bulk_operations.py` analyzed; direct Rust translation documented                             |
| INTEG-07 | Ouroboros mode hardcoded to "aion" -- day-mode fully removed from codebase                                              | Already done in `row.rs` from Phase 34; no additional work needed                                              |

</phase_requirements>

## Architecture Patterns

### Current Bar Dispatch Flow

```
WS Trade → symbol_task() → process_trade_through_processors()
    → processor.process_single_trade(&trade)
        → Ok(Some(bar)) → CompletedBar { symbol, threshold, bar }
            → bar_buffer.push(completed)  // ConcurrentRingBuffer
            → metrics.bars_emitted++
```

The ring buffer (`ConcurrentRingBuffer<CompletedBar>`) is the ONLY output from `trade_dispatch.rs`. Python consumes bars via `LiveBarEngine::next_bar()` which polls this ring buffer.

### Target Bar Dispatch Flow (Fan-Out)

```
WS Trade → symbol_task() → process_trade_through_processors()
    → processor.process_single_trade(&trade)
        → Ok(Some(bar)) → CompletedBar { symbol, threshold, bar }
            → pre_write_guard(&bar)  // Rust-side validation
            → bar_buffer.push(completed.clone())  // ChannelSink equivalent
            → for sink in extra_sinks { sink.on_bar(&completed) }  // CH sink
            → metrics.bars_emitted++
```

### Fan-Out Integration Point

**File:** `crates/opendeviationbar-streaming/src/live_engine/trade_dispatch.rs`

**Function:** `process_trade_through_processors()` -- lines 80-96 (normal bar emission) and lines 43-56 (midnight orphan emission).

Both code paths push to `bar_buffer` and must ALSO dispatch to optional sinks. The function already takes many parameters; adding `extra_sinks: &mut [Box<dyn BarSink>]` is the cleanest approach.

**Alternative approach:** Add sinks to `symbol_task()` instead. The `symbol_task` function in `symbol_task.rs` spawns per-symbol and calls `process_trade_through_processors()`. It could hold the sinks and call them after `process_trade_through_processors()` returns. However, this loses the granularity of per-bar dispatch since `process_trade_through_processors()` pushes directly to the ring buffer. The cleaner pattern is passing sinks into the dispatch function.

### OdbEngine Fan-Out Reference Pattern

`OdbEngine::fan_out()` in `engine/odb_engine.rs` (lines 189-200) provides the exact fault-isolation pattern:

```rust
fn fan_out(&mut self, bar: &CompletedBar) {
    for sink in &mut self.sinks {
        match sink.on_bar(bar) {
            Ok(()) => {}
            Err(SinkError::Recoverable(msg)) => {
                self.metrics.bars_dropped.fetch_add(1, Ordering::Relaxed);
                tracing::warn!(sink = sink.name(), %msg, "sink recoverable error");
            }
            Err(SinkError::Unrecoverable(msg)) => {
                tracing::error!(sink = sink.name(), %msg, "sink unrecoverable error");
            }
        }
    }
}
```

Key properties:

1. Each sink is called independently -- one failure does not prevent others
2. `Recoverable` errors (channel full, backpressure) are logged and counted
3. `Unrecoverable` errors are logged at ERROR level
4. The bar is passed by reference (`&CompletedBar`) -- no extra cloning needed since `ClickHouseWriterSink::on_bar()` takes `&CompletedBar` and serializes immediately

### ClickHouseWriterSink Non-Blocking Guarantee

`ClickHouseWriterSink::on_bar()` (in `sink.rs`) uses `try_send()` which is non-blocking:

- **Channel has space:** Serializes to `ClickHouseBarRow`, sends `FlushCommand::Bar(Box::new(row))` to flush thread -- returns immediately
- **Channel full:** Dead-letters to Parquet, returns `SinkError::Recoverable` -- the ring buffer push already happened so Python gets the bar
- **Channel closed:** Returns `SinkError::Unrecoverable` -- flush thread died

This guarantees the fan-out cannot block the Python bar path.

### Recommended Project Structure Changes

```
crates/opendeviationbar-streaming/src/
├── live_engine/
│   ├── mod.rs              # Add optional Vec<Box<dyn BarSink>> to LiveBarEngine
│   ├── trade_dispatch.rs   # Add sink dispatch + pre-write guard calls
│   └── symbol_task.rs      # Thread sinks through to dispatch function
├── clickhouse_writer/
│   ├── guards.rs           # NEW: Pre-write guard functions (INTEG-05)
│   └── ...                 # Existing Phase 34/35 files
├── engine/
│   └── mod.rs              # Already re-exports ClickHouseWriterSink
└── stream_manager.rs       # Add CH writer metrics to get_combined_metrics()

src/
├── stream_manager_bindings.rs  # Extend get_metrics() to include CH writer fields
└── lib.rs                      # Wire clickhouse-sink feature flag to PyO3 build
```

## Standard Stack

### Core (Already in Workspace)

| Library                      | Version   | Purpose                                            | Why Standard            |
| ---------------------------- | --------- | -------------------------------------------------- | ----------------------- |
| `opendeviationbar-streaming` | workspace | LiveBarEngine, BarSink trait, ClickHouseWriterSink | Core streaming crate    |
| `pyo3`                       | workspace | Python bindings                                    | Existing PyO3 bridge    |
| `tracing`                    | workspace | Structured logging                                 | Already used throughout |

### Supporting

No new dependencies needed. All required functionality exists in the workspace.

## Don't Hand-Roll

| Problem                   | Don't Build               | Use Instead                                                        | Why                                       |
| ------------------------- | ------------------------- | ------------------------------------------------------------------ | ----------------------------------------- |
| Fan-out error isolation   | Custom try/catch per sink | `OdbEngine::fan_out()` pattern (iterate sinks, match on SinkError) | Proven pattern already in codebase        |
| Env var config parsing    | PyO3 config bridge        | `ClickHouseWriterConfig::from_env()`                               | Already reads OPENDEVIATIONBAR_CH_HOSTS   |
| Non-blocking bar dispatch | Async channel to CH       | `SyncSender::try_send()` in ClickHouseWriterSink                   | Already implemented, backpressure handled |
| Metrics atomics           | Mutex-based metrics       | `AtomicU64` counters (existing FlushThreadMetrics)                 | Lock-free, matches existing pattern       |

## Common Pitfalls

### Pitfall 1: Cloning CompletedBar for Multiple Sinks

**What goes wrong:** `CompletedBar` contains `Arc<str>` (symbol) and `OpenDeviationBar` (large struct). Cloning per sink adds allocation overhead.
**Why it happens:** Instinct to clone before passing to each sink.
**How to avoid:** `BarSink::on_bar()` takes `&CompletedBar` -- no clone needed. The ring buffer push (which needs ownership) happens first, then sinks get references. If the ring buffer already consumes the value, consider cloning once for the ring buffer and using the original for sinks.
**Warning signs:** `clone()` calls in the hot path per bar per sink.

### Pitfall 2: Feature Flag Compilation Errors

**What goes wrong:** `#[cfg(feature = "clickhouse-sink")]` in `live_engine/` causes import errors when the feature is disabled because `clickhouse_writer` module doesn't exist.
**Why it happens:** The `clickhouse-sink` feature lives in `opendeviationbar-streaming` but `live_engine` is gated by `binance-integration` only.
**How to avoid:** Use conditional imports with `#[cfg(feature = "clickhouse-sink")]` at the use-site, not at the module level. The `engine/mod.rs` already shows the correct pattern (line 21-22).
**Warning signs:** `cargo build` without `clickhouse-sink` feature fails.

### Pitfall 3: Thread Safety of Sinks in Symbol Tasks

**What goes wrong:** `symbol_task()` runs as a tokio task (async). `BarSink` is `Send` but not `Sync`. Passing `&mut sink` across await points requires care.
**Why it happens:** Each `symbol_task` needs its own set of sinks -- sinks cannot be shared across symbols.
**How to avoid:** Create one `ClickHouseWriterSink` per engine (not per symbol). Pass it through as `Arc<Mutex<...>>` or create per-engine and fan-out at the `process_trade_through_processors()` level which is synchronous. Better yet: since the sink's `on_bar()` just does a `try_send()` to a channel, a single sink shared across symbol tasks via a cloneable sender would work. But the `BarSink` trait uses `&mut self`. Best approach: one sink instance owned by the engine, not by individual symbol tasks. The `process_trade_through_processors()` function receives sinks as a mutable slice reference.
**Warning signs:** Borrow checker errors with `&mut` across `.await`.

### Pitfall 4: Metrics Key Collision

**What goes wrong:** Adding CH writer metrics with keys that collide with existing `get_combined_metrics()` keys.
**Why it happens:** Both engine metrics and CH writer use terms like "bars_dropped".
**How to avoid:** Prefix CH writer metrics: `ch_writer_bars_flushed`, `ch_writer_bars_failed`, `ch_writer_flush_count`, `ch_writer_last_flush_latency_us`, `ch_writer_dead_letter_count`, `ch_writer_dead_letter_replayed`.
**Warning signs:** Python heartbeat reading wrong counter values.

### Pitfall 5: Pre-Write Guards Rejecting Valid Bars

**What goes wrong:** Rust pre-write guard rejects a bar that the Python path would have accepted.
**Why it happens:** Different precision or edge case handling between Python and Rust.
**How to avoid:** Port guards with exact same thresholds: `close_time_us` must be in range [1e15, 2.5e15], volumes must be >= 0. The `_guard_bar_spread_invariant()` only LOGS (never blocks) in Python, so the Rust version should also only log. The `_guard_threshold_registry()` is a registry check that the Rust side doesn't need (all bars in the engine already passed symbol registration at startup).
**Warning signs:** Bars appearing in Python's ClickHouse writes but not Rust's.

## Code Examples

### Pre-Write Guard in Rust (INTEG-05)

Porting from `bulk_operations.py` lines 42-76:

```rust
// crates/opendeviationbar-streaming/src/clickhouse_writer/guards.rs

use crate::live_engine::CompletedBar;

/// Minimum valid microsecond timestamp (approx 2001-09-09).
const MIN_US: i64 = 1_000_000_000_000_000;
/// Maximum valid microsecond timestamp (approx 2049-03-22).
const MAX_US: i64 = 2_500_000_000_000_000;

/// Pre-write validation for a completed bar before CH serialization.
/// Returns `true` if the bar passes all guards.
///
/// Guards ported from Python `_run_write_guards()` in bulk_operations.py.
pub fn validate_bar_for_write(bar: &CompletedBar) -> bool {
    let b = &bar.bar;

    // Guard 1: close_time_us scale (USEC-03)
    // Rust internal timestamps are microseconds (i64).
    if b.close_time < MIN_US || b.close_time > MAX_US {
        tracing::error!(
            symbol = %bar.symbol,
            threshold = bar.threshold_decimal_bps,
            close_time_us = b.close_time,
            "pre-write guard: close_time_us out of microsecond range"
        );
        return false;
    }
    if b.open_time < MIN_US || b.open_time > MAX_US {
        tracing::error!(
            symbol = %bar.symbol,
            threshold = bar.threshold_decimal_bps,
            open_time_us = b.open_time,
            "pre-write guard: open_time_us out of microsecond range"
        );
        return false;
    }

    // Guard 2: volume non-negative (Issue #88)
    let vol = b.volume.to_f64();
    if vol < 0.0 {
        tracing::error!(
            symbol = %bar.symbol,
            threshold = bar.threshold_decimal_bps,
            volume = vol,
            "pre-write guard: negative volume (overflow)"
        );
        return false;
    }

    // Guard 3: bar spread invariant (Issue #112) -- LOG ONLY, never reject
    // (Wide bars are expected during flash crashes with open-anchored detection)

    true
}
```

### Fan-Out in trade_dispatch.rs

```rust
// In process_trade_through_processors(), after bar_buffer.push(completed):

// Fan-out to optional sinks (e.g., ClickHouseWriterSink)
#[cfg(feature = "clickhouse-sink")]
for sink in extra_sinks.iter_mut() {
    match sink.on_bar(&completed) {
        Ok(()) => {}
        Err(SinkError::Recoverable(msg)) => {
            tracing::warn!(
                sink = sink.name(), %msg,
                %symbol, threshold = *threshold,
                "sink recoverable error"
            );
        }
        Err(SinkError::Unrecoverable(msg)) => {
            tracing::error!(
                sink = sink.name(), %msg,
                %symbol, threshold = *threshold,
                "sink unrecoverable error"
            );
        }
    }
}
```

### Metrics Bridge in StreamManager

```rust
// In StreamManager::get_combined_metrics(), extend with CH writer metrics:

#[cfg(feature = "clickhouse-sink")]
if let Some(ref ch_metrics) = self.ch_writer_metrics {
    m.insert(
        "ch_writer_bars_flushed".into(),
        ch_metrics.bars_flushed.load(Ordering::Relaxed),
    );
    m.insert(
        "ch_writer_bars_failed".into(),
        ch_metrics.bars_failed.load(Ordering::Relaxed),
    );
    m.insert(
        "ch_writer_flush_count".into(),
        ch_metrics.flush_count.load(Ordering::Relaxed),
    );
    m.insert(
        "ch_writer_last_flush_latency_us".into(),
        ch_metrics.last_flush_latency_us.load(Ordering::Relaxed),
    );
    m.insert(
        "ch_writer_dead_letter_count".into(),
        ch_metrics.dead_letter_count.load(Ordering::Relaxed),
    );
    m.insert(
        "ch_writer_dead_letter_replayed".into(),
        ch_metrics.dead_letter_replayed.load(Ordering::Relaxed),
    );
}
```

### PyO3 get_metrics() Extension

The existing `PyStreamManager::get_metrics()` already calls `manager.get_combined_metrics()` and converts to a Python dict. No PyO3 binding changes needed -- the Rust-side `get_combined_metrics()` extension automatically flows to Python.

## State of the Art

| Old Approach                  | Current Approach             | When Changed                           | Impact                                     |
| ----------------------------- | ---------------------------- | -------------------------------------- | ------------------------------------------ |
| Single ring buffer output     | Ring buffer + BarSink vector | Phase 36 (this phase)                  | Enables dual-write without blocking Python |
| Python CH writes (sync_flush) | Rust ClickHouseWriterSink    | Phase 34 built it, Phase 36 integrates | Eliminates Python from CH hot path         |
| Python pre-write guards       | Rust pre-write guards        | Phase 36 (this phase)                  | Guards run before serialization            |

## Open Questions

1. **Sink ownership for symbol tasks**
   - What we know: `symbol_task()` runs as a tokio task per symbol. `BarSink` requires `&mut self`.
   - What's unclear: Whether to create one shared sink (needs `Arc<Mutex>` or channel cloning) or per-symbol sinks (wasteful -- each spawns a flush thread).
   - Recommendation: One `ClickHouseWriterSink` per engine. The sink's `on_bar()` does `try_send()` to a single flush thread channel. Pass sinks through `symbol_task()` as `Arc<Mutex<Vec<Box<dyn BarSink>>>>` or better: since `process_trade_through_processors()` is synchronous (not async), the sink can be passed as `&mut [Box<dyn BarSink>]` within each symbol_task's loop body. The flush thread is shared -- bars from all symbols go to the same bounded channel.

2. **OPENDEVIATIONBAR_MODE check for enabling CH sink**
   - What we know: `ClickHouseWriterConfig::from_env()` reads `OPENDEVIATIONBAR_CH_HOSTS`.
   - What's unclear: Should the Rust writer check `OPENDEVIATIONBAR_MODE` to decide whether to activate? In Python, `OPENDEVIATIONBAR_MODE=remote` means "use bigblack."
   - Recommendation: The CH writer activates if `OPENDEVIATIONBAR_CH_HOSTS` is set. The `OPENDEVIATIONBAR_USE_RUST_CH_WRITER` feature flag (INTEG-03, Phase 37) will control actual activation. For Phase 36, the writer is always created when the feature is compiled in and env vars are present.

## Validation Architecture

### Test Framework

| Property           | Value                                                                                                                           |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| Framework          | cargo nextest + pytest                                                                                                          |
| Config file        | `.config/nextest.toml` (Rust), `pyproject.toml` (Python)                                                                        |
| Quick run command  | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(fan_out) or test(guard) or test(metrics)'` |
| Full suite command | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink`                                                    |

### Phase Requirements -> Test Map

| Req ID   | Behavior                                                  | Test Type | Automated Command                                                                                        | File Exists?       |
| -------- | --------------------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------- | ------------------ |
| INTEG-01 | Fan-out dispatches to both ring buffer AND BarSink        | unit      | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(fan_out)'`          | Wave 0             |
| INTEG-02 | ClickHouseWriterConfig::from_env() reads correct env vars | unit      | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(config_from_env)'`  | Exists (config.rs) |
| INTEG-04 | get*combined_metrics() includes ch_writer*\* keys         | unit      | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(combined_metrics)'` | Wave 0             |
| INTEG-05 | Pre-write guards reject invalid bars                      | unit      | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(guard)'`            | Wave 0             |
| INTEG-07 | Ouroboros mode hardcoded to "aion"                        | unit      | Already tested in Phase 34 row.rs tests                                                                  | Exists             |

### Sampling Rate

- **Per task commit:** `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink`
- **Per wave merge:** Full `cargo nextest run --workspace` + `mise run test-py`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `crates/opendeviationbar-streaming/src/clickhouse_writer/guards.rs` -- pre-write guard functions + tests
- [ ] Fan-out unit tests in `trade_dispatch.rs` or a dedicated test module
- [ ] Metrics integration test verifying `ch_writer_*` keys in `get_combined_metrics()`

## Sources

### Primary (HIGH confidence)

- `crates/opendeviationbar-streaming/src/engine/odb_engine.rs` -- fan-out pattern (lines 189-200)
- `crates/opendeviationbar-streaming/src/engine/traits.rs` -- BarSink trait definition
- `crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs` -- ClickHouseWriterSink implementation
- `crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs` -- ClickHouseWriterConfig::from_env()
- `crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs` -- FlushThreadMetrics fields
- `crates/opendeviationbar-streaming/src/live_engine/trade_dispatch.rs` -- bar emission point
- `crates/opendeviationbar-streaming/src/live_engine/mod.rs` -- LiveBarEngine structure
- `crates/opendeviationbar-streaming/src/stream_manager.rs` -- get_combined_metrics() pattern
- `src/stream_manager_bindings.rs` -- PyStreamManager::get_metrics() PyO3 bridge
- `python/opendeviationbar/clickhouse/bulk_operations.py` -- Python pre-write guards (lines 42-166)

### Secondary (MEDIUM confidence)

- `python/opendeviationbar/sidecar/__init__.py` -- Python heartbeat metrics consumption pattern

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- all code exists in workspace, no new dependencies
- Architecture: HIGH -- fan-out pattern proven in OdbEngine, BarSink trait stable
- Pitfalls: HIGH -- identified from direct code analysis, ownership patterns well-understood

**Research date:** 2026-03-29
**Valid until:** 2026-04-28 (stable internal architecture)

## Project Constraints (from CLAUDE.md)

- Python 3.13 ONLY (no impact on Rust-side work)
- ClickHouse COMMENT = SSoT (no schema changes in this phase)
- Leverage Rust: bar validation moves from Python to Rust (aligns with principle)
- Spot-only data source policy (no impact on writer integration)
- Aion is the only active mode (INTEG-07 already satisfied in row.rs)
- `maturin develop` required after any Rust change (standard build step)
- Feature flag: `clickhouse-sink` in `opendeviationbar-streaming` Cargo.toml already defined
- PyO3 `streaming` feature in root Cargo.toml needs extension to include `clickhouse-sink` when enabled
