# Project Research Summary

**Project:** opendeviationbar-py v9.0 — Rust-Native ClickHouse Writer
**Domain:** Real-time bar writer replacing Python httpx in sidecar hot path
**Researched:** 2026-03-28
**Confidence:** HIGH

## Executive Summary

The v9.0 milestone replaces the Python ClickHouse write path in the streaming sidecar with a Rust-native `ClickHouseWriterSink` implementing the existing `BarSink` trait. This is a well-scoped refactor of a single component in a mature 42M-bar production system. The recommended approach: implement a flush-thread-based sink that buffers `CompletedBar` structs internally, serializes to JSONEachRow, and POSTs to ClickHouse via `reqwest::blocking`, using the official `clickhouse` crate (v0.14.2) for type-safe RowBinary serialization. Three new workspace dependencies are needed (`clickhouse`, `parquet`, `md5`); all other required crates (`reqwest`, `backon`, `arrow`, `tokio`, `serde`) are already present.

The primary value of this rewrite is eliminating GIL contention and Python httpx overhead on the sidecar hot path. The write rate (~50 bars/sec for 16 symbols) is well below ClickHouse's INSERT capacity — the bottleneck is Python serialization and GIL release latency, not throughput. The Rust writer reduces per-batch latency from 5-50ms to under 1ms and removes the last blocking Python operation from the `on_bar()` dispatch loop.

The biggest risks are correctness risks, not performance risks. Specifically: dedup token divergence between Python and Rust MD5 computation (silent duplicate rows), silent HTTP 200 responses hiding zero-row writes, and pre-write guard bypass allowing corrupt bars into ClickHouse. All three have real incident history in this codebase. Mitigation requires cross-language parity tests before any production writes and a once-at-startup feature flag to prevent mid-session writer switching. A shadow mode (`OPENDEVIATIONBAR_USE_RUST_CH_WRITER=shadow`) is recommended for initial staging validation.

## Key Findings

### Recommended Stack

The `clickhouse` crate (v0.14.2, official ClickHouse Inc. client) is the right choice over raw `reqwest` POST. It provides a typed `#[derive(Row, Serialize)]` macro for zero-boilerplate RowBinary serialization and a built-in `Inserter` for batch buffering — exactly matching the Python `_flush_all_batches()` pattern. All other needed stack components are already workspace dependencies. Total new dependency surface is minimal: `clickhouse`, `parquet` (arrow-rs v58, matching the workspace), and `md5` (must match Python's `hashlib.md5` output byte-for-byte).

**Core technologies:**

- `clickhouse` 0.14.2: typed CH HTTP client — official crate, RowBinary format, built-in Inserter, rustls-tls support (no OpenSSL)
- `reqwest` 0.12 (existing): HTTP transport for ClickHouse INSERTs via blocking client in flush thread
- `backon` 1.6 (existing): exponential backoff retry — already used by providers, same pattern
- `parquet` 58 (new, matches arrow-rs): dead-letter Parquet serialization — exact version lock with arrow 58 in workspace
- `md5` 0.7 (new): dedup token generation — must produce identical hex output to Python `hashlib.md5().hexdigest()`

**Critical version requirement:** `clickhouse` 0.14.2 requires `rust-version = "1.93"` and `edition = "2024"` — workspace already meets both. `parquet` must be version 58 to match the arrow-rs release train already in workspace.

### Expected Features

The Rust writer achieves parity with the Python writer first, then adds observability differentiators. Day-mode filtering, Redis write locks, circuit breaker, plugin column discovery, and Telegram notifications are explicitly excluded — all either deprecated, not applicable to the sidecar hot path, or already handled by surrounding Python layers.

**Must have (table stakes — production parity):**

- HTTP POST to CH port 8123 with JSONEachRow format
- Batch buffering (10 bars OR 1s timeout, matching Python defaults at `sidecar/__init__.py:862-864`)
- Retry with exponential backoff (3 attempts, 30s total, matching stamina config)
- Niffler dead-letter to Parquet on exhausted retries (compatible with Charon replay)
- Dedup token generation — MD5 parity with Python `bulk_operations.py:282-299`
- Full column schema serialization (60+ columns matching schema.sql)
- `BarSink` trait implementation (`on_bar`, `flush`, `name`)
- Graceful shutdown flush on SIGTERM
- Pre-write guards: inverted TIDs, timestamp scale, non-negative volume
- `ouroboros_mode` field in every INSERT ("aion" for all active symbols)

**Should have (differentiators justifying the rewrite):**

- Feature flag `OPENDEVIATIONBAR_USE_RUST_CH_WRITER` checked once at startup
- Error classification (Recoverable vs Unrecoverable for CH HTTP responses)
- `WriterMetrics` exposed via PyO3 (`bars_written`, `bars_failed`, `dead_letters`, `last_flush_latency_us`)
- Per-partition batching grouping by `(symbol, threshold_decimal_bps)` before each INSERT

**Defer to post-validation:**

- Backpressure signaling (only needed if write latency spikes above 1s sustained)
- Batch-level Stathera diagnostic (heartbeat already covers this externally)

### Architecture Approach

The `ClickHouseWriterSink` uses a flush-thread pattern: `on_bar()` buffers synchronously in a `Vec<CompletedBar>` (sub-microsecond, non-blocking on the Tokio hot path), then sends full batches to a dedicated `std::thread` via a bounded `SyncSender`. The flush thread handles HTTP POST, retry, and dead-letter. This keeps the WS->processor->`on_bar()` critical path unblocked. The `LiveBarEngine` fans out to both the existing `ChannelSink` (Python orchestration still needs bars for SSE, checkpoints, Argus, Lethe) and the new `ClickHouseWriterSink`. Python config flows one-way into Rust via PyO3 `configure_ch_writer(dict)` at sidecar startup — Rust never reads env vars directly.

**Major components:**

1. `ClickHouseWriterSink` (new, `sink_clickhouse.rs`) — BarSink impl: buffer + flush thread + retry + dead-letter
2. `ClickHouseWriterConfig` (new) — CH connection params, buffer thresholds, dedup settings; created by Python, passed via PyO3
3. `WriterMetrics` (new) — AtomicU64 counters; read by Python heartbeat and `/health` endpoint
4. `LiveBarEngine` (modified) — fan-out to multiple sinks (ChannelSink + ClickHouseWriterSink)
5. `StreamManager` + PyO3 bridge (modified) — expose `configure_ch_writer()`, `get_metrics()` additions, `writer_health()`
6. Python sidecar `__init__.py` (modified) — feature-flag dispatch gates `_write_batch_with_retry()` calls; bar poll loop kept intact

### Critical Pitfalls

1. **Dedup token incompatibility** — Rust MD5 computation differs from Python (column selection order, integer formatting, nullable min/max) creating silent duplicate rows. Prevention: cross-language test comparing tokens for N known batches before any production writes. Use `first_agg_trade_id` as token base if timestamp formatting is ambiguous.

2. **Silent write failure (HTTP 200, zero rows written)** — ClickHouse returns HTTP 200 when dedup silently drops a block; without parsing `X-ClickHouse-Summary` header (`written_rows`), the writer reports success while bars are lost. Prevention: always set `send_progress_in_http_headers=1`, parse `written_rows` on every INSERT response, trigger dead-letter on unexpected zero-row writes.

3. **Pre-write guard bypass** — The Rust writer bypasses all 6 Python guards in `bulk_operations.py` (timestamp scale, volume sign, TID ordering, ouroboros consistency). A bar with millisecond timestamps (1000x too small) or inverted trade IDs enters ClickHouse silently. Prevention: port all 6 guards to Rust `on_bar()` before buffering.

4. **Feature flag race condition** — Flag evaluated per-flush rather than once at startup allows mid-session Rust writer init failure + Python fallback, creating two concurrent writers for overlapping TID ranges. Prevention: evaluate flag exactly once in `_create_engine()`, fall back to Python for the entire session on any Rust writer init failure.

5. **Dead-letter format incompatibility** — Rust `CompletedBar` field names (`open_time`) differ from ClickHouse column names Python replay expects (`open_time_us`). Charon replay fails silently, bars accumulate in dead-letter forever. Prevention: round-trip test (Rust writes Parquet, Python calls `store_bars_batch()`).

## Implications for Roadmap

Five phases are required. Phases 1-3 are pure Rust with no Python changes and are independently testable in isolation. Phase 4 wires Python and must be validated in staging. Phase 5 removes the Python write path only after staging confirms zero regressions.

### Phase 1: Core ClickHouseWriter (Rust, isolated)

**Rationale:** All downstream phases depend on a correct, tested writer. The six critical pitfalls (dedup token, silent failure, guard bypass, schema drift, ouroboros mode, URL encoding) must be solved here before any integration. Zero Python changes, fully unit-testable in isolation with a mock HTTP server.

**Delivers:** `ClickHouseWriterSink` implementing `BarSink` with: batch buffer (10 bars OR 1s), flush thread, HTTP POST to CH (JSONEachRow), MD5 dedup token matching Python exactly, 6 pre-write guards, exponential backoff retry (3 attempts, 30s), `WriterMetrics` AtomicU64 counters, `ouroboros_mode` in every INSERT, `X-ClickHouse-Summary` response validation, `std::panic::catch_unwind` at flush thread boundary.

**Addresses features:** HTTP POST, batch buffering, retry, dedup token, column schema, BarSink impl, graceful shutdown, pre-write guards, source version tagging, per-partition batching.

**Avoids:** Pitfalls 1 (dedup token), 2 (silent failure), 3 (guard bypass), 6 (schema drift), 11 (ouroboros mode), 12 (URL encoding), 14 (panic kills sidecar silently).

**New files:** `crates/opendeviationbar-streaming/src/engine/sink_clickhouse.rs`, `ch_serializer.rs`

**New deps:** `clickhouse = { version = "0.14", features = ["inserter", "rustls-tls"] }`, `md5 = "0.7"`

### Phase 2: Niffler Dead-Letter (Rust, isolated)

**Rationale:** Must ship before production integration. Dead-letter files must be replay-compatible with Python's existing `replay_dead_letters()` (Charon already watches `/tmp/opendeviationbar-dead-letter/`). Testable via a standalone round-trip test: Rust writes Parquet, Python reads it and calls `store_bars_batch()`.

**Delivers:** Parquet dead-letter serialization from `CompletedBar` using exact ClickHouse column names (`open_time_us` not `open_time`, `_us` suffix for microsecond timestamps). Dead-letter triggered on: exhausted retries, flush channel full (backpressure), flush thread panic. Filename pattern matching Charon's glob.

**Addresses features:** Niffler dead-letter on exhausted retries (table stakes), partial batch safety.

**Avoids:** Pitfalls 4 (partial batch without dead-letter), 8 (dead-letter format mismatch with Python replay).

**New deps:** `parquet = { version = "58", features = ["snap", "zstd"] }`

### Phase 3: Fan-Out in LiveBarEngine + PyO3 Bridge

**Rationale:** LiveBarEngine modification and PyO3 bridge are tightly coupled — engine exposes metrics via stream manager. Still no Python sidecar changes. Must complete before Python integration can wire the feature flag.

**Delivers:** Multi-sink fan-out in `symbol_task()` (ring_buffer push + ClickHouseWriterSink.on_bar()). PyO3 methods on `StreamManager`: `configure_ch_writer(dict)`, `writer_health() -> bool`, `writer_flush()`. Writer metric keys in `get_metrics()` response (`ch_bars_written`, `ch_bars_failed`, `ch_dead_letters`, `ch_last_flush_latency_us`). Single `reqwest::Client` instance constructed once at writer creation time.

**Addresses features:** Connection pooling with keep-alive, metrics reporting via `get_metrics()`, error classification.

**Avoids:** Pitfalls 7 (connection pool exhaustion), 9 (metrics blackout during cutover).

**Modified files:** `crates/opendeviationbar-streaming/src/live_engine/mod.rs`, `stream_manager.rs`, `src/stream_manager_bindings.rs`

### Phase 4: Python Integration + Staging Validation

**Rationale:** First phase with Python sidecar changes. Feature flag evaluated exactly once in `_create_engine()`. Python main loop gates `_write_batch_with_retry()` calls behind the flag but keeps the `engine.next_bar()` poll loop intact for SSE, checkpoints, Argus, and Lethe. Validated in staging (bigblack) against live Binance stream for a minimum of one week before Phase 5.

**Delivers:** `OPENDEVIATIONBAR_USE_RUST_CH_WRITER` feature flag (startup-only evaluation, never re-evaluated mid-session), sidecar main loop with gated Python write calls, Python-side logging of `ch_bars_written` / `ch_dead_letters` after each flush cycle, heartbeat `stathera.py` monitoring `ch_dead_letters == 0`, integration test against real CH.

**Addresses features:** Feature flag for zero-risk rollout, Python fallback kept intact.

**Avoids:** Pitfalls 5 (feature flag race — startup-only check prevents mid-session writer switching).

**Modified files:** `python/opendeviationbar/sidecar/__init__.py`, `scripts/heartbeat/checks/stathera.py`

### Phase 5: REST Bar Flush via Rust Writer

**Rationale:** Final phase. Routes `fill_from_rest` startup bars through the Rust writer, eliminating the last Python CH write on the startup path. Deferred until Phase 4 is validated in staging — changing the startup sequence is higher risk than the streaming hot path.

**Delivers:** `_sync_flush_rest_bars()` drained through Rust sink, removing `_write_bars_batch_to_clickhouse()` hot-path calls from `arrow_flush.py` when flag is active. After this phase, `clickhouse_write.py` becomes fallback-only code.

**Avoids:** Pitfall 5 (no writer switching mid-session — startup path uses the same writer selected in `_create_engine()`).

**Modified files:** `python/opendeviationbar/sidecar/arrow_flush.py`, `python/opendeviationbar/sidecar/midnight.py`

### Phase Ordering Rationale

- Phases 1-3 are pure Rust with no Python sidecar changes. Each is independently testable and mergeable without operational risk. This ordering eliminates correctness risk before touching the live sidecar.
- Phase 4 must follow Phase 3 because the PyO3 bridge (`configure_ch_writer`, writer metrics in `get_metrics()`) must exist before Python can wire the feature flag and log metrics.
- Phase 5 must follow Phase 4 validation. The REST bar flush is a startup sequence change — higher operational risk than the hot path. Running Phase 4 in staging for at least one week before Phase 5 is strongly recommended.
- Dead-letter (Phase 2) is separated from the core writer (Phase 1) to keep each phase reviewable independently and because dead-letter correctness requires a cross-language integration test that is separate from the core writer unit tests.

### Research Flags

Phases likely needing deeper research during planning:

- **Phase 4 (Python integration):** The exact `__init__.py` call sites that must be gated need auditing — the sidecar main loop is ~1300 lines. Targeted audit of `_write_batch_with_retry()` call sites, `_sync_flush_rest_bars_arrow()`, and valid_pairs filtering location needed before planning this phase.
- **Phase 3 (LiveBarEngine fan-out):** The `symbol_task()` architecture in `live_engine/mod.rs` must be read to confirm BarSink ownership model. `BarSink` is `&mut self` — the sink must be owned by the task, not shared. If tasks share state, the design changes.

Phases with standard patterns (skip research-phase):

- **Phase 1 (core writer):** `clickhouse` crate API is well-documented with official examples. RowBinary serialization, Inserter pattern, and retry with `backon` are straightforward. STACK.md has code-level implementation snippets ready to use.
- **Phase 2 (dead-letter):** `parquet` crate + ArrowWriter is a standard 5-line pattern. Column naming requirements are fully specified in PITFALLS.md Pitfall 8.

## Confidence Assessment

| Area         | Confidence | Notes                                                                                                                                                    |
| ------------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Stack        | HIGH       | Official ClickHouse Rust client docs verified, workspace deps audited against Cargo.toml, edition/rust-version compatibility confirmed                   |
| Features     | HIGH       | Directly derived from existing production Python writer at known file:line locations; BarSink trait interface is stable                                  |
| Architecture | HIGH       | Based on live production code topology: BarSink trait, ChannelSink reference impl, Python main loop structure all confirmed                              |
| Pitfalls     | HIGH       | Grounded in actual production incidents: Issue #269 phantom gaps, Issue #275 fd exhaustion, Issue #280 concurrent writers, Feb 2026 silent write failure |

**Overall confidence:** HIGH

### Gaps to Address

- **Inserter per-commit dedup token support:** The `clickhouse` crate `Inserter` may not support per-commit `insert_deduplication_token` options. If not, Phase 1 must use raw `client.insert()` with manual 500-row batching instead of `Inserter`. Quick API verification needed at Phase 1 start.
- **LiveBarEngine symbol_task() ownership model:** Phase 3 fan-out requires `ClickHouseWriterSink` to be owned by the symbol_task Tokio task. Audit `live_engine/mod.rs` before Phase 3 planning to confirm ownership model and whether `Arc<Mutex<>>` is needed.
- **Exact column order for JSONEachRow:** The INSERT column order must match the ClickHouse ORDER BY key position. `schema.sql` is the source of truth but mapping all 60+ columns to `CompletedBar` field order must be done precisely during Phase 1. Mechanical but error-prone.

## Sources

### Primary (HIGH confidence)

- `crates/opendeviationbar-streaming/src/engine/traits.rs` — BarSink trait, SinkError types
- `python/opendeviationbar/clickhouse/bulk_operations.py` — pre-write guards, dedup token generation, cache_key formula
- `python/opendeviationbar/sidecar/__init__.py` — batch config (lines 862-864), main loop structure, feature flag insertion points
- `python/opendeviationbar/sidecar/clickhouse_write.py` — stamina retry config, Niffler dead-letter integration
- `python/opendeviationbar/sidecar/dead_letter.py` — Niffler format, filename convention, replay path
- [clickhouse-rs GitHub](https://github.com/ClickHouse/clickhouse-rs) — Inserter API, Row derive, RowBinary format
- [ClickHouse Rust integration docs](https://clickhouse.com/docs/integrations/rust) — official guide
- Project incident history (Issue #269, #275, #280, #90, Feb 2026 silent write failure)

### Secondary (MEDIUM confidence)

- `python/opendeviationbar/clickhouse/schema.sql` — 60+ column schema for BarRow mapping (referenced in research, column order needs direct verification in Phase 1)
- `crates/opendeviationbar-streaming/src/live_engine/types.rs` — CompletedBar field list (referenced, ownership model needs Phase 3 audit)

### Tertiary (LOW confidence — validate during implementation)

- `clickhouse` crate Inserter per-commit options support — flagged in STACK.md as needing API verification before committing to Inserter vs raw insert() approach
- Arrow IPC format version compatibility between arrow-rs 58 and ClickHouse v24.x on bigblack — needs a test INSERT before Phase 2 commits to Arrow format for dead-letter

---

_Research completed: 2026-03-28_
_Ready for roadmap: yes_
