# Feature Landscape: Rust-Native ClickHouse Writer (v9.0)

**Domain:** Real-time bar writer replacing Python httpx in sidecar hot path
**Researched:** 2026-03-28
**Confidence:** HIGH (based on existing production Python writer + BarSink trait)

## Table Stakes

Features the Rust writer MUST have for production parity with the Python writer. Missing any of these = cannot ship.

| Feature                                    | Why Expected                                                                                                   | Complexity | Notes                                                                                                                                      |
| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| HTTP POST to CH (port 8123)                | Current writer uses `clickhouse-connect` over HTTP; Rust uses `reqwest::Client` (already in workspace)         | Low        | `reqwest` with `rustls-tls` already a workspace dep. POST to `/?query=INSERT INTO ... FORMAT JSONEachRow`                                  |
| Batch buffering (count + time)             | Python buffers 10 bars OR 1s timeout (`batch_max_size=10`, `batch_timeout_s=1.0`). Rust writer must match      | Med        | Configurable `max_bars: usize` + `max_delay: Duration`. Flush on whichever triggers first                                                  |
| Retry with exponential backoff             | Python uses `stamina.retry(attempts=3, timeout=30s)` on `(OSError, RuntimeError, ConnectionError, ValueError)` | Med        | Classify CH HTTP responses: 200=ok, 5xx/timeout=transient retry, 4xx=fatal. Use `tokio::time::sleep` for backoff                           |
| Niffler dead-letter on exhausted retries   | Python serializes failed bars to `/tmp/opendeviationbar-dead-letter/{sym}_{thr}_{ts}.parquet`                  | Med        | Use `arrow-rs` or `parquet` crate to write Parquet. Extract TID range for diagnostics. Existing Charon replay reads these files            |
| Dedup token generation (MD5 of cache_key)  | Python: `MD5(f"{symbol}_{threshold}_{start_ts}_{end_ts}_{ouroboros_mode}")` as `insert_deduplication_token`    | Low        | Must produce identical MD5 strings. Use `md5` crate. Send as `?insert_deduplicate=1&insert_deduplication_token={token}` query params       |
| Column schema parity                       | Python inserts ~25+ columns (OHLCV, microstructure, trade IDs, timestamps, metadata). Rust must match exactly  | Med        | Serialize `CompletedBar` + metadata (symbol, threshold, ouroboros_mode, version, cache_key, source_start_ts, source_end_ts) to JSONEachRow |
| `BarSink` trait implementation             | `ClickHouseWriter` implements existing `BarSink { on_bar, flush, name }` from `opendeviationbar-streaming`     | Low        | `on_bar` buffers; `flush` forces write. `SinkError::Recoverable` vs `Unrecoverable` already defined                                        |
| Graceful shutdown flush                    | Python `_flush_all_batches()` in `finally` block on SIGTERM/SIGINT                                             | Low        | `flush()` method on `BarSink` already designed for this. Drain internal buffer before returning                                            |
| Source version tagging                     | Python sets `opendeviationbar_version = f"{__version__}+sidecar"` on every batch                               | Low        | Accept version string at construction time                                                                                                 |
| Pre-write guards (physical invariant gate) | Python rejects bars where `first_agg_trade_id > last_agg_trade_id` or `open_time_us > close_time_us`           | Low        | Validate in `on_bar()` before buffering. Log + skip invalid bars                                                                           |

## Differentiators

Features that go beyond Python parity and justify the Rust rewrite.

| Feature                                             | Value Proposition                                                                                                                                    | Complexity | Notes                                                                                                                                                                             |
| --------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Feature flag for rollout                            | `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=true` env var. Python fallback when false or on error                                                           | Low        | Wire in sidecar `__init__.py`: if flag set, pass `ClickHouseWriterSink` to engine; else keep `ChannelSink` + Python write path. Zero-risk rollout                                 |
| Error classification (Recoverable vs Unrecoverable) | Python treats all errors the same (retry then dead-letter). Rust can distinguish: transient (5xx, timeout, DNS) vs fatal (4xx auth, schema mismatch) | Med        | Return `SinkError::Recoverable` for transient (triggers retry), `SinkError::Unrecoverable` for fatal (skip retries, dead-letter immediately). Prevents wasting 30s retrying a 403 |
| Metrics reporting via `get_metrics()`               | Expose `bars_buffered`, `bars_written`, `bars_dead_lettered`, `write_latency_us`, `retries_total`, `last_write_epoch_ms`                             | Med        | AtomicU64 counters (like `ChannelSink.bars_sent/bars_dropped`). Python reads via PyO3 binding on `engine.get_metrics()`. Feeds `/health` endpoint and heartbeat                   |
| Connection pooling with keep-alive                  | `reqwest::Client` reuses TCP connections by default. Python creates new connections or uses connection pooling less efficiently                      | Low        | Single `reqwest::Client` constructed once, shared across all writes. Eliminates per-batch TCP handshake                                                                           |
| Backpressure signaling                              | If internal buffer exceeds high-water mark, signal back to engine that writes are falling behind                                                     | Med        | If buffer > 2x `max_bars`, return `SinkError::Recoverable("backpressure")`. Engine can log/alert without blocking                                                                 |
| Batch-level Stathera diagnostic                     | Python runs `_stathera_diagnostic_polars()` per sync_flush. Rust can do it per batch with zero overhead                                              | Low        | Check `bars[i].last_agg_trade_id + 1 == bars[i+1].first_agg_trade_id` within each buffered batch. Log gaps, never block writes                                                    |

## Anti-Features

Features to explicitly NOT build in the Rust writer.

| Anti-Feature                         | Why Avoid                                                                                                                                                                                      | What to Do Instead                                                                                                                                                                                         |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Redis write lock                     | Sidecar is single-writer per (symbol, threshold). Lock exists only for multi-writer paths (kintsugi, populate). Adding Redis dep to Rust hot path adds latency + failure mode for zero benefit | Rely on writer ownership model (#280): sidecar owns live stream forward, kintsugi owns historical gaps                                                                                                     |
| Circuit breaker in Rust              | Python circuit breaker (`fatal_write_breaker`) exists only for `populate_cache_resumable()` paths, NOT sidecar. Sidecar uses retry+dead-letter instead                                         | Keep retry + Niffler dead-letter. If CH is down for extended period, Charon replays. No state machine needed                                                                                               |
| Overlap strategy handling            | Python `store_bars_batch` supports 4 overlap strategies (off/warn/skip/replace). Sidecar always uses `overlap_strategy="off"` (single writer). No need in Rust                                 | Hardcode `overlap_strategy=off` behavior. No overlap checks                                                                                                                                                |
| Cross-midnight bar filtering         | Python `arrow_flush.py` filters cross-midnight Day-mode bars. Day-mode is fully deprecated (0 day-mode symbols). Dead code path                                                                | All symbols are Aion. No midnight logic in Rust writer                                                                                                                                                     |
| Plugin feature columns               | Python dynamically discovers plugin columns via entry-points. No plugins in production sidecar                                                                                                 | If plugin columns appear in `CompletedBar`, serialize them. No dynamic discovery needed                                                                                                                    |
| Telegram notifications from Rust     | Python Niffler sends Telegram alerts on dead-letter via `send_telegram()`. Adding Telegram HTTP client to Rust = unnecessary complexity                                                        | Dead-letter to Parquet. Python layer already monitors `/tmp/opendeviationbar-dead-letter/` via Charon. Charon sends Telegram alerts                                                                        |
| Niffler immediate replay from Rust   | Python spawns a daemon thread to immediately replay dead-letters. Complex Python/kintsugi interaction                                                                                          | Let Charon handle replay (5-min interval, already proven). Rust only needs to write the Parquet dead-letter file                                                                                           |
| Dynamic column discovery             | Python iterates `MICROSTRUCTURE_COLUMNS`, `TRADE_ID_RANGE_COLUMNS`, etc. to build column list dynamically                                                                                      | Define column schema statically in Rust. `CompletedBar` + `OpenDeviationBar` fields are known at compile time                                                                                              |
| Async HTTP writes on background task | Decoupling `on_bar()` from HTTP POST via internal channel adds complexity, ordering risks, and shutdown edge cases                                                                             | `on_bar()` buffers synchronously. `flush()` does blocking HTTP POST. The 5-50ms write latency is acceptable given 10-bar batching (amortized <5ms/bar). Revisit only if metrics show bar processing stalls |

## Feature Dependencies

```
BarSink trait (exists) --> ClickHouseWriter implements BarSink
    |
    v
reqwest::Client (exists in workspace) --> HTTP POST to CH
    |
    v
Batch buffer (new) --> Triggers flush on count or timeout
    |
    v
Retry logic (new) --> Classifies CH HTTP responses
    |
    +--> Success --> Increment metrics counters
    |
    +--> Exhausted retries --> Niffler dead-letter (Parquet write)
    |
    v
Feature flag (new, Python-side) --> Wires ClickHouseWriterSink vs ChannelSink
    |
    v
Metrics (new) --> Exposed via engine.get_metrics() to Python /health endpoint
```

**Critical path**: BarSink impl --> HTTP POST --> Batch buffer --> Retry --> Dead-letter --> Feature flag --> Metrics

## MVP Recommendation

### Phase 1: Core Writer (ship first)

1. `ClickHouseWriter` implementing `BarSink` with `reqwest::Client` HTTP POST
2. Batch buffering (10 bars or 1s, matching Python defaults)
3. Retry with exponential backoff (3 attempts, 30s total timeout)
4. Dedup token generation (MD5 parity with Python)
5. Column schema serialization (JSONEachRow format)
6. Pre-write guards (TID ordering, timestamp ordering)
7. Graceful shutdown flush

### Phase 2: Resilience + Observability

1. Niffler dead-letter to Parquet on exhausted retries
2. Error classification (transient vs fatal)
3. Metrics counters (written, dead-lettered, retries, latency)
4. Feature flag (`OPENDEVIATIONBAR_USE_RUST_CH_WRITER`)
5. PyO3 binding to wire into sidecar

### Defer

- **Backpressure signaling**: Only needed if CH write latency spikes above 1s sustained. Monitor first via metrics. Add if data shows need.
- **Batch-level Stathera diagnostic**: Nice-to-have. Python already runs Stathera checks in heartbeat. Add after core writer is stable.

## Existing Dependencies (Already Available)

| Dependency         | Where                                                          | What It Provides                                                          |
| ------------------ | -------------------------------------------------------------- | ------------------------------------------------------------------------- |
| `BarSink` trait    | `crates/opendeviationbar-streaming/src/engine/traits.rs`       | `on_bar()`, `flush()`, `name()`, `SinkError` enum                         |
| `CompletedBar`     | `crates/opendeviationbar-streaming/src/live_engine/types.rs`   | `symbol: Arc<str>`, `threshold_decimal_bps: u32`, `bar: OpenDeviationBar` |
| `reqwest`          | Workspace `Cargo.toml` (v0.12, rustls-tls)                     | HTTP client with connection pooling                                       |
| `OpenDeviationBar` | `opendeviationbar-core/src/types.rs`                           | All OHLCV + microstructure fields, `first/last_agg_trade_id`, timestamps  |
| `SinkError`        | `crates/opendeviationbar-streaming/src/engine/traits.rs`       | `Recoverable(String)` / `Unrecoverable(String)`                           |
| `ChannelSink`      | `crates/opendeviationbar-streaming/src/engine/sink_channel.rs` | Reference impl pattern (AtomicU64 counters, try_send non-blocking)        |

## Configuration Parity Matrix

Current Python config values the Rust writer must respect:

| Parameter       | Python Source                        | Current Default                     | Rust Writer                                                   |
| --------------- | ------------------------------------ | ----------------------------------- | ------------------------------------------------------------- |
| Batch max size  | `sidecar/__init__.py:863`            | 10 bars                             | Constructor param, env override                               |
| Batch timeout   | `sidecar/__init__.py:864`            | 1.0s                                | Constructor param, env override                               |
| Max retries     | `ResilienceConfig.flush_max_retries` | 3                                   | Constructor param                                             |
| Retry timeout   | `stamina.retry(timeout=30.0)`        | 30s total                           | Constructor param                                             |
| CH host         | `OPENDEVIATIONBAR_CH_HOSTS`          | bigblack                            | Read from env at construction                                 |
| CH port         | Default                              | 8123                                | Read from env at construction                                 |
| Dead-letter dir | `dead_letter.py:_DEAD_LETTER_DIR`    | `/tmp/opendeviationbar-dead-letter` | Same path (Charon reads these)                                |
| Ouroboros mode  | `symbol_modes` dict                  | "aion"                              | Passed per-bar from `CompletedBar` context or at construction |
| Source version  | `__version__ + "+sidecar"`           | Dynamic                             | Passed at construction                                        |

## Sources

- Python writer: `python/opendeviationbar/sidecar/clickhouse_write.py` (production, stamina retry)
- Dead-letter: `python/opendeviationbar/sidecar/dead_letter.py` (Niffler serialization)
- Arrow flush: `python/opendeviationbar/sidecar/arrow_flush.py` (zero-copy drain path)
- BarSink trait: `crates/opendeviationbar-streaming/src/engine/traits.rs` (interface contract)
- ChannelSink: `crates/opendeviationbar-streaming/src/engine/sink_channel.rs` (reference impl)
- Batch config: `python/opendeviationbar/sidecar/__init__.py:862-864` (10 bars, 1s timeout)
- Resilience: `python/opendeviationbar/config/resilience.py` (flush_max_retries=3)
- Dedup: `python/opendeviationbar/clickhouse/bulk_operations.py:282-299` (MD5 token)
- Cache key: `python/opendeviationbar/clickhouse/bulk_operations.py:873-888` (key_str format)
- PROJECT.md: `.planning/PROJECT.md` (v9.0 milestone scope)
