# Requirements: opendeviationbar-py v9.0

**Defined:** 2026-03-28
**Core Value:** Eliminate Python from the WS streaming hot path -- zero-gap, zero-overlap bars written directly from Rust to ClickHouse

## v9.0 Requirements

Requirements for Rust-native ClickHouse writer milestone. Each maps to roadmap phases.

### Core Writer

- [x] **WRITER-01**: Rust `ClickHouseWriterSink` implements `BarSink` trait (`on_bar`, `flush`, `name`) with `SinkError` classification
- [ ] **WRITER-02**: ClickHouseWriterSink serializes `CompletedBar` to RowBinary format matching all columns in `schema.sql` (60+ fields including 16 Nullable inter-bar features)
- [x] **WRITER-03**: Batch buffering with configurable thresholds: N bars (default 500), T milliseconds (default 100ms), max bytes (default 5MB) -- flush on whichever fires first
- [ ] **WRITER-04**: Dedup token generation produces MD5 hash identical to Python's `MD5(f"{symbol}_{threshold}_{start_ts}_{end_ts}_{ouroboros_mode}")` -- verified by cross-language parity test
- [x] **WRITER-05**: Retry with backoff on transient errors (HTTP 429, 503, network timeout) using existing `backon` crate -- fatal errors (400, 401, schema mismatch) fail immediately
- [x] **WRITER-06**: Non-blocking `on_bar()` via flush thread pattern -- `SyncSender` bounded channel to dedicated `std::thread` with `reqwest::blocking` POST
- [x] **WRITER-07**: `X-ClickHouse-Summary` header parsed after INSERT to verify `written_rows == expected_rows` -- log warning on mismatch (silent dedup drops)
- [ ] **WRITER-08**: Computed columns (`is_liquidation_cascade`) are never inserted -- Rust writer omits them, letting ClickHouse evaluate DEFAULT expressions
- [x] **WRITER-09**: Plugin column discovery via `system.columns` query at startup -- dynamic column list, not hardcoded schema

### Resilience

- [ ] **RESIL-01**: Dead-letter fallback serializes failed `CompletedBar` batches to Parquet files in `/tmp/opendeviationbar-dead-letter/` with filename `{symbol}_{threshold}_{timestamp}.parquet`
- [x] **RESIL-02**: Dead-letter Parquet schema matches Python Charon replay expectations -- cross-language round-trip test gates this requirement
- [ ] **RESIL-03**: Error classification distinguishes transient (retry), fatal (dead-letter + alert), and backpressure (channel full) via `SinkError` variants
- [ ] **RESIL-04**: Backpressure detection via bounded `SyncSender` -- when flush thread can't keep up, `on_bar()` logs warning and dead-letters overflow bars
- [x] **RESIL-05**: Rust-side immediate replay of dead-letter files on flush thread recovery (Niffler pattern)

### Integration

- [x] **INTEG-01**: Fan-out in `LiveBarEngine` -- `ClickHouseWriterSink` runs alongside `ChannelSink` (Python still needs bars for checkpoints, SSE, Argus, Lethe)
- [x] **INTEG-02**: Configuration via environment variables (`OPENDEVIATIONBAR_CH_HOSTS`, `OPENDEVIATIONBAR_MODE`) -- no PyO3 config bridge, Rust reads same env vars as Python
- [x] **INTEG-03**: Feature flag `OPENDEVIATIONBAR_USE_RUST_CH_WRITER` with three modes: `off` (Python only), `shadow` (Rust writes + Python verifies), `live` (Rust writes, Python skips CH)
- [x] **INTEG-04**: Metrics bridge to Python -- Rust writer exposes write latency, batch sizes, error counts, dead-letter count via StreamManager metrics dict consumed by heartbeat
- [x] **INTEG-05**: Pre-write guards ported to Rust: close_time_us scale validation, volume non-negative, threshold registry check -- run before serialization
- [ ] **INTEG-06**: REST bar flush via Rust writer -- `_sync_flush_rest_bars()` uses Rust path when feature flag is `live`, eliminating last Python CH write on startup
- [x] **INTEG-07**: Ouroboros mode hardcoded to `"aion"` -- day-mode fully removed from codebase, no mode switching logic needed

### Validation

- [x] **VALID-01**: Shadow mode dual-write -- Rust and Python both write bars, Python queries CH and compares Rust-written vs Python-written bars (zero diff)
- [x] **VALID-02**: Dedup token parity test in Rust test suite -- fixed inputs produce identical MD5 hashes to Python `hashlib.md5().hexdigest()`
- [ ] **VALID-03**: Staging validation on bigblack with shadow mode for minimum 24 hours before enabling `live` mode
- [x] **VALID-04**: Stathera audit after cutover shows zero new gaps or overlaps -- same data integrity guarantees as Python writer
- [x] **VALID-05**: Dead-letter round-trip test -- Rust serializes to Parquet, Python Charon replays successfully, bars appear in CH with correct values

## Future Requirements

Deferred to future release. Tracked but not in current roadmap.

### Performance Optimization

- **PERF-01**: RowBinary streaming directly to HTTP request body (no intermediate `Vec<u8>` buffer)
- **PERF-02**: Connection pooling tuning for 16-symbol streaming load
- **PERF-03**: Pure Rust sidecar (no Python orchestrator)

## Out of Scope

| Feature                                       | Reason                                                                                                  |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| Rewriting kintsugi/heartbeat/seeder CH writes | Network I/O dominates, Python overhead negligible -- writer ownership model keeps these Python          |
| Replacing Python sidecar orchestrator         | Only the CH write path moves to Rust -- Python handles checkpoints, SSE, health, config                 |
| Redis write lock in Rust                      | Sidecar is sole live writer (writer ownership #280) -- lock exists for kintsugi overlap prevention only |
| Circuit breaker in Rust                       | Used only by `populate_cache_resumable()`, not sidecar hot path -- stays Python                         |
| Day-mode support                              | Fully removed from codebase -- Aion-only                                                                |
| ClickHouse native protocol (port 9000)        | HTTP interface (port 8123) is proven, no protocol divergence risk                                       |

## Traceability

| Requirement | Phase | Status  |
| ----------- | ----- | ------- |
| WRITER-01   | 34    | Complete |
| WRITER-02   | 34    | Pending |
| WRITER-03   | 34    | Complete |
| WRITER-04   | 34    | Pending |
| WRITER-05   | 34    | Complete |
| WRITER-06   | 34    | Complete |
| WRITER-07   | 34    | Complete |
| WRITER-08   | 34    | Pending |
| WRITER-09   | 34    | Complete |
| RESIL-01    | 35    | Pending |
| RESIL-02    | 35    | Complete |
| RESIL-03    | 35    | Pending |
| RESIL-04    | 35    | Pending |
| RESIL-05    | 35    | Complete |
| INTEG-01    | 36    | Complete |
| INTEG-02    | 36    | Complete |
| INTEG-03    | 37    | Complete |
| INTEG-04    | 36    | Complete |
| INTEG-05    | 36    | Complete |
| INTEG-06    | 38    | Pending |
| INTEG-07    | 36    | Complete |
| VALID-01    | 37    | Complete |
| VALID-02    | 37    | Complete |
| VALID-03    | 37    | Pending |
| VALID-04    | 37    | Complete |
| VALID-05    | 37    | Complete |

**Coverage:**

- v9.0 requirements: 26 total
- Mapped to phases: 26
- Unmapped: 0

---

_Requirements defined: 2026-03-28_
_Last updated: 2026-03-28 after roadmap creation_
