# Domain Pitfalls: Rust-Native ClickHouse Writer for Production Sidecar

**Domain:** Adding Rust ClickHouse HTTP writer to existing Python/Rust hybrid real-time pipeline
**Researched:** 2026-03-28
**System context:** 42M bars, 16 symbols, zero-gap/zero-overlap Stathera invariant, 24/7 Binance crypto stream

## Critical Pitfalls

Mistakes that cause data loss, Stathera violations, or require production rollback.

### Pitfall 1: Dedup Token Incompatibility — Silent Duplicate Rows

**What goes wrong:** The Rust writer generates `insert_deduplication_token` values that differ from the Python writer's MD5(cache_key) computation. During the feature-flag rollout period, the same batch retried by different writers produces different tokens. ClickHouse treats them as distinct INSERTs, creating duplicate rows that persist until the next OPTIMIZE FINAL. Stathera gap detection reports phantom overlaps. Heartbeat fires LOUD alerts.

**Why it happens:** The Python cache*key is computed as `MD5("{symbol}*{threshold}_{start_ts}_{end*ts}*{ouroboros_mode}")`where`start_ts`and`end_ts`are derived from the DataFrame's min/max`open_time_us`/`close_time_us`. The exact values depend on:

- Column selection order (Polars vs pandas min/max on nullable columns)
- Whether `open_time_us` exists (falls back to `close_time_us` for `start_ts`)
- String formatting of integers (Python `f"{int}"` vs Rust `format!("{}")`)

If the Rust implementation computes any of these differently — even a trailing space or different min/max on a nullable column — the MD5 diverges.

**Consequences:** Duplicate bars in ClickHouse. `SELECT ... FINAL` hides them at query time but they accumulate in storage. If Stathera queries run without FINAL (heartbeat fast path), phantom overlaps trigger false alerts. Worse: if Python fallback retries a batch that Rust already wrote with a different token, both succeed — 2x data.

**Prevention:**

1. Extract the exact cache_key formula into a shared specification (not just "MD5 of these fields" — specify the exact string template, integer formatting, column precedence)
2. Write a cross-language test: Python generates cache_key for N known batches, Rust generates the same, assert byte-identical MD5
3. During rollout, log both Python and Rust tokens for the same batch and diff them
4. Consider: use `first_agg_trade_id` as the dedup token base instead of timestamp ranges — trade IDs are deterministic integers with no formatting ambiguity

**Detection:** `SELECT cache_key, count() FROM open_deviation_bars WHERE (symbol, threshold_decimal_bps) = (?, ?) GROUP BY cache_key HAVING count() > 1` — any result with count > 1 indicates token divergence.

**Phase:** Must be solved in Phase 1 (core writer implementation) before any production writes.

---

### Pitfall 2: Silent Write Failure — HTTP 200 with Zero Rows Written

**What goes wrong:** ClickHouse HTTP interface returns HTTP 200 for INSERT statements where `insert_deduplicate=1` silently drops the block (token collision). The Rust writer sees a successful HTTP response but zero rows were written. If the writer doesn't parse and validate `written_rows` from the response, it reports success to the sidecar, which advances its state — bars are permanently lost.

**Why it happens:** The Python `clickhouse-connect` library returns a `QuerySummary` object with `written_rows`. The Rust `reqwest` POST to `http://host:8123/` gets a raw HTTP response. ClickHouse's HTTP interface returns the row count in the `X-ClickHouse-Summary` header (JSON with `written_rows`, `written_bytes`, etc.) but only when `send_progress_in_http_headers=1` is set. Without this setting, a 200 response with an empty body could mean "wrote 0 rows due to dedup" or "wrote N rows successfully" — indistinguishable.

**Consequences:** Data loss. The sidecar believes the write succeeded. Niffler dead-letter is not triggered. Bars are gone. Stathera eventually detects the gap, kintsugi repairs it hours later, but the real-time guarantee is broken.

**Prevention:**

1. Always set `send_progress_in_http_headers=1` in query params
2. Parse `X-ClickHouse-Summary` header from every INSERT response
3. If `written_rows == 0` and this was not an expected dedup collision (i.e., `skip_dedup=false` and we expected rows to be written), log a warning and trigger dead-letter
4. Integration test: INSERT with known dedup token twice, verify second returns `written_rows=0`, verify the writer correctly reports this

**Detection:** Compare sidecar `bars_drained` metric against ClickHouse `count(*)` for the same time window. Divergence = silent write loss.

**Phase:** Phase 1 (core writer). The HTTP response parsing is foundational.

---

### Pitfall 3: Pre-Write Guard Bypass — Corrupt Bars Enter ClickHouse

**What goes wrong:** The Python write path runs 6 pre-write guards before INSERT: `_guard_close_time_us_scale`, `_guard_volume_non_negative`, `_guard_bar_spread_invariant`, `_guard_threshold_registry`, `_guard_ouroboros_mode_consistency`, and the physical invariant gate (inverted trade IDs / timestamps). The Rust writer, being "just an HTTP POST", skips all of them. A bar with negative volume (overflow from f64 precision loss), timestamps in milliseconds instead of microseconds, or inverted first/last trade IDs enters ClickHouse and corrupts downstream analytics.

**Why it happens:** The guards are implemented in Python (`bulk_operations.py`) as part of the `BulkStoreMixin`. When the Rust writer bypasses `store_bars_batch()`, it bypasses all guards. The developer implementing the Rust writer focuses on the HTTP transport and forgets that the Python path was doing validation, not just INSERT.

**Consequences:**

- Microsecond/millisecond confusion: timestamps 1000x too small, `close_time_us < 1e15` — heartbeat Stathera queries return garbage
- Negative volumes: overflow bars cause NaN in ML features
- Inverted trade IDs: `first_agg_trade_id > last_agg_trade_id` — Stathera continuity chain breaks

**Prevention:**

1. Port all 6 guards to Rust as validation on the `CompletedBar` before buffering
2. The physical invariant gate (inverted TIDs/timestamps) is the most critical — implement it as `debug_assert!` in the Rust bar construction and as a runtime check in the writer
3. Unit test: construct bars with each invalid condition, verify the Rust writer rejects them
4. Keep the Python guards active during the feature-flag rollout period as a safety net (Python fallback path still validates)

**Detection:** `SELECT count() FROM open_deviation_bars WHERE close_time_us < 1000000000000000 OR volume < 0 OR (first_agg_trade_id > last_agg_trade_id AND first_agg_trade_id > 0)` — should always be 0.

**Phase:** Phase 1 (core writer). Guards must be implemented alongside the writer, not deferred.

---

### Pitfall 4: Partial Batch Write Without Dead-Letter — Bars Lost in Limbo

**What goes wrong:** The Rust writer POSTs a batch of 50 bars to ClickHouse. The TCP connection drops mid-transfer (SSH tunnel flap, ClickHouse OOM, network partition). ClickHouse may have consumed 30 of the 50 bars before the connection died. The Rust writer sees a connection error and triggers dead-letter for all 50 bars. When Niffler replays, 30 of those bars are duplicates (handled by ReplacingMergeTree) but the remaining 20 are legitimately lost if the dead-letter serialization itself fails.

**Why it happens:** ClickHouse HTTP INSERT is not transactional — there is no rollback. Once ClickHouse starts reading the request body, it may INSERT rows before the connection breaks. The Python path has the same issue but mitigates it with `stamina` retry (3 attempts) + Niffler dead-letter + Charon periodic ferry. If the Rust writer's dead-letter path has a bug (e.g., Parquet serialization panics on an unexpected type), the bars are gone.

**Consequences:** Stathera gap at the boundary of the partial write. The 30 written bars create a "half-batch" that doesn't align with the dead-lettered 50 bars. On replay, 30 duplicates are written (benign — ReplacingMergeTree handles it) and 20 bars fill the gap (correct). But if dead-letter serialization fails, those 20 bars are permanently lost.

**Prevention:**

1. Implement Rust dead-letter as Parquet serialization using `arrow-rs` — simpler and more reliable than custom format
2. Dead-letter format must be compatible with Python's Niffler replay path (`replay_dead_letters()` reads Polars DataFrames from `/tmp/opendeviationbar-dead-letter/`)
3. Test the dead-letter path with adversarial inputs: bars with NaN values, empty strings, i64::MAX trade IDs
4. Log the exact TID range of every dead-lettered batch for kintsugi targeted repair
5. Keep Charon periodic ferry running during rollout — it's the safety net that catches any dead-letter files the Rust writer creates

**Detection:** Monitor `/tmp/opendeviationbar-dead-letter/` for files. Any file older than 10 minutes = Niffler replay failed.

**Phase:** Phase 2 (Niffler dead-letter). Must be implemented before wiring into the sidecar main loop.

---

### Pitfall 5: Feature Flag Race Condition — Both Writers Active Simultaneously

**What goes wrong:** `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=true` is set, but during sidecar startup the Rust writer isn't initialized yet (e.g., `reqwest::Client` creation fails due to TLS config). The sidecar falls back to the Python writer. Mid-session, a hot-reload or env re-read re-enables the Rust writer. Now both writers are active for different batches in the same flush cycle. Two INSERT statements for overlapping TID ranges hit ClickHouse concurrently.

**Why it happens:** The feature flag is checked per-flush, not per-session. If the Rust writer becomes available after initial failure, the switching logic must be atomic. The Python path uses overlap_strategy="off" (trust mode — single writer assumption). Two concurrent writers violate this assumption.

**Consequences:** Duplicate rows with different cache_keys (different dedup tokens). ReplacingMergeTree eventually merges them but Stathera queries between merges see overlaps. Heartbeat fires false LOUD alerts. Operator panics, restarts sidecar, creating more gaps.

**Prevention:**

1. Feature flag is checked ONCE at sidecar startup, not per-flush
2. If Rust writer init fails at startup, fall back to Python for the entire session — no mid-session switching
3. Log which writer is active at startup with `logger.info("CH writer: rust" / "CH writer: python")`
4. Add a `/status` endpoint field showing the active writer for operational visibility
5. During rollout, use `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=shadow` mode: Rust writer runs in parallel but discards results (validates it would have succeeded without actually writing)

**Detection:** `grep "CH writer:" sidecar.log` — if both "rust" and "python" appear in the same session, race condition occurred.

**Phase:** Phase 3 (sidecar integration). The feature flag design is an integration concern.

---

### Pitfall 6: Column Schema Drift Between Rust and Python Writers

**What goes wrong:** The Python `store_bars_batch()` dynamically selects columns from the DataFrame based on what's available (microstructure, inter-bar features, plugin columns, exchange sessions). It adds metadata columns (`symbol`, `threshold_decimal_bps`, `ouroboros_mode`, `opendeviationbar_version`, `source_start_ts`, `source_end_ts`, `cache_key`). The Rust writer hardcodes a column list at compile time. When a new column is added to the ClickHouse schema (e.g., a new microstructure feature), the Python writer automatically includes it but the Rust writer doesn't — creating bars with NULL values for the new column.

**Why it happens:** The Python path is dynamic: `columns = [c for c in columns if c in df.columns]`. The Rust path must know the schema at compile time (Arrow schema for HTTP POST with `FORMAT Arrow`). Schema evolution requires Rust code changes + recompile + redeploy, while Python handles it automatically.

**Consequences:** Bars written by Rust have NULL for columns that Python would have populated. Mixed NULL/non-NULL values for the same column in the same time range. ML training pipelines break on unexpected NULLs. Worse: if the Rust writer sends columns in the wrong order or with wrong types, ClickHouse silently truncates or casts — data corruption without errors.

**Prevention:**

1. Define the column schema as a shared constant. In Rust, derive it from the `CompletedBar` struct fields. In Python, import it from the same source (or validate at startup that Rust and Python agree)
2. At sidecar startup, query ClickHouse `system.columns` and validate the Rust writer's schema matches
3. Use `FORMAT ArrowStream` for Rust INSERTs — Arrow schema is self-describing, ClickHouse validates column names and types on receipt
4. Add a version tag to bars written by Rust (e.g., `opendeviationbar_version = "{version}+rust-sidecar"`) so schema drift is attributable

**Detection:** `SELECT opendeviationbar_version, count() FROM open_deviation_bars GROUP BY opendeviationbar_version` — sudden appearance of a new version tag with fewer non-NULL columns = drift.

**Phase:** Phase 1 (core writer). Schema definition is part of the writer implementation.

## Moderate Pitfalls

Mistakes that cause operational issues, performance degradation, or delayed detection.

### Pitfall 7: Connection Pool Exhaustion from Unbounded reqwest::Client

**What goes wrong:** The Rust `reqwest::Client` is created per-flush or per-batch instead of being shared across the sidecar lifetime. Each client creates a new connection pool. Under load (16 symbols x 4 thresholds = 64 pairs), each sync_flush creates 64 HTTP connections. The ClickHouse server hits its max connections limit. New connections are rejected. The Rust writer falls back to dead-letter for all pairs.

**Why it happens:** The Python path had the same issue (Issue #275: fd exhaustion from per-loop `OpenDeviationBarCache()` creation). The fix was thread-local reuse. In Rust, `reqwest::Client` already uses connection pooling internally but only if the same `Client` instance is reused. Creating a new `Client` per write defeats this.

**Prevention:**

1. Create a single `reqwest::Client` at `ClickHouseWriter` construction time, reuse for all writes
2. Configure `pool_max_idle_per_host` to match expected concurrency (4-8 connections to ClickHouse is sufficient)
3. Set connection timeout (5s) and request timeout (30s) to match Python's stamina config
4. Monitor fd count: `ls /proc/{PID}/fd | wc -l` (bigblack) — should stay under 100

**Phase:** Phase 1 (core writer). Client lifecycle is a construction concern.

---

### Pitfall 8: Dead-Letter Format Incompatibility with Python Niffler Replay

**What goes wrong:** The Rust dead-letter writes Parquet files with Arrow-native column names (e.g., `open_time` in microseconds) while Python's `replay_dead_letters()` expects columns matching the ClickHouse schema (e.g., `open_time_us`, `close_time_ms`). The replay reads the Parquet file, constructs a Polars DataFrame, and calls `store_bars_batch()` — which fails because the column names don't match.

**Why it happens:** The `CompletedBar` struct uses Rust field names. The dict export path (`opendeviationbar_to_dict()` in `src/helpers.rs`) converts `open_time` to `open_time_ms` (dividing by 1000). The Arrow export path uses raw field names. If the Rust dead-letter uses Arrow serialization directly from `CompletedBar`, the column names won't match the Python replay expectation.

**Consequences:** Dead-letter files accumulate in `/tmp/opendeviationbar-dead-letter/`. Niffler replay fails silently (catches all exceptions). Charon periodic ferry also fails. The bars are "saved" but can never be replayed. They sit there until an operator manually investigates. Meanwhile, Stathera gaps persist.

**Prevention:**

1. Rust dead-letter must produce Parquet files with the EXACT column names that Python's `store_bars_batch()` expects (lowercase, `_us` suffix for microsecond timestamps, `_ms` suffix for millisecond timestamps)
2. Write a round-trip test: Rust writes dead-letter Parquet, Python reads it and calls `store_bars_batch()` — must succeed
3. Include `_symbol` and `_threshold_decimal_bps` columns (or encode them in the filename, as Python's `_niffler_dead_letter()` does: `{symbol}_{threshold}_{timestamp}.parquet`)
4. Match the exact filename pattern so Charon's glob finds them

**Phase:** Phase 2 (Niffler dead-letter). Must be validated with a cross-language integration test.

---

### Pitfall 9: Metrics Blackout During Cutover — No Visibility Into Write Performance

**What goes wrong:** The Python write path logs extensively: `cache_write: ouroboros_mode=... symbol=... threshold=... bars=...`, `Bulk cached N bars`, sync_flush diagnostics. The Rust writer handles HTTP transport but doesn't emit equivalent Python-visible logs. During rollout, operators lose visibility into write latency, success rate, and batch sizes. When something goes wrong, there's no log trail.

**Why it happens:** Rust logging (via `tracing` or `log` crate) doesn't automatically surface in the Python logger. The sidecar's log aggregation (file + Telegram) only captures Python `logging` output. Rust-side metrics are invisible unless explicitly bridged.

**Consequences:** Debugging production issues becomes guesswork. "Did the bars get written?" requires ClickHouse queries instead of log grep. The 5-50ms per-batch latency improvement is unmeasurable without metrics. Regression detection is impossible.

**Prevention:**

1. The Rust `ClickHouseWriter` must expose metrics via PyO3: `bars_written`, `bars_failed`, `batches_flushed`, `avg_write_latency_ms`, `last_write_at`
2. The Python integration layer logs these metrics after each flush cycle
3. Add Rust writer stats to the `/status` HTTP endpoint
4. During shadow mode, log both Python and Rust latencies side-by-side for comparison

**Phase:** Phase 1 (core writer) for metric struct, Phase 3 (integration) for Python bridging.

---

### Pitfall 10: ClickHouse HTTP Chunked Encoding Edge Cases

**What goes wrong:** Large batch POSTs (>1MB) use HTTP chunked transfer encoding. ClickHouse may return an error mid-stream (e.g., "Too many parts" during an ongoing mutation, or "Memory limit exceeded"). The error response arrives after the client has already sent the full request body. `reqwest` reports this as a successful send + error response, but the exact behavior depends on HTTP/1.1 vs HTTP/2 and the ClickHouse version.

**Why it happens:** ClickHouse's HTTP interface processes the INSERT as it reads the request body. If an error occurs after partial consumption, the behavior is undefined — some rows may have been inserted, the response may be a 500 with partial error text, or the connection may reset.

**Prevention:**

1. Cap batch size at 10,000 bars (~500KB in Arrow format) — well under chunked encoding thresholds
2. Use `Content-Length` header (not chunked encoding) by buffering the full body before sending
3. On any non-200 response, treat the entire batch as failed and dead-letter it
4. Set `max_insert_block_size` in query params to match the batch size cap
5. Test with ClickHouse under memory pressure (set `max_memory_usage=100000000` in query params for testing)

**Phase:** Phase 1 (core writer). HTTP transport details.

---

### Pitfall 11: Ouroboros Mode Not Passed Through Rust Writer

**What goes wrong:** The Python `store_bars_batch()` resolves `ouroboros_mode` from config if not specified, checks the symbol registry, logs warnings on mismatch, and runs `_guard_ouroboros_mode_consistency()`. The Rust writer only knows about `CompletedBar` (which has `symbol` and `threshold_decimal_bps` but no `ouroboros_mode`). The Rust INSERT omits the `ouroboros_mode` column or defaults to the wrong value.

**Why it happens:** `ouroboros_mode` is a Python-layer concept from `symbols.toml`. It's not in the `CompletedBar` struct because the Rust streaming engine doesn't need it for bar construction. But ClickHouse's ORDER BY includes `ouroboros_mode`, so it's required for correct ReplacingMergeTree dedup.

**Consequences:** Bars written with wrong `ouroboros_mode` sort into wrong ORDER BY groups. ReplacingMergeTree can't dedup them against existing bars. Mixed modes in the same partition cause `_guard_ouroboros_mode_consistency()` to fire when Python writers later write to the same partition.

**Prevention:**

1. Add `ouroboros_mode: &str` to the `ClickHouseWriter` constructor, populated from `symbol_modes` dict at sidecar startup
2. Or: add `ouroboros_mode` field to `CompletedBar` (upstream change in Rust streaming engine)
3. Validate at startup: for each symbol, assert the mode from `symbols.toml` matches what's in ClickHouse

**Phase:** Phase 1 (core writer). The mode must be included in every INSERT.

## Minor Pitfalls

### Pitfall 12: ClickHouse HTTP URL Query Parameter Encoding

**What goes wrong:** INSERT settings (`insert_deduplicate=1`, `insert_deduplication_token=...`) are passed as URL query parameters. If the dedup token contains special characters (the MD5 hex string is safe, but future token formats might not be), incorrect URL encoding causes ClickHouse to reject the request or misparse the token.

**Prevention:** Always URL-encode query parameters. Use `reqwest`'s `.query()` builder, not manual string concatenation.

**Phase:** Phase 1.

---

### Pitfall 13: Arrow Format Version Mismatch

**What goes wrong:** The Rust writer uses `arrow-rs` to serialize bars as Arrow IPC. ClickHouse expects a specific Arrow IPC format version. If `arrow-rs` defaults to Arrow IPC v5 (the "new" format) but ClickHouse only supports v4, the INSERT fails with a cryptic parsing error.

**Prevention:** Explicitly set Arrow IPC writer options to use the format version ClickHouse supports. Test with the exact ClickHouse version running on bigblack.

**Phase:** Phase 1.

---

### Pitfall 14: Rust Panic in Writer Thread Kills Sidecar Silently

**What goes wrong:** An unwrap/expect in the Rust writer panics (e.g., on an unexpected None value). If the writer runs in a Rust thread (not tokio), the panic unwinds and the thread dies. Python doesn't know the thread died. The sidecar continues running but all CH writes silently fail because the writer channel is disconnected.

**Prevention:** Use `std::panic::catch_unwind` at the writer thread boundary. On panic, log the error, dead-letter the batch, and signal the Python layer to fall back. Never use `.unwrap()` on network responses.

**Phase:** Phase 1.

## Phase-Specific Warnings

| Phase Topic                    | Likely Pitfall                                                                                    | Mitigation                                                                                       |
| ------------------------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| Phase 1: Core ClickHouseWriter | Dedup token incompatibility (#1), Silent write failure (#2), Guard bypass (#3), Schema drift (#6) | Cross-language dedup test, parse X-ClickHouse-Summary, port all 6 guards, shared schema constant |
| Phase 2: Niffler dead-letter   | Dead-letter format mismatch (#8), Partial batch (#4)                                              | Round-trip test (Rust write, Python read), Parquet column name exact match                       |
| Phase 3: Sidecar integration   | Feature flag race (#5), Metrics blackout (#9)                                                     | Once-at-startup flag check, PyO3 metric bridge                                                   |
| Phase 4: Production rollout    | Both writers active (#5), Connection exhaustion (#7)                                              | Shadow mode validation, single reqwest::Client lifecycle                                         |
| Phase 5: Python writer removal | Schema drift discovered late (#6), Guard coverage gaps (#3)                                       | Keep Python guards as assertions for 2 weeks after Rust-only cutover                             |

## Historical Precedent in This Codebase

These pitfalls are grounded in actual incidents and architectural decisions from the project history:

| Precedent                                                               | Relevant Pitfall              | Lesson                                                                        |
| ----------------------------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------------------- |
| Feb 2026 silent write failure (try_cache_write swallowed exceptions)    | #2 Silent write failure       | NEVER swallow write errors on the hot path — use fatal writes                 |
| Issue #269 phantom gaps from lightweight DELETE (banned since v13.49.0) | #1 Dedup token divergence     | Small semantic differences in CH operations cause downstream chaos            |
| Issue #275 fd exhaustion from per-loop OpenDeviationBarCache            | #7 Connection exhaustion      | Connection lifecycle must be explicit and tested under load                   |
| Issue #90 dedup token collision (empty dict left server-default active) | #1, #2                        | Dedup settings must be explicit, never rely on server defaults                |
| Issue #280 concurrent writer races (sidecar vs kintsugi)                | #5 Feature flag race          | Writer ownership must be exclusive — never two writers for the same TID range |
| Day-mode/Aion mode confusion across callers                             | #11 Ouroboros mode not passed | Metadata that's "obvious" in Python gets lost at Rust boundary                |

## Sources

- `/python/opendeviationbar/clickhouse/bulk_operations.py` — pre-write guards, dedup token generation, cache_key formula
- `/python/opendeviationbar/sidecar/clickhouse_write.py` — current stamina retry + Niffler dead-letter integration
- `/python/opendeviationbar/sidecar/dead_letter.py` — Niffler dead-letter format, filename convention, replay path
- `/python/opendeviationbar/clickhouse/cache.py` — CacheKey hash_key formula (MD5 of formatted string)
- `/crates/opendeviationbar-streaming/src/engine/traits.rs` — BarSink trait, SinkError types
- `/crates/opendeviationbar-streaming/src/live_engine/types.rs` — CompletedBar struct (symbol, threshold, bar)
- `.planning/quick/260327-dedup-collision-fix-validation/PLAN.md` — dedup token collision incident
- `.planning/quick/guard-investigation/ch-write-findings.md` — written_rows=0 silent drop behavior
- ClickHouse HTTP interface documentation — HIGH confidence (verified against deployed v24.x)
- Project incident history (Issue #269, #275, #280, #90) — HIGH confidence (lived experience)
