---
phase: 35-dead-letter-resilience
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - crates/opendeviationbar-streaming/Cargo.toml
  - crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
  - Cargo.toml
autonomous: true
requirements: [RESIL-01, RESIL-02, RESIL-03, RESIL-04]

must_haves:
  truths:
    - "When do_flush() fails after all retries, the failed batch is serialized to a Parquet file in /tmp/opendeviationbar-dead-letter/"
    - "When on_bar() gets TrySendError::Full, the overflow bar is dead-lettered to Parquet instead of silently dropped"
    - "Dead-letter Parquet files use {symbol}_{threshold}_{unix_timestamp}.parquet naming"
    - "Dead-letter Parquet schema uses exact column names from CORE_COLUMNS with correct nullable Arrow types for Option<T> fields"
  artifacts:
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs"
      provides: "Parquet dead-letter writer with Arrow schema matching Python Charon expectations"
      exports: ["write_dead_letter", "dead_letter_schema"]
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs"
      provides: "Flush failure path calls dead_letter::write_dead_letter instead of dropping bars"
      contains: "dead_letter::write_dead_letter"
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs"
      provides: "Backpressure path dead-letters overflow bars"
      contains: "dead_letter::write_dead_letter"
  key_links:
    - from: "flush_thread.rs"
      to: "dead_letter.rs"
      via: "write_dead_letter call on flush failure"
      pattern: "dead_letter::write_dead_letter"
    - from: "sink.rs"
      to: "dead_letter.rs"
      via: "write_dead_letter_single call on backpressure"
      pattern: "dead_letter"
---

<objective>
Create the Parquet dead-letter writer module and wire it into both the flush failure path and backpressure path, so no bars are ever silently lost.

Purpose: Eliminate silent bar loss when ClickHouse is unreachable or flush thread falls behind. Failed bars serialize to Parquet files compatible with Python Charon replay.
Output: `dead_letter.rs` module with Arrow schema, flush_thread.rs and sink.rs wired to dead-letter on failure.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/35-dead-letter-resilience/35-RESEARCH.md

<interfaces>
<!-- Key types and contracts the executor needs. -->

From crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs:

```rust
pub const CORE_COLUMNS: &[&str] = &[
    "symbol", "threshold_decimal_bps", "close_time_us", "open_time_us",
    "open", "high", "low", "close", "volume", "vwap", "buy_volume", "sell_volume",
    "individual_trade_count", "agg_record_count", "duration_us", "ofi",
    "vwap_close_deviation", "price_impact", "kyle_lambda_proxy", "trade_intensity",
    "volume_per_trade", "aggression_ratio", "aggregation_density", "turnover_imbalance",
    "ouroboros_mode",
    "exchange_session_sydney", "exchange_session_tokyo", "exchange_session_london", "exchange_session_newyork",
    "lookback_trade_count", "lookback_ofi", "lookback_duration_us", "lookback_intensity",
    "lookback_vwap_raw", "lookback_vwap_position", "lookback_count_imbalance",
    "lookback_kyle_lambda", "lookback_burstiness", "lookback_volume_skew", "lookback_volume_kurt",
    "lookback_price_range", "lookback_kaufman_er", "lookback_garman_klass_vol",
    "lookback_hurst", "lookback_permutation_entropy",
    "intra_bull_epoch_density", "intra_bear_epoch_density", "intra_bull_excess_gain",
    "intra_bear_excess_gain", "intra_bull_cv", "intra_bear_cv",
    "intra_max_drawdown", "intra_max_runup", "intra_trade_count", "intra_ofi",
    "intra_duration_us", "intra_intensity", "intra_vwap_position", "intra_count_imbalance",
    "intra_kyle_lambda", "intra_burstiness", "intra_volume_skew", "intra_volume_kurt",
    "intra_kaufman_er", "intra_garman_klass_vol", "intra_hurst", "intra_permutation_entropy",
    "has_gap", "gap_trade_count", "max_gap_duration_us", "is_exchange_gap",
    "first_agg_trade_id", "last_agg_trade_id", "is_orphan",
    "cache_key", "opendeviationbar_version", "source_start_ts", "source_end_ts",
];

#[derive(Debug, Clone, clickhouse::Row, serde::Serialize)]
pub struct ClickHouseBarRow {
    pub symbol: String,
    pub threshold_decimal_bps: u32,
    // ... 77 fields total, see row.rs for full struct
}

impl ClickHouseBarRow {
    pub fn from_completed_bar(bar: &CompletedBar) -> Self;
}
```

From crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs:

```rust
pub enum FlushCommand {
    Bar(Box<ClickHouseBarRow>),
    Flush,
    Shutdown,
}

pub struct FlushThreadMetrics {
    pub bars_flushed: AtomicU64,
    pub bars_failed: AtomicU64,
    pub flush_count: AtomicU64,
    pub last_flush_latency_us: AtomicU64,
    pub dead_letter_count: AtomicU64,
}
```

From crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs:

```rust
pub struct ClickHouseWriterSink {
    tx: SyncSender<FlushCommand>,
    // ...
}

impl BarSink for ClickHouseWriterSink {
    fn on_bar(&mut self, bar: &CompletedBar) -> Result<(), SinkError>;
}
```

From crates/opendeviationbar-streaming/Cargo.toml (clickhouse-sink feature):

```toml
clickhouse-sink = ["dep:clickhouse", "dep:md5", "dep:serde_json", "dep:reqwest", "dep:backon", "binance-integration"]
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Create dead_letter.rs Parquet writer module</name>
  <files>
    crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs,
    crates/opendeviationbar-streaming/Cargo.toml,
    Cargo.toml
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs,
    crates/opendeviationbar-streaming/Cargo.toml,
    Cargo.toml,
    python/opendeviationbar/sidecar/dead_letter.py,
    python/opendeviationbar/kintsugi/dead_letter.py
  </read_first>
  <behavior>
    - Test 1: write_dead_letter with 3 rows creates a Parquet file in /tmp/opendeviationbar-dead-letter/ with {symbol}_{threshold}_{timestamp}.parquet naming
    - Test 2: write_dead_letter_single with 1 row creates a valid Parquet file
    - Test 3: dead_letter_schema() returns Schema with correct nullable flags -- all lookback_* and intra_* fields are nullable, all core fields are non-nullable
    - Test 4: Parquet file contains correct column count matching CORE_COLUMNS length (77 columns)
    - Test 5: Parquet file written with ZSTD compression (verify via parquet metadata)
    - Test 6: write_dead_letter with empty slice returns Ok without creating a file
  </behavior>
  <action>
    1. Add workspace dependency for parquet crate in root Cargo.toml:
       ```toml
       parquet = { version = "58", default-features = false, features = ["snap", "zstd"] }
       ```

    2. Add to crates/opendeviationbar-streaming/Cargo.toml:
       - Add `parquet = { workspace = true, optional = true }` to [dependencies]
       - Update clickhouse-sink feature to include: `"dep:parquet", "dep:arrow-array", "dep:arrow-schema"`

    3. Create `dead_letter.rs` with:
       - `const DEAD_LETTER_DIR: &str = "/tmp/opendeviationbar-dead-letter"` -- matches Python `_DEAD_LETTER_DIR`
       - `pub fn dead_letter_schema() -> Schema` -- builds Arrow Schema with all 77 columns from CORE_COLUMNS. Field types:
         - String fields (symbol, ouroboros_mode, cache_key, opendeviationbar_version): Utf8, non-nullable
         - u32 fields (threshold_decimal_bps, individual_trade_count, agg_record_count): UInt32, non-nullable
         - i64 fields (close_time_us, open_time_us, duration_us, etc.): Int64, non-nullable
         - f64 fields (open, high, low, close, volume, etc.): Float64, non-nullable
         - bool fields (has_gap, is_exchange_gap): Boolean, non-nullable
         - u8 fields (exchange_session_*, is_orphan): UInt8, non-nullable
         - Option<u32> lookback_trade_count: UInt32, NULLABLE
         - Option<f64> lookback/intra features: Float64, NULLABLE
         - Option<i64> lookback_duration_us, intra_duration_us: Int64, NULLABLE
         - Option<u32> intra_trade_count: UInt32, NULLABLE

       - `fn rows_to_record_batch(rows: &[ClickHouseBarRow], schema: &Schema) -> Result<RecordBatch, DeadLetterError>` -- builds columnar Arrow arrays from row slice. For each field, iterate all rows and collect into the appropriate Arrow array builder. Use `map(|r| r.field)` pattern for non-nullable, `map(|r| r.field.as_ref().copied())` for nullable.

       - `pub fn write_dead_letter(rows: &[ClickHouseBarRow]) -> Result<PathBuf, DeadLetterError>`:
         - Return Ok(PathBuf::new()) if rows is empty
         - Create dir with create_dir_all
         - Filename: `{symbol}_{threshold}_{unix_secs}.parquet` using rows[0].symbol and rows[0].threshold_decimal_bps
         - Use `ArrowWriter::try_new(file, schema, Some(props))` with WriterProperties using ZSTD compression
         - Write single RecordBatch, close writer
         - Log with tracing::warn (dead-letter is a warning-level event)
         - Return path

       - `pub fn write_dead_letter_single(row: ClickHouseBarRow) -> Result<PathBuf, DeadLetterError>`:
         - Delegates to `write_dead_letter(&[row])`

       - `pub enum DeadLetterError` with IoError, ArrowError variants, implements Display and From conversions

    4. Update mod.rs to add `pub mod dead_letter;` (conditionally behind clickhouse-sink feature, same as other modules)

    CRITICAL per RESIL-02: Column names MUST exactly match CORE_COLUMNS from row.rs. Python Charon reads dead-letter Parquet via `polars.read_parquet()` and passes to `store_bars_batch()` which expects these exact column names. Any mismatch causes silent data loss.

    CRITICAL per research pitfall 5 (buffer ownership): The flush_thread.rs currently does `std::mem::take(buffer)` at line 199, clearing the buffer BEFORE the POST result. This must be fixed in Task 2 so failed rows are available for dead-lettering. For now, just create the dead_letter module.

  </action>
  <verify>
    <automated>cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(dead_letter)' -x</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "pub fn write_dead_letter" crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
    - grep -q "pub fn dead_letter_schema" crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
    - grep -q "pub mod dead_letter" crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
    - grep -q 'dep:parquet' crates/opendeviationbar-streaming/Cargo.toml
    - grep -q "DEAD_LETTER_DIR" crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
    - grep -q "Zstd\|ZSTD\|zstd" crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
  </acceptance_criteria>
  <done>dead_letter.rs module exists with write_dead_letter, write_dead_letter_single, dead_letter_schema functions. All tests pass. Parquet files use ZSTD compression, correct Arrow types with nullable flags for Option fields, and CORE_COLUMNS-exact naming.</done>
</task>

<task type="auto">
  <name>Task 2: Wire dead-letter into flush failure and backpressure paths</name>
  <files>
    crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
  </read_first>
  <action>
    1. **Fix buffer ownership in flush_thread.rs** (research pitfall 5):
       - Currently line 199 does `let _ = std::mem::take(buffer)` BEFORE the HTTP POST result. The buffer is gone by the time we know if the POST failed.
       - Restructure: serialize JSONEachRow body from `&buffer` (borrow, not take), then POST. On success: `buffer.clear()`. On failure: pass `std::mem::take(buffer)` to `dead_letter::write_dead_letter()`, then the buffer is empty either way.
       - Also handle serialization failure: if `serialize_json_each_row` fails, dead-letter the buffer contents as well (they're still valid ClickHouseBarRows even if JSON serialization failed).

    2. **Wire dead-letter into do_flush failure path** (per RESIL-01, RESIL-03):
       - In the `Err(e)` branch (lines 293-305): call `dead_letter::write_dead_letter()` with the failed rows
       - Log the dead-letter path on success, log CRITICAL if dead-letter write also fails (bars truly lost)
       - The existing `metrics.dead_letter_count.fetch_add(1, ...)` stays

    3. **Wire dead-letter into serialization failure path**:
       - In the serialization error path (lines 186-196): call `dead_letter::write_dead_letter()` before clearing buffer
       - Same pattern: log path on success, CRITICAL on double failure

    4. **Wire dead-letter into sink.rs backpressure path** (per RESIL-04):
       - In `on_bar()` TrySendError::Full branch: extract the `ClickHouseBarRow` from the `FlushCommand::Bar(boxed_row)` and call `dead_letter::write_dead_letter_single(*boxed_row)`
       - Pattern match on `TrySendError::Full(FlushCommand::Bar(boxed_row))` to recover the row
       - Note: TrySendError::Full returns the value that couldn't be sent, so we can extract it
       - Log warning with dead-letter path
       - Keep the existing `bars_dropped` counter increment and `SinkError::Recoverable` return

    5. **Update existing tests**:
       - The test_sink_backpressure test should now verify a dead-letter file was created (check /tmp/opendeviationbar-dead-letter/ for .parquet files, clean up after)
       - test_flush_on_row_count_triggers and similar failure tests should verify dead_letter_count metric and that Parquet files appear

    CRITICAL: When pattern matching TrySendError::Full, the contained value is the FlushCommand that was being sent. Since we only send FlushCommand::Bar in on_bar(), matching `Full(FlushCommand::Bar(boxed_row))` is safe. For Flush/Shutdown variants in the Full arm, just log and ignore (should not happen).

  </action>
  <verify>
    <automated>cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -x</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "dead_letter::write_dead_letter" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q "dead_letter" crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
    - grep -rqv "let _ = std::mem::take(buffer)" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs || echo "buffer take removed"
  </acceptance_criteria>
  <done>Flush failure path dead-letters rows to Parquet instead of dropping them. Backpressure path dead-letters overflow bars. Buffer ownership restructured so rows are available for dead-lettering on POST failure. All existing tests pass, backpressure test verifies dead-letter file creation.</done>
</task>

</tasks>

<verification>
- `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -x` -- all tests pass
- `cargo clippy -p opendeviationbar-streaming --features clickhouse-sink -- -D warnings` -- no warnings
- Dead-letter files appear in `/tmp/opendeviationbar-dead-letter/` when flush fails (verified by tests)
</verification>

<success_criteria>

- No bar is silently dropped on flush failure -- all failed batches produce Parquet dead-letter files
- No bar is silently dropped on backpressure -- overflow bars produce Parquet dead-letter files
- Dead-letter Parquet schema has exactly 77 columns matching CORE_COLUMNS with correct nullable types
- ZSTD compression used for Parquet files
- Filename pattern: {symbol}_{threshold}_{unix_timestamp}.parquet
  </success_criteria>

<output>
After completion, create `.planning/phases/35-dead-letter-resilience/35-01-SUMMARY.md`
</output>
