---
phase: 35-dead-letter-resilience
plan: 02
type: execute
wave: 2
depends_on: ["35-01"]
files_modified:
  - crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
  - tests/test_dead_letter_roundtrip.py
autonomous: true
requirements: [RESIL-05, RESIL-02]

must_haves:
  truths:
    - "After a successful flush, the flush thread periodically scans /tmp/opendeviationbar-dead-letter/ for Parquet files and replays them to ClickHouse"
    - "Niffler replay reads Parquet back into Arrow, converts to JSONEachRow, POSTs to ClickHouse, and deletes the file on success"
    - "A Parquet file written by Rust dead_letter::write_dead_letter can be read by Python polars.read_parquet with all columns intact"
    - "Replay stops on first failure (ClickHouse likely down)"
  artifacts:
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs"
      provides: "niffler_replay function for scanning and replaying dead-letter files"
      exports: ["niffler_replay"]
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs"
      provides: "Periodic niffler replay call after successful flushes"
      contains: "niffler_replay"
    - path: "tests/test_dead_letter_roundtrip.py"
      provides: "Cross-language round-trip test: Rust writes Parquet, Python reads with Polars"
  key_links:
    - from: "flush_thread.rs"
      to: "dead_letter.rs"
      via: "niffler_replay call after successful flush"
      pattern: "dead_letter::niffler_replay"
    - from: "dead_letter.rs"
      to: "ClickHouse HTTP"
      via: "reqwest POST with JSONEachRow format"
      pattern: "reqwest.*POST"
---

<objective>
Add Niffler replay (automatic dead-letter file recovery) to the flush thread, and create a cross-language round-trip test proving Rust-written Parquet files are readable by Python Charon.

Purpose: Complete the dead-letter lifecycle -- bars are saved on failure (Plan 01) and automatically replayed on recovery (this plan). The cross-language test gates RESIL-02.
Output: niffler_replay function in dead_letter.rs, flush thread integration, Python round-trip test.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/35-dead-letter-resilience/35-RESEARCH.md
@.planning/phases/35-dead-letter-resilience/35-01-SUMMARY.md

<interfaces>
<!-- Interfaces from Plan 01 that this plan builds on -->

From crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs (created in Plan 01):

```rust
pub const DEAD_LETTER_DIR: &str = "/tmp/opendeviationbar-dead-letter";
pub fn write_dead_letter(rows: &[ClickHouseBarRow]) -> Result<PathBuf, DeadLetterError>;
pub fn write_dead_letter_single(row: ClickHouseBarRow) -> Result<PathBuf, DeadLetterError>;
pub fn dead_letter_schema() -> Schema;
pub enum DeadLetterError { IoError(std::io::Error), ArrowError(String) }
```

From crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs:

```rust
pub fn spawn_flush_thread(rx, config, metrics) -> JoinHandle<()>;
fn do_flush(rt, http_client, buffer, config, metrics);
fn serialize_json_each_row(rows: &[ClickHouseBarRow]) -> Result<String, FlushError>;
```

From crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs:

```rust
pub struct ClickHouseWriterConfig {
    pub url: String,
    pub database: String,
    pub table: String,
    // ...
}
```

From python/opendeviationbar/kintsugi/dead_letter.py:

```python
def replay_dead_letters() -> int:
    # Reads *.parquet from _DEAD_LETTER_DIR
    # Parses filename: parts = path.stem.split("_"), parts[0]=symbol, parts[1]=threshold
    # Reads with polars.read_parquet(path)
    # Writes via cache.store_bars_batch(symbol, threshold, bars_df, ...)
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Implement Niffler replay in dead_letter.rs and wire into flush thread</name>
  <files>
    crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs
  </read_first>
  <behavior>
    - Test 1: niffler_replay with no dead-letter directory returns 0
    - Test 2: niffler_replay with empty directory returns 0
    - Test 3: niffler_replay with a .parquet file + mock HTTP 200 reads the file, POSTs to CH, deletes the file, returns row count
    - Test 4: niffler_replay with a .parquet file + mock HTTP 503 stops and leaves the file intact
    - Test 5: niffler_replay ignores non-.parquet files in the directory
    - Test 6: niffler_replay uses .replaying extension as advisory lock during replay
  </behavior>
  <action>
    1. **Add niffler_replay function to dead_letter.rs**:

       ```rust
       pub fn niffler_replay(
           rt: &tokio::runtime::Runtime,
           http_client: &reqwest::Client,
           config: &ClickHouseWriterConfig,
       ) -> usize
       ```

       Implementation:
       - Check if DEAD_LETTER_DIR exists, return 0 if not
       - Read directory entries sorted alphabetically (oldest first by timestamp in filename)
       - For each .parquet file:
         a. Rename to `{path}.replaying` as advisory lock (prevents Python Charon race -- research pitfall 6)
         b. Read Parquet file using `parquet::arrow::arrow_reader::ParquetRecordBatchReaderBuilder`
         c. For each RecordBatch: convert to JSONEachRow by iterating columns and building JSON objects matching column names from the Arrow schema
         d. POST to ClickHouse using same URL/database/table format as do_flush
         e. On success: delete the `.replaying` file, increment counter
         f. On failure: rename `.replaying` back to `.parquet`, break (stop on first failure -- CH likely down)
       - Return total rows replayed

    2. **Add helper: record_batch_to_json_each_row**:
       - Takes a RecordBatch, returns a String of newline-separated JSON objects
       - For each row index, build a JSON object with field_name: value for all columns
       - Handle Arrow types: Utf8 -> string, Float64 -> number, Int64 -> number, UInt32 -> number, UInt8 -> number, Boolean -> bool
       - Handle nullable columns: if null, emit JSON null

    3. **Wire niffler_replay into flush_thread.rs**:
       - Add an `Instant` timer (`last_niffler_check`) initialized to `Instant::now()` before the main loop
       - After each successful do_flush (in the `Ok(response)` branch), check if 60 seconds have elapsed since `last_niffler_check`
       - If yes: call `dead_letter::niffler_replay(rt, &http_client, &config)`, reset timer
       - Log replay count if > 0
       - Update `metrics.dead_letter_count` appropriately (subtract replayed count? No -- dead_letter_count tracks total dead-lettered, not current. Add a new `dead_letter_replayed` AtomicU64 to FlushThreadMetrics if needed, or just log)

    4. **Add `dead_letter_replayed: AtomicU64` to FlushThreadMetrics** for observability.

    CRITICAL per research anti-pattern: Niffler replay must NOT block normal flush flow. The 60-second timer ensures replay is opportunistic. If replay takes too long, the flush thread's normal recv_timeout continues after.

    CRITICAL per research pitfall 6 (race condition): Use rename-to-.replaying as advisory lock. ClickHouse dedup tokens prevent duplicate inserts even if both Rust Niffler and Python Charon replay the same file.

  </action>
  <verify>
    <automated>cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(niffler) | test(dead_letter)' -x</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "pub fn niffler_replay" crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
    - grep -q "niffler_replay" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q "last_niffler_check\|niffler" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q "dead_letter_replayed" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q "replaying" crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
  </acceptance_criteria>
  <done>niffler_replay reads Parquet dead-letter files, converts to JSONEachRow, POSTs to ClickHouse, deletes on success. Flush thread calls it every 60 seconds after successful flushes. Advisory .replaying lock prevents race with Python Charon. All tests pass.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Cross-language round-trip test (Rust writes Parquet, Python reads)</name>
  <files>
    tests/test_dead_letter_roundtrip.py
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs,
    python/opendeviationbar/kintsugi/dead_letter.py,
    python/opendeviationbar/clickhouse/bulk_operations.py
  </read_first>
  <behavior>
    - Test 1: Rust-written dead-letter Parquet readable by polars.read_parquet with all 77 columns present
    - Test 2: Column names exactly match CORE_COLUMNS (no renaming, no missing, no extra)
    - Test 3: Nullable columns (lookback_*, intra_*) correctly represent null as Python None
    - Test 4: Non-nullable numeric fields have correct types (f64 -> Float64, u32 -> UInt32, etc.)
    - Test 5: Filename follows {symbol}_{threshold}_{timestamp}.parquet pattern parseable by Python Charon
  </behavior>
  <action>
    1. Create `tests/test_dead_letter_roundtrip.py`:

       ```python
       """Cross-language dead-letter round-trip test (RESIL-02).

       Verifies Rust-written Parquet dead-letter files are readable by Python
       polars.read_parquet() with correct column names, types, and nullable handling.
       """
       import subprocess
       import tempfile
       from pathlib import Path

       import polars as pl
       import pytest
       ```

    2. **Write a Rust test binary** that creates a dead-letter Parquet file with known values:
       - Add a `#[test]` in dead_letter.rs that writes a known ClickHouseBarRow to a temp directory
       - Use `DEAD_LETTER_DIR` override via a helper that accepts a custom path
       - OR: more practical approach -- use the existing write_dead_letter function from the Rust test suite and read the output path

    3. **Practical approach for the Python test**:
       - The test calls `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(write_dead_letter_for_roundtrip)' -x` which writes a known Parquet file to a temp path and prints the path to stdout
       - OR (simpler): The Rust dead_letter tests already create Parquet files in /tmp/opendeviationbar-dead-letter/. The Python test can:
         a. Call a small Rust test helper that writes a dead-letter file and prints the path
         b. Read that file with polars
         c. Verify columns, types, nullability

    4. **Simplest approach** (recommended):
       - Add a `write_test_dead_letter` function exposed to Python via PyO3 in the test, OR
       - Better: Create the test Parquet file using `cargo test` first, then read with Python
       - Best: Add an integration test that uses subprocess to run a small Rust binary that writes the file, then reads with Python

    5. **Actual implementation** (most practical):
       - In dead_letter.rs tests, add `test_write_dead_letter_roundtrip_artifact` that writes a Parquet file to a known temp path and asserts it exists
       - In test_dead_letter_roundtrip.py:
         a. Run `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(roundtrip_artifact)' -x` to generate the file
         b. Find the most recent .parquet file in /tmp/opendeviationbar-dead-letter/
         c. Read with `pl.read_parquet(path)`
         d. Assert all 77 column names present and match CORE_COLUMNS
         e. Assert nullable columns have correct Polars types (Float64, UInt32, Int64 with null)
         f. Assert non-nullable columns have values (no nulls)
         g. Parse filename to verify {symbol}_{threshold}_{timestamp} format
         h. Clean up the test file

    6. **Column name verification**: Import or hardcode the expected CORE_COLUMNS list in the Python test. Compare against `df.columns`. They must be identical in order and naming.

    7. **Type verification**: Check key columns:
       - `symbol` -> pl.Utf8
       - `threshold_decimal_bps` -> pl.UInt32
       - `close_time_us` -> pl.Int64
       - `open` -> pl.Float64
       - `has_gap` -> pl.Boolean
       - `lookback_ofi` -> pl.Float64 (nullable)
       - `exchange_session_sydney` -> pl.UInt8

    Note: Use `--python 3.13` for any uv run commands per project policy.

  </action>
  <verify>
    <automated>uv run --python 3.13 pytest tests/test_dead_letter_roundtrip.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "CORE_COLUMNS\|core_columns\|RESIL-02" tests/test_dead_letter_roundtrip.py
    - grep -q "polars\|pl.read_parquet" tests/test_dead_letter_roundtrip.py
    - grep -q "def test_" tests/test_dead_letter_roundtrip.py
  </acceptance_criteria>
  <done>Cross-language round-trip test passes: Rust writes a dead-letter Parquet, Python reads it with Polars, all 77 columns present with correct names, types, and nullable handling. Filename parseable by Charon. RESIL-02 gated.</done>
</task>

</tasks>

<verification>
- `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -x` -- all Rust tests pass including niffler replay
- `uv run --python 3.13 pytest tests/test_dead_letter_roundtrip.py -x` -- cross-language round-trip passes
- `cargo clippy -p opendeviationbar-streaming --features clickhouse-sink -- -D warnings` -- no warnings
</verification>

<success_criteria>

- Niffler replay scans dead-letter dir every 60s after successful flushes and replays files to ClickHouse
- Replay reads Parquet back, converts to JSONEachRow, POSTs, deletes on success
- Replay stops on first failure (ClickHouse likely down)
- Advisory .replaying lock prevents race with Python Charon
- Cross-language round-trip test proves Rust Parquet readable by Python Polars with all 77 columns
  </success_criteria>

<output>
After completion, create `.planning/phases/35-dead-letter-resilience/35-02-SUMMARY.md`
</output>
