---
phase: 35-dead-letter-resilience
plan: "02"
status: complete
started: "2026-03-29"
completed: "2026-03-29"
subsystem: clickhouse-writer
tags: [dead-letter, niffler, replay, parquet, cross-language]
dependency_graph:
  requires: [35-01]
  provides: [niffler-replay, dead-letter-roundtrip]
  affects: [flush-thread, kintsugi-dead-letter]
tech_stack:
  added: []
  patterns:
    [advisory-lock-rename, arrow-to-json-each-row, background-mock-server]
key_files:
  created:
    - crates/opendeviationbar-streaming/src/clickhouse_writer/niffler.rs
    - tests/test_dead_letter_roundtrip.py
  modified:
    - crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
    - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs
decisions:
  - Niffler replay split into dedicated niffler.rs module (dead_letter.rs stayed under 1000 lines)
  - Mock server tests use separate thread+runtime to avoid nested block_on panic
  - Column count is 78 (not 77 as stated in plan) -- matches actual CORE_COLUMNS in row.rs
metrics:
  duration: 15m
  completed: "2026-03-29"
  tasks: 2
  files_created: 2
  files_modified: 3
---

# Phase 35 Plan 02: Niffler Replay + Cross-Language Round-Trip Summary

Niffler replay module reads dead-letter Parquet files, converts Arrow RecordBatches to JSONEachRow, POSTs to ClickHouse with advisory .replaying lock, wired into flush thread every 60 seconds; Python round-trip test proves Rust Parquet readable by Polars with all 78 columns intact.

## Tasks Completed

| #   | Task                                                               | Commit  | Status   |
| --- | ------------------------------------------------------------------ | ------- | -------- |
| 1   | Implement Niffler replay in niffler.rs and wire into flush thread  | fa07f68 | Complete |
| 2   | Cross-language round-trip test (Rust writes Parquet, Python reads) | 5472910 | Complete |

## Key Implementation Details

### Task 1: Niffler Replay

- `niffler.rs`: `niffler_replay()` scans dead-letter dir, sorts alphabetically (oldest first), replays to ClickHouse
- `record_batch_to_json_each_row()` converts Arrow RecordBatch to newline-separated JSON objects, handling Utf8/Float64/Int64/UInt32/UInt8/Boolean types and null values
- Advisory `.replaying` rename prevents race with Python Charon -- both can coexist safely
- Stops on first failure (ClickHouse likely down) to avoid cascading errors
- Flush thread calls niffler every 60 seconds after successful flushes via `maybe_niffler!()` macro
- Final niffler pass runs on shutdown
- `dead_letter_replayed` AtomicU64 metric added to FlushThreadMetrics

### Task 2: Cross-Language Round-Trip Test

- Rust `test_write_roundtrip_artifact` writes a known ClickHouseBarRow with specific nullable/non-nullable values to a temp Parquet file
- Python test runs Rust test via subprocess to generate artifact, then reads with `polars.read_parquet()`
- Verifies: column names match CORE_COLUMNS exactly (order + naming), correct Polars types, null handling, known values round-trip, filename parseable by Charon
- 78 columns total (plan said 77 -- corrected to match actual CORE_COLUMNS)

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 3 - Blocking] Nested tokio runtime panic in niffler tests**

- **Found during:** Task 1
- **Issue:** Tests wrapping wiremock's `block_on` around `niffler_replay_dir` (which also calls `block_on`) caused "Cannot start a runtime from within a runtime" panic
- **Fix:** Start mock server on a separate background thread with its own tokio runtime; pass URI back via channel
- **Files modified:** `niffler.rs` (test code)
- **Commit:** fa07f68

**2. [Rule 1 - Bug] Column count 78 not 77**

- **Found during:** Task 2
- **Issue:** Plan stated 77 columns but actual CORE_COLUMNS in row.rs has 78 entries
- **Fix:** Updated Python test to assert 78 columns
- **Files modified:** `tests/test_dead_letter_roundtrip.py`
- **Commit:** 5472910

**3. [Rule 2 - Missing] Niffler module split from dead_letter.rs**

- **Found during:** Task 1
- **Issue:** Adding niffler_replay + tests to dead_letter.rs would exceed 1000-line file size limit
- **Fix:** Created dedicated `niffler.rs` module; added `rows_to_record_batch_public()` accessor to dead_letter.rs for test usage
- **Files modified:** `niffler.rs` (new), `dead_letter.rs`, `mod.rs`
- **Commit:** fa07f68

## Known Stubs

None.

## Verification Results

- `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink`: 174 tests pass
- `pytest tests/test_dead_letter_roundtrip.py -v`: 9 tests pass
- `cargo clippy -p opendeviationbar-streaming --features clickhouse-sink -- -D warnings`: clean

## Self-Check: PASSED

- [x] `niffler.rs` exists
- [x] `tests/test_dead_letter_roundtrip.py` exists
- [x] Commit fa07f68 exists
- [x] Commit 5472910 exists
