# Phase 35: Dead-Letter Resilience - Research

**Researched:** 2026-03-28
**Domain:** Rust Parquet serialization, cross-language dead-letter compatibility
**Confidence:** HIGH

## Summary

Phase 35 adds dead-letter fallback to the Rust ClickHouse writer. When the flush thread fails to write bars after all retries (or when backpressure causes overflow), failed bars are serialized to Parquet files in `/tmp/opendeviationbar-dead-letter/` using the same filename pattern and schema that Python's Charon replay expects. This ensures zero bar loss even during ClickHouse outages.

The existing Python dead-letter system (`dead_letter.py`) writes bar dicts to Parquet via `polars.DataFrame(bar_dicts).write_parquet(path)`. The Charon replay (`kintsugi/dead_letter.py`) reads these with `polars.read_parquet(path)` and calls `store_bars_batch()`. The Rust implementation must produce Parquet files that Polars can read with identical column names and compatible types. The arrow-rs `parquet` crate v58 aligns with the project's existing arrow v58 workspace dependencies.

**Primary recommendation:** Use the `parquet` crate (v58, arrow-rs ecosystem) with `ArrowWriter` to serialize `Vec<ClickHouseBarRow>` to Parquet files matching the Python dead-letter schema exactly.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None explicitly locked -- all implementation choices at Claude's discretion.

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure phase. Key guidance from CONTEXT.md:

- Dead-letter directory: `/tmp/opendeviationbar-dead-letter/`
- Filename pattern: `{symbol}_{threshold}_{timestamp}.parquet`
- Use `parquet` crate (arrow-rs ecosystem, version 58) for Parquet serialization
- Dead-letter schema must match Python Charon replay expectations
- Niffler immediate replay pattern: scan dead-letter dir on flush thread recovery
- Backpressure: on_bar() already returns SinkError::Recoverable on channel full -- extend to dead-letter overflow bars
- Cross-language round-trip test: Rust writes Parquet, Python reads via polars/pyarrow

### Deferred Ideas (OUT OF SCOPE)

None.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID       | Description                                                                                       | Research Support                                                                               |
| -------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
| RESIL-01 | Dead-letter fallback serializes failed batches to Parquet in `/tmp/opendeviationbar-dead-letter/` | Parquet crate API, schema mapping from ClickHouseBarRow, Python compatibility analysis         |
| RESIL-02 | Dead-letter Parquet schema matches Python Charon replay expectations                              | Column-by-column schema analysis of store_bars_batch() expectations vs ClickHouseBarRow fields |
| RESIL-03 | Error classification: transient (retry), fatal (dead-letter + alert), backpressure (channel full) | Existing SinkError enum analysis, flush_thread error classification                            |
| RESIL-04 | Backpressure detection via bounded SyncSender -- dead-letter overflow bars                        | sink.rs on_bar() TrySendError::Full path, ClickHouseBarRow available at call site              |
| RESIL-05 | Rust-side immediate replay of dead-letter files on flush thread recovery                          | Niffler pattern analysis, flush thread scan-replay-delete cycle                                |

</phase_requirements>

## Standard Stack

### Core

| Library        | Version        | Purpose                                     | Why Standard                                    |
| -------------- | -------------- | ------------------------------------------- | ----------------------------------------------- |
| `parquet`      | 58.1.0         | Parquet file writing                        | arrow-rs ecosystem, matches workspace arrow v58 |
| `arrow-array`  | 58 (workspace) | Arrow array construction for Parquet writer | Already in workspace, required by ArrowWriter   |
| `arrow-schema` | 58 (workspace) | Schema definition for Parquet               | Already in workspace                            |

### Supporting

| Library  | Version     | Purpose                     | When to Use          |
| -------- | ----------- | --------------------------- | -------------------- |
| `chrono` | (workspace) | Unix timestamp for filename | Already in workspace |

### Alternatives Considered

| Instead of            | Could Use                     | Tradeoff                                                     |
| --------------------- | ----------------------------- | ------------------------------------------------------------ |
| `parquet` ArrowWriter | Raw Parquet row-by-row writer | ArrowWriter is columnar (standard), row-by-row is deprecated |
| JSON dead-letter      | Parquet dead-letter           | Parquet matches existing Python system, more compact, typed  |

**Installation:**
Add to `Cargo.toml` workspace deps:

```toml
parquet = { version = "58", default-features = false, features = ["snap", "zstd"] }
```

Add to `crates/opendeviationbar-streaming/Cargo.toml`:

```toml
parquet = { workspace = true, optional = true }
```

Update `clickhouse-sink` feature:

```toml
clickhouse-sink = ["dep:clickhouse", "dep:md5", "dep:serde_json", "dep:reqwest", "dep:backon", "dep:parquet", "dep:arrow-array", "dep:arrow-schema", "binance-integration"]
```

**Version verification:** parquet 58.1.0 is current (matches arrow-rs 58 ecosystem in workspace).

## Architecture Patterns

### Recommended Module Structure

```
crates/opendeviationbar-streaming/src/clickhouse_writer/
├── mod.rs               # Re-exports (add dead_letter)
├── config.rs            # Existing config
├── dedup.rs             # Existing dedup
├── flush_thread.rs      # Modified: call dead_letter on failure, niffler replay
├── row.rs               # Existing row mapping
├── schema.rs            # Existing plugin discovery
├── sink.rs              # Modified: dead-letter on backpressure
├── summary.rs           # Existing summary parsing
└── dead_letter.rs       # NEW: Parquet serialization + Niffler replay
```

### Pattern 1: Dead-Letter on Flush Failure

**What:** When `do_flush()` fails after all retries, serialize the buffer to a Parquet file instead of dropping bars.
**When to use:** Every flush failure path in `flush_thread.rs`.
**Current code (flush_thread.rs lines 293-305):**

```rust
// Current: bars are lost on failure
Err(e) => {
    metrics.bars_failed.fetch_add(batch_size, Ordering::Relaxed);
    metrics.dead_letter_count.fetch_add(1, Ordering::Relaxed);
    tracing::error!(..., "flush failed after all retries - bars dead-lettered");
}
```

**New pattern:**

```rust
Err(e) => {
    // Serialize failed batch to Parquet dead-letter
    if let Err(dl_err) = dead_letter::write_dead_letter(&failed_rows) {
        tracing::error!(error = %dl_err, "dead-letter write ALSO failed - bars lost");
    }
    metrics.bars_failed.fetch_add(batch_size, Ordering::Relaxed);
    metrics.dead_letter_count.fetch_add(1, Ordering::Relaxed);
}
```

### Pattern 2: Dead-Letter on Backpressure

**What:** When `on_bar()` gets `TrySendError::Full`, dead-letter the bar instead of just dropping it.
**Current code (sink.rs line 70-75):**

```rust
Err(std::sync::mpsc::TrySendError::Full(_)) => {
    self.bars_dropped.fetch_add(1, Ordering::Relaxed);
    tracing::warn!("clickhouse flush thread backpressure - bar dropped");
    Err(SinkError::Recoverable("clickhouse flush thread backpressure".into()))
}
```

**New pattern:**

```rust
Err(std::sync::mpsc::TrySendError::Full(FlushCommand::Bar(boxed_row))) => {
    // Dead-letter the overflow bar
    dead_letter::write_dead_letter_single(*boxed_row);
    self.bars_dropped.fetch_add(1, Ordering::Relaxed);
    Err(SinkError::Recoverable("backpressure - bar dead-lettered".into()))
}
```

### Pattern 3: Niffler Immediate Replay

**What:** After a successful flush, scan the dead-letter directory for Parquet files. For each file: read it, attempt to POST to ClickHouse, delete on success.
**When to use:** After each successful `do_flush()` in the flush thread.
**Key constraint:** Must be non-blocking relative to normal flush flow. Run as a periodic check (e.g., every N successful flushes or every T seconds).

### Anti-Patterns to Avoid

- **Blocking replay on the flush thread hot path:** Niffler replay should be opportunistic, not blocking normal flushes. Use a counter/timer to throttle replay attempts.
- **Reading Parquet files from Rust for replay:** The Niffler replay reads Parquet and POSTs to ClickHouse. Since the Parquet schema may differ between Rust-written and Python-written files, replay should use the existing JSON POST path (read Parquet into arrow, convert to JSONEachRow, POST).
- **Building a custom Parquet writer per-field:** Use Arrow's columnar `ArrowWriter` with a pre-built schema, not manual field-by-field serialization.

## Don't Hand-Roll

| Problem                    | Don't Build                  | Use Instead                                           | Why                                                   |
| -------------------------- | ---------------------------- | ----------------------------------------------------- | ----------------------------------------------------- |
| Parquet writing            | Manual binary Parquet format | `parquet::arrow::ArrowWriter`                         | Compression, encoding, metadata handled automatically |
| Arrow schema construction  | Manual field-by-field Schema | Static `Schema` from `ClickHouseBarRow` field list    | Compile-time safety, matches CORE_COLUMNS             |
| Cross-language type compat | Guess-and-check types        | Arrow canonical types (Float64, Int64, Utf8, Boolean) | Arrow types are language-neutral by design            |

**Key insight:** Arrow/Parquet is the cross-language bridge. Rust's `arrow-rs` and Python's `pyarrow`/`polars` both implement the same Arrow spec. As long as types are canonical Arrow types (Float64, Int64, Utf8, Boolean for the ClickHouseBarRow fields), round-trip compatibility is guaranteed.

## Common Pitfalls

### Pitfall 1: Nullable vs Non-Nullable Arrow Fields

**What goes wrong:** ClickHouseBarRow has 16 inter-bar + 22 intra-bar `Option<T>` fields. If the Arrow schema marks these as non-nullable, Parquet write panics on None values.
**Why it happens:** Default Arrow Field construction creates non-nullable fields.
**How to avoid:** Explicitly set `nullable = true` for all `Option<T>` fields in the Arrow schema.
**Warning signs:** Panic in `ArrowWriter::write()` with "column is not nullable" error.

### Pitfall 2: Type Mismatch Between Rust and Python Column Names

**What goes wrong:** Python Charon's `store_bars_batch()` expects lowercase column names matching `MICROSTRUCTURE_COLUMNS`, `INTER_BAR_FEATURE_COLUMNS`, etc. from `constants.py`. If Rust Parquet uses different names, Charon silently drops columns.
**Why it happens:** Rust struct field names may differ from ClickHouse column names.
**How to avoid:** Use `serde(rename)` or explicit column name mapping in the Arrow schema. The `CORE_COLUMNS` constant in `row.rs` already lists all 77 column names in exact ClickHouse order -- use these as the Parquet schema field names.
**Warning signs:** Replayed dead-letter bars missing columns in ClickHouse.

### Pitfall 3: Filename Parsing in Charon

**What goes wrong:** `replay_dead_letters()` parses the filename with `parts = path.stem.split("_")` and expects `parts[0]` = symbol, `parts[1]` = threshold, rest = timestamp. Symbols like `BTCUSDT` are fine, but if the filename format changes, parsing breaks.
**Why it happens:** Filename is the metadata carrier for symbol/threshold.
**How to avoid:** Match exact Python pattern: `{symbol}_{threshold}_{unix_timestamp}.parquet`. The `_kintsugi_cfg().min_dead_letter_parts` (default 3) validates the split count.
**Warning signs:** "skipping malformed dead-letter" log messages from Charon.

### Pitfall 4: Parquet Compression Compatibility

**What goes wrong:** If Rust uses a compression codec that Python's Polars doesn't support, `pl.read_parquet()` fails.
**Why it happens:** Not all Parquet compressors are universally supported.
**How to avoid:** Use ZSTD or Snappy compression (both universally supported by Polars, pyarrow, and the Rust parquet crate). ZSTD is preferred (better ratio).
**Warning signs:** "unsupported codec" error in Python Polars.

### Pitfall 5: Buffer Ownership in do_flush

**What goes wrong:** `do_flush()` currently does `let _ = std::mem::take(buffer)` at line 199, clearing the buffer before the POST. If the POST fails, the buffer contents are already gone and cannot be dead-lettered.
**Why it happens:** Current design didn't need the buffer contents after serialization.
**How to avoid:** Keep ownership of the rows until after the POST result is known. Either: (a) clone the buffer before take, or (b) restructure to take ownership only on success, dead-letter on failure. Option (b) is better -- serialize to JSONEachRow body first, keep the `Vec<ClickHouseBarRow>` alive until result is known.
**Warning signs:** Dead-letter files are empty or have zero rows.

### Pitfall 6: Niffler Replay Race Condition

**What goes wrong:** If the flush thread is replaying a dead-letter file while Python's Charon also attempts replay, both may insert the same bars.
**Why it happens:** No cross-language coordination between Rust Niffler and Python Charon.
**How to avoid:** This is actually safe because: (1) ClickHouse dedup token prevents duplicate inserts, and (2) the replay deletes the file on success, so the second reader will find the file gone. Use `rename` to a `.replaying` extension as an advisory lock, then delete on success.
**Warning signs:** Duplicate bars (but dedup handles this automatically).

## Code Examples

### Parquet Dead-Letter Writer (Rust)

```rust
// Source: arrow-rs parquet crate docs + project ClickHouseBarRow
use arrow_array::{
    ArrayRef, BooleanArray, Float64Array, Int32Array, Int64Array,
    RecordBatch, StringArray, UInt32Array, UInt8Array,
};
use arrow_schema::{DataType, Field, Schema};
use parquet::arrow::ArrowWriter;
use std::fs::File;
use std::sync::Arc;

/// Build the Arrow schema matching Python dead-letter expectations.
/// Column names match CORE_COLUMNS from row.rs exactly.
fn dead_letter_schema() -> Schema {
    Schema::new(vec![
        Field::new("symbol", DataType::Utf8, false),
        Field::new("threshold_decimal_bps", DataType::UInt32, false),
        Field::new("close_time_us", DataType::Int64, false),
        Field::new("open_time_us", DataType::Int64, false),
        Field::new("open", DataType::Float64, false),
        Field::new("high", DataType::Float64, false),
        Field::new("low", DataType::Float64, false),
        Field::new("close", DataType::Float64, false),
        Field::new("volume", DataType::Float64, false),
        Field::new("vwap", DataType::Float64, false),
        // ... all other core fields ...
        // Inter-bar features: nullable
        Field::new("lookback_trade_count", DataType::UInt32, true),
        Field::new("lookback_ofi", DataType::Float64, true),
        // ... etc
    ])
}

/// Write a batch of failed rows to a dead-letter Parquet file.
fn write_dead_letter(rows: &[ClickHouseBarRow]) -> Result<PathBuf, std::io::Error> {
    let dir = PathBuf::from("/tmp/opendeviationbar-dead-letter");
    std::fs::create_dir_all(&dir)?;

    let symbol = &rows[0].symbol;
    let threshold = rows[0].threshold_decimal_bps;
    let ts = std::time::SystemTime::now()
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap()
        .as_secs();
    let path = dir.join(format!("{symbol}_{threshold}_{ts}.parquet"));

    let schema = Arc::new(dead_letter_schema());
    let file = File::create(&path)?;
    let mut writer = ArrowWriter::try_new(file, schema.clone(), None)?;

    // Build columnar arrays from rows
    let batch = rows_to_record_batch(rows, &schema)?;
    writer.write(&batch)?;
    writer.close()?;

    Ok(path)
}
```

### Niffler Replay (Rust, on flush thread)

```rust
/// Scan dead-letter directory and replay files to ClickHouse.
/// Called periodically from flush thread after successful flushes.
fn niffler_replay(
    rt: &tokio::runtime::Runtime,
    http_client: &reqwest::Client,
    config: &ClickHouseWriterConfig,
) -> usize {
    let dir = PathBuf::from("/tmp/opendeviationbar-dead-letter");
    if !dir.exists() { return 0; }

    let mut replayed = 0;
    for entry in std::fs::read_dir(&dir).into_iter().flatten().flatten() {
        let path = entry.path();
        if path.extension().map_or(true, |e| e != "parquet") { continue; }

        // Read Parquet -> Arrow -> JSONEachRow -> POST
        match replay_parquet_file(rt, http_client, config, &path) {
            Ok(n) => {
                std::fs::remove_file(&path).ok();
                replayed += n;
            }
            Err(e) => {
                tracing::warn!(path = %path.display(), error = %e, "niffler replay failed");
                break; // Stop on first failure (CH likely down)
            }
        }
    }
    replayed
}
```

### Cross-Language Schema Mapping

| ClickHouseBarRow Field         | Rust Type           | Arrow DataType       | Polars Type          | Nullable |
| ------------------------------ | ------------------- | -------------------- | -------------------- | -------- |
| symbol                         | String              | Utf8                 | Utf8                 | No       |
| threshold_decimal_bps          | u32                 | UInt32               | UInt32               | No       |
| close_time_us                  | i64                 | Int64                | Int64                | No       |
| open_time_us                   | i64                 | Int64                | Int64                | No       |
| open/high/low/close            | f64                 | Float64              | Float64              | No       |
| volume/buy_volume/sell_volume  | f64                 | Float64              | Float64              | No       |
| individual_trade_count         | u32                 | UInt32               | UInt32               | No       |
| agg_record_count               | u32                 | UInt32               | UInt32               | No       |
| duration_us                    | i64                 | Int64                | Int64                | No       |
| ofi, vwap_close_deviation, ... | f64                 | Float64              | Float64              | No       |
| ouroboros_mode                 | String              | Utf8                 | Utf8                 | No       |
| exchange*session*\*            | u8                  | UInt8                | UInt8                | No       |
| lookback\_\* (16 features)     | Option<f64/u32/i64> | Float64/UInt32/Int64 | Float64/UInt32/Int64 | **Yes**  |
| intra\_\* (22 features)        | Option<f64/u32/i64> | Float64/UInt32/Int64 | Float64/UInt32/Int64 | **Yes**  |
| has_gap                        | bool                | Boolean              | Boolean              | No       |
| gap_trade_count                | i64                 | Int64                | Int64                | No       |
| max_gap_duration_us            | i64                 | Int64                | Int64                | No       |
| is_exchange_gap                | bool                | Boolean              | Boolean              | No       |
| first_agg_trade_id             | i64                 | Int64                | Int64                | No       |
| last_agg_trade_id              | i64                 | Int64                | Int64                | No       |
| is_orphan                      | u8                  | UInt8                | UInt8                | No       |
| cache_key                      | String              | Utf8                 | Utf8                 | No       |
| opendeviationbar_version       | String              | Utf8                 | Utf8                 | No       |
| source_start_ts                | i64                 | Int64                | Int64                | No       |
| source_end_ts                  | i64                 | Int64                | Int64                | No       |

## State of the Art

| Old Approach                                   | Current Approach                  | When Changed | Impact                                |
| ---------------------------------------------- | --------------------------------- | ------------ | ------------------------------------- |
| Python-only dead-letter (polars dict->parquet) | Rust Parquet via arrow-rs         | Phase 35     | Eliminates Python from failure path   |
| Bars dropped on flush failure (Rust side)      | Dead-letter to Parquet on failure | Phase 35     | Zero bar loss guarantee               |
| Python-only Niffler replay                     | Dual Rust+Python replay           | Phase 35     | Faster recovery, no Python dependency |

## Open Questions

1. **Niffler replay frequency**
   - What we know: Python Niffler replays immediately in a daemon thread. Charon runs every 5 minutes as backup.
   - What's unclear: How often should the Rust flush thread attempt replay? Every N flushes? Every T seconds?
   - Recommendation: Check for dead-letter files every 60 seconds (use an `Instant` timer in the flush thread loop), matching the spirit of Charon's periodic approach without being too aggressive.

2. **Should Rust Niffler replay read Parquet back or just POST raw JSON?**
   - What we know: Writing to Parquet is for Python Charon compatibility. The Rust side has the original rows in JSONEachRow form.
   - What's unclear: Should Rust replay read its own Parquet files back, or should it only attempt replay when it still has the rows in memory?
   - Recommendation: Read Parquet back into Arrow RecordBatches, convert to JSONEachRow, POST. This handles both Rust-written and Python-written dead-letter files, and works across restarts.

## Validation Architecture

### Test Framework

| Property           | Value                                                                        |
| ------------------ | ---------------------------------------------------------------------------- |
| Framework          | cargo nextest + pytest                                                       |
| Config file        | `.config/nextest.toml` (Rust), `pytest.ini` (Python)                         |
| Quick run command  | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink` |
| Full suite command | `mise run check-full`                                                        |

### Phase Requirements to Test Map

| Req ID   | Behavior                                    | Test Type   | Automated Command                                                                                       | File Exists?                         |
| -------- | ------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------- | ------------------------------------ |
| RESIL-01 | Dead-letter writes Parquet on flush failure | unit        | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(dead_letter)' -x`  | Wave 0                               |
| RESIL-02 | Parquet schema readable by Python Polars    | integration | `pytest tests/test_dead_letter_roundtrip.py -x`                                                         | Wave 0                               |
| RESIL-03 | Error classification in SinkError variants  | unit        | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(error_class)' -x`  | Partial (existing transient tests)   |
| RESIL-04 | Backpressure dead-letters overflow bars     | unit        | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(backpressure)' -x` | Partial (existing backpressure test) |
| RESIL-05 | Niffler replay of dead-letter files         | unit        | `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(niffler)' -x`      | Wave 0                               |

### Sampling Rate

- **Per task commit:** `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `crates/opendeviationbar-streaming/src/clickhouse_writer/dead_letter.rs` -- new module, covers RESIL-01, RESIL-05
- [ ] `tests/test_dead_letter_roundtrip.py` -- cross-language round-trip test, covers RESIL-02
- [ ] Update existing `test_sink_backpressure` to verify dead-letter write, covers RESIL-04

## Project Constraints (from CLAUDE.md)

- **Python 3.13 ONLY** -- cross-language round-trip tests must use Python 3.13
- **Local-first CI/CD** -- no GitHub Actions for testing, all quality gates via `mise run check-full`
- **Leverage Rust** -- dead-letter serialization happens in Rust, Python handles replay only as backup
- **ClickHouse COMMENT = SSoT** -- not directly relevant but dead-letter schema should document column purposes
- **Aion-only** -- ouroboros_mode always "aion" in dead-letter files

## Sources

### Primary (HIGH confidence)

- `crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs` -- current flush failure handling, line 293-305
- `crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs` -- ClickHouseBarRow struct, 77 fields, CORE_COLUMNS constant
- `crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs` -- backpressure handling, TrySendError::Full path
- `python/opendeviationbar/sidecar/dead_letter.py` -- Python Niffler: filename pattern, dir path, Polars write
- `python/opendeviationbar/kintsugi/dead_letter.py` -- Charon replay: polars.read_parquet, store_bars_batch
- `python/opendeviationbar/clickhouse/bulk_operations.py` -- store_bars_batch column expectations (lines 891-948)
- `python/opendeviationbar/sidecar/clickhouse_write.py` -- \_write_batch_with_retry dead-letter fallback path

### Secondary (MEDIUM confidence)

- parquet crate v58.1.0 (cargo search verified) -- aligned with workspace arrow v58

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - parquet crate v58 aligns with existing arrow workspace deps, verified via cargo search
- Architecture: HIGH - clear patterns from existing Python dead-letter system, well-understood flush thread structure
- Pitfalls: HIGH - all identified from actual code analysis (buffer ownership, nullable fields, filename parsing)

**Research date:** 2026-03-28
**Valid until:** 2026-04-28 (stable infrastructure, slow-moving domain)
