---
phase: 34-core-clickhouse-writer
plan: 03
subsystem: streaming/clickhouse_writer
tags: [rust, clickhouse, reqwest, wiremock, streaming, verification]
dependency_graph:
  requires:
    [
      ClickHouseBarRow,
      compute_dedup_token,
      ClickHouseWriterConfig,
      ClickHouseSummary,
      spawn_flush_thread,
      FlushThreadMetrics,
    ]
  provides:
    [
      raw-reqwest-flush,
      summary-verification-hot-path,
      SUMMARY_HEADER,
      FlushError,
    ]
  affects: [crates/opendeviationbar-streaming]
tech_stack:
  added: [wiremock-0.6]
  patterns: [raw-http-post-with-header-parsing, json-each-row-serialization]
key_files:
  created: []
  modified:
    - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs
    - crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
    - crates/opendeviationbar-streaming/Cargo.toml
    - Cargo.toml
decisions:
  - Raw reqwest POST replaces clickhouse crate client.insert() for header access
  - FlushError enum replaces clickhouse::error::Error for cleaner error classification
  - JSONEachRow format via serde_json::to_string() per row (ClickHouseBarRow already has Serialize)
  - wiremock for integration tests (async mock HTTP server)
patterns_established:
  - "Raw HTTP POST to ClickHouse with query params for INSERT + dedup token"
  - "X-ClickHouse-Summary header parsed on every successful flush for row count verification"
requirements-completed: [WRITER-07]
metrics:
  duration: 756s
  completed: "2026-03-29T09:24:19Z"
  tasks_completed: 1
  tasks_total: 1
  tests_added: 3
  tests_updated: 7
  files_modified: 5
---

# Phase 34 Plan 03: WRITER-07 Gap Closure Summary

Raw reqwest POST replacing clickhouse crate INSERT in do_flush(), enabling X-ClickHouse-Summary header parsing and verify_written_rows() on every production flush -- closing the WRITER-07 dead code gap.

## Performance

- **Duration:** 756s (~13 min)
- **Started:** 2026-03-29T09:11:43Z
- **Completed:** 2026-03-29T09:24:19Z
- **Tasks:** 1
- **Files modified:** 5

## Accomplishments

- Replaced `clickhouse` crate `client.insert()` with raw `reqwest::Client` POST using JSONEachRow format, giving direct access to HTTP response headers
- `parse_summary()` and `verify_written_rows()` are now called in the production `do_flush()` code path (were previously dead code behind `#[allow(unused_imports)]`)
- Added `FlushError` enum for cleaner error classification (HTTP status, network, serialization) replacing `clickhouse::error::Error` string parsing
- Added `SUMMARY_HEADER` constant and 3 wiremock integration tests proving the summary verification hot path works
- All 159 streaming crate tests pass (was 156, +3 new wiremock tests)

## Task Commits

Each task was committed atomically:

1. **Task 1: Replace clickhouse crate INSERT with raw reqwest POST + summary verification** - `64c9fde` (feat)

## Files Created/Modified

- `crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs` - Rewrote do_flush() to use raw reqwest POST with JSONEachRow format, added FlushError enum, removed #[allow(unused_imports)], added wiremock integration tests
- `crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs` - Added SUMMARY_HEADER constant
- `crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs` - Re-exported SUMMARY_HEADER
- `crates/opendeviationbar-streaming/Cargo.toml` - Added wiremock dev-dependency
- `Cargo.toml` - Added wiremock to workspace dependencies

## Decisions Made

- **Raw reqwest over clickhouse crate for INSERT**: The `clickhouse` crate's `Insert<T>::end()` returns `Result<()>` with no access to HTTP response headers. Raw reqwest POST with JSONEachRow format gives direct access to `X-ClickHouse-Summary` while `ClickHouseBarRow` already derives `serde::Serialize`.
- **FlushError enum**: Replaced `clickhouse::error::Error` dependency in error classification with a purpose-built `FlushError` enum (Http/Network/Serialization). Cleaner than string parsing `BadResponse` messages for status codes.
- **Kept clickhouse crate as dependency**: Still needed for `ClickHouseBarRow`'s `#[derive(clickhouse::Row)]` in row.rs and `discover_plugin_columns()` in schema.rs.

## Deviations from Plan

None - plan executed exactly as written.

## Issues Encountered

None.

## Known Stubs

None - all functionality is fully implemented with no placeholder data.

## Next Phase Readiness

- WRITER-07 gap fully closed: summary verification is on the production hot path
- All 3 plans of Phase 34 (foundational types, flush thread + BarSink, gap closure) are complete
- Ready for Phase 35: dead-letter Parquet fallback (Niffler)

---

_Phase: 34-core-clickhouse-writer_
_Completed: 2026-03-29_
