---
phase: 34-core-clickhouse-writer
plan: 03
type: execute
wave: 1
depends_on: ["34-02"]
files_modified:
  - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs
  - crates/opendeviationbar-streaming/Cargo.toml
autonomous: true
gap_closure: true
requirements: [WRITER-07]

must_haves:
  truths:
    - "do_flush() calls verify_written_rows() after every successful INSERT and logs tracing::warn on mismatch"
    - "X-ClickHouse-Summary header is parsed from the raw HTTP response in the production code path"
    - "All 156+ existing tests still pass with the new code path"
  artifacts:
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs"
      provides: "Raw reqwest POST with X-ClickHouse-Summary parsing in do_flush()"
      contains: "verify_written_rows"
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs"
      provides: "SUMMARY_HEADER constant and parse helper"
      contains: "X-ClickHouse-Summary"
  key_links:
    - from: "flush_thread.rs do_flush()"
      to: "summary.rs verify_written_rows()"
      via: "called after successful reqwest POST"
      pattern: "verify_written_rows"
---

<objective>
Wire X-ClickHouse-Summary verification into the production do_flush() code path.

Purpose: Close WRITER-07 gap -- parse_summary() and verify_written_rows() exist and are tested but are dead code in production because the `clickhouse` crate's `Insert<T>::end()` returns `()`, not HTTP response headers.

Output: do_flush() uses raw `reqwest` POST with JSONEachRow format to INSERT bars, parses X-ClickHouse-Summary from response headers, and calls verify_written_rows(). The `clickhouse` crate `client.insert()` path is removed (it provides no value over raw reqwest and blocks summary verification).
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/34-core-clickhouse-writer/34-01-SUMMARY.md
@.planning/phases/34-core-clickhouse-writer/34-02-SUMMARY.md

<interfaces>
<!-- Key types and contracts the executor needs. Extracted from codebase. -->

From crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs:

```rust
pub struct ClickHouseSummary {
    pub read_rows: u64,
    pub written_rows: u64,
    pub written_bytes: u64,
    pub elapsed_ns: u64,
}
pub fn parse_summary(header_value: &str) -> Option<ClickHouseSummary>;
pub fn verify_written_rows(summary: &ClickHouseSummary, expected: u64) -> bool;
```

From crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs:

```rust
#[derive(Debug, Clone, clickhouse::Row, serde::Serialize)]
pub struct ClickHouseBarRow { /* 78 fields */ }
```

From crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs:

```rust
pub struct ClickHouseWriterConfig {
    pub url: String,
    pub database: String,
    pub table: String,
    pub max_rows: usize,
    pub flush_period_ms: u64,
    pub max_bytes: usize,
    pub channel_capacity: usize,
    pub max_retries: u32,
}
```

From crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs:

```rust
pub fn spawn_flush_thread(rx, config, metrics) -> JoinHandle<()>;
fn do_flush(rt, client, buffer, config, metrics);
fn is_transient_error(err: &clickhouse::error::Error) -> bool;
```

Workspace Cargo.toml reqwest entry:

```toml
reqwest = { version = "0.12", default-features = false, features = ["stream", "rustls-tls", "json"] }
```

Note: `blocking` feature NOT enabled. Flush thread already has a tokio runtime via `rt.block_on()`, so use async reqwest inside that.
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Replace clickhouse crate INSERT with raw reqwest POST + summary verification</name>
  <files>
    crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs
    crates/opendeviationbar-streaming/Cargo.toml
  </files>
  <read_first>
    - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs (full file — understand do_flush, spawn_flush_thread, is_transient_error, all tests)
    - crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs (full file — parse_summary, verify_written_rows, ClickHouseSummary)
    - crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs (full file — ClickHouseBarRow struct, CORE_COLUMNS)
    - crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs (full file — ClickHouseWriterConfig)
    - crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs (compute_batch_dedup_token signature)
    - crates/opendeviationbar-streaming/Cargo.toml (current deps and features)
    - Cargo.toml (workspace deps — reqwest features)
  </read_first>
  <behavior>
    - Test 1: do_flush() with a mock HTTP server returning 200 + X-ClickHouse-Summary header with written_rows matching batch size -> flush succeeds, bars_flushed incremented, no warning logged
    - Test 2: do_flush() with mock returning 200 + X-ClickHouse-Summary with written_rows < batch size (silent dedup) -> flush succeeds but tracing::warn emitted with mismatch details
    - Test 3: do_flush() with mock returning 200 but NO X-ClickHouse-Summary header -> flush succeeds, tracing::warn about missing summary header
    - Test 4: All existing flush_thread tests still pass (shutdown drains, row_count trigger, timeout trigger, explicit flush, channel disconnect, dedup token set, summary mismatch detection, error classification)
  </behavior>
  <action>
    **Approach: Replace the `clickhouse` crate `client.insert()` call in `do_flush()` with a raw `reqwest` POST using JSONEachRow format.**

    The `clickhouse` crate's `Insert<T>::end()` returns `Result<()>` — no access to HTTP response headers. Since `ClickHouseBarRow` already derives `serde::Serialize`, we can serialize to JSON trivially. The flush thread already has a tokio runtime (`rt.block_on()`), so use async `reqwest::Client`.

    **Step 1: Update summary.rs**

    Add a public constant for the header name:
    ```rust
    pub const SUMMARY_HEADER: &str = "x-clickhouse-summary";
    ```

    **Step 2: Rewrite do_flush() in flush_thread.rs**

    Remove the `clickhouse::Client` from `spawn_flush_thread()`. Replace with `reqwest::Client` (async, used inside `rt.block_on()`).

    The new `do_flush()` should:

    1. Compute dedup token and set metadata (existing code, unchanged)
    2. Build INSERT SQL: `INSERT INTO {database}.{table} FORMAT JSONEachRow`
    3. Serialize each `ClickHouseBarRow` to JSON via `serde_json::to_string()`, join with newlines
    4. POST via `reqwest::Client` to `{config.url}/?database={config.database}&query={INSERT_SQL}&wait_end_of_query=1&insert_deduplication_token={dedup_token}`
    5. On success (HTTP 2xx):
       - Get `X-ClickHouse-Summary` header from response
       - If present: call `parse_summary()` then `verify_written_rows(summary, batch_size)`
       - If missing: `tracing::warn!("X-ClickHouse-Summary header missing from response")`
    6. On failure: classify as transient/fatal using HTTP status code directly (no `clickhouse::error::Error` — just check status code)

    **Step 3: Update error classification**

    Replace `is_transient_error(err: &clickhouse::error::Error)` with `is_transient_status(status: u16) -> bool` that classifies 429/500/502/503 as transient. Keep the existing function for backward compat in tests if needed, or update all call sites.

    Actually, since the retry uses `backon`, the retry closure now returns `Result<reqwest::Response, FlushError>` where `FlushError` wraps status code or network error. Define:
    ```rust
    #[derive(Debug)]
    enum FlushError {
        Http { status: u16, body: String },
        Network(reqwest::Error),
    }
    ```
    Then `is_transient_flush_error(err: &FlushError) -> bool` classifies transient cases.

    **Step 4: Remove `#[allow(unused_imports)]`**

    Remove the `#[allow(unused_imports)]` on line 28 and the comment on lines 25-27. `parse_summary` and `verify_written_rows` are now used in production.

    **Step 5: Remove `clickhouse` crate dependency from flush_thread.rs**

    The `clickhouse` crate is still needed for `ClickHouseBarRow`'s `#[derive(clickhouse::Row)]` in row.rs and for `discover_plugin_columns()` in schema.rs. But flush_thread.rs no longer imports `clickhouse::Client` or `clickhouse::error::Error`.

    **Step 6: Add wiremock integration tests**

    Add `wiremock` to dev-dependencies if not present. Write tests using `wiremock::MockServer` that:
    - Mock the ClickHouse HTTP endpoint
    - Return appropriate `X-ClickHouse-Summary` headers
    - Verify the flush thread calls verify_written_rows

    The test should use `spawn_flush_thread()` with the mock server URL, send bars, trigger flush, and check metrics.

    **IMPORTANT**: Keep all existing tests passing. The `test_is_transient_error_*` tests need updating since the error type changes from `clickhouse::error::Error` to `FlushError`. Update them to test `is_transient_flush_error()` instead.

    **IMPORTANT**: The `test_flush_on_shutdown_drains_buffer` and similar tests use `url: "http://127.0.0.1:1"` (unreachable). These should still work — reqwest will return a network error, classified as transient, retried, then failed. Update assertions if error classification changes.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink 2>&1 | tail -20</automated>
  </verify>
  <acceptance_criteria>
    - `#[allow(unused_imports)]` on parse_summary/verify_written_rows is REMOVED from flush_thread.rs
    - `do_flush()` contains a call to `parse_summary()` on the response header value
    - `do_flush()` contains a call to `verify_written_rows()` with the parsed summary and batch_size
    - `tracing::warn!` emitted when summary header is missing or written_rows mismatches
    - All existing tests pass (156+)
    - New test proves summary verification is called on the production hot path with a mock server
    - `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink` exits 0
  </acceptance_criteria>
  <done>
    do_flush() calls verify_written_rows() after every successful INSERT via raw reqwest POST. Silent dedup drops now produce tracing::warn alerts. All tests pass including new integration test with wiremock mock server returning X-ClickHouse-Summary headers.
  </done>
</task>

</tasks>

<verification>
1. `cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink` — all tests pass
2. `grep -n "verify_written_rows" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs` — shows call in do_flush(), NOT just imports
3. `grep -n "allow(unused_imports)" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs` — returns EMPTY (removed)
4. `grep -n "parse_summary" crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs` — shows call in do_flush()
5. `cargo nextest run --workspace --exclude opendeviationbar-py` — full workspace compiles and tests pass
</verification>

<success_criteria>

- WRITER-07 gap closed: verify_written_rows() is called in production do_flush() code path, not just tests
- X-ClickHouse-Summary header is parsed from actual HTTP response (not from clickhouse crate abstraction)
- Mismatch between written_rows and expected_rows produces tracing::warn
- Missing summary header produces tracing::warn
- All 156+ existing tests pass
- New integration test with wiremock proves the summary verification hot path
  </success_criteria>

<output>
After completion, create `.planning/phases/34-core-clickhouse-writer/34-03-SUMMARY.md`
</output>
