---
phase: 34-core-clickhouse-writer
plan: 02
type: execute
wave: 2
depends_on: ["34-01"]
files_modified:
  - crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/schema.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
  - crates/opendeviationbar-streaming/src/engine/mod.rs
autonomous: true
requirements:
  - WRITER-01
  - WRITER-03
  - WRITER-05
  - WRITER-06
  - WRITER-07
  - WRITER-09

must_haves:
  truths:
    - "on_bar() returns immediately without blocking on HTTP -- bars are sent to flush thread via bounded channel"
    - "Flush thread POSTs bars to ClickHouse when batch hits N=500 bars, T=100ms timeout, or max_bytes=5MB"
    - "HTTP 429/503 retries with exponential backoff; HTTP 400/401 fails immediately"
    - "X-ClickHouse-Summary written_rows is verified after every INSERT and mismatch logs a warning"
    - "Plugin columns are discovered via system.columns at startup, not hardcoded"
    - "When the bounded channel is full, on_bar() returns SinkError::Recoverable (no block)"
  artifacts:
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs"
      provides: "ClickHouseWriterSink implementing BarSink trait"
      contains: "impl BarSink for ClickHouseWriterSink"
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs"
      provides: "Dedicated std::thread flush loop with batch buffering and HTTP POST"
      contains: "pub fn spawn_flush_thread"
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/schema.rs"
      provides: "system.columns plugin column discovery"
      contains: "pub async fn discover_plugin_columns"
  key_links:
    - from: "crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs"
      to: "crates/opendeviationbar-streaming/src/engine/traits.rs"
      via: "BarSink trait implementation"
      pattern: "impl BarSink for ClickHouseWriterSink"
    - from: "crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs"
      to: "crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs"
      via: "serializes ClickHouseBarRow to ClickHouse"
      pattern: "ClickHouseBarRow"
    - from: "crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs"
      to: "crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs"
      via: "computes per-batch dedup token"
      pattern: "compute_batch_dedup_token"
---

<objective>
Implement the ClickHouseWriterSink that implements BarSink, with a dedicated flush thread for non-blocking writes, batch buffering with triple-trigger flush (N bars, T ms, max bytes), HTTP POST with retry/backoff, X-ClickHouse-Summary verification, and plugin column discovery.

Purpose: This is the core write path -- the sink that will eventually run alongside ChannelSink in the LiveBarEngine fan-out (Phase 36).
Output: Working ClickHouseWriterSink that can POST bars to a real or mock ClickHouse.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/34-core-clickhouse-writer/34-RESEARCH.md
@.planning/phases/34-core-clickhouse-writer/34-01-SUMMARY.md

@crates/opendeviationbar-streaming/src/engine/traits.rs
@crates/opendeviationbar-streaming/src/engine/sink_channel.rs
@crates/opendeviationbar-streaming/src/engine/mod.rs
@crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
@crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs
@crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs
@crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs
@crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs
@crates/opendeviationbar-streaming/Cargo.toml

<interfaces>
<!-- From Plan 01 artifacts -->

From clickhouse_writer/row.rs:

```rust
#[derive(Debug, Clone, clickhouse::Row, serde::Serialize)]
pub struct ClickHouseBarRow { /* 60+ fields */ }

impl ClickHouseBarRow {
    pub fn from_completed_bar(bar: &CompletedBar) -> Self;
}

pub const CORE_COLUMNS: &[&str] = &["symbol", "threshold_decimal_bps", ...];
```

From clickhouse_writer/dedup.rs:

```rust
pub fn compute_dedup_token(symbol: &str, threshold_decimal_bps: u32, start_ts: i64, end_ts: i64, ouroboros_mode: &str) -> String;
pub fn compute_batch_dedup_token(rows: &[ClickHouseBarRow]) -> String;
```

From clickhouse_writer/config.rs:

```rust
pub struct ClickHouseWriterConfig {
    pub url: String,
    pub database: String,
    pub table: String,
    pub max_rows: usize,
    pub flush_period_ms: u64,
    pub max_bytes: usize,
    pub channel_capacity: usize,
    pub max_retries: u32,
}
```

From clickhouse_writer/summary.rs:

```rust
pub fn parse_summary(header_value: &str) -> Option<ClickHouseSummary>;
pub fn verify_written_rows(summary: &ClickHouseSummary, expected: u64) -> bool;
```

From engine/traits.rs:

```rust
pub trait BarSink: Send {
    fn on_bar(&mut self, bar: &CompletedBar) -> Result<(), SinkError>;
    fn flush(&mut self) -> Result<(), SinkError>;
    fn name(&self) -> &str;
}
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Flush thread with batch buffering, HTTP POST, retry, and summary verification</name>
  <files>
    crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs,
    crates/opendeviationbar-streaming/src/engine/traits.rs,
    crates/opendeviationbar-streaming/Cargo.toml
  </read_first>
  <behavior>
    - Test: flush thread drains buffer when row count >= max_rows (500)
    - Test: flush thread drains buffer on timeout (flush_period_ms)
    - Test: flush thread drains buffer on Shutdown command even if below threshold
    - Test: HTTP POST to mock server succeeds and returns X-ClickHouse-Summary
    - Test: HTTP 429 response triggers retry with backoff (verified by request count on mock)
    - Test: HTTP 503 response triggers retry
    - Test: HTTP 400 response does NOT retry (fails immediately)
    - Test: X-ClickHouse-Summary with mismatched written_rows logs warning
    - Test: dedup token is set as INSERT query parameter
  </behavior>
  <action>
1. Create `flush_thread.rs` with the following structure:

**FlushCommand enum:**

```rust
pub(crate) enum FlushCommand {
    Bar(ClickHouseBarRow),
    Flush,
    Shutdown,
}
```

**FlushThreadMetrics struct (atomic counters):**

```rust
pub struct FlushThreadMetrics {
    pub bars_flushed: AtomicU64,
    pub bars_failed: AtomicU64,
    pub flush_count: AtomicU64,
    pub last_flush_latency_us: AtomicU64,
    pub dead_letter_count: AtomicU64,
}
```

**spawn_flush_thread function:**

```rust
pub fn spawn_flush_thread(
    rx: Receiver<FlushCommand>,
    config: ClickHouseWriterConfig,
    metrics: Arc<FlushThreadMetrics>,
) -> std::thread::JoinHandle<()>
```

Inside the thread:

- Create a `tokio::runtime::Builder::new_current_thread().enable_all().build()` runtime (the `clickhouse` crate is async, so we need a runtime inside the std::thread)
- Create a `clickhouse::Client::default().with_url(&config.url).with_database(&config.database).with_option("wait_end_of_query", "1")`
- Maintain `buffer: Vec<ClickHouseBarRow>` with capacity `config.max_rows`
- Loop on `rx.recv_timeout(Duration::from_millis(config.flush_period_ms))`:
  - `Ok(FlushCommand::Bar(row))`: push to buffer, check if `buffer.len() >= config.max_rows`, if so flush
  - `Ok(FlushCommand::Flush)` or `Err(Timeout)`: if buffer non-empty, flush
  - `Ok(FlushCommand::Shutdown)`: flush remaining, break
  - `Err(Disconnected)`: flush remaining, break

    **do_flush function (private):**

  ```rust
  fn do_flush(
      rt: &tokio::runtime::Runtime,
      client: &clickhouse::Client,
      buffer: &mut Vec<ClickHouseBarRow>,
      config: &ClickHouseWriterConfig,
      metrics: &FlushThreadMetrics,
  )
  ```

- Compute dedup token via `compute_batch_dedup_token(&buffer)`
- Set `source_start_ts` = min(open_time_us) and `source_end_ts` = max(close_time_us) on all rows
- Set `cache_key` = dedup token on all rows
- Use `backon::ExponentialBuilder::default().with_max_times(config.max_retries as usize)` for retry
- The retry closure:
  - `rt.block_on(async { client.insert(&config.table)?.with_option("insert_deduplication_token", &dedup_token).write(rows).end().await })`
  - On success: parse `X-ClickHouse-Summary` from response (if available via the client), call `verify_written_rows`
  - On error: classify -- if status 429 or 503, return retryable error; if 400 or 401, return permanent error
- After flush: update metrics (bars_flushed, flush_count, last_flush_latency_us)
- On all-retries-exhausted: increment bars_failed, log error (dead-letter is Phase 35)

  **NOTE on clickhouse crate insert API:**
  The `clickhouse` crate's `client.insert()` returns an `Insert<T>` that you `.write(&row)` per row then `.end()`. The pattern:

  ```rust
  let mut insert = client.insert(&table_name)?;
  insert = insert.with_option("insert_deduplication_token", &token);
  for row in buffer.drain(..) {
      insert.write(&row).await?;
  }
  insert.end().await?;
  ```

  If the `with_option` is not available on `Insert`, fall back to setting it on the client before the insert call:

  ```rust
  let insert_client = client.clone().with_option("insert_deduplication_token", &token);
  let mut insert = insert_client.insert(&table_name)?;
  ```

  **Retry strategy with backon:**

  ```rust
  use backon::{ExponentialBuilder, Retryable};
  let result = (|| async {
      // insert logic
  })
  .retry(ExponentialBuilder::default()
      .with_min_delay(Duration::from_secs(1))
      .with_max_delay(Duration::from_secs(16))
      .with_max_times(config.max_retries as usize))
  .when(|e| is_transient_error(e))
  .await;
  ```

  **is_transient_error classification:**

- HTTP 429 (rate limit) -> transient (retry)
- HTTP 503 (service unavailable) -> transient (retry)
- Network timeout/connection error -> transient (retry)
- HTTP 400 (bad request) -> fatal (no retry)
- HTTP 401 (unauthorized) -> fatal (no retry)
- Any other 4xx -> fatal (no retry)
- HTTP 500, 502 -> transient (retry)

1. Update `mod.rs` to add `pub mod flush_thread;` and re-export `FlushThreadMetrics`, `FlushCommand`.

2. Write tests using `wiremock`:
   - `test_flush_on_row_count`: send max_rows bars, verify mock server received 1 POST
   - `test_flush_on_timeout`: send 1 bar, wait flush_period_ms, verify mock server received POST
   - `test_flush_on_shutdown`: send 3 bars, send Shutdown, verify mock server received POST with 3 rows
   - `test_retry_on_503`: mock returns 503 twice then 200, verify 3 total requests
   - `test_no_retry_on_400`: mock returns 400, verify only 1 request
   - `test_dedup_token_in_query`: verify the INSERT request URL/body contains the dedup token
   - `test_summary_mismatch_warning`: mock returns summary with written_rows=0 for 5 bars, verify warning logged
     </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(flush)' 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - grep -q 'pub fn spawn_flush_thread' crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q 'enum FlushCommand' crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q 'pub struct FlushThreadMetrics' crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q 'insert_deduplication_token' crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q 'backon' crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - grep -q 'wait_end_of_query' crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs
    - cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(flush)' exits 0
  </acceptance_criteria>
  <done>
    Flush thread buffers rows and POSTs to ClickHouse when N/T/bytes threshold fires.
    HTTP 429/503 retry with exponential backoff via backon crate.
    HTTP 400/401 fail immediately without retry.
    Per-batch dedup token sent as insert_deduplication_token.
    X-ClickHouse-Summary parsed and written_rows verified.
    All wiremock integration tests pass.
  </done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: ClickHouseWriterSink BarSink implementation + plugin column discovery</name>
  <files>
    crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/schema.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs,
    crates/opendeviationbar-streaming/src/engine/mod.rs
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/flush_thread.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs,
    crates/opendeviationbar-streaming/src/engine/traits.rs,
    crates/opendeviationbar-streaming/src/engine/mod.rs,
    crates/opendeviationbar-streaming/src/engine/sink_channel.rs
  </read_first>
  <behavior>
    - Test: ClickHouseWriterSink::new() spawns a flush thread
    - Test: on_bar() sends to channel and returns Ok immediately
    - Test: on_bar() returns SinkError::Recoverable when channel is full (no blocking)
    - Test: on_bar() returns SinkError::Unrecoverable when flush thread has died
    - Test: flush() sends FlushCommand::Flush and returns Ok
    - Test: name() returns "clickhouse"
    - Test: drop() sends Shutdown and joins the flush thread
    - Test: discover_plugin_columns returns column names not in CORE_COLUMNS
    - Test: discover_plugin_columns returns empty vec when no plugin columns exist
  </behavior>
  <action>
1. Create `sink.rs` with `ClickHouseWriterSink`:
   ```rust
   pub struct ClickHouseWriterSink {
       tx: std::sync::mpsc::SyncSender<FlushCommand>,
       flush_thread: Option<std::thread::JoinHandle<()>>,
       metrics: Arc<FlushThreadMetrics>,
       bars_sent: AtomicU64,
       bars_dropped: AtomicU64,
   }
   ```

**Constructor:**

```rust
impl ClickHouseWriterSink {
    pub fn new(config: ClickHouseWriterConfig) -> Self {
        let (tx, rx) = std::sync::mpsc::sync_channel(config.channel_capacity);
        let metrics = Arc::new(FlushThreadMetrics::default());
        let flush_thread = spawn_flush_thread(rx, config, Arc::clone(&metrics));
        Self {
            tx,
            flush_thread: Some(flush_thread),
            metrics,
            bars_sent: AtomicU64::new(0),
            bars_dropped: AtomicU64::new(0),
        }
    }

    pub fn metrics(&self) -> &FlushThreadMetrics { &self.metrics }
    pub fn bars_sent(&self) -> u64 { self.bars_sent.load(Ordering::Relaxed) }
    pub fn bars_dropped(&self) -> u64 { self.bars_dropped.load(Ordering::Relaxed) }
}
```

**BarSink implementation:**

```rust
impl BarSink for ClickHouseWriterSink {
    fn on_bar(&mut self, bar: &CompletedBar) -> Result<(), SinkError> {
        let row = ClickHouseBarRow::from_completed_bar(bar);
        match self.tx.try_send(FlushCommand::Bar(row)) {
            Ok(()) => {
                self.bars_sent.fetch_add(1, Ordering::Relaxed);
                Ok(())
            }
            Err(std::sync::mpsc::TrySendError::Full(_)) => {
                self.bars_dropped.fetch_add(1, Ordering::Relaxed);
                tracing::warn!("clickhouse flush thread backpressure - bar dropped");
                Err(SinkError::Recoverable("clickhouse flush thread backpressure".into()))
            }
            Err(std::sync::mpsc::TrySendError::Disconnected(_)) => {
                Err(SinkError::Unrecoverable("clickhouse flush thread died".into()))
            }
        }
    }

    fn flush(&mut self) -> Result<(), SinkError> {
        self.tx.send(FlushCommand::Flush)
            .map_err(|_| SinkError::Unrecoverable("clickhouse flush thread died".into()))
    }

    fn name(&self) -> &str { "clickhouse" }
}
```

**Drop implementation** for graceful shutdown:

```rust
impl Drop for ClickHouseWriterSink {
    fn drop(&mut self) {
        let _ = self.tx.send(FlushCommand::Shutdown);
        if let Some(handle) = self.flush_thread.take() {
            let _ = handle.join();
        }
    }
}
```

1. Create `schema.rs` with plugin column discovery:

   ```rust
   use clickhouse::Client;

   /// Discover plugin-added columns by querying system.columns
   /// and filtering out CORE_COLUMNS.
   pub async fn discover_plugin_columns(
       client: &Client,
       database: &str,
       table: &str,
   ) -> Result<Vec<String>, clickhouse::error::Error> {
       #[derive(clickhouse::Row, serde::Deserialize)]
       struct ColumnRow {
           name: String,
       }

       let query = format!(
           "SELECT name FROM system.columns WHERE database = '{}' AND table = '{}' ORDER BY position",
           database, table
       );
       let rows: Vec<ColumnRow> = client.query(&query).fetch_all().await?;

       let core_set: std::collections::HashSet<&str> = super::row::CORE_COLUMNS.iter().copied().collect();
       // Also exclude computed columns
       let exclude: std::collections::HashSet<&str> = ["is_liquidation_cascade", "computed_at"].iter().copied().collect();

       Ok(rows.into_iter()
           .map(|r| r.name)
           .filter(|name| !core_set.contains(name.as_str()) && !exclude.contains(name.as_str()))
           .collect())
   }
   ```

2. Update `mod.rs` to add:
   - `pub mod sink;`
   - `pub mod schema;`
   - Re-export `ClickHouseWriterSink`

3. Update `crates/opendeviationbar-streaming/src/engine/mod.rs` to conditionally re-export:

   ```rust
   #[cfg(feature = "clickhouse-sink")]
   pub use super::clickhouse_writer::sink::ClickHouseWriterSink;
   ```

4. Write tests:
   - `test_sink_name`: verify name() returns "clickhouse"
   - `test_sink_on_bar_sends_to_channel`: create sink with mock config (url pointing to wiremock), send a bar, verify bars_sent increments
   - `test_sink_backpressure`: create sink with channel_capacity=1, fill it, verify next on_bar returns Recoverable and bars_dropped increments
   - `test_sink_flush_thread_died`: create sink, drop the receiver manually (tricky -- may need to test via shutdown), verify Unrecoverable
   - `test_sink_graceful_shutdown`: create sink, send bars, drop sink, verify flush thread joins cleanly
   - `test_discover_plugin_columns_filters_core`: mock system.columns query returning core + 2 plugin columns, verify only plugin columns returned
   - `test_discover_plugin_columns_empty`: mock system.columns returning only core columns, verify empty vec returned
     </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(sink) | test(schema) | test(plugin_discovery)' 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - grep -q 'impl BarSink for ClickHouseWriterSink' crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
    - grep -q 'pub fn new' crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
    - grep -q 'try_send' crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
    - grep -q 'pub async fn discover_plugin_columns' crates/opendeviationbar-streaming/src/clickhouse_writer/schema.rs
    - grep -q 'system.columns' crates/opendeviationbar-streaming/src/clickhouse_writer/schema.rs
    - grep -q 'CORE_COLUMNS' crates/opendeviationbar-streaming/src/clickhouse_writer/schema.rs
    - grep -q '"clickhouse"' crates/opendeviationbar-streaming/src/clickhouse_writer/sink.rs
    - cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink exits 0
    - cargo check -p opendeviationbar-streaming (without feature still compiles)
  </acceptance_criteria>
  <done>
    ClickHouseWriterSink implements BarSink with non-blocking on_bar() via SyncSender.
    Channel full returns SinkError::Recoverable, channel closed returns Unrecoverable.
    Drop sends Shutdown and joins flush thread for graceful cleanup.
    Plugin columns discovered from system.columns minus CORE_COLUMNS.
    All tests pass, workspace compiles with and without clickhouse-sink feature.
  </done>
</task>

</tasks>

<verification>
```bash
# All clickhouse_writer tests pass (including wiremock integration tests)
cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink

# Workspace still compiles without the feature

cargo check -p opendeviationbar-streaming

# Full workspace check (no regressions)

cargo nextest run --workspace --exclude opendeviationbar-py

# Clippy clean

cargo clippy -p opendeviationbar-streaming --features clickhouse-sink -- -D warnings

```
</verification>

<success_criteria>
1. ClickHouseWriterSink implements BarSink trait with non-blocking on_bar()
2. Flush thread POSTs batches to ClickHouse at N/T/bytes thresholds
3. Retry on HTTP 429/503 with exponential backoff, fail immediately on 400/401
4. Per-batch dedup token via insert_deduplication_token setting
5. X-ClickHouse-Summary parsed and written_rows verified after each INSERT
6. Plugin columns discovered from system.columns at startup
7. Graceful shutdown via Drop: Shutdown command + thread join
8. All integration tests pass with wiremock
</success_criteria>

<output>
After completion, create `.planning/phases/34-core-clickhouse-writer/34-02-SUMMARY.md`
</output>
```
