---
phase: 34-core-clickhouse-writer
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - Cargo.toml
  - crates/opendeviationbar-streaming/Cargo.toml
  - crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs
  - crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs
  - crates/opendeviationbar-streaming/src/lib.rs
autonomous: true
requirements:
  - WRITER-02
  - WRITER-04
  - WRITER-08

must_haves:
  truths:
    - "ClickHouseBarRow can be constructed from any CompletedBar with correct field mapping"
    - "Dedup token from Rust is byte-identical to Python hashlib.md5 for same inputs"
    - "Computed columns (is_liquidation_cascade, computed_at) are never included in the row struct"
    - "FixedPoint prices convert via to_f64(), i128 volumes convert via as f64 / SCALE"
  artifacts:
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs"
      provides: "ClickHouseBarRow struct with #[derive(Row, Serialize)] and from_completed_bar()"
      contains: "pub struct ClickHouseBarRow"
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs"
      provides: "MD5 dedup token generation with Python parity"
      contains: "pub fn compute_dedup_token"
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs"
      provides: "ClickHouseWriterConfig with batch thresholds"
      contains: "pub struct ClickHouseWriterConfig"
    - path: "crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs"
      provides: "X-ClickHouse-Summary JSON parser"
      contains: "pub fn parse_summary"
  key_links:
    - from: "crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs"
      to: "crates/opendeviationbar-core/src/types.rs"
      via: "CompletedBar.bar fields"
      pattern: "bar\\.open\\.to_f64\\(\\)"
    - from: "crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs"
      to: "python/opendeviationbar/clickhouse/bulk_operations.py"
      via: "identical MD5 formula"
      pattern: "md5::compute"
---

<objective>
Create the foundational types for the ClickHouse writer: the ClickHouseBarRow struct that maps all 60+ CompletedBar fields to ClickHouse columns, the MD5 dedup token generator with Python parity, the writer config struct, and the X-ClickHouse-Summary parser.

Purpose: These types are consumed by the flush thread (Plan 02). Defining them first ensures the executor for Plan 02 has concrete contracts to build against.
Output: `clickhouse_writer/` module with row.rs, dedup.rs, config.rs, summary.rs
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/34-core-clickhouse-writer/34-RESEARCH.md

@crates/opendeviationbar-streaming/src/engine/traits.rs
@crates/opendeviationbar-streaming/src/engine/sink_channel.rs
@crates/opendeviationbar-streaming/src/live_engine/types.rs
@crates/opendeviationbar-streaming/Cargo.toml
@crates/opendeviationbar-core/src/types.rs
@python/opendeviationbar/clickhouse/schema.sql
@python/opendeviationbar/clickhouse/bulk_operations.py
@Cargo.toml

<interfaces>
<!-- Key types the executor needs. -->

From crates/opendeviationbar-streaming/src/live_engine/types.rs:

```rust
#[derive(Debug, Clone)]
pub struct CompletedBar {
    pub symbol: Arc<str>,
    pub threshold_decimal_bps: u32,
    pub bar: OpenDeviationBar,
}
```

From crates/opendeviationbar-streaming/src/engine/traits.rs:

```rust
pub trait BarSink: Send {
    fn on_bar(&mut self, bar: &CompletedBar) -> Result<(), SinkError>;
    fn flush(&mut self) -> Result<(), SinkError>;
    fn name(&self) -> &str;
}

#[derive(Debug, thiserror::Error)]
pub enum SinkError {
    #[error("recoverable: {0}")]
    Recoverable(String),
    #[error("unrecoverable: {0}")]
    Unrecoverable(String),
}
```

From python/opendeviationbar/clickhouse/bulk_operations.py (line 557-560):

```python
key_str = f"{symbol}_{threshold_decimal_bps}_{start_ts}_{end_ts}_{ouroboros_mode}"
df["cache_key"] = hashlib.md5(key_str.encode()).hexdigest()
```

ClickHouse schema columns to OMIT from Row struct:

- `is_liquidation_cascade` (DEFAULT expression, computed by CH)
- `computed_at` (DEFAULT now64(3, 'UTC'))
- All plugin columns (DEFAULT NULL, discovered at runtime)
  </interfaces>

</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Cargo deps + feature gate + module scaffold + ClickHouseBarRow + dedup token</name>
  <files>
    Cargo.toml,
    crates/opendeviationbar-streaming/Cargo.toml,
    crates/opendeviationbar-streaming/src/lib.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs
  </files>
  <read_first>
    Cargo.toml,
    crates/opendeviationbar-streaming/Cargo.toml,
    crates/opendeviationbar-streaming/src/lib.rs,
    crates/opendeviationbar-core/src/types.rs,
    crates/opendeviationbar-core/src/arrow_export.rs,
    python/opendeviationbar/clickhouse/schema.sql,
    python/opendeviationbar/clickhouse/bulk_operations.py
  </read_first>
  <behavior>
    - Test: ClickHouseBarRow::from_completed_bar correctly maps open/high/low/close via to_f64()
    - Test: volume, buy_volume, sell_volume use `as f64 / 100_000_000.0` (SCALE)
    - Test: lookback_vwap_raw converts from Option<i64> to Option<f64> via SCALE
    - Test: aggregation_density field maps from bar.aggregation_density_f64
    - Test: ouroboros_mode is hardcoded "aion"
    - Test: exchange_session_* fields are all 0
    - Test: is_orphan is always 0
    - Test: is_liquidation_cascade and computed_at NOT present in struct
    - Test: compute_dedup_token("BTCUSDT", 250, 1700000000000000, 1700000100000000, "aion") matches Python hashlib.md5
  </behavior>
  <action>
1. Add workspace dependencies to root Cargo.toml `[workspace.dependencies]`:
   ```toml
   clickhouse = { version = "0.14", features = ["inserter", "lz4"] }
   md5 = "0.8"
   ```
   NOTE: Do NOT add `rustls-tls-native-roots` to clickhouse features -- the clickhouse crate uses hyper/h2 internally, not reqwest. TLS for HTTP transport is handled separately.

1. Add to `crates/opendeviationbar-streaming/Cargo.toml`:
   Under `[dependencies]`:

   ```toml
   clickhouse = { workspace = true, optional = true }
   md5 = { workspace = true, optional = true }
   serde.workspace = true
   serde_json = { workspace = true, optional = true }
   reqwest = { workspace = true, optional = true, features = ["blocking"] }
   ```

   Under `[features]`:

   ```toml
   clickhouse-sink = ["dep:clickhouse", "dep:md5", "dep:serde_json", "dep:reqwest"]
   ```

   Under `[dev-dependencies]`:

   ```toml
   wiremock = "0.6"
   tokio = { workspace = true, features = ["rt-multi-thread", "macros"] }
   ```

2. Create `crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs`:
   - `#[cfg(feature = "clickhouse-sink")]` gated
   - Re-export: ClickHouseBarRow, compute_dedup_token, ClickHouseWriterConfig, parse_summary
   - Submodules: row, dedup, config, summary

3. Add to `crates/opendeviationbar-streaming/src/lib.rs`:

   ```rust
   #[cfg(feature = "clickhouse-sink")]
   pub mod clickhouse_writer;
   ```

4. Create `row.rs` with `ClickHouseBarRow` struct:
   - Use `#[derive(Debug, Clone, clickhouse::Row, serde::Serialize)]`
   - All fields from schema.sql EXCEPT `is_liquidation_cascade` and `computed_at`
   - Field mapping (critical conversions):
     - Prices (open, high, low, close, vwap): `bar.open.to_f64()`
     - Volumes (volume, buy_volume, sell_volume): `bar.volume as f64 / 100_000_000.0`
     - `aggregation_density`: map from `bar.aggregation_density_f64` (name mismatch!)
     - `lookback_vwap_raw`: `bar.lookback_vwap_raw.map(|v| v as f64 / 100_000_000.0)` (i64 -> f64)
     - `ouroboros_mode`: hardcoded `"aion".to_string()`
     - `exchange_session_*`: hardcoded `0u8`
     - `is_orphan`: hardcoded `0u8`
     - `cache_key`: empty string (computed per-batch in flush thread)
     - `opendeviationbar_version`: `env!("CARGO_PKG_VERSION").to_string()`
     - `source_start_ts` / `source_end_ts`: `0i64` (computed per-batch)
   - Implement `from_completed_bar(bar: &CompletedBar) -> Self`
   - Add `CORE_COLUMNS: &[&str]` constant listing all column names in the struct (used by plugin discovery in Plan 02)

5. Create `dedup.rs`:

   ```rust
   pub fn compute_dedup_token(
       symbol: &str,
       threshold_decimal_bps: u32,
       start_ts: i64,
       end_ts: i64,
       ouroboros_mode: &str,
   ) -> String {
       let key_str = format!("{}_{}_{}_{}_{}",
           symbol, threshold_decimal_bps, start_ts, end_ts, ouroboros_mode);
       let digest = md5::compute(key_str.as_bytes());
       format!("{:x}", digest)
   }
   ```

   Also add `compute_batch_dedup_token(rows: &[ClickHouseBarRow]) -> String` that computes token from the batch's symbol, threshold, min(open_time_us), max(close_time_us), "aion".

6. Write tests:
   - `test_from_completed_bar_prices`: verify open/high/low/close/vwap are correct f64 values
   - `test_from_completed_bar_volumes`: verify volume/buy_volume/sell_volume use SCALE division
   - `test_from_completed_bar_lookback_vwap`: verify Option<i64> -> Option<f64> conversion
   - `test_from_completed_bar_aggregation_density`: verify field name mapping
   - `test_from_completed_bar_hardcoded_fields`: ouroboros_mode="aion", sessions=0, is_orphan=0
   - `test_computed_columns_not_in_struct`: compile-time check that struct has no is_liquidation_cascade field
   - `test_dedup_token_parity_with_python`: compute token for fixed inputs and verify against Python output. Use: `python3 -c "import hashlib; print(hashlib.md5('BTCUSDT_250_1700000000000000_1700000100000000_aion'.encode()).hexdigest())"` to get the expected hash, then hardcode it.
   - `test_batch_dedup_token`: verify batch token uses min/max timestamps
     </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(clickhouse_writer)' 2>&1 | tail -20</automated>
  </verify>
  <acceptance_criteria>
    - grep -q 'pub struct ClickHouseBarRow' crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs
    - grep -q 'pub fn compute_dedup_token' crates/opendeviationbar-streaming/src/clickhouse_writer/dedup.rs
    - grep -q 'clickhouse-sink' crates/opendeviationbar-streaming/Cargo.toml
    - grep -q 'pub mod clickhouse_writer' crates/opendeviationbar-streaming/src/lib.rs
    - grep -q 'aggregation_density' crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs (NOT aggregation_density_f64 as column name)
    - grep -q '"aion"' crates/opendeviationbar-streaming/src/clickhouse_writer/row.rs
    - NO match for 'is_liquidation_cascade' in row.rs
    - NO match for 'computed_at' in row.rs
    - cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink exits 0
  </acceptance_criteria>
  <done>
    ClickHouseBarRow struct compiles with all 60+ fields mapping correctly from CompletedBar.
    Dedup token produces MD5 hex identical to Python hashlib.md5 for fixed test inputs.
    Computed columns (is_liquidation_cascade, computed_at) are absent from the struct.
    All unit tests pass.
  </done>
</task>

<task type="auto">
  <name>Task 2: Config struct + X-ClickHouse-Summary parser</name>
  <files>
    crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs,
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs
  </files>
  <read_first>
    crates/opendeviationbar-streaming/src/clickhouse_writer/mod.rs,
    python/opendeviationbar/clickhouse/schema.sql
  </read_first>
  <action>
1. Create `config.rs` with `ClickHouseWriterConfig`:
   ```rust
   #[derive(Debug, Clone)]
   pub struct ClickHouseWriterConfig {
       /// ClickHouse HTTP URL (e.g., "http://localhost:8123")
       pub url: String,
       /// Database name (default: "opendeviationbar_cache")
       pub database: String,
       /// Table name (default: "open_deviation_bars")
       pub table: String,
       /// Max bars before flush (default: 500)
       pub max_rows: usize,
       /// Max milliseconds before flush (default: 100)
       pub flush_period_ms: u64,
       /// Max bytes before flush (default: 5MB = 5_242_880)
       pub max_bytes: usize,
       /// Bounded channel capacity for FlushCommand (default: 10_000)
       pub channel_capacity: usize,
       /// Max retry attempts for transient errors (default: 5)
       pub max_retries: u32,
   }
   ```
   Implement `Default` with the values above.
   Implement `from_env()` that reads:
   - `OPENDEVIATIONBAR_CH_HOSTS` -> url (prepend "http://" and append ":8123" if just hostname)
   - `OPENDEVIATIONBAR_CH_WRITER_MAX_ROWS` -> max_rows
   - `OPENDEVIATIONBAR_CH_WRITER_FLUSH_MS` -> flush_period_ms
   - `OPENDEVIATIONBAR_CH_WRITER_MAX_BYTES` -> max_bytes
   - Falls back to defaults for any unset var

1. Create `summary.rs` with X-ClickHouse-Summary parsing:

   ```rust
   #[derive(Debug, serde::Deserialize)]
   pub struct ClickHouseSummary {
       #[serde(deserialize_with = "deserialize_string_u64")]
       pub read_rows: u64,
       #[serde(deserialize_with = "deserialize_string_u64")]
       pub written_rows: u64,
       #[serde(deserialize_with = "deserialize_string_u64")]
       pub written_bytes: u64,
       #[serde(deserialize_with = "deserialize_string_u64")]
       pub elapsed_ns: u64,
   }
   ```

   The JSON values are strings (ClickHouse returns `"written_rows":"500"` not `"written_rows":500`), so implement a custom deserializer `deserialize_string_u64` that parses the string to u64.

   Add `pub fn parse_summary(header_value: &str) -> Option<ClickHouseSummary>` that calls `serde_json::from_str`.
   Add `pub fn verify_written_rows(summary: &ClickHouseSummary, expected: u64) -> bool` that returns true if written_rows == expected, logs warning via tracing::warn! if mismatch.

2. Update `mod.rs` to add `pub mod config;` and `pub mod summary;` and re-export the public types.

3. Write tests:
   - `test_config_defaults`: verify default values match spec (500 rows, 100ms, 5MB)
   - `test_parse_summary_valid`: parse `{"read_rows":"0","written_rows":"500","written_bytes":"12345","elapsed_ns":"1000000"}` succeeds
   - `test_parse_summary_invalid`: parse garbage returns None
   - `test_verify_written_rows_match`: returns true when equal
   - `test_verify_written_rows_mismatch`: returns false when not equal
     </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink -E 'test(config) | test(summary)' 2>&1 | tail -20</automated>
  </verify>
  <acceptance_criteria>
    - grep -q 'pub struct ClickHouseWriterConfig' crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs
    - grep -q 'pub struct ClickHouseSummary' crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs
    - grep -q 'pub fn parse_summary' crates/opendeviationbar-streaming/src/clickhouse_writer/summary.rs
    - grep -q 'max_rows: usize' crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs
    - grep -q 'flush_period_ms: u64' crates/opendeviationbar-streaming/src/clickhouse_writer/config.rs
    - cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink exits 0
  </acceptance_criteria>
  <done>
    ClickHouseWriterConfig has all batch thresholds (N=500, T=100ms, bytes=5MB) with env var overrides.
    ClickHouseSummary correctly parses X-ClickHouse-Summary JSON with string-encoded integers.
    verify_written_rows returns false and logs warning on mismatch.
  </done>
</task>

</tasks>

<verification>
```bash
# All clickhouse_writer module tests pass
cargo nextest run -p opendeviationbar-streaming --features clickhouse-sink

# Workspace still compiles without the feature

cargo check -p opendeviationbar-streaming

# No clippy warnings

cargo clippy -p opendeviationbar-streaming --features clickhouse-sink -- -D warnings

```
</verification>

<success_criteria>
1. `clickhouse-sink` feature gate compiles cleanly
2. ClickHouseBarRow maps all 60+ fields from CompletedBar correctly (verified by unit tests)
3. Dedup token is byte-identical to Python for fixed inputs (cross-language parity test)
4. Computed columns (is_liquidation_cascade, computed_at) absent from Row struct
5. Config defaults match spec: 500 rows, 100ms, 5MB
6. Summary parser handles string-encoded integer JSON fields
</success_criteria>

<output>
After completion, create `.planning/phases/34-core-clickhouse-writer/34-01-SUMMARY.md`
</output>
```
