---
phase: 14-rust-core-gap-fields-conditional-reset
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - crates/opendeviationbar-core/src/types.rs
  - crates/opendeviationbar-core/src/processor.rs
  - crates/opendeviationbar-core/src/arrow_export.rs
  - src/helpers.rs
autonomous: true
requirements: [CORE-02, CORE-03]

must_haves:
  truths:
    - "A bar constructed from trades with a TID gap has has_gap=true, gap_trade_count>0, max_gap_duration_us>0"
    - "A bar constructed from continuous trades has has_gap=false, gap_trade_count=0, max_gap_duration_us=0"
    - "is_exchange_gap defaults to false and is never set by Rust processor"
    - "Arrow export includes 4 new gap fields in the schema and data"
    - "PyO3 dict export includes 4 new gap fields"
  artifacts:
    - path: "crates/opendeviationbar-core/src/types.rs"
      provides: "Gap fields on OpenDeviationBar struct"
      contains: "has_gap"
    - path: "crates/opendeviationbar-core/src/processor.rs"
      provides: "Gap detection logic in update_with_trade path"
      contains: "gap_trade_count"
    - path: "crates/opendeviationbar-core/src/arrow_export.rs"
      provides: "Arrow schema and builder for gap fields"
      contains: "has_gap"
    - path: "src/helpers.rs"
      provides: "PyO3 dict export for gap fields"
      contains: "has_gap"
  key_links:
    - from: "crates/opendeviationbar-core/src/processor.rs"
      to: "crates/opendeviationbar-core/src/types.rs"
      via: "update_with_trade sets gap fields on OpenDeviationBar"
      pattern: "has_gap.*true"
    - from: "crates/opendeviationbar-core/src/arrow_export.rs"
      to: "crates/opendeviationbar-core/src/types.rs"
      via: "Arrow schema includes gap fields"
      pattern: "has_gap"
---

<objective>
Add gap awareness fields to the OpenDeviationBar struct and wire gap detection into the bar construction path (both batch and streaming). Export gap fields through Arrow and PyO3 dict paths.

Purpose: Bars need to carry metadata about trade-ID gaps so downstream healers (Puck, Kintsugi) know which bars were built from incomplete data.
Output: OpenDeviationBar with 4 new fields, gap detection in update_with_trade, Arrow + dict export updated.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/14-rust-core-gap-fields-conditional-reset/14-CONTEXT.md
@crates/opendeviationbar-core/src/types.rs
@crates/opendeviationbar-core/src/processor.rs
@crates/opendeviationbar-core/src/arrow_export.rs
@src/helpers.rs

<interfaces>
<!-- Key types and contracts the executor needs -->

From crates/opendeviationbar-core/src/types.rs (OpenDeviationBar struct, line ~34-329):

- Struct uses `#[derive(Debug, Clone, Default, Serialize, Deserialize)]`
- Fields use `#[serde(default)]` for backward compat
- Tier organization: OHLCV Core, Volume, Trade Tracking, Price Context, Microstructure, Inter-Bar, Intra-Bar
- `update_with_trade(&mut self, trade: &AggTrade)` at line ~477 updates price/volume/TID tracking
- `new(trade: &AggTrade) -> Self` at line ~335 initializes all fields from first trade
- `compute_microstructure_features(&mut self)` at line ~537 runs at bar finalization

From crates/opendeviationbar-core/src/processor.rs:

- `process_single_trade(&mut self, trade: &AggTrade)` — streaming path, line ~436
- `process_agg_trade_records_with_options(...)` — batch path, line ~649
- Both paths call `bar_state.bar.update_with_trade(trade)` on subsequent trades
- `self.last_trade_id: Option<i64>` tracks last processed agg_trade_id

From crates/opendeviationbar-core/src/arrow_export.rs:

- `opendeviationbar_schema() -> Schema` at line ~390 defines Arrow field list
- `opendeviationbar_vec_to_record_batch(bars: &[OpenDeviationBar])` at line ~450 builds RecordBatch
- Uses BooleanBuilder, Int64Builder, Float64Builder, UInt32Builder patterns

From src/helpers.rs:

- `opendeviationbar_to_dict(py, bar: &OpenDeviationBar) -> PyResult<PyObject>` at line ~95
- Uses `ikey(py, "field_name")` for interned string keys
- Pattern: `dict.set_item(ikey(py, "field"), bar.field)?;`
  </interfaces>

</context>

<tasks>

<task type="auto">
  <name>Task 1: Add gap fields to OpenDeviationBar struct + gap detection in update_with_trade</name>
  <read_first>
    - crates/opendeviationbar-core/src/types.rs (full file — struct, new(), update_with_trade(), Default derive)
    - crates/opendeviationbar-core/src/processor.rs (lines 431-530 for process_single_trade, lines 649-780 for batch path)
  </read_first>
  <files>crates/opendeviationbar-core/src/types.rs, crates/opendeviationbar-core/src/processor.rs</files>
  <action>
    **In types.rs — Add 4 fields to OpenDeviationBar struct** (after Tier 3 Trade Tracking, before Tier 4 Price Context):

    Add a new section comment `// === TIER 3b: GAP AWARENESS (v7.0 Aion) ===` with these fields:
    ```rust
    /// Whether this bar contains a trade-ID gap (incoming_tid > last_tid + 1)
    /// Transient flag — all healers fire immediately to repair gaps
    #[serde(default)]
    pub has_gap: bool,

    /// Count of missing trades across all gaps within this bar
    /// Computed as sum of (incoming_tid - last_tid - 1) for each gap
    #[serde(default)]
    pub gap_trade_count: i64,

    /// Maximum gap duration in microseconds across all gaps within this bar
    /// Estimated from timestamp jump: incoming_timestamp - last_timestamp for the widest gap
    #[serde(default)]
    pub max_gap_duration_us: i64,

    /// Whether the gap is confirmed as an exchange-side gap (REST backfill found no trades)
    /// Set by Python layer AFTER repair attempt fails — NOT set by Rust processor
    #[serde(default)]
    pub is_exchange_gap: bool,
    ```

    The `#[derive(Default)]` on the struct handles initialization — `bool` defaults to `false`, `i64` defaults to `0`. No changes needed to `OpenDeviationBar::new()` since the gap fields should be zero/false for the opening trade.

    **In types.rs — Add gap tracking to update_with_trade():**

    In `update_with_trade(&mut self, trade: &AggTrade)`, AFTER the existing `last_agg_trade_id` update (line ~497) and BEFORE the volume accumulation section, add gap detection:
    ```rust
    // Gap detection: check if incoming trade has a TID discontinuity
    // last_agg_trade_id was already updated above via monotonic guard,
    // so compare against the PREVIOUS last_agg_trade_id value
    // We need the OLD value before the monotonic update, so restructure:
    ```

    IMPORTANT: The current code updates `self.last_agg_trade_id` with a monotonic guard BEFORE we can detect the gap. Restructure the agg_trade_id update section as follows:

    Replace the existing block:
    ```rust
    if trade.agg_trade_id > self.last_agg_trade_id {
        self.last_agg_trade_id = trade.agg_trade_id;
    }
    ```

    With:
    ```rust
    if trade.agg_trade_id > self.last_agg_trade_id {
        // Gap detection: if incoming TID is not consecutive, mark gap
        let expected_next = self.last_agg_trade_id + 1;
        if trade.agg_trade_id > expected_next && self.last_agg_trade_id > 0 {
            self.has_gap = true;
            let this_gap_count = trade.agg_trade_id - expected_next;
            self.gap_trade_count += this_gap_count;
            // Track max gap duration from timestamp jump
            let this_gap_duration = trade.timestamp - self.close_time;
            if this_gap_duration > self.max_gap_duration_us {
                self.max_gap_duration_us = this_gap_duration;
            }
        }
        self.last_agg_trade_id = trade.agg_trade_id;
    }
    ```

    Note: `self.close_time` is the timestamp of the previous trade (updated just above in the same function), so the duration calculation is correct — it uses the time difference between the last trade before the gap and the first trade after.

    **No changes to processor.rs** — the gap detection lives entirely in `update_with_trade()` which is called by both `process_single_trade` and `process_agg_trade_records` paths. The batch path's loop calls `bar_state.bar.update_with_trade(agg_record)` for both breach and non-breach cases.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo check -p opendeviationbar-core 2>&1 | tail -5</automated>
  </verify>
  <acceptance_criteria>
    - `grep -n "has_gap" crates/opendeviationbar-core/src/types.rs` returns field definition AND gap detection logic
    - `grep -n "gap_trade_count" crates/opendeviationbar-core/src/types.rs` returns field AND accumulation logic
    - `grep -n "max_gap_duration_us" crates/opendeviationbar-core/src/types.rs` returns field AND tracking logic
    - `grep -n "is_exchange_gap" crates/opendeviationbar-core/src/types.rs` returns field definition only (no detection logic)
    - `cargo check -p opendeviationbar-core` succeeds with no errors
  </acceptance_criteria>
  <done>OpenDeviationBar struct has 4 gap awareness fields. Gap detection fires in update_with_trade when TID gap detected. is_exchange_gap defaults to false with no Rust-side setter.</done>
</task>

<task type="auto">
  <name>Task 2: Wire gap fields into Arrow export and PyO3 dict export</name>
  <read_first>
    - crates/opendeviationbar-core/src/arrow_export.rs (full file — schema, builders, RecordBatch construction)
    - src/helpers.rs (full file — opendeviationbar_to_dict, dict_to_opendeviationbar_full)
    - src/arrow_bindings.rs (dict_to_opendeviationbar_full if it exists there)
  </read_first>
  <files>crates/opendeviationbar-core/src/arrow_export.rs, src/helpers.rs</files>
  <action>
    **In arrow_export.rs — Update opendeviationbar_schema():**

    Add 4 fields AFTER the microstructure features section (after `turnover_imbalance`), with a comment:
    ```rust
    // Gap awareness (v7.0 Aion)
    Field::new("has_gap", DataType::Boolean, false),
    Field::new("gap_trade_count", DataType::Int64, false),
    Field::new("max_gap_duration_us", DataType::Int64, false),
    Field::new("is_exchange_gap", DataType::Boolean, false),
    ```

    **In arrow_export.rs — Update opendeviationbar_vec_to_record_batch():**

    1. Add builders after `turnover_imbalance` builder:
    ```rust
    let mut has_gap = BooleanBuilder::with_capacity(n);
    let mut gap_trade_count = Int64Builder::with_capacity(n);
    let mut max_gap_duration_us = Int64Builder::with_capacity(n);
    let mut is_exchange_gap = BooleanBuilder::with_capacity(n);
    ```

    2. In the `for bar in bars` loop, after `turnover_imbalance.append_value(...)`:
    ```rust
    has_gap.append_value(bar.has_gap);
    gap_trade_count.append_value(bar.gap_trade_count);
    max_gap_duration_us.append_value(bar.max_gap_duration_us);
    is_exchange_gap.append_value(bar.is_exchange_gap);
    ```

    3. In the `RecordBatch::try_new(schema, vec![...])` call, add after `turnover_imbalance.finish()`:
    ```rust
    Arc::new(has_gap.finish()),
    Arc::new(gap_trade_count.finish()),
    Arc::new(max_gap_duration_us.finish()),
    Arc::new(is_exchange_gap.finish()),
    ```

    **In helpers.rs — Update opendeviationbar_to_dict():**

    Add after the Microstructure Features section (after `turnover_imbalance` set_item), before Inter-Bar Core Features:
    ```rust
    // Gap Awareness (v7.0 Aion)
    dict.set_item(ikey(py, "has_gap"), bar.has_gap)?;
    dict.set_item(ikey(py, "gap_trade_count"), bar.gap_trade_count)?;
    dict.set_item(ikey(py, "max_gap_duration_us"), bar.max_gap_duration_us)?;
    dict.set_item(ikey(py, "is_exchange_gap"), bar.is_exchange_gap)?;
    ```

    **In arrow_bindings.rs — Update dict_to_opendeviationbar_full() if it exists:**

    Search for `dict_to_opendeviationbar_full` in `src/arrow_bindings.rs`. If it reconstructs an OpenDeviationBar from a Python dict (reverse of opendeviationbar_to_dict), add extraction for the 4 new fields using the existing `get_*_optional` helper pattern:
    ```rust
    has_gap: get_bool_optional(dict, "has_gap", false)?,
    gap_trade_count: get_i64_optional(dict, "gap_trade_count", 0)?,
    max_gap_duration_us: get_i64_optional(dict, "max_gap_duration_us", 0)?,
    is_exchange_gap: get_bool_optional(dict, "is_exchange_gap", false)?,
    ```
    If `get_bool_optional` doesn't exist, create it following the pattern of `get_u32_optional`.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && cargo check -p opendeviationbar-core --features arrow-export && cargo check -p opendeviationbar-py 2>&1 | tail -5</automated>
  </verify>
  <acceptance_criteria>
    - `grep -n "has_gap" crates/opendeviationbar-core/src/arrow_export.rs` returns schema field AND builder AND append_value
    - `grep -n "has_gap" src/helpers.rs` returns dict.set_item line
    - `cargo check -p opendeviationbar-core --features arrow-export` succeeds
    - `cargo check -p opendeviationbar-py` succeeds (full PyO3 build check)
  </acceptance_criteria>
  <done>Gap fields exported through Arrow RecordBatch (schema + data) and PyO3 dict. Both export paths include has_gap, gap_trade_count, max_gap_duration_us, is_exchange_gap.</done>
</task>

</tasks>

<verification>
1. `cargo check -p opendeviationbar-core` — struct compiles with gap fields
2. `cargo check -p opendeviationbar-core --features arrow-export` — Arrow export compiles with gap fields
3. `cargo check -p opendeviationbar-py` — full PyO3 build compiles
4. `cargo nextest run -p opendeviationbar-core` — existing tests pass (gap fields default to false/0, backward compatible)
</verification>

<success_criteria>

1. OpenDeviationBar struct has has_gap, gap_trade_count, max_gap_duration_us, is_exchange_gap fields
2. update_with_trade() detects TID gaps and sets has_gap=true with counts and duration
3. Arrow schema includes 4 new columns, RecordBatch builder populates them
4. PyO3 dict export includes 4 new keys
5. All existing tests pass (backward compatible — gap fields default to false/0)
   </success_criteria>

<output>
After completion, create `.planning/phases/14-rust-core-gap-fields-conditional-reset/14-01-SUMMARY.md`
</output>
