# FFI Data Integrity Findings

**Date**: 2026-03-21
**Scope**: PyO3 FFI boundary in `src/stream_manager_bindings.rs`, `src/streaming_bindings.rs`, `src/helpers.rs`
**Perspective**: Can data be corrupted or lost when crossing the Rust-Python boundary?

---

## TGI-1: FFI Boundary Method Audit

### Methods Crossing the Boundary

| Method                  | Direction      | Data Type                              | Validated? |
| ----------------------- | -------------- | -------------------------------------- | ---------- |
| `drain_bars()`          | Rust -> Python | `Vec<CompletedBar>` -> list of dicts   | Partial    |
| `next_bar()`            | Rust -> Python | `Option<CompletedBar>` -> dict or None | Partial    |
| `seed_trade_id()`       | Python -> Rust | i64                                    | **NO**     |
| `set_checkpoint()`      | Python -> Rust | PyDict -> Checkpoint                   | Yes        |
| `fill_from_rest()`      | Rust -> Python | u64 (bar count)                        | Ambiguous  |
| `collect_checkpoints()` | Rust -> Python | `HashMap<String, Checkpoint>` -> dict  | Yes        |
| `get_forming_bars()`    | Rust -> Python | `Vec<FormingBar>` -> list of dicts     | Partial    |
| `fill_gap()`            | Python -> Rust | i64, i64 -> `GapFillResult`            | **NO**     |

---

## Finding 1: `seed_trade_id()` Accepts Negative Values (MEDIUM)

**File**: `src/stream_manager_bindings.rs:171-186`

```rust
pub fn seed_trade_id(&mut self, symbol: &str, threshold: u32, last_agg_trade_id: i64) -> PyResult<()> {
    // No validation on last_agg_trade_id
    manager.seed_trade_id(symbol, threshold, last_agg_trade_id);
}
```

**Rust side** (`stream_manager.rs:182-187`): Uses `if last_agg_trade_id > *entry` — a negative value would be silently ignored (entry defaults to 0). This is **accidentally safe** but not intentionally validated.

**Risk**: LOW in practice. A negative seed would be ignored because 0 > negative. But the lack of explicit validation means the contract is implicit, not enforced. If the default ever changes from 0, negative seeds could corrupt state.

**Recommendation**: Add `if last_agg_trade_id < 0 { return Err(...) }` at the FFI boundary.

---

## Finding 2: `fill_from_rest()` Return Value is PRODUCED Count, Not PUSHED Count (HIGH)

**File**: `crates/opendeviationbar-streaming/src/live_engine.rs:479-604`

The return value `symbol_bars` increments on EVERY bar produced (line 587, 604), regardless of whether the ring buffer push succeeded:

```rust
if !bar_buffer.push(completed) {
    metrics.dropped_bars.fetch_add(1, Ordering::Relaxed);
    tracing::warn!("fill_from_rest: ring buffer full, bar dropped");
}
symbol_bars += 1;  // Counted EVEN IF DROPPED
```

This means `fill_from_rest()` returns the number of bars **produced**, NOT the number successfully pushed to the buffer. If the ring buffer overflows, the Python side sees a count that includes dropped bars.

**Impact**: The Python sidecar uses this count for logging/diagnostics. If it trusts this count as "bars available to drain", it will have fewer bars than expected. The `dropped_bars` metric is tracked in Rust but NOT returned to Python through this method.

**Recommendation**: Either return a struct `{produced: u64, dropped: u64}` or count only successfully pushed bars.

---

## Finding 3: `drain_bars()` / `next_bar()` Bar-to-Dict Conversion is Lossless for Numeric Types (OK)

**File**: `src/helpers.rs:95-259`

The `opendeviationbar_to_dict()` conversion:

- **Timestamps**: `bar.open_time / 1000` (us -> ms). Integer division, no precision loss for ms-aligned values. Binance trades are ms-precision, so the `* 1000` on ingest and `/ 1000` on export is a clean round-trip.
- **Prices (FixedPoint)**: `bar.open.to_f64()` converts i64 scaled by 10^8 to f64. For crypto prices (max ~100,000 with 8 decimals), this fits well within f64's 53-bit mantissa (max exact integer ~9e15, vs max scaled value ~10^13). **No precision loss.**
- **Volumes (i128)**: `bar.volume as f64 * volume_scale_inv` where volume is i128 scaled by 10^8. For volumes up to ~10^10 (typical crypto), scaled value is ~10^18. f64 mantissa is 53 bits (~9e15). **Precision loss possible for very large volumes** (>90M units scaled by 10^8 = 9e15). This would affect only extreme edge cases.
- **Trade IDs (i64)**: Passed directly as i64 to Python. PyO3 maps i64 to Python int. **No precision loss.**
- **Microstructure features (f64)**: Passed as-is. **No conversion loss.**

---

## Finding 4: Checkpoint Round-Trip Has FixedPoint -> f64 -> FixedPoint Precision Risk (LOW)

**File**: `src/helpers.rs:262-483`

Checkpoint export (`checkpoint_to_dict`): `bar.open.to_f64()` converts FixedPoint(i64) to f64.
Checkpoint import (`dict_to_checkpoint`): `f64_to_fixed_point(open)` converts back via `(value * 100_000_000.0).round() as i64`.

**Round-trip analysis**: FixedPoint stores i64 with 8 decimal places (scale 10^8). The max safe exact integer in f64 is 2^53 = 9,007,199,254,740,992. So FixedPoint values up to ~90,071,992.54740992 round-trip exactly. For crypto prices (BTC at ~100,000 = FixedPoint(10,000,000,000,000)), the scaled value is 10^13, well within f64 range. **Safe for all realistic crypto prices.**

For the checkpoint's volume fields (i128 -> f64 -> i128): Same analysis applies. `(volume * SCALE).round() as i128`. Risk only for volumes exceeding ~90M units, which is unrealistic for a single incomplete bar.

---

## Finding 5: `drain_bars()` Cannot Lose Bars in Conversion (OK)

**File**: `src/stream_manager_bindings.rs:461-481`

```rust
let bars = manager.drain_bars();  // Takes ownership of Vec
let list = pyo3::types::PyList::empty(py);
for bar in &bars {
    let dict = helpers::opendeviationbar_to_dict(py, &bar.bar)?;  // ? propagates errors
    // ...
    list.append(dict)?;
}
```

The `?` operator means if ANY bar fails to convert, the entire call returns an error to Python. No bar is silently skipped. The drain operation under `ConcurrentRingBuffer` holds the mutex for the entire drain, so no concurrent pop can steal items.

**However**: If conversion fails mid-way, bars already drained from the ring buffer but not yet appended to the Python list are lost. The ring buffer's `drain_all()` empties the buffer before conversion starts. This is an edge case that would only trigger on a PyO3 allocation failure.

---

## Finding 6: `set_checkpoint()` Has Strong Validation (OK)

**File**: `src/helpers.rs:354-483`

`dict_to_checkpoint()` validates:

- Required fields: `symbol`, `threshold_decimal_bps`, `last_timestamp_us`, `price_hash` — all raise `PyKeyError` if missing
- Threshold range: 1-100,000 dbps — raises `PyValueError` if out of range
- Type extraction: PyO3's `.extract()` raises `TypeError` on wrong type
- Optional fields default gracefully: `anomaly_summary` defaults, `defer_open` defaults to false, `last_completed_bar_tid` defaults to None

**No gaps found** in checkpoint validation.

---

## Finding 7: `fill_gap()` Accepts Negative Trade IDs Without Validation (LOW)

**File**: `src/stream_manager_bindings.rs:383-411`

```rust
pub fn fill_gap(&self, py: Python, symbol: &str, from_tid: i64, to_tid: i64) -> PyResult<...> {
    // No validation that from_tid > 0 or to_tid > from_tid
    runtime.block_on(manager.fill_gap(&symbol_upper, from_tid, to_tid))
}
```

No boundary validation. Negative TIDs or `from_tid >= to_tid` would be passed to the Rust fill logic, which would likely fetch 0 trades and return a benign result, but the contract is implicit.

---

## Finding 8: Ring Buffer Overflow During `fill_from_rest` is Silent to Python (HIGH)

**File**: `crates/opendeviationbar-streaming/src/live_engine.rs:583-603`

Default ring buffer capacity is 10,000 bars (from `OPENDEVIATIONBAR_MAX_PENDING_BARS` env var or default). During `fill_from_rest`, with 18 symbols x 9 thresholds processing potentially thousands of bars each, the buffer can overflow.

When it overflows:

1. Oldest bars are dropped (ring buffer wraps)
2. `metrics.dropped_bars` increments (Rust-side only)
3. A WARN log is emitted
4. The return count still includes dropped bars (Finding 2)
5. Python `drain_bars()` will return fewer bars than `fill_from_rest()` reported

**The Python sidecar calls `_sync_flush_rest_bars()` which calls `drain_bars()` after `fill_from_rest()`.** The sidecar does log the drain count, and recent commit `33009a9` added overflow logging. But there's no mechanism to detect the discrepancy programmatically.

---

## Summary

| #   | Finding                                             | Severity | Data Loss Risk                |
| --- | --------------------------------------------------- | -------- | ----------------------------- |
| 1   | `seed_trade_id` no negative validation              | MEDIUM   | None (accidentally safe)      |
| 2   | `fill_from_rest` returns produced, not pushed count | HIGH     | Misleading count              |
| 3   | Numeric conversion in bar-to-dict                   | OK       | None for realistic values     |
| 4   | Checkpoint FixedPoint round-trip                    | LOW      | None for realistic prices     |
| 5   | `drain_bars` conversion atomicity                   | OK       | Edge case on alloc failure    |
| 6   | `set_checkpoint` validation                         | OK       | Well validated                |
| 7   | `fill_gap` no TID validation                        | LOW      | Benign (0 trades fetched)     |
| 8   | Ring buffer overflow silent to Python               | HIGH     | Bars dropped during REST fill |

**Critical path for data loss**: Finding 2 + Finding 8 combine. During a sidecar restart with large REST backfill, `fill_from_rest` can produce more bars than the 10K ring buffer holds. Dropped bars are counted in the return value but absent from `drain_bars()`. The sidecar would write fewer bars to ClickHouse than produced, creating Stathera gaps that require kintsugi repair.

**Mitigation already in place**: Commit `33009a9` added overflow logging to `fill_from_rest` ring buffer pushes. The `dropped_bars` metric exists in Rust. Kintsugi will repair any resulting gaps. But the gap between "bars produced" and "bars available" is not surfaced to Python code.
