# Rust Ring Buffer Safety Guard Investigation

**Date**: 2026-03-21
**Scope**: Can `ConcurrentRingBuffer` silently lose bars? What guards exist and what is missing?

---

## Ring Buffer Design

- **Location**: `crates/opendeviationbar-streaming/src/ring_buffer.rs`
- **Type**: Fixed-capacity circular buffer, `parking_lot::Mutex`-guarded
- **Default capacity**: 10,000 bars (`OPENDEVIATIONBAR_MAX_PENDING_BARS` env override)
- **Overflow behavior**: Drops OLDEST bar, returns `false`, increments `total_dropped` metric

## Push Site Catalog (8 total, ALL guarded)

Every `bar_buffer.push()` call in `live_engine.rs` checks the return value and logs/metrics on drop.

### fill_from_rest (2 sites)

| Line | Context                          | Guard                                   |
| ---- | -------------------------------- | --------------------------------------- |
| 583  | Midnight orphan during REST fill | `if !push` -> `dropped_bars++`, `warn!` |
| 600  | Normal bar during REST fill      | `if !push` -> `dropped_bars++`, `warn!` |

### symbol_task / process_trade_through_processors (4 sites)

| Line | Context                     | Guard                                                                 |
| ---- | --------------------------- | --------------------------------------------------------------------- |
| 1093 | Midnight orphan (WS trade)  | `if !was_added` -> `dropped_bars++`, `backpressure_events++`, `warn!` |
| 1129 | Normal bar (WS trade)       | `if !was_added` -> `dropped_bars++`, `backpressure_events++`, `warn!` |
| 1256 | Midnight orphan (puck_fill) | `if !push` -> `dropped_bars++`, `backpressure_events++`               |
| 1274 | Normal bar (puck_fill)      | `if !push` -> `dropped_bars++`, `backpressure_events++`               |

**Verdict**: No unguarded push sites remain. The original `let _ = bar_buffer.push()` pattern has been fully replaced.

---

## Bar-Loss Path Analysis: Rust -> Python

### Path 1: Ring buffer overflow (MITIGATED, NOT ELIMINATED)

**Risk**: HIGH during `fill_from_rest` with many symbols.

- `fill_from_rest` spawns up to 4 concurrent symbol tasks, ALL pushing into the SAME 10K ring buffer.
- With 18 symbols x 4 thresholds = 72 (symbol, threshold) pairs, even 4 concurrent fills can produce thousands of bars.
- The 8,233 bar count from discovery context fits: if production exceeds 10K before `drain_bars()` is called, oldest bars are silently overwritten.

**Current mitigation**:

- Logging on every drop (`tracing::warn!`)
- `dropped_bars` metric counter
- Python-side Lethe detection polls `get_metrics()` periodically and sends Telegram alert

**Gap**: Lethe only runs during the WS streaming phase (inside the `while running:` loop on timeout heartbeats). During `fill_from_rest` + `_sync_flush_rest_bars`, there is NO Python-side overflow detection. The bars are produced entirely in Rust, then drained once. If overflow happens during fill, the log warning is the only signal.

### Path 2: drain_bars() bypasses StreamManager continuity tracking (MINOR)

**Risk**: LOW (telemetry gap, not data loss).

- `StreamManager.drain_bars()` (line 374) delegates directly to `engine.drain_bars()` without updating `trade_id_state`.
- `StreamManager.next_bar()` (line 328) tracks Stathera continuity per symbol.
- Bars drained via `drain_bars()` are invisible to the continuity tracker, so the first WS bar after start may appear as a "gap" in SM metrics.

**Impact**: False continuity gap metric, not actual bar loss. The bars themselves are written to ClickHouse correctly via `_sync_flush_rest_bars`.

### Path 3: next_bar() timeout returns None (BY DESIGN)

**Risk**: NONE for bar loss.

- `next_bar(timeout)` returns `None` on timeout. The bar stays in the ring buffer until the next `pop()`.
- Bars are NOT consumed on timeout -- they persist.
- The Python sidecar loop simply retries.

### Path 4: Cross-midnight filter in \_sync_flush_rest_bars (BY DESIGN)

**Risk**: Intentional discard, not silent loss.

- Lines 800-808: Bars spanning midnight are discarded with `logger.warning`.
- These bars violate the day-mode invariant and will be repaired by kintsugi.

### Path 5: valid_pairs filter in_sync_flush_rest_bars (BY DESIGN)

**Risk**: Intentional discard.

- Lines 796-797: Bars for unregistered (symbol, threshold) pairs are silently skipped.
- The `filtered_invalid_pairs` count is logged at INFO level (line 813).

### Path 6: SSE client backpressure (NON-BLOCKING, ISOLATED)

**Risk**: NONE for ring buffer path.

- SSE broadcast uses `queue.put_nowait()` with bounded 256-item queues per client.
- Slow SSE clients lose events, but this does NOT affect ClickHouse writes.

---

## Structural Risk: fill_from_rest Capacity vs Production

The core risk is arithmetic:

| Factor                         | Value                    |
| ------------------------------ | ------------------------ |
| Ring buffer capacity           | 10,000 (default)         |
| Symbols in production          | 18                       |
| Thresholds per symbol          | 4 (250, 500, 750, 1000)  |
| Concurrent fill tasks          | 4 (semaphore-limited)    |
| Bars per symbol-day (250 dbps) | ~2,000-4,000 for BTCUSDT |

**Worst case**: 4 active symbols x 4 thresholds x ~2,000 bars = 32,000 bars pushed before drain.
With 10K capacity, up to 22,000 bars could be dropped.

**Actual case**: The semaphore limits concurrency to 4 symbols at a time, and each symbol processes sequentially across chunks. But bars accumulate across ALL completed symbol tasks before `drain_bars()` is called.

---

## Recommendations

### P0: Increase ring buffer capacity for fill_from_rest

The 10K default was sized for WS streaming (slow trickle). `fill_from_rest` is a burst producer.
Options:

1. **Dedicated capacity override**: `with_bar_channel_capacity(100_000)` before fill_from_rest, then shrink after drain.
2. **Dynamic sizing**: `symbols.len() * thresholds.len() * estimated_bars_per_symbol_day`.
3. **Inline drain**: Drain the ring buffer after each symbol completes fill_from_rest, not after all symbols finish.

### P1: Add drain_bars() continuity tracking

`StreamManager.drain_bars()` should update `trade_id_state` for each drained bar, mirroring what `next_bar()` does. This prevents false gap metrics after the REST-to-WS transition.

### P2: Python-side overflow detection during fill_from_rest

After `drain_bars()`, check `engine.get_metrics()["dropped_bars"]`. If > 0, log LOUD warning + consider Telegram alert. Currently only Lethe detects this, and Lethe runs only during WS phase.

---

## Files Examined

| File                                                      | Lines                                                                                                                           |
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `crates/opendeviationbar-streaming/src/ring_buffer.rs`    | 1-341 (full)                                                                                                                    |
| `crates/opendeviationbar-streaming/src/live_engine.rs`    | 400-620 (fill_from_rest), 850-893 (next_bar/drain_bars), 1065-1300 (symbol_task + puck_fill), 1370-1670 (symbol_task main loop) |
| `crates/opendeviationbar-streaming/src/stream_manager.rs` | 320-380 (next_bar/drain_bars)                                                                                                   |
| `src/stream_manager_bindings.rs`                          | 309-481 (PyO3 next_bar/drain_bars)                                                                                              |
| `python/opendeviationbar/sidecar.py`                      | 750-830 (\_sync_flush_rest_bars), 2486-2500 (main loop), 2635-2653 (Lethe)                                                      |
