# Python Sidecar Startup Sequence: Safety Analysis

**Date**: 2026-03-21
**Scope**: `_create_engine()` in `python/opendeviationbar/sidecar.py` (lines 911-1079)
**Perspective**: Can the startup sequence produce gaps due to ordering, timing, or missed steps?

---

## Startup Sequence Map

```
_create_engine(config, trace_id, gap_fill_results, valid_pairs)
│
├── 1. Create StreamManager/OdbEngine
├── 2. engine.start_ws()                    ← WS buffering begins
├── 3. _inject_checkpoints()                ← Returns (loaded, loaded_pairs)
│
├── 4. Compute needs_fill = (loaded < n_expected)
│
├── [IF NOT needs_fill — ALL checkpoints loaded]:
│   └── 5a. _seed_trade_ids_from_clickhouse()   ← For gap detector only
│
├── [IF needs_fill — SOME or ALL pairs missing checkpoints]:
│   ├── 5b. _delete_today_bars(valid_pairs)      ← BLANKET delete (no symbol filter!)
│   ├── 6b. _seed_trade_ids_from_clickhouse()    ← Seeds from post-delete CH
│   ├── 7b. _run_midnight_preflight()            ← Overrides seeds with crossover TID
│   ├── 8b. engine.fill_from_rest()              ← REST catch-up using seeds
│   └── 9b. _sync_flush_rest_bars()              ← Drain REST bars to CH
│
├── 10. Build min_tid_guard for unloaded pairs
└── 11. engine.start()                           ← Drain WS buffer, go live
```

Post-`_create_engine`:

```
12. _heal_day_boundary_gaps()   ← Checks CH for Stathera gaps in today's bars
```

---

## Findings

### FINDING-1: Blanket DELETE destroys checkpointed pairs' bars (MEDIUM-HIGH)

**Location**: `_delete_today_bars()` (line 735-740)

When `needs_fill=True` (even one pair missing a checkpoint), `_delete_today_bars` executes:

```sql
DELETE FROM opendeviationbar_cache.open_deviation_bars
WHERE ouroboros_mode = '{ouroboros_mode}'
AND toDate(open_time_ms / 1000) = today()
```

This deletes ALL today's bars for ALL symbols and thresholds — not just the pairs without checkpoints. If 17/18 symbols have checkpoints and 1 doesn't, all 17 symbols' today bars are deleted.

**Impact**: The 17 checkpointed pairs then go through `fill_from_rest()` which rebuilds from REST. This is wasteful but not gap-producing because `fill_from_rest` creates processors from checkpoints (line 431-445 in `live_engine.rs`). However, the window between DELETE and sync_flush completion leaves a period where CH has no today bars — any heartbeat or external query during this window sees a false gap.

**Risk**: Low for actual gaps (fill_from_rest compensates), but the blanket DELETE is unnecessarily aggressive and creates a visible hole in observability.

**Recommendation**: Add symbol/threshold filter to DELETE query:

```sql
AND (symbol, threshold_decimal_bps) IN (...)
```

Limited to only the unloaded pairs.

---

### FINDING-2: `_delete_today_bars` failure is non-fatal — silent gap risk (MEDIUM)

**Location**: Line 745-746

```python
except Exception:
    logger.warning("clean_slate: failed to delete today's bars (non-fatal)", exc_info=True)
```

If delete fails (CH timeout, network issue), the sequence continues with `fill_from_rest()`. The REST bars are then flushed via `_sync_flush_rest_bars()` → `store_bars_replacing()`. Since the old bars weren't deleted, the new bars interleave with old bars from a prior session that have different TID boundaries.

**Impact**: RMT(computed_at) dedup should handle this IF the new bars cover the same TID ranges. But if the prior session's bars had different TID splits (different bar boundaries due to different checkpoint state), you get overlapping bars that RMT won't deduplicate (different `first_agg_trade_id` values = different rows in the ORDER BY).

**Risk**: Medium. Overlapping bars → Stathera anomalies → kintsugi repair cycle. The explicit comment says this scenario should trigger a clean rebuild, but the non-fatal handling undermines it.

**Recommendation**: If `_delete_today_bars` fails and `needs_fill` is True, either:

- Retry the delete with backoff, OR
- Skip `fill_from_rest` entirely and rely on kintsugi (accepting a temporary gap)

---

### FINDING-3: `_seed_trade_ids_from_clickhouse` silently leaves pairs unseeded (MEDIUM)

**Location**: Lines 471-481

```python
except (OSError, RuntimeError):
    logger.debug(  # <-- debug level, not warning
        "trade-ID seed query failed for %s@%d", symbol, threshold
    )
```

Per-pair seed failures are logged at DEBUG level only. If a single pair's query fails (e.g., CH returns an error for one symbol), that pair starts with seed=0.

**Impact with MAX semantics**: A seed of 0 means `fill_from_rest` uses `from_tid = max(checkpoint_tid, 0)`. If the pair has a checkpoint, the checkpoint TID takes over (safe). If NOT, `from_tid = None` and no REST fill occurs (line 482: `if let Some(from_id) = from_tid`). That pair has ZERO today bars, producing a gap.

**Scenario**: New symbol added, no checkpoint, CH seed query fails → `from_tid = None` → no REST fill → gap until first WS trade closes a bar.

**Recommendation**: Elevate per-pair seed failures to WARNING and track unseeded pairs. After seeding, verify all `needs_fill` pairs have a seed > 0, and log a LOUD warning if any don't.

---

### FINDING-4: Preflight failure leaves symbol at CH seed (not crossover TID) (LOW-MEDIUM)

**Location**: `_run_midnight_preflight()` lines 882-889

```python
if crossover_tid is None:
    logger.warning("preflight skipped for %s: could not determine crossover trade ID", symbol)
    continue
```

If Binance REST is down or the midnight query fails, the symbol keeps its CH seed (yesterday's `last_agg_trade_id` from the last bar). This seed may be correct (if yesterday's last bar was the orphan bar), or it may be too low (if yesterday had gaps that kintsugi hasn't repaired yet).

**Impact**: `fill_from_rest` starts from the CH seed instead of the crossover TID. If CH seed < crossover_tid - 1, REST fetches some yesterday trades that produce yesterday bars in today's session → cross-midnight filter discards them → those trades are wasted but no gap. If CH seed > crossover_tid - 1, some today-morning trades are skipped → gap until Puck detects discontinuity.

**Risk**: Low under normal conditions (CH seed is usually close to crossover). Higher risk after manual bar deletions or kintsugi failures.

---

### FINDING-5: `fill_from_rest` error swallowed — falls back to WS-only (MEDIUM)

**Location**: Lines 1028-1030

```python
except (RuntimeError, OSError) as e:
    logger.warning("fill_from_rest failed: %s (falling back to WS-only start)", e)
    rest_bars = 0
```

If `fill_from_rest` completely fails (all REST requests error), the engine starts WS-only from the first live trade. All today's bars from midnight to now are lost (they were already deleted at step 5b).

**Impact**: Gap from midnight (or last CH bar time) until first WS trade arrival. This is the WORST case: bars deleted + no REST fill + WS starts cold.

**Risk**: Medium-high in practice because it requires both conditions: needs_fill=True (common on restarts) + REST total failure (rare but happens during Binance maintenance). The `_heal_day_boundary_gaps` call after `_create_engine` (line 2337) should detect and fill this gap, BUT the engine just failed REST — `fill_gap()` uses the same REST endpoint and will likely fail too.

**Recommendation**: If `fill_from_rest` fails AND `_delete_today_bars` succeeded, send a LOUD alert. The system is now in a known-bad state: today's bars deleted, no replacement available.

---

### FINDING-6: `_sync_flush_rest_bars` failure partially handled (LOW-MEDIUM)

**Location**: Lines 848-857

Individual pair flush failures fall back to `_write_batch_with_retry`. But if the entire sync_flush fails (e.g., all CH connections down), `_create_engine` continues to `engine.start()` without REST bars in CH.

The bars were produced by `fill_from_rest` and are in the Rust ring buffer. After `engine.start()`, the normal polling loop should eventually drain them. But:

- The `_heal_day_boundary_gaps` query runs AFTER `_create_engine` returns (line 2337)
- It queries CH, which doesn't have the REST bars yet → false gaps → spurious `fill_gap` calls

**Risk**: Self-healing but noisy. Kintsugi will eventually clean up.

---

### FINDING-7: Watchdog restart passes `gap_fill_results=None` — stale checkpoints reinjected (LOW)

**Location**: Line 1228

```python
new_engine, _, new_min_tid_guard = _create_engine(
    config, trace_id, None, valid_pairs=valid_pairs,
)
```

On watchdog restart, `gap_fill_results=None` means `_inject_checkpoints` doesn't know that gap-fill already wrote bars. However, `_extract_checkpoints` (line 1212) saved fresh checkpoints from the dying engine. If extraction succeeded, the new checkpoints ARE fresh. If extraction failed (line 1215), the OLD checkpoint files (from initial startup) are reinjected.

**Impact**: Stale checkpoint reinjection → `needs_fill=False` (if most pairs had checkpoints) → no delete, no fill_from_rest → engine resumes from stale checkpoint that may overlap with bars already in CH.

**Risk**: Low because RMT dedup handles most overlaps, and Puck detects the first WS gap.

---

### FINDING-8: `min_tid_guard` queries CH AFTER sync_flush but before WS starts (SAFE)

**Location**: Lines 1046-1074

The guard is built after `fill_from_rest` and `sync_flush`, so it reflects the current CH state. This is correct. No issue found.

---

### FINDING-9: Race window between WS buffer start and `engine.start()` (LOW)

**Location**: Lines 964-1078

WS starts buffering immediately (line 965). The entire sequence (inject checkpoints, delete, seed, preflight, fill_from_rest, sync_flush, build guard) runs before `engine.start()` drains the buffer.

**Answer**: WS buffer is 50,000 trades per symbol (`mpsc::channel::<AggTrade>(50_000)` in `live_engine.rs:324`). For BTCUSDT at peak (~50 trades/sec), this gives ~1000 seconds (16 minutes) before overflow. The startup sequence typically completes in 10-60 seconds (REST fill being the bottleneck). Overflow is unlikely.

**Impact**: With tokio bounded mpsc, when the channel is full the sender's `.send()` awaits (backpressure). This means the WS connection handler blocks, which could cause WS heartbeat timeout and reconnection. But 50K capacity provides ample headroom.

**Risk**: Very low in practice. Only a concern if REST fill takes >15 minutes (extreme network issues).

---

### FINDING-10: `needs_fill` threshold is binary — partial checkpoint scenarios suboptimal (LOW)

**Location**: Lines 991-992

```python
n_expected = len(valid_pairs) if valid_pairs else len(config.symbols) * len(config.thresholds)
needs_fill = hasattr(engine, "fill_from_rest") and loaded < n_expected
```

If 71 out of 72 pairs have checkpoints, `needs_fill=True` triggers blanket DELETE + full REST fill for all pairs. The 71 checkpointed pairs had valid state that is now discarded and rebuilt from REST.

**Impact**: Unnecessary work + larger gap window (REST fill takes longer for 18 symbols than for 1). Not a correctness issue — more of a performance/availability concern.

**Recommendation**: Consider a targeted fill: only delete and fill for unloaded pairs, keep checkpointed pairs running from checkpoints.

---

## Risk Summary

| #   | Finding                                              | Severity    | Gap Risk                | Fix Complexity              |
| --- | ---------------------------------------------------- | ----------- | ----------------------- | --------------------------- |
| 1   | Blanket DELETE destroys checkpointed pairs' bars     | MEDIUM-HIGH | Observability hole      | Low (add WHERE filter)      |
| 2   | Delete failure + fill continues → overlapping bars   | MEDIUM      | Overlaps → kintsugi     | Low (retry or gate)         |
| 3   | Per-pair seed failure at DEBUG level → unseeded pair | MEDIUM      | Cold start gap          | Low (elevate log + verify)  |
| 4   | Preflight failure → CH seed used (may be stale)      | LOW-MEDIUM  | Possible morning gap    | Low (already has fallback)  |
| 5   | fill_from_rest total failure after DELETE → cold WS  | MEDIUM-HIGH | Full today gap          | Medium (LOUD alert + guard) |
| 6   | sync_flush failure → false gaps in heal query        | LOW-MEDIUM  | Noisy but self-healing  | Low                         |
| 7   | Watchdog restart stale checkpoint reinjection        | LOW         | RMT handles overlaps    | Low                         |
| 9   | WS buffer overflow during long startup               | VERY LOW    | Unlikely (50K capacity) | N/A                         |
| 10  | Binary needs_fill → unnecessary blanket rebuild      | LOW         | Performance only        | Medium (targeted fill)      |

## Priority Recommendations

1. **FINDING-5 + FINDING-2 combined**: The worst case is DELETE succeeds + fill_from_rest fails → empty today. Add a pre-condition: if delete succeeded, fill_from_rest failure should be LOUD (Telegram alert) and potentially trigger immediate kintsugi.

2. **FINDING-1**: Scope `_delete_today_bars` to only unloaded pairs. This is a low-risk, high-reward fix.

3. **FINDING-3**: Elevate per-pair seed failures and verify all needs_fill pairs have seeds after the seeding step.
