# Guard Investigation Synthesis

**Date**: 2026-03-21
**Agents**: 8 (rust-ringbuf, rust-processor, py-syncflush, py-startup, ch-write, deploy-safety, heartbeat-gaps, ffi-integrity)
**Raw findings**: 48 | **After dedup**: 23 unique findings

---

## Deduplication Map

Findings discovered by multiple agents were merged into single entries:

| Merged Finding                                             | Discovered By                                                                 |
| ---------------------------------------------------------- | ----------------------------------------------------------------------------- |
| Ring buffer overflow during fill_from_rest                 | rust-ringbuf (Path 1), ffi-integrity (F2+F8), py-startup (F9)                 |
| `_delete_today_bars` failure swallowed                     | py-syncflush (F2 context), py-startup (F2), ch-write (F3), deploy-safety (F4) |
| DELETE-then-INSERT non-atomicity                           | py-syncflush (F2), ch-write (F2 ghost bar), deploy-safety (F5 straddle)       |
| `fill_from_rest` failure after delete = gap                | py-startup (F5), deploy-safety (TGI-2), heartbeat-gaps (BS2)                  |
| Seed TID inflation / ordering sensitivity                  | rust-processor (F3), py-startup (F3), heartbeat-gaps (BS5)                    |
| Heartbeat suppresses today's overlaps via kintsugi_healing | heartbeat-gaps (BS3), deploy-safety (F4 context)                              |

---

## Prioritized Findings

### P0: Can Cause Data Loss in Production TODAY

| ID       | Finding                                                                                                                                                                    | Layer           | File:Line                                                  | Proposed Fix                                                                                                                                                                           |
| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **P0-1** | Ring buffer overflow during `fill_from_rest` drops bars silently — 10K capacity vs 32K worst-case burst. Python sees inflated count (produced, not pushed).                | Rust Core / FFI | `live_engine.rs:583-604`, `stream_manager_bindings.rs:461` | Inline drain after each symbol completes fill_from_rest; OR return `{produced, dropped}` struct; OR increase capacity to 100K during REST fill                                         |
| **P0-2** | `_do_replace()` DELETE-then-INSERT: sync DELETE succeeds, INSERT fails = bars permanently deleted. Fallback also fails on same guard = dead-letter but original bars gone. | Python Layer    | `maintenance.py:395-414`, `sidecar.py:848-857`             | Reverse to INSERT-first, then targeted DELETE of stale rows (aligns with #293 direction)                                                                                               |
| **P0-3** | Heartbeat suppresses today's overlap/gap alerts when `kintsugi_healing=True`, but kintsugi CANNOT heal today (#280 today-guard). 495-overlap incident proof.               | Monitoring      | `heartbeat/checks/stathera.py:99`                          | Split health logic by writer ownership: if anomalies are in today's bars, `kintsugi_healing` must NOT suppress. Add velocity threshold (`abs(delta) > 10` per 5-min cycle = CRITICAL). |
| **P0-4** | `_delete_today_bars` failure silently swallowed → stale bars + inflated seed TID → overlapping bars and/or skipped trades.                                                 | Python Layer    | `sidecar.py:745-746`                                       | Return failure signal; if delete fails AND `needs_fill=True`, abort startup or skip CH seed step.                                                                                      |

### P1: Can Cause Data Loss Under Specific Conditions

| ID       | Finding                                                                                                                                                                                 | Layer        | File:Line                                    | Proposed Fix                                                                           |
| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | -------------------------------------------- | -------------------------------------------------------------------------------------- |
| **P1-1** | `fill_from_rest` total failure AFTER `_delete_today_bars` succeeded = empty today. Kintsugi can't repair (writer ownership). Gap persists 0-24h.                                        | Python Layer | `sidecar.py:1028-1030`                       | Send LOUD Telegram alert. Consider immediate kintsugi override for this specific case. |
| **P1-2** | `_delete_ghost_bar()` fully fire-and-forget (no `mutations_sync`, no wait). Ghost bars persist indefinitely alongside corrected bars — RMT cannot dedup different `first_agg_trade_id`. | Python Layer | `backfill.py:1428-1456`                      | Add `SETTINGS mutations_sync=1` to the DELETE statement.                               |
| **P1-3** | Blanket `_delete_today_bars` destroys ALL pairs' bars when only 1 pair lacks checkpoint. 71/72 checkpointed pairs lose bars unnecessarily.                                              | Python Layer | `sidecar.py:735-740`                         | Scope DELETE to unloaded pairs only: `AND (symbol, threshold_decimal_bps) IN (...)`    |
| **P1-4** | Per-pair CH seed failure logged at DEBUG level only → pair starts unseeded → 0 REST bars → gap.                                                                                         | Python Layer | `sidecar.py:471-481`                         | Elevate to WARNING. After seeding, verify all `needs_fill` pairs have seed > 0.        |
| **P1-5** | Deploy: `.venv-old` destroyed immediately in swap command — no rollback path if new version crashes.                                                                                    | Deploy/Ops   | `.claude/skills/odb-ship/SKILL.md:447`       | Remove trailing `&& rm -rf .venv-old`. Keep until next deploy.                         |
| **P1-6** | Deploy: SSH drop between monit unmonitor and re-monitor = services down indefinitely with no auto-recovery.                                                                             | Deploy/Ops   | `.claude/skills/odb-ship/SKILL.md` (Phase 4) | Add deploy sentinel file + monit timeout check. Or re-monitor immediately after stop.  |
| **P1-7** | No backward trade-ID guard in `process_single_trade()`. Out-of-order trades corrupt `last_trade_id`, `committed_floors`.                                                                | Rust Core    | `processor.rs:442`                           | Add `debug_assert!` or warn on backward trade ID in single-trade path.                 |

### P2: Operational Risk (Cascade, Blind Spots, Dead Code)

| ID       | Finding                                                                                                                                                                    | Layer        | File:Line                      |
| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ------------------------------ |
| **P2-1** | Heartbeat has no `sidecar_metrics` check — `dropped_bars`, `backpressure_events`, per-symbol trade freshness, WS reconnection count all invisible to independent observer. | Monitoring   | `heartbeat/checks/` (absent)   |
| **P2-2** | `drain_bars()` bypasses StreamManager continuity tracking → false gap metric after REST-to-WS transition.                                                                  | Rust Core    | `stream_manager.rs:374`        |
| **P2-3** | No Lethe (overflow detection) during `fill_from_rest` + `_sync_flush_rest_bars` — only runs in WS streaming loop.                                                          | Python Layer | `sidecar.py:2635-2653`         |
| **P2-4** | `mutations_healthy` exists as heartbeat result but is NOT in CRITICAL_KEYS gate set. Pending mutations during writes = undetected.                                         | Monitoring   | `heartbeat/checks/stathera.py` |
| **P2-5** | Deploy mutation kill loop has no timeout — infinite loop if mutations regenerate.                                                                                          | Deploy/Ops   | SKILL.md Phase 4.1b            |
| **P2-6** | OPTIMIZE FINAL in deploy does not resolve straddle bars (different `first_agg_trade_id`, overlapping TID range).                                                           | Deploy/Ops   | `maintenance.py:141-200`       |
| **P2-7** | `store_bars_batch()` has no try/except — raw exceptions propagate, inconsistent with `store_bars_bulk()` pattern.                                                          | Python Layer | `bulk_operations.py:978-984`   |
| **P2-8** | `valid_pairs` filter silently drops bars with no dead-letter, no Telegram alert.                                                                                           | Python Layer | `sidecar.py:796`               |
| **P2-9** | Deploy checkpoint clear BEFORE venv swap — if swap fails, checkpoints gone, next start does unnecessary full rebuild.                                                      | Deploy/Ops   | SKILL.md Phase 4.3→4.4         |

### P3: Code Quality / Consistency

| ID       | Finding                                                                                                                       | Layer        | File:Line                            |
| -------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------ | ------------------------------------ |
| **P3-1** | Batch path `defer_open` desync — local variable not synced to `self.defer_open`. Accidentally correct but invariant violated. | Rust Core    | `processor.rs:649-860`               |
| **P3-2** | `seed_trade_id()` and `fill_gap()` accept negative trade IDs without validation at FFI boundary.                              | FFI          | `stream_manager_bindings.rs:171,383` |
| **P3-3** | SQL f-string interpolation in DELETE queries (internal-only, no attack surface).                                              | Python Layer | `maintenance.py:249,396-404`         |
| **P3-4** | Fallback `written` count uses `len(bars)` instead of CH-confirmed count.                                                      | Python Layer | `sidecar.py:857`                     |

---

## Cross-References

- **#293** (INSERT-only replace): P0-2 is the flip side — `_do_replace()` still does DELETE-first despite #293's INSERT-only direction. Completing #293 would eliminate P0-2.
- **#294** (seed TID inflation): P1-4 (seed failure at DEBUG) is a contributing factor. P0-4 (delete failure swallowed) exacerbates the seed ordering issue #294 documents.
- **#280** (writer ownership): P0-3 (heartbeat blind spot) is a direct consequence — heartbeat doesn't respect the ownership boundary.

---

## Implementation Order

1. **Sprint 1 (P0s)**: P0-3 (heartbeat ownership split), P0-4 (delete failure escalation), P0-2 (INSERT-first in \_do_replace), P0-1 (ring buffer capacity/drain strategy)
2. **Sprint 2 (P1s)**: P1-1 (fill failure alert), P1-2 (ghost bar sync), P1-3 (scoped delete), P1-5+P1-6 (deploy hardening)
3. **Sprint 3 (P2s)**: P2-1 (sidecar_metrics heartbeat), P2-4 (mutations gate), P2-3 (Lethe during REST), remaining P2s
4. **Backlog (P3s)**: All P3s as opportunistic fixes
