# Deploy Safety Findings: odb-ship Pipeline Inconsistency Analysis

**Date**: 2026-03-21
**Scope**: Can the odb-ship deploy pipeline leave bigblack in an inconsistent state?
**Method**: Static analysis of SKILL.md Phase 4 + sidecar.py startup sequence + maintenance.py

---

## TGI-1: Phase 4 (Service Orchestration) — SSH Drop Interruption Points

### Pipeline Step Sequence (Phase 4)

```
4.1  monit unmonitor + systemctl stop
4.1b KILL pending CH mutations (loop until 0)
4.2  OPTIMIZE TABLE FINAL (force RMT dedup)
4.3  Clear streaming checkpoints
4.4  Atomic venv swap (.venv-staging -> .venv)
4.5  Sync systemd units + daemon-reload
4.6  Start services (sidecar first, then kintsugi)
4.7  Re-monitor in monit
4.8  Post-start Stathera audit
4.9  Verify services active
```

### Finding 1: CRITICAL — `.venv-old` Destroyed Immediately, Rollback Impossible

**Location**: SKILL.md line 447

```bash
rm -rf .venv-old && mv .venv .venv-old 2>/dev/null; mv .venv-staging .venv && rm -rf .venv-old
```

The `rm -rf .venv-old` at the END of the SAME command chain destroys the rollback venv immediately after the swap. The rollback section (line 621) says "if .venv-old exists" — but it never exists because line 447 deletes it.

**Impact**: If the new version crashes on startup (bad .so, bad Python code), there is no fast rollback path. The operator must `uv pip install opendeviationbar==PREVIOUS_VERSION` which requires PyPI connectivity and time.

**Fix**: Remove the trailing `&& rm -rf .venv-old`. Keep `.venv-old` alive until the NEXT deploy. The leading `rm -rf .venv-old` already handles cleanup of the previous generation.

### Finding 2: HIGH — SSH Drop Between Steps 4.1 and 4.4 = Services Down, No Restart

If SSH drops after step 4.1 (services stopped, monit unmonitored) but before step 4.7 (monit re-monitored):

- Sidecar and kintsugi are stopped
- Monit won't auto-restart them (unmonitored)
- No watchdog will recover — the system is in a dead state
- Data gap accumulates silently until operator notices

**Impact**: Unmonitored downtime of unknown duration. Every minute of sidecar downtime = potential Stathera gaps.

**Fix**: Add a deploy sentinel file (e.g., `/tmp/opendeviationbar-deploy-in-progress`) that monit checks. If deploy started >10 minutes ago and services are down, monit re-monitors and restarts. Or: re-monitor monit as the FIRST step after service stop (monit's 120s cycle provides a natural restart delay that may be acceptable).

### Finding 3: HIGH — SSH Drop After 4.3 (Checkpoints Cleared) But Before 4.6 (Services Start)

If checkpoints are cleared (step 4.3) and the venv is swapped (step 4.4), but the SSH connection drops before services start:

- Checkpoints are gone
- When an operator manually starts the sidecar, it enters `fill_from_rest` mode
- `fill_from_rest` triggers `_delete_today_bars` which DELETEs all of today's bars
- This is CORRECT behavior (clean slate), but the gap between checkpoint clear and service start is unprotected

**Mitigation already exists**: The sidecar's autonomous recovery handles this gracefully — `fill_from_rest` rebuilds from REST data. The gap is the downtime itself, not data corruption.

### Finding 4: MEDIUM — `_delete_today_bars` Failure is Silently Swallowed

**Location**: sidecar.py line 745

```python
except Exception:
    logger.warning("clean_slate: failed to delete today's bars (non-fatal)", exc_info=True)
```

If `_delete_today_bars` fails (e.g., ClickHouse timeout, mutation stuck), `fill_from_rest` proceeds anyway. The new bars from REST will interleave with old bars that should have been deleted. This produces overlapping bars.

**Impact**: Silent overlap creation that requires OPTIMIZE FINAL or kintsugi to clean up.

**Fix**: Make `_delete_today_bars` failure fatal when called during startup (not during normal operation). If today's bars can't be deleted, `fill_from_rest` should not proceed.

### Finding 5: MEDIUM — OPTIMIZE FINAL Does Not Resolve Straddle Bars

**Location**: maintenance.py line 418-485

RMT dedup keys on `(symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)`. OPTIMIZE FINAL only deduplicates rows with the SAME `first_agg_trade_id`.

Straddle bars — bars from different processor sessions that start at different trade IDs but overlap in trade-ID range — are NOT resolved by OPTIMIZE FINAL. They have different `first_agg_trade_id` values, so RMT treats them as distinct rows.

The `store_bars_replacing()` function handles this via `_find_straddle_bars()` + targeted DELETE. But the deploy pipeline's step 4.2 (OPTIMIZE FINAL) does NOT run straddle detection. It assumes OPTIMIZE FINAL is sufficient.

**Impact**: Pre-existing straddle bars survive the deploy. The sidecar starts with stale overlapping data. The post-start Stathera audit (step 4.8) may detect overlaps, but the remediation is "run OPTIMIZE FINAL again" — which won't fix straddle bars either.

**Fix**: After OPTIMIZE FINAL in step 4.2, run a straddle bar detection query. If straddle bars exist, run targeted DELETEs before starting services.

### Finding 6: MEDIUM — Step 4.1b Mutation Kill Loop Has No Timeout

```bash
ssh bigblack 'while true; do
  COUNT=$(curl ...)
  if [ "$COUNT" = "0" ]; then break; fi
  curl -s ... "KILL MUTATION ..." > /dev/null
  sleep 2
done'
```

If mutations keep being created (e.g., by a process that wasn't properly stopped), this loop runs forever. There's no max-iteration guard.

**Fix**: Add a max iteration count (e.g., 30 iterations = 60 seconds). After that, warn and proceed.

### Finding 7: LOW — Checkpoint Clear (4.3) Before Venv Swap (4.4) Creates Ordering Risk

Checkpoints are cleared BEFORE the venv swap. If the swap fails (e.g., `.venv-staging` doesn't exist because step 3.3 failed), the system is left with:

- Old venv still in place
- Checkpoints deleted
- Next sidecar start will do full `fill_from_rest` unnecessarily

**Fix**: Move checkpoint clearing to AFTER venv swap verification, or make it conditional on swap success.

---

## TGI-2: Sidecar Startup Sequence — Seed Ordering Invariants

### The Invariant (from CLAUDE.md)

```
1. Start WS (buffered mode)
2. Inject checkpoints
3. Delete today's bars
4. Seed from ClickHouse (MUST be AFTER step 3)
5. Midnight preflight (overrides seeds with crossover TID)
6. fill_from_rest
```

### Verified in Code (sidecar.py:961-1078)

The ordering is correctly implemented. Key observations:

- **Steps 3-4-5 are ONLY executed when `needs_fill=True`** (i.e., some pairs lack checkpoints). When all checkpoints load, only step 4 runs (seeding), and steps 3 and 5 are skipped.
- **The non-fill path** (line 994-995) runs `_seed_trade_ids_from_clickhouse` WITHOUT deleting today's bars first. This is correct because checkpoints are valid — no need to delete.
- **fill_from_rest failure** (line 1028-1030) is caught and falls back to WS-only start with `rest_bars = 0`. This means today's bars were already deleted (step 3) but no REST bars replaced them. The min-TID guard (lines 1046-1074) provides defense-in-depth against overlaps, but there IS a gap until kintsugi repairs it.

### Risk: fill_from_rest Failure After \_delete_today_bars

If `_delete_today_bars` succeeds but `fill_from_rest` fails:

- Today's bars are deleted
- No REST bars replace them
- WS starts from now, creating a gap from midnight to now
- Kintsugi won't repair today's gaps (writer ownership model: sidecar owns today)
- Gap persists until next day when kintsugi takes ownership

**Impact**: Potential 0-24h gap on all streaming pairs if REST is down during deploy.

---

## TGI-3: Deploy-Time Guards — What's Missing

### Existing Guards

| Guard                       | Location            | Purpose                                 |
| --------------------------- | ------------------- | --------------------------------------- |
| Monit unmonitor before stop | Step 4.1            | Prevents monit auto-restart during swap |
| Mutation kill loop          | Step 4.1b           | Clears blocking mutations               |
| OPTIMIZE FINAL              | Step 4.2            | RMT dedup (but not straddle)            |
| Checkpoint clear            | Step 4.3            | Clean slate for new session             |
| Staging venv verification   | Step 3.5            | Validates new code before swap          |
| Post-start Stathera audit   | Step 4.8            | Detects overlaps after start            |
| min-TID guard               | sidecar.py:1046     | Prevents overlapping bars from WS       |
| committed_floors            | live_engine.rs (L8) | Dedup floor per threshold               |

### Missing Guards

| Missing Guard                              | Risk                                                                | Severity |
| ------------------------------------------ | ------------------------------------------------------------------- | -------- |
| **Deploy timeout watchdog**                | SSH drop leaves system in unmonitored stopped state indefinitely    | HIGH     |
| **Rollback venv preservation**             | `.venv-old` destroyed immediately, no fast rollback                 | CRITICAL |
| **Straddle bar detection** at deploy time  | OPTIMIZE FINAL doesn't fix straddle overlaps                        | MEDIUM   |
| **fill_from_rest failure escalation**      | Delete succeeds + fill fails = gap that kintsugi can't repair today | MEDIUM   |
| **Mutation kill loop timeout**             | Infinite loop if mutations regenerate                               | MEDIUM   |
| **\_delete_today_bars failure escalation** | Silent swallow leads to interleaved bars                            | MEDIUM   |
| **Pre-deploy Stathera snapshot**           | No baseline to compare post-deploy health against                   | LOW      |
| **Service start verification timeout**     | `sleep 8` is arbitrary, no health poll loop                         | LOW      |

---

## Summary

The odb-ship pipeline is well-designed for the happy path but has several interruption vulnerabilities:

1. **The most critical issue is the immediate destruction of `.venv-old`** — a one-line fix (`rm -rf .venv-old` at the end of the swap command should be removed).

2. **The second most critical issue is the lack of a deploy timeout watchdog** — if SSH drops between monit unmonitor and monit re-monitor, the system stays down indefinitely with no automatic recovery.

3. **OPTIMIZE FINAL is insufficient for straddle bars** — the deploy pipeline assumes it resolves all overlaps, but straddle bars (different `first_agg_trade_id`, overlapping trade-ID ranges) survive OPTIMIZE FINAL. Deploy should run straddle detection + targeted DELETE.

4. **`_delete_today_bars` + `fill_from_rest` failure combination** creates an unrecoverable same-day gap. The writer ownership model prevents kintsugi from repairing it.
