---
name: sidecar-restart-zero-overlap-zero-gap-recovery-playbook
description: >-
  Operational playbook for sidecar restart recovery with zero overlaps and zero gaps.
  Covers checkpoint preservation, min-TID overlap guard, correct stop-delete-backfill-start
  sequencing, and every anti-pattern that caused cascading data corruption.
  Use when deploying new versions, recovering from sidecar crashes, repairing today's
  Stathera anomalies, or planning any operation that stops the sidecar.
  SSoT for restart recovery patterns — other skills link here instead of duplicating.
allowed-tools: Read, Bash, Grep, Glob, Agent
---

# Sidecar Restart Zero-Overlap Zero-Gap Recovery Playbook

ultrathink

SSoT for all operational patterns involving sidecar restarts, today's data repair, and checkpoint continuity. Every pattern here was validated in production; every anti-pattern caused real data corruption.

## When to Use

- `/sidecar-restart-zero-overlap-zero-gap-recovery-playbook` — Before any sidecar restart or today's data repair
- Before deploying a new version via `/odb-ship`
- After a sidecar crash or OOM kill
- When heartbeat reports overlaps or gaps on streaming symbols
- When planning manual bar deletion or CH maintenance

---

## Autonomous Recovery (v13.28+)

**The sidecar now handles recovery autonomously.** When checkpoints are missing or stale, `_inline_startup_backfill()` runs `backfill_gap()` for each affected pair BEFORE the engine starts. No manual intervention needed.

```
Sidecar startup (autonomous):
1. Check checkpoints for all pairs
2. IF all valid → skip backfill, seamless continuation
3. IF any missing/stale → run backfill_gap() per pair (fills CH → now via REST)
4. start_ws() → WS trades buffer
5. engine.start() → drains buffer, Puck fills tiny transition gap
6. Min-TID guard as defense-in-depth
```

**This eliminates all manual sequencing.** The golden rule below is for manual repairs only (when you need to DELETE corrupted data).

---

## The Golden Rule (Manual Repair Only)

**Stop → Delete → Backfill → Start.** Always in this order. Never skip a step, never reorder.

```
1. Stop sidecar          (saves checkpoints on shutdown)
2. Delete corrupted bars  (ALTER TABLE DELETE, wait for mutations)
3. Backfill trailing edge (backfill_gap from CH last bar to now)
4. Start sidecar          (loads checkpoints or min-TID guard activates)
```

Violating this order causes cascading failures:

| Violation                      | Consequence                                                                                   |
| ------------------------------ | --------------------------------------------------------------------------------------------- |
| Start before backfill          | Sidecar writes bars from NOW; backfill only fills tiny trailing edge; 18h gap from midnight   |
| Delete without stopping        | Kintsugi's concurrent writes hit 300s CH timeout during DELETE mutation scan                  |
| Backfill while sidecar running | Concurrent writer race → overlapping bars (the exact problem #280 solved)                     |
| Skip backfill entirely         | Gap from yesterday's last bar to sidecar's first WS trade — Puck can't fill 18h gaps reliably |

---

## Validated Patterns

### Pattern 1: Clean Deploy (Normal — No Corruption)

The sidecar had valid checkpoints and no Stathera anomalies. Standard deploy.

```
1. monit unmonitor sidecar + kintsugi
2. systemctl stop sidecar + kintsugi
3. Atomic venv swap (.venv-staging → .venv)
4. systemctl start sidecar     ← checkpoints load, bars_emitted resumes
5. Verify: checkpoints_loaded > 0 in logs
6. systemctl start kintsugi
7. monit monitor sidecar + kintsugi
```

**Key check**: `checkpoints_loaded=8` (or however many streaming pairs). If 0, the min-TID guard activates and prevents overlaps — but you'll have a gap from the last backfill bar to the first WS trade.

### Pattern 2: Repair Today's Overlaps (Stathera Shows Overlaps)

Today's bars have overlapping trade-ID ranges from a bad restart.

```
1. monit unmonitor sidecar
2. systemctl stop sidecar              ← saves checkpoints
3. DELETE today's bars:
     ALTER TABLE ... DELETE WHERE symbol='X' AND toDate(open_time_ms/1000)='YYYY-MM-DD'
4. Wait for mutations:
     SELECT count() FROM system.mutations WHERE is_done=0   ← must be 0
5. Clear streaming checkpoints:
     rm ~/.cache/opendeviationbar/checkpoints/streaming_*.json
6. backfill_gap(force=True, overlap_strategy="replace") for each pair
     ← fills from yesterday's last bar to NOW (covers entire day)
7. Verify Stathera: overlaps=0, gaps=0
8. systemctl start sidecar
9. monit monitor sidecar
```

**Critical**: Step 6 MUST happen before step 8. If reversed, sidecar writes bars from NOW and backfill only fills a tiny trailing edge.

### Pattern 3: Sidecar OOM/Crash Recovery (No Manual Intervention)

Sidecar was killed by systemd OOM or crashed. Checkpoints were saved on previous clean shutdown (or periodically).

```
1. systemd auto-restarts sidecar (Restart=on-failure)
2. Sidecar loads checkpoints:
   a. checkpoints_loaded > 0 → clean continuation, Puck fills any small gap
   b. checkpoints_loaded = 0 → min-TID guard activates, prevents overlaps
3. If (b): small gap from CH last bar to first WS trade
   → kintsugi repairs it tomorrow (today-guard)
   → OR manually run Pattern 2 if immediate fix needed
```

**No manual intervention needed** for normal crashes. The min-TID guard + kintsugi-tomorrow is the autonomous safety net.

### Pattern 4: Version Deploy with Rust Changes (Full Wheel)

New `.so` binary — must install via wheel, not rsync.

```
1. Full /odb-ship pipeline (build wheels, publish, deploy)
2. During Phase 4 service orchestration:
   a. Stop sidecar (saves checkpoints)
   b. Atomic venv swap (new wheel)
   c. Start sidecar
3. If checkpoints load: done (Pattern 1)
4. If checkpoints rejected (stale):
   → min-TID guard prevents overlaps
   → backfill_gap before start if you want zero gaps immediately
   → OR let kintsugi heal tomorrow
```

### Pattern 5: Version Deploy with Python-Only Changes (Hotfix)

No Rust changes — rsync Python files to site-packages.

```
1. Push to origin/main
2. Git sync on bigblack: git fetch && git reset --hard origin/main
3. rsync Python files (exclude _core*.so)
4. Clear __pycache__
5. Clear streaming checkpoints:
     rm -f ~/.cache/opendeviationbar/checkpoints/streaming_*.json
6. Restart sidecar
```

**Why always clear checkpoints**: Even though the .so doesn't change, Python-only hotfixes can change checkpoint schema or startup code. Stale checkpoints with incompatible schema cause crash-loops that prevent the clean-slate DELETE from running (#sidecar-restart-overlaps, 2026-03-28). Clearing checkpoints is a safe no-op when they're compatible.

**Sidecar handles clean-slate autonomously** (v13.45.0+): On every startup, `_create_engine()` deletes stale bars before `fill_from_rest`. No manual DELETE step needed.

> **v8.0 Aion-native**: For crypto symbols, the sidecar resumes from `min(max(last_agg_trade_id))` across thresholds per symbol (NOT midnight crossover TID). DELETE uses TID-based `WHERE first_agg_trade_id > resume_tid` (NOT date-based). The midnight crossover TID path only applies to day-mode forex symbols.

---

### Pattern 6: Post-Reboot Recovery (Full Machine Restart)

After bigblack reboots (power loss, kernel update, network issue):

1. All services auto-restart via systemd + monit
2. Sidecar's inline backfill fills today for streaming symbols
3. **But**: if previous session wrote corrupted bars (overlaps from pre-reboot), inline backfill's `overlap_strategy="skip"` preserves them
4. **Fix**: One Pattern 2 pass on today's streaming symbols:

```bash
# Stop sidecar, delete today, restart (inline backfill covers full day)
sudo monit unmonitor sidecar
systemctl --user stop opendeviationbar-sidecar
for sym in BTCUSDT ETHUSDT; do
  curl -s "http://localhost:8123/" --data-binary \
    "ALTER TABLE opendeviationbar_cache.open_deviation_bars DELETE WHERE symbol='$sym' AND toDate(open_time_ms/1000)='$(date -u +%Y-%m-%d)' SETTINGS mutations_sync=1"
done
# Wait for mutations: SELECT count() FROM system.mutations WHERE is_done=0
rm -f ~/.cache/opendeviationbar/checkpoints/streaming_*.json
systemctl --user start opendeviationbar-sidecar
sudo monit monitor sidecar
```

1. Kintsugi handles yesterday+ automatically — no action needed
2. Verify: `overlaps=0, gaps=0` for streaming symbols via Stathera query

---

## Anti-Patterns (Never Do These)

### Anti-Pattern 1: Auto-Delete Checkpoint Files

**What**: Stale checkpoint guard deletes checkpoint files when `cp_first_tid > ch_last_tid + 1`.

**Why it fails**: Next restart has zero checkpoints → fresh processor → bars overlap with CH's existing data. Cascading: every subsequent restart also has zero checkpoints (deleted files can't be recovered).

**The fix (v13.27.0)**: Skip loading + warn, but **preserve the file**. On shutdown, the session writes fresh checkpoints that will be valid next restart.

**Incident**: Mar 18, 2026 — 7/8 checkpoints auto-deleted on first restart, causing 826 overlapping bars across all thresholds.

### Anti-Pattern 2: Start Sidecar Before Backfill

**What**: After deleting today's bars, immediately start sidecar without running backfill first.

**Why it fails**: Sidecar writes bars from NOW (e.g., 18:20 UTC). Then backfill runs but CH already has recent bars — it only fills a tiny trailing edge. Result: 18-hour gap from midnight to 18:20.

**The fix**: Always run `backfill_gap(force=True)` BEFORE starting sidecar. Backfill fills from yesterday's last bar to now, covering the entire day.

**Incident**: Mar 18, 2026 — 6 gaps including 908K and 975K trade-ID gaps from midnight to sidecar start.

### Anti-Pattern 3: Individual Bar Deletion

**What**: Delete specific overlapping bars by trade-ID range to "fix" overlaps.

**Why it fails**: Each deletion creates NEW gaps adjacent to the deleted bars. The gap-overlap-gap cascade makes the situation worse.

**The fix**: Only kintsugi's full day-recompute (DELETE entire day + reprocess from scratch) can fix complex corruption. For streaming pairs, use Pattern 2.

**Incident**: Mar 16, 2026 — individual bar deletion cascade created 665 overlaps + 169 gaps.

### Anti-Pattern 4: Asclepius (Intra-Day Self-Heal on Live Engine)

**What**: Attempt to fill OLD gaps on a live sidecar engine using `backfill_gap()` or `fill_gap()`.

**Why it fails**:

- `backfill_gap()` creates a fresh stateless processor → produces malformed bars (wrong threshold)
- `fill_gap()` injects old trades into an advanced processor → produces bars that overlap with existing data

**The fix**: Only two safe gap-fill mechanisms exist:

- **Puck** (inline real-time, sub-3s) — for gaps detected on live WS trades
- **Kintsugi** (full day-recompute, runs tomorrow) — for historical gaps

**Incident**: Mar 16, 2026 — Asclepius caused 665 overlaps + 169 gaps in 10 hours. Permanently removed (#284).

### Anti-Pattern 5: DELETE Concurrent with Kintsugi

**What**: Run `ALTER TABLE DELETE` while kintsugi is actively writing bars.

**Why it fails**: DELETE scans all parts (3000+). During the scan, kintsugi's ClickHouse writes timeout at 300s. This triggers REST fallback warnings, Telegram notification storms, and potential data loss.

**The fix**: Stop kintsugi before DELETE. Or at minimum, time the DELETE during kintsugi's sleep interval (60-300s between passes).

**Incident**: Mar 18, 2026 — storm of CH write timeout warnings + Telegram SSL errors during concurrent DELETE + kintsugi writes.

### Anti-Pattern 6: Clean-Slate DELETE on Sidecar Restart (RESOLVED v13.45.0)

**Historical context (v13.27.0)**: Clean-slate DELETE was an anti-pattern because it created a gap from midnight to first WS trade — no mechanism existed to fill that gap during startup.

**Resolved (v13.45.0)**: The pre-fill clean-slate approach (DELETE + `fill_from_rest`) closes the midnight gap entirely. The sidecar now uses this exact approach on every startup: DELETE today's bars via partition-scoped `DELETE IN PARTITION`, then `fill_from_rest` from midnight crossover TID, then `sync_flush` to ClickHouse. Fresh processors recompute on an empty day — zero conflicts, restartable any number of times.

**This is no longer an anti-pattern.** It is the current production startup sequence. For crypto (Aion-native, v8.0): TID-based DELETE + resume from `min(max(last_agg_trade_id))`. For day-mode forex: date-based DELETE + midnight crossover TID.

### Anti-Pattern 7: Stale Checkpoints with fill_from_rest (#295)

**What**: Inject saved checkpoints before running fill_from_rest.

**Why it fails**: Checkpoint TIDs override midnight preflight seeds. Rust `fill_from_rest` uses `max(checkpoint_tid, ch_seed_tid)` — stale checkpoint always wins over the midnight crossover TID → fill_from_rest skips trades from midnight to the checkpoint → gaps.

**The fix (v13.43.3)**: Skip checkpoint injection entirely when `fill_from_rest` is available. Fresh processors start from the midnight crossover TID, guaranteeing zero-gap coverage from midnight forward.

### Anti-Pattern 8: Any Post-Write DELETE Strategy (#295 i1-10)

**What**: Delete stale bars AFTER `fill_from_rest` + `sync_flush` writes new bars.

**Why it fails**: 10 iterations proved every post-write DELETE variant fails. Lightweight DELETE masks persist and eat new bars. Traditional DELETE is too slow (2+ min) and catches sync_flush bars during the scan. Verification gates and per-pair selective DELETE add complexity but can't handle the fundamental issue: different processor instances produce bars with different TID boundaries, making selective cleanup impossible for high-threshold pairs that don't breach during fill_from_rest.

**The fix (iteration 11, v13.45.0)**: Pre-fill clean-slate `ALTER TABLE DELETE` — delete ALL today's bars BEFORE fill_from_rest. Fresh processors recompute from midnight crossover on an empty day. Uses traditional mutation with `mutations_sync=1` (physically rewrites parts, no mask). Pre-fill timing is safe: sidecar hasn't started streaming, so 1-2 min scan doesn't race anything. Zero conflicts, restartable any number of times.

**For manual recovery (Pattern 2)**: Same approach — `ALTER TABLE ... DELETE` (traditional mutation) with `mutations_sync=1`.

### Anti-Pattern 9: Starting Kintsugi Before sync_flush Completes (#295)

**What**: Start kintsugi immediately after sidecar, without waiting for fill_from_rest + sync_flush.

**Why it fails**: Kintsugi's `_do_replace()` creates DELETE mutations that scan ~5000 parts. If today's bars haven't been written yet (sync_flush pending), kintsugi mutations can delete them as collateral when they eventually land in shared parts. Even with TID-filtered DELETEs, the mutation queue serialization causes timing-dependent bar loss.

**The fix**: After starting sidecar, poll logs for `sync_flush: drained and wrote`. Verify zero Stathera anomalies on today. THEN start kintsugi.

---

## Decision Tree

```
Heartbeat shows Stathera anomalies on streaming symbols?
│
├── Overlaps only (tid_delta < 0)?
│   ├── Are they on today's date?
│   │   ├── YES → Pattern 2 (stop, delete today, backfill, start)
│   │   └── NO  → Wait for kintsugi (owns yesterday+)
│   └── Is sidecar running cleanly now?
│       ├── YES → RMT will merge bars with same first_tid; kintsugi fixes the rest tomorrow
│       └── NO  → Fix sidecar first, then assess
│
├── Gaps only (tid_delta > 0)?
│   ├── Day-boundary gaps (day_gap = 1)?
│   │   └── Normal structural gaps — kintsugi heals tomorrow
│   ├── Intra-day gaps on today?
│   │   ├── Small (<1000 trades) → Puck should fill on next WS trade
│   │   └── Large (>1000 trades) → Pattern 2 (stop, delete, backfill, start)
│   └── Intra-day gaps on yesterday+?
│       └── Kintsugi's domain — check kintsugi logs
│
└── Both overlaps AND gaps?
    └── Pattern 2 is the only safe fix (full day-recompute via delete + backfill)
```

---

## Stathera Verification Queries

### Check overlaps + gaps for streaming symbols

```sql
WITH bars AS (
    SELECT symbol, threshold_decimal_bps,
           first_agg_trade_id, last_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER w AS prev_last_tid,
           first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta
    FROM opendeviationbar_cache.open_deviation_bars FINAL
    WHERE symbol IN ('BTCUSDT', 'ETHUSDT')
    WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode
                 ORDER BY first_agg_trade_id)
)
SELECT
    countIf(tid_delta < 0) AS overlaps,
    countIf(tid_delta > 0) AS gaps
FROM bars WHERE prev_last_tid > 0
```

**Success criteria**: `overlaps = 0`. Gaps may exist at day boundaries (structural) — check with the detail query below.

### Detail: identify each anomaly with dates

```sql
WITH bars AS (
    SELECT symbol, threshold_decimal_bps,
           first_agg_trade_id, last_agg_trade_id,
           lagInFrame(last_agg_trade_id, 1) OVER w AS prev_last_tid,
           first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta,
           toDate(close_time_ms/1000) AS bar_date,
           toDate(lagInFrame(close_time_ms, 1) OVER w / 1000) AS prev_bar_date
    FROM opendeviationbar_cache.open_deviation_bars FINAL
    WHERE symbol IN ('BTCUSDT', 'ETHUSDT')
    WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode
                 ORDER BY first_agg_trade_id)
)
SELECT symbol, threshold_decimal_bps, tid_delta, bar_date, prev_bar_date,
       bar_date - prev_bar_date AS day_gap,
       if(tid_delta < 0, 'OVERLAP', 'GAP') AS anomaly_type
FROM bars
WHERE (tid_delta < 0 OR (tid_delta > 0 AND bar_date = prev_bar_date))
  AND prev_last_tid > 0
ORDER BY symbol, threshold_decimal_bps
```

### Check pending mutations (before backfill)

```sql
SELECT count() FROM system.mutations
WHERE database = 'opendeviationbar_cache' AND is_done = 0
```

Must return `0` before proceeding with backfill.

---

## Backfill Script (Copy-Paste Ready)

Save as `/tmp/odb_fill_today_gap.py` on bigblack:

```python
"""Fill trailing edge from CH last bar to now for streaming pairs.

Run AFTER deleting corrupted bars and BEFORE starting sidecar.
"""
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)s | %(message)s")

from opendeviationbar.backfill import backfill_gap

pairs = [
    ("BTCUSDT", 250), ("BTCUSDT", 500), ("BTCUSDT", 750), ("BTCUSDT", 1000),
    ("ETHUSDT", 250), ("ETHUSDT", 500), ("ETHUSDT", 750), ("ETHUSDT", 1000),
]

for sym, thr in pairs:
    try:
        logging.info("filling %s@%d...", sym, thr)
        result = backfill_gap(sym, thr, overlap_strategy="replace", force=True)
        logging.info("  %s@%d: %d bars", sym, thr, getattr(result, "bars_written", 0))
    except Exception:
        logging.exception("  FAILED: %s@%d", sym, thr)
```

Run: `~/opendeviationbar-py/.venv/bin/python3 /tmp/odb_fill_today_gap.py`

---

## Defense Layers (How They Interact)

| Layer  | Mechanism                    | Prevents                            | Added    |
| ------ | ---------------------------- | ----------------------------------- | -------- |
| **L1** | Cross-midnight bar filter    | Bars spanning day boundary          | v13.10.0 |
| **L2** | Pre-write gate (filter+warn) | Cross-midnight bars reaching CH     | v13.10.0 |
| **L3** | Charon dead-letter cleanup   | Permanent replay failures           | v13.10.0 |
| **L4** | Physical invariant gate      | Inverted trade IDs/timestamps       | v13.15.0 |
| **L5** | Min-TID overlap guard        | Fresh-processor bars overlapping CH | v13.27.0 |
| **L6** | Checkpoint file preservation | Cascading zero-checkpoint restarts  | v13.27.0 |
| **L7** | ReplacingMergeTree ORDER BY  | Same first_agg_trade_id dedup       | Schema   |
| **L8** | Kintsugi day-recompute       | Yesterday's complex corruption      | v12.65.0 |
| **L9** | Kintsugi today-guard         | Concurrent writer races             | v13.5.2  |

**L5 + L6 are the v13.27.0 additions** that close the restart overlap gap.

---

## Related

| Resource                                                                                                         | What to Find There                                     |
| ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| [/CLAUDE.md § Writer Ownership Model](/CLAUDE.md)                                                                | Sidecar owns today, kintsugi owns yesterday+           |
| [/.claude/skills/odb-ship/SKILL.md](/.claude/skills/odb-ship/SKILL.md)                                           | Full deploy pipeline (links here for restart patterns) |
| [/.claude/skills/odb-ship/references/lessons-learned.md](/.claude/skills/odb-ship/references/lessons-learned.md) | Individual incident write-ups                          |
| [/scripts/CLAUDE.md § Kintsugi](/scripts/CLAUDE.md)                                                              | Today-guard, parallelism tuning                        |
| [/deploy/CLAUDE.md](/deploy/CLAUDE.md)                                                                           | Env files, monit config, defense-in-depth              |
| [/python/opendeviationbar/validation/day_mode/CLAUDE.md](/python/opendeviationbar/validation/day_mode/CLAUDE.md) | Day-mode invariant enforcement                         |
