---
status: awaiting_human_verify
trigger: "22 Stathera overlaps after multiple sidecar restarts during v7.2/v7.3 deploy session"
created: 2026-03-28T00:00:00Z
updated: 2026-03-28T04:30:00Z
---

## Current Focus

hypothesis: CONFIRMED — Aion path ordering bug: \_inject_checkpoints() runs BEFORE DELETE in Aion path, so checkpoint crashes prevent DELETE from ever running. Day path has correct ordering (DELETE before checkpoint-dependent code).
test: Code review confirmed
expecting: n/a — confirmed
next_action: Fix code ordering + update documentation

## Symptoms

expected: Zero overlaps after sidecar restart. The odb-ship skill documents a clean-slate DELETE sequence that should prevent this.
actual: 22 overlaps (exactly 1 per symbol/threshold pair). All on today's data. RMT dedup via OPTIMIZE FINAL did not resolve them because bars have different first_agg_trade_id (different processor sessions produce different TID boundaries).
errors: Heartbeat showed "22 overlaps" in Stathera. All pairs affected equally.
reproduction: Three sidecar restarts in one session: (1) v7.2 full wheel deploy with odb-ship, (2) v7.3 Python-only hotfix via rsync, (3) another rsync for TID-sequential code. Restart #1 used full odb-ship (clean-slate). Restarts #2 and #3 used the hotfix rsync path which does NOT delete today's bars.
started: 2026-03-28 deploy session

## Eliminated

- hypothesis: The sidecar code does NOT delete today's bars on startup
  evidence: It DOES delete for both Aion and Day paths, but the Aion path ordering makes the DELETE fragile — checkpoint injection crash prevents DELETE from running.
  timestamp: 2026-03-28T03:30:00Z

- hypothesis: Hotfix rsync path itself causes overlaps
  evidence: The hotfix path simply restarts the sidecar. The sidecar's own \_create_engine() handles clean-slate. The issue is not the hotfix path but the sidecar code's ordering vulnerability that the hotfix path exposed.
  timestamp: 2026-03-28T04:00:00Z

## Evidence

- timestamp: 2026-03-28T03:00:00Z
  checked: odb-ship SKILL.md hotfix path (lines 778-820)
  found: Hotfix path does rsync + stop + start. Relies on sidecar startup for clean-slate.
  implication: Correct design IF sidecar startup always reaches DELETE.

- timestamp: 2026-03-28T03:10:00Z
  checked: sidecar/**init**.py \_create_engine() Aion path (lines 264-318)
  found: Aion path ordering: (1) \_inject_checkpoints() (2) \_seed_trade_ids_from_clickhouse() (3) DELETE today. If step 1 crashes, step 3 never runs.
  implication: ROOT CAUSE — stale checkpoints crash step 1, DELETE at step 3 never executes, old bars persist.

- timestamp: 2026-03-28T03:12:00Z
  checked: sidecar/**init**.py \_create_engine() Day path (lines 320-425)
  found: Day path ordering: (1) midnight preflight (2) yesterday integrity (3) KILL + OPTIMIZE + DELETE today (4) parquet seeds (5) fill_from_rest. DELETE runs BEFORE any checkpoint-dependent code.
  implication: Day path has correct ordering. Aion path should match.

- timestamp: 2026-03-28T03:15:00Z
  checked: symbols.toml for ouroboros_mode distribution
  found: ALL Binance symbols are ouroboros_mode = "aion". Only Exness forex symbols are "day". So the Aion path vulnerability affects ALL streaming symbols.
  implication: The Day path's correct ordering is never used for Binance symbols.

- timestamp: 2026-03-28T03:20:00Z
  checked: Aion DELETE exception handling (line 317)
  found: DELETE failures caught as non-fatal: except (OSError, RuntimeError, ImportError). If DELETE fails, sidecar continues with fill_from_rest, writing bars ON TOP of old bars.
  implication: Secondary vulnerability — DELETE failures should be fatal, not non-fatal.

- timestamp: 2026-03-28T03:25:00Z
  checked: restart playbook Anti-Pattern 6
  found: Lists "Clean-Slate DELETE on Sidecar Restart" as anti-pattern, but this was resolved in v13.45.0. The sidecar now uses this exact approach. Anti-pattern is stale.
  implication: Documentation contradicts code.

- timestamp: 2026-03-28T03:30:00Z
  checked: restart playbook Pattern 5 (Python-only hotfix)
  found: Does not mention that sidecar startup handles clean-slate autonomously. Does not mention checkpoint clearing.
  implication: Documentation is outdated.

- timestamp: 2026-03-28T04:00:00Z
  checked: odb-ship hotfix path checkpoint clearing
  found: odb-ship documents checkpoint clearing only when checkpoint code changed (gate at lines 822-833). Does not clear checkpoints by default.
  implication: Hotfix path should always clear checkpoints (safe no-op if not stale).

## Resolution

root_cause: Aion startup path ordering bug in `_create_engine()`. The Aion path runs `_inject_checkpoints()` BEFORE the DELETE today operation (lines 264-318). When checkpoint files have a stale schema (e.g., after a hotfix rsync changes checkpoint.py), `_inject_checkpoints()` crashes, and the DELETE at lines 285-318 never executes. The sidecar exits, systemd restarts it, same crash repeats. Bars from the previous successful fill_from_rest persist in ClickHouse. After 5 crash-restarts, the bars have accumulated from multiple processor sessions with different TID boundaries. When the user manually clears checkpoints and restarts, the final successful run's DELETE should clean up — BUT if checkpoint clearing happened AFTER the crash-loop and BEFORE the TID-sequential fix at ~03:34, there was a window where bars from multiple sessions coexisted. The Day path (lines 320-425) has correct ordering: DELETE runs BEFORE any checkpoint-dependent code. The Aion path should match. Contributing factor: Aion DELETE exception handler (line 317) catches failures as non-fatal, allowing fill_from_rest to write on top of existing bars if DELETE fails for any reason.

fix:

1. [CODE] Move Aion DELETE BEFORE \_inject_checkpoints() in \_create_engine() — match Day path ordering
2. [CODE] Make Aion DELETE failure fatal (raise, not log warning) — prevent writing on top of existing bars
3. [DOC] Update odb-ship hotfix path to always clear checkpoints before restart
4. [DOC] Update restart playbook Pattern 5 to document sidecar's autonomous clean-slate
5. [DOC] Update restart playbook Anti-Pattern 6 to note resolution in v13.45.0

verification: Code review + ruff check passes. Syntax validated. Awaiting production deploy verification.
files_changed:

- python/opendeviationbar/sidecar/**init**.py
- python/opendeviationbar/sidecar/CLAUDE.md
- .claude/skills/odb-ship/SKILL.md
- .claude/skills/sidecar-restart-zero-overlap-zero-gap-recovery-playbook/SKILL.md
