---
status: diagnosed
trigger: "3 Stathera anomalies detected in first heartbeat after v13.43.1 deploy: 2 AVAXUSDT overlaps, 1 LINKUSDT gap, 1 XRPUSDT gap"
created: 2026-03-22T04:30:00Z
updated: 2026-03-22T05:15:00Z
---

## Current Focus

hypothesis: CONFIRMED - All 3 anomalies are PRE-EXISTING, created by v13.40.1/v13.41.0 sidecar sessions where \_delete_today_bars FAILED. v13.43.1 deploy did NOT cause them.
test: n/a - root cause confirmed
expecting: n/a
next_action: Return diagnosis. Kintsugi will auto-repair LINKUSDT and XRPUSDT gaps (yesterday's data). AVAXUSDT overlaps need manual kintsugi repair.

## Symptoms

expected: Zero Stathera anomalies after deploy with OPTIMIZE FINAL + clean restart
actual: |

- AVAXUSDT@500: 2 overlaps near midnight (tid_delta -40 and -184, bars at 23:57-23:59 UTC)
- LINKUSDT@500: 1 gap of 7,376 trades (gap at 20:26 UTC)
- XRPUSDT@500: 1 gap of 53,683 trades (gap at 15:18 UTC)
  errors: No errors from sidecar or kintsugi
  reproduction: Detected in first heartbeat after v13.43.1 deploy
  started: Created by v13.40.1 (20:01 Mar 21) and v13.41.0 (23:54 Mar 21) sidecar restarts

## Eliminated

- hypothesis: v13.43.1 deploy caused the anomalies
  evidence: v13.43.1 startup at 04:02 shows clean_slate SUCCESS, 17/17 preflight, all bars computed_at before 04:02
  timestamp: 2026-03-22T05:00:00Z

- hypothesis: OPTIMIZE FINAL should have deduped the AVAXUSDT overlaps
  evidence: Overlapping bars have different first_agg_trade_id (232238341 vs 232238381) which is part of ORDER BY key, so RMT cannot collapse them
  timestamp: 2026-03-22T05:05:00Z

## Evidence

- timestamp: 2026-03-22T04:35:00Z
  checked: AVAXUSDT computed_at timestamps
  found: |
  Bar [232238341,232238564] computed_at 2026-03-22 00:12:45 (sidecar PID 1948271, v13.41.0)
  Bar [232238381,232238470] computed_at 2026-03-22 00:31:07 (kintsugi repair)
  Bar [232238471,232238586] computed_at 2026-03-22 00:31:07 (kintsugi repair)
  implication: Sidecar wrote one large bar, kintsugi wrote two smaller bars from same trade range. Concurrent writer race.

- timestamp: 2026-03-22T04:37:00Z
  checked: LINKUSDT and XRPUSDT gap boundaries
  found: |
  LINKUSDT: pre-gap bar (last_tid=286497186) written by kintsugi v13.38.0 at 02:52 on Mar 21
  LINKUSDT: post-gap bar (first_tid=286504563) written at 00:12 on Mar 22 - gap covers 00:00-20:26 on Mar 21
  XRPUSDT: pre-gap bar (last_tid=697168853) written by kintsugi v13.39.0 at 18:33 on Mar 21
  XRPUSDT: post-gap bar (first_tid=697222537) written at 00:12 on Mar 22 - gap covers 00:00-15:18 on Mar 21
  NO bars exist in the gap ranges (never written, not deleted)
  implication: fill_from_rest started too late, skipping the beginning of 2026-03-21

- timestamp: 2026-03-22T04:40:00Z
  checked: PID 1781048 (v13.40.1) startup sequence at 20:01
  found: |
  "clean_slate: failed to delete today's bars (non-fatal)" at 20:01:55
  Seeded from ClickHouse at 20:01:57 (AFTER failed delete - picked up stale bars)
  Preflight overrode XRPUSDT to 697168854 but CH seed was already higher (697222536)
  fill_from_rest started from 697222536 instead of 697168854
  LINKUSDT preflight SKIPPED ("could not determine crossover trade ID")
  implication: Failed delete_today + CH seed = inflated seed that overrides preflight crossover TID

- timestamp: 2026-03-22T04:42:00Z
  checked: PID 1948271 (v13.41.0) startup sequence at 23:54
  found: |
  "clean_slate: failed to delete today's bars (non-fatal)" at 23:57:07
  Same pattern: stale CH seeds, 11/17 preflight only
  AVAXUSDT preflight SKIPPED ("could not determine crossover trade ID")
  fill_from_rest for AVAXUSDT started from 232228901 (CH seed)
  Sidecar and kintsugi both wrote bars for 2026-03-21 concurrently
  implication: AVAXUSDT overlap is concurrent writer race between sidecar fill_from_rest and kintsugi repair

- timestamp: 2026-03-22T04:45:00Z
  checked: v13.43.1 startup sequence at 04:02
  found: |
  "clean_slate: deleted today's bars for 45 pairs" at 04:07:36 - SUCCESS
  "seeded trade-ID continuity for 45 pairs from ClickHouse" at 04:07:40 - AFTER delete
  "midnight preflight: verified 17/17 symbols, overrode seeds for 68 pairs" - ALL symbols
  implication: v13.43.1 has the fix (fatal delete_today). Anomalies are legacy from v13.40.1/v13.41.0.

- timestamp: 2026-03-22T04:50:00Z
  checked: Current code vs deployed v13.40.1/v13.41.0
  found: |
  Current code (v13.43.1): \_delete_today_bars docstring says "Failure is FATAL (#295 P0-4)"
  Previous versions: treated failure as "non-fatal" (warning + continue)
  This was the exact bug that #295 fixed
  implication: The anomalies are the last artifacts of the pre-#295 bug

- timestamp: 2026-03-22T05:00:00Z
  checked: Anomaly location (2026-03-21 = yesterday)
  found: v13.43.1 delete_today only cleaned 2026-03-22. Anomalies on 2026-03-21 untouched.
  implication: These are in kintsugi territory now. Kintsugi should auto-repair the gaps. Overlaps need kintsugi too.

## Resolution

root_cause: |
All 3 anomalies are PRE-EXISTING artifacts from v13.40.1 and v13.41.0 sidecar sessions that had a BUG:
`_delete_today_bars` failed but was treated as non-fatal. This violated the startup sequence invariant (#294):

1. **LINKUSDT gap (7,376 trades) + XRPUSDT gap (53,683 trades):**
   - \_delete_today_bars FAILED at v13.40.1 startup (20:01:55 on Mar 21)
   - CH seed picked up stale bars from the previous session, inflating seeds above preflight override
   - fill_from_rest started from the inflated seed instead of midnight crossover TID
   - Result: bars from start-of-day to the stale seed point were never written

2. **AVAXUSDT overlaps (tid_delta -40, -184):**
   - \_delete_today_bars FAILED at v13.41.0 startup (23:57:07 on Mar 21)
   - Sidecar fill_from_rest produced bar [232238341,232238564] at 00:12:45
   - Kintsugi repair produced bars [232238381,232238470] and [232238471,232238586] at 00:31:07
   - Both writers operated on 2026-03-21 data concurrently → overlapping bars with different first_agg_trade_id
   - OPTIMIZE FINAL cannot dedup because different first_agg_trade_id = different ORDER BY key

v13.43.1 has the fix (#295): \_delete_today_bars failure is now FATAL. The v13.43.1 startup at 04:02 succeeded cleanly.
These anomalies are on 2026-03-21 (kintsugi territory) and were not cleaned by today's delete.

fix:
verification:
files_changed: []
