---
status: investigating
trigger: "fix REST->WS junction overlaps/gaps at sidecar restart — 4 gaps + 6 overlaps at the fill_from_rest->WS boundary"
created: 2026-03-21T00:00:00Z
updated: 2026-03-21T00:00:00Z
---

## Current Focus

hypothesis: puck_fill bars emitted during junction fill bypass committed_floors dedup and are pushed to ring buffer, but sync_flush already wrote the REST bars to CH — so when Python main loop picks up junction-fill bars they overlap with already-committed REST bars in CH
test: trace committed_floors initialization in symbol_task vs bars emitted by puck_fill during junction fill
expecting: junction fill bars go to ring buffer WITHOUT committed_floors check, creating overlaps with sync_flushed REST bars
next_action: compare puck_fill bar emission (no committed_floors) vs process_trade_through_processors (has committed_floors)

## Symptoms

expected: After sidecar restart, fill_from_rest produces bars that seamlessly connect to the WS-produced bars, with zero Stathera gaps or overlaps at the junction.
actual: 4 gaps + 6 overlaps appear at the fill_from_rest->WS boundary. sync_flush bars written to ClickHouse from the REST fill overlap with bars produced by the WS stream after it starts. committed_floors doesn't prevent the overlap. Puck gap-fill doesn't bridge the transition.
errors: Stathera violations detected by heartbeat — overlapping trade-ID ranges between REST-filled bars and WS bars.
reproduction: Restart sidecar on bigblack. After restart, fill_from_rest runs, then WS starts producing bars. The junction between these two sources has overlaps/gaps.
started: Persistent issue. v1.4 fixed committed_floors initialization and added midnight preflight, but the REST->WS junction problem remains.

## Eliminated

## Evidence

- timestamp: 2026-03-21T00:10:00Z
  checked: fill_from_rest bar emission path (live_engine.rs:591-604)
  found: Bars pushed to ring buffer without committed_floors check. No dedup at all during fill_from_rest.
  implication: REST bars go directly to ring buffer, then sync_flush writes them to CH. This is correct — no dedup needed here since these are the first bars.

- timestamp: 2026-03-21T00:12:00Z
  checked: committed_floors initialization in symbol_task (live_engine.rs:1387-1396)
  found: Floor initialized from max(processor.last_completed_bar_tid, ch_seed_tid). After fill_from_rest, processors have accumulated state. last_completed_bar_tid reflects bars completed during REST fill. ch_seed_tid is the pre-delete seed.
  implication: The floor SHOULD be set to the last REST-completed bar's TID. But ch_seed_tid might still be the old pre-delete value if seeding happened before fill_from_rest.

- timestamp: 2026-03-21T00:15:00Z
  checked: puck_fill bar emission path (live_engine.rs:1249-1278)
  found: puck_fill does NOT check committed_floors. Bars emitted by puck_fill go straight to ring buffer without dedup.
  implication: Junction fill bars produced by puck_fill at the REST->WS boundary will be pushed to ring buffer even if they overlap with bars already written to CH by sync_flush. This is ROOT CAUSE #1 for overlaps.

- timestamp: 2026-03-21T00:18:00Z
  checked: sync_flush writes REST bars to CH (sidecar.py:838-845) using store_bars_replacing
  found: sync_flush uses store_bars_replacing which does synchronous DELETE-then-INSERT for the pair. This deletes ALL today's bars for the pair and re-inserts the REST bars.
  implication: After sync_flush, CH has only REST bars for today. Then engine.start() spawns symbol_task.

- timestamp: 2026-03-21T00:20:00Z
  checked: symbol_task junction fill (live_engine.rs:1401-1502)
  found: Junction fill calls puck_fill to bridge REST last TID to first WS TID. puck_fill pushes bars to ring buffer. These bars then get picked up by Python main loop and written to CH. But these junction bars OVERLAP with REST bars because the REST processor already processed those trades.
  implication: WAIT — puck_fill uses the SAME processors that were used by fill_from_rest. The processors track last_agg_trade_id. The junction fill fetches trades from rest_last_tid+1 to first_ws_tid. These are NEW trades not yet processed. So junction fill bars should NOT overlap with REST bars. Need to re-examine.

- timestamp: 2026-03-21T00:25:00Z
  checked: committed_floors vs ring buffer during junction fill
  found: Junction fill bars are pushed to ring buffer by puck_fill (no committed_floors check). Then process_trade_through_processors for the first WS trade DOES check committed_floors. But the junction fill bars in the ring buffer will be picked up by Python's main loop and written to CH via the normal batch path — which uses store_bars_batch (INSERT), NOT store_bars_replacing (DELETE+INSERT).
  implication: The junction fill bars are written via INSERT. If they share the same ORDER BY key as existing bars, RMT dedup handles it. But if they have DIFFERENT first_agg_trade_id, they create overlapping ranges.

- timestamp: 2026-03-21T00:30:00Z
  checked: Ring buffer consumption after engine.start()
  found: After engine.start(), symbol_task runs junction fill which pushes bars to ring buffer. These bars are mixed with WS-produced bars. Python main loop drains ring buffer via next_bar() and writes ALL bars to CH via store_bars_batch. But sync_flush already wrote REST bars to CH. Junction fill bars cover trades BETWEEN REST and WS — these should be fine. The OVERLAP problem is: WS trades that arrived during fill_from_rest are ALSO processed by symbol_task.
  implication: Need to trace exactly what happens with WS buffered trades.

- timestamp: 2026-03-21T00:35:00Z
  checked: WS buffer at junction (live_engine.rs:1406-1416)
  found: symbol_task receives first WS trade, then drains all immediately-available trades into ws_buffer. These WS trades were buffered since start_ws() was called (step 1 of startup). fill_from_rest runs between start_ws() and start(). So the WS buffer contains ALL trades since engine creation.
  implication: The first WS trade's TID could be BEFORE the REST fill's last TID if the WS connection started buffering before REST fill caught up.

- timestamp: 2026-03-21T00:40:00Z
  checked: JUNC-04 monotonicity guard (live_engine.rs:1479-1501)
  found: After junction fill, first_ws_trade is only processed if its TID > current_last_tid (processor's last_agg_trade_id). WS buffer trades are also processed through gap detector + process_trade_through_processors. BUT process_trade_through_processors checks committed_floors. The floor was initialized from max(last_completed_bar_tid, ch_seed_tid).
  implication: committed_floors should prevent duplicate bar emission for WS trades already covered by REST fill. But the floor is initialized BEFORE junction fill. After junction fill completes, new bars are emitted via puck_fill to the ring buffer. committed_floors is NOT updated by puck_fill — only by process_trade_through_processors. So after junction fill, the floor is still at the REST-completed level, not the junction-fill level.

- timestamp: 2026-03-21T00:45:00Z
  checked: KEY FINDING — puck_fill does not update committed_floors
  found: puck_fill (live_engine.rs:1191-1308) pushes bars to ring buffer but never updates committed_floors. process_trade_through_processors (line 1092, 1128) updates committed_floors after each bar emission. Junction fill uses puck_fill, which emits bars WITHOUT updating committed_floors. Then WS trades are processed via process_trade_through_processors which checks stale committed_floors.
  implication: If puck_fill junction fill produces bars, they go to ring buffer. committed_floors stays at REST level. Then WS trades arrive that were already processed by puck_fill → process_trade_through_processors produces the SAME bars again (since processor state advanced, it won't re-produce, but the gap detector may re-fire puck on overlapping WS trades). Actually wait — the processor is the SAME instance. After puck_fill processes trades through it, the processor's state has advanced. So WS trades with lower TIDs won't produce new bars — the processor has already seen them. The monotonicity guard (JUNC-04) handles the first WS trade. But what about ws_buffer trades?

- timestamp: 2026-03-21T00:50:00Z
  checked: ws_buffer processing (live_engine.rs:1504-1523)
  found: ws_buffer trades are processed with gap detector check + process_trade_through_processors. The gap detector was re-seeded at JUNC-03 to first_ws_trade.agg_trade_id. So gaps in ws_buffer trades (vs first_ws_trade) will trigger puck_fill again. Process_trade_through_processors will skip trades already in the processor (they won't produce bars since processor already processed them during junction fill). BUT — the processor may have advanced PAST some ws_buffer trades during junction fill.
  implication: The processor state is: last_agg_trade_id = max(rest_fill, junction_fill). WS buffer trades with TID < this are processed through processor but don't produce new bars (processor has already seen them or they're out of order). The real question is: are bars from sync_flush (REST bars in CH) overlapping with bars from the main loop (junction fill + WS bars)?

## Resolution

root_cause: TWO root causes identified:

1. **puck_fill bars bypass committed_floors** (live_engine.rs:1249-1278): puck_fill pushes bars to ring buffer without checking or updating committed_floors. When junction fill runs and produces bars, those bars are emitted to ring buffer. Python main loop picks them up and writes them to CH. But sync_flush already wrote REST bars to CH. If junction fill's puck_fill processes trades that overlap with the last REST fill chunk (due to parallel REST fetch boundaries), bars can overlap.

2. **sync_flush uses store_bars_replacing (DELETE+INSERT) but main loop uses store_bars_batch (INSERT only)**: sync_flush atomically replaces today's bars with REST bars. Then junction fill + WS bars are INSERTed via the main loop's store_bars_batch. If junction fill produces bars covering the same trade range as the tail end of REST bars, those bars get INSERTed alongside existing REST bars. RMT dedup only works if ORDER BY keys match exactly. If the bar boundaries differ (different first_agg_trade_id), both bars survive → overlap.

fix:
verification:
files_changed: []
