# Roadmap: v1.4 Zero-Gap Zero-Overlap Guarantee

## Overview

Three phases that fix the two root causes of Stathera gaps (committed_floors dedup bug, midnight processor seam) and then instrument the system to prove they stay fixed. Phase 1 fixes the Rust dedup floor so bars are never suppressed at the REST-to-WS junction. Phase 2 uses Binance REST to verify the exact midnight crossover trade ID and anchor fill_from_rest from it, closing the multi-processor seam. Phase 3 overhauls heartbeat telemetry to continuously validate zero gaps across every symbol, every day.

## Phases

**Phase Numbering:**

- Integer phases (1, 2, 3): Planned milestone work
- Decimal phases (2.1, 2.2): Urgent insertions (marked with INSERTED)

Decimal phases appear between their surrounding integers in numeric order.

- [ ] **Phase 1: Dedup Floor Fix** - Fix committed_floors to use last completed bar, eliminating intra-day junction gaps
- [ ] **Phase 2: Midnight Preflight Verification** - Query Binance REST for exact midnight crossover trade ID, verify yesterday's orphan bar, anchor today's fill_from_rest
- [ ] **Phase 3: Heartbeat Telemetry Overhaul** - Unified per-symbol Stathera check, yesterday completion, junction telemetry, LOUD regression alerts
- [ ] **Phase 4: WS Resilience Enhancements** - Adopt unicorn-binance-websocket-api patterns: ping/pong keepalive tuning, rate limit enforcement, error classification, IP ban handling, stream state signals, buffer OOM prevention
- [ ] **Phase 5: Seeder REST Integration & Rate Limit Coordination** - Add REST API fallback to seeder for recent days not yet on Vision, unify rate limit coordination across sidecar/kintsugi/seeder via shared AdaptiveRateLimiter
- [ ] **Phase 6: Memory Efficiency Refactoring** - 8 P0 findings from Issue #300 with Oracle bit-exact regression tests

## Phase Details

### Phase 1: Dedup Floor Fix

**Goal**: Bars at the fill_from_rest to WS junction are never silently suppressed by the dedup floor
**Depends on**: Nothing (first phase)
**Requirements**: RUST-01, RUST-02
**Success Criteria** (what must be TRUE):

1. `committed_floors` in `symbol_task()` reads from the last COMPLETED bar's `last_agg_trade_id`, not from `processor.last_agg_trade_id()` which includes the forming bar tail
2. After a fill_from_rest completes and WS trades resume, Stathera audit of today's bars shows zero intra-day gaps (`tid_delta == 0` for all consecutive bar pairs within a single UTC day)
3. No regression in existing bar processing -- cargo nextest and pytest suites pass with zero failures
   **Plans**: 2 plans

Plans:

- [x] 01-01-PLAN.md -- Add last_completed_bar_tid field to processor and checkpoint (RUST-01)
- [x] 01-02-PLAN.md -- Wire into committed_floors, add PyO3 bindings, full regression (RUST-01, RUST-02)

### Phase 2: Midnight Preflight Verification

**Goal**: Every sidecar restart anchors today's bars from a verified midnight crossover trade ID, closing the multi-processor seam between kintsugi (yesterday) and sidecar (today)
**Depends on**: Phase 1 (correct dedup floor needed to verify junction bars are not suppressed after anchoring)
**Requirements**: PREFLIGHT-01, PREFLIGHT-02, PREFLIGHT-03
**Success Criteria** (what must be TRUE):

1. On checkpoint-cleared restart, sidecar queries Binance REST for each streaming symbol's first trade at or after UTC midnight (`startTime=midnight_ms&limit=1`) and obtains the crossover trade ID
2. Sidecar verifies that yesterday's orphan bar in ClickHouse has `last_agg_trade_id == crossover_trade_id - 1`. If verification fails, sidecar logs a WARNING and defers repair to kintsugi (no data corruption, no crash)
3. `fill_from_rest` uses the verified crossover trade ID as its anchor (not ClickHouse seed), and the resulting first bar of today satisfies `yesterday_orphan.last_agg_trade_id + 1 == today_first_bar.first_agg_trade_id`
4. For symbols where yesterday has no orphan bar in ClickHouse (gap or fresh deploy), sidecar falls back to the current fill_from_rest behavior gracefully -- no crash, no hang
   **Plans**: 2 plans

Plans:

- [x] 02-01-PLAN.md -- Add crossover query, orphan verification, and ClickHouse query (PREFLIGHT-01, PREFLIGHT-02)
- [x] 02-02-PLAN.md -- Wire preflight into \_create_engine, override fill_from_rest seeds (PREFLIGHT-03)

### Phase 3: Heartbeat Telemetry Overhaul

**Goal**: Heartbeat continuously validates zero-gap zero-overlap across every symbol with unified Stathera checks and LOUD alerts on any regression
**Depends on**: Phase 2 (telemetry must validate the fixes from Phase 1 and Phase 2 are working in production)
**Requirements**: TELEM-01, TELEM-02, TELEM-03, TELEM-04
**Success Criteria** (what must be TRUE):

1. Heartbeat reports a single unified Stathera continuity result per symbol -- no separate "intra-day" vs "midnight-boundary" classification. A gap is a gap, regardless of where it occurs
2. Heartbeat reports yesterday completion status per symbol: whether yesterday's orphan bar's `last_agg_trade_id` matches `crossover_trade_id - 1` (the verified midnight boundary)
3. After a sidecar restart, heartbeat reports junction telemetry: REST last trade ID, WS first trade ID, forming bar state, and committed_floors value -- sufficient to diagnose any junction issue without SSH
4. If any new Stathera gap appears in today's bars for any symbol, heartbeat sends a LOUD Telegram alert within 5 minutes containing exact (symbol, threshold, tid_delta, trade ID range)
   **Plans**: 2 plans

Plans:

- [x] 03-01-PLAN.md -- Unify Stathera check to per-symbol results, add today-gap LOUD alert (TELEM-01, TELEM-04)
- [x] 03-02-PLAN.md -- Add yesterday completion and junction telemetry checks (TELEM-02, TELEM-03)

### Phase 4: WS Resilience Enhancements

**Goal**: Harden the sidecar's Binance WebSocket layer by adopting battle-tested patterns from unicorn-binance-websocket-api
**Depends on**: Phase 1 (dedup floor needed for correct gap detection during reconnects)
**Plans:** 2 plans
**Success Criteria** (what must be TRUE):

1. Ping/pong keepalive configured at 5s interval / 10s timeout across all combined WS streams
2. Subscribe/unsubscribe operations enforce max 5 msg/sec (3 effective after 2-msg reserve for pings)
3. Heartbeat classifies 429 (fatal rate limit) vs 500 (transient) WebSocket errors distinctly in Telegram alerts
4. IP ban duration extracted from Binance error responses; sidecar sleeps until ban expires rather than retry-storming
5. Stream state signals (CONNECT, FIRST_DATA, DISCONNECT) emitted for consumer API SSE observability
6. Ring buffer with configurable maxlen prevents OOM on slow consumers

Plans:

- [x] 04-01-PLAN.md -- Rust WS protocol hardening: ping/pong keepalive, error variants, ban handling, subscribe rate limiter, stream signals
- [x] 04-02-PLAN.md -- Python integration: stream signal tracking in sidecar health, heartbeat 429 vs 500 error classification

### Phase 5: Seeder REST Integration & Rate Limit Coordination

**Goal**: Seeder can fill recent days via REST API (not just Vision CSV), and all REST consumers (sidecar, kintsugi, seeder) coordinate through a unified rate limit budget to prevent 429s
**Depends on**: Phase 4 (WS resilience provides the stream signal infrastructure that rate limiter telemetry builds on)
**Requirements**: SEEDER-01, SEEDER-02, SEEDER-03, SEEDER-04
**Plans:** 4 plans
**Success Criteria** (what must be TRUE):

1. Seeder attempts REST API fallback for days where Vision CSV is not yet available (typically today and yesterday), using the same `aggTrades` endpoint as kintsugi
2. All three REST consumers (sidecar Puck/fill_from_rest, kintsugi, seeder) share rate limit state via a common mechanism (file-based or Redis), preventing any single consumer from starving others
3. Total REST API weight consumption across all processes stays below 80% of Binance's 6000 weight/min IP-wide budget under normal operation
4. Seeder REST path respects writer ownership model: only seeds Parquet ticks (no ClickHouse writes), only for days kintsugi hasn't yet processed
5. Heartbeat reports aggregate REST API utilization across all consumers

Plans:

- [x] 05-01-PLAN.md -- Parquet trade-ID continuity validation, seeder write_ticks() routing, systemd Conflicts= (SEEDER-04)
- [x] 05-02-PLAN.md -- Cross-process rate limit coordinator, per-process ceiling env vars in systemd (SEEDER-02, SEEDER-03)
- [x] 05-03-PLAN.md -- Seeder REST fallback for yesterday, batch rotation, missing-days cache (SEEDER-01, SEEDER-04)
- [x] 05-04-PLAN.md -- Heartbeat REST utilization telemetry, seeder days-behind metrics (SEEDER-02, SEEDER-03)

### Phase 6: Memory Efficiency Refactoring — P0 Findings from Issue #300

**Goal**: Implement 8 P0 memory efficiency findings from the 9-agent audit (Issue #300) with zero behavior change, proven by Oracle bit-exact regression tests
**Depends on**: Phase 5
**Requirements**: MEM-01, MEM-02, MEM-03, MEM-04, MEM-05, MEM-06, MEM-07, MEM-08
**Plans:** 4 plans
**Success Criteria** (what must be TRUE):

1. AggTrade derives Copy — no clone overhead on hot path trade accumulation
2. Arrow input path uses stack-local AggTrade iteration — no Vec<AggTrade> heap allocation (14-56 MB savings per chunk)
3. GIL released during pure Rust computation — other Python threads can execute during 50-200ms processing
4. Plugin enrichment skips Polars-to-Pandas roundtrip when no plugins installed (common case)
5. get_stats() uses Parquet footer metadata — never materializes file data (99.9% I/O reduction)
6. REST fallback uses Arrow streaming — no 400-600 MB/day list[dict] materialization
7. store_bars_batch() uses single with_columns() call — 5+ fewer intermediate DataFrames per write
8. FormingBar watch updates throttled to 1 Hz — 99.3% fewer OpenDeviationBar clones in streaming
9. ALL oracle regression tests pass — bit-exact output for identical input

Plans:

- [x] 06-01-PLAN.md -- Golden snapshot oracle capture: Rust + Python fixtures for BTCUSDT 2024-01-01 @250 (ALL MEM-\*)
- [x] 06-02-PLAN.md -- Rust core: Copy on AggTrade, Arrow zero-copy iterator, FormingBar 1Hz throttle (MEM-01, MEM-02, MEM-08)
- [x] 06-03-PLAN.md -- PyO3 GIL release in Arrow processing methods (MEM-03)
- [ ] 06-04-PLAN.md -- Python pipeline: skip plugin roundtrip, footer row counting, REST Arrow path, merge with_columns (MEM-04, MEM-05, MEM-06, MEM-07)

## Progress

**Execution Order:**
Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6

| Phase                              | Plans Complete | Status   | Completed  |
| ---------------------------------- | -------------- | -------- | ---------- |
| 1. Dedup Floor Fix                 | 2/2            | Complete | 2026-03-21 |
| 2. Midnight Preflight Verification | 2/2            | Complete | 2026-03-21 |
| 3. Heartbeat Telemetry Overhaul    | 2/2            | Complete | 2026-03-22 |
| 4. WS Resilience Enhancements      | 2/2            | Complete | 2026-03-22 |
| 5. Seeder REST & Rate Coordination | 4/4            | Complete | 2026-03-22 |
| 6. Memory Efficiency Refactoring   | 0/4            | Planned  | -          |
