# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** -- Phases 1-3 (shipped 2026-03-22) -- [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** -- Phases 1-6 (shipped 2026-03-23) -- [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** -- Phases 1-3 (shipped 2026-03-23) -- [archive](milestones/v1.5-ROADMAP.md)
- ✅ **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** -- Phases 1-5 (shipped 2026-03-24) -- [archive](milestones/v2.0-ROADMAP.md)
- ✅ **v3.0 Streaming Reliability + Pipeline Cleanup** -- Phases 1-5.1 (shipped 2026-03-24) -- [archive](milestones/v3.0-ROADMAP.md)
- ✅ **v4.0 Microsecond Column Rename + Deploy Gap Repair** -- Phases 1-5 (shipped 2026-03-25) -- [archive](milestones/v4.0-ROADMAP.md)
- 🚧 **v5.0 Seeder Day-Boundary Completeness + Kintsugi Parquet Trust** -- Phases 1-5 (in progress)

## Phases

- [ ] **Phase 1: Crossover TID Utility** - Extract shared crossover lookup callable by seeder, kintsugi, and heartbeat
- [ ] **Phase 2: Seeder Day-Boundary Backfill** - Seeder backfills yesterday's Parquet file after midnight to crossover TID boundary
- [ ] **Phase 3: Kintsugi Parquet Trust** - Kintsugi verifies Parquet completeness before using as repair source
- [ ] **Phase 4: Heartbeat Coverage** - Yesterday completion check expanded to all streaming symbols
- [ ] **Phase 5: Documentation** - Root cause and fix documented across CLAUDE.md network

## Phase Details

### Phase 1: Crossover TID Utility

**Goal**: Any module can look up the crossover TID for a given symbol and date without importing sidecar internals
**Depends on**: Nothing (first phase)
**Requirements**: XOVER-01, XOVER-02
**Plans:** 1 plan

Plans:

- [x] 01-01-PLAN.md -- Extract crossover TID lookup into shared module, rewire sidecar and heartbeat

**Success Criteria** (what must be TRUE):

1. Seeder, kintsugi, and heartbeat can call a shared `get_crossover_tid(symbol, date)` without importing anything from `sidecar.py`
2. The utility reuses the existing `_get_midnight_crossover_tid` + `_verify_crossover_from_seed` logic (refactored out, not reimplemented)
3. Existing sidecar midnight preflight still works identically after the extraction (no behavioral change)

### Phase 2: Seeder Day-Boundary Backfill

**Goal**: Yesterday's Parquet file contains every trade from 00:00:00.000 through the last trade before midnight crossover
**Depends on**: Phase 1
**Requirements**: SEED-01, SEED-02
**Plans:** 1 plan

Plans:

- [x] 02-01-PLAN.md -- Day-boundary backfill: detect incomplete yesterday Parquet, fetch missing trailing trades via REST

**Success Criteria** (what must be TRUE):

1. After the seeder's post-midnight run, yesterday's Parquet `max(agg_trade_id)` equals `crossover_tid - 1`
2. The seeder detects an incomplete yesterday file and backfills only the missing trailing trades (the ~5 min gap)
3. A complete yesterday file (max TID already at crossover boundary) is not re-fetched or modified

### Phase 3: Kintsugi Parquet Trust

**Goal**: Kintsugi only uses Parquet as a repair source when it can prove the file covers the full day
**Depends on**: Phase 1
**Requirements**: TRUST-01, TRUST-02
**Plans:** 1 plan

**Success Criteria** (what must be TRUE):

1. Before day-recompute, kintsugi verifies `max(agg_trade_id) >= crossover_tid - 1` on the Parquet file
2. If Parquet verification fails, kintsugi falls back to Vision/REST instead of using incomplete data
3. Orphan bars emitted from Parquet and Vision repair paths have `is_orphan=True` set correctly

Plans:

- [x] 03-01-PLAN.md -- Parquet completeness gate + orphan bar marking in Parquet/Vision repair paths

### Phase 4: Heartbeat Coverage

**Goal**: Yesterday completion monitoring covers all streaming symbols, not just a hardcoded subset
**Depends on**: Phase 1
**Requirements**: HB-01
**Plans:** 1 plan

Plans:

- [x] 04-01-PLAN.md -- Dynamic symbol list from registry for yesterday completion check

**Success Criteria** (what must be TRUE):

1. Yesterday completion check runs for every symbol in the streaming config (all 17+ Binance symbols)
2. An incomplete yesterday for any symbol triggers the existing LOUD Telegram alert

### Phase 5: Documentation

**Goal**: Root cause of the day-boundary gap and the v5.0 fix are documented across the CLAUDE.md network
**Depends on**: Phase 2, Phase 3, Phase 4
**Requirements**: DOC-01
**Success Criteria** (what must be TRUE):

1. `scripts/CLAUDE.md` documents seeder day-boundary semantics and the post-midnight backfill mechanism
2. `python/opendeviationbar/kintsugi/CLAUDE.md` documents the Parquet trust verification gate
3. `scripts/heartbeat/CLAUDE.md` documents the expanded symbol coverage (no more hardcoded pair)
4. Root `CLAUDE.md` references the day-boundary completeness guarantee in the monitoring table
   **Plans:** 1 plan

Plans:

- [x] 05-01-PLAN.md -- Update CLAUDE.md network with v5.0 day-boundary completeness docs

## Backlog

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 0/2 plans executed

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

## Progress

**Execution Order:** Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5
Phase 1 is the foundation. Phases 2-4 depend on Phase 1 (could parallelize but sequential is safer). Phase 5 depends on 2-4.

| Phase                           | Milestone | Plans Complete | Status      | Completed |
| ------------------------------- | --------- | -------------- | ----------- | --------- |
| 1. Crossover TID Utility        | v5.0      | 0/1            | Not started | -         |
| 2. Seeder Day-Boundary Backfill | v5.0      | 0/1            | Not started | -         |
| 3. Kintsugi Parquet Trust       | v5.0      | 0/1            | Not started | -         |
| 4. Heartbeat Coverage           | v5.0      | 0/1            | Not started | -         |
| 5. Documentation                | v5.0      | 0/1            | Not started | -         |
