# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** -- Phases 1-3 (shipped 2026-03-22) -- [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** -- Phases 1-6 (shipped 2026-03-23) -- [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** -- Phases 1-3 (shipped 2026-03-23) -- [archive](milestones/v1.5-ROADMAP.md)
- ✅ **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** -- Phases 1-5 (shipped 2026-03-24) -- [archive](milestones/v2.0-ROADMAP.md)
- ✅ **v3.0 Streaming Reliability + Pipeline Cleanup** -- Phases 1-5.1 (shipped 2026-03-24) -- [archive](milestones/v3.0-ROADMAP.md)
- ✅ **v4.0 Microsecond Column Rename + Deploy Gap Repair** -- Phases 1-5 (shipped 2026-03-25) -- [archive](milestones/v4.0-ROADMAP.md)
- ✅ **v5.0 Seeder Day-Boundary Completeness + Kintsugi Parquet Trust** -- Phases 1-5 (shipped 2026-03-25, v13.54.0)
- ✅ **v5.1 SEED-001 Fix + Heartbeat Unified View** -- Phases 6-7 (shipped 2026-03-25, v13.55.0)
- 🚧 **v6.0 Data Integrity & Checksum Verification** -- Phases 8-12 (in progress)

## Phases

<details>
<summary>✅ v5.0 Seeder Day-Boundary Completeness (Phases 1-5) - SHIPPED 2026-03-25</summary>

- [x] **Phase 1: Crossover TID Utility** - Extract shared crossover lookup callable by seeder, kintsugi, and heartbeat
- [x] **Phase 2: Seeder Day-Boundary Backfill** - Seeder backfills yesterday's Parquet file after midnight to crossover TID boundary
- [x] **Phase 3: Kintsugi Parquet Trust** - Kintsugi verifies Parquet completeness before using as repair source
- [x] **Phase 4: Heartbeat Coverage** - Yesterday completion check expanded to all streaming symbols
- [x] **Phase 5: Documentation** - Root cause and fix documented across CLAUDE.md network

</details>

<details>
<summary>✅ v5.1 SEED-001 Fix + Heartbeat Unified View (Phases 6-7) - SHIPPED 2026-03-25</summary>

- [x] **Phase 6: Fix SEED-001 Parquet-First Bar Processing** - Process Parquet ticks into bars instead of skipping past them
- [x] **Phase 7: Heartbeat Unified Per-Symbol View** - Merge Yesterday + Junction into exception-only display with TID deltas

</details>

### v6.0 Data Integrity & Checksum Verification

- [ ] **Phase 8: Checksum Spike** - Prove Rust checksum.rs SHA-256 is callable from Python with correct results, acceptable perf, and edge case handling
- [ ] **Phase 9: Verification Registry** - Centralized per-file verification state store as SSoT for seeder tracking and sweep
- [ ] **Phase 10: Seeder Checksum Integration** - Wire SHA-256 verification into tick_cache_seeder download pipeline with retry and audit trail
- [ ] **Phase 11: Integrity Sweep** - Audit existing Parquet cache against Vision CHECKSUM files with incremental mode
- [ ] **Phase 12: Telemetry & Heartbeat** - Heartbeat reports checksum status per symbol, LOUD alerts on failures, local telemetry log

## Phase Details

<details>
<summary>✅ v5.0 Phase Details (Phases 1-5)</summary>

### Phase 1: Crossover TID Utility

**Goal**: Any module can look up the crossover TID for a given symbol and date without importing sidecar internals
**Depends on**: Nothing (first phase)
**Requirements**: XOVER-01, XOVER-02
**Plans:** 1 plan

Plans:

- [x] 01-01-PLAN.md -- Extract crossover TID lookup into shared module, rewire sidecar and heartbeat

**Success Criteria** (what must be TRUE):

1. Seeder, kintsugi, and heartbeat can call a shared `get_crossover_tid(symbol, date)` without importing anything from `sidecar.py`
2. The utility reuses the existing `_get_midnight_crossover_tid` + `_verify_crossover_from_seed` logic (refactored out, not reimplemented)
3. Existing sidecar midnight preflight still works identically after the extraction (no behavioral change)

### Phase 2: Seeder Day-Boundary Backfill

**Goal**: Yesterday's Parquet file contains every trade from 00:00:00.000 through the last trade before midnight crossover
**Depends on**: Phase 1
**Requirements**: SEED-01, SEED-02
**Plans:** 1 plan

Plans:

- [x] 02-01-PLAN.md -- Day-boundary backfill: detect incomplete yesterday Parquet, fetch missing trailing trades via REST

**Success Criteria** (what must be TRUE):

1. After the seeder's post-midnight run, yesterday's Parquet `max(agg_trade_id)` equals `crossover_tid - 1`
2. The seeder detects an incomplete yesterday file and backfills only the missing trailing trades (the ~5 min gap)
3. A complete yesterday file (max TID already at crossover boundary) is not re-fetched or modified

### Phase 3: Kintsugi Parquet Trust

**Goal**: Kintsugi only uses Parquet as a repair source when it can prove the file covers the full day
**Depends on**: Phase 1
**Requirements**: TRUST-01, TRUST-02
**Plans:** 1 plan

Plans:

- [x] 03-01-PLAN.md -- Parquet completeness gate + orphan bar marking in Parquet/Vision repair paths

**Success Criteria** (what must be TRUE):

1. Before day-recompute, kintsugi verifies `max(agg_trade_id) >= crossover_tid - 1` on the Parquet file
2. If Parquet verification fails, kintsugi falls back to Vision/REST instead of using incomplete data
3. Orphan bars emitted from Parquet and Vision repair paths have `is_orphan=True` set correctly

### Phase 4: Heartbeat Coverage

**Goal**: Yesterday completion monitoring covers all streaming symbols, not just a hardcoded subset
**Depends on**: Phase 1
**Requirements**: HB-01
**Plans:** 1 plan

Plans:

- [x] 04-01-PLAN.md -- Dynamic symbol list from registry for yesterday completion check

**Success Criteria** (what must be TRUE):

1. Yesterday completion check runs for every symbol in the streaming config (all 17+ Binance symbols)
2. An incomplete yesterday for any symbol triggers the existing LOUD Telegram alert

### Phase 5: Documentation

**Goal**: Root cause of the day-boundary gap and the v5.0 fix are documented across the CLAUDE.md network
**Depends on**: Phase 2, Phase 3, Phase 4
**Requirements**: DOC-01

Plans:

- [x] 05-01-PLAN.md -- Update CLAUDE.md network with v5.0 day-boundary completeness docs

**Success Criteria** (what must be TRUE):

1. `scripts/CLAUDE.md` documents seeder day-boundary semantics and the post-midnight backfill mechanism
2. `python/opendeviationbar/kintsugi/CLAUDE.md` documents the Parquet trust verification gate
3. `scripts/heartbeat/CLAUDE.md` documents the expanded symbol coverage (no more hardcoded pair)
4. Root `CLAUDE.md` references the day-boundary completeness guarantee in the monitoring table

</details>

<details>
<summary>✅ v5.1 Phase Details (Phases 6-7)</summary>

### Phase 6: Fix SEED-001 Parquet-First Bar Processing

**Goal:** On sidecar restart, process Parquet ticks into bars via Rust processor instead of skipping past them. Eliminates 274K+ trade gaps caused by SEED-001 advancing the REST seed past deleted bars.
**Requirements**: SEED-FIX-01, SEED-FIX-02
**Depends on:** Phase 5 (v5.0 shipped)
**Plans:** 1 plan

Plans:

- [x] 06-01-PLAN.md -- Process Parquet ticks into bars and seed with last_completed_bar_tid

### Phase 7: Heartbeat Unified Per-Symbol View

**Goal:** Merge Yesterday + Junction sections into a single exception-only display covering all registry symbols. Yesterday expands from BTC/ETH to all enabled Binance spot symbols (from `symbols.toml`). Junction replaces raw TID dumps with delta-only (0=clean). When all symbols are clean, show one summary line. Only expand individual symbols when issues exist.
**Requirements**: HB-VIEW-01, HB-VIEW-02
**Depends on:** Phase 6 (v5.1)
**Plans:** 1 plan

Plans:

- [x] 07-01-PLAN.md -- Merge Yesterday + Junction into unified exception-only Per-Symbol Health section

</details>

### Phase 8: Checksum Spike

**Goal**: Rust checksum.rs SHA-256 fetch+verify is proven callable from Python with correct results, acceptable performance, and graceful edge case handling
**Depends on**: Phase 7 (v5.1 shipped)
**Requirements**: SPIKE-01, SPIKE-02, SPIKE-03
**Success Criteria** (what must be TRUE):

1. A Python script can call the Rust SHA-256 checksum function and verify a real Vision ZIP download against its .CHECKSUM file, producing a pass/fail result
2. Checksum verification adds less than 10% wall-clock overhead to a typical Vision ZIP download-and-extract cycle
3. A 404 CHECKSUM response (old data) results in a soft-skip (not a crash), and a corrupted download results in a hard fail with a clear error message
4. Network timeout during CHECKSUM fetch results in a soft-skip with a warning, not a pipeline halt
   **Plans:** 1/1 plans complete

Plans:

- [x] 08-01-PLAN.md — PyO3 checksum bindings + offline tests + network spike validation

### Phase 9: Verification Registry

**Goal**: A centralized store tracks per-file checksum verification state as the single source of truth, queryable by seeder, sweep, and heartbeat
**Depends on**: Phase 8
**Requirements**: TELEM-03
**Success Criteria** (what must be TRUE):

1. The registry stores per-file records (symbol, date, sha256, verified_at, source) and can be queried by symbol or date range
2. The seeder can write a verification record after each successful checksum, and the sweep can read existing records to skip already-verified files
3. The registry persists across process restarts (file-based JSON or SQLite, not in-memory)
   **Plans:** 1 plan

Plans:

- [x] 09-01: Verification registry -- data model, read/write API, persistence

### Phase 10: Seeder Checksum Integration

**Goal**: Every Vision ZIP downloaded by tick_cache_seeder is SHA-256 verified against the .CHECKSUM file before being accepted into the Parquet cache
**Depends on**: Phase 8, Phase 9
**Requirements**: SEEDER-01, SEEDER-02, SEEDER-03
**Success Criteria** (what must be TRUE):

1. Every Vision ZIP download is automatically verified against the .CHECKSUM file before the seeder processes it into Parquet
2. A checksum mismatch triggers up to 2 re-downloads; if all retries fail, a LOUD Telegram alert fires with symbol, date, and hash details
3. Each successful verification is recorded in the verification registry with symbol, date, checksum, and timestamp
4. The seeder's Telegram summary report includes checksum pass/fail/skip counts
   **Plans:** 1/1 plans complete

Plans:

- [x] 10-01: Wire checksum verification into seeder download pipeline with retry, registry write, and Telegram reporting

### Phase 11: Integrity Sweep

**Goal**: A script can audit all existing Parquet cache files against Vision CHECKSUM files, with incremental mode to avoid redundant re-verification
**Depends on**: Phase 9
**Requirements**: SWEEP-01, SWEEP-02, SWEEP-03
**Success Criteria** (what must be TRUE):

1. Running the sweep script re-downloads Vision CHECKSUM files and compares SHA-256 hashes against the original ZIPs used to create each Parquet file
2. Incremental mode reads the verification registry and skips files that are already verified, only checking new or unverified files
3. The sweep produces a summary report with pass/fail/skip counts broken down by symbol and date range
4. A failed file is flagged in the registry and optionally triggers a Telegram alert
   **Plans:** 1/1 plans complete

Plans:

- [x] 11-01: Integrity sweep script -- full and incremental modes, registry integration, summary report

### Phase 12: Telemetry & Heartbeat

**Goal**: Heartbeat reports checksum verification health per symbol with LOUD Telegram alerts on any integrity failure, and all checksum operations are logged locally for post-mortem
**Depends on**: Phase 9, Phase 10
**Requirements**: TELEM-01, TELEM-02, TELEM-04
**Success Criteria** (what must be TRUE):

1. The 5-minute heartbeat includes a checksum health section showing last verified date and any failures per symbol
2. Any checksum failure (seeder or sweep) triggers a LOUD Telegram alert with symbol, date, expected hash, and actual hash
3. All checksum operations (verify, skip, fail, retry) are written to a local telemetry log file for post-mortem analysis
4. When all symbols have clean checksums, the heartbeat shows a single summary line (exception-only pattern from Phase 7)
   **Plans:** 1/1 plans complete

Plans:

- [x] 12-01: Heartbeat checksum health check + LOUD alert + local telemetry log

## Backlog

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 0/2 plans executed

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

## Progress

**Execution Order:** Phases execute in numeric order: 8 -> 9 -> 10 -> 11 -> 12
Phase 8 is the spike (must complete first). Phase 9 is the registry foundation. Phase 10 depends on 8+9. Phase 11 depends on 9. Phase 12 depends on 9+10.

| Phase                                    | Milestone | Plans Complete | Status      | Completed  |
| ---------------------------------------- | --------- | -------------- | ----------- | ---------- |
| 1. Crossover TID Utility                 | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 2. Seeder Day-Boundary Backfill          | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 3. Kintsugi Parquet Trust                | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 4. Heartbeat Coverage                    | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 5. Documentation                         | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 6. Fix SEED-001 Parquet-First Processing | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 7. Heartbeat Unified Per-Symbol View     | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 8. Checksum Spike                        | v6.0      | 1/1 | Complete   | 2026-03-26 |
| 9. Verification Registry                 | v6.0      | 0/1            | Not started | -          |
| 10. Seeder Checksum Integration          | v6.0      | 1/1 | Complete    | 2026-03-26 |
| 11. Integrity Sweep                      | v6.0      | 1/1 | Complete    | 2026-03-26 |
| 12. Telemetry & Heartbeat                | v6.0      | 1/1 | Complete    | 2026-03-26 |
