# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** — Phases 1-3 (shipped 2026-03-22) — [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** — Phases 1-6 (shipped 2026-03-23) — [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** — Phases 1-3 (shipped 2026-03-23) — [archive](milestones/v1.5-ROADMAP.md)
- ✅ **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** — Phases 1-5 (shipped 2026-03-24) — [archive](milestones/v2.0-ROADMAP.md)
- 🚧 **v3.0 Streaming Reliability + Pipeline Cleanup** — Phases 1-5 (in progress)

## Phases

- [ ] **Phase 1: Mutation Safety** - Kintsugi DELETE uses partition-scoped mutations, eliminating 300s full-table scans under concurrent repair
- [ ] **Phase 2: Sidecar Fill Pipeline** - fill_from_rest returns computed bars directly with zero-trade gap at WS junction
- [ ] **Phase 3: Parquet-First Catchup** - Sidecar startup reads local Parquet tick cache before hitting REST API, reducing startup latency and API load
- [ ] **Phase 4: Combined WebSocket Streams** - Migrate from per-symbol WS connections to Binance combined streams (2x9 hybrid) with resilient reconnection
- [ ] **Phase 5: Dead Code Cleanup** - Remove 111 stale scripts, dead Rust modules, unused dependencies, and stale TODOs identified in adversarial audit (#298)

## Phase Details

### Phase 1: Mutation Safety

**Goal**: Kintsugi repairs complete reliably without mutation timeouts under production concurrency
**Depends on**: Nothing (first phase, P0 bug fix)
**Requirements**: MUT-01, MUT-02
**Success Criteria** (what must be TRUE):

1. Every kintsugi DELETE operation uses `DELETE IN PARTITION` syntax with explicit partition tuple -- no `ALTER TABLE DELETE` without partition scope in kintsugi code paths
2. 24 concurrent kintsugi workers repairing 100+ shards complete a full pass without any mutation exceeding 300s timeout
3. Mutation execution time for single-day shard repair drops from O(all parts) to O(1 partition) -- observable in ClickHouse system.mutations log
   **Plans**: 2 plans

Plans:

- [x] 01-01-PLAN.md -- Per-partition lock + sidecar DELETE migration
- [x] 01-02-PLAN.md -- Secondary DELETE call site audit (rectify, scripts)

### Phase 2: Sidecar Fill Pipeline

**Goal**: Sidecar startup produces zero-gap bar coverage from midnight to WS junction without intermediate trade hand-off
**Depends on**: Phase 1
**Requirements**: STREAM-01, STREAM-02
**Success Criteria** (what must be TRUE):

1. `fill_from_rest` returns a list of computed bars (not raw trades) that are immediately flushed to ClickHouse via `sync_flush`
2. Stathera continuity holds across the REST-fill-to-WS-junction boundary -- `bar[last_rest].last_agg_trade_id + 1 == bar[first_ws].first_agg_trade_id` for every symbol after sidecar startup
3. No intermediate trade DataFrame is materialized in Python between REST fetch and bar computation (trades stay in Rust)
   **Plans**: 2 plans

Plans:

- [x] 02-01-PLAN.md -- Pipeline hardening: dropped_bars diagnostic, Stathera validation, code audit
- [x] 02-02-PLAN.md -- Gap closure: ws_first_tid tracking + full junction telemetry in /status

### Phase 3: Parquet-First Catchup

**Goal**: Sidecar startup leverages the local Parquet tick cache to minimize REST API calls during fill_from_rest
**Depends on**: Phase 2
**Requirements**: PQFIRST-01, PQFIRST-02, PQFIRST-03
**Success Criteria** (what must be TRUE):

1. On startup, sidecar reads `max(agg_trade_id)` from today's Parquet tick cache per symbol before querying REST API
2. REST API is called only for the gap between Parquet's last trade ID and WS buffer start -- not for the full day
3. When Parquet data is stale (>10 min old) or missing, sidecar falls back to current full `fill_from_rest` with no behavior change (no regression)
4. Startup REST API weight consumption measurably decreases when fresh Parquet data exists (observable in rate limit coordinator logs)
   **Plans**: 1 plan

Plans:

- [x] 03-01-PLAN.md -- TDD: \_read_parquet_max_tids + sidecar startup integration

### Phase 4: Combined WebSocket Streams

**Goal**: All symbols stream via Binance combined streams with resilient reconnection maintaining Stathera continuity
**Depends on**: Nothing (independent of Phases 1-3)
**Requirements**: WS-01, WS-02, WS-03
**Success Criteria** (what must be TRUE):

1. Sidecar connects via `/stream?streams=sym1@aggTrade/sym2@aggTrade/...` using 2x9 hybrid architecture (two combined connections, 9 streams each) instead of per-symbol WebSocket connections
2. Combined stream reconnection uses Antaeus exponential backoff with jitter -- no fixed-delay reconnect loops
3. All 26 symbols maintain Stathera continuity (zero gaps, zero overlaps) for 24+ hours after migration to combined streams
4. Total WebSocket connection count drops from 18+ to 2 (observable in sidecar /health endpoint)
   **Plans**: 2 plans

Plans:

- [x] 04-01-PLAN.md -- WsMode feature flag + 2-connection split + per-symbol fallback
- [x] 04-02-PLAN.md -- Full validation + human review checkpoint

### Phase 5: Dead Code Cleanup

**Goal**: Production codebase contains no dead scripts, dead Rust modules, or unreachable code identified in the adversarial audit
**Depends on**: Nothing (independent, can run in parallel with any phase)
**Requirements**: CLEAN-01, CLEAN-02, CLEAN-03
**Success Criteria** (what must be TRUE):

1. All 111 stale scripts identified in audit are removed from `scripts/` directory
2. Dead Rust modules (streaming indicators/stats) and unused Cargo dependencies removed
3. Stale TODOs referencing closed issues cleaned up
4. Every removal is verified unreachable before deletion -- no import references, no dynamic loading, no CLI entry points
5. `mise run check-full` passes after all removals (no broken imports, no test failures)
   **Plans**: 2 plans

Plans:

- [ ] 05-01-PLAN.md -- Remove 111 unreferenced Python scripts from scripts/
- [ ] 05-02-PLAN.md -- Remove dead Rust code, unused deps, stale TODOs

## Backlog

### Phase 05.1: Post-Audit Polish (INSERTED)

**Goal:** Close MISS-01 integration gap (/health ws_mode observability) and fix test helper key mismatch from v3.0 milestone audit
**Requirements**: None (gap closure -- closing MISS-01 and test tech debt)
**Depends on:** Phase 5
**Plans:** 1/1 plans complete

Plans:

- [x] 05.1-01-PLAN.md -- /health ws_mode field + test helper open_time_ms key fix

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 0/2 plans executed

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

## Progress

**Execution Order:** Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5
Phase 4 and Phase 5 are independent and can execute in parallel with earlier phases.

| Phase                         | Milestone | Plans Complete | Status      | Completed |
| ----------------------------- | --------- | -------------- | ----------- | --------- |
| 1. Mutation Safety            | v3.0      | 2/2            | Complete    | -         |
| 2. Sidecar Fill Pipeline      | v3.0      | 2/2            | Complete    | -         |
| 3. Parquet-First Catchup      | v3.0      | 0/1            | In progress | -         |
| 4. Combined WebSocket Streams | v3.0      | 0/2            | Not started | -         |
| 5. Dead Code Cleanup          | v3.0      | 0/2            | Planned     |           |
