# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** — Phases 1-3 (shipped 2026-03-22) — [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** — Phases 1-6 (shipped 2026-03-23) — [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** — Phases 1-3 (shipped 2026-03-23) — [archive](milestones/v1.5-ROADMAP.md)
- 🚧 **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** — Phases 1-5 (in progress)

## Phases

- [x] **Phase 1: Pre-Migration Prerequisites** - Fix forex naming, reallocate rate limits, audit Parquet coverage
- [x] **Phase 2: Kintsugi Parquet-Only Migration** - Strip REST/Vision fallback, add cache-miss deferral, always 64 workers
- [x] **Phase 3: Kintsugi OOM Hardening** - Adopt repair_direct_parquet.py memory discipline for 64-worker operation
- [x] **Phase 4: Microsecond Data Layer** - Rust processor + Parquet seeder + reseed to native microsecond timestamps
- [ ] **Phase 5: Microsecond ClickHouse Rebuild** - CH schema migration, Python alignment, full 35M+ bar rebuild

## Phase Details

### Phase 1: Pre-Migration Prerequisites

**Goal**: All infrastructure preconditions satisfied so kintsugi can safely become Parquet-only
**Depends on**: Nothing (first phase)
**Requirements**: INFRA-01, INFRA-02, INFRA-03
**Success Criteria** (what must be TRUE):

1. Kintsugi filters out non-Binance (forex/Exness) symbols entirely — only crypto symbols enter the repair queue
2. Rate limit budget reflects seeder as sole REST owner (seeder ceiling raised, kintsugi ceiling removed)
3. All 18+ enabled symbols have verified Parquet tick cache coverage with no date gaps that would block repair
4. Any coverage gaps identified in the audit have been backfilled by a seeder sprint before proceeding
   **Plans**: TBD

### Phase 2: Kintsugi Parquet-Only Migration

**Goal**: Kintsugi reads exclusively from the Parquet tick cache -- no REST API calls, no Vision downloads
**Depends on**: Phase 1
**Requirements**: KPAQ-01, KPAQ-02, KPAQ-03
**Success Criteria** (what must be TRUE):

1. `_repair_under_lock()` has no REST or Vision code paths -- Parquet is the only data source
2. Cache-miss shards return `deferred_cache_miss` sentinel (-2) and do NOT write to `kintsugi_log` (no 30-day dismiss window)
3. Kintsugi always launches 64 Parquet workers -- no dynamic 24-worker network path exists
4. Deferred shards re-attempt on next pass after seeder fills the gap (verified by observing a deferred shard succeed after seeder run)
5. No Binance REST 400/404 errors from kintsugi in production logs
   **Plans**: TBD

### Phase 3: Kintsugi OOM Hardening

**Goal**: Kintsugi can sustain 64 concurrent workers without OOM under production workloads
**Depends on**: Phase 2
**Requirements**: KPAQ-04
**Success Criteria** (what must be TRUE):

1. Kintsugi uses `process_trades_arrow()` zero-copy path (not pandas DataFrame conversion)
2. mimalloc allocator purge runs between repair cycles to prevent RSS creep
3. RSS stays below systemd MemoryMax (4 GB) during sustained 64-worker repair passes with 100+ shards
   **Plans**: TBD

### Phase 4: Microsecond Data Layer

**Goal**: All tick data and bar timestamps use native microsecond precision from Rust through Parquet
**Depends on**: Phase 3
**Requirements**: USEC-01, USEC-02, USEC-04
**Success Criteria** (what must be TRUE):

1. Seeder writes 16-digit microsecond timestamps to Parquet for all post-2025-01-01 data (no ms truncation)
2. Pre-2025 Parquet files are reseeded with ms-to-us upscaled timestamps (uniform 16-digit across all dates)
3. Rust `OpenDeviationBarProcessor` accepts and outputs microsecond timestamps throughout (no internal ms conversion)
4. Bars produced from reseeded Parquet data pass Stathera continuity validation
   **Plans**: 2 plans

Plans:

- [x] 04-01-PLAN.md — TickStorage us-native storage + seeder + pipeline callers
- [ ] 04-02-PLAN.md — Reseed pre-2025 Parquet files + Stathera verification

### Phase 5: Microsecond ClickHouse Rebuild

**Goal**: ClickHouse schema and all 35M+ bars store microsecond timestamps, Python code aligned
**Depends on**: Phase 4
**Requirements**: USEC-03, USEC-05, USEC-06
**Success Criteria** (what must be TRUE):

1. ClickHouse timestamp columns store microsecond values with updated COMMENTs documenting the unit change
2. All Python timestamp arithmetic uses microseconds (DAY_US, HOUR_US, etc.) -- no ms constants remain in active code paths
3. Sidecar, kintsugi, and heartbeat operate correctly with microsecond timestamps end-to-end
4. Full ClickHouse rebuild completes via repair_direct_parquet.py with 48 workers, all bars have microsecond precision
5. Stathera continuity validated across the entire rebuilt dataset (zero gaps, zero overlaps)
   **Plans**: 4 plans

Plans:

- [x] 05-01-PLAN.md — Rust dict output + Python contracts + tests migration to \_us
- [x] 05-02-PLAN.md — ClickHouse schema DDL + CH Python layer + repair_direct_parquet.py
- [x] 05-03-PLAN.md — Sidecar + kintsugi + heartbeat + production scripts migration
- [ ] 05-04-PLAN.md — Full 35M+ bar rebuild + Stathera validation + docs update

## Backlog

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 0 plans

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

## Progress

**Execution Order:** Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5

| Phase                              | Milestone | Plans Complete | Status      | Completed  |
| ---------------------------------- | --------- | -------------- | ----------- | ---------- |
| 1. Pre-Migration Prerequisites     | v2.0      | 3/3            | Complete    | 2026-03-24 |
| 2. Kintsugi Parquet-Only Migration | v2.0      | 3/3            | Complete    | 2026-03-24 |
| 3. Kintsugi OOM Hardening          | v2.0      | 4/4            | Complete    | 2026-03-24 |
| 4. Microsecond Data Layer          | v2.0      | 1/2            | In progress | -          |
| 5. Microsecond ClickHouse Rebuild  | v2.0      | 0/4            | Not started | -          |
