# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** -- Phases 1-3 (shipped 2026-03-22) -- [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** -- Phases 1-6 (shipped 2026-03-23) -- [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** -- Phases 1-3 (shipped 2026-03-23) -- [archive](milestones/v1.5-ROADMAP.md)
- ✅ **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** -- Phases 1-5 (shipped 2026-03-24) -- [archive](milestones/v2.0-ROADMAP.md)
- ✅ **v3.0 Streaming Reliability + Pipeline Cleanup** -- Phases 1-5.1 (shipped 2026-03-24) -- [archive](milestones/v3.0-ROADMAP.md)
- 🚧 **v4.0 Microsecond Column Rename + Deploy Gap Repair** -- Phases 1-5 (in progress)

## Phases

- [ ] **Phase 1: Test Hygiene** - Remove dead test file and fix schema validation so check-full baseline is clean before bulk renames
- [ ] **Phase 2: Rust Dict Export Rename** - Rust helpers.rs emits `_us` keys instead of `_ms`, since Rust produces the dict keys that Python consumes
- [ ] **Phase 3: ClickHouse Schema Migration** - Recreate open_deviation_bars table with `close_time_us`/`open_time_us` columns, zero data loss
- [ ] **Phase 4: Python + SQL Rename** - Update all Python (~232) and SQL (~60) references from `_ms` to `_us` with full regression validation
- [ ] **Phase 5: Deploy Gap Repair** - Fix backfill_gap() trailing edge and integrate cross-midnight gap repair into odb-ship deploy pipeline

## Phase Details

### Phase 1: Test Hygiene

**Goal**: Test suite is clean -- no failing tests from stale code references, no tests for removed functionality
**Depends on**: Nothing (first phase, independent quick wins)
**Requirements**: TEST-01, TEST-02
**Success Criteria** (what must be TRUE):

1. `test_data_contracts.py` passes -- schema validation matches current Rust output
2. `test_heal_scoping.py` no longer exists in the test suite (tests removed `_check_and_heal_overlaps` code)
3. `pytest` runs with zero unexpected failures from these two test files
   **Plans**: 1 plan

Plans:

- [x] 01-01-PLAN.md -- Fix BarOutputSchema \_ms to_us naming and delete dead test file

### Phase 2: Rust Dict Export Rename

**Goal**: Rust-side bar dict export uses `_us` column names, establishing the source-of-truth naming that Python and SQL will follow
**Depends on**: Phase 1
**Requirements**: RENAME-03
**Success Criteria** (what must be TRUE):

1. `helpers.rs` dict export emits keys `close_time_us` and `open_time_us` instead of `close_time_ms` and `open_time_ms`
2. `cargo test` passes with updated key names (Rust-side tests reference new names)
3. Arrow schema exported by PyO3 bindings uses `close_time_us`/`open_time_us` field names
   **Plans**: 1 plan

Plans:

- [x] 02-01-PLAN.md -- Rename dict export keys from \_ms to_us in helpers.rs

### Phase 3: ClickHouse Schema Migration

**Goal**: ClickHouse `open_deviation_bars` table uses `close_time_us`/`open_time_us` column names with zero data loss
**Depends on**: Phase 2 (Rust must emit new names before CH schema changes, so inserts use correct columns)
**Requirements**: RENAME-01
**Success Criteria** (what must be TRUE):

1. `DESCRIBE open_deviation_bars` shows `close_time_us` and `open_time_us` -- no `_ms` columns remain
2. Row count before and after migration is identical (zero data loss verified by `SELECT count()`)
3. Stathera continuity audit passes on migrated data (no gaps introduced by migration)
4. `ORDER BY` key includes the renamed columns (table recreation, not `RENAME COLUMN` which fails on ORDER BY keys)
   **Plans**: 1 plan

Plans:

- [x] 03-01-PLAN.md -- Update schema.sql, create migration script, update \_ensure_schema() DDL

### Phase 4: Python + SQL Rename

**Goal**: Every Python file, SQL query, and configuration reference uses `_us` naming -- the codebase speaks one language
**Depends on**: Phase 2 (Rust keys), Phase 3 (CH columns)
**Requirements**: RENAME-02, RENAME-04, RENAME-05
**Success Criteria** (what must be TRUE):

1. Zero occurrences of `close_time_ms` or `open_time_ms` in Python source files (verified by grep)
2. Zero occurrences of `close_time_ms` or `open_time_ms` in SQL strings/queries (verified by grep)
3. `mise run check-full` passes -- all tests, lints, and formatting clean after bulk rename
4. Sidecar `/health` endpoint and heartbeat queries return data using `_us` column names
   **Plans**: 1 plan

Plans:

- [x] 04-01-PLAN.md -- Bulk rename \_ms to_us across 38 Python/SQL files + check-full validation

### Phase 5: Deploy Gap Repair

**Goal**: Deploys that cross midnight produce zero-gap bar coverage via working `backfill_gap()` integrated into the ship pipeline
**Depends on**: Phase 4 (all renames complete -- backfill_gap reads `close_time_us`)
**Requirements**: BFILL-01, BFILL-02
**Success Criteria** (what must be TRUE):

1. `backfill_gap()` trailing edge calculation correctly reads `close_time_us` microsecond values (not stale `_ms` reference)
2. `backfill_gap(force=True)` completes successfully for all 18 streaming symbol/threshold pairs without error
3. After a simulated deploy gap, Stathera continuity holds across the gap boundary (no missing trade IDs)
4. odb-ship deploy pipeline invokes cross-midnight gap repair automatically when deploy spans UTC midnight
   **Plans**: 1 plan

Plans:

- [x] 05-01-PLAN.md -- Fix backfill_gap() ms-to-us unit mismatch and update tests

## Backlog

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 0/2 plans executed

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

## Progress

**Execution Order:** Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5
Phase 1 is independent (test hygiene). Phases 2-5 are sequential (rename chain).

| Phase                          | Milestone | Plans Complete | Status      | Completed |
| ------------------------------ | --------- | -------------- | ----------- | --------- |
| 1. Test Hygiene                | v4.0      | 1/1            | Complete    | -         |
| 2. Rust Dict Export Rename     | v4.0      | 1/1            | Complete    | -         |
| 3. ClickHouse Schema Migration | v4.0      | 1/1            | Complete    | -         |
| 4. Python + SQL Rename         | v4.0      | 1/1            | Complete    | -         |
| 5. Deploy Gap Repair           | v4.0      | 0/1            | Not started | -         |
