# Roadmap: opendeviationbar-py

## Milestones

- ✅ **v1.0 Junction Fix** -- Phases 1-3 (shipped 2026-03-22) -- [archive](milestones/v1.0-ROADMAP.md)
- ✅ **v1.4 Zero-Gap Zero-Overlap Guarantee** -- Phases 1-6 (shipped 2026-03-23) -- [archive](milestones/v1.4-ROADMAP.md)
- ✅ **v1.5 Sidecar Startup Architecture** -- Phases 1-3 (shipped 2026-03-23) -- [archive](milestones/v1.5-ROADMAP.md)
- ✅ **v2.0 Seeder as Sole REST Owner + Microsecond Alignment** -- Phases 1-5 (shipped 2026-03-24) -- [archive](milestones/v2.0-ROADMAP.md)
- ✅ **v3.0 Streaming Reliability + Pipeline Cleanup** -- Phases 1-5.1 (shipped 2026-03-24) -- [archive](milestones/v3.0-ROADMAP.md)
- ✅ **v4.0 Microsecond Column Rename + Deploy Gap Repair** -- Phases 1-5 (shipped 2026-03-25) -- [archive](milestones/v4.0-ROADMAP.md)
- ✅ **v5.0 Seeder Day-Boundary Completeness + Kintsugi Parquet Trust** -- Phases 1-5 (shipped 2026-03-25, v13.54.0)
- ✅ **v5.1 SEED-001 Fix + Heartbeat Unified View** -- Phases 6-7 (shipped 2026-03-25, v13.55.0)
- ✅ **v6.0 Data Integrity & Checksum Verification** -- Phases 8-12 (shipped 2026-03-26, v13.56.1)
- ✅ **v7.0 Aion Mode -- Genesis-Continuous for Crypto** -- Phases 13-19 (shipped 2026-03-27) -- [archive](milestones/v7.0-ROADMAP.md)
- ✅ **v7.1 Aion Hardening -- Footgun Elimination + Startup Fix** -- Phases 20-22 (shipped 2026-03-27) -- [archive](milestones/v7.1-ROADMAP.md)
- ✅ **v7.2 Memory Efficiency -- Zero-OOM Sidecar** -- Phases 23-25 (shipped 2026-03-27) -- [archive](milestones/v7.2-ROADMAP.md)
- ✅ **v7.3 Exchange Gap Classification** -- Phases 26-28 (shipped 2026-03-28)
- ✅ **v8.0 Aion-Native Pipeline -- Zero-Gap Restarts** -- Phases 29-33 (shipped 2026-03-28)
- 🚧 **v9.0 Rust-Native ClickHouse Writer** -- Phases 34-38 (in progress)

## Phases

<details>
<summary>✅ v5.0 Seeder Day-Boundary Completeness (Phases 1-5) - SHIPPED 2026-03-25</summary>

- [x] **Phase 1: Crossover TID Utility** - Extract shared crossover lookup callable by seeder, kintsugi, and heartbeat
- [x] **Phase 2: Seeder Day-Boundary Backfill** - Seeder backfills yesterday's Parquet file after midnight to crossover TID boundary
- [x] **Phase 3: Kintsugi Parquet Trust** - Kintsugi verifies Parquet completeness before using as repair source
- [x] **Phase 4: Heartbeat Coverage** - Yesterday completion check expanded to all streaming symbols
- [x] **Phase 5: Documentation** - Root cause and fix documented across CLAUDE.md network

</details>

<details>
<summary>✅ v5.1 SEED-001 Fix + Heartbeat Unified View (Phases 6-7) - SHIPPED 2026-03-25</summary>

- [x] **Phase 6: Fix SEED-001 Parquet-First Bar Processing** - Process Parquet ticks into bars instead of skipping past them
- [x] **Phase 7: Heartbeat Unified Per-Symbol View** - Merge Yesterday + Junction into exception-only display with TID deltas

</details>

<details>
<summary>✅ v6.0 Data Integrity & Checksum Verification (Phases 8-12) - SHIPPED 2026-03-26</summary>

- [x] **Phase 8: Checksum Spike** - Prove Rust SHA-256 callable from Python with correct results and edge case handling
- [x] **Phase 9: Verification Registry** - Centralized per-file verification state store as SSoT
- [x] **Phase 10: Seeder Checksum Integration** - Wire SHA-256 into seeder download pipeline with retry and audit trail
- [x] **Phase 11: Integrity Sweep** - Audit existing Parquet cache against Vision CHECKSUM files
- [x] **Phase 12: Telemetry & Heartbeat** - Heartbeat checksum health, LOUD alerts, local telemetry log

</details>

<details>
<summary>✅ v7.0 Aion Mode (Phases 13-19) - SHIPPED 2026-03-27</summary>

- [x] **Phase 13: Registry + Mode Enum** - Ouroboros mode SSoT in symbols.toml with Aion variant
- [x] **Phase 14: Rust Core -- Gap Fields + Conditional Reset** - Bar struct gap awareness and conditional midnight reset
- [x] **Phase 15: Oracle Validation** - Bit-exact oracle tests proving Aion correctness
- [x] **Phase 16: Sidecar Aion Startup** - Checkpoint-resume startup path for Aion symbols
- [x] **Phase 17: Kintsugi Aion Repair** - Genesis-forward reconciliation with TID-based writer boundaries
- [x] **Phase 18: Monitor + Validation Gates** - Rolling-window freshness, conditional day-mode gate, mode-agnostic Stathera
- [x] **Phase 19: Fleet Recompute + Telemetry** - Wipe Day-mode crypto bars, recompute as Aion on bigblack

</details>

<details>
<summary>✅ v7.1 Aion Hardening (Phases 20-22) - SHIPPED 2026-03-27</summary>

- [x] **Phase 20: Aion-Aware Sidecar Startup** - Preserve pre-midnight bars, seed from max(committed_tid, crossover_tid)
- [x] **Phase 21: Universal Write Path ouroboros_mode** - Wire per-symbol mode in all write call sites
- [x] **Phase 22: Monitor Hardening + Schema** - Freshness query filter, sidecar restart investigation, CH validation

</details>

<details>
<summary>✅ v7.2 Memory Efficiency (Phases 23-25) - SHIPPED 2026-03-27</summary>

- [x] **Phase 23: Arrow FFI drain_bars** - Arrow RecordBatch FFI for zero-copy sync_flush
- [x] **Phase 24: Hot-Path Allocation Reduction** - Single CH cache, eliminate per-flush dict conversions
- [x] **Phase 25: Lazy Evaluation + Cache Efficiency** - scan_parquet for dedup, in-place sort

</details>

<details>
<summary>✅ v7.3 Exchange Gap Classification (Phases 26-28) - SHIPPED 2026-03-28</summary>

- [x] **Phase 26: Exchange Gap Classification** - Classify gaps as pipeline vs exchange
- [x] **Phase 27: Kintsugi Exchange Gap Awareness** - Skip unfillable exchange gaps in repair
- [x] **Phase 28: Stathera Exchange Gap Resolution** - Permanent Stathera resolution for exchange gaps

</details>

<details>
<summary>✅ v8.0 Aion-Native Pipeline (Phases 29-33) - SHIPPED 2026-03-28</summary>

- [x] **Phase 29: DELETE Strategy + Exchange Gap Floor** - Spike safest DELETE approach and clamp resume TID above exchange gaps
- [x] **Phase 30: Parquet-First Gap Fill + Junction Telemetry** - Read local tick cache before REST API, measure handoff quality
- [x] **Phase 31: Aion-First Defaults** - Change default ouroboros mode from "day" to "aion" across all call sites
- [x] **Phase 32: Historical Gap Cleanup** - Bulk reprocess 98K day-boundary gaps as continuous Aion streams
- [x] **Phase 33: Dead Code Removal** - Remove day-mode code paths unreachable with Aion-only crypto

</details>

### v9.0 Rust-Native ClickHouse Writer

- [x] **Phase 34: Core ClickHouse Writer** - Rust BarSink with batch buffering, dedup tokens, pre-write guards, and retry (completed 2026-03-29)
- [x] **Phase 35: Dead-Letter Resilience** - Niffler Parquet dead-letter with Charon-compatible format, backpressure, and replay (completed 2026-03-29)
- [x] **Phase 36: Fan-Out + PyO3 Bridge** - LiveBarEngine multi-sink dispatch with metrics, config, and pre-write guards in Rust (completed 2026-03-29)
- [ ] **Phase 37: Python Integration + Staging Validation** - Feature flag, shadow mode dual-write, 24h staging on bigblack
- [ ] **Phase 38: REST Bar Flush Cutover** - Route startup fill_from_rest bars through Rust writer

## Phase Details

### Phase 34: Core ClickHouse Writer

**Goal**: A fully tested Rust `ClickHouseWriterSink` can serialize bars and POST them to ClickHouse with correct dedup tokens, batch buffering, retry, and response validation -- all in isolation without touching the sidecar
**Depends on**: Phase 33 (v8.0 shipped)
**Requirements**: WRITER-01, WRITER-02, WRITER-03, WRITER-04, WRITER-05, WRITER-06, WRITER-07, WRITER-08, WRITER-09
**Success Criteria** (what must be TRUE):

1. Running the Rust test suite with a mock HTTP server shows `ClickHouseWriterSink` buffers bars and flushes at the configured threshold (N bars, T ms, or max bytes -- whichever fires first)
2. A cross-language parity test confirms the Rust MD5 dedup token for a fixed set of inputs is byte-identical to Python's `hashlib.md5(f"{symbol}_{threshold}_{start_ts}_{end_ts}_{ouroboros_mode}").hexdigest()`
3. The writer parses `X-ClickHouse-Summary` after every INSERT and logs a warning when `written_rows` differs from `expected_rows` (verifiable in test output)
4. Computed columns (`is_liquidation_cascade`) are never sent in the INSERT payload -- ClickHouse evaluates DEFAULT expressions
5. Plugin columns are discovered via `system.columns` query at startup, not hardcoded -- adding a plugin column requires zero Rust code changes
   **Plans**: 3 plans

Plans:

- [x] 34-01-PLAN.md — Foundation: ClickHouseBarRow, dedup token, config, summary parser
- [x] 34-02-PLAN.md — ClickHouseWriterSink BarSink impl, flush thread, retry, plugin discovery
- [x] 34-03-PLAN.md — Gap closure: wire X-ClickHouse-Summary verification into production do_flush()

### Phase 35: Dead-Letter Resilience

**Goal**: When ClickHouse is unreachable or the flush thread falls behind, bars are serialized to Parquet dead-letter files that Python's Charon can replay without modification
**Depends on**: Phase 34
**Requirements**: RESIL-01, RESIL-02, RESIL-03, RESIL-04, RESIL-05
**Success Criteria** (what must be TRUE):

1. When the mock HTTP server returns 503 for all retries, a Parquet file appears in `/tmp/opendeviationbar-dead-letter/` with the correct `{symbol}_{threshold}_{timestamp}.parquet` naming
2. A cross-language round-trip test passes: Rust writes a dead-letter Parquet, Python reads it via `store_bars_batch()`, and the bars appear in ClickHouse with correct values (column names use `_us` suffix)
3. When the flush thread recovers (HTTP starts succeeding), accumulated dead-letter files are replayed automatically before new bars are written (Niffler pattern)
4. When the bounded SyncSender channel is full, `on_bar()` does not block -- overflow bars are dead-lettered with a warning log
   **Plans**: 2 plans

Plans:

- [x] 35-01-PLAN.md — Parquet dead-letter writer module + flush/backpressure wiring
- [x] 35-02-PLAN.md — Niffler replay on flush thread + cross-language round-trip test

### Phase 36: Fan-Out + PyO3 Bridge

**Goal**: LiveBarEngine dispatches every completed bar to both ChannelSink (for Python) and ClickHouseWriterSink (for CH), with writer metrics and config flowing through the PyO3 bridge
**Depends on**: Phase 35
**Requirements**: INTEG-01, INTEG-02, INTEG-04, INTEG-05, INTEG-07
**Success Criteria** (what must be TRUE):

1. With fan-out enabled, Python's `engine.next_bar()` still receives every bar via ChannelSink while ClickHouseWriterSink independently writes to CH -- neither sink blocks the other
2. Python can call `stream_manager.get_metrics()` and see Rust writer counters (`ch_bars_written`, `ch_bars_failed`, `ch_dead_letters`, `ch_last_flush_latency_us`) updating in real time
3. Pre-write guards in Rust reject bars with inverted timestamps, negative volume, or unregistered thresholds before serialization -- verifiable via a guard-violation test that checks no HTTP POST is made
4. `ouroboros_mode` is hardcoded to `"aion"` in every INSERT -- no mode-switching logic exists in the Rust writer
   **Plans**: 2 plans

Plans:

- [x] 36-01-PLAN.md — Pre-write guards + fan-out wiring in LiveBarEngine and trade_dispatch
- [x] 36-02-PLAN.md — Metrics bridge in StreamManager + Cargo.toml clickhouse-sink wiring

### Phase 37: Python Integration + Staging Validation

**Goal**: The sidecar runs in shadow mode on bigblack for 24+ hours, dual-writing via both Rust and Python, with zero diff between the two and zero dead-letter files
**Depends on**: Phase 36
**Requirements**: INTEG-03, VALID-01, VALID-02, VALID-03, VALID-04, VALID-05
**Success Criteria** (what must be TRUE):

1. Setting `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=shadow` causes both Rust and Python to write bars, and a comparison query shows zero row differences between them
2. Setting `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=live` causes Python to skip `_write_batch_with_retry()` entirely -- only Rust writes to CH
3. After 24 hours of shadow mode on bigblack, Stathera audit shows zero new gaps or overlaps -- same data integrity as Python-only operation
4. Heartbeat reports `ch_dead_letters == 0` throughout the staging period -- any dead-letter file triggers a LOUD alert
5. The feature flag is evaluated exactly once at `_create_engine()` startup -- no mid-session writer switching is possible
   **Plans**: 2 plans

Plans:

- [x] 37-01-PLAN.md — Feature flag, shadow compare, heartbeat monitoring, deploy env
- [ ] 37-02-PLAN.md — Validation script + 24h staging checkpoint on bigblack

### Phase 38: REST Bar Flush Cutover

**Goal**: The startup `fill_from_rest` bars flow through the Rust ClickHouse writer, eliminating the last Python CH write on the hot path
**Depends on**: Phase 37 (staging validation must pass)
**Requirements**: INTEG-06
**Success Criteria** (what must be TRUE):

1. With `OPENDEVIATIONBAR_USE_RUST_CH_WRITER=live`, sidecar startup `_sync_flush_rest_bars()` drains bars through the Rust writer -- no calls to `_write_bars_batch_to_clickhouse()` in `arrow_flush.py`
2. After restart with Rust REST flush, Stathera audit shows zero gaps at the REST-to-WS junction -- same quality as the Python path
3. `clickhouse_write.py` is reachable only when the feature flag is `off` or `shadow` -- it becomes fallback-only code
   **Plans**: 2 plans

Plans:

- [ ] 36-01-PLAN.md — Pre-write guards + fan-out wiring in LiveBarEngine and trade_dispatch
- [ ] 36-02-PLAN.md — Metrics bridge in StreamManager + Cargo.toml clickhouse-sink wiring

## Backlog

### Phase 999.1: Rust multi-threshold Arrow API for repair pipeline (BACKLOG)

**Goal:** Add `process_trades_arrow_multi()` to Rust processor -- accept multiple thresholds in one call, eliminate Python GIL round-trips. Benchmarked 2026-03-24: 48 workers hit GIL ceiling at 469% CPU (15/32 cores). Est. 2-3x speedup (3,200 -> 6,000-10,000 days/min).
**Requirements:** TBD
**Plans:** 1/2 plans executed

Plans:

- [ ] TBD (promote with /gsd:review-backlog when ready)

### Phase 999.2: Default ouroboros mode from "day" to "aion" (BACKLOG -- PROMOTED to Phase 31)

**Goal:** Promoted to Phase 31 in v8.0 milestone.

## Progress

**Execution Order:** Phases 34-38 execute sequentially. Phase 34 is the isolated Rust writer. Phase 35 adds dead-letter. Phase 36 wires fan-out and PyO3 bridge. Phase 37 is Python integration + shadow-mode staging. Phase 38 is final REST flush cutover.

| Phase                                           | Milestone | Plans Complete | Status      | Completed  |
| ----------------------------------------------- | --------- | -------------- | ----------- | ---------- |
| 1. Crossover TID Utility                        | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 2. Seeder Day-Boundary Backfill                 | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 3. Kintsugi Parquet Trust                       | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 4. Heartbeat Coverage                           | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 5. Documentation                                | v5.0      | 1/1            | Complete    | 2026-03-25 |
| 6. Fix SEED-001 Parquet-First Processing        | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 7. Heartbeat Unified Per-Symbol View            | v5.1      | 1/1            | Complete    | 2026-03-25 |
| 8. Checksum Spike                               | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 9. Verification Registry                        | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 10. Seeder Checksum Integration                 | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 11. Integrity Sweep                             | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 12. Telemetry & Heartbeat                       | v6.0      | 1/1            | Complete    | 2026-03-26 |
| 13. Registry + Mode Enum                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 14. Rust Core -- Gap Fields + Conditional Reset | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 15. Oracle Validation                           | v7.0      | 1/1            | Complete    | 2026-03-27 |
| 16. Sidecar Aion Startup                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 17. Kintsugi Aion Repair                        | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 18. Monitor + Validation Gates                  | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 19. Fleet Recompute + Telemetry                 | v7.0      | 2/2            | Complete    | 2026-03-27 |
| 20. Aion-Aware Sidecar Startup                  | v7.1      | 1/1            | Complete    | 2026-03-27 |
| 21. Universal Write Path ouroboros_mode         | v7.1      | 1/1            | Complete    | 2026-03-27 |
| 22. Monitor Hardening + Schema                  | v7.1      | 2/2            | Complete    | 2026-03-27 |
| 23. Arrow FFI drain_bars                        | v7.2      | 1/1            | Complete    | 2026-03-27 |
| 24. Hot-Path Allocation Reduction               | v7.2      | 1/1            | Complete    | 2026-03-27 |
| 25. Lazy Evaluation + Cache Efficiency          | v7.2      | 1/1            | Complete    | 2026-03-27 |
| 26. Exchange Gap Classification                 | v7.3      | -/-            | Complete    | 2026-03-28 |
| 27. Kintsugi Exchange Gap Awareness             | v7.3      | -/-            | Complete    | 2026-03-28 |
| 28. Stathera Exchange Gap Resolution            | v7.3      | -/-            | Complete    | 2026-03-28 |
| 29. DELETE Strategy + Exchange Gap Floor        | v8.0      | 2/2            | Complete    | 2026-03-28 |
| 30. Parquet-First Gap Fill + Junction Telemetry | v8.0      | 2/2            | Complete    | 2026-03-28 |
| 31. Aion-First Defaults                         | v8.0      | 1/1            | Complete    | 2026-03-28 |
| 32. Historical Gap Cleanup                      | v8.0      | 1/1            | Complete    | 2026-03-28 |
| 33. Dead Code Removal                           | v8.0      | 1/1            | Complete    | 2026-03-28 |
| 34. Core ClickHouse Writer                      | v9.0      | 3/3            | Complete    | 2026-03-29 |
| 35. Dead-Letter Resilience                      | v9.0      | 2/2 | Complete    | 2026-03-29 |
| 36. Fan-Out + PyO3 Bridge                       | v9.0      | 2/2 | Complete    | 2026-03-29 |
| 37. Python Integration + Staging Validation     | v9.0      | 1/2 | In Progress|  |
| 38. REST Bar Flush Cutover                      | v9.0      | 0/0            | Not started | -          |
