# Milestones

## v7.3 Exchange Gap Classification -- Permanent Stathera Resolution (Shipped: 2026-03-28)

**Phases completed:** 0 phases, 0 plans, 0 tasks

**Key accomplishments:**

- (none recorded)

---

## v7.2 Memory Efficiency -- Zero-OOM Sidecar (Shipped: 2026-03-27)

**Phases completed:** 10 phases, 17 plans, 27 tasks

**Key accomplishments:**

- OuroborosMode.AION enum variant with symbols.toml wiring, crypto-only validation, and boundary function dispatch
- Expanded all public Python API ouroboros_mode parameters from Literal["day"] to Literal["day", "aion"] with matching type stubs
- 4 gap awareness fields on OpenDeviationBar with TID-based gap detection in update_with_trade, exported through Arrow and PyO3 dict paths
- 1. [Rule 3 - Blocking] Fixed duplicate import conflict
- 1. [Rule 3 - Blocking] OuroborosMode not re-exported from streaming crate
- Per-symbol ouroboros_mode wired through all 6 kintsugi call sites with conditional today-guard for Aion symbols
- TID-range gap repair for Aion symbols via \_repair_aion() with Parquet-first sourcing and single continuous processor
- Per-symbol mode-aware pre-write gate filtering only day-mode cross-midnight bars, with Aion exclusion in Stathera cross-midnight oracle
- Rolling-window freshness check for Aion symbols replacing meaningless crossover_tid orphan check, with configurable AION_FRESHNESS_WINDOW_MIN and safe day-mode fallback
- Diagnostic scripts, deploy config, and schema comments updated to support Aion ouroboros mode with per-symbol registry auto-detection
- Coordinated stop-drop-wipe-recompute script with 3-phase CLI (preflight/execute/verify), dynamic partition discovery, and Stathera audit for bigblack Aion migration
- Crossover-TID max-seed + targeted today-DELETE for Aion symbols to eliminate 8 midnight-boundary Stathera gaps
- Per-symbol ouroboros_mode wired into all 6 store_bars_batch call sites across backfill.py, dead_letter.py, and count_bounded.py -- eliminating global get_operational_ouroboros_mode() fallback for crypto write paths
- Aion freshness query filters by ouroboros_mode='aion' (MON-04) and both store methods warn on mode mismatch (SCHEMA-01)
- Sidecar restart root cause identified as operator-induced (v7.0 deploy); PID age + restart count telemetry added to heartbeat

---

## v7.1 Aion Hardening -- Footgun Elimination + Startup Fix (Shipped: 2026-03-27)

**Phases completed:** 11 phases, 17 plans, 27 tasks

**Key accomplishments:**

- OuroborosMode.AION enum variant with symbols.toml wiring, crypto-only validation, and boundary function dispatch
- Expanded all public Python API ouroboros_mode parameters from Literal["day"] to Literal["day", "aion"] with matching type stubs
- 4 gap awareness fields on OpenDeviationBar with TID-based gap detection in update_with_trade, exported through Arrow and PyO3 dict paths
- 1. [Rule 3 - Blocking] Fixed duplicate import conflict
- 1. [Rule 3 - Blocking] OuroborosMode not re-exported from streaming crate
- Per-symbol ouroboros_mode wired through all 6 kintsugi call sites with conditional today-guard for Aion symbols
- TID-range gap repair for Aion symbols via \_repair_aion() with Parquet-first sourcing and single continuous processor
- Per-symbol mode-aware pre-write gate filtering only day-mode cross-midnight bars, with Aion exclusion in Stathera cross-midnight oracle
- Rolling-window freshness check for Aion symbols replacing meaningless crossover_tid orphan check, with configurable AION_FRESHNESS_WINDOW_MIN and safe day-mode fallback
- Diagnostic scripts, deploy config, and schema comments updated to support Aion ouroboros mode with per-symbol registry auto-detection
- Coordinated stop-drop-wipe-recompute script with 3-phase CLI (preflight/execute/verify), dynamic partition discovery, and Stathera audit for bigblack Aion migration
- Crossover-TID max-seed + targeted today-DELETE for Aion symbols to eliminate 8 midnight-boundary Stathera gaps
- Per-symbol ouroboros_mode wired into all 6 store_bars_batch call sites across backfill.py, dead_letter.py, and count_bounded.py -- eliminating global get_operational_ouroboros_mode() fallback for crypto write paths
- Aion freshness query filters by ouroboros_mode='aion' (MON-04) and both store methods warn on mode mismatch (SCHEMA-01)
- Sidecar restart root cause identified as operator-induced (v7.0 deploy); PID age + restart count telemetry added to heartbeat

---

## v7.0 Aion Mode -- Genesis-Continuous for Crypto (Shipped: 2026-03-27)

**Phases completed:** 15 phases, 20 plans, 32 tasks

**Key accomplishments:**

- Replaced broken SEED-001 "skip Parquet range" with "process Parquet ticks into bars via Rust processor, seed with last_completed_bar_tid"
- Merged Yesterday + Junction heartbeat sections into exception-only Per-Symbol Health display with per-symbol TID delta computation
- Rust SHA-256 checksum bindings proven callable from Python with correct results, 13.5% overhead (HTTP-bound, acceptable), and graceful edge case handling (404/timeout soft-skip, mismatch/invalid hard-fail)
- ChecksumRegistry with per-file SHA-256 verification state, upsert dict storage, and atomic JSON persistence
- SHA-256 checksum verification wired into tick_cache_seeder Vision download pipeline with retry, LOUD alerts, and ChecksumRegistry persistence
- Retroactive Parquet cache auditing via Vision ZIP SHA-256 verification with incremental registry tracking and concurrent downloads
- Checksum verification health monitoring in heartbeat with JSONL telemetry log and LOUD Telegram alerts on integrity failures
- OuroborosMode.AION enum variant with symbols.toml wiring, crypto-only validation, and boundary function dispatch
- Expanded all public Python API ouroboros_mode parameters from Literal["day"] to Literal["day", "aion"] with matching type stubs
- 4 gap awareness fields on OpenDeviationBar with TID-based gap detection in update_with_trade, exported through Arrow and PyO3 dict paths
- 1. [Rule 3 - Blocking] Fixed duplicate import conflict
- 1. [Rule 3 - Blocking] OuroborosMode not re-exported from streaming crate
- Per-symbol ouroboros_mode wired through all 6 kintsugi call sites with conditional today-guard for Aion symbols
- TID-range gap repair for Aion symbols via \_repair_aion() with Parquet-first sourcing and single continuous processor
- Per-symbol mode-aware pre-write gate filtering only day-mode cross-midnight bars, with Aion exclusion in Stathera cross-midnight oracle
- Rolling-window freshness check for Aion symbols replacing meaningless crossover_tid orphan check, with configurable AION_FRESHNESS_WINDOW_MIN and safe day-mode fallback
- Diagnostic scripts, deploy config, and schema comments updated to support Aion ouroboros mode with per-symbol registry auto-detection
- Coordinated stop-drop-wipe-recompute script with 3-phase CLI (preflight/execute/verify), dynamic partition discovery, and Stathera audit for bigblack Aion migration

---

## v6.0 v6.0 (Shipped: 2026-03-26)

**Phases completed:** 8 phases, 7 plans, 13 tasks

**Key accomplishments:**

- Replaced broken SEED-001 "skip Parquet range" with "process Parquet ticks into bars via Rust processor, seed with last_completed_bar_tid"
- Merged Yesterday + Junction heartbeat sections into exception-only Per-Symbol Health display with per-symbol TID delta computation
- Rust SHA-256 checksum bindings proven callable from Python with correct results, 13.5% overhead (HTTP-bound, acceptable), and graceful edge case handling (404/timeout soft-skip, mismatch/invalid hard-fail)
- ChecksumRegistry with per-file SHA-256 verification state, upsert dict storage, and atomic JSON persistence
- SHA-256 checksum verification wired into tick_cache_seeder Vision download pipeline with retry, LOUD alerts, and ChecksumRegistry persistence
- Retroactive Parquet cache auditing via Vision ZIP SHA-256 verification with incremental registry tracking and concurrent downloads
- Checksum verification health monitoring in heartbeat with JSONL telemetry log and LOUD Telegram alerts on integrity failures

---

## v5.0 Seeder Day-Boundary Completeness + Kintsugi Parquet Trust (Shipped: 2026-03-25)

**Phases completed:** 6 phases, 5 plans, 7 tasks

**Key accomplishments:**

- Shared get_crossover_tid(symbol, date) utility extracted from sidecar internals with 2-tier CH+REST lookup, date parameterization, and backward-compatible sidecar wrapper
- Post-midnight backfill of yesterday's Parquet file to crossover_tid - 1 using shared crossover utility, with state marker dedup and 5 unit tests
- Parquet completeness gate via crossover TID boundary check and orphan bar marking in both Parquet and Vision kintsugi repair paths
- Yesterday completion check expanded from 2 hardcoded symbols to all enabled Binance spot symbols via registry lookup with graceful fallback
- v5.0 day-boundary completeness documented across 5 CLAUDE.md files: seeder backfill, kintsugi Parquet trust gate, heartbeat dynamic symbols, crossover.py module mapping, and L9 monitoring layer

---

## v4.0 Microsecond Column Rename + Deploy Gap Repair (Shipped: 2026-03-25)

**Phases completed:** 6 phases, 5 plans, 8 tasks

**Key accomplishments:**

- BarOutputSchema columns renamed from \_ms to \_us and dead test_heal_scoping.py deleted -- 17 tests pass green
- Renamed open_time_ms/close_time_ms to open_time_us/close_time_us in helpers.rs PyO3 dict export -- truthful keys matching actual microsecond values
- ClickHouse schema DDL and migration script updated from \_ms to \_us column naming with idempotent dry-run-default migration for bigblack
- Mechanical bulk rename of 244 close_time_ms/open_time_ms/last_close_time_ms occurrences to \_us across 38 Python/SQL files, verified by grep and full test suite
- Fixed backfill_gap() ms-to-us unit mismatch that caused negative gap_seconds and broke all trailing-edge repairs

---

## v4.0 Microsecond Column Rename + Deploy Gap Repair (Shipped: 2026-03-25)

**Phases completed:** 6 phases, 5 plans, 8 tasks

**Key accomplishments:**

- BarOutputSchema columns renamed from \_ms to \_us and dead test_heal_scoping.py deleted -- 17 tests pass green
- Renamed open_time_ms/close_time_ms to open_time_us/close_time_us in helpers.rs PyO3 dict export -- truthful keys matching actual microsecond values
- ClickHouse schema DDL and migration script updated from \_ms to \_us column naming with idempotent dry-run-default migration for bigblack
- Mechanical bulk rename of 244 close_time_ms/open_time_ms/last_close_time_ms occurrences to \_us across 38 Python/SQL files, verified by grep and full test suite
- Fixed backfill_gap() ms-to-us unit mismatch that caused negative gap_seconds and broke all trailing-edge repairs

---

## v3.0 Streaming Reliability + Pipeline Cleanup (Shipped: 2026-03-24)

**Phases completed:** 7 phases, 10 plans, 20 tasks

**Key accomplishments:**

- Per-partition lock replaces global \_replace_lock enabling concurrent kintsugi workers; sidecar pre-fill migrated to O(1) DELETE IN PARTITION per pair
- Migrated all remaining open_deviation_bars DELETE call sites to partition-scoped DELETE IN PARTITION across 8 files, with kintsugi_log exception documented
- Stathera continuity validation and dropped_bars diagnostic in sidecar fill pipeline, with junction telemetry enrichment and STREAM-01 contract documentation
- ws_first_tid one-shot tracking in \_main_loop with full REST-to-WS junction telemetry exposed via /status endpoint
- Sidecar startup reads max(agg_trade_id) from today's Parquet tick cache per symbol, advancing REST fill start point past cached data to reduce API weight from ~500 to ~50
- WsMode enum with 2-connection combined split (volume-ranked) and per-symbol fallback, plumbed end-to-end from OPENDEVIATIONBAR_STREAMING_WS_MODE env var through Python config and PyO3 to Rust LiveEngineConfig
- Full quality gate validation (1094 tests, cargo deny, Python smoke test) confirms combined WS 2-connection split is deployment-ready with per-symbol rollback via OPENDEVIATIONBAR_STREAMING_WS_MODE env var
- 79 stale scripts removed (35,476 LOC) -- research patterns, one-off benchmarks, superseded maintenance -- verified unreachable via cross-reference audit against mise tasks, systemd units, and Python imports
- Removed ~1094 LOC dead streaming modules (indicators, stats), unused workspace deps (rolling-stats, tdigests), and stale TODOs referencing closed issues #59/#197
- /health endpoint now exposes ws_mode and expected_ws_connections; test helper \_make_bar() keys corrected from open_time_us to open_time_ms matching production Rust output

---

## v1.0 v1.0 (Shipped: 2026-03-24)

**Phases completed:** 7 phases, 12 plans, 24 tasks

**Key accomplishments:**

- Split 2,436-line live_engine.rs monolith into 5 focused submodules under live_engine/ directory with zero test regressions
- Split 2,188-line checkpoint.py monolith into 4-file checkpoint/ package (models, storage, engine, orchestrators) with full backward compatibility
- Extracted 3 orthogonal concerns (rectification, memory guard, vision cache) from backfill.py into sibling modules, reducing it from 1423 to 980 lines with full backward compatibility
- 1. [Rule 3 - Blocking] Renamed signal.py to yesterday_signal.py
- 10 Exness forex symbols registered in symbols.toml with ExnessSeederConfig pydantic-settings class for env-driven configuration
- Exness forex tick cache seeder downloading daily ZIPs from ticks.ex2archive.com with 3-column Parquet output, idempotent operation, and Telegram reporting
- systemd oneshot service with 1h timer, monit 2h freshness monitor, and 4 CLAUDE.md files updated for Exness forex seeder deployment infrastructure
- Kintsugi consumes sidecar's yesterday-repair signal file, injects flagged symbols at HEAD of repair queue with dedup, and clears signal after all shards processed
- 4 CLAUDE.md spokes for highest-complexity directories (src/, sidecar/, kintsugi/, checkpoint/) with source-derived file tables, domain-specific flow diagrams, and bidirectional parent links
- 4 new CLAUDE.md spoke files for validation (intermediate parent), processors, heartbeat, and heartbeat/checks (14-module index), plus day_mode parent chain fix
- Three CLAUDE.md spokes for compat (alpha-forge panel/manifest/availability), notify (Telegram @obdpy_bot integration), and ratelimit (file-based 6000 weight/min coordinator) with source-derived content
- Root CLAUDE.md hub table expanded from 13 to 24 entries with source-derived SSoT descriptions, full network verification: 24 files, 0 orphans, 0 broken parent links
- Plan:

---

## v1.4 Zero-Gap Zero-Overlap Guarantee (Shipped: 2026-03-23)

**Phases completed:** 6 phases, 16 plans, 32 tasks

**Key accomplishments:**

- Added last_completed_bar_tid field to OpenDeviationBarProcessor and Checkpoint, tracking completed bar trade IDs separately from forming bar tail for dedup floor initialization
- Fixed committed_floors dedup floor to use last_completed_bar_tid instead of last_agg_trade_id, eliminating junction bar suppression at REST-to-WS transition
- Binance REST midnight crossover query and ClickHouse orphan verification for Stathera continuity at day boundaries
- \_run_midnight_preflight() orchestrator wired into \_create_engine(), overriding CH seeds with verified crossover trade IDs for zero-gap midnight boundaries
- Unified per-symbol Stathera continuity check replacing intra-day/adjacent-day classification, with LOUD Telegram alert on new today gaps
- Per-symbol yesterday orphan completion via crossover TID validation, junction telemetry from sidecar HTTP+journal, both visible in Telegram heartbeat
- Client-side ping with real 10s pong deadline, Binance 429/ban error classification, and CONNECT/FIRST_DATA/DISCONNECT stream signal emission on both per-symbol and combined WebSocket paths
- Sidecar /health endpoint with WS connection lifecycle tracking (CONNECT/FIRST_DATA/DISCONNECT) and heartbeat Telegram alerts with 429-fatal vs transient-disconnect classification
- Trade-ID continuity validation in write_ticks(), seeder routed through dedup path, systemd overlap prevention via Conflicts=
- File-based RateLimitCoordinator + PyAdaptiveRateLimiter PyO3 binding + kintsugi rate limit gating + per-process systemd ceiling env vars
- REST API fallback for yesterday's Vision gaps using two-step fetch, 6-symbol batch rotation with BTCUSDT priority, and kintsugi-processed day guard
- Aggregate REST utilization monitoring with sustained-90% LOUD alert, seeder days-behind metric, and weekly Vision vs REST oracle check
- Captured bit-exact golden snapshot fixtures (46 bars each) for BTCUSDT 2024-01-01 at threshold=250 in both Rust (JSON) and Python (Parquet) before any memory refactoring changes
- AggTrade Copy derive, zero-copy Arrow iterator adapter, and FormingBar 1 Hz throttle -- three Rust-side P0 memory optimizations with bit-exact oracle verification
- GIL released via py.detach() during pure Rust computation in process_trades_arrow() and process_trades_arrow_native() -- Python threads unblocked during 50-200ms bar processing
- Skip plugin Polars-Pandas roundtrip, footer-only Parquet row counting, Arrow REST fallback, and merged with_columns calls

---

## v1.4 v1.4 (Shipped: 2026-03-21)

**Phases completed:** 3 phases, 6 plans, 11 tasks

**Key accomplishments:**

- Added last_completed_bar_tid field to OpenDeviationBarProcessor and Checkpoint, tracking completed bar trade IDs separately from forming bar tail for dedup floor initialization
- Fixed committed_floors dedup floor to use last_completed_bar_tid instead of last_agg_trade_id, eliminating junction bar suppression at REST-to-WS transition
- Binance REST midnight crossover query and ClickHouse orphan verification for Stathera continuity at day boundaries
- \_run_midnight_preflight() orchestrator wired into \_create_engine(), overriding CH seeds with verified crossover trade IDs for zero-gap midnight boundaries
- Unified per-symbol Stathera continuity check replacing intra-day/adjacent-day classification, with LOUD Telegram alert on new today gaps
- Per-symbol yesterday orphan completion via crossover TID validation, junction telemetry from sidecar HTTP+journal, both visible in Telegram heartbeat

---

## v1.4 Zero-Gap Zero-Overlap Guarantee (In Progress)

**Goal:** Eliminate ALL Stathera gaps and overlaps -- both intra-day junction gaps (committed_floors bug) and midnight boundary gaps (multi-processor seam) -- with full telemetry to validate.

**Phases:** 3 phases

| Phase | Name                            | Requirements     |
| ----- | ------------------------------- | ---------------- |
| 1     | Dedup Floor Fix                 | RUST-01, RUST-02 |
| 2     | Midnight Preflight Verification | PREFLIGHT-01..03 |
| 3     | Heartbeat Telemetry Overhaul    | TELEM-01..04     |

---

## v1.3 Atomic Restart Recovery (Shipped: 2026-03-21)

**Phases completed:** 2 phases, 4 plans, 6 tasks

**Key accomplishments:**

- Atomic DELETE+INSERT via store_bars_replacing() in \_sync_flush_rest_bars() with Polars conversion, Redis lock, and \_write_batch_with_retry fallback
- Today-only date filter on \_heal_day_boundary_gaps() Stathera query and valid_pairs passthrough for overlap heal call site
- Verified \_inline_startup_backfill(), \_startup_tid_reconciliation(), and \_gap_fill() watchdog call are already absent from sidecar.py -- all three PRUNE requirements satisfied
- Removed dead gap_fill_on_startup field from SidecarConfig, type stubs, CLI --no-gap-fill flag, and 9 test constructions

---

## v1.2 Performance (Shipped: 2026-03-20)

**Phases completed:** 4 phases, 5 plans, 8 tasks

**Key accomplishments:**

- Bounded ThreadPoolExecutor(max_workers=2) replaces unbounded threading.Thread in Telegram log sink, with dedicated 3s timeout to prevent thread exhaustion during kintsugi batch repairs
- Thread-local CH cache eliminates per-call fd churn in kintsugi workers, and 3-tier concurrency control prevents REST-only shards from diluting Parquet/Vision throughput
- Zero-allocation FixedPoint::from_f64() eliminates 4M allocs/day, cached CSV decompression removes O(N^2) ZIP re-parsing in IntraDayChunkIterator
- Eliminate per-iteration Vec::with_capacity(32) allocation in WS select loop via clear()+reuse pattern (RUST-03)
- Cached kintsugi config per repair (3 instantiations eliminated), dynamic Vision 404 TTL (15min for T-2 dates), and removed redundant DataFrame copy in bulk operations
