# opendeviationbar-py — Project

## What This Is

Rust workspace with Python bindings for open deviation bar construction. Performance-optimized production system processing 42M+ bars across 17 symbols on bigblack, with zero-gap zero-overlap guarantees validated by continuous telemetry.

## Core Value

Zero-gap, zero-overlap bars across every restart, every boundary, every deploy — verified by continuous telemetry. Now supports both Day-mode (UTC-midnight reset) and Aion-mode (continuous genesis-forward) per symbol.

## Requirements

### Validated

- ✓ Rust-side bar dedup prevents overlaps at source (v13.33.0)
- ✓ Sidecar overlap self-heal detects+repairs today's overlaps (v13.32.0)
- ✓ Deploy hardening: monit unmonitor, mutation cleanup, OPTIMIZE FINAL (v13.32.1)
- ✓ P0-P8: v1.2 Performance milestone (all 12 requirements)
- ✓ Atomic replace in sync_flush using store_bars_replacing() (v1.3)
- ✓ Legacy pruning: dead restart-gap code removed (v1.3)
- ✓ Clean slate DELETE today before fill_from_rest (v1.3)
- ✓ drain_bars() synchronous ring buffer drain (v1.3)
- ✓ 1000 dbps removed, 100 dbps added for BTC/ETH/SOL (v13.38.0)
- ✓ committed_floors from last completed bar, not forming bar tail — v1.4 (RUST-01, RUST-02)
- ✓ Midnight crossover trade ID verification via REST API preflight — v1.4 (PREFLIGHT-01-03)
- ✓ Heartbeat unified per-symbol Stathera continuity + LOUD alerts — v1.4 (TELEM-01-04)
- ✓ WS resilience: ping/pong keepalive, ban handling, stream signals — v1.4 (Phase 4)
- ✓ Seeder REST fallback + cross-process rate limit coordination — v1.4 (SEEDER-01-04)
- ✓ Memory efficiency: Copy AggTrade, Arrow streaming, GIL release, Python optimizations — v1.4 (MEM-01-08)
- ✓ Yesterday integrity check + signal file — v1.5 (STARTUP-01-02)
- ✓ Kintsugi yesterday priority queue — v1.5 (STARTUP-03)
- ✓ Sub-minute sidecar startup (27s measured) — v1.5 (STARTUP-04)

- ✓ Skip-if-clean DELETE strategy with pre-check query — v8.0 (RESUME-04, RESUME-01)
- ✓ Exchange gap floor clamping for resume TID — v8.0 (RESUME-01)
- ✓ Parquet-first Aion gap fill before REST API — v8.0 (RESUME-02)
- ✓ Junction telemetry at REST-to-WS handoff — v8.0 (RESUME-03)
- ✓ Bulk Aion repair of 98K historical day-boundary gaps — v8.0 (CLEANUP-01)
- ✓ Day-mode dead code removal with forex guards — v8.0 (CLEANUP-02-04)
- ✓ Aion-first defaults across all function signatures — v8.0 (DEFAULT-01-05)

- ✓ End-to-end SHA-256 checksum verification for Vision ZIP downloads — v6.0 (SPIKE-01..03, SEEDER-01..03)
- ✓ Centralized ChecksumRegistry with per-file verification state as SSoT — v6.0 (TELEM-03)
- ✓ Integrity sweep audits existing Parquet cache against Vision CHECKSUMs — v6.0 (SWEEP-01..03)
- ✓ Heartbeat checksum health + LOUD Telegram alerts + JSONL telemetry log — v6.0 (TELEM-01, 02, 04)

- ✓ OuroborosMode.AION enum + symbols.toml per-symbol SSoT — v7.0 (REG-01..05)
- ✓ Gap awareness on bars: `has_gap`, `gap_trade_count`, `max_gap_duration_us`, `is_exchange_gap` — v7.0 (CORE-01..04)
- ✓ Bit-exact oracle: Day==Aion single-day equivalence, midnight-spanning continuity — v7.0 (VAL-01..03)
- ✓ Sidecar Aion startup: checkpoint-resume, no clean-slate DELETE — v7.0 (SIDE-01..03)
- ✓ Kintsugi Aion repair: TID-range reprocessing, conditional today-guard — v7.0 (KINT-01..03)
- ✓ Monitor + Validation: per-symbol pre-write gate, Aion freshness, Stathera oracle — v7.0 (MON-01..03)
- ✓ Fleet recompute script + diagnostic updates for bigblack migration — v7.0 (RECOMP-01..05, VAL-04)

### Active

- [ ] Rust `ClickHouseWriter` implementing `BarSink` trait with `reqwest::Client` POST to CH HTTP interface
- [ ] Batch buffering in Rust (configurable: N bars or T ms threshold)
- [ ] Niffler dead-letter fallback in Rust (Parquet serialization on write failure)
- [ ] Sidecar integration: wire Rust writer into WS streaming path, replacing Python httpx `_sync_flush`
- [ ] Dedup token generation in Rust (MD5 of cache_key for INSERT dedup)
- [ ] Feature flag for Rust/Python writer fallback during rollout

## Current Milestone: v9.0 Rust-Native ClickHouse Writer

**Goal:** Eliminate Python from the WS streaming hot path by implementing a Rust-native ClickHouse writer — the last Python bottleneck in the real-time sidecar pipeline.

**Target features:**

- Rust `ClickHouseWriter` implementing `BarSink` trait — direct `reqwest::Client` POST to ClickHouse HTTP interface (port 8123)
- Batch buffering in Rust (configurable: N bars or T ms threshold)
- Niffler dead-letter fallback in Rust (Parquet serialization on write failure)
- Sidecar integration: wire Rust writer into the WS streaming path, replacing Python httpx
- Dedup token generation matching Python's MD5(cache_key) for INSERT dedup continuity
- Feature flag (`OPENDEVIATIONBAR_USE_RUST_CH_WRITER`) for safe rollout with Python fallback

## Current State

v8.0 Aion-Native Pipeline shipped (2026-03-28). 14 milestones complete.
All crypto symbols Aion-only (16 symbols, 0 day-mode). Day-mode tests removed.
Historical day-boundary gaps bulk-repaired. Zero pipeline gaps, zero overlaps.
Sidecar hot path: WS→Rust bars (<1ms) → Python httpx→CH (5-50ms per batch).
BarSink trait exists in opendeviationbar-streaming with ChannelSink reference impl.
Current Python write path: clickhouse_write.py via_write_batch_with_retry() + stamina retry.

### Out of Scope

- Rewriting heartbeat/seeder/kintsugi CH writes — network I/O dominates, Python overhead negligible
- Replacing Python sidecar orchestrator — only the CH write path moves to Rust
- OpenDeviationBar struct split (P9) — major refactor, separate milestone
- Full 18-symbol streaming — planned for later milestone
- Plugin Polars->Pandas — no plugins in production

## Key Decisions

| Decision                                       | Rationale                                               | Outcome                                 |
| ---------------------------------------------- | ------------------------------------------------------- | --------------------------------------- |
| Clean slate DELETE today on restart            | Proven zero-gap pattern (v1.3)                          | ✓ Good — 0/0 Stathera on restart        |
| store_bars_replacing for sync_flush            | Proven kintsugi mechanism                               | ✓ Good — atomic per-chunk               |
| REST API crossover verification                | User insight — pinpoint exact midnight trade ID         | ✓ Good — verified in production         |
| committed_floors from completed bar            | 6-agent spike root cause (Bug 1)                        | ✓ Good — junction gaps eliminated       |
| Unified Stathera in heartbeat                  | No more intra-day vs midnight classification            | ✓ Good — single per-symbol check        |
| File-based rate limit coordination             | Cross-process budget sharing without Redis              | ✓ Good — atomic JSON, stale expiry      |
| Oracle golden snapshots before refactoring     | Bit-exact regression baseline                           | ✓ Good — caught 0 regressions           |
| AggTrade Copy + Arrow zero-copy                | Eliminate clone overhead + heap allocation              | ✓ Good — verified bit-exact             |
| Aion = continuous Ouroboros variant            | CEO: crypto is 24/7, orphan bars unnecessary            | ✓ Good — shipped v7.0, 27/27 reqs       |
| Gap taxonomy: intra-bar flag, inter-bar query  | Marking individual bars for inter-bar gaps is ambiguous | ✓ Good — G1/G2 flagged, G3/G4/O1 query  |
| symbols.toml SSoT for ouroboros_mode           | Registry already controls thresholds, market, priority  | ✓ Good — per-symbol mode control        |
| has_gap is transient, healers fire immediately | Mark-and-continue, not accept-and-ignore                | ✓ Good — Puck/Kintsugi repair ASAP      |
| Per-symbol mode wiring (not global)            | Mixed Day+Aion fleet in one sidecar instance            | ✓ Good — HashMap<String, OuroborosMode> |
| Aion repair = TID-range, not day-recompute     | Continuous bars have no day boundaries                  | ✓ Good — \_repair_aion() in kintsugi    |

## Context

v8.0 shipped with 5 phases (29-33), completing Aion-native pipeline with zero-gap restarts.
Sidecar hot path bottleneck identified: Python httpx write path is 5-50ms per batch (Issue #318).
BarSink trait + ChannelSink reference impl ready in opendeviationbar-streaming crate.

**Known tech debt:**

- Bigblack fleet recompute pending execution (`scripts/fleet_recompute_aion.py`)
- TELEM-03: `ws_first_tid` and `committed_floors` junction telemetry sub-fields not yet populated
- Phase 14 VERIFICATION had gaps_found (prev_close_time — fixed inline)
- Phase 18 test regression fixed inline (get_symbol_entries mock)

## Constraints

- Python 3.13 only
- No Rust API changes that break PyO3 bindings
- Spot-only data sources (Binance)
- 10 hard constraints from git history (see v1.3 debug findings)

## Evolution

This document evolves at phase transitions and milestone boundaries.

**After each phase transition** (via `/gsd:transition`):

1. Requirements invalidated? → Move to Out of Scope with reason
2. Requirements validated? → Move to Validated with phase reference
3. New requirements emerged? → Add to Active
4. Decisions to log? → Add to Key Decisions
5. "What This Is" still accurate? → Update if drifted

**After each milestone** (via `/gsd:complete-milestone`):

1. Full review of all sections
2. Core Value check — still the right priority?
3. Audit Out of Scope — reasons still valid?
4. Update Context with current state

---

_Last updated: 2026-03-28 — v9.0 Rust-Native ClickHouse Writer milestone started_
