# Roadmap: v1.2 — Performance

**Created:** 2026-03-20
**Phases:** 4
**Requirements:** 12/12 mapped

---

## Phase 1: Telegram Thread Leak Fix (P0)

**Goal:** Eliminate thread exhaustion from unbounded Telegram log sink threads.

**Requirements:** STAB-01, STAB-02

**Plans:** 1/1 plans complete

Plans:

- [x] 01-01-PLAN.md — Replace threading.Thread with bounded ThreadPoolExecutor + log sink timeout

**Location:** `python/opendeviationbar/logging.py` lines 177-180

**Changes:**

1. Replace `threading.Thread(target=self._send, ...).start()` with a module-level `ThreadPoolExecutor(max_workers=2, thread_name_prefix="telegram-log")`
2. Use `executor.submit()` — if pool is busy, the 2 workers handle backpressure naturally (queue depth bounded by GC)
3. Reduce notification timeout from 10s to 3s for the log sink path (not direct `send_telegram()` calls)

**Success Criteria:**

1. `threading.Thread` never called directly for Telegram log sink
2. Max 2 concurrent Telegram threads regardless of log volume
3. Kintsugi 30-shard batch with Telegram SSL errors does NOT crash with `can't start new thread`
4. Existing `send_telegram()` direct calls unchanged (sidecar, kintsugi explicit notifications)

---

## Phase 2: Kintsugi REST + CH Connection Fix (P1-P2)

**Goal:** Eliminate hour-long REST repairs and connection churn.

**Requirements:** STAB-03, STAB-04, STAB-05, STAB-06

**Plans:** 1 plan

Plans:

- [x] 02-01-PLAN.md — Thread-local CH cache + 3-tier concurrency control (Parquet/Vision/REST)

**Changes:**

1. **Thread-local CH cache** (`kintsugi.py`): Add `_get_worker_cache()` using `threading.local()` in `_try_parquet_day_replace()` and `_try_vision_day_replace()` — same pattern as `repair_direct_parquet.py`
2. **REST concurrency limit**: Add `OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REST` env var (default 2). In `kintsugi_pass()`, when candidates include REST-only shards, cap concurrent workers to this value instead of `MAX_CONCURRENT_REPAIRS=24`
3. **Detect REST vs Parquet candidates**: Extend `_shard_is_parquet_cached()` to also check Vision availability. Group candidates into Parquet-cached (high parallelism) and REST-only (low parallelism)

**Success Criteria:**

1. Thread-local cache: `ls /proc/PID/fd | wc -l` shows ~4N+50 fds for N workers (not 96x20)
2. Single BTCUSDT REST repair completes in <30 minutes (not 1h+)
3. REST workers don't dilute Parquet workers' rate limit budget
4. No `RuntimeError: can't start new thread` from kintsugi

---

## Phase 3: Rust Hot Path Optimization (P3-P5)

**Goal:** Eliminate unnecessary allocations in trade processing and bar construction.

**Requirements:** RUST-01, RUST-02, RUST-03

**Plans:** 1/2 plans executed

Plans:

- [x] 03-01-PLAN.md — FixedPoint::from_f64() + IntraDayChunkIterator CSV decompression cache
- [x] 03-02-PLAN.md — WS burst frame Vec reuse via clear()

**Changes:**

1. **CSV FixedPoint** (`historical.rs:211`): Replace `FixedPoint::from_str(&csv_trade.1.to_string())` with `FixedPoint::from_f64()` or change `CsvAggTrade` fields to `String` with direct deserialization
2. **Chunk iterator** (`historical.rs:1129`): Store `csv::Reader` in `IntraDayChunkIterator` instead of re-opening ZIP per chunk. Save byte offset between chunks for O(N) instead of O(N²)
3. **Burst frame** (`live_engine.rs:1579`): Move `Vec<AggTrade>` outside the select loop, call `frame.clear()` instead of `Vec::with_capacity(32)` per iteration

**Success Criteria:**

1. Vision CSV load benchmark: measurable improvement for 20M-trade BTCUSDT day
2. No per-trade heap allocation in the WS select loop (verified via profiling)
3. All existing tests pass (`cargo nextest run -p opendeviationbar-streaming --lib`)

---

## Phase 4: Python Efficiency (P6-P8)

**Goal:** Eliminate redundant config instantiation, cache misses, and DataFrame copies.

**Requirements:** PYEF-01, PYEF-02, PYEF-03

**Plans:** 1 plan

Plans:

- [x] 04-01-PLAN.md — Config caching in \_repair() + dynamic Vision TTL + remove DataFrame copy

**Changes:**

1. **Config caching** (`kintsugi.py`): Call `_kintsugi_cfg()` once at top of `_repair()`, bind to local `cfg`. Replace 3 subsequent calls at lines 951, 993, 1025 with `cfg.batch_days`, `cfg.batch_workers`
2. **Vision TTL** (`kintsugi.py`): For dates within 2 days of today, use 15-minute Redis TTL instead of 1-hour. Allows faster retry when Vision publishes
3. **Remove copy** (`bulk_operations.py:503`): Change `df = bars.copy()` to `df = bars` — subsequent column mutations replace the reference, caller unaffected

**Success Criteria:**

1. `_kintsugi_cfg()` called <=2x per `_repair()` invocation (down from 3+)
2. Vision availability detected within 15 minutes of Binance publish (not 1 hour)
3. No redundant DataFrame copy in `store_bars_bulk()` non-DatetimeIndex path

---

## Dependency Graph

```
Phase 1 (Telegram) ─────┐
Phase 2 (REST + CH) ─────┤── Phase 5 (Deploy + Verify)
Phase 3 (Rust) ──────────┤
Phase 4 (Python) ────────┘
```

Phases 1-4 are independent — can execute in any order or parallel.

---

## Coverage

| Category  | Requirements | Count  | Phases |
| --------- | ------------ | ------ | ------ |
| Stability | STAB-01..06  | 6      | 1, 2   |
| Rust      | RUST-01..03  | 3      | 3      |
| Python    | PYEF-01..03  | 3      | 4      |
| **Total** |              | **12** | 1-4    |

**v1 requirements mapped: 12/12. Zero unmapped.**

---

_Roadmap created: 2026-03-20_
