# Technical Debt & Concerns Report: opendeviationbar-py

**Date**: 2026-03-19
**Codebase Size**: 33.6K LOC (Python), 2.7K LOC (sidecar alone), 10 Rust crates
**Complexity Areas**: Streaming architecture, gap reconciliation, memory management

---

## HIGH PRIORITY CONCERNS

### 1. Sidecar Complexity & Fragility (HIGH)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/sidecar.py` (2,655 lines)

**Issue**: Monolithic 2,655-line file couples watchdog logic, streaming, checkpoint persistence, gap-fill, dead-letter handling, and HTTP API endpoints.

**Risks**:

- Single change cascades across multiple subsystems (watchdog, replay, checkpoint inject)
- Difficult to unit-test individual components
- High cognitive load for code review
- Checkpoint injection guards stale checkpoints (lines 301-335) but relies on ClickHouse availability to validate
- Multiple fallback paths for missing data create subtle interaction bugs

**References**:

- Line 1: "FILE-SIZE-OK: Watchdog logic (Issue #107) is tightly coupled with main loop"
- Lines 2378-2655: Kintsugi module is also large (2,378 lines)

**Recommendation**: Plan modularization: split into `streaming.py`, `checkpoint.py`, `dead_letter.py`, `http_api.py`, `health.py` (Phase 4 of Issue #178 unified engine).

---

### 2. REST→WS Junction Forming-Bar Abandonment Bug (HIGH)

**Files**:

- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/sidecar.py` (lines 1180-1200)
- `/Users/terryli/eon/opendeviationbar-py/.claude/projects/-Users-terryli-eon-opendeviationbar-py/memory/rest-ws-junction-fix.md`

**Issue**: When sidecar restarts, the REST→WS transition at `fill_from_rest()` can abandon the forming bar being processed at the junction.

**Root Cause**:

- Old approach (pre-v13.29): Fresh processor via `backfill_gap()` → different state from WS processor
- Creating a "forming-bar abandonment gap" (trades not consumed by either processor)
- WS continues with its processor, losing trade continuity

**v13.29 Fix**:

- `fill_from_rest()` now preserves same processor for WS pipeline
- Trades flow through same stateful processor → zero-gap restart
- See line 1180-1200: "Architecture: WS buffers in parallel while REST catches up"

**Residual Risk**:

- Cloudflare/network failures during junction can still create edge cases
- Dead-letter replay must fill these gaps via Charon (5-min retry) + kintsugi

**Recommendation**:

- Add instrumentation to detect forming-bar abandonment (per-symbol tid_delta anomaly detection)
- Audit for edge cases where `fill_from_rest()` early-returns without flushing all trades

---

### 3. Memory Pressure from OOM History (HIGH)

**Files**:

- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/backfill.py` (lines 900+)
- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/kintsugi.py` (memory checks)
- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/config/backfill.py` (memory config)

**History**:

- v13.30.0: Rust `fill_from_rest()` OOMed on large trade batches → reverted to chunked Python version (50K trades/chunk)
- v13.26: `reconstruct_today()` OOMed at 2.5GB with 18 symbols → function removed
- Multiple past incidents requiring MemoryMax tuning (2.5GB → 4GB → 6GB for seeder)

**Current Mitigations**:

- Dual-layer memory guards:
  - Soft: `_check_memory_watermark()` (65% → gc, 90% → abort)
  - Hard: systemd `MemoryMax` kills process on breach
- Chunked backfill every `OPENDEVIATIONBAR_FLUSH_THRESHOLD` bars (default 10K)
- Tier-1 Parquet lazy-loading instead of buffering full DataFrames

**Remaining Risks**:

- Multi-threshold fan-out (BTCUSDT@250/500/750/1000) can multiply memory usage
- Polars eager loading in `_process_day()` can spike unexpectedly
- Redis client instantiation (backfill.py line 73) creates implicit connection per call — could exhaust FDs
- Seeder module still tuning RSS (recent MEM-015-MEM-024 optimizations target 2.5GB from 7GB)

**Recommendation**:

- Implement per-day memory budgeting (split long ranges into smaller day chunks earlier)
- Add memory telemetry dashboard (RSS %, arena fragmentation)
- Consider limiting multi-threshold parallelism

---

### 4. Asclepius Permanently Disabled (HIGH)

**Files**:

- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/sidecar.py` (comments)
- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/kintsugi.py` (lines ~400-500)

**Issue**: Issue #284 removed all intra-day gap-fill from sidecar (`Asclepius fill_gap()` on live engine).

**Why Disabled**:

- `fill_gap()` on advancing processor creates overlapping bars (stateless new processor ≠ streaming processor state)
- v13.21 incident: 665 overlaps + 169 gaps in 10 hours during live gap-fill attempt
- Only safe patterns:
  - Puck: inline REST fill on WS discontinuity (same processor, immediate)
  - Kintsugi: tomorrow's complete day backfill (no live engine interference)

**Current Workaround**:

- Sidecar gaps → Puck (inline, <3s)
- Old gaps → deferred to kintsugi next day
- Heartbeat monitors and Kintsugi heals today's gaps (v13.11.1: fixed health logic inverted line)

**Risk**:

- If gap persists through end-of-day boundary → must wait for kintsugi (5+ hours)
- Live trading systems relying on today's data miss the gap

**Recommendation**:

- Upstream: Add midnight reset to Rust `LiveBarEngine` (Issue pending)
- Emit heartbeat alert if gap persists > 60 min during trading hours
- Document as "known limitation: gaps >1h fixed on T+1 by kintsugi"

---

### 5. Cross-Midnight Bar Handling Complexity (HIGH)

**Files**:

- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/sidecar.py` (lines 2415+)
- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/clickhouse/bulk_operations.py` (lines 617, 949)
- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/kintsugi.py` (lines 2308+)

**Issue**: Issue #264/#263 — bars straddling UTC midnight violate day-mode invariant, causing cascading retry storms.

**v13.10.0 Fixes** (3-layer defense):

| Layer  | Mechanism                            | Impact                                                            |
| ------ | ------------------------------------ | ----------------------------------------------------------------- |
| **L1** | Stream filter in sidecar (line 2415) | Skip cross-midnight bars before batch accumulation                |
| **L2** | Pre-write gate (bulk_operations.py)  | Changed from raise → filter + warn; filter bars, write valid ones |
| **L3** | Dead-letter cleanup (kintsugi.py)    | Detect unplayable bars (all filtered) and delete instead of retry |

**Residual Risks**:

- If all bars in a batch are cross-midnight → batch becomes empty → 0 rows written → checkpoint may not advance consistently
- Rust-side midnight reset still missing (LiveBarEngine doesn't reset at day boundary)
- Stathera gap detection must account for legitimate day boundaries vs. real gaps

**Recommendation**:

- Implement Rust-side `maybe_reset_at_midnight()` in `LiveBarEngine` (separate issue)
- Add batch-level telemetry: track filtered-out counts per day
- Audit `checkpoint.py` for edge cases where empty batches leave checkpoints in limbo

---

## MEDIUM PRIORITY CONCERNS

### 6. Writer Ownership Model Requires Absolute Discipline (MEDIUM)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/kintsugi.py` (writer ownership checks)

**Issue**: Issue #280 — Sidecar owns today (UTC), Kintsugi owns yesterday+. Clean boundary prevents races, but requires NO exceptions.

**Current State**:

- Kintsugi today-guard is ABSOLUTE (no interior-gap exception)
- All gaps touching today clamp to yesterday or defer (returns -1 sentinel)
- Heartbeat fix: `stathera_all_gaps_known` str (not bool) so doesn't gate health

**Fragility**:

- If kintsugi accidentally repairs today's gap → concurrent writer race → ReplacingMergeTree confusion
- Sidecar checkpoints might become stale if kintsugi defers → next restart could re-process yesterday's data
- No automated enforcement: relies on code discipline

**Recommendation**:

- Add Redis-backed write lock per (symbol, threshold) for additional layer (already in place via `write_lock.py`)
- Add daily audit: detect if any kintsugi repairs touched today's data
- Document in runbook: "If sidecar + kintsugi both write today → delete ClickHouse data for today manually + restart"

---

### 7. Symbol Registry Gate Coverage (MEDIUM)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/symbol_registry.py` (839 lines)

**Issue**: Symbol registry (Issue #79) must be validated at all entry points. Risk: disabled symbols can slip through.

**Gaps**:

- `populate_cache_resumable()` calls `validate_symbol_registered()` BUT gate is "off" during tests
- v13.11.0 fixed: tests now use `filter_pairs_by_registry()` to pre-filter disabled symbols
- v13.11.1 fix: added `filter_pairs_by_registry()` to `_gap_fill()` to skip disabled symbols on sidecar startup

**Remaining Risk**:

- If a symbol is disabled mid-operation (e.g., USDC bridge token deprecated) → already-running sidecars keep streaming
- No runtime reload of `symbols.toml` — requires restart to apply registry changes

**Recommendation**:

- Add watch-mode to symbol registry (detect `symbols.toml` changes, hot-reload)
- Sidecar startup to check if ANY configured symbol is newly disabled → warn + skip

---

### 8. Checkpoint Staleness Detection (MEDIUM)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/sidecar.py` (lines 301-335)

**Issue**: Checkpoints saved during a previous session may reference trades that are no longer in ClickHouse (manual deletes, corruption).

**Current Guard**:

- Check if `cp_first_tid > ch_last_tid + 1` → reject checkpoint
- Preserve file (don't auto-delete) for next restart

**Weakness**:

- Relies on ClickHouse being available (`try/except` on CH query)
- If CH unavailable → injected anyway ("better than nothing")
- Edge case: checkpoint has `first_agg_trade_id=0` (no trade ID) → guard skipped, could be incorrect

**Recommendation**:

- Require ClickHouse query success (don't inject on CH unavailable)
- Add minimum freshness requirement: `cp_age_s < 7 days` (reject stale checkpoint files)
- Log checkpoint injection to kintsugi_log for debugging

---

### 9. ClickHouse Connection Pool Exhaustion Risk (MEDIUM)

**Files**:

- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/clickhouse/connection_pool.py`
- `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/backfill.py` (implicit Redis connections)

**Issue**: Each `_get_redis_client()` call creates new connection without pooling (backfill.py lines 71-82).

**Risk**:

- FD exhaustion if many concurrent backfill/kintsugi processes
- Redis errors silently degrade to no negative caching

**Recommendation**:

- Use Redis connection pooling (singleton pattern with `redis.ConnectionPool`)
- Add FD monitoring to systemd service

---

### 10. Polars Deprecation Risk (MEDIUM)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/storage/parquet.py` (764 lines)

**Issue**: Polars deprecated `is_in()` method (v0.20+). v13.17.0 changelog mentions avoiding deprecation path.

**Current**: Using anti-join instead of `is_in()` (MEM-015 optimization)

**Risk**:

- Polars version upgrades could break code silently
- Arrow compatibility also evolving (Arrow export paths added v13.x)

**Recommendation**:

- Pin Polars version in `pyproject.toml` (currently likely unpinned or loosely pinned)
- Add CI test: attempt upgrade to next major Polars version
- Monitor Polars changelog for breaking changes

---

## LOW PRIORITY CONCERNS

### 11. Large Script Collection (LOW)

**Directory**: `/scripts/` (140 Python files)

**Issue**: 140 scripts, many research/audit related. Risk of dead code.

**Examples**:

- Research scripts: `tda_*.py`, `regime_analysis.py`, etc. (20+ files)
- Audit scripts: `validate_*.py`, `combined_*.py` (30+ files)
- Maintenance: `kintsugi.py`, `populate_*.py`, `tick_cache_seeder.py`, etc.

**Impact**: Low (scripts are not imported by production code) but makes codebase harder to navigate.

**Recommendation**:

- Create `scripts/ARCHIVE/` and move research/one-off audit scripts there
- Keep only operational scripts (kintsugi, populate, monitoring, heartbeat, seeder)

---

### 12. Error Handling in HTTP API (LOW)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/sidecar.py` (lines 1300-1600)

**Issue**: HTTP endpoints (`/ariadne`, `/trades/gap-fill`, `/catchup`) catch broad `Exception` and return 500.

**Risks**:

- Bugs in error paths are invisible (swallowed by catch-all)
- Rate limiting (`_gap_fill_bucket`, `_catchup_bucket`) could silently fail if bucket creation breaks

**Recommendation**:

- Log full traceback for 500 errors (currently only `logger.exception()` which may not be captured)
- Add circuit breaker for endpoint failures (3 failures → 60s backoff)

---

### 13. Dead Code from Asclepius Removal (LOW)

**Files**: Various (sidecar.py, kintsugi.py)

**Issue**: Comments reference old Asclepius `fill_gap()` pattern. Code likely still present but unreachable.

**Examples**:

- `reconstruct_today()` comment in sidecar.py (removed in v13.26)
- Old "intra-day self-heal" references

**Recommendation**:

- Audit for dead function definitions that reference Asclepius
- Remove unreachable code paths

---

### 14. SQL Query Construction (LOW)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/clickhouse/query_operations.py` (lines 149, 176, 653)

**Issue**: Uses f-strings for SQL query construction (e.g., `f"SELECT ... WHERE symbol = '{symbol}'"`).

**Risk**: SQL injection if symbol/threshold parameters not validated.

**Current State**:

- Symbols validated via registry (issue #79)
- Thresholds are integers (safe from injection)
- Parameters come from API requests (parse_qs → validate before use)

**Recommendation**:

- Replace f-strings with parameterized queries (ClickHouse `pyformat` style: `?`)
- Add SQL query audit script to detect f-string patterns

---

### 15. Rate Limiting Implementation (LOW)

**File**: `/Users/terryli/eon/opendeviationbar-py/python/opendeviationbar/sidecar.py` (lines 1400-1410)

**Issue**: Token bucket rate limiter (`_gap_fill_bucket`, `_catchup_bucket`) implemented manually.

**Implementation**:

- `try_acquire()` calls tokens; 429 on failure
- Fixed capacity (likely hard-coded)

**Risk**:

- No metrics/monitoring of rate limit hits
- Capacity may be wrong for production load
- No way to adjust without code change + restart

**Recommendation**:

- Make bucket capacity configurable via env var
- Add Prometheus metrics: `sidecar_ratelimit_hits_total` per endpoint

---

## SUMMARY TABLE

| #   | Category        | Severity | Issue                                      | File(s)                  | Status                  |
| --- | --------------- | -------- | ------------------------------------------ | ------------------------ | ----------------------- |
| 1   | Architecture    | HIGH     | Sidecar monolith (2,655 LOC)               | sidecar.py               | Tracked: Issue #178     |
| 2   | Data Integrity  | HIGH     | REST→WS junction forming-bar abandonment   | sidecar.py L1180         | Fixed v13.29            |
| 3   | Operations      | HIGH     | Memory OOM history + current pressure      | backfill.py, kintsugi.py | Ongoing monitoring      |
| 4   | Operations      | HIGH     | Asclepius disabled (no intra-day gap fix)  | kintsugi.py              | Tracked: Issue #284     |
| 5   | Data Integrity  | HIGH     | Cross-midnight bars cascade fix (v13.10.0) | sidecar.py L2415         | Fixed, L1-L3 defense    |
| 6   | Architecture    | MEDIUM   | Writer ownership requires discipline       | kintsugi.py              | Tracked: Issue #280     |
| 7   | Operations      | MEDIUM   | Symbol registry coverage incomplete        | symbol_registry.py       | Fixed v13.11.0/v13.11.1 |
| 8   | Operations      | MEDIUM   | Checkpoint staleness detection weak        | sidecar.py L301          | Needs hardening         |
| 9   | Operations      | MEDIUM   | ClickHouse connection pool risk            | backfill.py L71          | Needs pooling           |
| 10  | Dependencies    | MEDIUM   | Polars deprecation risk                    | parquet.py               | Ongoing                 |
| 11  | Codebase Health | LOW      | 140 scripts, many dead                     | scripts/                 | Needs cleanup           |
| 12  | API             | LOW      | Broad exception catching in HTTP           | sidecar.py L1500         | Minor                   |
| 13  | Code Quality    | LOW      | Dead code from Asclepius                   | Various                  | Needs audit             |
| 14  | Security        | LOW      | SQL f-string construction                  | query_operations.py      | Low risk (validated)    |
| 15  | API             | LOW      | Rate limiter not configurable              | sidecar.py L1400         | Minor                   |

---

## RECOMMENDATIONS (Priority Order)

### Immediate (This Sprint)

1. **Add REST→WS junction telemetry** — Monitor for forming-bar abandonment gaps
2. **Audit checkpoint staleness** — Make CH query required (don't inject on unavailable)
3. **Document Asclepius limitation** — Update runbook with "gaps >1h fixed T+1" policy

### Next Sprint

1. **Modularize sidecar** — Split into 5 modules (start Phase 4 unified engine)
2. **Memory telemetry dashboard** — Add RSS/arena fragmentation monitoring
3. **Symbol registry hot-reload** — Watch `symbols.toml`, reload without restart

### Medium Term

1. **Rust-side midnight reset** — Upstream `LiveBarEngine` midnight boundary fix
2. **Redis connection pooling** — Prevent FD exhaustion
3. **SQL parameterization audit** — Replace f-strings with safe queries

### Long Term

1. **Script archive cleanup** — Move research scripts to `/scripts/ARCHIVE/`
2. **Polars version pinning** — Lock version, CI upgrade testing
3. **Observability improvements** — Structured logging, metrics export

---

## APPENDIX: Issue Reference Map

**Known Issues Referenced in Code**:

- Issue #59 — Exness inter-bar features (TODO)
- Issue #79 — Symbol registry gate (v13.11.0+ fixed)
- Issue #88 — Volume overflow detection (validation added)
- Issue #96 — Arrow export + junction issues (fixed v13.29)
- Issue #107 — Sidecar watchdog coupling (acknowledged, awaiting refactor)
- Issue #111 — Ariadne trade-ID lookup (implemented #217)
- Issue #178 — Unified engine (phases 0-3 complete, 4-7 pending)
- Issue #213-#228 — Consumer API (completed)
- Issue #257 — Rust gap detection (fixed v13.29)
- Issue #264/#263 — Cross-midnight bars (fixed v13.10.0)
- Issue #279 — Per-symbol → combined WS (planned, not critical)
- Issue #280 — Writer ownership model (sealed v13.5.2+)
- Issue #284 — Asclepius permanent disable (sealed)

**Ended Issues (Closed)**:

- #238 — Process consolidation (3 phases, all complete v12.65.0)
- #281 — Kintsugi seal guard
- #282 — Asclepius failure modes
- #283 — Documentation updates

---

## Data Sources

This report gathered concerns from:

- Source code grep: TODOs, FIXMEs, Issue references
- File size analysis: large files (sidecar 2,655 LOC, kintsugi 2,378 LOC, checkpoint 2,195 LOC)
- CHANGELOG.md: OOM history, breaking changes, recent fixes
- CLAUDE.md network: project memory, known limitations, principles
- Memory guards: watermark checks, MemoryMax tuning
- Exception handling: broad catches, error propagation paths
- API surface: endpoint rate limiting, validation gaps
