# Phase 19: Fleet Recompute + Telemetry - Research

**Researched:** 2026-03-27
**Domain:** ClickHouse data migration, systemd/monit operations, heartbeat telemetry
**Confidence:** HIGH

## Summary

Phase 19 is a purely operational/infrastructure phase. All 30 crypto symbols already have `ouroboros_mode = "aion"` in `symbols.toml` (Phase 13 REG-01). The code changes for Aion mode (Phases 14-18) provide the runtime behavior. This phase executes the fleet-wide data migration: drop all existing Day-mode crypto bar partitions from ClickHouse, recompute as Aion, update operational artifacts (deploy env, monit, heartbeat messages, diagnostic scripts), and wipe stale Day-mode telemetry.

The critical risk is the recompute duration (~1.3h estimated for 18 enabled crypto symbols x 4 thresholds = 72 partition pairs). The DROP PARTITION operation itself is instant (metadata-only), but the subsequent `populate_cache_resumable()` or kintsugi-driven recompute is the bottleneck. The sidecar must be stopped during recompute to prevent concurrent writes to the same partitions.

**Primary recommendation:** Execute as a coordinated stop-drop-recompute-verify-restart sequence on bigblack. Use `DROP PARTITION` (instant, metadata-only) for deletion. Recompute via kintsugi catch-up mode (handles all symbols, all thresholds, parallelized). Verify via heartbeat Stathera check before restarting the sidecar.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all implementation choices at Claude's discretion (infrastructure phase).

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure/operational phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

### Deferred Ideas (OUT OF SCOPE)

None -- infrastructure phase stays within phase scope.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID        | Description                                                                                          | Research Support                                                                                                                                                                                                  |
| --------- | ---------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| RECOMP-01 | DELETE all Day-mode crypto bars via DROP PARTITION per (symbol, threshold) pair                      | DROP PARTITION is instant metadata-only op; partition key is `(symbol, threshold_decimal_bps)` -- 72 crypto partitions (18 enabled symbols x 4 thresholds)                                                        |
| RECOMP-02 | Full fleet recompute on bigblack with Aion mode for all 18 crypto symbols                            | Kintsugi catch-up mode handles parallelized recompute; symbols.toml already has `ouroboros_mode = "aion"` for all crypto                                                                                          |
| RECOMP-03 | Wipe Day-mode telemetry artifacts (kintsugi_log, population_checkpoints, heartbeat state for crypto) | Three CH tables + local filesystem state files need cleanup                                                                                                                                                       |
| RECOMP-04 | Update monit monitors, systemd configs, Telegram heartbeat message formats for Aion                  | `deploy/opendeviationbar.env` has `OPENDEVIATIONBAR_OUROBOROS_MODE=day`; heartbeat formatter references "orphan" for Day-mode; monit conf is mode-agnostic (no changes needed)                                    |
| RECOMP-05 | All internal Day-mode references for crypto symbols removed/updated                                  | `detect_gaps.py`, `populate_full_cache.py`, `populate_range.py` use `_resolve_ouroboros_mode()` from env var; `day_mode/oracle.py` queries hardcode `ouroboros_mode = 'day'`; `yesterday.py` already handles Aion |
| VAL-04    | Heartbeat confirms Stathera 0-gap 0-overlap for all Aion crypto symbols after recompute              | `stathera.py` check is mode-agnostic (trade-ID continuity works identically); `yesterday.py` already routes Aion to freshness check                                                                               |

</phase_requirements>

## Standard Stack

### Core

| Library            | Version    | Purpose                                        | Why Standard                    |
| ------------------ | ---------- | ---------------------------------------------- | ------------------------------- |
| clickhouse-connect | (existing) | ClickHouse client for DROP PARTITION + queries | Already in project dependencies |
| systemctl          | (system)   | Service management during recompute            | Standard systemd user services  |
| monit              | (system)   | Watchdog management during maintenance         | Already deployed on bigblack    |

No new dependencies required. This phase uses exclusively existing infrastructure.

## Architecture Patterns

### Recompute Sequence (Stop-Drop-Recompute-Verify-Restart)

The recompute must follow a strict sequence to avoid data races:

```
1. Stop sidecar (prevents new writes during migration)
2. Stop kintsugi (prevents concurrent repair writes)
3. DROP PARTITION for all 72 crypto partitions (instant)
4. Wipe telemetry tables (kintsugi_log, population_checkpoints for crypto)
5. Wipe local heartbeat state files
6. Update deploy/opendeviationbar.env (remove OPENDEVIATIONBAR_OUROBOROS_MODE=day)
7. Start kintsugi catch-up (recompute from scratch, Aion mode)
8. Wait for recompute completion (~1.3h)
9. Run Stathera audit (verify 0-gap 0-overlap)
10. Start sidecar (resume real-time streaming)
```

### DROP PARTITION Pattern

The partition key is `(symbol, threshold_decimal_bps)` -- 64-72 total partitions. For crypto-only drop:

```sql
-- Per (symbol, threshold) pair -- instant metadata-only operation
ALTER TABLE opendeviationbar_cache.open_deviation_bars
    DROP PARTITION ('BTCUSDT', 250);
ALTER TABLE opendeviationbar_cache.open_deviation_bars
    DROP PARTITION ('BTCUSDT', 500);
-- ... repeat for all 18 symbols x 4 thresholds = 72 partitions
```

**Source:** CLAUDE.md Section 7 (ClickHouse DELETE strategies) -- DROP PARTITION is the fastest strategy, ranked #1.

### Environment Variable Update

`deploy/opendeviationbar.env` line 11 currently sets:

```ini
OPENDEVIATIONBAR_OUROBOROS_MODE=day
```

This global env var is used by `get_operational_ouroboros_mode()` as fallback. With Aion being the default for crypto (per-symbol registry), and Day remaining for forex, the correct approach is:

- **Remove** `OPENDEVIATIONBAR_OUROBOROS_MODE=day` from `opendeviationbar.env` entirely, OR
- **Keep it** as a fallback default since `symbols.toml` per-symbol `ouroboros_mode` takes precedence in all modern code paths

The safer approach: keep the env var but update scripts that use `_resolve_ouroboros_mode()` to prefer per-symbol registry lookup over global env var.

### Telemetry Artifact Cleanup

Three categories of stale artifacts:

| Artifact                            | Location                                                   | Cleanup Method                                                 |
| ----------------------------------- | ---------------------------------------------------------- | -------------------------------------------------------------- |
| `kintsugi_log` entries for crypto   | ClickHouse table                                           | `ALTER TABLE DELETE WHERE symbol IN (...)` with crypto symbols |
| `population_checkpoints` for crypto | ClickHouse table                                           | `ALTER TABLE DELETE WHERE symbol IN (...)` with crypto symbols |
| `backfill_requests` for crypto      | ClickHouse table                                           | `ALTER TABLE DELETE WHERE symbol IN (...)` with crypto symbols |
| Heartbeat Stathera state            | `~/.cache/opendeviationbar/heartbeat/stathera_counts.json` | Delete file (regenerated on next run)                          |
| Local checkpoints                   | `~/.cache/opendeviationbar/checkpoints/`                   | Delete crypto-related checkpoint files                         |

### Diagnostic Script Updates (RECOMP-05)

Files that reference Day-mode for crypto and need updating:

| File                                                         | Current Behavior                                                                                | Required Change                                                                                   |
| ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `scripts/detect_gaps.py`                                     | Uses `_resolve_ouroboros_mode()` from env var, hardcodes `ouroboros_mode = 'day'` in some paths | Should respect per-symbol registry; add `--ouroboros-mode` CLI flag or auto-detect from registry  |
| `scripts/populate_full_cache.py`                             | Accepts `--ouroboros` CLI flag, defaults to env var                                             | Should auto-detect from registry when not specified                                               |
| `scripts/populate_range.py`                                  | Accepts `--ouroboros` CLI flag                                                                  | Should auto-detect from registry                                                                  |
| `python/opendeviationbar/validation/day_mode/oracle.py`      | Queries hardcode `ouroboros_mode = 'day'`                                                       | Already correct -- day_mode oracle is specific to Day-mode symbols only (forex). No change needed |
| `python/opendeviationbar/validation/day_mode/guards.py`      | Pre-write gate for Day-mode bars                                                                | Already conditional on mode (MON-02 from Phase 18)                                                |
| `python/opendeviationbar/validation/day_mode/observation.py` | CLI observation tool                                                                            | Already Day-mode specific -- no change needed for Aion migration                                  |
| `deploy/opendeviationbar.env`                                | `OPENDEVIATIONBAR_OUROBOROS_MODE=day`                                                           | Remove or update comment to clarify it's forex-only fallback                                      |
| `deploy/CLAUDE.md`                                           | Documents `OPENDEVIATIONBAR_OUROBOROS_MODE=day`                                                 | Update documentation                                                                              |
| `scripts/heartbeat/formatter.py`                             | Remediation text: "orphan bar doesn't match crossover TID"                                      | Already handles Aion via yesterday.py routing                                                     |

### Schema Comment Update

The ClickHouse schema (`schema.sql`) has:

```sql
-- Day mode is the only supported mode.
ouroboros_mode LowCardinality(String) DEFAULT 'day',
```

This comment needs updating to reflect Aion support. The DEFAULT can remain 'day' for backward compatibility (new bars from the sidecar will explicitly set the mode from per-symbol registry).

## Don't Hand-Roll

| Problem               | Don't Build                     | Use Instead                                              | Why                                                                                                         |
| --------------------- | ------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Bar recompute         | Custom recompute script         | Kintsugi catch-up (`mise run kintsugi:catchup`)          | Kintsugi already handles parallel recompute with all safety guards (Stathera, write locks, circuit breaker) |
| Partition enumeration | Hardcoded list of 72 partitions | Query `system.parts` for crypto partitions               | Dynamic, handles any future symbol additions                                                                |
| Stathera verification | Custom post-recompute audit     | Heartbeat Stathera check (`scripts/health_heartbeat.py`) | Already implements the exact VAL-04 requirement                                                             |
| Telemetry cleanup     | Per-row DELETE loops            | Batch `ALTER TABLE DELETE WHERE symbol IN (...)`         | ClickHouse-efficient batch deletion                                                                         |

## Common Pitfalls

### Pitfall 1: Recompute Without Stopping Sidecar

**What goes wrong:** Sidecar writes today's bars while kintsugi is recomputing historical data. The clean-slate DELETE in sidecar startup conflicts with kintsugi's writes.
**Why it happens:** Forgetting the writer ownership model (sidecar owns today, kintsugi owns yesterday+).
**How to avoid:** Stop sidecar FIRST, then kintsugi, then DROP PARTITION, then start kintsugi for recompute. Only restart sidecar AFTER recompute verification.
**Warning signs:** Stathera gaps appearing during recompute, circuit breaker tripping.

### Pitfall 2: DROP PARTITION Leaves Forex Data Intact

**What goes wrong:** Accidentally dropping forex symbol partitions (EURUSD, GBPUSD, etc.) which should remain Day-mode.
**Why it happens:** Using a blanket "drop all partitions" instead of crypto-only.
**How to avoid:** Enumerate partitions from `system.parts WHERE partition LIKE '%USDT%'` or explicitly list only the 18 enabled crypto symbols.
**Warning signs:** Forex bar counts dropping to 0 after DROP.

### Pitfall 3: Global OUROBOROS_MODE Env Var Conflicts with Per-Symbol Registry

**What goes wrong:** Scripts using `get_operational_ouroboros_mode()` return "day" from env var, overriding per-symbol registry "aion" setting.
**Why it happens:** The env var `OPENDEVIATIONBAR_OUROBOROS_MODE=day` in `deploy/opendeviationbar.env` takes precedence in scripts that use the global resolver.
**How to avoid:** Scripts should prefer per-symbol registry lookup. Keep env var only as a last-resort fallback for contexts where no symbol is available.
**Warning signs:** Recomputed bars having `ouroboros_mode='day'` instead of `'aion'` in ClickHouse.

### Pitfall 4: Stale KNOWN_GAP_FINGERPRINTS After Recompute

**What goes wrong:** After recompute, all bars have new trade-ID boundaries. The `KNOWN_GAP_FINGERPRINTS` in `heartbeat/config.py` reference specific old trade IDs that no longer apply.
**Why it happens:** Fingerprints are keyed by `(symbol, threshold, first_agg_trade_id, prev_last_tid)` -- after recompute these exact IDs may still exist but the gap pattern changes.
**How to avoid:** Clear `KNOWN_GAP_FINGERPRINTS` dict before recompute verification. Re-add only if gaps reappear.
**Warning signs:** Heartbeat showing "known root cause" blocks for gaps that shouldn't exist.

### Pitfall 5: Kintsugi Recompute Respects Today-Guard

**What goes wrong:** Kintsugi refuses to repair today's data due to writer ownership model (sidecar owns today).
**Why it happens:** The today-guard in kintsugi is absolute for Day-mode. For Aion mode (KINT-02), the guard is TID-based, not UTC-day-based.
**How to avoid:** Ensure kintsugi Aion repair path is active (Phase 17 KINT-01/KINT-02 already complete). Verify kintsugi catch-up processes crypto symbols with Aion strategy.
**Warning signs:** Kintsugi skipping recent bars, gaps remaining in the most recent data.

## Code Examples

### DROP PARTITION for All Crypto Symbols

```python
# Source: CLAUDE.md Section 7 + schema.sql PARTITION BY
import clickhouse_connect

client = clickhouse_connect.get_client(host='localhost')

# Get all crypto symbols from registry
from opendeviationbar.symbol_registry import get_symbol_entries
entries = get_symbol_entries()
crypto_symbols = [
    name for name, entry in entries.items()
    if entry.asset_class == "crypto" and entry.exchange == "binance" and entry.market == "spot"
]

# Standard thresholds on bigblack
thresholds = [250, 500, 750, 1000]

for symbol in crypto_symbols:
    for threshold in thresholds:
        client.command(
            f"ALTER TABLE opendeviationbar_cache.open_deviation_bars "
            f"DROP PARTITION ('{symbol}', {threshold})"
        )
        print(f"Dropped partition ({symbol}, {threshold})")
```

### Wipe Crypto Telemetry from kintsugi_log

```sql
-- Batch DELETE with mutations_sync=1 (required for all DELETEs)
ALTER TABLE opendeviationbar_cache.kintsugi_log
DELETE WHERE symbol IN ('BTCUSDT','ETHUSDT','BNBUSDT','XRPUSDT','ADAUSDT',
    'DOGEUSDT','SOLUSDT','AVAXUSDT','LTCUSDT','LINKUSDT',
    'UNIUSDT','NEARUSDT','AAVEUSDT','BCHUSDT','FILUSDT','SUIUSDT',
    'DOTUSDT','SHIBUSDT')
SETTINGS mutations_sync = 1;
```

### Verify Stathera Post-Recompute

```sql
-- VAL-04: Should return 0 rows (no gaps, no overlaps)
SELECT symbol, threshold_decimal_bps,
       countIf(tid_delta < 0) AS overlaps,
       countIf(tid_delta > 0) AS gaps
FROM (
    SELECT symbol, threshold_decimal_bps,
           first_agg_trade_id - lagInFrame(last_agg_trade_id, 1)
               OVER (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode
                     ORDER BY first_agg_trade_id, close_time_us) - 1 AS tid_delta,
           row_number() OVER (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode
                              ORDER BY first_agg_trade_id, close_time_us) AS rn
    FROM opendeviationbar_cache.open_deviation_bars FINAL
    WHERE first_agg_trade_id > 0 AND last_agg_trade_id > 0
)
WHERE rn > 1 AND tid_delta != 0
GROUP BY symbol, threshold_decimal_bps
```

## State of the Art

| Old Approach                                 | Current Approach                            | When Changed             | Impact                                    |
| -------------------------------------------- | ------------------------------------------- | ------------------------ | ----------------------------------------- |
| Global `OPENDEVIATIONBAR_OUROBOROS_MODE=day` | Per-symbol `ouroboros_mode` in symbols.toml | Phase 13 (REG-01/REG-02) | Scripts should prefer registry            |
| Day-mode midnight reset for crypto           | Aion continuous mode                        | Phases 14-18             | No orphan bars, no midnight boundary gaps |
| Yesterday orphan completion check            | Rolling-window freshness check for Aion     | Phase 18 (MON-01)        | Heartbeat already routes correctly        |

## Open Questions

1. **BTCUSDT has `thresholds = [100, 250, 500, 750]` in symbols.toml but bigblack only caches [250, 500, 750, 1000]**
   - What we know: `OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD=250` in `.mise.toml` prevents 100 dbps processing. Some symbols may have different threshold lists.
   - What's unclear: Whether DROP PARTITION should target all thresholds in symbols.toml or only the 4 standard ones on bigblack.
   - Recommendation: Query `system.parts` to discover ACTUAL partitions on bigblack before dropping. This avoids both over-dropping (nonexistent partitions) and under-dropping (unexpected thresholds).

2. **18 vs 30 crypto symbols**
   - What we know: 30 symbols in symbols.toml have `ouroboros_mode = "aion"`. But only ~18 are enabled Binance spot symbols (the rest include disabled symbols, BTCUSD_PERP, etc.).
   - What's unclear: Whether disabled symbols have partitions on bigblack.
   - Recommendation: Query `system.parts` for all crypto-asset partitions. Drop what exists.

3. **Recompute duration estimate**
   - What we know: 18 symbols x 4 thresholds. Kintsugi at 24 concurrent workers does ~44 repairs/30s. Total depends on date range per symbol.
   - What's unclear: Exact recompute time depends on per-symbol data volume and Parquet cache coverage.
   - Recommendation: Start kintsugi catch-up and monitor via heartbeat. Estimate ~1-3 hours based on 42M bars.

## Environment Availability

| Dependency         | Required By                  | Available                        | Version      | Fallback                 |
| ------------------ | ---------------------------- | -------------------------------- | ------------ | ------------------------ |
| ClickHouse         | DROP PARTITION, recompute    | Yes (bigblack)                   | Native       | --                       |
| systemctl --user   | Service management           | Yes (bigblack)                   | systemd 247+ | --                       |
| monit              | Watchdog management          | Yes (bigblack)                   | --           | --                       |
| Redis              | Write locks during recompute | Yes (bigblack)                   | --           | Graceful degradation     |
| Parquet tick cache | Kintsugi recompute source    | Yes (bigblack, seeder populates) | --           | Vision download fallback |

**Missing dependencies with no fallback:** None.
**Missing dependencies with fallback:** None.

## Validation Architecture

### Test Framework

| Property           | Value                                   |
| ------------------ | --------------------------------------- |
| Framework          | pytest 9.0.2                            |
| Config file        | `pyproject.toml`                        |
| Quick run command  | `mise run test-py -- -x -k "test_name"` |
| Full suite command | `mise run test-py`                      |

### Phase Requirements -> Test Map

| Req ID    | Behavior                                | Test Type   | Automated Command                                   | File Exists?      |
| --------- | --------------------------------------- | ----------- | --------------------------------------------------- | ----------------- |
| RECOMP-01 | DROP PARTITION for crypto bars          | manual-only | SSH to bigblack, run DROP PARTITION script          | N/A (operational) |
| RECOMP-02 | Full fleet recompute with Aion mode     | manual-only | `mise run kintsugi:catchup` on bigblack             | N/A (operational) |
| RECOMP-03 | Wipe Day-mode telemetry artifacts       | manual-only | SQL DELETE + local file cleanup on bigblack         | N/A (operational) |
| RECOMP-04 | Update monit/systemd/heartbeat for Aion | unit        | `mise run test-py -- -x -k "heartbeat"`             | Yes (multiple)    |
| RECOMP-05 | Remove Day-mode crypto references       | unit        | `mise run test-py -- -x -k "day_mode or ouroboros"` | Yes               |
| VAL-04    | Stathera 0-gap 0-overlap post-recompute | manual-only | Heartbeat Stathera check on bigblack                | N/A (operational) |

### Sampling Rate

- **Per task commit:** `mise run test-py -- -x -k "heartbeat or day_mode or ouroboros"`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green + bigblack Stathera audit green

### Wave 0 Gaps

None -- existing test infrastructure covers all phase requirements. RECOMP-01/02/03/04(operational) are verified via bigblack post-recompute checks, not unit tests.

## Sources

### Primary (HIGH confidence)

- Codebase analysis: `schema.sql`, `stathera_query.py`, `deploy/opendeviationbar.env`, `symbols.toml`, `heartbeat/checks/yesterday.py`, `heartbeat/config.py`
- CLAUDE.md hub: Section 7 (ClickHouse DELETE strategies), Section 5 (Writer Ownership Model)

### Secondary (MEDIUM confidence)

- deploy/CLAUDE.md, scripts/systemd/CLAUDE.md, scripts/CLAUDE.md -- operational documentation

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - no new dependencies, all existing infrastructure
- Architecture: HIGH - recompute sequence follows established patterns (DROP PARTITION, kintsugi catch-up)
- Pitfalls: HIGH - based on documented production incidents and codebase conventions

**Research date:** 2026-03-27
**Valid until:** 2026-04-27 (stable infrastructure, no external dependency changes)
