---
name: ch-clean-slate
description: Rebuild ClickHouse open_deviation_bars from scratch using local Parquet tick cache and repair_direct_parquet.py. Use whenever the user wants to start fresh, nuke the database, rebuild from ground up, clean slate ClickHouse, or when the table has become corrupted beyond incremental repair (e.g., 50K+ unmerged parts, stuck mutations, epoch-zero partition pollution). Also use after partition key migrations.
---

# ClickHouse Clean Slate Rebuild

Stepwise procedure to rebuild the `open_deviation_bars` table from scratch. Drops everything, recreates from schema, and repopulates from the local Parquet tick cache using `repair_direct_parquet.py` (24-worker parallel, Arrow zero-copy, ~200 days/min).

**When to use**: Corruption beyond incremental repair, partition key migrations, stuck mutation queues, or when you just want a guaranteed-clean starting point.

**Time estimate**: ~2-3 hours for BTCUSDT+ETHUSDT (~3000 days each, 4 thresholds). Other symbols require kintsugi Vision downloads (hours to days depending on count).

---

## Phase 1: Kill Everything (Brute Force)

No waiting, no graceful shutdown. Kill all ODB processes, unmonitor everything, stop the heartbeat timer. Leave nothing running that could interfere.

```bash
# Unmonitor FIRST (prevents monit from restarting things we kill)
ssh bigblack 'sudo monit unmonitor all 2>/dev/null; echo "Monit unmonitored all"'

# Stop systemd services
ssh bigblack 'systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-kintsugi-catchup opendeviationbar-heartbeat.timer opendeviationbar-seeder.timer 2>/dev/null; echo "Services stopped"'

# Kill ANY surviving ODB process (SIGKILL, no mercy)
ssh bigblack 'pkill -9 -f "streaming_sidecar|kintsugi|heartbeat|seeder|repair_direct|populate_cache" 2>/dev/null; sleep 1; pgrep -af "streaming_sidecar|kintsugi|heartbeat|seeder|repair" | grep -v pgrep || echo "All killed"'
```

## Phase 2: Nuclear Drop

Restart ClickHouse to clear stuck mutations, then DROP all tables. `DROP TABLE` is instant (metadata-only) — no waiting for mutation queues to drain.

```bash
# Restart CH to clear mutation state
ssh bigblack 'sudo systemctl restart clickhouse-server && sleep 5 && curl -s "http://localhost:8123/ping" && echo " CH up"'

# DROP all ODB tables (instant, no mutations)
ssh bigblack 'clickhouse-client -q "
DROP TABLE IF EXISTS opendeviationbar_cache.open_deviation_bars SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.open_deviation_bars_monthly_backup SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.open_deviation_bars_backup SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.kintsugi_log SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.population_checkpoints SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.backfill_requests SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.session_labels SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.data_freshness SYNC;
DROP TABLE IF EXISTS opendeviationbar_cache.plugin_migrations SYNC;
" && echo "All tables dropped"'

# Verify empty
ssh bigblack 'clickhouse-client -q "SELECT name FROM system.tables WHERE database = '"'"'opendeviationbar_cache'"'"' ORDER BY name"'
```

**Why restart CH first**: Pending mutations from the old session block DROP TABLE. Restarting CH is the fastest way to clear the mutation queue. This is safe because we're dropping everything anyway.

## Phase 3: Recreate Schema

Recreate all tables from `schema.sql`. This is the single source of truth for table definitions including the partition key.

```bash
# Apply schema (uses clickhouse-client for multiquery support)
ssh bigblack 'clickhouse-client --multiquery < ~/opendeviationbar-py/python/opendeviationbar/clickhouse/schema.sql && echo "Schema applied"'

# Verify tables + partition key
ssh bigblack 'clickhouse-client -q "SELECT name FROM system.tables WHERE database = '"'"'opendeviationbar_cache'"'"' ORDER BY name"'
ssh bigblack 'clickhouse-client -q "SELECT partition_key FROM system.tables WHERE database = '"'"'opendeviationbar_cache'"'"' AND name = '"'"'open_deviation_bars'"'"'"'

# Confirm zero parts
ssh bigblack 'clickhouse-client -q "SELECT count() FROM system.parts WHERE database = '"'"'opendeviationbar_cache'"'"' AND table = '"'"'open_deviation_bars'"'"' AND active"'
```

The partition key should show `(symbol, threshold_decimal_bps)` — 64 partitions total (no time component).

## Phase 4: Repopulate from Parquet Tick Cache

Use `repair_direct_parquet.py` — the proven parallel repair tool. It reads from the local Parquet tick cache, processes through Rust (Arrow zero-copy), and writes via plain INSERT (no DELETE mutations).

### Check available Parquet cache

```bash
ssh bigblack 'for dir in ~/.cache/opendeviationbar/ticks/BINANCE_SPOT_*/; do
  sym=$(basename "$dir" | sed "s/BINANCE_SPOT_//")
  count=$(ls "$dir"/*.parquet 2>/dev/null | wc -l)
  if [ "$count" -gt 0 ]; then
    first=$(ls "$dir"/*.parquet | head -1 | xargs basename | sed "s/.parquet//")
    last=$(ls "$dir"/*.parquet | tail -1 | xargs basename | sed "s/.parquet//")
    echo "$sym: $count days ($first to $last)"
  fi
done'
```

### Run repairs: ONE symbol at a time

Run sequentially per symbol — NOT in parallel. Two concurrent repair processes (2x24 = 48 workers) exceed the ClickHouse query scheduler cap (~143 concurrent queries), causing contention collapse.

**Discover symbols from the registry** (no hardcoding):

```bash
# Get all enabled symbols with Parquet cache
ssh bigblack 'cd ~/opendeviationbar-py && .venv/bin/python3 -c "
from opendeviationbar.symbol_registry import get_registered_symbols, get_thresholds_for_symbol
for sym in get_registered_symbols(enabled_only=True):
    thresholds = get_thresholds_for_symbol(sym)
    print(f\"{sym} thresholds={list(thresholds)}\")
"'
```

**Run repair per symbol** (use the thresholds from the registry):

```bash
# Example for each symbol — adapt thresholds from registry output above
ssh bigblack 'cd ~/opendeviationbar-py && .venv/bin/python3 scripts/repair_direct_parquet.py \
    --execute --symbol BTCUSDT --threshold 100 250 500 750 --workers 24'
```

Wait for each symbol to complete before starting the next. At ~200 days/min:

- BTCUSDT (~3000 days): ~15 min
- ETHUSDT (~3000 days): ~15 min
- Symbols without Parquet cache: skip (kintsugi handles via Vision download after restart)

### Monitor progress

```bash
ssh bigblack 'clickhouse-client -q "
SELECT symbol, count() as bars, count(DISTINCT threshold_decimal_bps) as thresholds
FROM opendeviationbar_cache.open_deviation_bars
GROUP BY symbol ORDER BY bars DESC
"'
```

## Phase 5: OPTIMIZE TABLE FINAL

After bulk inserts, many small parts accumulate. OPTIMIZE FINAL merges them down for faster queries and mutations. This step is MANDATORY — without it, kintsugi mutations scan all parts and time out.

```bash
ssh bigblack 'clickhouse-client -q "OPTIMIZE TABLE opendeviationbar_cache.open_deviation_bars FINAL" && echo "OPTIMIZE done"'

# Verify reasonable parts count
ssh bigblack 'clickhouse-client -q "SELECT count() FROM system.parts WHERE database = '"'"'opendeviationbar_cache'"'"' AND table = '"'"'open_deviation_bars'"'"' AND active"'
```

**Learned 2026-03-23**: Skipping this step after a partition migration left 56K unmerged parts. Every kintsugi mutation timed out at 300s for 5+ hours. The parts count is the root cause, not the partition count.

## Phase 6: Restart Services

```bash
# Start sidecar first (live streaming)
ssh bigblack 'systemctl --user start opendeviationbar-sidecar'

# Wait for sync_flush before starting kintsugi (#295)
echo "Waiting for sidecar sync_flush..."
for i in $(seq 1 30); do
  if ssh bigblack 'journalctl --user -u opendeviationbar-sidecar --since "5 min ago" --no-pager 2>/dev/null | grep -q "sync_flush"'; then
    echo "sync_flush complete"
    break
  fi
  echo "  waiting... ($i/30)"
  sleep 10
done

# Start kintsugi (will discover and repair gaps for symbols without Parquet cache)
ssh bigblack 'systemctl --user start opendeviationbar-kintsugi'

# Re-enable monit
ssh bigblack 'sudo monit monitor sidecar 2>/dev/null; sudo monit monitor kintsugi 2>/dev/null'
```

## Phase 7: Verify

```bash
# Version alignment
ssh bigblack 'VER=$(.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)") && echo "Version: $VER"'

# Bar counts
ssh bigblack 'clickhouse-client -q "
SELECT symbol, threshold_decimal_bps, count() as bars
FROM opendeviationbar_cache.open_deviation_bars FINAL
GROUP BY symbol, threshold_decimal_bps
ORDER BY symbol, threshold_decimal_bps
"'

# Monit health
ssh bigblack 'sudo monit summary 2>/dev/null'

# Sidecar health
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -m json.tool'
```

Kintsugi will automatically discover and repair gaps for symbols that don't have local Parquet cache (downloads from Binance Vision). This is the steady-state gap-healing mechanism — no manual intervention needed.

---

## Anti-Patterns (Learned from Production)

| Anti-Pattern                                                | What Happens                                                         | Do Instead                                                  |
| ----------------------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------------- |
| Skip OPTIMIZE FINAL after bulk insert                       | 50K+ unmerged parts, all mutations time out at 300s                  | Always OPTIMIZE FINAL and verify parts <10K                 |
| Run 2+ repair processes simultaneously                      | 48+ workers exceed CH query scheduler cap (143), contention collapse | One symbol at a time, 24 workers max                        |
| Use `populate_cache_resumable(force_refresh=True)`          | Creates DELETE mutations that scan all parts                         | Use `repair_direct_parquet.py` (plain INSERT, no mutations) |
| Hardcode symbol list                                        | Misses new symbols, processes disabled ones                          | Read from `get_registered_symbols(enabled_only=True)`       |
| Use TRUNCATE TABLE with pending mutations                   | TRUNCATE blocks behind mutation queue, hangs forever                 | Restart CH first, then DROP TABLE (instant)                 |
| Use `/tmp` or `/var/tmp` for temp files during CH heavy I/O | Root filesystem goes transiently read-only                           | Use `~/.cache/opendeviationbar/`                            |

---

## What This Skill Does NOT Cover

- **Adding new symbols**: Use `/symbol-onboarding` skill
- **Sidecar restart recovery**: Use `/sidecar-restart-zero-overlap-zero-gap-recovery-playbook`
- **Incremental gap repair**: Use kintsugi daemon (this skill is for full rebuilds only)
- **Schema changes**: Edit `schema.sql` first, then use this skill to rebuild
