# Agent 3: Immediate Triage -- Safely Unblocking ClickHouse Mutation Deadlock

**Date**: 2026-03-21
**Situation**: 131 pending mutations, 30 stuck DELETEs (2,633+ sec), 120,450 parts, 7.84 GiB CH memory, 132 active queries. All kintsugi repairs blocked.

---

## 1. ClickHouse Mutation Mechanics (Research Findings)

### How Mutations Work in ClickHouse

ClickHouse mutations (`ALTER TABLE DELETE`, `DELETE FROM`) are **not** transactional. They are asynchronous background operations that rewrite data parts. Key behaviors:

1. **Mutations are ordered**: They execute sequentially per table (FIFO). Mutation #5 cannot start until #1-#4 complete.
2. **Each mutation scans all parts**: A DELETE mutation must check every data part to find matching rows. With 120,450 parts, each mutation scans ALL of them.
3. **`mutations_sync = 1` blocks the client**: The issuing query waits until the mutation completes. The mutation itself runs in the background mutation pipeline -- `mutations_sync` only controls whether the client blocks.
4. **Mutations queue behind each other**: 131 pending mutations means mutation #131 must wait for 1-130 to finish. This is the deadlock -- each mutation scans 120K parts, taking 40+ minutes each. 131 x 40 min = ~87 hours to drain.

### What Happens When You KILL a Mutation

`KILL MUTATION WHERE mutation_id = 'xxx'`:

- **Cancels the mutation** -- removes it from the queue.
- **Does NOT roll back**: If the mutation partially applied to some parts, those parts keep the change. But for DELETE mutations that have `read_rows=0`, **no parts have been modified yet** -- the mutation hasn't started processing.
- **Data safety**: Since these stuck mutations have `read_rows=0`, killing them means the DELETE never happened. The old data remains intact. This is **safe**.
- **Sequential unblock**: Killing mutation N allows mutation N+1 to proceed. To fully unblock, you must kill ALL stuck mutations in the queue.

### What Happens When You KILL a Query

`KILL QUERY WHERE query_id = 'xxx'`:

- Cancels the running query (the `client.command()` call in Python).
- For `mutations_sync = 1` queries, the **query** is killed but the **mutation** may remain in the queue.
- You need to kill BOTH: the query (to unblock the Python thread) AND the mutation (to unblock the mutation pipeline).

---

## 2. Root Cause Analysis

### Why 131 Mutations Piled Up

The `_do_replace()` function in `maintenance.py:443` issues:

```sql
DELETE FROM opendeviationbar_cache.open_deviation_bars
WHERE symbol = '...' AND threshold_decimal_bps = ...
  AND first_agg_trade_id <= ... AND last_agg_trade_id >= ...
SETTINGS lightweight_mutation_projection_mode = 'drop', mutations_sync = 1
```

With `mutations_sync = 1`, each kintsugi repair thread BLOCKS waiting for its DELETE to complete. But mutations are sequential -- only one runs at a time. With 24 concurrent kintsugi workers (`MAX_CONCURRENT_REPAIRS=24`), they all queue up:

1. Worker 1: DELETE (running, scanning 120K parts ~40 min)
2. Worker 2-24: DELETE (queued, blocked by Worker 1's `mutations_sync = 1`)
3. Worker 1 completes, Worker 2 starts, takes another 40 min...
4. Meanwhile, more kintsugi passes queue more DELETEs

**The 120,450 parts is the amplifier.** Normal operation has ~1,600-2,000 parts. At 120K parts, each mutation scan takes 50-100x longer than normal. This likely happened because:

- Heavy kintsugi repair activity created many small INSERT-only parts
- ClickHouse merges couldn't keep up with the insert rate
- Parts count snowballed, making each mutation slower, which made the queue grow longer

### The Cascade

```
High insert rate -> many parts -> slow mutations -> blocked workers ->
more queued mutations -> even slower -> deadlock
```

---

## 3. Data Safety Assessment

### Is There a Partial DELETE + No INSERT State?

**No, and here's why:**

The `_do_replace()` flow is:

1. `DELETE ... SETTINGS mutations_sync = 1` (blocks until DELETE done)
2. `store_bars_batch(...)` (INSERT new bars)
3. On INSERT failure: `_dead_letter_replace_failure()` saves to Parquet

Since the stuck DELETEs have `read_rows=0`, they haven't started. If we kill them:

- The DELETE never happened
- The INSERT never happened (Python thread is blocked waiting for DELETE)
- Old data remains intact
- Kintsugi will retry the shard on the next pass

**Key insight**: `mutations_sync = 1` means the Python code is stuck at step 1. Step 2 (INSERT) never executes. There is NO partial state.

### What About `_delete_today_bars()`?

The sidecar's `_delete_today_bars()` also uses `mutations_sync = 1`. If a sidecar restart happened during this deadlock, that DELETE is also stuck in the queue. Killing it means today's bars are NOT deleted, and the sidecar startup sequence will be inconsistent.

**Mitigation**: After clearing mutations, restart the sidecar. The startup sequence re-runs `_delete_today_bars()` fresh.

---

## 4. Recovery Plan: Order of Operations

### Why Stop Kintsugi First

If we kill mutations without stopping kintsugi, it will immediately:

1. Run a new `kintsugi_pass()`
2. Issue 24 more DELETEs via `_do_replace()`
3. Re-create the same deadlock

**Kintsugi must be stopped before clearing mutations.**

### Why NOT Stop the Sidecar

The sidecar handles real-time data. Stopping it causes data gaps that require kintsugi to fix later. The sidecar is only blocked if it tried to restart during the deadlock. If it's running normally (streaming WS data), leave it alone.

### Parts Compaction

After clearing mutations, the 120K parts must be compacted before re-enabling kintsugi. Otherwise the first new kintsugi DELETE will take 40+ min again. `OPTIMIZE TABLE ... FINAL` forces merges.

---

## 5. READY-TO-RUN Recovery Script

**REVIEW CAREFULLY BEFORE EXECUTING. DO NOT RUN BLINDLY.**

```bash
#!/usr/bin/env bash
# ============================================================
# ClickHouse Mutation Deadlock Recovery Script
# Date: 2026-03-21
# Situation: 131 pending mutations, 30 stuck DELETEs, 120K parts
#
# SAFETY: All stuck DELETEs have read_rows=0 (not started).
#         Killing them leaves old data intact.
#         Kintsugi will retry affected shards on next pass.
# ============================================================
set -euo pipefail

echo "=== PHASE 1: ASSESS (read-only) ==="

# 1a. Snapshot current state
echo "--- Pending mutations ---"
ssh bigblack "clickhouse-client --query \"
  SELECT mutation_id, command, create_time, parts_to_do, is_done
  FROM system.mutations
  WHERE database = 'opendeviationbar_cache'
    AND table = 'open_deviation_bars'
    AND is_done = 0
  ORDER BY create_time
  FORMAT PrettyCompactMonoBlock
\""

echo ""
echo "--- Stuck queries (>60s) ---"
ssh bigblack "clickhouse-client --query \"
  SELECT query_id, substring(query, 1, 120) as query_prefix,
         elapsed, read_rows, memory_usage
  FROM system.processes
  WHERE elapsed > 60
    AND query LIKE '%open_deviation_bars%'
  ORDER BY elapsed DESC
  FORMAT PrettyCompactMonoBlock
\""

echo ""
echo "--- Parts count ---"
ssh bigblack "clickhouse-client --query \"
  SELECT count() as parts,
         sum(rows) as total_rows,
         formatReadableSize(sum(bytes_on_disk)) as disk_size
  FROM system.parts
  WHERE database = 'opendeviationbar_cache'
    AND table = 'open_deviation_bars'
    AND active = 1
\""

echo ""
echo "--- Active merges ---"
ssh bigblack "clickhouse-client --query \"
  SELECT count() as active_merges,
         sum(num_parts) as parts_in_merge
  FROM system.merges
  WHERE database = 'opendeviationbar_cache'
    AND table = 'open_deviation_bars'
\""

echo ""
echo "=== PHASE 1 COMPLETE. Review output above. ==="
echo "Press Ctrl+C to abort, or Enter to proceed to Phase 2..."
read -r


echo "=== PHASE 2: STOP WRITERS ==="

# 2a. Stop kintsugi (prevents re-queuing DELETEs)
echo "Stopping kintsugi..."
ssh bigblack "systemctl --user stop opendeviationbar-kintsugi"

# 2b. Stop kintsugi-catchup if running
echo "Stopping kintsugi-catchup..."
ssh bigblack "systemctl --user stop opendeviationbar-kintsugi-catchup 2>/dev/null || true"

# 2c. Verify stopped
echo "Verifying kintsugi stopped..."
ssh bigblack "systemctl --user is-active opendeviationbar-kintsugi || true"
ssh bigblack "systemctl --user is-active opendeviationbar-kintsugi-catchup || true"

echo ""
echo "=== PHASE 2 COMPLETE. Kintsugi stopped. ==="
echo "Press Enter to proceed to Phase 3 (kill mutations)..."
read -r


echo "=== PHASE 3: KILL STUCK QUERIES ==="

# 3a. Kill all stuck DELETE queries on open_deviation_bars
# This unblocks the Python threads waiting on mutations_sync=1
echo "Killing stuck DELETE queries..."
ssh bigblack "clickhouse-client --query \"
  KILL QUERY WHERE query LIKE '%DELETE FROM opendeviationbar_cache.open_deviation_bars%'
    AND elapsed > 60
  ASYNC
\""

# Wait a moment for queries to die
sleep 3

# 3b. Verify queries killed
echo "Remaining long queries:"
ssh bigblack "clickhouse-client --query \"
  SELECT count() as stuck_queries
  FROM system.processes
  WHERE elapsed > 60
    AND query LIKE '%open_deviation_bars%'
\""

echo ""
echo "=== PHASE 3 COMPLETE. ==="
echo "Press Enter to proceed to Phase 4 (kill mutations)..."
read -r


echo "=== PHASE 4: KILL ALL PENDING MUTATIONS ==="

# 4a. Kill ALL pending mutations on the table
# This clears the entire mutation queue.
# Safe because: read_rows=0 means no partial application.
echo "Killing all pending mutations on open_deviation_bars..."
ssh bigblack "clickhouse-client --query \"
  KILL MUTATION WHERE database = 'opendeviationbar_cache'
    AND table = 'open_deviation_bars'
    AND is_done = 0
\""

sleep 2

# 4b. Verify mutations cleared
echo "Remaining pending mutations:"
ssh bigblack "clickhouse-client --query \"
  SELECT count() as pending_mutations
  FROM system.mutations
  WHERE database = 'opendeviationbar_cache'
    AND table = 'open_deviation_bars'
    AND is_done = 0
\""

# 4c. Also clear kintsugi_log mutations if any
echo "Checking kintsugi_log mutations..."
ssh bigblack "clickhouse-client --query \"
  KILL MUTATION WHERE database = 'opendeviationbar_cache'
    AND table = 'kintsugi_log'
    AND is_done = 0
\""

echo ""
echo "=== PHASE 4 COMPLETE. Mutation queue cleared. ==="
echo "Press Enter to proceed to Phase 5 (compact parts)..."
read -r


echo "=== PHASE 5: COMPACT PARTS ==="

# 5a. Check current parts count
echo "Parts before compaction:"
ssh bigblack "clickhouse-client --query \"
  SELECT count() as parts
  FROM system.parts
  WHERE database = 'opendeviationbar_cache'
    AND table = 'open_deviation_bars'
    AND active = 1
\""

# 5b. Trigger OPTIMIZE to merge parts
# This runs in background. We do NOT use FINAL (would take hours on 120K parts).
# Instead, let ClickHouse's natural merge strategy compact incrementally.
echo "Triggering background merges (non-blocking)..."
ssh bigblack "clickhouse-client --query \"
  OPTIMIZE TABLE opendeviationbar_cache.open_deviation_bars
\" --send_timeout 10 --receive_timeout 10 2>/dev/null || echo '(OPTIMIZE sent, continuing)'"

# 5c. Monitor merge progress (non-blocking check)
echo ""
echo "Active merges (will continue in background):"
ssh bigblack "clickhouse-client --query \"
  SELECT count() as active_merges,
         sum(num_parts) as merging_parts,
         formatReadableSize(sum(total_size_bytes_compressed)) as merging_size
  FROM system.merges
  WHERE database = 'opendeviationbar_cache'
    AND table = 'open_deviation_bars'
\""

echo ""
echo "=== PHASE 5 COMPLETE. ==="
echo ""
echo "=== IMPORTANT: DO NOT restart kintsugi yet. ==="
echo ""
echo "Wait for parts count to drop below 5,000 before restarting kintsugi."
echo "Monitor with:"
echo ""
echo "  ssh bigblack \"watch -n 10 'clickhouse-client --query \\\"SELECT count() FROM system.parts WHERE database=\\\\\\\"opendeviationbar_cache\\\\\\\" AND table=\\\\\\\"open_deviation_bars\\\\\\\" AND active=1\\\"'\""
echo ""
echo "Once parts < 5,000, restart kintsugi:"
echo "  ssh bigblack 'systemctl --user start opendeviationbar-kintsugi'"
echo ""
echo "If sidecar is also stuck, restart it:"
echo "  ssh bigblack 'systemctl --user restart opendeviationbar-sidecar'"
echo ""


echo "=== PHASE 6: VERIFY DATA INTEGRITY ==="

# 6a. Quick bar count sanity check
echo "Bar count by symbol:"
ssh bigblack "clickhouse-client --query \"
  SELECT symbol, threshold_decimal_bps, count() as bars,
         max(close_time_ms) as latest_close_ms
  FROM opendeviationbar_cache.open_deviation_bars FINAL
  GROUP BY symbol, threshold_decimal_bps
  ORDER BY symbol, threshold_decimal_bps
  FORMAT PrettyCompactMonoBlock
\""

echo ""
echo "=== RECOVERY COMPLETE ==="
echo ""
echo "Next steps:"
echo "1. Wait for parts count < 5,000 (monitor above)"
echo "2. Restart kintsugi"
echo "3. Watch next heartbeat (5 min) for Stathera status"
echo "4. Consider reducing MAX_CONCURRENT_REPAIRS to prevent recurrence"
```

---

## 6. Post-Recovery: Preventing Recurrence

### Root Cause: 120K Parts

The mutation deadlock is a symptom, not the cause. The real issue is 120,450 active parts. This happened because inserts outpaced merges. Prevention:

| Action                                 | Priority | Details                                                                              |
| -------------------------------------- | -------- | ------------------------------------------------------------------------------------ |
| **Monitor parts count**                | P0       | Add to heartbeat: alert if parts > 10,000                                            |
| **Reduce kintsugi concurrency**        | P1       | Lower `MAX_CONCURRENT_REPAIRS` from 24 to 12 temporarily until parts stabilize       |
| **Consider `max_parts_in_total`**      | P2       | ClickHouse setting to reject inserts when parts exceed threshold (prevents snowball) |
| **Investigate why merges fell behind** | P2       | Check `system.part_log` for merge failures, resource contention                      |

### Why `mutations_sync = 1` Amplifies the Problem

With `mutations_sync = 0` (async), the Python thread would return immediately and the mutation would run in background. The kintsugi worker would proceed to INSERT, which could cause the race condition documented in #293 (DELETE runs after INSERT, deleting new data).

The `mutations_sync = 1` fix is correct for data safety but creates the deadlock risk when mutations are slow. The solution is NOT to remove `mutations_sync = 1` but to **ensure parts count stays manageable** so mutations complete quickly.

### Alternative: Partitioned DELETE

Instead of a table-wide mutation, the DELETE in `_do_replace()` could use partition-level lightweight deletes:

```sql
ALTER TABLE ... DELETE ... IN PARTITION ...
```

This only scans parts in the target partition, not all 120K parts. But this requires knowing the partition key, and the current table uses `toYYYYMM(toDateTime(open_time_ms / 1000))` partitioning, which would need the date range of the affected bars.

---

## 7. Quick Reference: Key Commands

```bash
# Check mutation queue
ssh bigblack "clickhouse-client --query 'SELECT count() FROM system.mutations WHERE database=\"opendeviationbar_cache\" AND table=\"open_deviation_bars\" AND is_done=0'"

# Check parts count
ssh bigblack "clickhouse-client --query 'SELECT count() FROM system.parts WHERE database=\"opendeviationbar_cache\" AND table=\"open_deviation_bars\" AND active=1'"

# Check active merges
ssh bigblack "clickhouse-client --query 'SELECT count(), sum(num_parts) FROM system.merges WHERE database=\"opendeviationbar_cache\" AND table=\"open_deviation_bars\"'"

# Kill all pending mutations
ssh bigblack "clickhouse-client --query 'KILL MUTATION WHERE database=\"opendeviationbar_cache\" AND table=\"open_deviation_bars\" AND is_done=0'"

# Kill stuck queries
ssh bigblack "clickhouse-client --query \"KILL QUERY WHERE query LIKE '%DELETE FROM opendeviationbar_cache.open_deviation_bars%' AND elapsed > 60 ASYNC\""
```
