# Synthesis: ClickHouse Mutation Deadlock Investigation

**Date**: 2026-03-21
**Agent**: 9/9 (Synthesis)
**Sources**: 8 agent reports in `.planning/quick/ch-mutation-investigation/`

---

## 1. Root Cause

Kintsugi runs up to 24 concurrent shard repairs, each calling `_do_replace()` which issues a synchronous lightweight `DELETE FROM ... SETTINGS mutations_sync=1`. ClickHouse mutations are **totally ordered per table** (not per partition), so all 24 DELETEs serialize into a single FIFO queue. Each mutation must process all 120,450 active parts (vs the healthy ~5,760), taking 270-315 seconds per mutation. The 24th worker waits ~2,633 seconds for mutations 1-23 to complete. The `mutations_sync=1` setting is required for correctness (prevents the INSERT-before-DELETE race that caused bar loss in v13.39-v13.40), but it makes each Python thread block for the full mutation duration. The 120K part count is itself a symptom: heavy kintsugi INSERT activity created many small parts, merges fell behind, mutation scans slowed, which caused more mutations to queue, creating a positive feedback loop.

---

## 2. Why Previous Approaches Failed

| Approach                                | Phase               | Failure Mode                                                                                                                                                                                            | Lesson                                                                                                                                                               |
| --------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Async ALTER TABLE DELETE                | Phase 1 (pre-Mar 9) | Heavyweight mutations queued faster than processing. Workers crashed after 120s timeout. Monit restart = death spiral. 3-day outage.                                                                    | Heavyweight mutations (full part rewrite) are unacceptable under concurrency.                                                                                        |
| Async lightweight DELETE FROM           | Phase 2 (Mar 9-12)  | DELETE mask not visible before INSERT completes. `SELECT ... FINAL` shows both old and new bars. Stathera overlaps (tid_delta < 0).                                                                     | Async DELETE + immediate INSERT = race condition.                                                                                                                    |
| INSERT-only with RMT(computed_at) dedup | Phase 3 (Mar 21 AM) | Kintsugi recomputes produce bars with different `first_agg_trade_id`. RMT cannot dedup rows with different ORDER BY keys. 495 overlaps cascaded via straddle DELETEs + overlap self-heal amplification. | **No stable per-bar identity exists across sidecar vs kintsugi recompute.** This is inherent: bar boundaries depend on processor state from previous day's last bar. |
| Sync DELETE-then-INSERT (current)       | Phase 4 (Mar 21 PM) | Correct semantics but 24 concurrent sync DELETEs serialize on single table. Deadlock under 120K parts.                                                                                                  | `mutations_sync=1` is correct; the problem is concurrency, not the mechanism.                                                                                        |

---

## 3. Invariants Any Solution Must Preserve

From Agent 2's historical analysis, proven by 4 failure cycles:

| ID  | Invariant                                                                                                                                                            | Proven By                          |
| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- |
| I1  | **No async race between DELETE and INSERT** — DELETE must be confirmed complete before INSERT                                                                        | Phase 2 failure (bar loss)         |
| I2  | **Straddle bar handling** — old bars with different `first_agg_trade_id` whose trade-ID range overlaps new bars MUST be removed                                      | Phase 3 failure (495 overlaps)     |
| I3  | **No mutation storm under concurrency** — approach must not generate O(N) mutations for N bars; blanket DELETE (1 per repair) is acceptable, per-bar DELETE cascades | Phase 3 failure (targeted DELETEs) |
| I4  | **No data-loss window** — at no point should bars be deleted without replacement                                                                                     | Foundational                       |
| I5  | **Must use lightweight DELETE FROM** (not ALTER TABLE DELETE) to avoid full part rewrites                                                                            | Phase 1 failure (3-day outage)     |
| I6  | **Crash safety** — kill at any point must not cause permanent data loss; worst case = temporary Stathera gap that kintsugi repairs                                   | Foundational                       |

---

## 4. Key Contradiction Resolved: Daily Partitions

**Agent 7** says daily partitions would create ~187K partitions, far exceeding ClickHouse's "recommended <1,000 limit." **Agent 8** claims ClickHouse "handles millions" and recommends daily partitioned REPLACE PARTITION as the long-term solution.

**Resolution**: Both are partially correct.

- The <1,000 recommendation is a **guideline for general OLAP workloads**, not a hard limit. ClickHouse's internal materialized views routinely create thousands of partitions.
- With `do_not_merge_across_partitions_select_final=1` already enabled on this table, merge scheduling across partitions is disabled, eliminating the primary concern.
- However, 144-187K partitions **do** increase: (a) server startup time (metadata loading), (b) baseline memory usage, (c) `system.parts` query latency for monitoring, (d) INSERT overhead (partition routing per batch).
- This project runs on a **single dedicated node** (bigblack) with a workload that is batch-heavy and read-infrequent. This profile tolerates more partitions than a multi-tenant OLAP system.

**Verdict**: Daily partitions are viable but require **empirical validation** on bigblack before committing. The P1 write-queue fix should be deployed first (zero schema risk), with daily partition migration as a validated P2.

---

## 5. Solution Rankings

### Tier 1: P0 — Immediate Triage (today, 30 min)

**Action**: Kill stuck mutations + OPTIMIZE + reduce concurrency. Recovery script provided by Agent 3.

| Step | Action                                                               | Risk                                                      | Reversible?            |
| ---- | -------------------------------------------------------------------- | --------------------------------------------------------- | ---------------------- |
| 1    | Stop kintsugi (`systemctl --user stop opendeviationbar-kintsugi`)    | None                                                      | Yes (restart)          |
| 2    | Kill stuck DELETE queries (`KILL QUERY WHERE ...`)                   | None — DELETEs have `read_rows=0`, no partial application | Yes (no state change)  |
| 3    | Kill all pending mutations (`KILL MUTATION WHERE ... AND is_done=0`) | None — `read_rows=0` means no parts modified              | Yes (kintsugi retries) |
| 4    | Trigger `OPTIMIZE TABLE` (non-blocking)                              | None                                                      | N/A                    |
| 5    | Wait for parts < 5,000, then restart kintsugi                        | None                                                      | N/A                    |

**Effectiveness**: Unblocks the system. Does NOT prevent recurrence.

---

### Tier 2: P1 — Prevent Recurrence (this week, 2-4 hours)

**Recommended: threading.Lock() around `_do_replace()` + config tuning** (Agent 6 Option A+D)

| Aspect                   | Detail                                                                                                             |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------ |
| **Implementation**       | Add `_mutation_lock = threading.Lock()` in `maintenance.py`, wrap `_do_replace()` body with `with _mutation_lock:` |
| **Lines changed**        | ~5                                                                                                                 |
| **Effort**               | 2-4 hours (implement + test + deploy)                                                                              |
| **Risk**                 | Extremely low — no schema change, no data migration, no ClickHouse config change                                   |
| **Effectiveness**        | Eliminates mutation queue contention entirely. Max 1 pending mutation at any time vs 24-96 currently.              |
| **Reversibility**        | Remove the lock.                                                                                                   |
| **Preserves invariants** | All 6. DELETE+INSERT semantics unchanged; only concurrency reduced.                                                |
| **Complementary config** | `BATCH_WORKERS=1` (inner parallelism off), `MAX_CONCURRENT_PARQUET=8` (outer fetch parallelism reduced)            |

**Why this beats alternatives at P1**:

- Producer-consumer queue (Agent 6 Option B): ~60-80 lines, complex error propagation, Redis lock semantics change. Over-engineered.
- Batch mutations (Agent 6 Option C): Crash safety regression (N days lost vs 1). Complex compound DELETE predicates.
- Throttle concurrency only (Agent 6 Option D): Does not eliminate the queue; 2 workers still queue behind each other with 270s waits.

**What P1 does NOT fix**: Each individual `DELETE ... mutations_sync=1` still takes 270-315s with 120K parts. P1 eliminates queue waste but not per-mutation duration. The OPTIMIZE from P0 addresses this by reducing parts count.

---

### Tier 3: P2 — Long-Term Architecture (next sprint, 1-2 weeks)

Two viable options, ranked:

#### P2-A: Daily Partitioned REPLACE PARTITION (Agent 8 Approach 8) -- RECOMMENDED

| Aspect                   | Detail                                                                                                                                                                                                  |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Concept**              | Change partition key from `toYYYYMM` to `toYYYYMMDD`. Use `ALTER TABLE REPLACE PARTITION ... FROM staging` instead of DELETE+INSERT.                                                                    |
| **Implementation**       | New table creation, data migration, `_do_replace()` rewrite to use staging table + REPLACE PARTITION                                                                                                    |
| **Effort**               | 1-2 weeks (schema migration + code + validation)                                                                                                                                                        |
| **Risk**                 | Medium — requires data migration of ~42M rows. Mitigated by zero-downtime migration path (create new, copy, RENAME).                                                                                    |
| **Effectiveness**        | **Eliminates mutations entirely.** REPLACE PARTITION is metadata-only. 24 workers on 24 different daily partitions = true parallelism with zero serialization.                                          |
| **Reversibility**        | Keep backup table; RENAME back if issues.                                                                                                                                                               |
| **Preserves invariants** | All 6. Atomic swap = no data-loss window. No async race. Straddle bars replaced by full-day recompute.                                                                                                  |
| **Partition count**      | ~144K partitions. Validated viable for this workload profile (single-node, batch-heavy, `do_not_merge_across_partitions` already set). **Requires empirical validation on bigblack before production.** |

**Migration path** (zero downtime):

1. CREATE new table with daily partition key, same ORDER BY
2. `INSERT INTO new_table SELECT * FROM old_table` (background, can take hours)
3. Pause kintsugi + sidecar momentarily
4. `RENAME TABLE old_table TO backup, new_table TO open_deviation_bars`
5. Resume services
6. After 7 days validation, DROP backup

#### P2-B: Per-Symbol Table Sharding (Agent 7 Section 5b)

| Aspect            | Detail                                                                                                                           |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| **Concept**       | Split into 18 tables (one per symbol). Mutations serialize per-table, so 18 repairs for different symbols run truly in parallel. |
| **Effort**        | 2-4 weeks (query routing, schema management, monitoring, migration)                                                              |
| **Risk**          | High — every query, migration, backup, monitoring tool must be updated for 18 tables                                             |
| **Effectiveness** | True parallelism for cross-symbol repairs. Same-symbol repairs still serialize.                                                  |
| **Reversibility** | Very difficult                                                                                                                   |
| **Verdict**       | Last resort. Complexity far exceeds P2-A for similar benefit.                                                                    |

---

### Not Recommended (eliminated by investigation)

| Approach                                        | Agent | Why Eliminated                                                    |
| ----------------------------------------------- | ----- | ----------------------------------------------------------------- |
| CollapsingMergeTree                             | 5     | Same ORDER BY key instability; collapse requires matching keys    |
| VersionedCollapsingMergeTree                    | 5     | Same as above                                                     |
| Change ORDER BY to exclude `first_agg_trade_id` | 5     | No stable per-bar identity across sidecar vs kintsugi recompute   |
| INSERT + SELECT FINAL (no DELETE ever)          | 5     | Ghost rows accumulate unboundedly; Stathera invariant breaks      |
| argMax() dedup view                             | 8     | No stable grouping key; 50+ argMax columns = performance disaster |
| Two-table versioned architecture                | 8     | Read-time anti-join expensive; compaction reintroduces mutations  |
| ClickHouse projections for dedup                | 8     | Projections are read-only; cannot solve write-side dedup          |
| DROP PARTITION (with monthly partitions)        | 7     | Granularity mismatch: destroys 30 days to fix 1 day               |

---

## 6. Quick Win: Gap-Only Fast Path (Agent 8)

Orthogonal to the above tiers, applicable immediately:

```python
# In _do_replace(), before the DELETE:
existing = self.client.query(
    "SELECT count() FROM ... FINAL "
    "WHERE symbol=... AND first_agg_trade_id <= {new_last} AND last_agg_trade_id >= {new_first}"
).first_row[0]
if existing == 0:
    # Pure gap fill -- just INSERT, no DELETE needed
    return self.store_bars_batch(...)
```

Eliminates mutations for ~30-50% of kintsugi repairs (pure gap fills vs mismatch corrections). ~5 lines, zero risk. Can be deployed alongside P1.

---

## 7. Recommended Execution Order

1. **Now**: Execute P0 triage (Agent 3 recovery script)
2. **Today/tomorrow**: Implement P1 (`threading.Lock()` + `BATCH_WORKERS=1`)
3. **This week**: Implement gap-only fast path (5 lines)
4. **Next sprint**: Prototype daily partition table on bigblack, benchmark partition count impact, implement P2-A if validated
5. **Monitor**: Add parts count to heartbeat (alert if > 10,000)

---

## 8. Agent Report Index

| Agent                                           | Focus                     | Key Finding                                                                                                |
| ----------------------------------------------- | ------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [Agent 1](agent-1-mutation-internals.md)        | Mutation mechanics        | Lightweight DELETEs still create mutations; `mutations_sync=1` blocks caller; mutations are FIFO per-table |
| [Agent 2](agent-2-historical-context.md)        | Historical context        | 4 failed approaches over 12 days; 6 invariants any solution must preserve                                  |
| [Agent 3](agent-3-immediate-triage.md)          | Immediate triage          | Ready-to-run recovery script; stuck DELETEs have `read_rows=0` (safe to kill)                              |
| [Agent 4](agent-4-benchmark-baseline.md)        | Benchmark baseline        | 270-315s per mutation (70-column rewrite); 24x concurrency = same throughput as 1x                         |
| [Agent 5](agent-5-insert-only-revisited.md)     | INSERT-only revisited     | No MergeTree engine variant solves the key instability problem; write queue is most practical              |
| [Agent 6](agent-6-serialized-queue.md)          | Serialized queue design   | `threading.Lock()` around `_do_replace()` = 5 lines, eliminates queue waste                                |
| [Agent 7](agent-7-table-partitioning.md)        | Partition analysis        | Daily partitions = 187K (too many for general use); partition key is not the bottleneck                    |
| [Agent 8](agent-8-alternative-architectures.md) | Alternative architectures | Daily REPLACE PARTITION eliminates mutations entirely; true parallelism                                    |
