# Agent 1: ClickHouse Mutation Internals Investigation

## Table Schema Context

The table `opendeviationbar_cache.open_deviation_bars` uses:

- **Engine**: `ReplacingMergeTree(computed_at)`
- **Partition**: `(symbol, threshold_decimal_bps, toYYYYMM(toDateTime(close_time_ms / 1000, 'UTC')))`
- **ORDER BY**: `(symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)`
- **~42M rows**, ~120K parts (from diagnostic), ~18 symbols x 4 thresholds x ~months of data

The partition key `(symbol, threshold_decimal_bps, toYYYYMM(...))` means the table has **hundreds of partitions** (18 symbols x 4 thresholds x N months). Each partition can have multiple parts.

---

## 1. How ClickHouse Executes Mutations

### Traditional Mutations (ALTER TABLE DELETE/UPDATE)

- Mutations **rewrite entire data parts**. Every column in affected parts is rewritten, even if only one column changes.
- Mutations are **totally ordered by creation time**. They apply to each part in creation order.
- ClickHouse can **batch multiple mutations** into a single pass per part, applying all pending mutations at once during a merge.
- Parts are processed **in parallel** — ClickHouse can apply a mutation to multiple parts concurrently. The parallelism is controlled by `background_pool_size` and `background_merges_mutations_concurrency_ratio`.
- Mutations are NOT sequential per-table in the sense of "mutation 1 must finish on all parts before mutation 2 starts." Rather, **for any given part**, mutations are applied in order. Different parts can be at different mutation stages simultaneously.

### Lightweight Deletes (DELETE FROM)

- `DELETE FROM table WHERE condition` is internally translated to: `ALTER TABLE table UPDATE _row_exists = 0 WHERE condition`
- Only the `_row_exists` mask column is written — **not all columns**. Much cheaper per-part than a traditional mutation.
- Lightweight deletes are **synchronous by default** (controlled by `lightweight_deletes_sync`, default=2 for replicated, effectively synchronous).
- The `mutations_sync = 1` setting also applies to lightweight deletes, making the query block until the mutation completes on the current node.
- Affected parts are mutated (mask written); unaffected parts get hardlinks (zero-copy).

### Critical Insight: The Project Uses Lightweight DELETE

The `_do_replace()` method at `maintenance.py:443` uses:

```sql
DELETE FROM opendeviationbar_cache.open_deviation_bars
WHERE symbol = '...' AND threshold_decimal_bps = ... AND ...
SETTINGS lightweight_mutation_projection_mode = 'drop', mutations_sync = 1
```

This is a **lightweight delete** (DELETE FROM), NOT an ALTER TABLE DELETE. This means:

- Only the `_row_exists` mask column is written per affected part
- But the mutation still must be **applied to every part** that could potentially match the WHERE clause
- With `mutations_sync = 1`, the query **blocks** until the mutation is applied to ALL parts

---

## 2. Are Mutations Truly Sequential?

### Per-Part: Ordered

For any single part, mutations execute in creation order. Mutation N must complete on a part before mutation N+1 starts on that part.

### Across Parts: Parallel

Multiple parts can be mutated concurrently. The degree of parallelism is governed by:

- `background_pool_size` (default: 16) — total threads for merges + mutations
- `background_merges_mutations_concurrency_ratio` (default: 2) — ratio of mutation threads to merge threads
- `number_of_free_entries_in_pool_to_execute_mutation` (default: 20) — won't start mutations if too few free pool entries

### Across Mutations: Batched

ClickHouse can **combine multiple pending mutations** into a single pass per part. If mutations 1-5 are all pending for the same part, it may apply all 5 in one rewrite pass.

### The Deadlock Mechanism

With 30 concurrent `DELETE FROM ... SETTINGS mutations_sync=1` queries:

1. Each DELETE creates a mutation entry and blocks the caller until completion
2. Each mutation must be applied to **every part** in the table (120K parts)
3. Mutations are ordered — mutation 1 must complete on a given part before mutation 2 can start on that part
4. The background pool has limited threads (default 16) for processing parts
5. With 30 queued mutations, each blocking synchronously, the total wall time is approximately:
   - **Best case (batched)**: If ClickHouse batches all 30 mutations into one pass, it processes 120K parts once = ~270-315s total
   - **Worst case (serial)**: 30 mutations x 270-315s each = **2.25-2.6 hours** (matches the observed 2,633s+ stuck queries)
   - **Actual behavior**: Between these extremes. Earlier mutations complete, releasing their callers, but later ones must wait.

**The `mutations_sync=1` setting is the blocker** — it forces each DELETE caller to wait for its mutation to finish on all parts. With 30 concurrent callers, mutation 30 must wait for mutations 1-29 to process on every part first.

---

## 3. What is `mutations_sync` and Its Levels?

| Value | Behavior                                               | Use Case                                         |
| ----- | ------------------------------------------------------ | ------------------------------------------------ |
| **0** | Async — returns immediately after queuing the mutation | High throughput, eventual consistency            |
| **1** | Blocks until mutation completes on **current node**    | Data safety (prevents INSERT-before-DELETE race) |
| **2** | Blocks until mutation completes on **all replicas**    | Strong consistency across replicas               |

For this project, `mutations_sync=1` is REQUIRED because:

- Without it, INSERT can execute before DELETE completes, and the async DELETE can erase the freshly-inserted rows
- This was the root cause of bar loss in v13.39-v13.40 (#293)

---

## 4. Settings That Control Mutation Parallelism

| Setting                                              | Default | Description                                              |
| ---------------------------------------------------- | ------- | -------------------------------------------------------- |
| `background_pool_size`                               | 16      | Total threads for background merges + mutations          |
| `background_merges_mutations_concurrency_ratio`      | 2       | How many mutation threads relative to merge threads      |
| `number_of_free_entries_in_pool_to_execute_mutation` | 20      | Min free pool entries before mutations start             |
| `max_replicated_mutations_in_queue`                  | 8       | Max concurrent mutation tasks (ReplicatedMergeTree only) |
| `number_of_mutations_to_delay`                       | 500     | Start slowing down new mutations at this queue depth     |
| `number_of_mutations_to_throw`                       | 1000    | Reject new mutations at this queue depth                 |

---

## 5. Key Questions Answered

### Q1: Is there ANY way to parallelize mutations on the same table?

**Within a single mutation**: Yes — parts are processed in parallel (up to `background_pool_size` threads).

**Across different mutations**: Partially — ClickHouse can batch multiple mutations into one pass per part, effectively parallelizing them. But with `mutations_sync=1`, each caller blocks independently, so the 30th caller must wait for all 29 prior mutations to complete.

**True parallelism of independent mutations**: No. Mutations on the same table are strictly ordered. Even if mutation 1 targets `symbol='BTCUSDT'` and mutation 2 targets `symbol='ETHUSDT'` (completely disjoint data), mutation 2 still waits for mutation 1 on every part. **ClickHouse does not partition-scope mutations.**

### Q2: Theoretical minimum time for 24 concurrent DELETEs?

- **If perfectly batched**: One pass through 120K parts with all 24 mutations combined = **270-315s** (single mutation time)
- **If serial**: 24 x 270-315s = **1.8-2.1 hours**
- **Observed**: ~2,633s (44 min) with 30 DELETEs stuck, suggesting partial batching but significant serialization
- **Reality**: The first few complete quickly; later ones queue. The 24th DELETE waits for the full queue. With `mutations_sync=1` the Python caller blocks the entire time.

### Q3: What happens to INSERTs while mutations are pending?

**INSERTs are NOT blocked by mutations.** Data inserted after a mutation was submitted will NOT be affected by that mutation. This is critical — the INSERT in step 2 of `_do_replace()` proceeds immediately after the DELETE returns.

However, if mutations are consuming all background threads, INSERT performance may degrade slightly due to resource contention (merges compete with mutations for the same thread pool).

### Q4: Do lightweight DELETEs behave differently from traditional DELETE mutations?

**Yes, significantly:**

| Aspect            | Lightweight DELETE (DELETE FROM)                           | Traditional DELETE (ALTER TABLE DELETE) |
| ----------------- | ---------------------------------------------------------- | --------------------------------------- |
| Mechanism         | Sets `_row_exists = 0` mask                                | Rewrites all columns in affected parts  |
| I/O cost per part | Low (one small column)                                     | High (all columns rewritten)            |
| Visibility        | Rows hidden immediately on read                            | Rows physically removed                 |
| Disk reclaim      | Deferred to next merge                                     | Immediate on mutation completion        |
| Default sync      | Synchronous (`lightweight_deletes_sync=2`)                 | Async (`mutations_sync=0`)              |
| Still a mutation? | **Yes** — creates a mutation entry, follows ordering rules | Yes                                     |

**Critical**: Even though lightweight DELETEs are cheaper per-part, they still create mutation entries that follow the same ordering rules. 30 lightweight DELETEs queue just like 30 traditional DELETEs.

### Q5: Relationship between parts count and mutation speed?

**Direct linear relationship.** Each mutation must process every part (at minimum, check if the part could match the WHERE clause). With 120K parts:

- Each part check/mutation is fast (especially for lightweight deletes — just write a small mask file)
- But 120K parts x per-part overhead = significant total time
- The partition key helps somewhat — ClickHouse can skip parts whose partition min/max values exclude the WHERE predicate
- However, the WHERE clause in `_do_replace()` uses `first_agg_trade_id` ranges which span across partitions, limiting partition pruning

---

## 6. Root Cause Analysis

The deadlock is caused by the combination of:

1. **24 concurrent kintsugi repairs** each calling `_do_replace()` simultaneously
2. **Each `_do_replace()` issues a synchronous lightweight DELETE** (`mutations_sync=1`)
3. **Mutations are ordered per-table** — each mutation must wait for all prior mutations to complete on every part
4. **120K parts** means each mutation takes ~270-315s even without contention
5. **30 queued mutations** (24 repairs + possibly sidecar DELETEs) create a serial pipeline where later mutations wait for all earlier ones

**The serial bottleneck is mutations_sync=1 + mutation ordering, NOT the lightweight delete mechanism itself.**

---

## 7. Potential Mitigations (for other agents)

### Reduce mutation count

- Batch all kintsugi repairs into fewer DELETE operations (e.g., one per symbol instead of one per shard)
- Use a single DELETE with an OR-combined WHERE clause

### Avoid mutations entirely

- Use INSERT-only with ReplacingMergeTree dedup (the `computed_at` version column already exists)
- Accept eventual consistency — old bars visible until next merge
- Use `OPTIMIZE TABLE ... FINAL` after INSERT to force dedup

### Reduce parts count

- Aggressive part merging to reduce 120K → fewer, larger parts
- Consider partition key redesign (fewer partitions = fewer parts)

### Serialization at application level

- Limit concurrent repairs (e.g., MAX_CONCURRENT_REPAIRS=4 instead of 24)
- Queue repairs and issue DELETEs sequentially with a single connection

### Use async mutations

- `mutations_sync=0` + post-INSERT verification instead of synchronous blocking
- Requires careful handling to prevent the INSERT-before-DELETE race (#293)

---

## Sources

- [ClickHouse: Avoid Mutations](https://clickhouse.com/docs/optimize/avoid-mutations)
- [ClickHouse: Handling Updates and Deletes](https://clickhouse.com/blog/handling-updates-and-deletes-in-clickhouse)
- [ClickHouse: Lightweight DELETE Docs](https://clickhouse.com/docs/sql-reference/statements/delete)
- [ClickHouse: Delete Mutations Docs](https://clickhouse.com/docs/managing-data/delete_mutations)
- [ClickHouse: On-the-fly Mutations](https://clickhouse.com/docs/guides/developer/on-the-fly-mutations)
- [ClickHouse: SQL-style Updates (Part 2)](https://clickhouse.com/blog/updates-in-clickhouse-2-sql-style-updates)
- [ClickHouse: Updates Benchmarks (Part 3)](https://clickhouse.com/blog/updates-in-clickhouse-3-benchmarks)
- [Altinity KB: Mutations](https://kb.altinity.com/altinity-kb-queries-and-syntax/mutations/)
- [Altinity KB: How ALTERs Work](https://kb.altinity.com/altinity-kb-setup-and-maintenance/alters/)
- [OneUpTime: Handle Large Mutations](https://oneuptime.com/blog/post/2026-01-21-clickhouse-mutations/view)
- [QuantRail: Lightweight DELETE in ClickHouse](https://quantrail-data.com/lightweight-delete-in-clickhouse/)
- [GitHub Discussion #62292: Lightweight deletes and mutations_sync](https://github.com/ClickHouse/ClickHouse/discussions/62292)
- [GitHub Issue #59225: Lightweight deletes blocking and slow](https://github.com/ClickHouse/ClickHouse/issues/59225)
- [GitHub PR #49117: Settings to delay/throw on too many mutations](https://github.com/ClickHouse/ClickHouse/pull/49117)
- [GitHub PR #62195: lightweight_deletes_sync setting](https://github.com/ClickHouse/ClickHouse/pull/62195)
