# Agent 2: Historical Context — Why mutations_sync=1 Was Introduced

## Executive Summary

The current `DELETE ... SETTINGS mutations_sync=1` approach is the result of **four failed alternatives** over ~12 days (Mar 9-21, 2026). Each failure taught a specific lesson. The problem is NOT with synchronous DELETE per se — it is with **24 concurrent synchronous DELETEs** hitting the same table. Understanding the full history reveals a viable middle ground.

---

## Timeline of Approaches

### Phase 1: Async ALTER TABLE DELETE (original, pre-Mar 9)

**Approach**: `ALTER TABLE ... DELETE` (heavyweight mutation) without `mutations_sync`.

**Failure**: Issue #269 — 3-day outage (2026-03-09). With 13 concurrent kintsugi workers, heavyweight mutations queued faster than ClickHouse could process. Each mutation rewrites parts on disk. Workers waited 120s in `_wait_for_mutations()` then crashed (code 209). Monit restarted them, creating a death spiral.

**Commit**: `c9cb5c5` — switched from `ALTER TABLE ... DELETE` to lightweight `DELETE FROM` (ClickHouse 23.3+). Lightweight delete uses a delete mask instead of rewriting parts. Executes in milliseconds, does NOT queue a mutation.

### Phase 2: Async Lightweight DELETE FROM (Mar 9-12)

**Approach**: Lightweight `DELETE FROM` without `mutations_sync`. Crash-safe overlapping predicate.

**Commits**: `9b343b4` (crash-safe replace), `c9cb5c5` (lightweight delete), `ad43bd1` (projection mode fix).

**Failure**: Issue #299 — async race condition. The lightweight DELETE submits a mutation that runs asynchronously. With 649+ parts on the table, the DELETE mask may not be visible before the INSERT completes. Under `SELECT ... FINAL`, both old and new bars coexist, creating Stathera overlaps (tid_delta < 0).

**Commit**: `ad06650` — added `_cleanup_stale_left_straddle()` post-INSERT to detect and surgically delete the one stale bar that straddles the new range. This was a patch, not a fix.

### Phase 3: INSERT-only with RMT(computed_at) dedup (Mar 21, ~11am)

**Approach**: Eliminate DELETE entirely. INSERT new bars, rely on ReplacingMergeTree with `computed_at` version column to deduplicate same-key bars. For straddle bars (different `first_agg_trade_id`), detect post-INSERT via cheap FINAL query and do targeted single-row DELETE.

**Commit**: `7a9f01c` — the INSERT-only refactor.

**Why it was tried**: Eliminates the blanket DELETE (300s timeout), the async race, and the data-loss window between DELETE and INSERT. Spike-validated: RMT dedup works for same-key bars, straddle detection in 33ms.

**Failure (cascading, 3 stages)**:

1. **Straddle bars everywhere** (fundamental problem): Kintsugi recomputes entire Ouroboros days from raw trades. The new bars have ENTIRELY DIFFERENT `first_agg_trade_id` values from old bars because bar boundaries shift with threshold and processor state. Since `first_agg_trade_id` is part of the ORDER BY key, RMT CANNOT deduplicate them — they are different rows by definition. Every kintsugi repair produced straddle bars.

2. **Targeted DELETEs cascaded** (~Mar 21, 12-5pm): Each straddle bar required a targeted DELETE. With 24 concurrent kintsugi workers repairing historical gaps, hundreds of targeted per-bar DELETEs piled up. Under 4800+ parts, mutations queued faster than processing. The targeted DELETE for bar X would fail, leaving overlaps that triggered the overlap self-heal loop.

3. **Overlap self-heal amplified** (commit `2eff178`): `_check_and_heal_overlaps()` detected the overlaps from failed straddle DELETEs and called `store_bars_replacing()`, which created MORE targeted DELETEs. Positive feedback loop: 122 overlaps and 24 pending mutations observed. This was disabled.

**Result**: 495 overlaps cascade. INSERT-only approach reverted.

### Phase 4: Synchronous DELETE-then-INSERT (Mar 21, ~6pm) — CURRENT

**Approach**: Blanket lightweight `DELETE FROM` with `mutations_sync=1` (blocks until complete), then INSERT.

**Commits**: `941aabc` (\_delete_today_bars sync), `49e2ebf` (store_bars_replacing sync).

**Why it works**:

- Single blanket DELETE catches both fully-contained AND straddle bars in one operation (no per-bar mutations)
- `mutations_sync=1` blocks until DELETE completes — no async race
- INSERT follows only after DELETE is confirmed — no data loss window
- Lightweight delete (not ALTER TABLE) — no part rewrite, just delete mask

**Current problem**: With 24 concurrent kintsugi workers, each doing a synchronous DELETE, ClickHouse processes mutations sequentially per table. Workers block on each other.

---

## Root Cause Analysis: Why INSERT-only Failed (The Core Issue)

The INSERT-only approach failed because of a **fundamental mismatch** between RMT dedup and kintsugi's repair model:

1. **RMT deduplicates by ORDER BY key**: `(symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)`
2. **Kintsugi recomputes entire days**: fetches raw trades and processes through the bar algorithm from scratch
3. **Bar boundaries are threshold-dependent**: The same raw trades produce bars with different `first_agg_trade_id` depending on processor state and threshold
4. **Result**: Old bars and new bars have DIFFERENT ORDER BY keys. RMT treats them as different rows. Both coexist after merge.

This is **inherent**, not a bug. The only way INSERT-only could work is if bar boundaries were deterministic given the same input — which they ARE for a single threshold+day, but the `first_agg_trade_id` of the FIRST bar in a day depends on the LAST bar of the previous day (processor state carry-over). A kintsugi repair that re-processes from the start of a day will necessarily produce a different first bar boundary than the original streaming session.

---

## Invariants Any Solution MUST Preserve

Based on this history, any replacement approach must satisfy:

### I1: No async race between DELETE and INSERT

The DELETE must be confirmed complete before INSERT. Async DELETE + immediate INSERT = bar loss (proven in Phase 2, Phase 3).

### I2: Straddle bar handling

Old bars with different `first_agg_trade_id` whose trade-ID range overlaps new bars MUST be removed. RMT cannot handle this (proven in Phase 3).

### I3: No mutation storm under concurrency

The approach must not generate O(N) mutations for N bars. Blanket DELETE (1 mutation per repair) is acceptable. Per-bar DELETE (N mutations per repair) cascades (proven in Phase 3).

### I4: No data-loss window

At no point should bars be deleted without replacement. Either INSERT-first (with dedup) or synchronous DELETE-then-immediate-INSERT.

### I5: Works with lightweight DELETE FROM

Must use `DELETE FROM` (not `ALTER TABLE ... DELETE`) to avoid part rewrites. The table has 4800+ parts; heavyweight mutations caused a 3-day outage (proven in Phase 1).

### I6: Crash safety

A kill at any point must not cause permanent data loss. Worst case = temporary Stathera gap that kintsugi repairs on next pass.

---

## Current Architecture: Who DELETEs and When?

### Two DELETE call sites (today)

| Call Site                               | When                  | Scope                              | Concurrency         |
| --------------------------------------- | --------------------- | ---------------------------------- | ------------------- |
| `_delete_today_bars()` (sidecar.py:717) | Sidecar startup only  | ALL today's bars, ALL pairs        | 1 (singleton)       |
| `_do_replace()` (maintenance.py:418)    | Every kintsugi repair | One (symbol, threshold, day) range | Up to 24 concurrent |

### The deadlock is in `_do_replace()`

- Kintsugi runs up to 24 concurrent repair workers
- Each calls `store_bars_replacing()` -> `_do_replace()`
- Each does `DELETE FROM ... SETTINGS mutations_sync=1`
- ClickHouse processes mutations sequentially per table
- 24 workers block on each other, each waiting for all prior mutations to complete

### `_delete_today_bars()` is NOT part of the deadlock

- Called once at sidecar startup, not concurrently
- Deletes ALL today's bars in a single operation (no per-pair iteration)
- Not involved in kintsugi's concurrent repair pattern

---

## Middle Ground: Potential Solutions

Based on the history, the problem is NOT "synchronous DELETE is wrong" but "24 concurrent synchronous DELETEs serialize." Potential approaches:

### A. Serialize kintsugi repairs (concurrency=1)

**Pro**: Zero mutations concurrent. Each DELETE+INSERT is atomic.
**Con**: Throughput drops 24x. Kintsugi already takes hours for large backlogs.

### B. Batch kintsugi DELETEs

Instead of 24 individual DELETE+INSERT cycles, collect all repairs for a pass, issue ONE blanket DELETE covering all ranges, then batch INSERT.
**Pro**: Single mutation per kintsugi pass.
**Con**: Larger data-loss window if crash between DELETE and INSERT. More complex orchestration.

### C. Mutex around \_do_replace()

Serialize just the DELETE+INSERT, but let trade fetching run in parallel.
**Pro**: Fetching (slow part) stays parallel. DELETE+INSERT (fast part) serialized.
**Con**: Still sequential mutations, but without 24 blocked threads.

### D. Partition-level isolation

If kintsugi repairs are scoped to specific (symbol, threshold) partitions, ClickHouse can process mutations in parallel across partitions (mutations are sequential per-partition, not per-table).
**Pro**: True parallelism for non-overlapping repairs.
**Con**: Requires ClickHouse partitioning change. Current partitioning is by month, not by symbol.

### E. Use ALTER TABLE ... UPDATE instead of DELETE+INSERT

Mark old bars as "superseded" with an UPDATE, then INSERT new bars. RMT eventually merges.
**Pro**: No DELETE mutation at all.
**Con**: UPDATE is also a mutation. And it doesn't remove old rows — they coexist until merge.

### F. Separate the fetching from the writing with a write queue

All workers fetch and compute in parallel, push results to a single-writer queue. One writer thread does sequential DELETE+INSERT.
**Pro**: Maximum fetch parallelism. Zero write contention.
**Con**: Single-writer bottleneck. Must handle backpressure.

---

## Key Commit References

| Commit    | Date   | What                                                      |
| --------- | ------ | --------------------------------------------------------- |
| `c9cb5c5` | Mar 12 | Switch from ALTER TABLE DELETE to lightweight DELETE FROM |
| `9b343b4` | Mar 9  | Crash-safe replace with overlapping DELETE predicate      |
| `ad43bd1` | Mar 12 | Add lightweight_mutation_projection_mode=drop             |
| `ad06650` | Mar 12 | Auto-delete stale left-straddling bars (async race patch) |
| `7a9f01c` | Mar 21 | INSERT-only replace (later reverted)                      |
| `e407dca` | Mar 19 | Sidecar overlap_strategy=replace on every write           |
| `20d64c8` | Mar 19 | Revert overlap_strategy to "off" (mutation storms)        |
| `317a3a9` | Mar 19 | Sidecar overlap self-heal (later disabled)                |
| `2eff178` | Mar 21 | Disable overlap self-heal                                 |
| `941aabc` | Mar 21 | Make \_delete_today_bars synchronous                      |
| `49e2ebf` | Mar 21 | Revert to synchronous DELETE-then-INSERT (CURRENT)        |

---

## Conclusion

The synchronous DELETE-then-INSERT approach is correct in principle. The original bugs were:

1. **Async DELETE** (mutations_sync=0) allowing race conditions
2. **Per-bar targeted DELETE** creating mutation storms
3. **INSERT-only** not handling bars with different ORDER BY keys

The CURRENT problem is purely a **concurrency bottleneck**: 24 workers issuing 24 synchronous DELETEs that serialize. The solution should preserve synchronous DELETE+INSERT semantics while reducing mutation contention — either through write serialization (options C, F), mutation batching (option B), or partition-level isolation (option D).
