# Agent 4: Benchmark Baseline -- What Does DELETE Actually Cost?

## 1. Table Schema Analysis

**Source**: `python/opendeviationbar/clickhouse/schema.sql`

```
ENGINE = ReplacingMergeTree(computed_at)
PARTITION BY (symbol, threshold_decimal_bps, toYYYYMM(toDateTime(close_time_ms / 1000, 'UTC')))
ORDER BY (symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)
```

### Partition Key: (symbol, threshold_decimal_bps, toYYYYMM)

This is a **composite 3-part partition key**. Estimated partition count:

- 18 symbols x 4 thresholds x ~80 months (2019-2025) = **~5,760 partitions**
- Actual count likely lower (not all symbols have data for all months), but still in the thousands.

### Primary Key = ORDER BY

`(symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)`

The DELETE predicate in `_do_replace()` is:

```sql
WHERE symbol = '{symbol}'
  AND threshold_decimal_bps = {threshold}
  AND ouroboros_mode = '{ouroboros_mode}'
  AND first_agg_trade_id <= {new_last}
  AND last_agg_trade_id >= {new_first}
  AND first_agg_trade_id > 0
```

**Primary key utilization**: The first 3 columns (`symbol`, `threshold_decimal_bps`, `ouroboros_mode`) match the ORDER BY prefix exactly -- ClickHouse can skip to the right partitions and granules efficiently. `first_agg_trade_id <= {new_last}` can also use the primary key (4th column). However, `last_agg_trade_id >= {new_first}` is NOT in the primary key -- this requires scanning rows within matching granules.

### Projection Impact

```sql
PROJECTION prj_time_range (SELECT * ORDER BY (symbol, threshold_decimal_bps, close_time_ms))
```

With `lightweight_mutation_projection_mode = 'drop'` in the DELETE statement, the projection is dropped and rebuilt on next merge. This avoids the cost of updating the projection inline but means **every DELETE invalidates the projection** for affected parts, triggering a rebuild.

## 2. Why Does a DELETE of ~100-1000 Bars Take 270-315s?

### 2a. ClickHouse DELETE = Mutation (Full Part Rewrite)

ClickHouse `DELETE FROM` is NOT a row-level delete. It is a **lightweight mutation** that:

1. Scans EVERY part in matching partitions
2. For each part: reads all rows, evaluates the WHERE predicate, writes a new part WITHOUT the matching rows
3. Atomically swaps the old part for the new part

Even deleting 1 row from a 42M-row table rewrites the parts containing that row.

### 2b. Partition Pruning Helps -- But Not Enough

The DELETE predicate includes `symbol` and `threshold_decimal_bps` which are partition key columns. ClickHouse CAN prune partitions. For a single kintsugi repair (one symbol, one threshold, one day):

- **Pruned to**: ~1 partition (1 month of 1 symbol at 1 threshold)
- **Typical partition size**: 42M bars / 5760 partitions ~ **7,300 bars/partition** (average)
- Some partitions (BTCUSDT@250) are much larger due to higher bar density.

So the partition pruning IS working. But the mutation still must:

1. Read all parts within that partition
2. Decompress all columns (the table has **~70 columns** including all microstructure features)
3. Evaluate the predicate on every row
4. Rewrite the part with matched rows removed
5. Drop and flag the projection for rebuild

### 2c. The 70-Column Tax

The table has approximately 70 columns. Each column is stored separately in ClickHouse. A mutation must:

- Read ALL 70 columns for matching parts
- Write ALL 70 columns back for the new part

This is the **dominant cost factor**. Even for a partition with ~7K rows, reading and writing 70 columns of compressed data is substantial I/O.

### 2d. Estimated Per-Mutation Cost Breakdown (270-315s)

| Phase                | Operation                                   | Estimated Time |
| -------------------- | ------------------------------------------- | -------------- |
| Partition selection  | Identify matching partitions from predicate | <1s            |
| Part enumeration     | List parts in matching partitions           | <1s            |
| Column read          | Decompress 70 columns for ~11 parts         | ~30-60s        |
| Predicate evaluation | Evaluate WHERE on all rows                  | ~5-10s         |
| Column rewrite       | Write new parts without deleted rows        | ~30-60s        |
| Projection drop      | Mark projection as needing rebuild          | <1s            |
| fsync + metadata     | Atomic swap, metadata update                | ~5-10s         |
| **Merge overhead**   | Background merges competing for I/O         | ~150-200s      |

**Key insight**: The 270-315s likely includes **I/O contention from concurrent operations**. The base cost of a single uncontended DELETE on one partition is probably 60-120s. The remaining time is I/O queuing.

## 3. Serial Mutation Queue: The Real Bottleneck

### 3a. ClickHouse Mutation Serialization

ClickHouse processes mutations **strictly serially per table**. Even though mutations CAN process multiple parts in parallel within a single mutation, **different mutations wait in a FIFO queue**.

With `mutations_sync = 1`, each `_do_replace()` call BLOCKS until its DELETE mutation completes. Meanwhile, all other DELETEs from concurrent kintsugi workers queue up.

### 3b. Throughput Calculation

**Single-threaded scenario** (no contention):

- 1 DELETE: ~120s (estimated uncontended)
- 1 INSERT: ~2-5s
- Total per repair: ~125s
- Throughput: ~0.48 repairs/min

**24 concurrent workers** (actual kintsugi scenario):

- Each worker submits DELETE with `mutations_sync=1`
- First worker's DELETE: ~120s (runs immediately)
- Workers 2-24: BLOCKED waiting in mutation queue
- Mutation queue depth: up to 24
- Worker 24's DELETE starts after workers 1-23 complete: 23 x 120s = **2,760s wait** (matches the observed 2,633s)
- Total time for 24 repairs: 24 x 125s = **3,000s (50 min)** serially
- Effective throughput: 24 repairs / 50 min = **0.48 repairs/min** (same as single-threaded!)

**Concurrency provides ZERO throughput benefit for mutations.** In fact, it's WORSE because:

1. I/O contention from 24 threads doing INSERTs between DELETEs
2. Background merges compete for disk bandwidth
3. Memory pressure from 24 blocked connections

### 3c. The mutations_sync=1 Amplifier

Without `mutations_sync=1`, the DELETE returns immediately and the mutation runs in the background. But this causes the INSERT-before-DELETE-completes race (root cause of v13.39-v13.40 bar loss). So `mutations_sync=1` is REQUIRED for correctness.

This means the serial mutation queue is an **inescapable bottleneck** with the current DELETE-then-INSERT architecture.

## 4. Partition Structure: Could It Help?

### 4a. Current Partition Key

`PARTITION BY (symbol, threshold_decimal_bps, toYYYYMM(...))`

Each kintsugi repair targets ONE (symbol, threshold, day). Since the partition is monthly, a single day's repair touches exactly **1 partition** (the month containing that day).

### 4b. Finer Partitioning (e.g., daily)?

If partitions were daily instead of monthly:

- Partition count: 18 x 4 x ~2500 days = **180,000 partitions** -- too many (ClickHouse recommends <1000)
- Would NOT help mutation speed because mutations are still serial per TABLE

### 4c. Multiple Tables (Table Sharding)?

If each (symbol, threshold) had its own table:

- Mutations would be parallel ACROSS tables
- 18 x 4 = 72 tables
- Each kintsugi repair's DELETE runs independently
- 24 concurrent repairs across different (symbol, threshold) pairs = TRUE parallelism

This is the only partition-based approach that would actually help.

## 5. Alternative Approaches (Performance Impact)

### 5a. INSERT-Only with RMT Dedup (Previously Tried, Failed -- #293)

The project already tried INSERT-only (no DELETE) relying on `ReplacingMergeTree(computed_at)` to dedup. **Failed** because kintsugi recomputes can produce bars with different `first_agg_trade_id` values, creating 495 overlaps that cascaded.

### 5b. Throttle Concurrency

Reduce `MAX_CONCURRENT_REPAIRS` from 24 to a lower value:

- At N=1: ~120s per repair, zero queue contention
- At N=4: 4 workers, 3 queue behind = max 360s wait, moderate I/O contention
- At N=8: diminishing returns, queue depth starts hurting

**Optimal N**: Probably 2-4. Above that, mutation queue serialization means extra workers just queue without throughput gain. The only benefit of N>1 is overlapping one worker's INSERT with another's DELETE.

### 5c. Batch DELETEs

Instead of 24 individual DELETE mutations, combine multiple repairs into one DELETE with OR predicates:

```sql
DELETE FROM ... WHERE
  (symbol='BTCUSDT' AND threshold=250 AND first_agg_trade_id <= X1 AND last_agg_trade_id >= Y1)
  OR (symbol='ETHUSDT' AND threshold=500 AND first_agg_trade_id <= X2 AND last_agg_trade_id >= Y2)
  ...
```

One mutation instead of 24. But: more complex, crosses partition boundaries, harder crash safety.

### 5d. DROP PARTITION + Re-INSERT

For whole-month repairs:

```sql
ALTER TABLE ... DROP PARTITION (symbol, threshold, YYYYMM)
```

`DROP PARTITION` is metadata-only (instant, no mutation). Then INSERT all bars for that month. Works only when the repair covers a full partition's worth of data.

For daily repairs within a monthly partition, this doesn't help without changing to daily partitioning.

## 6. Key Findings Summary

| Finding                     | Detail                                                              |
| --------------------------- | ------------------------------------------------------------------- |
| **Base DELETE cost**        | ~60-120s uncontended (70-column part rewrite)                       |
| **Observed DELETE cost**    | 270-315s (includes I/O contention)                                  |
| **Queue wait**              | Up to 2,633s (24 serial mutations x ~120s each)                     |
| **Concurrency benefit**     | ZERO for mutations (serial per table)                               |
| **Primary key utilization** | Good for first 3 cols, partial for trade-ID range                   |
| **Partition pruning**       | Works (prunes to 1 monthly partition per repair)                    |
| **Projection overhead**     | `lightweight_mutation_projection_mode='drop'` avoids inline rebuild |
| **Root cause**              | `mutations_sync=1` + serial mutation queue + 24 concurrent workers  |
| **The paradox**             | 24x parallelism produces same throughput as 1x, with worse latency  |

## 7. Recommended Investigation Paths

1. **Throttle concurrency to N=2-4** -- immediate, zero-code-risk improvement in latency without throughput loss
2. **Table sharding** -- separate tables per (symbol, threshold) enables true parallel mutations. Major refactor.
3. **Batch mutations** -- combine multiple repairs into single DELETE. Moderate refactor, significant queue reduction.
4. **Avoid DELETE entirely** -- requires solving the first_agg_trade_id divergence problem that killed INSERT-only (#293). Could use per-day DROP PARTITION if partition key changed to daily (but partition count explosion).
5. **Column reduction** -- fewer columns = faster part rewrite. Archive rarely-used microstructure columns to a separate table.
