# Agent 7: Can DROP PARTITION + INSERT Replace DELETE + INSERT?

## Executive Summary

**No — DROP PARTITION cannot replace DELETE + INSERT in the current architecture.** The fundamental mismatch is granularity: kintsugi repairs one day, but partitions span one month. DROP PARTITION would destroy 29 innocent days to fix 1 day. Changing to daily partitions would create ~180K partitions (ClickHouse limit: ~1000). REPLACE PARTITION suffers the same granularity problem. However, several partition-adjacent strategies are viable.

---

## 1. Current Partition Structure

From `schema.sql` (line 318):

```sql
PARTITION BY (symbol, threshold_decimal_bps, toYYYYMM(toDateTime(close_time_ms / 1000, 'UTC')))
```

**Partition tuple**: `('BTCUSDT', 250, 202003)` — one partition per (symbol, threshold, month).

**Estimated partition count**: 18 symbols x 4 thresholds x ~80 months = ~5,760 partitions. With 120K parts total, that's ~21 parts per partition on average.

**Key fact**: Each kintsugi repair targets one (symbol, threshold, **day**) — a subset of one monthly partition.

---

## 2. DROP PARTITION Feasibility Analysis

### 2a. How DROP PARTITION Works

`ALTER TABLE DROP PARTITION` is a metadata-only operation:

- Moves part directories to `detached/` or removes them
- **Instant** — no mutation created, no part rewrite, no column decompression
- No interaction with the mutation queue whatsoever
- Does NOT block or get blocked by pending mutations

### 2b. Why It Cannot Work Here

**Granularity mismatch**: DROP PARTITION operates on entire partitions. The current partition key is `(symbol, threshold, YYYYMM)`. Kintsugi repairs one **day**.

| Operation       | Scope                      | Collateral Damage        |
| --------------- | -------------------------- | ------------------------ |
| Kintsugi repair | 1 day (e.g., 2025-03-15)   | None intended            |
| DROP PARTITION  | 1 month (e.g., 2025-03-\*) | Destroys 30 days of bars |

To use DROP PARTITION for a day-repair, you would need to:

1. DROP the entire monthly partition (destroying ~30 days)
2. Re-INSERT bars for the 29 unaffected days
3. INSERT the repaired day's bars

This is worse than the current approach — you'd need to SELECT 29 days of bars into memory, drop the partition, then re-INSERT them. For BTCUSDT@250, a month can contain ~25K bars. The SELECT+re-INSERT overhead plus the atomicity risk (crash between DROP and INSERT = month of data lost) makes this a non-starter.

---

## 3. Daily Partition Key: Would It Fix the Granularity Problem?

### 3a. Partition Count Explosion

Changing `toYYYYMM` to `toYYYYMMDD`:

| Factor                     | Count        |
| -------------------------- | ------------ |
| Symbols                    | 18           |
| Thresholds                 | 4            |
| Days (~2019-01 to 2026-03) | ~2,600       |
| **Total partitions**       | **~187,200** |

ClickHouse strongly recommends **fewer than 1,000 partitions per table**. At 187K partitions:

### 3b. What Degrades With Too Many Partitions

1. **Server startup**: ClickHouse loads partition metadata into memory at startup. 187K partitions = slow cold start, high baseline memory.
2. **INSERT performance**: Each INSERT must determine the target partition. With a composite 3-part partition key, ClickHouse creates a part per partition per INSERT batch. If a single INSERT touches N partitions, it creates N parts → the "too many parts" error (`TOO_MANY_PARTS`).
3. **Merge overhead**: The merge scheduler must evaluate 187K partitions for merge candidates. Merges happen independently per partition — more partitions = more scheduling overhead + more small parts sitting unmerged.
4. **SELECT performance**: `SELECT ... FINAL` must scan and dedup across all parts in matching partitions. More partitions = more open files, more memory for merge cursors.
5. **system.parts table**: Becomes huge, slowing `system.parts` queries used for monitoring.

**ClickHouse source code enforces a default limit** of 100 parts per partition for INSERT (`max_parts_in_total` defaults to 100,000 across all partitions). With 187K partitions, even 1 part per partition nearly hits the global limit.

### 3c. Verdict on Daily Partitions

**Not viable.** 187K partitions would degrade overall system performance severely. The benefit (instant DROP PARTITION for day-repairs) does not justify the cost.

---

## 4. REPLACE PARTITION Analysis

### 4a. How REPLACE PARTITION Works

```sql
ALTER TABLE main REPLACE PARTITION partition_id FROM temp_table
```

- Creates a temp table with identical schema
- INSERT repaired bars into temp table
- REPLACE PARTITION atomically swaps partition data from temp to main
- **Metadata-only** operation — no mutation, no column rewrite
- The temp table partition and main table partition must have the same partition expression

### 4b. Same Granularity Problem

REPLACE PARTITION operates at partition granularity. With monthly partitions, replacing a partition replaces the entire month. To repair one day:

1. Create temp table with same schema
2. SELECT all bars for the month EXCEPT the target day → INSERT into temp
3. INSERT repaired day's bars into temp
4. `REPLACE PARTITION` — atomically swap

This is equivalent to DROP PARTITION + re-INSERT. Same problems: need to read/write ~29 days of innocent data, crash between steps = data loss, memory overhead.

### 4c. With Daily Partitions

If partitions were daily, REPLACE PARTITION would be perfect:

1. Create temp table → INSERT repaired day → REPLACE PARTITION → done
2. Atomic, instant, no mutation, no collateral damage

But daily partitions create 187K partitions, which is the same blocker from Section 3.

---

## 5. Hybrid and Alternative Approaches

### 5a. Partition by (symbol, toYYYYMM) — Remove threshold from partition key

Removing `threshold_decimal_bps` from the partition key:

- Partitions: 18 symbols x 80 months = **1,440 partitions** (closer to limit but still high)
- Benefit: fewer partitions overall
- Problem: doesn't solve the day-within-month granularity issue

### 5b. Per-Symbol Tables (Table Sharding)

Create a separate table per symbol (or per symbol+threshold):

| Approach                | Tables | Benefit                       | Cost                    |
| ----------------------- | ------ | ----------------------------- | ----------------------- |
| Per symbol              | 18     | Independent mutation queues   | Schema management       |
| Per (symbol, threshold) | 72     | True parallelism for kintsugi | Heavy schema management |

**How it helps**: Mutations are serial **per table**. With 18 tables, 18 kintsugi repairs for different symbols can run truly in parallel — each table has its own mutation queue. The deadlock only occurs when multiple mutations target the same table.

**Problems**:

- All query routing must include table dispatch logic
- Schema migrations must run 18-72 times
- Monitoring (heartbeat, Stathera audit) must query across all tables
- Backup/restore becomes complex
- ClickHouse views can unify reads but not writes

**Verdict**: High complexity. May be worth it if all other solutions fail, but should be a last resort.

### 5c. Partition by (symbol, toISOWeek) — Weekly Partitions

- Partitions: 18 x 4 x ~370 weeks = **~26,640** — still too many
- Would need to remove threshold: 18 x 370 = **~6,660** — borderline high

### 5d. Materialized View Trick (Partition Router)

A clever pattern: create a MV that routes each INSERT to a partitioned sub-table with daily granularity, while the main table retains monthly partitions for reads. But this doesn't help with DELETE — the DELETE still targets the main table.

### 5e. The Actually Viable Solution: Don't Touch Partitions At All

The partition structure is not the root cause. The root cause is:

1. 24 concurrent mutations on the same table
2. `mutations_sync=1` forces each caller to block

The fixes that don't require partition changes:

- **Application-level serialization**: Limit kintsugi to 1-4 concurrent repairs (not 24)
- **Batch DELETE**: Combine multiple repairs' DELETE predicates into one mutation
- **INSERT-only with forced OPTIMIZE**: Avoid DELETE entirely, use `OPTIMIZE ... FINAL` to force RMT dedup (but this was tried and failed — see Agent 2 Phase 3)

---

## 6. DROP PARTITION as a Surgical Tool (Not General Solution)

While DROP PARTITION cannot replace the general DELETE+INSERT pattern, it CAN be used for specific operations:

### 6a. `_delete_today_bars()` in sidecar.py

Current code (line 738-743):

```sql
DELETE FROM opendeviationbar_cache.open_deviation_bars
WHERE ouroboros_mode = 'day'
  AND toDate(open_time_ms / 1000) = today()
SETTINGS lightweight_mutation_projection_mode = 'drop', mutations_sync = 1
```

This deletes ALL today's bars across ALL symbols and thresholds. It's a full-day wipe. If we could identify the relevant partition IDs (the current month's partitions for all symbol+threshold combos), we could use:

```sql
ALTER TABLE opendeviationbar_cache.open_deviation_bars
  DROP PARTITION (symbol, threshold, toYYYYMM(now()))
```

But this drops the entire MONTH, not just today. So even for this use case, DROP PARTITION is too coarse.

### 6b. Full-Month Recompute (Future Use Case)

If kintsugi ever needed to recompute an entire month (currently it works day-by-day), DROP PARTITION would be ideal:

```sql
-- Instant, no mutation
ALTER TABLE opendeviationbar_cache.open_deviation_bars
  DROP PARTITION ('BTCUSDT', 250, 202503)
-- Then INSERT the recomputed month
```

This is not currently needed but worth keeping in mind for disaster recovery.

---

## 7. Recommendations for the Investigation

### DO NOT pursue

1. **Daily partitions** — 187K partitions is a non-starter
2. **DROP PARTITION for day-repairs** — granularity mismatch is fundamental
3. **REPLACE PARTITION with monthly partitions** — same problem as DROP PARTITION
4. **Table sharding** — complexity explosion for marginal benefit

### DO pursue (recommended to other agents)

1. **Application-level concurrency control** — the cheapest, most effective fix. Limit `MAX_CONCURRENT_REPAIRS` to 1-4 (serialization at app level eliminates mutation queue contention)
2. **Batch DELETE** — combine N repair DELETEs into one `DELETE FROM ... WHERE (symbol, threshold, first_agg_trade_id, last_agg_trade_id) IN ((...), (...), ...)` with a single `mutations_sync=1` call
3. **Parts count reduction** — `OPTIMIZE TABLE ... FINAL` to merge 120K parts down to ~5K. Each mutation processes fewer parts. This is orthogonal to the partition structure and directly reduces per-mutation time.
4. **Investigate why 120K parts exist** — healthy for ~5,760 partitions would be ~5,760-11,520 parts (1-2 per partition). 120K parts suggests merges are falling behind or parts are not merging. `SELECT partition, count() FROM system.parts WHERE table = 'open_deviation_bars' AND active GROUP BY partition ORDER BY count() DESC LIMIT 20` would reveal the distribution.

---

## 8. Key Insight

The partition key is not the bottleneck. The mutation serialization model is. No partition key change can make mutations parallel across the same table. The two escape hatches from mutation serialization are:

1. **Don't use mutations** (INSERT-only + RMT dedup) — tried and failed due to straddle bars (Agent 2 Phase 3)
2. **Use fewer mutations** (application-level batching/serialization) — the viable path

The partition key matters for query performance and data organization, but it cannot solve the concurrent-mutation deadlock.
