# Agent 5: Can INSERT-Only Work With a Different Engine or Approach?

## Executive Summary

**No INSERT-only approach can work with the current table, and alternative MergeTree engines do not solve the fundamental problem.** The root issue is that kintsugi recomputes entire days, producing bars with different `first_agg_trade_id` values. Any dedup strategy that relies on matching old and new rows by key fails because the keys change. The most viable mutation-free approach is **partition-level EXCHANGE TABLES** (two-table swap), which eliminates mutations entirely at the cost of additional orchestration.

---

## Approach 1: CollapsingMergeTree — REJECTED

**Concept**: Use a `Sign` column (+1 for live rows, -1 for cancellation). To replace bars, INSERT Sign=-1 copies of old bars (cancels them), then INSERT Sign=+1 new bars.

**Why it fails**:

1. **Requires knowing the exact old rows to cancel.** To emit Sign=-1 rows, kintsugi must read the old bars (including their `first_agg_trade_id`) and INSERT exact copies with Sign=-1. This means a SELECT before every replace — which kintsugi already does for Stathera verification, so the cost is acceptable.

2. **But the ORDER BY key must match.** CollapsingMergeTree collapses rows with the same ORDER BY key and opposite Sign values. If old and new bars have different `first_agg_trade_id` (which they do), the -1 row for the old bar and the +1 row for the new bar have DIFFERENT ORDER BY keys. They never collapse. Both accumulate forever.

3. **Workaround: Change ORDER BY to exclude `first_agg_trade_id`.** Possible (see Approach 3), but then CollapsingMergeTree offers no advantage over RMT — it just adds complexity with the Sign column.

**Verdict**: CollapsingMergeTree solves the wrong problem. It is designed for efficient row cancellation when the key is stable. Our key is not stable.

---

## Approach 2: VersionedCollapsingMergeTree — REJECTED

**Concept**: Like CollapsingMergeTree but adds a `Version` column. Rows with same key are collapsed in version order, handling out-of-order inserts.

**Why it fails**: Same fundamental problem as CollapsingMergeTree. The version column helps with out-of-order delivery of the SAME logical row, but old and new bars still have different ORDER BY keys. Different keys = different rows = no collapsing.

**Verdict**: Same rejection as Approach 1. VersionedCollapsing does not help when keys change.

---

## Approach 3: Change ORDER BY to Exclude `first_agg_trade_id` — HIGH RISK, THEORETICALLY POSSIBLE

**Concept**: Change ORDER BY from `(symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)` to something like `(symbol, threshold_decimal_bps, ouroboros_mode, open_time_ms)`. If `open_time_ms` is stable across recomputes, RMT can dedup naturally.

**Analysis of candidate keys**:

| Column                                          | Stable across recomputes?                                                                                                                                                              | Unique per bar?                                                   | Notes                                      |
| ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------ |
| `open_time_ms`                                  | **Mostly yes** — the first trade of a bar determines open_time_ms. But bar boundaries shift when processor state differs, so the first bar of a day may have a different open_time_ms. | No — two bars could share open_time_ms if a bar has zero duration | Risky for first bar of day                 |
| `close_time_ms`                                 | **No** — already proven unstable (#158 Phase 7 migration rationale). Different processors produce different close times.                                                               | No                                                                | Previously used, abandoned for good reason |
| `first_agg_trade_id`                            | **No** — this is the known problem                                                                                                                                                     | Yes (by construction)                                             | Current key, fails for kintsugi            |
| Composite: `(open_time_ms, first_agg_trade_id)` | No — either component can change                                                                                                                                                       | Yes                                                               | Does not help                              |

**The "mostly yes" problem with `open_time_ms`**: When kintsugi recomputes a day, the FIRST bar's `open_time_ms` is determined by the first trade of the day. This IS deterministic IF the processor is reset at the day boundary (Ouroboros reset). But the LAST bar of the previous day may have consumed some trades from the current day before the reset — the "straddle bar" problem. When the old session produced a straddle bar and the new session starts cleanly, bar 1 of the new session has a different `open_time_ms`.

**Critical risk**: Changing ORDER BY on a 42M-row table requires a full data migration (same as #158 Phase 7). The previous migration took significant effort and risk. Doing it again with a key that is only "mostly stable" could create subtle, hard-to-diagnose dedup failures.

**Verdict**: Theoretically possible but the key stability guarantee is not strong enough. The migration risk is high and the benefit is uncertain.

---

## Approach 4: INSERT + SELECT FINAL (No DELETE Ever) — REJECTED

**Concept**: Never DELETE. Just INSERT new bars. Queries always use `SELECT ... FINAL` which resolves RMT versions at read time. Old bars with different `first_agg_trade_id` coexist but are eventually merged away.

**Why it fails**:

1. **Old bars are NOT merged away.** RMT deduplicates by ORDER BY key. Old and new bars have different `first_agg_trade_id`, so they are different rows. `SELECT ... FINAL` keeps ALL of them — there is nothing to deduplicate. The table grows unboundedly with ghost rows.

2. **Stathera invariant breaks.** The old bars interleave with new bars in trade-ID space, creating overlaps (tid_delta < 0). Every downstream consumer (heartbeat, kintsugi gap detection, flowsurface) sees corrupted data.

3. **Read performance degrades.** `FINAL` already adds overhead. With ghost rows accumulating, the overhead grows linearly.

**Verdict**: INSERT+FINAL only works when old and new rows share the same ORDER BY key. They don't. This is exactly the #293 failure, just without the straddle cleanup attempt.

---

## Approach 5: Post-INSERT Dedup via GROUP BY / Materialized View — REJECTED

**Concept**: After INSERT, run a dedup query that keeps only the latest bars per logical position, either via a materialized view or a periodic cleanup query.

**Analysis**:

The fundamental question: what defines a "logical position"? For time-series bars, a bar's position is defined by the trades it contains. Two bars at the "same position" contain overlapping trades but have different `first_agg_trade_id`.

A GROUP BY dedup would need:

```sql
-- Which rows to keep? Need to identify "same logical bar" somehow.
-- Option A: Group by trade-ID overlap (expensive, requires self-join)
-- Option B: Group by time window (imprecise, bars vary in duration)
-- Option C: Group by date (too coarse, loses all bars in the day)
```

None of these provide a clean, deterministic dedup key. The problem is inherent: there IS no stable scalar key that identifies "the same bar" across recomputes.

**A materialized view** could aggregate INSERT events, but it faces the same problem — it cannot know which new rows should replace which old rows without the same key.

**Verdict**: The dedup problem is not solvable via SQL aggregation because there is no stable dedup key.

---

## Approach 6: Two-Table Swap (EXCHANGE TABLES) — VIABLE, RECOMMENDED FOR INVESTIGATION

**Concept**: Instead of mutating the live table, build replacement data in a staging table and atomically swap partitions.

### Mechanism

ClickHouse provides two relevant atomic operations:

1. **`ALTER TABLE ... REPLACE PARTITION ... FROM ...`** — atomically replaces one partition of the target table with the same partition from a source table. No mutations involved.

2. **`EXCHANGE TABLES`** — atomically swaps two tables (DDL-level rename). Overkill for per-repair operations but useful for full rebuilds.

### How It Would Work for Kintsugi

The partition key is `(symbol, threshold_decimal_bps, toYYYYMM(...))`. Kintsugi repairs operate on a specific `(symbol, threshold, day)`. A day falls within exactly ONE month partition.

**Per-repair flow**:

1. Worker fetches trades and computes new bars (parallel, no contention)
2. Worker creates a temporary table with identical schema
3. Worker INSERTs ALL bars for the affected partition (symbol, threshold, month) into the temp table — not just the repaired day, but the ENTIRE month partition
4. `ALTER TABLE open_deviation_bars REPLACE PARTITION (symbol, threshold, YYYYMM) FROM temp_table`
5. Drop temp table

**Advantages**:

- Zero mutations. `REPLACE PARTITION` is a metadata operation (swaps filesystem directories).
- Atomic. No data-loss window.
- Fast. Partition swap is ~milliseconds regardless of row count.
- No serialization bottleneck. Different partitions can be swapped concurrently.

**Disadvantages / Challenges**:

1. **Must reconstruct the entire partition (month) to replace it.** Kintsugi repairs one day, but the partition contains an entire month. The worker must read all other days in the month from the live table and combine them with the repaired day.

2. **Concurrent repairs in the same partition conflict.** If two workers repair different days in the same month, both build a temp table with their respective fix. The second `REPLACE PARTITION` overwrites the first's fix. Solution: serialize repairs within the same partition (same month), parallelize across partitions.

3. **Partition key includes `toYYYYMM(close_time_ms)` — bars near month boundaries could span months.** In practice, Ouroboros day mode prevents cross-day bars, so cross-month bars are also prevented. This is safe.

4. **120K parts in the current table.** Even reading the full partition data to reconstruct the temp table is expensive if the partition has many unmerged parts. `SELECT ... FINAL` is required.

5. **Schema must match exactly** between source and target tables for `REPLACE PARTITION` to work.

### Feasibility Assessment

The biggest obstacle is point 1: reconstructing the full month. For a partition with ~50K bars (typical month for one symbol+threshold), this means:

- SELECT ~50K bars from live table (fast with partition pruning)
- Remove the bars for the repaired day
- Add the new bars
- INSERT all ~50K bars into temp table
- REPLACE PARTITION

This is more I/O than the current approach but eliminates ALL mutations. The trade-off is worthwhile if mutation contention is the bottleneck (which it is).

### Optimization: Daily Sub-Partitioning

If the partition key were changed to `(symbol, threshold_decimal_bps, toDate(...))` instead of `toYYYYMM(...)`, each day would be its own partition. Then:

- `REPLACE PARTITION` would swap just one day's data
- No need to reconstruct the full month
- Kintsugi repairs are naturally day-scoped (Ouroboros day mode)

**Trade-off**: More partitions means more parts (already at 120K). ClickHouse recommends <1000 partitions per table. With 18 symbols x 4 thresholds x ~2500 days of data = 180K partitions — **too many**. Would need to reduce partition cardinality, e.g., `(toDate(close_time_ms))` without symbol/threshold, but then repairs must reconstruct all symbols/thresholds for a day.

### Alternative: Weekly or YYYYMMDD Partition with Cleanup

A pragmatic middle ground: keep monthly partitions but have kintsugi workers coordinate so only one worker operates on a given partition at a time. This is essentially Agent 2's "Option D" (partition-level isolation) combined with the swap approach.

---

## Approach 7: Hybrid — Serialize Writes, Parallelize Fetches (Write Queue) — MOST PRACTICAL

This is Agent 2's "Option F" but reframed as an INSERT-only discussion:

**Concept**: All workers fetch trades and compute bars in parallel. Completed results are pushed to a single-writer queue. The writer thread does sequential DELETE+INSERT operations — no concurrency on mutations.

**Why this deserves emphasis as the most practical path**:

1. **No schema changes.** Table stays as-is.
2. **No migration.** No data movement.
3. **No new engine.** ReplacingMergeTree continues to work.
4. **Trade-fetch parallelism preserved.** The slow part (REST/Parquet reads, bar computation) runs on all 24 workers.
5. **Mutation concurrency = 1.** No deadlock possible. Each DELETE+INSERT completes in ~270-315ms (not seconds — per-partition, the DELETE targets one (symbol, threshold, day) range which hits very few parts).
6. **Total throughput**: If each write takes 1-2 seconds and there are 100 repairs per pass, the write queue drains in 2-3 minutes. The fetch phase (parallel) is the dominant cost.

**Implementation**: Python `queue.Queue` + one consumer thread. Workers do `queue.put((symbol, threshold, bars_df, ...))`, consumer does `_do_replace()` in a loop. Backpressure via `queue.maxsize`.

---

## Comparative Summary

| Approach                          | Eliminates Mutations?  | Schema Change? | Migration?      | Handles Key Drift? | Practical? |
| --------------------------------- | ---------------------- | -------------- | --------------- | ------------------ | ---------- |
| 1. CollapsingMergeTree            | No (collapse = merge)  | Yes            | Full            | No                 | No         |
| 2. VersionedCollapsing            | No                     | Yes            | Full            | No                 | No         |
| 3. Change ORDER BY                | Partial (RMT dedup)    | Yes            | Full (42M rows) | Partially          | Risky      |
| 4. INSERT + FINAL                 | No                     | No             | No              | No                 | No         |
| 5. GROUP BY dedup                 | No                     | No             | No              | No                 | No         |
| 6. EXCHANGE/REPLACE PARTITION     | **Yes**                | Maybe          | Optional        | **Yes**            | Complex    |
| 7. Write queue (serialize writes) | No (still uses DELETE) | **No**         | **No**          | N/A (keeps DELETE) | **Yes**    |

---

## Recommendations

### Immediate (hours): Approach 7 — Write Queue

Serialize the mutation-generating writes through a single-writer queue while keeping trade fetching parallel. This is a Python-only change (no ClickHouse schema changes, no data migration) and directly eliminates the deadlock.

**Key insight**: The DELETE in `_do_replace()` is fast (~1-2s for a single (symbol, threshold, day) range because it targets a small number of parts). The deadlock comes from 24 of them queuing simultaneously. Serializing them costs O(24 \* 2s) = ~48s total write time per kintsugi pass, which is negligible compared to the parallel fetch time.

### Medium-term (weeks): Approach 6 — REPLACE PARTITION

If mutation-free operation is desired (e.g., for even higher concurrency or to eliminate `mutations_sync` entirely), investigate `ALTER TABLE ... REPLACE PARTITION`. This requires:

1. Prototype with a test table to validate partition swap semantics
2. Handle the "reconstruct full month" requirement
3. Add per-partition write coordination (only one worker per partition)

### Long-term (months): Consider Daily Partitioning

If kintsugi repair volume grows significantly, re-evaluate partition granularity. Daily partitions would make `REPLACE PARTITION` trivially aligned with kintsugi's day-scoped repairs, but the partition count must be managed (TTL old partitions, or use a different partition expression).

---

## Key Invariants Preserved

All approaches above preserve the 6 invariants from Agent 2:

- **I1 (no async race)**: Write queue serializes; REPLACE PARTITION is atomic
- **I2 (straddle handling)**: Blanket DELETE (write queue) or full partition replacement (REPLACE PARTITION) catch all straddle bars
- **I3 (no mutation storm)**: Max 1 concurrent mutation (write queue) or zero mutations (REPLACE PARTITION)
- **I4 (no data-loss window)**: Sequential DELETE+INSERT in write queue; atomic swap in REPLACE PARTITION
- **I5 (lightweight DELETE)**: Write queue keeps current approach; REPLACE PARTITION eliminates DELETE entirely
- **I6 (crash safety)**: Write queue = worst case temporary Stathera gap (kintsugi repairs next pass); REPLACE PARTITION = worst case temp table is orphaned (cleaned up on next pass)
