# Agent 8: Alternative Architectures That Sidestep Mutations Entirely

**Date**: 2026-03-21
**Perspective**: What architectures avoid ClickHouse mutations completely?

---

## Context: Why Mutations Are Toxic Here

The core problem: kintsugi runs 24 concurrent shard repairs, each doing a synchronous `DELETE FROM ... SETTINGS mutations_sync=1`. ClickHouse mutations are ordered per-part, so 24 mutations x 120K parts = serialized queue that takes hours. Four previous approaches failed (async ALTER DELETE, async lightweight DELETE, INSERT-only RMT, current sync DELETE).

**The fundamental constraint**: bars get different `first_agg_trade_id` on recompute because bar boundaries depend on processor state from the previous day's last bar. This means RMT's native dedup on the current ORDER BY key cannot handle kintsugi repairs.

---

## Approach 1: REPLACE PARTITION (Atomic Partition Swap)

### Concept

ClickHouse's `ALTER TABLE t1 REPLACE PARTITION 'id' FROM t2` atomically replaces an entire partition from a staging table. No mutations at all — it's a metadata-only operation that swaps filesystem directory pointers.

### How It Would Work

1. Kintsugi recomputes a day's bars for (symbol, threshold)
2. Current partitioning is `(symbol, threshold_decimal_bps, toYYYYMM(...))`
3. Write new bars into a staging table with identical schema + partitioning
4. `ALTER TABLE main REPLACE PARTITION 'partition_id' FROM staging`

### Assessment

| Criterion             | Rating          | Notes                                                   |
| --------------------- | --------------- | ------------------------------------------------------- |
| Eliminates mutations? | **YES**         | REPLACE PARTITION is metadata-only, no mutation queue   |
| Preserves Stathera?   | YES             | Full day recompute produces correct trade-ID continuity |
| Read performance      | **ZERO IMPACT** | No FINAL needed for this operation                      |
| Write performance     | **EXCELLENT**   | INSERT into staging + atomic swap, both fast            |
| Migration complexity  | LOW             | Same schema, just add staging table                     |
| Data safety           | **EXCELLENT**   | Atomic swap — old partition exists until swap completes |

### Critical Problem: Partition Granularity Mismatch

Current partition key: `(symbol, threshold_decimal_bps, toYYYYMM(close_time_ms))`. This means one partition = one symbol + one threshold + one **month** of data.

Kintsugi repairs **one day** at a time. To use REPLACE PARTITION, kintsugi would need to:

1. SELECT all non-repaired rows from the month partition
2. INSERT them into staging
3. INSERT the repaired day's bars into staging
4. REPLACE the month partition

For a month with ~150K bars, this copies 149K unchanged bars just to fix 500. Wasteful but not catastrophic — and **zero mutations**.

### Optimization: Change Partition Key to Daily

```sql
PARTITION BY (symbol, threshold_decimal_bps, toYYYYMMDD(toDateTime(close_time_ms / 1000, 'UTC')))
```

This makes REPLACE PARTITION exactly match kintsugi's repair unit (one day). The swap replaces ~500 bars, not 15K. More partitions (18 symbols x 4 thresholds x ~2000 days = ~144K partitions), but ClickHouse handles millions of partitions fine with the `do_not_merge_across_partitions_select_final` setting already in use.

**Migration path**: Create new table with daily partitions, INSERT from old, RENAME.

### Verdict: STRONGLY RECOMMENDED

This is the highest-confidence approach. REPLACE PARTITION is:

- Used by ClickHouse itself for materialized view updates
- Atomic and crash-safe (old partition only removed after new is linked)
- Zero mutation queue involvement
- Compatible with existing 24-worker concurrency (different partitions = different workers, true parallelism)

**Key insight**: With daily partitions, each kintsugi repair targets a different partition. 24 concurrent REPLACE PARTITION operations on 24 different partitions are **fully parallel** — no serialization at all.

---

## Approach 2: Materialized View with argMax() Dedup

### Concept

Never DELETE from the main table. INSERT all bars (including duplicates from recomputes). A materialized view or query-time dedup using `argMax()` keeps only the latest version.

### How It Would Work

Option A: **Query-time dedup view**

```sql
CREATE VIEW open_deviation_bars_deduped AS
SELECT
    symbol, threshold_decimal_bps, ouroboros_mode,
    argMax(open, computed_at) AS open,
    argMax(high, computed_at) AS high,
    argMax(low, computed_at) AS low,
    argMax(close, computed_at) AS close,
    argMax(volume, computed_at) AS volume,
    -- ... 50+ more columns with argMax()
FROM opendeviationbar_cache.open_deviation_bars
GROUP BY symbol, threshold_decimal_bps, ouroboros_mode, ??? -- grouping key
```

Option B: **AggregatingMergeTree materialized view**

```sql
CREATE MATERIALIZED VIEW open_deviation_bars_latest
ENGINE = AggregatingMergeTree()
ORDER BY (symbol, threshold_decimal_bps, ouroboros_mode, ???)
AS SELECT
    symbol, threshold_decimal_bps, ouroboros_mode,
    argMaxState(open, computed_at) AS open,
    -- ... 50+ columns
FROM opendeviationbar_cache.open_deviation_bars
GROUP BY ...
```

### Critical Problem: No Stable Grouping Key

The GROUP BY key must uniquely identify "the same bar across recomputes." Candidates:

- `first_agg_trade_id`: **CHANGES** on recompute (the whole reason we're here)
- `open_time_ms`: **CHANGES** on recompute (bar boundaries shift)
- `close_time_ms`: **CHANGES** on recompute
- Trade-ID range overlap: Not expressible as a GROUP BY key

There is **no stable column** that identifies "this bar and that bar represent the same logical position." This is the same fundamental issue that killed INSERT-only RMT dedup.

### Assessment

| Criterion             | Rating       | Notes                                   |
| --------------------- | ------------ | --------------------------------------- |
| Eliminates mutations? | YES          | INSERT-only, dedup on read              |
| Preserves Stathera?   | **NO**       | No stable grouping key for argMax()     |
| Read performance      | **TERRIBLE** | argMax() on 50+ columns, no index usage |
| Write performance     | EXCELLENT    | INSERT-only                             |
| Migration complexity  | HIGH         | Need to define 50+ argMax() columns     |
| Data safety           | YES          | Never deletes                           |

### Verdict: NOT VIABLE

The fundamental problem — no stable bar identity across recomputes — kills this approach. Even if we could express the dedup, 50+ argMax() columns would be a maintenance nightmare and a query performance disaster.

---

## Approach 3: Two-Table Versioned Architecture

### Concept

- `open_deviation_bars` (main): immutable, never mutated
- `open_deviation_bars_repairs`: receives kintsugi repairs
- Read path unions both tables with dedup logic
- Periodic compaction merges repairs into main

### How It Would Work

1. Kintsugi INSERTs repairs into `_repairs` table (with a `repair_generation` column)
2. Read queries: `SELECT * FROM main WHERE NOT superseded UNION ALL SELECT * FROM repairs`
3. "Superseded" detection: bars in main whose trade-ID range overlaps with any repair
4. Weekly compaction: rebuild main from repairs + non-superseded main rows

### Assessment

| Criterion             | Rating           | Notes                                           |
| --------------------- | ---------------- | ----------------------------------------------- |
| Eliminates mutations? | YES (for writes) | Compaction still needs mutations but is offline |
| Preserves Stathera?   | YES              | Repairs are authoritative, old bars excluded    |
| Read performance      | **BAD**          | Every query does UNION ALL + anti-join          |
| Write performance     | EXCELLENT        | INSERT-only                                     |
| Migration complexity  | HIGH             | New table, new read path, compaction job        |
| Data safety           | GOOD             | Both tables always have data                    |

### Verdict: OVERCOMPLICATED

The read-time anti-join (find main bars whose trade-ID range overlaps repairs) is expensive and error-prone. The compaction step reintroduces the mutation problem, just at a lower frequency. This adds complexity without solving the fundamental issue better than REPLACE PARTITION.

---

## Approach 4: Separate Repair Log with FINAL Dedup

### Concept

Similar to Approach 3 but uses a single RMT table with a different ORDER BY that enables FINAL dedup.

### The Key Question: Is There a Stable Bar Identity?

For this to work, we need an ORDER BY key where old and new versions of "the same bar" share the same key value. Candidates:

- **Approach 4a: ORDER BY (symbol, threshold, ouroboros, open_time_ms)**
  - `open_time_ms` = timestamp of the first trade in the bar
  - If bar boundaries shift on recompute, `open_time_ms` changes
  - **Not stable** — same problem as `first_agg_trade_id`

- **Approach 4b: ORDER BY (symbol, threshold, ouroboros, close_time_ms)**
  - Same issue — bar boundaries shift means different close times

- **Approach 4c: Use a day-level repair marker**
  - Instead of per-bar identity, mark entire days as "superseded"
  - Not expressible in RMT ORDER BY

### Verdict: NOT VIABLE (same fundamental problem)

---

## Approach 5: ClickHouse Projections for Dedup

### Research Findings

- Projections are **read-only optimization structures** — they do NOT participate in write dedup
- `SELECT ... FINAL` applies to the base table's RMT engine, projections are query acceleration only
- The `deduplicate_merge_projection_mode = 'drop'` setting (already in use) means projections are dropped during dedup merges and rebuilt
- Projections cannot solve the write-side dedup problem

### Verdict: NOT APPLICABLE

Projections optimize reads, not writes. They cannot replace mutations.

---

## Approach 6: External Dedup in Rust (Diff-Based INSERT)

### Concept

Before writing, query existing bars for the repair date range, compute a diff, and only INSERT truly new bars (deleting only changed ones).

### How It Would Work

1. Kintsugi recomputes day's bars (produces N new bars)
2. Query existing bars: `SELECT first_agg_trade_id, last_agg_trade_id FROM ... WHERE date = X`
3. Compare: if old bar's `first_agg_trade_id` matches a new bar's, skip (identical)
4. DELETE only the mismatched old bars (fewer mutations)
5. INSERT only new/changed bars

### Assessment

This doesn't eliminate mutations — it reduces them. Given that `first_agg_trade_id` changes on recompute (the fundamental constraint), the diff would find that ALL bars differ, resulting in the same blanket DELETE we have now.

**One exception**: When kintsugi repairs a gap (missing bars) rather than a mismatch (wrong bars), there's nothing to DELETE — just INSERT. Currently kintsugi uses the same blanket DELETE for both cases.

### Optimization: Detect gap-only repairs

```python
# Before _do_replace, check if any existing bars overlap
existing_count = self.client.query(
    "SELECT count() FROM ... FINAL WHERE symbol = ... AND first_agg_trade_id BETWEEN ... AND ..."
)
if existing_count == 0:
    # Pure gap fill — just INSERT, no DELETE needed
    return self.store_bars_batch(...)
else:
    # Overlap exists — must DELETE first
    return self._do_replace(...)
```

### Verdict: INCREMENTAL IMPROVEMENT ONLY

Helps for pure gap fills (no DELETE needed), but doesn't solve the concurrent mutation problem for overlapping repairs. Worth combining with other approaches.

---

## Approach 7: Change ORDER BY to Use Stable Key + RMT Dedup

### Concept

If we change the ORDER BY key to use columns that remain stable across recomputes, RMT's native dedup handles everything via INSERT-only.

### Analysis of Key Stability

The question: which columns are identical when the same logical bar is recomputed?

**Bar construction flow** (from `processor.rs`):

1. First trade after previous bar closes → opens new bar (`defer_open` mechanism)
2. `open_time_ms` = timestamp of the trade that opens the bar (first trade AFTER breach)
3. `first_agg_trade_id` = trade ID of that opening trade
4. Bar closes when price breaches threshold relative to open

**When kintsugi recomputes a day**:

- It calls `reset_at_ouroboros()` at the start of the day
- The first trade of the day opens the first bar
- Bar boundaries are deterministic given the same trades + same threshold

**Critical insight from agent-2**: "bar boundaries ARE deterministic given the same input — which they ARE for a single threshold+day, but the `first_agg_trade_id` of the FIRST bar in a day depends on the LAST bar of the previous day (processor state carry-over)."

Wait — kintsugi calls `reset_at_ouroboros()` at day start, which resets processor state. So the first bar of a kintsugi-recomputed day starts from trade #1 of the day. But the sidecar's original computation may have had a forming bar carrying over from yesterday, making the first bar start at a different trade.

**Result**: The first bar of the day has different `first_agg_trade_id` between sidecar (carry-over from yesterday) and kintsugi (fresh start). All subsequent bars in the day shift accordingly.

**No stable per-bar key exists across sidecar vs. kintsugi recompute.** This is inherent to the streaming vs. batch processing difference.

### Verdict: NOT VIABLE

The sidecar-vs-kintsugi processor state difference makes ALL per-bar columns unstable. The only stable unit is the entire day.

---

## Approach 8: Day-Level Partitioned REPLACE (Synthesis of Approaches 1 + 7)

### Concept

Combine the insight that "the day is the stable repair unit" with REPLACE PARTITION's atomic swap.

### Architecture

```
Current: PARTITION BY (symbol, threshold, YYYYMM)     -- monthly
Proposed: PARTITION BY (symbol, threshold, YYYYMMDD)   -- daily
```

With daily partitions:

1. Kintsugi creates a temp table with same schema
2. INSERTs recomputed bars for (symbol, threshold, day) into temp
3. `ALTER TABLE main REPLACE PARTITION 'partition_id' FROM temp`
4. DROP temp table

### Concurrency Model

- 24 workers repair 24 different (symbol, threshold, day) tuples
- Each targets a **different partition**
- REPLACE PARTITION is **per-partition**, not per-table
- True parallelism: 24 concurrent REPLACE PARTITION operations with zero serialization

### Partition Count Estimation

- 18 symbols x 4 thresholds x ~2000 days (2019-2026) = ~144,000 partitions
- ClickHouse handles this well with `do_not_merge_across_partitions_select_final = 1` (already set)
- Each daily partition: ~100-500 bars, very small

### Implementation Sketch

```python
def _do_replace_via_partition(self, symbol, threshold, bars_df, *, date_str):
    """Atomic partition replacement — zero mutations."""
    partition_id = self._compute_partition_id(symbol, threshold, date_str)
    staging_table = f"_staging_{uuid4().hex[:8]}"

    try:
        # 1. Create ephemeral staging table (same schema)
        self.client.command(
            f"CREATE TABLE opendeviationbar_cache.{staging_table} "
            f"AS opendeviationbar_cache.open_deviation_bars"
        )

        # 2. INSERT new bars into staging
        self.store_bars_batch(
            symbol, threshold, bars_df,
            table_name=staging_table, ...
        )

        # 3. Atomic partition swap (metadata only, no mutation)
        self.client.command(
            f"ALTER TABLE opendeviationbar_cache.open_deviation_bars "
            f"REPLACE PARTITION {partition_id} "
            f"FROM opendeviationbar_cache.{staging_table}"
        )
    finally:
        # 4. Cleanup staging
        self.client.command(
            f"DROP TABLE IF EXISTS opendeviationbar_cache.{staging_table}"
        )
```

### Migration Path

1. **Phase 1**: Create new table with daily partitions, same ORDER BY
2. **Phase 2**: `INSERT INTO new_table SELECT * FROM old_table` (one-time migration)
3. **Phase 3**: `RENAME TABLE old_table TO old_table_backup, new_table TO open_deviation_bars`
4. **Phase 4**: Update `_do_replace()` to use REPLACE PARTITION instead of DELETE
5. **Phase 5**: After validation, DROP backup table

Each phase is independently testable. No downtime required for phases 1-3 (can be done while sidecar writes to old table).

### Assessment

| Criterion             | Rating        | Notes                                                                             |
| --------------------- | ------------- | --------------------------------------------------------------------------------- |
| Eliminates mutations? | **YES**       | REPLACE PARTITION = metadata swap, zero mutations                                 |
| Preserves Stathera?   | **YES**       | Full day recompute maintains trade-ID continuity                                  |
| Read performance      | **EXCELLENT** | Daily partitions actually improve partition pruning                               |
| Write performance     | **EXCELLENT** | INSERT into staging + atomic swap                                                 |
| Concurrency           | **EXCELLENT** | 24 workers on 24 partitions = true parallelism                                    |
| Migration complexity  | MEDIUM        | One-time table recreation + data copy                                             |
| Data safety           | **EXCELLENT** | Atomic swap; crash between INSERT and REPLACE = staging orphan (cleaned up)       |
| Crash safety          | **EXCELLENT** | Worst case: staging table orphan, trivially cleaned. Main table never loses data. |

---

## Comparative Summary

| Approach                       | Eliminates Mutations | Stable Key | Read Perf     | Write Perf    | Concurrency   | Verdict                                 |
| ------------------------------ | -------------------- | ---------- | ------------- | ------------- | ------------- | --------------------------------------- |
| 1. REPLACE PARTITION (monthly) | YES                  | N/A        | GOOD          | GOOD          | YES           | Viable but wasteful (copies full month) |
| 2. argMax() MV                 | YES                  | **NO**     | TERRIBLE      | GOOD          | YES           | Dead: no stable grouping key            |
| 3. Two-table union             | YES\*                | N/A        | BAD           | GOOD          | YES           | Overcomplicated                         |
| 4. Stable ORDER BY             | YES                  | **NO**     | GOOD          | GOOD          | YES           | Dead: no stable per-bar key exists      |
| 5. Projections                 | NO                   | N/A        | N/A           | N/A           | N/A           | Wrong tool (read-only)                  |
| 6. Rust diff                   | Partial              | N/A        | GOOD          | GOOD          | Partial       | Incremental only                        |
| 7. Stable ORDER BY (variant)   | YES                  | **NO**     | GOOD          | GOOD          | YES           | Dead: sidecar vs kintsugi state         |
| **8. Daily REPLACE PARTITION** | **YES**              | **N/A**    | **EXCELLENT** | **EXCELLENT** | **EXCELLENT** | **STRONGLY RECOMMENDED**                |

---

## RECOMMENDATION: Approach 8 — Daily Partitioned REPLACE PARTITION

### Why This Is the Right Answer

1. **Eliminates mutations entirely**: REPLACE PARTITION is a metadata-only atomic swap. Zero mutation queue involvement. The 120K-part / serialization problem disappears completely.

2. **Matches the repair model perfectly**: Kintsugi repairs one (symbol, threshold, day) at a time. Daily partitions make this a 1:1 mapping. No wasteful copying of unchanged bars.

3. **True concurrency**: 24 workers repairing 24 different days operate on 24 different partitions. REPLACE PARTITION on partition A has zero interaction with REPLACE PARTITION on partition B. No serialization, no deadlock.

4. **Crash-safe by design**: If a worker crashes after creating the staging table but before REPLACE PARTITION, the main table is untouched. Orphaned staging tables are trivially cleaned. If it crashes after REPLACE PARTITION but before DROP staging, the staging table is a harmless orphan.

5. **No data-loss window**: The old partition exists until the atomic REPLACE completes. There is no moment where bars are deleted but not yet replaced.

6. **Backwards compatible**: The change is entirely in the write path. The read path (SELECT ... FINAL with partition pruning) works identically — in fact, daily partitions improve read performance through better partition pruning.

### Risks and Mitigations

| Risk                        | Mitigation                                                                          |
| --------------------------- | ----------------------------------------------------------------------------------- |
| 144K partitions too many    | ClickHouse routinely handles millions; `do_not_merge_across_partitions` already set |
| Staging table cleanup       | Use `finally` block + periodic cleanup of orphaned `_staging_*` tables              |
| Migration downtime          | Zero downtime: create new table, copy data, atomic RENAME                           |
| REPLACE PARTITION atomicity | ClickHouse guarantees atomicity — used internally for MV updates                    |
| Partition ID computation    | Must match ClickHouse's internal partition ID format exactly                        |

### Quick Win: Gap-Only Fast Path (Approach 6 hybrid)

Even before the partition migration, a quick improvement: detect when a kintsugi repair is filling a pure gap (no existing bars overlap) and skip the DELETE entirely. This is a 5-line change in `_do_replace()` that eliminates mutations for ~30-50% of kintsugi repairs (the ones filling genuine gaps rather than correcting mismatches).

```python
# In _do_replace(), before the DELETE:
existing = self.client.query(
    "SELECT count() FROM ... FINAL "
    "WHERE symbol = ... AND first_agg_trade_id <= {new_last} AND last_agg_trade_id >= {new_first}"
).first_row[0]

if existing == 0:
    # Pure gap fill — no overlapping bars to remove
    return self.store_bars_batch(symbol, threshold, bars_df, ...)
```

---

## Implementation Priority

1. **Immediate** (today): Gap-only fast path in `_do_replace()` — reduces mutation count with zero risk
2. **Short-term** (this week): Daily partition migration + REPLACE PARTITION implementation
3. **Medium-term**: Staging table lifecycle management (periodic cleanup cron)
