# Agent 6: Serialized Mutation Queue — Separate Fetch from Write

## Problem Recap

Kintsugi runs up to 24 concurrent shard repairs (`max_concurrent_parquet=64`, `max_concurrent_repairs=4`). Each repair calls `_do_replace()` which executes `DELETE ... mutations_sync=1` then INSERT. ClickHouse mutations are totally ordered per-table (not per-partition), so 24 concurrent DELETEs serialize into a single queue. Result: 24 workers produce identical throughput to 1 worker (~0.48 repairs/min), but with 24x the memory and queue latency.

## Architecture Analysis

### Two Levels of Parallelism

The code has **two nested ThreadPoolExecutors**:

1. **Outer (shard-level)**: `kintsugi_pass()` at line 1880 — up to `max_concurrent` workers across different `(symbol, threshold)` pairs
2. **Inner (day-level)**: `_repair()` at line 1114 — up to `batch_workers` (default 4) parallel day repairs within a single shard

Both levels end up calling `_do_replace()` → synchronous `DELETE ... mutations_sync=1`. With 24 outer shards x 4 inner days = potentially 96 concurrent mutation requests, all serialized by ClickHouse into a single queue.

### The Fetch-Write Split Point

Each repair has two distinct phases:

| Phase     | Code Path                                                                                                                         | Nature                                 | Duration                           |
| --------- | --------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | ---------------------------------- |
| **FETCH** | `_try_parquet_day_replace` (local I/O), `_try_vision_day_replace` (HTTP GET), `_process_gap_fallback_starttime` (REST pagination) | I/O-bound, embarrassingly parallel     | 0.5s (Parquet) to 12min (REST)     |
| **WRITE** | `_do_replace()` → `DELETE mutations_sync=1` + `store_bars_batch()`                                                                | Must be serial (ClickHouse constraint) | 270-315s per mutation (120K parts) |

The fetch phase produces a `polars.DataFrame` of bars. The write phase consumes it. These are cleanly separable in all three data source paths:

- **Parquet path** (line 1216): `bars_df` built, then `cache.store_bars_batch(..., overlap_strategy="replace")`
- **Vision path** (line 1350): `bars_df` built, then `cache.store_bars_batch(..., overlap_strategy="replace")`
- **REST path** (line 1466): calls `_process_gap_fallback_starttime()` which internally does fetch+write

## Option Analysis

### Option A: threading.Lock() around \_do_replace()

**Concept**: Add a module-level `threading.Lock()` so only one thread can execute `_do_replace()` at a time. Workers still fetch in parallel but queue for the write.

**Implementation sketch**:

```python
# In maintenance.py
_mutation_lock = threading.Lock()

def _do_replace(self, symbol, threshold_decimal_bps, bars_df, ...):
    with _mutation_lock:
        self.client.command("DELETE ... mutations_sync=1")
        return self.store_bars_batch(...)
```

**Lines changed**: ~5

**Pros**:

- Minimal code change. No new abstractions.
- Fetch parallelism preserved — workers download/process in parallel.
- ClickHouse mutation queue has at most 1 pending mutation at any time (vs 24).
- Existing error handling, retry, dead-letter, today-guard all untouched.
- Redis write lock still works (orthogonal — per-pair lock vs global mutation lock).

**Cons**:

- Workers block while holding fetched `bars_df` in memory. With 24 workers each holding a day's bars (~2-10MB each), this is 48-240MB — acceptable.
- Lock contention visible in thread dumps but not a real problem since writes must be serial anyway.
- Does NOT help the 120K parts problem — each mutation still takes 270-315s.
- The REST path (`_process_gap_fallback_starttime`) does fetch+write atomically inside `backfill.py`, so the lock would need to be inside `_do_replace()` (which it is), but the REST path also calls `store_bars_batch(overlap_strategy="replace")` which routes to `_do_replace()`. This works correctly.

**Interaction with Redis write lock**: No conflict. Redis lock is per-(symbol, threshold). Mutation lock is global. A worker acquires Redis lock first (or uses `_skip_lock=True`), then acquires mutation lock. No deadlock risk since acquisition order is always Redis → mutation.

**Verdict**: **RECOMMENDED as immediate tactical fix.** Eliminates mutation queue waste with ~5 lines of code. Does not address the root cause (120K parts), but stops the bleeding.

### Option B: Producer-Consumer Queue

**Concept**: Fetch workers push `(shard, repair_date, bars_df)` tuples to a `queue.Queue(maxsize=N)`. A single consumer thread dequeues and calls `_do_replace()` sequentially.

**Implementation sketch**:

```python
write_queue: queue.Queue = queue.Queue(maxsize=8)

# In _try_parquet_day_replace / _try_vision_day_replace:
# Instead of cache.store_bars_batch(..., overlap_strategy="replace")
write_queue.put(WriteRequest(symbol, threshold, bars_df, version))

# Consumer thread:
def _write_consumer():
    while True:
        req = write_queue.get()
        if req is SENTINEL:
            break
        cache = _get_worker_cache()
        cache._do_replace(req.symbol, req.threshold, req.bars_df, ...)
```

**Lines changed**: ~60-80 (new WriteRequest dataclass, consumer thread lifecycle, error propagation)

**Pros**:

- Workers never block — they push to queue and immediately start next fetch.
- Memory bounded by `maxsize` — backpressure when writes are slow.
- Clean separation of concerns.

**Cons**:

- Error propagation is complex. Currently, if `_do_replace()` fails, the exception propagates up through `_try_parquet_day_replace()` → `_repair_under_lock()` → `_repair()` → `repair_shard()`, where it's caught and produces a `KintsugiResult(error=...)`. With a queue, the consumer must propagate errors back to the producer somehow (Future, Event, callback).
- Dead-letter logic in `_do_replace()` saves bars to Parquet on INSERT failure. This still works from the consumer thread, but error attribution (which shard failed?) requires carrying context.
- `_repair_under_lock()` currently returns the bar count synchronously. With async writes, the caller doesn't know the result until the consumer processes it.
- Consumer thread lifecycle (start, stop, error handling, daemon vs join) adds complexity.
- The REST path is harder to split — `_process_gap_fallback_starttime()` does streamed fetch+write with per-chunk flushes. Would need refactoring to buffer all bars first.

**Interaction with Redis write lock**: Problematic. Currently `_skip_lock=True` is passed because "caller already holds cache_write_lock." With a consumer thread, the lock must be acquired by the consumer, not the producer. This changes the locking semantics.

**Verdict**: Over-engineered for the problem. The complexity of error propagation and lock semantics is not worth the marginal benefit over Option A. Workers holding 2-10MB each while blocked is not a real concern.

### Option C: Batch Mutations

**Concept**: Collect N repair results, then do ONE DELETE + INSERT covering all N repairs in a single mutation.

**Implementation sketch**:

```python
# Accumulate repairs
pending_writes: list[WriteRequest] = []

# After all fetches complete:
# Build compound DELETE predicate
predicates = " OR ".join(
    f"(symbol='{r.symbol}' AND threshold={r.threshold} AND first_agg_trade_id<={r.last} AND last_agg_trade_id>={r.first})"
    for r in pending_writes
)
client.command(f"DELETE FROM ... WHERE {predicates} SETTINGS mutations_sync=1")

# Single INSERT with all bars concatenated
all_bars = pl.concat([r.bars_df for r in pending_writes])
client.insert(all_bars)
```

**Lines changed**: ~100-120

**Pros**:

- Amortizes mutation cost: 1 mutation per batch instead of N.
- With 120K parts, fewer mutations = fewer part scans.
- Could reduce total write time significantly if mutations have high fixed overhead.

**Cons**:

- **Compound DELETE predicate is risky**: ClickHouse mutations with complex `OR` predicates scan ALL parts. The predicate grows linearly with batch size. With different (symbol, threshold) combinations in the same batch, the mutation cannot leverage partition pruning effectively.
- **Crash safety regression**: Current design is crash-safe per-chunk (line 1391-1395). Batching N repairs means if the process dies after DELETE but before INSERT, N days of bars are lost (vs 1 day currently). Dead-letter for N-day batches is much harder.
- **Memory spike**: Must hold all N repair results in memory simultaneously. 24 repairs x 10MB each = 240MB — manageable but defeats the current `del bars_df` cleanup pattern.
- **Cross-pair lock**: Would need to hold Redis locks for ALL pairs simultaneously during the batch write, creating a different deadlock risk.
- **Verification**: `verify_repair()` runs per-shard after write. With batched writes, verification must wait for the entire batch, delaying feedback.

**Interaction with 120K parts**: Marginally helps — fewer mutations means fewer full-table scans. But each compound mutation scans ALL 120K parts with a more complex predicate. Net benefit unclear without benchmarking.

**Verdict**: **Not recommended.** The crash safety regression and complexity far outweigh the marginal throughput gain. The 120K parts problem should be solved at the ClickHouse level (OPTIMIZE, TTL, partition strategy) not by batching mutations.

### Option D: Throttle MAX_CONCURRENT_REPAIRS to 2-4

**Concept**: Since throughput is identical regardless of concurrency, just reduce `max_concurrent_parquet` from 64 to 2-4 and `max_concurrent_repairs` from 4 to 2.

**Implementation**: Change 2 config defaults. Zero code changes.

**Lines changed**: 2 (config defaults) or 0 (env var override in deploy)

**Pros**:

- Simplest possible change. Zero risk.
- Reduces memory pressure (2 workers vs 24).
- Reduces ClickHouse mutation queue depth (2 pending vs 24).
- Reduces file descriptor usage (fewer concurrent connections).
- Reduces thread contention, context switching, log noise.

**Cons**:

- **Does NOT fix the root cause**: mutations still serialize. 2 workers with 270s mutations still means the second worker waits 270s for the first mutation to complete before its mutation starts.
- **Reduces fetch parallelism**: With only 2 workers, Vision/Parquet downloads are serialized more. For Parquet-cached repairs (local I/O, <1s), this barely matters. For Vision repairs (5s each), slight slowdown. For REST repairs (12min each), 2 workers means only 2 REST downloads at a time.
- **Does not address day-level inner parallelism**: `batch_workers=4` still spawns 4 day workers per shard. With 2 outer workers x 4 inner workers = 8 concurrent mutations.

**Interaction with 120K parts**: Does not help. Same mutation duration per repair.

**Verdict**: **Good as a complementary measure alongside Option A**, but insufficient alone. Reduces waste but doesn't eliminate the serialization penalty.

## Recommendation: Option A + D Combined

### Immediate Fix (Option A): Mutation Lock

Add a **module-level** `threading.Lock()` in `maintenance.py` around `_do_replace()`:

```python
import threading

_mutation_lock = threading.Lock()

def _do_replace(self, symbol, threshold_decimal_bps, bars_df, *,
                ouroboros_mode, version, new_first, new_last):
    with _mutation_lock:
        # Step 1: Synchronous blanket DELETE
        self.client.command(
            "DELETE FROM opendeviationbar_cache.open_deviation_bars ..."
        )
        # Step 2: INSERT new bars
        try:
            rows_inserted = self.store_bars_batch(...)
        except Exception:
            _dead_letter_replace_failure(...)
            raise
        return rows_inserted
```

**Why module-level, not instance-level**: `_get_worker_cache()` creates per-thread `OpenDeviationBarCache` instances. A per-instance lock would not prevent concurrent mutations across threads. The lock must be module-level to be effective across all worker threads.

**Why threading.Lock, not asyncio.Lock**: Kintsugi uses `ThreadPoolExecutor`, not asyncio. `threading.Lock()` is the correct primitive.

### Config Tuning (Option D): Reduce Concurrency

```bash
# In deploy/opendeviationbar.env (or .mise.toml):
OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_PARQUET=8   # was 64 → still good fetch parallelism
OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REPAIRS=4   # keep at 4 (Vision/mixed)
OPENDEVIATIONBAR_KINTSUGI_BATCH_WORKERS=1            # was 4 → serial day writes within shard
```

**Rationale for `batch_workers=1`**: Inner day-level parallelism creates the most mutation waste (4 day workers per shard, all hitting `_do_replace()`). Setting to 1 eliminates inner parallelism. The mutation lock (Option A) handles outer parallelism.

### Expected Impact

| Metric                      | Before                                               | After (A+D)                                                     |
| --------------------------- | ---------------------------------------------------- | --------------------------------------------------------------- |
| Concurrent mutations queued | Up to 96 (24 outer x 4 inner)                        | 1 (mutation lock)                                               |
| Mutation queue wait time    | 24 x 270s = 6,480s worst case                        | 0s (no queue)                                                   |
| Fetch parallelism           | 24 concurrent downloads                              | 8 concurrent downloads (Parquet)                                |
| Throughput per mutation     | ~0.48 repairs/min (same with or without parallelism) | ~0.48 repairs/min (unchanged — bottleneck is mutation duration) |
| Memory (worker threads)     | 24 threads x bars in memory                          | 8 threads x bars in memory                                      |

**Key insight**: The mutation lock does NOT improve per-mutation throughput. Each `DELETE ... mutations_sync=1` still takes 270-315s with 120K parts. What it eliminates is the **wasted queue latency** — mutations no longer pile up behind each other. The actual throughput improvement comes from solving the 120K parts problem (a separate investigation).

### What This Does NOT Fix

1. **120K parts problem**: Each mutation still scans 120K parts. This is the root cause of 270-315s mutation duration. Must be solved via `OPTIMIZE TABLE ... FINAL` or partition strategy changes.
2. **Mutation duration**: Individual `DELETE ... mutations_sync=1` operations are still slow.
3. **Total repair throughput**: Still bottlenecked by ClickHouse mutation speed. The fix just eliminates wasted concurrency.

### Risks

1. **Throughput ceiling**: With `_mutation_lock`, only one write happens at a time. If fetches are faster than writes (they are — Parquet fetch is <1s vs 270s write), workers will block holding bars in memory. With 8 workers, worst case is 8 x 10MB = 80MB — negligible.
2. **Lock starvation**: `threading.Lock()` does not guarantee FIFO ordering in CPython. In practice, with 270s hold times, starvation is not a concern — threads wake up approximately in order.
3. **Deadlock**: No risk. The lock acquisition order is always: Redis per-pair lock (optional) → mutation lock. No circular dependency.

### Alternative: Lock at store_bars_replacing() Level

Instead of locking inside `_do_replace()`, lock at `store_bars_replacing()` (line 314). This would serialize all "replace" strategy writes, including non-kintsugi callers. Since kintsugi is the only production user of `overlap_strategy="replace"`, this is equivalent but provides a wider safety margin.

```python
def store_bars_replacing(self, ...):
    with _mutation_lock:
        # ... existing logic including _do_replace()
```

This is slightly safer because it prevents any future caller of `store_bars_replacing()` from creating concurrent mutations. Recommended over locking inside `_do_replace()` for defense-in-depth.
