# Phase 1: Mutation Safety - Research

**Researched:** 2026-03-24
**Domain:** ClickHouse mutation performance, DELETE operation scoping, Python threading locks
**Confidence:** HIGH

## Summary

Phase 1 addresses a verified production bottleneck: kintsugi's `overlap_strategy="replace"` path triggers `ALTER TABLE DELETE` operations that scan all parts when not partition-scoped, causing 300s timeouts under 24-worker concurrency. The fix is straightforward -- migrate all DELETE call sites to `DELETE IN PARTITION` syntax where the partition key `(symbol, threshold_decimal_bps)` is known, and replace the global `_replace_lock` with per-partition locks.

The hot path (`_do_replace()` in `maintenance.py`) already uses `DELETE IN PARTITION` since v13.50.0. The remaining work is auditing and migrating 8+ secondary call sites across kintsugi, sidecar, rectify, and utility scripts, plus the lock granularity change. One notable exception: the `kintsugi_log` table uses `MergeTree()` with no explicit `PARTITION BY`, so its DELETEs cannot use `DELETE IN PARTITION` -- those DELETEs target a tiny table and are not performance-critical.

**Primary recommendation:** Audit all DELETE sites, migrate to `DELETE IN PARTITION ('{symbol}', {threshold_decimal_bps})` for `open_deviation_bars` table, replace `_replace_lock` with per-partition dict, verify on bigblack with 24-worker kintsugi pass.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

- D-01: Full audit of ALL DELETE call sites -- migrate every `ALTER TABLE DELETE` to `DELETE IN PARTITION` where partition key is known, including non-hot-path sites (sidecar, rectify, cache_clear, validate_dedup_hardening, kintsugi_log cleanup)
- D-02: Per-partition lock replaces global `_replace_lock` -- dict of locks keyed by `(symbol, threshold_decimal_bps)`, different partitions mutate concurrently, same partition serialized
- D-03: MUT-01 already mostly satisfied -- `_do_replace()` already uses `DELETE IN PARTITION` since v13.50.0, phase is primarily about completeness and concurrency
- D-04: Production verification on bigblack -- run normal kintsugi catchup with 24 workers, query `system.mutations` for max duration, no synthetic load test needed

### Claude's Discretion

- Lock implementation details (e.g., `defaultdict(threading.Lock)` vs explicit dict with `setdefault`)
- Whether to add telemetry logging for per-partition lock contention
- Order of migration across call sites

### Deferred Ideas (OUT OF SCOPE)

None -- discussion stayed within phase scope.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID     | Description                                                                                                                             | Research Support                                                                                                                                                                     |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| MUT-01 | Kintsugi DELETE operations use `DELETE IN PARTITION` with partition-level granularity exclusively -- no full-table `ALTER TABLE DELETE` | Full audit of 10+ call sites completed; partition key `(symbol, threshold_decimal_bps)` always available at every open_deviation_bars DELETE site; kintsugi_log exception documented |
| MUT-02 | Concurrent kintsugi repairs (24+ workers) complete without mutation timeout (300s)                                                      | Per-partition lock eliminates global serialization; `DELETE IN PARTITION` scans O(1 partition) not O(all parts); verified by `system.mutations` query on bigblack                    |

</phase_requirements>

## Standard Stack

No new dependencies. This phase modifies existing Python code only.

### Core (already in project)

| Library              | Version  | Purpose                 | Why Standard                     |
| -------------------- | -------- | ----------------------- | -------------------------------- |
| `threading`          | stdlib   | Per-partition lock dict | Already used for `_replace_lock` |
| `clickhouse-connect` | existing | ClickHouse mutations    | Already the project's CH client  |

## Architecture Patterns

### DELETE Call Site Inventory

Complete audit of every `ALTER TABLE DELETE` in the codebase, categorized by table and migration path:

#### Table: `open_deviation_bars` (PARTITION BY (symbol, threshold_decimal_bps))

| #   | File:Line                                          | Current Pattern                                                                           | Symbol/Threshold Available?                       | Migration                                                                                                                   |
| --- | -------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| 1   | `maintenance.py:325`                               | `DELETE IN PARTITION ('{sym}', {thr})`                                                    | Yes (params)                                      | **Already done** -- template for others                                                                                     |
| 2   | `maintenance.py:336`                               | `ALTER TABLE DELETE` (dead code under `if False:`)                                        | N/A                                               | Remove dead code                                                                                                            |
| 3   | `sidecar/__init__.py:344`                          | `ALTER TABLE DELETE WHERE (symbol, threshold_decimal_bps) IN (...)`                       | Yes (pairs list)                                  | Migrate to per-pair `DELETE IN PARTITION` loop, or single compound DELETE with partition hint                               |
| 4   | `rectify.py:240`                                   | `ALTER TABLE DELETE WHERE symbol=... AND threshold_decimal_bps=... AND close_time_ms=...` | Yes (params)                                      | Migrate to `DELETE IN PARTITION ('{sym}', {thr}) WHERE close_time_ms=...`                                                   |
| 5   | `scripts/cache_clear.py:48`                        | `ALTER TABLE DELETE WHERE symbol=...`                                                     | Symbol yes, threshold no (deletes all thresholds) | Migrate: query distinct thresholds first, then `DELETE IN PARTITION` per (sym, thr), OR use `DROP PARTITION` per (sym, thr) |
| 6   | `scripts/validate_dedup_hardening.py:110`          | `ALTER TABLE DELETE WHERE symbol=...`                                                     | Uses test symbol `__TEST__`                       | Migrate: use known test threshold, `DELETE IN PARTITION ('__TEST__', {thr})`                                                |
| 7   | `scripts/validate_microstructure_roundtrip.py:294` | `ALTER TABLE DELETE WHERE symbol=...`                                                     | Uses test symbol                                  | Same as #6                                                                                                                  |
| 8   | `scripts/cleanup_oversized_bars.py:96`             | `ALTER TABLE DELETE WHERE {where_sql}`                                                    | Yes (symbol and threshold in params)              | Migrate to `DELETE IN PARTITION`                                                                                            |
| 9   | `scripts/validate_kintsugi_fixes.py:83,231`        | `ALTER TABLE kintsugi_log DELETE WHERE symbol='__PROBE__'`                                | kintsugi_log table                                | See kintsugi_log section below                                                                                              |
| 10  | `scripts/purge_crypto_low_threshold.py:109`        | `ALTER TABLE DELETE WHERE symbol=...`                                                     | Yes                                               | Migrate to `DELETE IN PARTITION` per (sym, thr) or `DROP PARTITION`                                                         |
| 11  | `scripts/cleanup_extra_symbols.py:114`             | `ALTER TABLE DELETE WHERE symbol=...`                                                     | Yes                                               | Same as #10                                                                                                                 |

#### Table: `kintsugi_log` (MergeTree, NO explicit PARTITION BY)

| #   | File:Line                                   | Current Pattern                                            | Can Use DELETE IN PARTITION?                                                                                                        |
| --- | ------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| 12  | `kintsugi/discovery.py:443`                 | `ALTER TABLE kintsugi_log DELETE WHERE result='clean'...`  | **NO** -- table has no explicit partition key (single `all` partition). `DELETE IN PARTITION` requires knowing the partition tuple. |
| 13  | `scripts/validate_kintsugi_fixes.py:83,231` | `ALTER TABLE kintsugi_log DELETE WHERE symbol='__PROBE__'` | **NO** -- same reason                                                                                                               |

**kintsugi_log exception:** This table uses `MergeTree()` with `ORDER BY (timestamp, symbol, threshold_decimal_bps)` and no `PARTITION BY` clause. ClickHouse places all data in a single implicit partition (`tuple()`). The DELETE is already fast because the table is tiny (90-day TTL, small row count). No migration needed for these sites -- they are not performance-critical and don't touch `open_deviation_bars`.

### Pattern: DELETE IN PARTITION Syntax

The existing `_do_replace()` provides the canonical pattern:

```python
# Source: maintenance.py:325 (already correct)
self.client.command(
    "ALTER TABLE opendeviationbar_cache.open_deviation_bars "
    f"DELETE IN PARTITION ('{symbol}', {threshold_decimal_bps}) "
    f"WHERE first_agg_trade_id <= {new_last} "
    f"AND last_agg_trade_id >= {new_first} "
    "AND first_agg_trade_id > 0 "
    "SETTINGS mutations_sync = 1"
)
```

Key elements:

- Partition tuple: `('{symbol}', {threshold_decimal_bps})` -- string value quoted, int not
- `SETTINGS mutations_sync = 1` -- REQUIRED on all DELETEs (blocks until complete)
- WHERE clause further narrows within the partition (ORDER BY index prunes to matching granules)

### Pattern: Per-Partition Lock

Replace the module-level global lock:

```python
# BEFORE (maintenance.py:29)
_replace_lock = threading.Lock()

# AFTER
_replace_locks: dict[tuple[str, int], threading.Lock] = {}
_replace_locks_guard = threading.Lock()  # protects dict creation only

def _get_partition_lock(symbol: str, threshold_decimal_bps: int) -> threading.Lock:
    key = (symbol, threshold_decimal_bps)
    lock = _replace_locks.get(key)
    if lock is not None:
        return lock
    with _replace_locks_guard:
        # Double-check after acquiring guard
        return _replace_locks.setdefault(key, threading.Lock())
```

**Why `setdefault` over `defaultdict`:** `defaultdict(threading.Lock)` auto-creates locks on any key access including reads, and cannot be used with a guard lock for thread-safe initialization. `setdefault` is atomic for dict insertion and integrates cleanly with the double-check pattern.

**Impact:** `_do_replace()` changes from `with _replace_lock:` to `with _get_partition_lock(symbol, threshold_decimal_bps):`. All callers benefit automatically since they flow through `_do_replace()`.

### Sidecar Compound DELETE Strategy

The sidecar pre-fill (site #3) currently uses a single compound `ALTER TABLE DELETE` for all 60+ (symbol, threshold) pairs. Two migration options:

**Option A: Loop of DELETE IN PARTITION** (recommended)

```python
for sym, thr in pairs:
    cache.client.command(
        "ALTER TABLE opendeviationbar_cache.open_deviation_bars "
        f"DELETE IN PARTITION ('{sym}', {thr}) "
        f"WHERE ouroboros_mode = '{mode}' "
        f"AND toDate(open_time_ms / 1000000) = '{today_str}' "
        "SETTINGS mutations_sync = 1"
    )
```

Pro: Each DELETE scans only its partition. Con: 60+ sequential network round-trips.

**Option B: Parallel DELETE IN PARTITION** (alternative)
Use ThreadPoolExecutor to issue DELETEs concurrently across partitions. Pro: faster wall time. Con: 60+ concurrent mutations may hit CH scheduler limits.

**Recommendation:** Option A. The sidecar pre-fill runs once at startup, not in a hot loop. 60 sequential partition-scoped DELETEs will each complete in milliseconds (scanning 1 partition vs scanning all 6000+ parts). Total time: seconds, not the current 300s+ timeout.

### Anti-Patterns to Avoid

- **Using `DELETE FROM` (lightweight DELETE):** Permanently banned. Creates `_row_exists=0` ghost rows causing false Stathera anomalies.
- **Omitting `mutations_sync = 1`:** Creates race conditions where INSERT runs before DELETE completes.
- **Using `defaultdict(threading.Lock)`:** Not thread-safe for creation; prefer `setdefault` with guard lock.

## Don't Hand-Roll

| Problem                  | Don't Build             | Use Instead                                | Why                                                              |
| ------------------------ | ----------------------- | ------------------------------------------ | ---------------------------------------------------------------- |
| Thread-safe per-key lock | Custom locking protocol | `dict.setdefault()` + guard lock           | Standard Python pattern, proven thread-safe                      |
| Mutation monitoring      | Custom polling loop     | `system.mutations` ClickHouse system table | Built-in, authoritative, already used by `_wait_for_mutations()` |

## Common Pitfalls

### Pitfall 1: kintsugi_log Has No Partition Key

**What goes wrong:** Attempting `DELETE IN PARTITION` on `kintsugi_log` fails because the table has no `PARTITION BY` clause.
**Why it happens:** Natural assumption that all tables have partition keys.
**How to avoid:** Audit each table's DDL before migrating. `kintsugi_log` uses `MergeTree()` without `PARTITION BY` -- its DELETEs stay as `ALTER TABLE DELETE`.
**Warning signs:** ClickHouse error about invalid partition expression.

### Pitfall 2: Sidecar Column Name Mismatch

**What goes wrong:** The sidecar compound DELETE references `open_time_ms` but the schema column is `open_time_us` (microseconds).
**Why it happens:** Legacy column name from before the ms->us migration.
**How to avoid:** Verify column names against `schema.sql` when migrating DELETE syntax. The sidecar code may have a compatibility alias or the field may actually work due to ClickHouse's column naming.
**Warning signs:** DELETE silently matches no rows, or ClickHouse error about unknown column.

### Pitfall 3: Lock Dict Memory Growth

**What goes wrong:** Per-partition lock dict grows unbounded if symbols/thresholds are dynamic.
**Why it happens:** New lock created for every unique (symbol, threshold) pair.
**How to avoid:** Not a real concern here -- the symbol registry has ~26 symbols x 4 thresholds = ~104 partitions maximum. Lock objects are tiny (~100 bytes each).

### Pitfall 4: Test Scripts Using Synthetic Symbols

**What goes wrong:** Test scripts use `__TEST__` or `__PROBE__` as symbol names, which may not match any partition.
**Why it happens:** Test data is inserted then deleted.
**How to avoid:** For `DELETE IN PARTITION`, the test must know both symbol and threshold. Use a fixed test threshold (e.g., 9999) so the partition tuple is known.

## Code Examples

### Migrating rectify.py Ghost Bar DELETE

```python
# BEFORE (rectify.py:240)
cache.client.command(
    "ALTER TABLE opendeviationbar_cache.open_deviation_bars "
    "DELETE WHERE symbol = {symbol:String} "
    "  AND threshold_decimal_bps = {threshold:UInt32} "
    "  AND ouroboros_mode = {mode:String} "
    "  AND close_time_ms = {close_time_ms:Int64}",
    parameters={...},
)

# AFTER
cache.client.command(
    "ALTER TABLE opendeviationbar_cache.open_deviation_bars "
    f"DELETE IN PARTITION ('{symbol}', {threshold_decimal_bps}) "
    "WHERE ouroboros_mode = {mode:String} "
    "  AND close_time_ms = {close_time_ms:Int64} "
    "SETTINGS mutations_sync = 1",
    parameters={"mode": ouroboros_mode, "close_time_ms": close_time_ms},
)
```

**Note:** `DELETE IN PARTITION` does not support parameterized partition tuples in clickhouse-connect -- the partition expression must be interpolated directly (f-string), while the WHERE clause can still use parameters.

### Migrating cache_clear.py (Symbol-Only DELETE)

```python
# BEFORE: Deletes all thresholds for a symbol in one unscoped DELETE
cache.client.command(
    f"ALTER TABLE {table} DELETE WHERE symbol = '{symbol}'"
)

# AFTER: Query partitions, then delete each one
partitions = cache.client.query(
    "SELECT DISTINCT partition "
    "FROM system.parts "
    "WHERE database = 'opendeviationbar_cache' "
    "AND table = 'open_deviation_bars' "
    "AND active "
    f"AND partition LIKE '(''{symbol}'',%'"
).result_rows
for (part_id,) in partitions:
    cache.client.command(
        f"ALTER TABLE {table} DELETE IN PARTITION {part_id} "
        f"WHERE symbol = '{symbol}' "
        "SETTINGS mutations_sync = 1"
    )
```

### Production Verification Query

```sql
-- Run on bigblack after kintsugi pass with 24 workers
SELECT
    command,
    create_time,
    parts_to_do,
    latest_fail_reason,
    formatReadableTimeDelta(
        date_diff('second', create_time, if(is_done, latest_fail_time, now()))
    ) AS duration
FROM system.mutations
WHERE database = 'opendeviationbar_cache'
  AND table = 'open_deviation_bars'
ORDER BY create_time DESC
LIMIT 50;

-- Verify no mutation exceeds 300s
SELECT
    max(date_diff('second', create_time,
        if(is_done, latest_fail_time, now()))) AS max_duration_s
FROM system.mutations
WHERE database = 'opendeviationbar_cache'
  AND table = 'open_deviation_bars'
  AND create_time > now() - INTERVAL 1 HOUR;
```

## Validation Architecture

### Test Framework

| Property           | Value                                      |
| ------------------ | ------------------------------------------ |
| Framework          | pytest 9.0.2                               |
| Config file        | `pyproject.toml` [tool.pytest.ini_options] |
| Quick run command  | `mise run test-py`                         |
| Full suite command | `mise run check-full`                      |

### Phase Requirements -> Test Map

| Req ID | Behavior                                                               | Test Type   | Automated Command                                                              | File Exists?                  |
| ------ | ---------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------ | ----------------------------- |
| MUT-01 | All open_deviation_bars DELETEs use DELETE IN PARTITION                | unit (mock) | `pytest tests/test_mutation_safety.py::test_delete_in_partition_syntax -x`     | Wave 0                        |
| MUT-01 | kintsugi_log DELETEs remain ALTER TABLE DELETE (no partition key)      | unit (mock) | `pytest tests/test_mutation_safety.py::test_kintsugi_log_no_partition -x`      | Wave 0                        |
| MUT-02 | Per-partition lock allows concurrent mutations on different partitions | unit        | `pytest tests/test_mutation_safety.py::test_per_partition_lock_concurrency -x` | Wave 0                        |
| MUT-02 | Per-partition lock serializes same-partition mutations                 | unit        | `pytest tests/test_mutation_safety.py::test_same_partition_serialized -x`      | Wave 0                        |
| MUT-02 | Production verification: 24 workers, max mutation <300s                | manual-only | `system.mutations` query on bigblack                                           | N/A (production verification) |

### Sampling Rate

- **Per task commit:** `mise run test-py`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green + bigblack production verification before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_mutation_safety.py` -- covers MUT-01, MUT-02 (lock behavior, SQL syntax assertions)
- No framework install needed -- pytest already configured

## Open Questions

1. **Sidecar `open_time_ms` column reference**
   - What we know: The sidecar compound DELETE references `open_time_ms` but the schema defines `open_time_us`. There may be an alias or compatibility layer.
   - What's unclear: Whether this is a bug that has been working by coincidence or an intentional alias.
   - Recommendation: Verify the actual column name during implementation. If it's `open_time_us`, fix the reference as part of the migration.

2. **Parameterized partition tuples in clickhouse-connect**
   - What we know: `DELETE IN PARTITION` partition expressions are parsed by ClickHouse, not as query parameters.
   - What's unclear: Whether clickhouse-connect supports `{param:Type}` syntax in the partition tuple position.
   - Recommendation: Use f-string interpolation for partition tuples (safe -- symbol is from the registry, threshold is an int). WHERE clauses can continue using parameterized queries where applicable.

## Sources

### Primary (HIGH confidence)

- Codebase audit: `maintenance.py`, `sidecar/__init__.py`, `rectify.py`, `discovery.py`, all scripts with DELETE operations
- ClickHouse schema: `schema.sql` -- partition key is `(symbol, threshold_decimal_bps)`, kintsugi_log has no partition key
- Production data: STATE.md documents the 300s timeout root cause from v13.50.0 partition migration

### Secondary (MEDIUM confidence)

- ClickHouse `DELETE IN PARTITION` syntax: verified from existing working code in `_do_replace()` (line 325)
- Python `dict.setdefault` thread safety: standard CPython GIL guarantee for dict operations

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- no new dependencies, modifying existing code
- Architecture: HIGH -- pattern already proven in `_do_replace()`, full call site audit completed
- Pitfalls: HIGH -- identified from direct code reading and schema analysis

**Research date:** 2026-03-24
**Valid until:** 2026-04-24 (stable domain, no external dependencies changing)
