# ClickHouse Write Path Safety Audit

**Date**: 2026-03-21
**Scope**: All ClickHouse write commands in `python/opendeviationbar/clickhouse/` and callers

---

## 1. Complete ClickHouse Command Catalog

### INSERT Paths (2 entry points)

| Entry Point                   | File                     | Method                  | Protocol | Error Handling                                                  |
| ----------------------------- | ------------------------ | ----------------------- | -------- | --------------------------------------------------------------- |
| `store_bars_bulk()`           | `bulk_operations.py:371` | `client.insert_df()`    | HTTP     | Catches `CLICKHOUSE_TRANSIENT_ERRORS`, raises `CacheWriteError` |
| `store_bars_batch()`          | `bulk_operations.py:687` | `client.insert_arrow()` | HTTP     | **NO try/except** -- exception propagates raw to caller         |
| `store_open_deviation_bars()` | `cache.py:433`           | `client.insert_df()`    | HTTP     | Catches `CLICKHOUSE_TRANSIENT_ERRORS`, raises `CacheWriteError` |

### DELETE Paths (4 distinct patterns)

| Path                        | File:Line            | Command                                     | mutations_sync | Wait Mechanism                                 | Silent Fail Risk                                                                 |
| --------------------------- | -------------------- | ------------------------------------------- | -------------- | ---------------------------------------------- | -------------------------------------------------------------------------------- |
| `_do_replace()`             | `maintenance.py:395` | `DELETE FROM ... SETTINGS mutations_sync=1` | **YES (1)**    | Synchronous (blocks)                           | LOW -- sync guarantees completion before INSERT                                  |
| `_delete_today_bars()`      | `sidecar.py:735`     | `DELETE FROM ... SETTINGS mutations_sync=1` | **YES (1)**    | Synchronous (blocks)                           | LOW -- but wrapped in bare `except Exception` that logs warning and continues    |
| `delete_bars_by_identity()` | `maintenance.py:255` | `ALTER TABLE DELETE`                        | **NO**         | `_wait_for_mutations()` polling (300s timeout) | **MEDIUM** -- timeout logs warning but returns, caller assumes deletion complete |
| `_delete_ghost_bar()`       | `backfill.py:1430`   | `ALTER TABLE DELETE`                        | **NO**         | **NONE**                                       | **HIGH** -- fire-and-forget, no sync, no wait, no verification                   |
| `_aletheia_purge()`         | `kintsugi.py:600`    | `ALTER TABLE kintsugi_log DELETE`           | **NO**         | **NONE**                                       | LOW -- only affects log table, not bar data                                      |

### OPTIMIZE Paths (1 pattern)

| Path                 | File:Line                | Command                    | Risk                                                                              |
| -------------------- | ------------------------ | -------------------------- | --------------------------------------------------------------------------------- |
| `deduplicate_bars()` | `maintenance.py:141-200` | `OPTIMIZE TABLE ... FINAL` | LOW -- fire-and-forget by design, 60s client timeout, merge continues server-side |

---

## 2. Findings: Can Writes Silently Lose or Corrupt Data?

### FINDING-1: `store_bars_batch()` has NO exception handling (MEDIUM severity)

**Location**: `bulk_operations.py:978-984`

```python
summary = self.client.insert_arrow(
    "opendeviationbar_cache.open_deviation_bars",
    arrow_table,
    settings=insert_settings,
)
return summary.written_rows
```

Unlike `store_bars_bulk()` (which catches `CLICKHOUSE_TRANSIENT_ERRORS` and wraps them in `CacheWriteError`), `store_bars_batch()` lets any exception propagate raw. This is not "silent" data loss, but it means:

- The circuit breaker in `_fatal_cache_write()` may not categorize the error correctly
- Callers (kintsugi, sidecar) each implement their own error handling, leading to inconsistency

**Impact**: Not silent loss, but inconsistent error surface. Callers like kintsugi catch `Exception` broadly (kintsugi.py:1222, 1354), which masks the root cause.

### FINDING-2: `_delete_ghost_bar()` is fully fire-and-forget (HIGH severity)

**Location**: `backfill.py:1428-1456`

Uses `ALTER TABLE DELETE` without `mutations_sync` and without any wait. The DELETE is asynchronous. Immediately after, the caller writes new bars (backfill.py:1367-1380). If:

1. The DELETE hasn't executed yet when the INSERT arrives
2. The old bar has a different `first_agg_trade_id` from the new bar (which is the entire point of ghost bar cleanup)
3. Both bars coexist until background merge

**Result**: Ghost bar persists alongside the corrected bar until RMT merge. Since they have different `first_agg_trade_id` (different ORDER BY key), RMT **cannot** deduplicate them. The ghost bar persists **indefinitely** until an explicit `OPTIMIZE FINAL`.

**Mitigation**: This is used by `rectify_recent_bars()` which is a maintenance function, not a hot path. But the ghost bar it's trying to delete will survive.

### FINDING-3: `_delete_today_bars()` swallows DELETE failure (MEDIUM severity)

**Location**: `sidecar.py:745-746`

```python
except Exception:
    logger.warning("clean_slate: failed to delete today's bars (non-fatal)", exc_info=True)
```

If the DELETE fails, the sidecar continues with `fill_from_rest()`, which writes new bars. The old (stale) bars from a previous sidecar session coexist with new bars. They may have overlapping trade-ID ranges but different `first_agg_trade_id` values (different processor sessions), meaning RMT cannot dedup them.

**Impact**: Duplicate bars for today until next `OPTIMIZE FINAL` or kintsugi repair. The startup sequence doc says step 3 (delete today) MUST happen before step 4 (seed from CH). If delete fails silently, the seed reads stale high-water marks, inflating the seed TID and potentially skipping trades.

### FINDING-4: `delete_bars_by_identity()` timeout is non-fatal (LOW severity)

**Location**: `maintenance.py:493-521`

`_wait_for_mutations()` polls for 300s, then logs a warning and returns. The caller (`delete_bars_by_identity`) logs "Deleted N bars" regardless. The mutation may still be running.

**Impact**: Low in practice because this is used for Stathera cleanup (old overlapping bars), and the mutations typically complete within seconds. But the API contract is misleading -- it returns a count that implies completion.

### FINDING-5: `_do_replace()` DELETE predicate can be broader than intended (LOW severity)

**Location**: `maintenance.py:395-405`

The DELETE predicate is:

```sql
AND first_agg_trade_id <= {new_last}
AND last_agg_trade_id >= {new_first}
AND first_agg_trade_id > 0
```

This catches **any** bar whose trade-ID range overlaps the new bars, including straddle bars that extend beyond the new range. This is intentional (the docstring says "straddle bars"), but if new bars cover only part of a day, bars from adjacent time periods within that day could be deleted if their trade IDs happen to overlap.

**Mitigation**: In practice, kintsugi repairs full UTC days, so the new bars always cover the complete trade-ID range for the day. The DELETE is scoped to (symbol, threshold, ouroboros_mode).

### FINDING-6: SQL injection via f-string interpolation (LOW severity, internal-only)

**Locations**:

- `maintenance.py:249`: `f"DELETE WHERE symbol = '{symbol}'"` -- `symbol` comes from internal callers
- `maintenance.py:396-404`: `_do_replace()` uses f-strings for symbol, threshold, ouroboros_mode, trade IDs
- `sidecar.py:737-739`: `_delete_today_bars()` uses f-string for ouroboros_mode

All values come from internal Python code (symbol registry, config), not user input. No real attack surface. But it breaks the pattern of parameterized queries used elsewhere (e.g., `_find_straddle_bars` uses `{symbol:String}` parameters).

### FINDING-7: INSERT dedup token collision risk (VERY LOW severity)

**Location**: `bulk_operations.py:280-297`

The dedup token is derived from `cache_key`, which is `MD5(symbol_threshold_startTs_endTs_ouroboros)`. If two different bar batches have the same time range (e.g., kintsugi recompute produces same start/end timestamps), the second INSERT is silently dropped by ClickHouse's `non_replicated_deduplication_window`.

**Mitigation**: `_do_replace()` passes `skip_dedup=True` to avoid this. Normal writes use dedup as a retry-safety net. The window is 1000 blocks, so collisions age out.

### FINDING-8: `store_bars_batch()` returns `summary.written_rows` without verifying it matches input (INFO)

**Location**: `bulk_operations.py:984`

After filtering (physical invariant gate, day-mode gate), the actual rows sent to ClickHouse may differ from the input. The return value is from `insert_arrow().written_rows`, which reflects what ClickHouse accepted. This is correct behavior. However:

- If `insert_deduplicate` drops the entire block (token collision), `written_rows` = 0 but no error is raised
- Callers (kintsugi) use `written` to determine success, and `written=0` on a dedup collision looks like "no bars produced"

---

## 3. Risk Summary

| Finding                                   | Severity | Silent Data Loss?          | Fix Complexity                               |
| ----------------------------------------- | -------- | -------------------------- | -------------------------------------------- |
| F1: `store_bars_batch` no try/except      | MEDIUM   | No (exception propagates)  | LOW -- add same pattern as `store_bars_bulk` |
| F2: `_delete_ghost_bar` fire-and-forget   | HIGH     | Ghost bars persist         | LOW -- add `mutations_sync=1`                |
| F3: `_delete_today_bars` swallows failure | MEDIUM   | Stale bars + inflated seed | MEDIUM -- fail startup if delete fails       |
| F4: `_wait_for_mutations` timeout         | LOW      | Mutation may not finish    | LOW -- return bool                           |
| F5: `_do_replace` broad predicate         | LOW      | No (intentional design)    | N/A                                          |
| F6: SQL f-string interpolation            | LOW      | No (internal-only)         | LOW -- switch to parameters                  |
| F7: Dedup token collision                 | VERY LOW | Block silently dropped     | N/A (by design)                              |
| F8: written_rows=0 on dedup               | INFO     | No (correct behavior)      | N/A                                          |

---

## 4. Recommendations (Priority Order)

### P0: Fix `_delete_ghost_bar` (F2)

Add `SETTINGS mutations_sync=1` or switch to `DELETE FROM ... SETTINGS mutations_sync=1`. Without this, ghost bars are guaranteed to persist until manual OPTIMIZE.

### P1: Harden `_delete_today_bars` failure handling (F3)

The bare `except Exception` that logs-and-continues is dangerous because it silently allows stale seed TIDs. At minimum, if DELETE fails, the function should return a failure signal so `_create_engine()` can abort or skip the CH seed step.

### P2: Add consistent error handling to `store_bars_batch` (F1)

Wrap `insert_arrow()` in the same `CLICKHOUSE_TRANSIENT_ERRORS` catch + `CacheWriteError` raise pattern used by `store_bars_bulk()`.

### P3: Normalize f-string SQL to parameterized queries (F6)

Low priority but improves code consistency and prevents future mistakes if symbols ever come from external input.
