# ClickHouse Cache Layer

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Schema**: [schema.sql](/python/opendeviationbar/clickhouse/schema.sql)

---

## Cache Population

### Resumable Population (Recommended for Long Ranges)

For date ranges > 30 days, use `populate_cache_resumable()`:

```python
from opendeviationbar import populate_cache_resumable

# Incremental, resumable, memory-safe
populate_cache_resumable(
    "BTCUSDT",
    start_date="2019-01-01",
    end_date="2025-12-31",
    threshold_decimal_bps=100,
)

# Force restart (wipe cache and checkpoint)
populate_cache_resumable(
    "BTCUSDT",
    start_date="2019-01-01",
    end_date="2025-12-31",
    force_refresh=True,
)
```

**How it works**:

1. Resumes from last checkpoint (local or ClickHouse)
2. Fetches raw tick data day-by-day
3. Processes with stateful `OpenDeviationBarProcessor` (preserves incomplete bars)
4. Stores bars incrementally to `opendeviationbar_cache.open_deviation_bars`
5. Saves checkpoint after each day (both local and ClickHouse)

### Native Method (Short Ranges ≤ 30 days)

Use `get_open_deviation_bars()` with `fetch_if_missing=True` to populate the cache:

```python
from opendeviationbar import get_open_deviation_bars

# Populate threshold 100 data for BTCUSDT
df = get_open_deviation_bars(
    "BTCUSDT",
    start_date="2023-06-01",
    end_date="2025-12-01",
    threshold_decimal_bps=100,
    use_cache=True,
    fetch_if_missing=True,
    include_microstructure=False,
)
print(f"Fetched {len(df):,} bars")
```

**How it works**:

1. Checks ClickHouse cache for existing data
2. If missing, fetches raw tick data from Binance (day-by-day to prevent OOM)
3. Computes open deviation bars via Rust backend
4. Stores results in `opendeviationbar_cache.open_deviation_bars` table
5. Returns pandas DataFrame

### Populating Remote Hosts

#### Prerequisites: ClickHouse Setup

**Docker-based setup** (recommended for hosts without sudo):

```bash
# SSH to host
ssh <host>

# Create data directory
mkdir -p ~/clickhouse-data

# Start ClickHouse container with passwordless HTTP access
# CRITICAL: -e CLICKHOUSE_PASSWORD= (empty) enables passwordless default user
docker run -d \
  --name clickhouse-server \
  -p 8123:8123 -p 9000:9000 \
  -v ~/clickhouse-data:/var/lib/clickhouse \
  -e CLICKHOUSE_PASSWORD= \
  --restart unless-stopped \
  clickhouse/clickhouse-server:latest

# Verify connection
curl "http://localhost:8123/?query=SELECT%201"
```

**Create schema** (run once after container starts):

```bash
# Download and execute schema
curl -s "http://localhost:8123/" --data-binary @- << 'EOF'
CREATE DATABASE IF NOT EXISTS opendeviationbar_cache
EOF

# Copy schema.sql content or use setup script
# See: scripts/setup-littleblack-clickhouse.sh for full schema
```

#### Install opendeviationbar

**CRITICAL**: Use `uv` instead of `pip` (Python 3.13 externally-managed-environment).

```bash
# Install uv if not present
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install opendeviationbar (latest version)
~/.local/bin/uv pip install opendeviationbar
```

#### Environment Variables

**CRITICAL**: mise SSoT pattern - check `mise.toml` for hardcoded hosts.

```bash
# Required for remote hosts not in default OPENDEVIATIONBAR_CH_HOSTS
export OPENDEVIATIONBAR_CH_HOSTS=localhost

# Required for low thresholds (below default minimum)
export OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD=1
```

**Gotcha**: If `mise.toml` sets `OPENDEVIATIONBAR_CH_HOSTS=bigblack`, remote hosts will fail silently. Override explicitly.

#### Run Population

```bash
# Start tmux session (long-running, survives SSH disconnect)
tmux new-session -d -s opendeviationbar-populate

# Attach and run
tmux attach -t opendeviationbar-populate

# Inside tmux:
export OPENDEVIATIONBAR_CH_HOSTS=localhost
export OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD=1

~/.local/bin/uv run python3 << 'PYEOF'
from opendeviationbar import populate_cache_resumable

populate_cache_resumable(
    "BTCUSDT",
    start_date="2023-06-01",
    end_date="2025-12-31",
    threshold_decimal_bps=100,
)
PYEOF

# Detach: Ctrl+B, D
# Reattach later: tmux attach -t opendeviationbar-populate
```

#### Troubleshooting Remote Setup

| Error                            | Cause                                       | Fix                                           |
| -------------------------------- | ------------------------------------------- | --------------------------------------------- |
| "Authentication failed"          | Container missing `-e CLICKHOUSE_PASSWORD=` | Recreate container with empty password flag   |
| "externally-managed-environment" | pip blocked on Python 3.13                  | Use `uv pip install` instead                  |
| ThresholdError "below minimum"   | Default minimum threshold validation        | Set `OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD=1` |
| Connection refused               | Wrong host in environment                   | Set `OPENDEVIATIONBAR_CH_HOSTS=localhost`     |
| "command not found: mise"        | mise not installed on remote                | Use explicit uv/python paths                  |

---

## Checking Cache Status

### Using cache_status.py script

```bash
python scripts/cache_status.py
```

### Direct ClickHouse query

```python
import clickhouse_connect
client = clickhouse_connect.get_client(host='localhost')
result = client.query('''
    SELECT symbol, threshold_decimal_bps, count(*) as bars,
           min(close_time_ms) as earliest, max(close_time_ms) as latest
    FROM opendeviationbar_cache.open_deviation_bars
    GROUP BY symbol, threshold_decimal_bps
''')
for row in result.result_rows:
    print(f"{row[0]} @ {row[1]} dbps: {row[2]:,} bars")
```

---

## Threshold Presets

| Preset       | Value | Use Case         |
| ------------ | ----- | ---------------- |
| `"micro"`    | 10    | Scalping         |
| `"tight"`    | 50    | Day trading      |
| `"standard"` | 100   | Swing trading    |
| `"medium"`   | 250   | Default          |
| `"wide"`     | 500   | Position trading |
| `"macro"`    | 1000  | Long-term        |

---

## Connection Mode

`OPENDEVIATIONBAR_MODE` env var (set in `.mise.toml`) controls host resolution in `preflight.py:get_available_clickhouse_host()`:

| Mode     | Behavior                                                          | Use Case                      |
| -------- | ----------------------------------------------------------------- | ----------------------------- |
| `auto`   | Try localhost first, then `OPENDEVIATIONBAR_CH_HOSTS` via SSH     | Default (auto-detect)         |
| `local`  | Force `localhost:8123` only                                       | Local ClickHouse only         |
| `remote` | Skip localhost, use `OPENDEVIATIONBAR_CH_HOSTS` via direct or SSH | **Current** (always bigblack) |
| `cloud`  | Require `CLICKHOUSE_HOST` env var                                 | Managed ClickHouse Cloud      |

**Current config** (`.mise.toml`): `OPENDEVIATIONBAR_MODE = "remote"` — all queries route through bigblack via SSH tunnel. No local ClickHouse.

**Preflight gate**: `mise run db:ensure` verifies bigblack reachable. All test tasks (`test-py`, `test:e2e`, `test:clickhouse`, `test:all`) depend on `db:ensure`.

---

## Host-Specific Cache Status

| Host        | Symbols                  | Thresholds Cached (dbps) | ClickHouse Setup | Notes             |
| ----------- | ------------------------ | ------------------------ | ---------------- | ----------------- |
| bigblack    | 15 Tier 1 crypto symbols | 250, 500, 750, 1000      | Native           | Primary GPU host  |
| bigblack    | EURUSD (forex)           | 50, 100, 200             | Native           | Exness Raw_Spread |
| littleblack | BTCUSDT                  | 100                      | Docker           | Secondary host    |

**Note**: Local ClickHouse has been removed. All data served from bigblack via SSH tunnel (`localhost:18123 → bigblack:8123`). The flowsurface app auto-starts the tunnel via `mise run preflight`.

**Minimum crypto threshold**: 250 dbps (enforced by `OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD` in `.mise.toml`)

**Total cached**: ~42M bars (growing — kintsugi backfilling historical gaps)

To add a threshold to a host, run the population script above on that host.

**Docker vs Native**: Use Docker when sudo unavailable. Native preferred for production (lower overhead).

---

## Dedup Hardening (Issue #90)

Five-layer protection against duplicate rows from OPTIMIZE timeout crashes and retry storms. Validated on bigblack (~42M rows, ALL LAYERS PASS).

### Five-Layer Architecture

| Layer | Mechanism                                                | Where                      | Scope        |
| ----- | -------------------------------------------------------- | -------------------------- | ------------ |
| 1     | `non_replicated_deduplication_window = 1000`             | `_ensure_schema()` ALTER   | Engine-level |
| 2     | INSERT dedup token (MD5 of `cache_key`)                  | `_build_insert_settings()` | Per-INSERT   |
| 3     | Fire-and-forget `OPTIMIZE` + merge polling               | `deduplicate_bars()`       | Post-write   |
| 4     | `SELECT ... FINAL` with partition parallelism            | `_FINAL_READ_SETTINGS`     | Query-time   |
| 5     | Schema migration on every `OpenDeviationBarCache()` init | `_ensure_schema()`         | Bootstrap    |

### Write Entry Point Map

All ClickHouse write paths emit INSERT dedup tokens:

| Caller                                 | Store Method                  | Dedup Token | `skip_dedup`            |
| -------------------------------------- | ----------------------------- | ----------- | ----------------------- |
| `get_open_deviation_bars()`            | `store_open_deviation_bars()` | Yes         | No (always `False`)     |
| `populate_cache_resumable()`           | `store_bars_batch()`          | Yes         | Param (default `False`) |
| `get_n_open_deviation_bars()`          | `store_bars_bulk()`           | Yes         | Param (default `False`) |
| `precompute_open_deviation_bars()`     | `store_bars_bulk()`           | Yes         | Param (default `False`) |
| `process_trades_to_dataframe_cached()` | `store_open_deviation_bars()` | Yes         | No (always `False`)     |

### Key Functions and Constants

| Symbol                     | File                 | Purpose                                                       |
| -------------------------- | -------------------- | ------------------------------------------------------------- |
| `_build_insert_settings()` | `bulk_operations.py` | Build `{insert_deduplicate, insert_deduplication_token}` dict |
| `_FINAL_READ_SETTINGS`     | `constants.py`       | `{"do_not_merge_across_partitions_select_final": 1}`          |
| `deduplicate_bars()`       | `maintenance.py`     | Fire-and-forget OPTIMIZE per partition (60s timeout)          |
| `_wait_for_merges()`       | `maintenance.py`     | Poll `system.merges` until OPTIMIZE completes                 |
| `_ensure_schema()`         | `cache.py`           | ALTER TABLE MODIFY SETTING (idempotent migration)             |

### `skip_dedup` Parameter

`store_bars_bulk()` and `store_bars_batch()` accept `skip_dedup: bool = False`:

- **`False` (default)**: Generates dedup token from `cache_key` column. Retrying the same INSERT is silently dropped by ClickHouse.
- **`True`**: Skips token. Used for `force_refresh=True` paths where cache was already deleted.

`store_open_deviation_bars()` always passes `False` (no `force_refresh` pathway).

### Validation

```bash
mise run cache:validate-dedup    # Run on bigblack (requires ClickHouse)
```

Validates all 4 runtime layers against live data. See `scripts/validate_dedup_hardening.py`.

---

## Gap Detection & Write Safety

### Silent Write Failure (Feb 2026 Incident)

`try_cache_write()` in `orchestration/open_deviation_bars_cache.py` catches all exceptions and returns `None` — correct for `get_open_deviation_bars()` (cache is optional) but **wrong** for `populate_cache_resumable()` (ClickHouse IS the destination).

**Fix**: `populate_cache_resumable()` uses `_fatal_cache_write()` which raises on failure and returns confirmed `written_rows`. The checkpoint only advances for successfully written days.

### `force_refresh` Atomicity Warning

`force_refresh=True` deletes ALL existing bars BEFORE writing new ones. If the write phase fails (connection drop, T-1 error), deleted data is unrecoverable. Mitigations:

- Fatal write path ensures early failure detection
- T-1 guard prevents end-of-range crash
- Checkpoint tracks `force_refresh_pending` state

### Gap Detection Query

`scripts/detect_gaps.py` uses ClickHouse `lagInFrame()` window function:

```sql
SELECT close_time_ms,
       lagInFrame(close_time_ms, 1) OVER (ORDER BY close_time_ms) AS prev_ts,
       (close_time_ms - prev_ts) / 3600000 AS gap_hours
FROM opendeviationbar_cache.open_deviation_bars FINAL
WHERE (symbol, threshold_decimal_bps) = ({symbol:String}, {threshold:UInt32})
HAVING gap_hours > {min_gap_hours:Float64}
```

**Note**: `neighbor()` is deprecated in ClickHouse 24.x+. Use `lagInFrame()` / `leadInFrame()`.

---

## Overlap Prevention (Issue #158 Phase 7-8)

### Overlap Strategy Semantics

| Strategy    | Semantic      | Behavior                                                                                                                                                                                  |
| ----------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"off"`     | Trust         | No overlap check. Single-writer (sidecar).                                                                                                                                                |
| `"warn"`    | Append        | Log warning, write anyway. L2 ReplacingMergeTree dedup.                                                                                                                                   |
| `"skip"`    | Defer         | Skip write if overlap exists. Another writer covered it.                                                                                                                                  |
| `"replace"` | Authoritative | `DELETE IN PARTITION` then INSERT (`mutations_sync=1`). Partition key is `(symbol, threshold_decimal_bps)` only — no date component. ORDER BY index prunes to matching trade-ID granules. |

**Routing**: `store_bars_batch(overlap_strategy="replace")` routes to `store_bars_replacing()` → `_do_replace()` in `maintenance.py`. Uses `DELETE IN PARTITION (symbol, threshold_decimal_bps)` with trade-ID WHERE clause for targeted deletion. `store_bars_bulk()` raises `ValueError` for "replace" (requires Polars).

**Anti-pattern (NEVER USE):** `DELETE FROM` (lightweight DELETE) — creates `_row_exists=0` ghost rows that persist until CH background merge, causing false Stathera anomalies. Banned since v13.49.0. See `/CLAUDE.md` section 7 for full strategy table.

### ORDER BY Change

```sql
-- Old: ORDER BY (symbol, threshold_decimal_bps, ouroboros_mode, close_time_ms)
-- New: ORDER BY (symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)
```

**Why**: Different processors produce different `close_time_ms` for the same trades. `first_agg_trade_id` uniquely identifies a bar's starting trade, so ReplacingMergeTree natively deduplicates overlapping bars on merge (keeping latest `computed_at`).

**Migration**: `scripts/migrate_order_by_first_agg_trade_id.py` (dry-run by default, `--apply` to execute).

**Skip index**: `idx_close_time` (minmax, granularity 8192) added via `_ensure_schema()` for time-range query performance.

### Redis Write Lock

`cache_write_lock(symbol, threshold_decimal_bps)` in `write_lock.py` — Redis-backed distributed lock prevents concurrent writes to the same (symbol, threshold). Gracefully degrades to no-op if Redis unavailable.

**Integration**: Wraps `store_bars_bulk()`, `store_bars_batch()`, and `store_bars_replacing()`. The `_skip_lock=True` parameter prevents deadlock when `store_bars_replacing()` calls `store_bars_batch()` internally.

**Config**: `OPENDEVIATIONBAR_REDIS_*` env vars (see `config/redis.py`).

---

## Files

| File                  | Purpose                                                                       |
| --------------------- | ----------------------------------------------------------------------------- |
| `cache.py`            | OpenDeviationBarCache class, core cache operations, dedup Layer 3-5           |
| `bulk_operations.py`  | BulkStoreMixin (store_bars_bulk, store_bars_batch), dedup Layer 2             |
| `maintenance.py`      | MaintenanceMixin (deduplicate_bars,\_wait_for_merges, optimize)               |
| `write_lock.py`       | Redis write lock for overlap prevention (Issue #158 Phase 7, L1)              |
| `query_operations.py` | QueryOperationsMixin (get_n_bars, get_bars_by_timestamp_range)                |
| `schema.sql`          | ClickHouse table schema (open_deviation_bars + population_checkpoints tables) |
| `stathera_query.py`   | Shared Stathera SQL builder (kintsugi, detect_gaps, heartbeat) (#259)         |
| `config.py`           | Connection configuration                                                      |
| `preflight.py`        | Installation checks                                                           |
| `tunnel.py`           | SSH tunnel support                                                            |

---

## Circuit Breaker (`fatal_write_breaker`)

Module-level circuit breaker in `orchestration/open_deviation_bars_cache.py` (Issue #108) protects against cascading ClickHouse write failures.

| Parameter            | Value             | Description                           |
| -------------------- | ----------------- | ------------------------------------- |
| `failure_threshold`  | 3                 | Consecutive failures before OPEN      |
| `recovery_timeout`   | 60s               | Time in OPEN before probing HALF_OPEN |
| `expected_exception` | `CacheWriteError` | Only these count toward threshold     |

**State machine**: CLOSED → (3 failures) → OPEN → (60s) → HALF_OPEN → (success) → CLOSED

**Telegram alert**: Sent on OPEN transition via `_on_circuit_state_change()` callback.

**When to use each write path**:

| Function               | Use Case                                                     | Raises on Failure             |
| ---------------------- | ------------------------------------------------------------ | ----------------------------- |
| `_fatal_cache_write()` | `populate_cache_resumable()` — ClickHouse IS the destination | Yes (circuit breaker wrapped) |
| `try_cache_write()`    | `get_open_deviation_bars()` — cache is optional              | No (logs warning)             |

---

## fd Exhaustion in Multi-Threaded Scripts (Issue #275)

**Anti-pattern**: Creating `OpenDeviationBarCache()` per loop iteration in multi-threaded code. Each instance creates a `urllib3.PoolManager(maxsize=20)`, holding up to 20 file descriptors. With N workers x 4 thresholds x 20 fds = 80N fds, this quickly exceeds `ulimit -n`.

**Best practice**: Thread-local cache reuse via `threading.local()`. One `OpenDeviationBarCache()` per worker thread, reused across all iterations.

The connection churn anti-pattern was verified empirically on bigblack: old code = 1,777 socket fds and 9 days/min; fixed code = 25 sockets and 24 days/min. The 2.7x speedup came entirely from eliminating connection setup overhead, not from any ClickHouse tuning.

```python
_thread_local = threading.local()

def _get_worker_cache() -> OpenDeviationBarCache:
    if not hasattr(_thread_local, "cache"):
        _thread_local.cache = OpenDeviationBarCache()
    return _thread_local.cache
```

**fd budget**: `LimitNOFILE=65536` is set in systemd services but **not** in ad-hoc shell sessions (default `ulimit -n` = 1024 on bigblack). Thread-local reuse keeps fd count at ~4N+50 for N workers, safely under the 1024 shell limit.

---

## Deploy Secrets (`local.env` Pattern)

On bigblack, secrets that must survive `git reset --hard` during deploys live in `deploy/opendeviationbar.local.env` (NOT git-tracked). All systemd services load this via `EnvironmentFile=-` (dash prefix = ignore if missing).

Contains: `OPENDEVIATIONBAR_TELEGRAM_TOKEN`, any other deploy-specific secrets.

See [scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md) for the full environment file pattern.

---

## Pending Migrations

### Delta+ZSTD Codec (Issue #202)

Four timestamp/trade-ID columns benefit from Delta encoding (sequential values compress dramatically). Run on bigblack when connectivity is restored:

```sql
ALTER TABLE opendeviationbar_cache.open_deviation_bars
    MODIFY COLUMN close_time_ms CODEC(Delta, ZSTD),
    MODIFY COLUMN open_time_ms CODEC(Delta, ZSTD),
    MODIFY COLUMN first_agg_trade_id CODEC(Delta, ZSTD),
    MODIFY COLUMN last_agg_trade_id CODEC(Delta, ZSTD);
```

**Notes**:

- Non-blocking (`ALTER TABLE ... MODIFY COLUMN` is metadata-only; data re-encoded on next merge)
- Verify after: `SELECT name, compression_codec FROM system.columns WHERE table = 'open_deviation_bars' AND name IN ('close_time_ms', 'open_time_ms', 'first_agg_trade_id', 'last_agg_trade_id')`
- Expected storage reduction: ~30-50% on these columns (monotonically increasing values)

---

## Related

- [/CLAUDE.md](/CLAUDE.md) - Project hub (architecture, principles, terminology)
- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) - Python API, memory guards, validation
- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) - Pueue ops, per-year parallelization (default strategy), anti-patterns
- [/scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md) - systemd services, timers, dependency chain
- [/crates/CLAUDE.md](/crates/CLAUDE.md) - Rust crates, microstructure feature formulas
