---
name: bulk-backfill
description: >-
  High-throughput historical bar backfill on bigblack using repair_direct_parquet.py
  (200 days/min, 500x faster than kintsugi genesis). Use whenever the user needs to
  seed bars for a new threshold, backfill thousands of days of historical data, or
  accelerate any large-scale bar population that kintsugi would take days/weeks to
  complete. Also use when the user mentions repair_direct_parquet, fast backfill,
  bulk seeding, or asks why kintsugi is slow for genesis/historical data.
  TRIGGERS - bulk backfill, historical backfill, fast backfill, seed bars, 100dbps
  backfill, repair parquet, high-throughput backfill, accelerate backfill, fast
  seeding, genesis slow, kintsugi too slow, repair direct parquet.
allowed-tools: Read, Bash, Grep, Glob, AskUserQuestion
---

# Bulk Backfill: High-Throughput Historical Bar Seeding

## Why This Exists

Kintsugi's genesis discovery algorithm creates **1 shard covering 1 day** per (symbol, threshold) per pass. With 60s polling between passes, this means ~0.4 days/min for a single pair. For a 2,555-day gap, that's **4-5 days**.

`repair_direct_parquet.py` bypasses kintsugi entirely. It reads raw trades from the Parquet tick cache, processes them through the Rust bar constructor with 24 parallel workers, and bulk-INSERTs to ClickHouse. Measured throughput: **100-200 days/min** depending on bar density.

| Method                   | Throughput       | 2,555 days    | Why                                      |
| ------------------------ | ---------------- | ------------- | ---------------------------------------- |
| Kintsugi genesis         | 0.4 days/min     | ~4.5 days     | 1-day shard discovery per pass           |
| repair_direct_parquet.py | 100-200 days/min | **13-25 min** | 24 parallel workers, direct Parquet read |

Use kintsugi for ongoing gap healing (it's designed for that). Use this skill for bulk historical seeding of new thresholds or symbols.

---

## Prerequisites

Before starting, verify these conditions on bigblack:

### 1. Parquet Tick Cache

The script reads from `~/.cache/opendeviationbar/ticks/BINANCE_SPOT_{SYMBOL}/`. If ticks aren't cached, the script logs `no_parquet` and skips those days.

```bash
# Check coverage
ssh bigblack 'ls ~/.cache/opendeviationbar/ticks/BINANCE_SPOT_{SYMBOL}/ | wc -l'

# Check specific date range
ssh bigblack 'ls ~/.cache/opendeviationbar/ticks/BINANCE_SPOT_BTCUSDT/ | head -5 && echo "..." && ls ~/.cache/opendeviationbar/ticks/BINANCE_SPOT_BTCUSDT/ | tail -5'
```

If coverage is incomplete, seed first:

```bash
ssh bigblack 'systemctl --user start opendeviationbar-seeder.timer'
# Or one-off: .venv/bin/python3 scripts/tick_cache_seeder.py
```

The seeder downloads from Binance Vision at ~48 files/min with 12 concurrent downloads. A full 8-year history takes ~2 hours.

### 2. Symbol Registration

The symbol must be enabled in `symbols.toml`. Check:

```bash
ssh bigblack 'grep -A5 "BTCUSDT" ~/opendeviationbar-py/symbols.toml | head -10'
```

### 3. ClickHouse Running

```bash
ssh bigblack 'clickhouse-client -q "SELECT 1"'
```

### 4. System Resources

```bash
ssh bigblack 'uptime && free -h | head -2'
```

The script needs ~13 GB RSS at peak. Bigblack has 61 GB total. Check that load average is reasonable (< 10) and at least 15 GB RAM is available.

---

## Execution

### Step 1: Dry Run

Always dry-run first. It checks how many days need repair and samples Parquet availability:

```bash
ssh bigblack 'cd ~/opendeviationbar-py && \
  .venv/bin/python3 scripts/repair_direct_parquet.py \
  --threshold {DBPS} --symbol {SYMBOL}'
```

Output tells you: `N days need repair` and `M/20 sampled days have Parquet`.

### Step 2: Execute

```bash
ssh bigblack 'cd ~/opendeviationbar-py && \
  nohup .venv/bin/python3 scripts/repair_direct_parquet.py \
  --threshold {DBPS} --symbol {SYMBOL} \
  --execute --workers 24 \
  > /tmp/repair-{SYMBOL}-{DBPS}dbps.log 2>&1 & \
  echo "PID: $!"'
```

**Parameters:**

- `--threshold`: one or more thresholds in dbps (e.g., `--threshold 100` or `--threshold 100 250 500`)
- `--symbol`: filter to one symbol (omit to process BTCUSDT + ETHUSDT)
- `--workers 24`: the empirical sweet spot on bigblack (do NOT exceed — see Resource Limits)
- `--limit N`: process at most N days (useful for testing)

### Step 3: Monitor Progress

```bash
# Throughput and completion
ssh bigblack 'grep -E "/min|complete:|days need" /tmp/repair-{SYMBOL}-{DBPS}dbps.log | tail -10'

# Process health (RSS in KB, CPU%, elapsed time)
ssh bigblack 'ps -p {PID} -o pid,rss,pcpu,etime --no-headers'

# ClickHouse query pressure (should be < 145)
ssh bigblack 'clickhouse-client -q "SELECT count() FROM system.processes"'

# Threshold distribution (verify correct threshold)
ssh bigblack 'grep "threshold=" /tmp/repair-{SYMBOL}-{DBPS}dbps.log | grep -oP "threshold=\d+" | sort | uniq -c | sort -rn'
```

**Expected throughput by bar density:**

| Threshold | Bars/day (BTC) | Days/min | Notes                        |
| --------- | -------------- | -------- | ---------------------------- |
| 100 dbps  | 2,000-25,000   | 93-200   | Slows on recent dense data   |
| 250 dbps  | 500-2,000      | 200+     | Original benchmark threshold |
| 500 dbps  | 100-500        | 200+     | Light processing             |
| 750 dbps  | 30-200         | 200+     | Lightest                     |

### Step 4: Verify Completion

After the script logs `{SYMBOL} complete:`, run Stathera audit:

```bash
ssh bigblack 'cd ~/opendeviationbar-py && \
  .venv/bin/python3 scripts/detect_gaps.py \
  --symbol {SYMBOL} --threshold {DBPS}'
```

Exit code 0 = clean. Exit code 1 = gaps found (kintsugi will heal these on its next pass).

Also check bar counts:

```bash
ssh bigblack "clickhouse-client -q \"
  SELECT symbol, threshold_decimal_bps, count() as bars,
         min(toDate(open_time_ms/1000, 'UTC')) as first_day,
         max(toDate(open_time_ms/1000, 'UTC')) as last_day
  FROM opendeviationbar_cache.open_deviation_bars
  WHERE symbol = '{SYMBOL}' AND threshold_decimal_bps = {DBPS}
  GROUP BY symbol, threshold_decimal_bps
\""
```

---

## Resource Limits (bigblack)

These limits are empirically validated. Violating them causes silent throughput collapse.

| Resource   | Limit              | What happens if exceeded                                                            |
| ---------- | ------------------ | ----------------------------------------------------------------------------------- |
| Workers    | **24 max**         | 32 workers = ClickHouse contention collapse (0 completions/30s, 271 active queries) |
| RSS        | ~13 GB peak        | Allocator purge every 50 days keeps it bounded                                      |
| FD budget  | ~146 at 24 workers | Thread-local CH cache: ~4N+50 fds. Safe under ulimit -n 1024                        |
| CH queries | ~143 concurrent    | Soft cap. Pipeline parallelism (Parquet+Rust+CH) overlaps cleanly at 24             |

### Cohabitation with Other Services

The script runs alongside the production stack. Resource awareness:

| Service           | Memory   | Impact                                                    |
| ----------------- | -------- | --------------------------------------------------------- |
| Sidecar           | ~1.2 GB  | Latency-critical, OOMScoreAdjust=-500 (survives pressure) |
| Main kintsugi     | ~1.5 GB  | Will contend on ClickHouse if processing same pairs       |
| Heartbeat         | ~100 MB  | 5-min timer, negligible                                   |
| **repair script** | ~3-13 GB | The big consumer during backfill                          |

**Total during backfill**: ~17 GB of 61 GB available. Safe.

**ClickHouse contention**: If main kintsugi is actively processing the SAME (symbol, threshold), Redis lock prevents conflicts — one will wait/skip. Different pairs are fully independent.

---

## Anti-Patterns

### Don't use kintsugi for bulk genesis seeding

Kintsugi's `_discover_genesis_shards()` creates exactly 1 shard covering 1 day per pass. `BATCH_DAYS=365` has no effect because genesis shards are always 1-day wide. The 24-worker pool is wasted (only 3 shards discovered). Use repair_direct_parquet.py instead.

### Don't exceed 24 workers

Empirically validated cliff at 32 workers: CPU paradoxically drops (14% vs 22%), ClickHouse active queries double to 271, and zero completions in a 4-minute observation window. The ClickHouse internal query scheduler is the bottleneck, not CPU or I/O.

### Don't run from /tmp

The #275 incident: copying scripts to `/tmp` and running from there caused "old code still running" bugs. After `git pull`, the repo script IS the latest version. Always run from `~/opendeviationbar-py/scripts/`.

### Don't skip the Parquet check

Days without Parquet cache log `no_parquet` and produce zero bars. If many days lack cache, you're wasting worker slots. Seed first with `tick_cache_seeder.py`.

### Don't use overlap_strategy="replace"

The script uses `overlap_strategy="warn"` (plain INSERT). The "replace" strategy runs a DELETE scan before each INSERT, which times out at 300s with 1,658+ active ClickHouse parts. ReplacingMergeTree dedup handles any duplicates on merge.

### Don't run alongside kintsugi-catchup on same pairs

Both hammer ClickHouse. Either stop kintsugi-catchup during the bulk run, or ensure they target different (symbol, threshold) pairs.

---

## Cleanup After Backfill

1. **Remove the log**: `ssh bigblack 'rm /tmp/repair-*.log'`
2. **Verify kintsugi sees clean data**: Wait for the next kintsugi pass and check heartbeat — Stathera should show zero anomalies for the backfilled pairs.
3. **Let ReplacingMergeTree settle**: Any overlap warnings during the run are benign — ClickHouse merges deduplicate by `first_agg_trade_id` within hours.

---

## Example: Full 100dbps Backfill (Issue #299)

This is the canonical example that motivated this skill.

```bash
# 1. Check Parquet coverage
ssh bigblack 'ls ~/.cache/opendeviationbar/ticks/BINANCE_SPOT_BTCUSDT/ | wc -l'
# Expected: ~2600+ files (one per day since 2018-01-16)

# 2. Dry run
ssh bigblack 'cd ~/opendeviationbar-py && \
  .venv/bin/python3 scripts/repair_direct_parquet.py --threshold 100'
# Expected: "BTCUSDT: ~2500 days need repair", "20/20 sampled days have Parquet"

# 3. Execute
ssh bigblack 'cd ~/opendeviationbar-py && \
  nohup .venv/bin/python3 scripts/repair_direct_parquet.py \
  --threshold 100 --execute --workers 24 \
  > /tmp/repair-100dbps.log 2>&1 & echo "PID: $!"'

# 4. Monitor (run periodically)
ssh bigblack 'grep -E "/min|complete:" /tmp/repair-100dbps.log | tail -5'
# Expected: 93-200 days/min, ~25 min total for BTC+ETH

# 5. Verify
ssh bigblack 'cd ~/opendeviationbar-py && \
  .venv/bin/python3 scripts/detect_gaps.py --threshold 100'
```

---

## Related Skills

- `Skill(remote-backfill)` — pueue-based backfill via `populate_cache_resumable()` (slower, for bounded ranges)
- `Skill(perf-analysis)` — resource-aware parallelism analysis, ClickHouse capacity model
- `Skill(opendeviationbar-job-safety)` — memory profiles, per-symbol thresholds
- `Skill(odb-telemetry)` — production telemetry, heartbeat interpretation
