---
name: perf-analysis
description: >-
  Resource-aware parallelism analysis for opendeviationbar-py backfill and cache population.
  Encodes ClickHouse capacity model (parts pressure, merge pool), network bandwidth model
  (Binance Vision S3, bigblack 1Gbps link), memory/CPU budgets, composite safety formula,
  monitoring queries, backfill ETA estimation, and escalation thresholds.
  TRIGGERS - how many workers, parallel workers, safe parallelism, backfill ETA,
  ClickHouse capacity, parts pressure, merge queue, network bandwidth, memory budget,
  resource analysis, performance analysis, capacity planning, worker count.
allowed-tools: Read, Bash, Grep, Glob
compatibility: ssh, clickhouse-client (bigblack host access required)
---

# Resource-Aware Parallelism Analysis

Determines the safe number of concurrent populate workers for opendeviationbar-py cache backfill on bigblack. Prevents ClickHouse parts pressure, OOM kills, network saturation, and CPU starvation through a composite capacity model validated against production measurements.

**Born from**: March 2026 capacity analysis where unconstrained parallelism caused parts accumulation and merge queue stalls on bigblack.

## Prerequisite Skills

- `Skill(remote-backfill)` -- command formulation, pueue groups, ouroboros strategy
- `Skill(opendeviationbar-job-safety)` -- memory profiles per threshold tier, cgroup limits

---

## Task Templates

### Calculate Safe Worker Count

```
[Preflight]
1. SSH to bigblack, run all 5 capacity probes (Section: Live Capacity Probes)
2. Record current service memory footprint
3. Check active parts count and merge queue depth

[Execute]
4. Run capacity model: python .claude/skills/perf-analysis/models/capacity_model.py
   OR apply Composite Safety Formula manually with measured values
5. Take the minimum across all 4 dimensions

[Verify]
6. Start workers at calculated count
7. Monitor for 15 minutes using Monitoring Queries (queries/ directory)
8. Check escalation thresholds -- if any Yellow, hold; if any Red, reduce
```

### Monitor Active Backfill Health

```
[Preflight]
1. Confirm backfill is running: ssh bigblack 'pueue status --group GROUP'

[Execute]
2. Run monitoring queries every 10-15 minutes:
   - queries/active_parts.sql -- parts accumulation trend
   - queries/merge_queue.sql -- merge backlog
   - queries/insert_latency.sql -- write performance
   - queries/memory_pressure.sql -- ClickHouse RSS
3. Compare against Escalation Thresholds table

[Verify]
4. All metrics in Green zone -- continue
5. Any Yellow -- hold parallelism, do not increase
6. Any Red -- reduce workers immediately (pueue parallel N-2 --group GROUP)
```

### Estimate Backfill ETA

```
[Preflight]
1. Identify symbols and date ranges to backfill
2. Look up volume class for each symbol (Section: Backfill ETA Model)

[Execute]
3. Calculate total symbol-days
4. Apply blended rate from ETA model
5. Divide by planned worker count

[Verify]
6. Cross-check against Speed Reference in Skill(remote-backfill)
7. Add 20% contingency for Vision download variability
```

---

## ⚠️ Scope: Two Distinct Workloads

This skill contains **two separate capacity models** for different workloads:

| Workload                                       | Bottleneck                                         | Safe Workers                  | Section                                                           |
| ---------------------------------------------- | -------------------------------------------------- | ----------------------------- | ----------------------------------------------------------------- |
| `populate_cache_resumable()` (Vision-download) | **Network** — USB ethernet / 1 Gbps link           | 6                             | Sections 1-4 below                                                |
| `kintsugi` daemon (Parquet-cached)             | **ClickHouse** query scheduler (~143-145 soft cap) | **24** (empirical hard limit) | [Kintsugi Parallelism](#kintsugi-parallelism-parquet-cached-mode) |

**Do NOT apply the Vision-download composite formula to kintsugi.** When BTCUSDT Parquet cache is populated, kintsugi has zero network I/O — all data is served from local NVMe page cache. The bottleneck is entirely different.

---

## Kintsugi Parallelism (Parquet-cached mode)

Empirical step-up test on bigblack (2026-03-13): `MAX_CONCURRENT_REPAIRS` governs outer parallelism for kintsugi when Parquet cache is populated.

| MAX_CONCURRENT_REPAIRS | CPU  | CH active queries | Repairs/30s | Verdict                       |
| ---------------------- | ---- | ----------------- | ----------- | ----------------------------- |
| 8                      | 8.8% | 130               | ~8          | baseline                      |
| 16                     | 19%  | 138               | 17          | linear ✓                      |
| **24**                 | 22%  | **143**           | **44**      | super-linear ✓ **sweet spot** |
| 32                     | 14%  | 271               | **0**       | cliff ✗ — contention          |

**Bottleneck**: ClickHouse internal query scheduler. Soft cap ≈143-145 concurrent active queries. 24 workers saturate it cleanly. CPU (22%), NVMe (page cache hits), and RAM (4.6 GB of 36 GB) are NOT the constraints.

**Why 32 fails**: CH active queries double to 271, exceeding the soft cap. CPU paradoxically drops (threads block waiting for CH scheduler). Zero completions in 4-minute observation.

**Hard rule**: `MAX_CONCURRENT_REPAIRS=24` on bigblack (Ryzen 9 7950X + local NVMe + local ClickHouse). **Do NOT increase to 32.** Env var: `OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REPAIRS` in `deploy/opendeviationbar.env`.

**Sustained throughput**: 44 repairs/30s window → ~5000+ repairs/hr (burst). Sustained rate depends on shard queue depth.

**SSoT**: [scripts/CLAUDE.md — Kintsugi Parallelism Tuning](/scripts/CLAUDE.md#kintsugi-parallelism-tuning-bigblack)

---

## Vision-Download Populate Workers: Composite Capacity Model

The following sections apply to `populate_cache_resumable()` / pueue backfill workers that download from Binance Vision S3.

---

## Capacity Models

### 1. ClickHouse Capacity Model

ClickHouse is the binding constraint for write-heavy backfill. Each INSERT creates new parts that must be merged in the background.

| Parameter               | Value   | Source                        |
| ----------------------- | ------- | ----------------------------- |
| `parts_to_delay_insert` | 1000    | ClickHouse server default     |
| `parts_to_throw_insert` | 3000    | ClickHouse server default     |
| `max_parts_in_total`    | 100000  | ClickHouse server default     |
| `background_pool_size`  | 16      | bigblack config (32-core box) |
| Insert latency (single) | 3ms avg | Measured baseline             |
| Insert latency (single) | 6ms max | Measured baseline             |

**Rule**: `max_workers <= background_pool_size * merge_efficiency_factor`

The merge efficiency factor (0.6) accounts for the fact that not all merge threads are available -- some handle mutations, projections, and other background operations.

```
ch_safe_workers = floor(16 * 0.6) = 9
```

**Monitoring queries**: See `queries/active_parts.sql`, `queries/merge_queue.sql`, `queries/insert_latency.sql`

### 2. Network Capacity Model

Binance Vision serves historical trade ZIPs from S3. No rate limiting, but bigblack's link is the bottleneck.

| Parameter              | Value      | Source                   |
| ---------------------- | ---------- | ------------------------ |
| Bigblack link          | 1 Gbps     | ISP contract             |
| Theoretical throughput | 125 MB/s   | 1 Gbps / 8               |
| Per-worker bandwidth   | ~15 MB/s   | Measured (Vision ZIPs)   |
| Per-ZIP size           | 15-25 MB   | BTCUSDT daily trade ZIPs |
| Per-ZIP download time  | ~1.3-1.5 s | Measured                 |

**Rule**: `max_workers <= link_bandwidth_mbps / per_worker_bandwidth * safety_margin`

Safety margin (0.8) reserves bandwidth for sidecar WebSocket streams, SSH sessions, and ClickHouse replication.

```
net_safe_workers = floor((125 / 15) * 0.8) = 6
```

### 3. Memory Capacity Model

Each populate worker holds a day of trades in memory during processing, plus the Rust processor state.

| Component           | Memory    | Source       |
| ------------------- | --------- | ------------ |
| Per populate worker | ~200 MB   | Peak RSS     |
| ClickHouse server   | ~2.3 GB   | Baseline RSS |
| Streaming sidecar   | ~326 MB   | Measured     |
| Kintsugi daemon     | ~1.6 GB   | Measured     |
| Backfill daemon     | ~591 MB   | Measured     |
| OS + headroom       | 4 GB      | Reserved     |
| **Bigblack total**  | **61 GB** | Physical RAM |

**Rule**: `max_workers <= (total_ram - services - headroom) / per_worker_mem`

```
available = 61 - 2.3 - 0.326 - 1.6 - 0.591 - 4.0 = 52.2 GB
mem_safe_workers = floor(52.2 / 0.2) = 261
```

Memory is rarely the binding constraint for populate workers (200 MB each), but becomes critical at lower thresholds (250/100 dbps) where per-worker memory is 1-5 GB. See `Skill(opendeviationbar-job-safety)` for threshold-specific memory profiles.

**With heavy thresholds (250 dbps)**:

```
available = 52.2 GB
mem_safe_workers_250 = floor(52.2 / 5.0) = 10  # BTC/ETH @250
mem_safe_workers_250 = floor(52.2 / 3.0) = 17  # Altcoins @250
```

### 4. CPU Capacity Model

Populate workers are I/O-bound (waiting on Vision downloads), but ClickHouse merge threads are CPU-bound.

| Component                     | CPU Usage    | Source                 |
| ----------------------------- | ------------ | ---------------------- |
| Per populate worker           | ~0.5 cores   | I/O-bound (network)    |
| ClickHouse merge threads      | 16 cores     | CPU-bound during merge |
| Sidecar + kintsugi + backfill | ~2 cores     | Measured               |
| **Bigblack total**            | **32 cores** | Physical               |

**Rule**: `max_workers <= (total_cores * 0.5 - service_cores) / per_worker_cores`

Reserve 50% of cores for ClickHouse merges and services.

```
cpu_safe_workers = floor((32 * 0.5 - 2) / 0.5) = 28
```

CPU is rarely the binding constraint for I/O-bound populate workers.

---

## Composite Safety Formula

The safe worker count is the **minimum** across all four dimensions:

```
safe_workers = min(
    floor(merge_pool_size * 0.6),                           # ClickHouse: 9
    floor(link_bw_mbps / per_worker_bw * 0.8),              # Network:    6
    floor(avail_ram_gb / per_worker_gb),                     # Memory:   261 (or 10 for heavy)
    floor((cores * 0.5 - service_cores) / per_worker_cores), # CPU:       28
)
```

**Typical result for light thresholds (1000/750/500 dbps)**: `min(9, 6, 261, 28) = 6`

**Typical result for heavy thresholds (250 dbps, BTC/ETH)**: `min(9, 6, 10, 28) = 6`

**Network is the binding constraint for Vision-download populate workers.** ClickHouse becomes binding when many workers produce high insert rates simultaneously.

> **This formula does NOT apply to kintsugi.** See [Kintsugi Parallelism](#kintsugi-parallelism-parquet-cached-mode) — kintsugi with Parquet cache is ClickHouse-bound, not network-bound, and the empirical sweet spot is 24 (not 6).

**Runnable model**: `python .claude/skills/perf-analysis/models/capacity_model.py` -- accepts measured values as arguments.

---

## Live Capacity Probes

Run these on bigblack before and during backfill to feed the capacity model with real values.

### Probe 1: ClickHouse Active Parts

```bash
ssh bigblack "clickhouse-client --query \"$(cat .claude/skills/perf-analysis/queries/active_parts.sql)\""
```

Healthy: <300. Yellow: >500. Red: >800.

### Probe 2: Active Merges

```bash
ssh bigblack "clickhouse-client --query \"$(cat .claude/skills/perf-analysis/queries/merge_queue.sql)\""
```

Healthy: <3 pending. Yellow: >5. Red: >10.

### Probe 3: Insert Latency

```bash
ssh bigblack "clickhouse-client --query \"$(cat .claude/skills/perf-analysis/queries/insert_latency.sql)\""
```

Healthy: <10ms avg. Yellow: >10ms. Red: >50ms.

### Probe 4: Memory Pressure

```bash
ssh bigblack "clickhouse-client --query \"$(cat .claude/skills/perf-analysis/queries/memory_pressure.sql)\""
ssh bigblack "free -h | head -2"
```

### Probe 5: Network Throughput

```bash
ssh bigblack "sar -n DEV 1 5 2>/dev/null | grep -E 'eth0|eno1' | tail -5"
```

Or estimate from Vision download times -- if per-ZIP time exceeds 3s (vs baseline 1.3-1.5s), network is saturated.

---

## Escalation Thresholds

| Metric         | Green   | Yellow     | Red         | Action on Red                   |
| -------------- | ------- | ---------- | ----------- | ------------------------------- |
| Active parts   | <300    | >500       | >800        | Reduce workers by 50%           |
| Insert latency | <10ms   | >10ms avg  | >50ms avg   | Reduce workers, check merges    |
| Merge queue    | <3      | >5 pending | >10 pending | Pause new inserts until cleared |
| Available RAM  | >20 GB  | <10 GB     | <5 GB       | Kill lowest-priority workers    |
| Disk util      | <40%    | >60%       | >80%        | Investigate, check TTL merges   |
| Vision DL time | <2s/ZIP | >3s/ZIP    | >5s/ZIP     | Reduce workers (network sat.)   |

**Response protocol**:

- **Green**: Safe to increase parallelism by 1-2 workers
- **Yellow**: Hold current parallelism, do not increase
- **Red**: Immediately reduce workers; wait 5 minutes; re-probe before resuming

---

## Backfill ETA Model

Empirically measured processing rates on bigblack (single worker, including Vision download + Rust processing + ClickHouse insert).

### Per-Symbol Processing Rates

| Symbol   | Rate (s/day) @250 dbps | Rate (s/day) @1000 dbps | Volume Class |
| -------- | ---------------------- | ----------------------- | ------------ |
| BTCUSDT  | 2.36                   | 0.8                     | Heavy        |
| ETHUSDT  | 1.9                    | 0.7                     | Heavy        |
| BNBUSDT  | 1.5                    | 0.5                     | Medium       |
| SOLUSDT  | 1.3                    | 0.4                     | Medium       |
| XRPUSDT  | 1.2                    | 0.4                     | Medium       |
| DOGEUSDT | 1.8                    | 0.6                     | Medium       |
| Others   | 1.0                    | 0.3                     | Light        |

**Blended rate** (weighted by volume across all 15 batch symbols): ~1.8 s/day @250 dbps.

### ETA Formula

```
total_symbol_days = sum(days_per_symbol for each symbol)
eta_seconds = total_symbol_days * blended_rate_per_day / num_workers
eta_hours = eta_seconds / 3600
```

**Example**: 15 symbols, 7 years each (2018-2025), 4 workers:

```
total_symbol_days = 15 * 365 * 7 = 38,325 symbol-days
eta_seconds = 38,325 * 1.8 / 4 = 17,246 seconds
eta_hours = 17,246 / 3600 = 4.8 hours
```

Add 20% contingency for Vision download variability: **~5.7 hours**.

### Calibration

After each major backfill, record actual vs estimated time to refine the per-symbol rates. Update this section with new measurements.

| Date    | Workload      | Estimated | Actual | Delta | Notes                |
| ------- | ------------- | --------- | ------ | ----- | -------------------- |
| 2026-03 | Initial model | --        | --     | --    | Baseline from probes |

---

## Anti-Patterns

### AP-1: Ignoring Parts Pressure

Parts accumulate faster than ClickHouse can merge them. At `parts_to_delay_insert` (1000), inserts slow down artificially. At `parts_to_throw_insert` (3000), inserts fail. **Fix**: Monitor parts count, reduce workers before reaching 500.

### AP-2: Flat Worker Count Across Thresholds

250 dbps workers use 5 GB RAM vs 200 MB for 1000 dbps. A flat "8 workers" safe at 1000 dbps causes OOM at 250 dbps. **Fix**: Always use the composite formula with threshold-appropriate `per_worker_gb`.

### AP-3: Not Accounting for Resident Services

Sidecar (326 MB), kintsugi (1.6 GB), and backfill daemon (591 MB) run continuously. Forgetting them overestimates available RAM by ~2.5 GB. **Fix**: Always subtract service footprint from available RAM.

### AP-4: Assuming Network is Unlimited

Binance Vision has no rate limits, but bigblack's 1 Gbps link saturates at ~6-8 concurrent ZIP downloads. Beyond that, per-worker download time degrades, giving diminishing returns. **Fix**: Cap workers at `floor(125 / 15 * 0.8) = 6` for network-bound workloads.

### AP-5: Scaling Workers Without Monitoring

Adding workers without checking merge queue depth and insert latency. By the time parts_to_delay_insert triggers, the backlog takes 10+ minutes to clear. **Fix**: Monitor for 15 minutes after each parallelism increase before adding more.

---

## Quick Decision Table

### Vision-Download Populate Workers (pueue / `populate_cache_resumable`)

For common scenarios on bigblack (32 cores, 61 GB, 1 Gbps):

| Scenario                        | Safe Workers | Binding Constraint | Notes                         |
| ------------------------------- | ------------ | ------------------ | ----------------------------- |
| Light threshold (1000/750/500)  | 6            | Network            | Memory ~200 MB/worker         |
| Heavy threshold (250, BTC/ETH)  | 4-5          | Network + Memory   | Memory ~5 GB/worker           |
| Heavy threshold (250, altcoins) | 6            | Network            | Memory ~3 GB/worker           |
| Very heavy threshold (100)      | 3            | Memory             | Memory ~8 GB/worker           |
| Mixed thresholds simultaneously | 4            | Combined load      | Conservative, monitor closely |
| Sidecar + kintsugi + backfill   | 4-5          | Memory (services)  | ~2.5 GB service overhead      |

### Kintsugi Daemon (`MAX_CONCURRENT_REPAIRS`)

When BTCUSDT Parquet cache is populated, kintsugi is ClickHouse-bound, not network-bound:

| Scenario                             | MAX_CONCURRENT_REPAIRS | Binding Constraint     | Notes                                                |
| ------------------------------------ | ---------------------- | ---------------------- | ---------------------------------------------------- |
| Kintsugi (Parquet-cached, bigblack)  | **24**                 | CH query scheduler     | Empirical hard limit. Do NOT use 32 (cliff)          |
| Kintsugi (Vision-download, no cache) | 2                      | Network (USB ethernet) | `OPENDEVIATIONBAR_SEEDER_MAX_CONCURRENT_DOWNLOADS=2` |
| Kintsugi focused on 1 symbol         | **24**                 | CH query scheduler     | Same limit; single symbol just reduces queue         |

---

## References

- `queries/active_parts.sql` -- ClickHouse active parts count
- `queries/merge_queue.sql` -- Active merge operations
- `queries/insert_latency.sql` -- Insert performance from query_log
- `queries/memory_pressure.sql` -- ClickHouse memory metrics
- `queries/parts_trend.sql` -- Parts count over time for trend analysis
- `models/capacity_model.py` -- Runnable composite safety formula
- `Skill(remote-backfill)` -- Command formulation, pueue groups
- `Skill(opendeviationbar-job-safety)` -- Memory profiles, cgroup limits
- `Skill(autonomous-agent-ops)` -- Multi-agent monitoring guardrails
- [/docs/development/PERFORMANCE.md](/docs/development/PERFORMANCE.md) -- Benchmarks, metrics

---

## Post-Execution Reflection

After this skill completes, reflect before closing the task:

0. **Locate yourself.** — Find this SKILL.md's canonical path (Glob for this skill's name) before editing. All corrections target THIS file and its sibling references/ — never other documentation.
1. **What failed?** — Fix the instruction that caused it. If it could recur, add it as an anti-pattern.
2. **What worked better than expected?** — Promote it to recommended practice. Document why.
3. **What drifted?** — Any script, reference, or external dependency that no longer matches reality gets fixed now.
4. **Log it.** — Every change gets an evolution-log entry with trigger, fix, and evidence.

Do NOT defer. The next invocation inherits whatever you leave behind.
---

## Post-Change Checklist

After modifying this skill:

- [ ] Escalation thresholds match current bigblack configuration
- [ ] Memory profiles consistent with `Skill(opendeviationbar-job-safety)`
- [ ] Quick Decision Table updated if hardware changed
- [ ] Calibration table updated with new backfill measurements
- [ ] All queries in `queries/` are syntactically valid
- [ ] `models/capacity_model.py` produces correct output

---

## Revision Log

| Date       | Rev | Change                                       | Author |
| ---------- | --- | -------------------------------------------- | ------ |
| 2026-03-05 | 1.0 | Initial skill from capacity analysis session | Claude |
