---
name: benchmark-fairness
description: >-
  A/B performance benchmark comparison checklist for Rust (Criterion) and Python code changes.
  Ensures apples-to-apples measurement with build parity, coverage validation, and statistical rigor.
  TRIGGERS - benchmark, performance comparison, before after, A/B test, criterion baseline,
  cargo bench, speed regression, is it faster, measure performance.
allowed-tools: Read, Bash, Grep, Glob, Edit, Write
---

# Benchmark Fairness Checklist

Structured A/B performance comparison protocol for Rust (Criterion) and Python changes. Born from lessons learned during the memory efficiency audit (Issue #95) where 2 of 3 runtime optimizations were not benchmarked.

## When to Use This Skill

Use this skill when:

- Comparing performance before and after code changes
- Running `cargo bench` with `--save-baseline` / `--baseline`
- Validating that optimization work actually improved speed
- Reviewing someone else's benchmark results for fairness

## Pre-Flight Checklist

Before running any benchmarks, verify all items. Skip = document why.

### 1. Build Parity (Must Pass)

| Check                                 | Command                                                        | What to Verify                                       |
| ------------------------------------- | -------------------------------------------------------------- | ---------------------------------------------------- |
| Bench profile identical               | `grep -A5 '\[profile.bench\]' ~/.cargo/config.toml Cargo.toml` | Same opt-level, LTO, codegen-units for both runs     |
| RUSTFLAGS identical                   | `grep -A3 'rustflags' .cargo/config.toml`                      | Same target-cpu, opt-level, strip settings           |
| Features identical                    | `grep '\[features\]' -A20 Cargo.toml`                          | Same feature flags enabled for both runs             |
| Cargo.lock unchanged                  | `git diff Cargo.lock`                                          | Same dependency versions (no silent upgrades)        |
| Dev-deps don't affect library codegen | Inspect `[dev-dependencies]`                                   | dev-deps only affect bench binary, not library crate |

**Anti-pattern**: Adding `[dev-dependencies]` to fix bench compilation is safe (does not affect library codegen), but document it.

### 2. Coverage Mapping (Must Pass)

For each optimization being benchmarked, verify there is a benchmark that **exercises that specific code path**.

| Question                                 | How to Check                                                                    |
| ---------------------------------------- | ------------------------------------------------------------------------------- |
| Which code paths were changed?           | `git diff --stat` + read each modified function                                 |
| Does each changed path have a benchmark? | Map each fix ID to a benchmark group                                            |
| Are conditional paths covered?           | Check if both branches of if/else are benchmarked                               |
| Are feature-gated paths covered?         | Check if benchmarks enable required features (e.g., `.with_inter_bar_config()`) |

**Create a coverage table** (this is the most critical step):

```
| Fix ID | Code Path Changed              | Benchmark Group          | Exercised? |
|--------|--------------------------------|--------------------------|------------|
| R1     | processor.rs bar close single  | bar_close_take/single    | YES        |
| R2     | processor.rs bar close batch   | opendeviationbar_processing/*    | YES        |
| R3     | processor.rs include_incomplete| ???                      | NO - GAP   |
| R5     | interbar_math.rs volume moments| ???                      | NO - GAP   |
```

**Rule**: If a changed code path has no benchmark, either add one or document why measurement is not applicable (e.g., compile-time-only changes like `#[derive(Default)]`).

### 3. Workload Fairness (Must Pass)

| Check                               | What to Verify                                                                                                            |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Deterministic data generation       | Same seed, same RNG, no system clock dependency                                                                           |
| Sufficient trigger rate             | The optimization being measured is actually exercised (e.g., bar close rate high enough for clone elimination benchmarks) |
| Fresh state per iteration           | No state leakage between iterations (`b.iter()` creates new processor each time)                                          |
| `black_box()` on inputs AND outputs | Prevents dead-code elimination and constant folding                                                                       |
| Setup cost excluded or consistent   | Processor construction, data preparation outside `b.iter()` or consistent between runs                                    |

**Anti-pattern**: Using threshold=8000 (8% range) with volatility=100 produces very few bar closes. Use threshold=250 with volatility=500 to stress-test bar close paths.

### 4. Measurement Protocol (Must Pass)

| Check                                | What to Verify                                                    |
| ------------------------------------ | ----------------------------------------------------------------- |
| Warmup adequate                      | Criterion default 3s is sufficient for most benchmarks            |
| Sample count adequate                | Criterion default 100 samples is sufficient                       |
| Outlier rate normal                  | 3-11% outliers is normal; >20% suggests environmental instability |
| Confidence intervals don't include 0 | For claimed improvements, the CI must exclude zero change         |
| Sub-nanosecond results ignored       | Changes <1ns are measurement noise, not real regressions          |

### 5. Environmental Controls (Should Pass)

| Check                | What to Verify                                                                                   |
| -------------------- | ------------------------------------------------------------------------------------------------ |
| Same machine         | Both runs on same hardware                                                                       |
| Thermal state        | Cooldown between baseline and comparison runs, or verify results are consistent in reverse order |
| Background processes | Close unnecessary apps; check `htop` for CPU-hungry processes                                    |
| Power state          | Plugged in (avoid thermal throttling on battery)                                                 |

**Gold standard**: Run benchmarks in reverse order (after-then-before) and verify consistent results. If improvements disappear when run order reverses, the result is thermal artifact.

---

## Execution Protocol

### Step 1 - Establish Baseline

```bash
# Option A: git stash (uncommitted changes)
git stash push -m "optimization-changes"
# Apply ONLY non-functional fixes needed to compile benchmarks
# (e.g., [dev-dependencies], .unwrap() on Result-returning constructors)
cargo bench --bench opendeviationbar_bench -- --save-baseline before
git stash pop

# Option B: git checkout (committed changes on branch)
git checkout main
cargo bench --bench opendeviationbar_bench -- --save-baseline before
git checkout feature-branch
```

**Critical**: If the baseline needs non-functional fixes to compile (like adding `[dev-dependencies]`), apply them to BOTH runs. These fixes must not affect benchmarked code paths.

### Step 2 - Run Comparison

```bash
cargo bench --bench opendeviationbar_bench -- --baseline before
```

### Step 3 - Interpret Results

| Criterion Output                       | Interpretation                                     |
| -------------------------------------- | -------------------------------------------------- |
| "Performance has improved" (p < 0.05)  | Statistically significant improvement              |
| "Performance has regressed" (p < 0.05) | Statistically significant regression - investigate |
| "No change in performance detected"    | Change is within noise floor                       |
| Change < 5% on sub-10ns measurements   | Measurement noise - ignore regardless of p-value   |

### Step 4 - Report

Structure results as:

```
| Benchmark | Before | After | Change | Verdict |
|-----------|--------|-------|--------|---------|
| name      | X us   | Y us  | -Z%    | Improved / Noise / Regressed |
```

Include:

- Coverage table showing which optimizations are measured
- Note any gaps (unmeasured code paths)
- Note any confounds (thermal, background processes)

---

## Common Anti-Patterns

| Anti-Pattern                             | Why It's Wrong                                                 | Fix                                         |
| ---------------------------------------- | -------------------------------------------------------------- | ------------------------------------------- |
| Benchmark only the happy path            | Missing conditional branches (e.g., `include_incomplete=true`) | Add benchmarks for all branches             |
| Uniform volume in synthetic data         | Volume moments optimization (skew/kurt) not exercised          | Vary volume across trades                   |
| Feature-gated code not enabled           | Inter-bar features require `.with_inter_bar_config()`          | Enable features in benchmark setup          |
| Claiming improvement from untested paths | Fold optimization not benchmarked but claimed as contributor   | Only claim what's measured                  |
| Running baseline after optimization      | Thermal throttling biases against baseline                     | Run baseline FIRST or run both orders       |
| Ignoring sub-ns changes                  | 0.1ns "regression" on 2ns measurement is 5% noise              | Set noise threshold or document as noise    |
| Monolithic benchmark only                | Single 1M-tick benchmark hides per-operation improvements      | Test at multiple scales (1K, 10K, 100K, 1M) |

---

## Criterion Quick Reference

```bash
# Save baseline
cargo bench --bench opendeviationbar_bench -- --save-baseline before

# Compare against baseline
cargo bench --bench opendeviationbar_bench -- --baseline before

# Run specific benchmark group
cargo bench --bench opendeviationbar_bench -- "opendeviationbar_processing"

# Dry-run (compile check only)
cargo bench --bench opendeviationbar_bench -- --test

# Open HTML report
open target/criterion/report/index.html

# Set noise threshold (in bench code)
# group.noise_threshold(0.05)  // Suppress <5% changes
```

---

## Post-Change Checklist (Self-Maintenance)

After modifying THIS skill:

1. [ ] Anti-patterns list reflects latest lessons learned
2. [ ] Coverage mapping table template is up to date
3. [ ] Criterion quick reference matches current version
4. [ ] All referenced files in references/ exist
5. [ ] Append changes to [evolution-log.md](./references/evolution-log.md)

---

## Reference Documentation

For detailed information, see:

- [Criterion Baseline Workflow](./references/criterion-baseline-workflow.md) - Step-by-step baseline comparison
- [Evolution Log](./references/evolution-log.md) - Change history for this skill

---

## Troubleshooting

| Issue                                 | Cause                              | Fix                                                                    |
| ------------------------------------- | ---------------------------------- | ---------------------------------------------------------------------- |
| Benchmark won't compile               | Missing `[dev-dependencies]`       | Add `criterion` and library crate to dev-deps                          |
| `OpenDeviationBarProcessor::new()` error      | Constructor returns `Result` now   | Add `.unwrap()`                                                        |
| Baseline panic on new benchmark group | Group only exists in "after" code  | Expected - Criterion can't compare groups that don't exist in baseline |
| All benchmarks show same % change     | Likely thermal or environmental    | Add cooldown, run in reverse order                                     |
| Improvement disappears on re-run      | First run had cold cache advantage | Run 3+ times, take median                                              |
| `Gnuplot not found` warning           | Cosmetic only                      | Install gnuplot or ignore (plotters backend works)                     |
