rerun-bench report
Same task, run N times. Generated 2026-10-03T08:26:33+00:00 by rerun-bench 0.1.0. 2 result set(s), 10 task(s), k = 3.
Leaderboard
Click a column to sort. CV columns are the mean within-task coefficient of variation (std / mean across reruns of the same task).
| Agent / model | CLI version | Runs/task | Pass rate [95% CI] | pass@k | pass^k | Flip rate | Flaky tasks | Cost/run | Cost CV | Tokens CV | Wall median | Wall CV | Approach sim. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| claude / claude-opus-5-5 claude | 2.1.288 (Claude Code) | 3 | 100% [89, 100] | 100% | 100% | 0% | 0% | $0.0938 | 0.05 | 0.04 | 12.6s | 0.18 | 0.81 |
| codex / gpt-6-luna codex | codex-cli 0.160.0 | 3 | 93% [79, 98] | 100% | 80% | 13% | 20% | n/a | n/a | 0.14 | 16.1s | 0.21 | 0.77 |
Per-task consistency
Each square is one run, in run order: green passed, red failed. Hover a cell for CI, flip rate, cost and wall-time CV, and approach similarity.
| Task | claude / claude-opus-5-5 | codex / gpt-6-luna |
|---|---|---|
| add-cli-flag | 100%flip 0% | 100%flip 0% |
| edit-config | 100%flip 0% | 100%flip 0% |
| fix-failing-test | 100%flip 0% | 100%flip 0% |
| follow-agents-md | 100%flip 0% | 100%flip 0% |
| implement-lru-cache | 100%flip 0% | 67%flip 67% |
| implement-slugify | 100%flip 0% | 100%flip 0% |
| minimal-fix | 100%flip 0% | 100%flip 0% |
| multi-file-rename | 100%flip 0% | 100%flip 0% |
| refactor-extract-helper | 100%flip 0% | 67%flip 67% |
| write-tests | 100%flip 0% | 100%flip 0% |
Metric definitions
- Pass rate
- passes / runs, pooled over tasks, with a Wilson 95% interval.
- pass@k
- P(at least one of k reruns passes), unbiased estimator, averaged over tasks.
- pass^k
- P(all k reruns pass), unbiased estimator, averaged over tasks.
- Flip rate
- P(two reruns of the same task disagree) = 2c(n-c)/(n(n-1)), averaged over tasks.
- Flaky tasks
- Share of tasks with at least one pass and at least one fail.
- Cost / tokens / wall CV
- Sample std / mean across reruns of one task, averaged over tasks.
- Approach sim.
- Mean pairwise Jaccard similarity of changed-line sets among passing runs. 1.0 means every passing run made the same edit.