The HarnessTax study measured how much the agent harness changes what a fixed model can do, by running seven models through Claude Code, Codex CLI and Pi on the same SWE-bench Lite and Terminal-Bench 2.0 tasks. We ran the same tasks, models, limits and graders through a fourth harness, KISS Sorcar, a general-purpose agent framework configured with four shell-and-file tools and a system prompt of about 200 words. Averaged over the seven models, KISS Sorcar solves 71.4% of SWE-bench Lite attempts (Pi: 71.9%, Codex: 73.7%, Claude Code: 73.4%) and 67.9% of Terminal-Bench 2.0 attempts (Pi: 70.0%, Codex: 65.7%, Claude Code: 65.1%), at a cost per attempt about the same as Pi's and half of Claude Code's on SWE-bench Lite. Every per-model success rate falls inside the blog's confidence interval for the same model on Pi.
The HarnessTax study asks how much the coding-agent harness matters once the model is fixed. It runs seven models through three harnesses (Claude Code, Codex CLI and Pi) on 30 randomly sampled SWE-bench Lite instances and 30 Terminal-Bench 2.0 tasks, three attempts each, and reports success rate and cost per attempt. This page adds a fourth harness to that grid, KISS Sorcar, and keeps everything else the same.
KISS Sorcar is a general-purpose agent framework, not a coding tool. For this comparison it
ran as a Sorcar Extension Agent (SEA): a small Python file that gives the framework's agent loop a
task prompt, a model and a set of tools. The SEA's tools are the four a minimal coding harness
needs, bash, read, write and edit, all executed
inside the benchmark's own container over docker exec, plus the framework's built-in
finish. The system prompt is short (about 200 words) and generic: work inside the
container, explore before you edit, verify with the repository's own tests, stop when done. Nothing
in it mentions either benchmark, and no benchmark-specific hints, retrieval, or test-oracle access
are used.
The protocol follows the blog point by point:
task.toml.git diff of the
container after the run, graded with the official swebench evaluation harness in the
official per-instance images with network access disabled. Terminal-Bench 2.0 tasks run under
Harbor 0.20.0 against the terminal-bench@2.0 registry with the official verifiers.The Claude Code, Codex and Pi numbers below are the blog's published values, not re-runs. The
KISS Sorcar numbers are new measurements on the same tasks. Everything needed to rerun them is in
benchmarkings/harnesstax/ of the KISS repository (see the last section).
SWE-bench Lite gives the agent a checked-out repository and a GitHub issue; the attempt counts as solved when the maintainers' hidden tests pass on the resulting patch. Across the seven models, KISS Sorcar solves 71.4% of attempts on average, against 71.9% for Pi, 73.4% for Claude Code and 73.7% for Codex. Cost is where the harnesses separate: taking the geometric mean over models, a KISS Sorcar attempt costs about the same as a Pi attempt, 23% less than a Codex attempt and half of a Claude Code attempt.
| Model · harness | Success | 95% CI | $ / attempt | 95% CI | $ / solve | Turns (mean) |
|---|---|---|---|---|---|---|
| Fable 5 · KISS Sorcar | 93.3% | 86.7%–98.9% | $0.559 lowest | $0.412–$0.731 | $0.599 | 13.8 |
| Fable 5 · Pi | 96.7% | 91.1%–100.0% | $0.666 | $0.494–$0.875 | $0.689 | — |
| Fable 5 · Claude Code | 97.8% best | 93.3%–100.0% | $1.329 | $1.101–$1.601 | $1.360 | — |
| Fable 5 · Codex | 96.7% | 91.1%–100.0% | $0.890 | $0.697–$1.127 | $0.921 | — |
| Opus 4.8 · KISS Sorcar | 81.1% | 67.8%–92.2% | $0.378 lowest | $0.214–$0.569 | $0.466 | 16.5 |
| Opus 4.8 · Pi | 82.2% | 71.1%–92.2% | $0.473 | $0.272–$0.725 | $0.575 | — |
| Opus 4.8 · Codex | 88.9% best | 77.8%–97.8% | $0.694 | $0.491–$0.946 | $0.780 | — |
| Opus 4.8 · Claude Code | 86.7% | 75.6%–95.6% | $0.976 | $0.733–$1.263 | $1.126 | — |
| Sonnet 4.6 · KISS Sorcar | 64.4% | 47.8%–80.0% | $0.519 lowest | $0.338–$0.721 | $0.805 | 29.6 |
| Sonnet 4.6 · Claude Code | 66.7% | 51.1%–81.1% | $0.669 | $0.499–$0.863 | $1.004 | — |
| Sonnet 4.6 · Pi | 64.4% | 47.8%–80.0% | $0.679 | $0.461–$0.926 | $1.054 | — |
| Sonnet 4.6 · Codex | 68.9% best | 53.3%–83.3% | $0.745 | $0.485–$1.100 | $1.081 | — |
| Haiku 4.5 · KISS Sorcar | 60.0% best | 43.3%–76.7% | $0.471 | $0.409–$0.535 | $0.785 | 73.1 |
| Haiku 4.5 · Pi | 60.0% best | 43.3%–75.6% | $0.374 lowest | $0.308–$0.441 | $0.623 | — |
| Haiku 4.5 · Codex | 57.8% | 41.1%–74.4% | $0.392 | $0.296–$0.510 | $0.678 | — |
| Haiku 4.5 · Claude Code | 52.2% | 36.7%–67.8% | $0.426 | $0.346–$0.507 | $0.816 | — |
| GPT-5.6 Sol · KISS Sorcar | 76.7% | 63.3%–88.9% | $0.615 | $0.476–$0.790 | $0.803 | 25.2 |
| GPT-5.6 Sol · Pi | 74.4% | 58.9%–88.9% | $0.441 lowest | $0.342–$0.553 | $0.592 | — |
| GPT-5.6 Sol · Codex | 73.3% | 57.8%–87.8% | $0.561 | $0.442–$0.702 | $0.765 | — |
| GPT-5.6 Sol · Claude Code | 77.8% best | 63.3%–90.0% | $1.540 | $1.184–$2.095 | $1.980 | — |
| GPT-5.6 Luna · KISS Sorcar | 53.3% | 36.7%–70.0% | $0.036 | $0.027–$0.047 | $0.068 | 28.6 |
| GPT-5.6 Luna · Pi | 53.3% | 35.6%–70.0% | $0.030 lowest | $0.024–$0.037 | $0.056 | — |
| GPT-5.6 Luna · Codex | 55.6% best | 37.8%–72.2% | $0.035 | $0.028–$0.044 | $0.064 | — |
| GPT-5.6 Luna · Claude Code | 55.6% best | 38.9%–72.2% | $0.152 | $0.120–$0.190 | $0.274 | — |
| Kimi K3 · KISS Sorcar | 71.1% | 56.7%–84.4% | $0.425 lowest | $0.300–$0.565 | $0.597 | 34.1 |
| Kimi K3 · Pi | 72.2% | 57.8%–85.6% | $0.455 | $0.323–$0.607 | $0.630 | — |
| Kimi K3 · Claude Code | 76.7% best | 62.2%–88.9% | $0.784 | $0.567–$1.045 | $1.023 | — |
| Kimi K3 · Codex | 74.4% | 60.0%–87.8% | $0.845 | $0.665–$1.043 | $1.135 | — |
Two model-level patterns stand out. With the two strongest models KISS Sorcar sits on the cost–success frontier: Fable 5 solves 93.3% at $0.56 per attempt (Pi: 96.7% at $0.67; Claude Code: 97.8% at $1.33) and Opus 4.8 solves 81.1% at $0.38 (Pi: 82.2% at $0.47; Codex: 88.9% at $0.69). These models finish in 14 to 16 model calls on average, close to the 15.4 turns the blog reports for Fable 5 on Pi, and about 85% of the tokens they consume are prompt-cache reads billed at a tenth of the input price. With Claude Haiku 4.5 and GPT-5.6 Sol the success rates match Pi (60.0% vs 60.0%, and 76.7% vs 74.4%) but KISS Sorcar pays more per attempt ($0.47 vs $0.37 for Haiku, $0.62 vs $0.44 for Sol). Haiku's trajectories explain its gap: it averages 73 model calls per attempt, writing and running many small diagnostic scripts before and after each edit, where the frontier models stop after a dozen. The harness does not stop it, so the extra turns become extra cost rather than extra solves.
Of the 180 unsolved KISS Sorcar attempts on this benchmark, 175 ended with the agent reporting success after its own checks and the hidden tests disagreeing, and 5 (all with Haiku) ran into the 100-turn cap. No attempt gave up or failed for infrastructure reasons. In the task-level grid below, 156 model–task pairs were solved at least once by both harnesses, 10 only by KISS Sorcar and 6 only by Pi; the remaining 38 pairs defeated both.
Terminal-Bench 2.0 tasks are open-ended terminal jobs: compile a compiler, train a model under a size limit, recover a corrupted git repository, reverse-engineer a rendered image. Each task has its own wall-clock limit and a verifier script. Averaged over the seven models, KISS Sorcar solves 67.9% of attempts, against 70.0% for Pi, 65.7% for Codex and 65.1% for Claude Code, at a cost per attempt about the same as Pi's, 9% less than Codex's and 32% less than Claude Code's.
| Model · harness | Success | 95% CI | $ / attempt | 95% CI | $ / solve | Turns (mean) |
|---|---|---|---|---|---|---|
| Fable 5 · KISS Sorcar | 71.1% | 55.6%–85.6% | $1.120 | $0.679–$1.686 | $1.576 | 13.5 |
| Fable 5 · Codex | 72.2% | 56.7%–86.7% | $0.976 lowest | $0.632–$1.406 | $1.351 | — |
| Fable 5 · Pi | 71.1% | 54.4%–85.6% | $1.079 | $0.571–$1.749 | $1.517 | — |
| Fable 5 · Claude Code | 75.6% best | 61.1%–88.9% | $1.554 | $0.993–$2.235 | $2.057 | — |
| Opus 4.8 · KISS Sorcar | 63.3% | 46.7%–78.9% | $0.866 | $0.479–$1.346 | $1.367 | 19.5 |
| Opus 4.8 · Pi | 72.2% best | 58.9%–84.4% | $0.758 lowest | $0.384–$1.245 | $1.049 | — |
| Opus 4.8 · Codex | 72.2% best | 56.7%–86.7% | $0.848 | $0.508–$1.269 | $1.174 | — |
| Opus 4.8 · Claude Code | 68.9% | 52.2%–84.4% | $0.899 | $0.590–$1.277 | $1.305 | — |
| Sonnet 4.6 · KISS Sorcar | 68.9% best | 54.4%–82.2% | $0.593 | $0.385–$0.821 | $0.861 | 23.6 |
| Sonnet 4.6 · Codex | 63.3% | 46.7%–78.9% | $0.552 lowest | $0.380–$0.741 | $0.872 | — |
| Sonnet 4.6 · Pi | 65.6% | 48.9%–82.2% | $0.614 | $0.391–$0.869 | $0.937 | — |
| Sonnet 4.6 · Claude Code | 62.2% | 46.7%–76.7% | $0.669 | $0.447–$0.917 | $1.075 | — |
| Haiku 4.5 · KISS Sorcar | 45.6% | 30.0%–61.1% | $0.297 | $0.215–$0.384 | $0.653 | 40.9 |
| Haiku 4.5 · Codex | 31.1% | 15.6%–47.8% | $0.214 lowest | $0.143–$0.293 | $0.687 | — |
| Haiku 4.5 · Pi | 47.8% best | 32.2%–63.3% | $0.250 | $0.160–$0.355 | $0.522 | — |
| Haiku 4.5 · Claude Code | 41.1% | 27.8%–55.6% | $0.263 | $0.162–$0.377 | $0.639 | — |
| GPT-5.6 Sol · KISS Sorcar | 78.9% | 65.6%–91.1% | $0.553 | $0.362–$0.793 | $0.700 | 18.5 |
| GPT-5.6 Sol · Pi | 83.3% best | 71.1%–94.4% | $0.421 lowest | $0.272–$0.612 | $0.505 | — |
| GPT-5.6 Sol · Codex | 78.9% | 65.6%–91.1% | $0.761 | $0.463–$1.131 | $0.965 | — |
| GPT-5.6 Sol · Claude Code | 71.1% | 55.6%–85.6% | $1.355 | $0.948–$1.853 | $1.905 | — |
| GPT-5.6 Luna · KISS Sorcar | 73.3% | 58.9%–86.7% | $0.045 | $0.026–$0.068 | $0.061 | 24.6 |
| GPT-5.6 Luna · Pi | 76.7% best | 62.2%–90.0% | $0.045 lowest | $0.026–$0.069 | $0.059 | — |
| GPT-5.6 Luna · Codex | 72.2% | 57.8%–85.6% | $0.064 | $0.039–$0.093 | $0.089 | — |
| GPT-5.6 Luna · Claude Code | 70.0% | 55.6%–83.3% | $0.098 | $0.068–$0.133 | $0.141 | — |
| Kimi K3 · KISS Sorcar | 74.4% best | 60.0%–86.7% | $0.259 lowest | $0.155–$0.383 | $0.348 | 18.3 |
| Kimi K3 · Pi | 73.3% | 60.0%–85.6% | $0.383 | $0.239–$0.543 | $0.523 | — |
| Kimi K3 · Codex | 70.0% | 55.6%–83.3% | $0.450 | $0.283–$0.657 | $0.643 | — |
| Kimi K3 · Claude Code | 66.7% | 52.2%–80.0% | $0.521 | $0.356–$0.707 | $0.782 | — |
Here the comparison splits by model rather than by harness. With Kimi K3, KISS Sorcar matches Pi's success rate (74.4% vs 73.3%) at $0.26 per attempt against Pi's $0.38, the largest cost gap in KISS Sorcar's favour in the study (Pi pays 1.48× as much). With Claude Sonnet 4.6 it posts the highest success rate of the four harnesses (68.9%; Pi 65.6%, Codex 63.3%, Claude Code 62.2%), and with Claude Haiku 4.5 it lands between Pi and Codex (45.6% vs 47.8% and 31.1%). GPT-5.6 Luna (73.3% vs 76.7%) and Claude Fable 5 (71.1% vs 71.1%) track Pi within a few points, at about Pi's cost.
The two models that fall short of Pi are Claude Opus 4.8 (63.3% vs
72.2%) and GPT-5.6 Sol (78.9% vs 83.3%, at
$0.55 vs $0.42 per attempt). Both gaps sit inside the confidence
intervals, but they have an identifiable cause. Opus's misses relative to Pi concentrate on
gcode-to-text and path-tracing, which Pi's Opus solves 3/3 and KISS
Sorcar's Opus solves 0/3 and
1/3, and every one of those misses
ended at the task's wall-clock limit (15 and 30 minutes respectively). The trajectories show two
different ways of losing to the clock. On gcode-to-text Opus made 38 to 56 model
calls in its 15 minutes but spent only about a minute of that inside tool calls; the rest was the
model's own reasoning between calls. On path-tracing it ran brute-force numerical
searches that hit the harness's 180-second default command timeout two to eight times per attempt.
Time limits matter on this benchmark generally: 66 of the 202 failed
KISS Sorcar attempts (33%) ended at the task's wall-clock limit,
55 of them on five CPU-bound tasks (train-fasttext 19, make-doom-for-mips 12, compile-compcert 10, gcode-to-text 8, path-tracing 6). Those tasks
ran on a shared 32-core host with up to 26 benchmark containers active at once; the blog does not
describe its hardware, so part of this gap may be machine rather than harness.
In the task-level grid, 129 model–task pairs were solved at least once by both harnesses, 7 only by KISS Sorcar and 8 only by Pi, with 36 pairs solved by neither. The blog publishes per-task Pi results for six of the seven models on this benchmark, so Kimi K3 is not in these counts.
The HarnessTax blog's conclusion was that the harness matters, but less than the model, and that a compact harness (Pi) can keep up with the tool vendors' own products. Adding KISS Sorcar confirms both points from outside the coding-tool world: a general-purpose agent framework with four container tools and a short prompt reaches the blog's Pi numbers on both benchmarks (71.4% vs 71.9% on SWE-bench Lite, 67.9% vs 70.0% on Terminal-Bench 2.0, averaged over seven models) and does so at half of Claude Code's cost per attempt on SWE-bench Lite and 32% less than it on Terminal-Bench 2.0. Where KISS Sorcar trails, the trajectories point at model habits the harness leaves alone (Haiku's long diagnostic loops, Opus's slow reasoning against short task clocks) rather than at missing capabilities.
Some caveats apply to all of the numbers above. Thirty tasks and three attempts give wide confidence intervals, roughly ±14 points, so differences of a few points between harnesses are not distinguishable from noise; that is true of the blog's own harness comparisons as well. The Claude Code, Codex and Pi values are the blog's published figures, not re-runs, and were priced with a price list we could not inspect. Kimi K3 was served by OpenRouter rather than Fireworks, and one of its 90 Terminal-Bench attempts (a re-run) was billed at OpenRouter's later list price of $1.70 / $8.50 per million tokens instead of $1.95 / $10.92; at the earlier price that attempt would cost at most 29% more, which moves Kimi's mean cost per attempt by well under 1%. The Terminal-Bench tasks with tight wall-clock limits ran on a busy machine. 21 of the 1,260 attempts were re-run once because the run itself was invalid: eight because the official grader exceeded its own time limit, two cancelled by an operator error, two in which a provider fallback briefly routed calls to a different model, and nine in which the harness let the agent execute a few tool calls past the 100-turn cap. No attempt was re-run because of its result. An attempt that reached the 100-turn cap is graded on the container's state at the cap and billed through its 100th model call; any further tool-less calls it made to write a summary are not counted.
# one-time: private daemon for the benchmark runs (KISS_HOME=~/.kiss-harnesstax)
uv run python -m benchmarkings.harnesstax.daemon &
# SWE-bench Lite: run the 30 instances for every model, three attempts, then grade
M="claude-fable-5,claude-opus-4-8,claude-sonnet-4-6,claude-haiku-4-5,gpt-5.6-sol,gpt-5.6-luna,openrouter/moonshotai/kimi-k3"
uv run python -m benchmarkings.harnesstax.swebench_runner --phase baseline --models "$M" --reps 3 --concurrency 10
uv run python -m benchmarkings.harnesstax.swebench_runner --phase baseline --evaluate
# Terminal-Bench 2.0 through Harbor (official verifiers), three attempts
uv run python -m benchmarkings.harnesstax.tb2_runner --phase baseline --models "$M" --reps 3 --concurrency 4
# attempts that failed for infrastructure reasons: set aside and re-run
uv run python -m benchmarkings.harnesstax.audit --phase baseline --quarantine --rerun-plan
# tables, confidence intervals, and this page
uv run python -m benchmarkings.harnesstax.aggregate --phase baseline
uv run python -m benchmarkings.harnesstax.report --phase baseline --out reports/harnesstax-kiss-sorcar.html
Every attempt leaves a trajectory.jsonl with all model messages and tool calls, a
result.json with turns, tokens and cost, and, for SWE-bench Lite, the extracted
model.patch and the official grader's per-run report; these live under
benchmarkings/harnesstax/results/.
{
"swebench-lite": {
"means": {
"Pi": {
"models": 7,
"kiss_success": 0.7142857142857143,
"success": 0.7188571428571429,
"cost_ratio": 0.9985236467408232
},
"Codex": {
"models": 7,
"kiss_success": 0.7142857142857143,
"success": 0.7365714285714287,
"cost_ratio": 1.2963005441391684
},
"Claude Code": {
"models": 7,
"kiss_success": 0.7142857142857143,
"success": 0.7335714285714285,
"cost_ratio": 2.024584517013977
}
},
"agreement": {
"both": 156,
"kiss_only": 10,
"pi_only": 6,
"neither": 38
},
"frontier": [
"Fable 5 \u00b7 KISS Sorcar",
"Opus 4.8 \u00b7 KISS Sorcar",
"Fable 5 \u00b7 Pi",
"Fable 5 \u00b7 Claude Code",
"Opus 4.8 \u00b7 Pi",
"Haiku 4.5 \u00b7 Pi",
"GPT-5.6 Luna \u00b7 Pi",
"GPT-5.6 Luna \u00b7 Codex"
],
"inside_pi_ci": [
7,
7
],
"attempts": 630,
"failures": 180,
"outcomes": {
"solved": 450,
"claimed but wrong": 175,
"turn cap": 5
},
"turn_cap_models": [
"claude-haiku-4-5"
],
"other_errors": 0,
"timeouts_by_task": {},
"pi_per_task": {
"claude-fable-5": {
"astropy__astropy-7746": [
1,
3
],
"django__django-11019": [
3,
3
],
"django__django-11564": [
3,
3
],
"django__django-11620": [
3,
3
],
"django__django-11630": [
3,
3
],
"django__django-11815": [
3,
3
],
"django__django-12308": [
3,
3
],
"django__django-12497": [
3,
3
],
"django__django-12747": [
3,
3
],
"django__django-14608": [
3,
3
],
"django__django-15400": [
3,
3
],
"django__django-15819": [
3,
3
],
"matplotlib__matplotlib-24334": [
3,
3
],
"matplotlib__matplotlib-25433": [
3,
3
],
"psf__requests-1963": [
3,
3
],
"pylint-dev__pylint-5859": [
3,
3
],
"pytest-dev__pytest-5103": [
3,
3
],
"pytest-dev__pytest-5495": [
3,
3
],
"scikit-learn__scikit-learn-13241": [
3,
3
],
"scikit-learn__scikit-learn-14983": [
3,
3
],
"sphinx-doc__sphinx-10325": [
3,
3
],
"sphinx-doc__sphinx-8435": [
3,
3
],
"sympy__sympy-11400": [
2,
3
],
"sympy__sympy-14024": [
3,
3
],
"sympy__sympy-14817": [
3,
3
],
"sympy__sympy-15345": [
3,
3
],
"sympy__sympy-16792": [
3,
3
],
"sympy__sympy-20154": [
3,
3
],
"sympy__sympy-20590": [
3,
3
],
"sympy__sympy-23117": [
3,
3
]
},
"claude-opus-4-8": {
"astropy__astropy-7746": [
1,
3
],
"django__django-11019": [
0,
3
],
"django__django-11564": [
1,
3
],
"django__django-11620": [
3,
3
],
"django__django-11630": [
2,
3
],
"django__django-11815": [
2,
3
],
"django__django-12308": [
3,
3
],
"django__django-12497": [
3,
3
],
"django__django-12747": [
2,
3
],
"django__django-14608": [
3,
3
],
"django__django-15400": [
2,
3
],
"django__django-15819": [
3,
3
],
"matplotlib__matplotlib-24334": [
3,
3
],
"matplotlib__matplotlib-25433": [
3,
3
],
"psf__requests-1963": [
3,
3
],
"pylint-dev__pylint-5859": [
3,
3
],
"pytest-dev__pytest-5103": [
3,
3
],
"pytest-dev__pytest-5495": [
3,
3
],
"scikit-learn__scikit-learn-13241": [
3,
3
],
"scikit-learn__scikit-learn-14983": [
3,
3
],
"sphinx-doc__sphinx-10325": [
3,
3
],
"sphinx-doc__sphinx-8435": [
3,
3
],
"sympy__sympy-11400": [
0,
3
],
"sympy__sympy-14024": [
1,
3
],
"sympy__sympy-14817": [
3,
3
],
"sympy__sympy-15345": [
3,
3
],
"sympy__sympy-16792": [
3,
3
],
"sympy__sympy-20154": [
3,
3
],
"sympy__sympy-20590": [
3,
3
],
"sympy__sympy-23117": [
3,
3
]
},
"claude-sonnet-4-6": {
"astropy__astropy-7746": [
0,
3
],
"django__django-11019": [
0,
3
],
"django__django-11564": [
0,
3
],
"django__django-11620": [
3,
3
],
"django__django-11630": [
0,
3
],
"django__django-11815": [
3,
3
],
"django__django-12308": [
2,
3
],
"django__django-12497": [
3,
3
],
"django__django-12747": [
3,
3
],
"django__django-14608": [
3,
3
],
"django__django-15400": [
0,
3
],
"django__django-15819": [
0,
3
],
"matplotlib__matplotlib-24334": [
0,
3
],
"matplotlib__matplotlib-25433": [
2,
3
],
"psf__requests-1963": [
3,
3
],
"pylint-dev__pylint-5859": [
3,
3
],
"pytest-dev__pytest-5103": [
3,
3
],
"pytest-dev__pytest-5495": [
0,
3
],
"scikit-learn__scikit-learn-13241": [
3,
3
],
"scikit-learn__scikit-learn-14983": [
3,
3
],
"sphinx-doc__sphinx-10325": [
3,
3
],
"sphinx-doc__sphinx-8435": [
3,
3
],
"sympy__sympy-11400": [
0,
3
],
"sympy__sympy-14024": [
0,
3
],
"sympy__sympy-14817": [
3,
3
],
"sympy__sympy-15345": [
3,
3
],
"sympy__sympy-16792": [
3,
3
],
"sympy__sympy-20154": [
3,
3
],
"sympy__sympy-20590": [
3,
3
],
"sympy__sympy-23117": [
3,
3
]
},
"claude-haiku-4-5": {
"astropy__astropy-7746": [
0,
3
],
"django__django-11019": [
0,
3
],
"django__django-11564": [
0,
3
],
"django__django-11620": [
3,
3
],
"django__django-11630": [
0,
3
],
"django__django-11815": [
3,
3
],
"django__django-12308": [
0,
3
],
"django__django-12497": [
3,
3
],
"django__django-12747": [
1,
3
],
"django__django-14608": [
3,
3
],
"django__django-15400": [
2,
3
],
"django__django-15819": [
0,
3
],
"matplotlib__matplotlib-24334": [
0,
3
],
"matplotlib__matplotlib-25433": [
2,
3
],
"psf__requests-1963": [
3,
3
],
"pylint-dev__pylint-5859": [
3,
3
],
"pytest-dev__pytest-5103": [
3,
3
],
"pytest-dev__pytest-5495": [
0,
3
],
"scikit-learn__scikit-learn-13241": [
0,
3
],
"scikit-learn__scikit-learn-14983": [
3,
3
],
"sphinx-doc__sphinx-10325": [
3,
3
],
"sphinx-doc__sphinx-8435": [
3,
3
],
"sympy__sympy-11400": [
1,
3
],
"sympy__sympy-14024": [
0,
3
],
"sympy__sympy-14817": [
3,
3
],
"sympy__sympy-15345": [
3,
3
],
"sympy__sympy-16792": [
3,
3
],
"sympy__sympy-20154": [
3,
3
],
"sympy__sympy-20590": [
3,
3
],
"sympy__sympy-23117": [
3,
3
]
},
"gpt-5.6-sol": {
"astropy__astropy-7746": [
0,
3
],
"django__django-11019": [
0,
3
],
"django__django-11564": [
2,
3
],
"django__django-11620": [
3,
3
],
"django__django-11630": [
3,
3
],
"django__django-11815": [
3,
3
],
"django__django-12308": [
0,
3
],
"django__django-12497": [
3,
3
],
"django__django-12747": [
3,
3
],
"django__django-14608": [
3,
3
],
"django__django-15400": [
0,
3
],
"django__django-15819": [
0,
3
],
"matplotlib__matplotlib-24334": [
3,
3
],
"matplotlib__matplotlib-25433": [
3,
3
],
"psf__requests-1963": [
3,
3
],
"pylint-dev__pylint-5859": [
3,
3
],
"pytest-dev__pytest-5103": [
3,
3
],
"pytest-dev__pytest-5495": [
2,
3
],
"scikit-learn__scikit-learn-13241": [
0,
3
],
"scikit-learn__scikit-learn-14983": [
3,
3
],
"sphinx-doc__sphinx-10325": [
3,
3
],
"sphinx-doc__sphinx-8435": [
3,
3
],
"sympy__sympy-11400": [
0,
3
],
"sympy__sympy-14024": [
3,
3
],
"sympy__sympy-14817": [
3,
3
],
"sympy__sympy-15345": [
3,
3
],
"sympy__sympy-16792": [
3,
3
],
"sympy__sympy-20154": [
3,
3
],
"sympy__sympy-20590": [
3,
3
],
"sympy__sympy-23117": [
3,
3
]
},
"gpt-5.6-luna": {
"astropy__astropy-7746": [
0,
3
],
"django__django-11019": [
0,
3
],
"django__django-11564": [
0,
3
],
"django__django-11620": [
3,
3
],
"django__django-11630": [
0,
3
],
"django__django-11815": [
2,
3
],
"django__django-12308": [
0,
3
],
"django__django-12497": [
3,
3
],
"django__django-12747": [
0,
3
],
"django__django-14608": [
3,
3
],
"django__django-15400": [
0,
3
],
"django__django-15819": [
0,
3
],
"matplotlib__matplotlib-24334": [
0,
3
],
"matplotlib__matplotlib-25433": [
3,
3
],
"psf__requests-1963": [
3,
3
],
"pylint-dev__pylint-5859": [
3,
3
],
"pytest-dev__pytest-5103": [
3,
3
],
"pytest-dev__pytest-5495": [
0,
3
],
"scikit-learn__scikit-learn-13241": [
0,
3
],
"scikit-learn__scikit-learn-14983": [
0,
3
],
"sphinx-doc__sphinx-10325": [
3,
3
],
"sphinx-doc__sphinx-8435": [
3,
3
],
"sympy__sympy-11400": [
0,
3
],
"sympy__sympy-14024": [
2,
3
],
"sympy__sympy-14817": [
2,
3
],
"sympy__sympy-15345": [
3,
3
],
"sympy__sympy-16792": [
3,
3
],
"sympy__sympy-20154": [
3,
3
],
"sympy__sympy-20590": [
3,
3
],
"sympy__sympy-23117": [
3,
3
]
},
"openrouter/moonshotai/kimi-k3": {
"astropy__astropy-7746": [
0,
3
],
"django__django-11019": [
0,
3
],
"django__django-11564": [
1,
3
],
"django__django-11620": [
3,
3
],
"django__django-11630": [
0,
3
],
"django__django-11815": [
3,
3
],
"django__django-12308": [
3,
3
],
"django__django-12497": [
3,
3
],
"django__django-12747": [
3,
3
],
"django__django-14608": [
3,
3
],
"django__django-15400": [
2,
3
],
"django__django-15819": [
0,
3
],
"matplotlib__matplotlib-24334": [
2,
3
],
"matplotlib__matplotlib-25433": [
3,
3
],
"psf__requests-1963": [
2,
3
],
"pylint-dev__pylint-5859": [
3,
3
],
"pytest-dev__pytest-5103": [
3,
3
],
"pytest-dev__pytest-5495": [
0,
3
],
"scikit-learn__scikit-learn-13241": [
3,
3
],
"scikit-learn__scikit-learn-14983": [
3,
3
],
"sphinx-doc__sphinx-10325": [
3,
3
],
"sphinx-doc__sphinx-8435": [
3,
3
],
"sympy__sympy-11400": [
1,
3
],
"sympy__sympy-14024": [
3,
3
],
"sympy__sympy-14817": [
0,
3
],
"sympy__sympy-15345": [
3,
3
],
"sympy__sympy-16792": [
3,
3
],
"sympy__sympy-20154": [
3,
3
],
"sympy__sympy-20590": [
3,
3
],
"sympy__sympy-23117": [
3,
3
]
}
},
"rerun": 0,
"clock_failures": {}
},
"tb2": {
"means": {
"Pi": {
"models": 7,
"kiss_success": 0.6793650793650794,
"success": 0.7,
"cost_ratio": 0.9730838816192142
},
"Codex": {
"models": 7,
"kiss_success": 0.6793650793650794,
"success": 0.657,
"cost_ratio": 1.099560967733926
},
"Claude Code": {
"models": 7,
"kiss_success": 0.6793650793650794,
"success": 0.6508571428571429,
"cost_ratio": 1.4782110957533698
}
},
"agreement": {
"both": 129,
"kiss_only": 7,
"pi_only": 8,
"neither": 36
},
"frontier": [
"GPT-5.6 Sol \u00b7 Pi",
"GPT-5.6 Luna \u00b7 Pi"
],
"inside_pi_ci": [
7,
7
],
"attempts": 630,
"failures": 202,
"outcomes": {
"solved": 428,
"claimed but wrong": 125,
"infra/timeout": 66,
"turn cap": 7,
"gave up": 4
},
"turn_cap_models": [
"claude-haiku-4-5",
"gpt-5.6-luna"
],
"other_errors": 0,
"timeouts_by_task": {
"train-fasttext": 19,
"make-doom-for-mips": 12,
"compile-compcert": 10,
"gcode-to-text": 8,
"path-tracing": 6,
"path-tracing-reverse": 4,
"feal-linear-cryptanalysis": 3,
"adaptive-rejection-sampler": 3,
"code-from-image": 1
},
"pi_per_task": {
"claude-fable-5": {
"adaptive-rejection-sampler": [
0,
3
],
"build-cython-ext": [
3,
3
],
"build-pmars": [
2,
3
],
"code-from-image": [
3,
3
],
"compile-compcert": [
2,
3
],
"configure-git-webserver": [
0,
3
],
"constraints-scheduling": [
3,
3
],
"feal-linear-cryptanalysis": [
3,
3
],
"financial-document-processor": [
3,
3
],
"fix-code-vulnerability": [
3,
3
],
"fix-git": [
3,
3
],
"gcode-to-text": [
3,
3
],
"headless-terminal": [
3,
3
],
"make-doom-for-mips": [
0,
3
],
"merge-diff-arc-agi-task": [
3,
3
],
"multi-source-data-merger": [
3,
3
],
"nginx-request-logging": [
3,
3
],
"openssl-selfsigned-cert": [
3,
3
],
"path-tracing": [
3,
3
],
"path-tracing-reverse": [
3,
3
],
"pypi-server": [
3,
3
],
"query-optimize": [
0,
3
],
"regex-log": [
3,
3
],
"reshard-c4-data": [
3,
3
],
"sam-cell-seg": [
0,
3
],
"sparql-university": [
3,
3
],
"sqlite-db-truncate": [
3,
3
],
"torch-tensor-parallelism": [
0,
3
],
"train-fasttext": [
0,
3
],
"video-processing": [
0,
3
]
},
"claude-opus-4-8": {
"adaptive-rejection-sampler": [
0,
3
],
"build-cython-ext": [
2,
3
],
"build-pmars": [
2,
3
],
"code-from-image": [
3,
3
],
"compile-compcert": [
2,
3
],
"configure-git-webserver": [
1,
3
],
"constraints-scheduling": [
2,
3
],
"feal-linear-cryptanalysis": [
3,
3
],
"financial-document-processor": [
3,
3
],
"fix-code-vulnerability": [
3,
3
],
"fix-git": [
3,
3
],
"gcode-to-text": [
3,
3
],
"headless-terminal": [
2,
3
],
"make-doom-for-mips": [
0,
3
],
"merge-diff-arc-agi-task": [
3,
3
],
"multi-source-data-merger": [
3,
3
],
"nginx-request-logging": [
3,
3
],
"openssl-selfsigned-cert": [
3,
3
],
"path-tracing": [
3,
3
],
"path-tracing-reverse": [
3,
3
],
"pypi-server": [
3,
3
],
"query-optimize": [
2,
3
],
"regex-log": [
3,
3
],
"reshard-c4-data": [
3,
3
],
"sam-cell-seg": [
0,
3
],
"sparql-university": [
3,
3
],
"sqlite-db-truncate": [
3,
3
],
"torch-tensor-parallelism": [
1,
3
],
"train-fasttext": [
0,
3
],
"video-processing": [
0,
3
]
},
"claude-sonnet-4-6": {
"adaptive-rejection-sampler": [
0,
3
],
"build-cython-ext": [
3,
3
],
"build-pmars": [
3,
3
],
"code-from-image": [
3,
3
],
"compile-compcert": [
3,
3
],
"configure-git-webserver": [
3,
3
],
"constraints-scheduling": [
3,
3
],
"feal-linear-cryptanalysis": [
0,
3
],
"financial-document-processor": [
3,
3
],
"fix-code-vulnerability": [
3,
3
],
"fix-git": [
3,
3
],
"gcode-to-text": [
0,
3
],
"headless-terminal": [
3,
3
],
"make-doom-for-mips": [
0,
3
],
"merge-diff-arc-agi-task": [
3,
3
],
"multi-source-data-merger": [
3,
3
],
"nginx-request-logging": [
3,
3
],
"openssl-selfsigned-cert": [
3,
3
],
"path-tracing": [
0,
3
],
"path-tracing-reverse": [
0,
3
],
"pypi-server": [
3,
3
],
"query-optimize": [
0,
3
],
"regex-log": [
3,
3
],
"reshard-c4-data": [
2,
3
],
"sam-cell-seg": [
0,
3
],
"sparql-university": [
3,
3
],
"sqlite-db-truncate": [
3,
3
],
"torch-tensor-parallelism": [
3,
3
],
"train-fasttext": [
0,
3
],
"video-processing": [
0,
3
]
},
"claude-haiku-4-5": {
"adaptive-rejection-sampler": [
0,
3
],
"build-cython-ext": [
0,
3
],
"build-pmars": [
2,
3
],
"code-from-image": [
3,
3
],
"compile-compcert": [
3,
3
],
"configure-git-webserver": [
1,
3
],
"constraints-scheduling": [
3,
3
],
"feal-linear-cryptanalysis": [
0,
3
],
"financial-document-processor": [
0,
3
],
"fix-code-vulnerability": [
3,
3
],
"fix-git": [
3,
3
],
"gcode-to-text": [
0,
3
],
"headless-terminal": [
3,
3
],
"make-doom-for-mips": [
0,
3
],
"merge-diff-arc-agi-task": [
2,
3
],
"multi-source-data-merger": [
3,
3
],
"nginx-request-logging": [
3,
3
],
"openssl-selfsigned-cert": [
2,
3
],
"path-tracing": [
0,
3
],
"path-tracing-reverse": [
0,
3
],
"pypi-server": [
3,
3
],
"query-optimize": [
2,
3
],
"regex-log": [
2,
3
],
"reshard-c4-data": [
3,
3
],
"sam-cell-seg": [
0,
3
],
"sparql-university": [
2,
3
],
"sqlite-db-truncate": [
0,
3
],
"torch-tensor-parallelism": [
0,
3
],
"train-fasttext": [
0,
3
],
"video-processing": [
0,
3
]
},
"gpt-5.6-sol": {
"adaptive-rejection-sampler": [
3,
3
],
"build-cython-ext": [
3,
3
],
"build-pmars": [
3,
3
],
"code-from-image": [
3,
3
],
"compile-compcert": [
3,
3
],
"configure-git-webserver": [
0,
3
],
"constraints-scheduling": [
3,
3
],
"feal-linear-cryptanalysis": [
3,
3
],
"financial-document-processor": [
3,
3
],
"fix-code-vulnerability": [
3,
3
],
"fix-git": [
3,
3
],
"gcode-to-text": [
1,
3
],
"headless-terminal": [
3,
3
],
"make-doom-for-mips": [
1,
3
],
"merge-diff-arc-agi-task": [
3,
3
],
"multi-source-data-merger": [
3,
3
],
"nginx-request-logging": [
3,
3
],
"openssl-selfsigned-cert": [
3,
3
],
"path-tracing": [
3,
3
],
"path-tracing-reverse": [
3,
3
],
"pypi-server": [
3,
3
],
"query-optimize": [
3,
3
],
"regex-log": [
3,
3
],
"reshard-c4-data": [
3,
3
],
"sam-cell-seg": [
0,
3
],
"sparql-university": [
3,
3
],
"sqlite-db-truncate": [
3,
3
],
"torch-tensor-parallelism": [
1,
3
],
"train-fasttext": [
2,
3
],
"video-processing": [
1,
3
]
},
"gpt-5.6-luna": {
"adaptive-rejection-sampler": [
3,
3
],
"build-cython-ext": [
3,
3
],
"build-pmars": [
3,
3
],
"code-from-image": [
3,
3
],
"compile-compcert": [
3,
3
],
"configure-git-webserver": [
0,
3
],
"constraints-scheduling": [
3,
3
],
"feal-linear-cryptanalysis": [
2,
3
],
"financial-document-processor": [
2,
3
],
"fix-code-vulnerability": [
3,
3
],
"fix-git": [
3,
3
],
"gcode-to-text": [
0,
3
],
"headless-terminal": [
3,
3
],
"make-doom-for-mips": [
2,
3
],
"merge-diff-arc-agi-task": [
3,
3
],
"multi-source-data-merger": [
3,
3
],
"nginx-request-logging": [
3,
3
],
"openssl-selfsigned-cert": [
3,
3
],
"path-tracing": [
3,
3
],
"path-tracing-reverse": [
2,
3
],
"pypi-server": [
3,
3
],
"query-optimize": [
3,
3
],
"regex-log": [
3,
3
],
"reshard-c4-data": [
3,
3
],
"sam-cell-seg": [
0,
3
],
"sparql-university": [
3,
3
],
"sqlite-db-truncate": [
3,
3
],
"torch-tensor-parallelism": [
1,
3
],
"train-fasttext": [
0,
3
],
"video-processing": [
0,
3
]
}
},
"rerun": 21,
"clock_failures": {
"claude-fable-5 / gcode-to-text": [
1,
1
],
"claude-fable-5 / compile-compcert": [
2,
2
],
"claude-fable-5 / make-doom-for-mips": [
3,
3
],
"claude-fable-5 / adaptive-rejection-sampler": [
3,
0
],
"claude-fable-5 / train-fasttext": [
3,
3
],
"claude-opus-4-8 / gcode-to-text": [
3,
3
],
"claude-opus-4-8 / compile-compcert": [
3,
3
],
"claude-opus-4-8 / make-doom-for-mips": [
3,
3
],
"claude-opus-4-8 / path-tracing": [
2,
2
],
"claude-opus-4-8 / adaptive-rejection-sampler": [
3,
0
],
"claude-opus-4-8 / train-fasttext": [
3,
3
],
"claude-sonnet-4-6 / feal-linear-cryptanalysis": [
1,
1
],
"claude-sonnet-4-6 / gcode-to-text": [
3,
3
],
"claude-sonnet-4-6 / compile-compcert": [
1,
1
],
"claude-sonnet-4-6 / make-doom-for-mips": [
3,
3
],
"claude-sonnet-4-6 / path-tracing": [
3,
3
],
"claude-sonnet-4-6 / path-tracing-reverse": [
3,
3
],
"claude-sonnet-4-6 / adaptive-rejection-sampler": [
2,
0
],
"claude-sonnet-4-6 / train-fasttext": [
3,
3
],
"claude-haiku-4-5 / feal-linear-cryptanalysis": [
3,
2
],
"claude-haiku-4-5 / gcode-to-text": [
3,
0
],
"claude-haiku-4-5 / compile-compcert": [
1,
0
],
"claude-haiku-4-5 / make-doom-for-mips": [
3,
0
],
"claude-haiku-4-5 / path-tracing": [
3,
0
],
"claude-haiku-4-5 / path-tracing-reverse": [
3,
0
],
"claude-haiku-4-5 / adaptive-rejection-sampler": [
3,
2
],
"claude-haiku-4-5 / train-fasttext": [
3,
3
],
"gpt-5.6-sol / gcode-to-text": [
3,
1
],
"gpt-5.6-sol / compile-compcert": [
1,
1
],
"gpt-5.6-sol / make-doom-for-mips": [
3,
0
],
"gpt-5.6-sol / path-tracing": [
1,
0
],
"gpt-5.6-sol / train-fasttext": [
2,
2
],
"gpt-5.6-luna / gcode-to-text": [
3,
0
],
"gpt-5.6-luna / make-doom-for-mips": [
3,
0
],
"gpt-5.6-luna / path-tracing": [
1,
1
],
"gpt-5.6-luna / adaptive-rejection-sampler": [
1,
0
],
"gpt-5.6-luna / train-fasttext": [
3,
2
],
"openrouter/moonshotai/kimi-k3 / gcode-to-text": [
1,
0
],
"openrouter/moonshotai/kimi-k3 / compile-compcert": [
3,
3
],
"openrouter/moonshotai/kimi-k3 / make-doom-for-mips": [
3,
3
],
"openrouter/moonshotai/kimi-k3 / code-from-image": [
1,
1
],
"openrouter/moonshotai/kimi-k3 / path-tracing-reverse": [
1,
1
],
"openrouter/moonshotai/kimi-k3 / adaptive-rejection-sampler": [
2,
1
],
"openrouter/moonshotai/kimi-k3 / train-fasttext": [
3,
3
]
}
}
}