TokenJam proof — humaneval

tokenjam 0.5.1 n=20 tasks · k=1 sample(s) · deepseek:deepseek-reasoner → deepseek:deepseek-chat

Verdict: No significant regression · McNemar p=1.000 (α=0.05) · candidate chosen by explicit --candidate override

-75.6%

cost delta (measured)

+5.0pp

pass-rate delta [95% CI -4.5, +14.6]

0 / 1

tasks broken / fixed by the swap

Pass rate (95% CI whiskers)

Original19/20 (95%)

Candidate20/20 (100%)

Original$0.015005

Candidate$0.003660

task	orig	cand	candidate detail
HumanEval/0	1/1	1/1	ok
HumanEval/1	1/1	1/1	ok
HumanEval/2	1/1	1/1	ok
HumanEval/3	1/1	1/1	ok
HumanEval/4	1/1	1/1	ok
HumanEval/5	1/1	1/1	ok
HumanEval/6	1/1	1/1	ok
HumanEval/7	1/1	1/1	ok
HumanEval/8	1/1	1/1	ok
HumanEval/9	1/1	1/1	ok
HumanEval/10	0/1	1/1	ok
HumanEval/11	1/1	1/1	ok
HumanEval/12	1/1	1/1	ok
HumanEval/13	1/1	1/1	ok
HumanEval/14	1/1	1/1	ok
HumanEval/15	1/1	1/1	ok
HumanEval/16	1/1	1/1	ok
HumanEval/17	1/1	1/1	ok
HumanEval/18	1/1	1/1	ok
HumanEval/19	1/1	1/1	ok

Accuracy is the pass-rate on THIS benchmark suite, not a general “quality preserved” claim. Confidence is the CI + p-value, not a single “safe %”.

Generated 2026-06-25 10:19 · tokenjam-bench · executable-accuracy proof