TokenJam proof — humaneval

tokenjam 0.5.2 n=50 tasks · k=1 sample(s) · openai:o3 → openai:o4-mini

Switching looks safe

We ran the same 50 humaneval task(s) through both models and graded every answer with the same automated judge. The cheaper candidate (o4-mini) cost 88% less than o3, but scored 4 points lower on this suite — 88% of tasks passed before, 84% after.

No statistically significant quality drop was detected, so moving to the cheaper model is a reasonable way to cut cost.

-88%

cheaper to run (measured API $)

-4 pts

accuracy vs the original model

4 worse · 2 better

tasks changed by the swap (of 50)

How each task did

Task	Original	Cheaper model	Why — the judge’s reason
0	pass	pass	ok
1	pass	pass	ok
2	pass	pass	ok
3	fail	fail	NameError: name 'List' is not defined. Did you mean: 'list'?
4	pass	pass	ok
5	pass	fail	NameError: name 'List' is not defined. Did you mean: 'list'?
6	fail	pass	ok
7	fail	pass	ok
8	pass	pass	ok
9	pass	pass	ok
10	pass	fail	NameError: name 'is_palindrome' is not defined. Did you mean: 'make_palindrome'?
11	pass	pass	ok
12	pass	fail	NameError: name 'List' is not defined. Did you mean: 'list'?
13	pass	pass	ok
14	pass	pass	ok
15	pass	pass	ok
16	pass	pass	ok
17	pass	pass	ok
18	pass	pass	ok
19	pass	pass	ok
20	fail	fail	NameError: name 'List' is not defined. Did you mean: 'list'?
21	pass	pass	ok
22	pass	pass	ok
23	pass	pass	ok
24	pass	pass	ok
25	pass	pass	ok
26	pass	pass	ok
27	pass	pass	ok
28	pass	fail	NameError: name 'List' is not defined. Did you mean: 'list'?
29	pass	pass	ok
30	pass	pass	ok
31	pass	pass	ok
32	fail	fail	NameError: name 'math' is not defined
33	pass	pass	ok
34	pass	pass	ok
35	pass	pass	ok
36	pass	pass	ok
37	pass	pass	ok
38	fail	fail	NameError: name 'encode_cyclic' is not defined. Did you mean: 'decode_cyclic'?
39	pass	pass	ok
40	pass	pass	ok
41	pass	pass	ok
42	pass	pass	ok
43	pass	pass	ok
44	pass	pass	ok
45	pass	pass	ok
46	pass	pass	ok
47	pass	pass	ok
48	pass	pass	ok
49	pass	pass	ok

The statistics behind it

Verdict: No significant regression · McNemar p=0.688 (α=0.05) · candidate chosen by explicit --candidate override

McNemar’s test asks whether the difference between the two models is bigger than chance: a p-value above 0.05 means the change is not statistically significant. The 95% CI on the pass-rate delta is the range the true difference is likely in — if it crosses zero, the direction isn’t certain.

-4.0pp

pass-rate delta [95% CI -13.5, +5.5]

Pass rate (95% CI whiskers)

Original44/50 (88%)

Candidate42/50 (84%)

Cost (measured)

Original$1.192450

Candidate$0.148118

How to read this

Accuracy is the pass-rate on THIS benchmark suite, not a general “quality preserved” claim. Confidence is the CI + p-value, not a single “safe %”.

Generated 2026-06-26 13:22 · tokenjam-bench · proof report