tokenjam 0.5.2 n=50 tasks · k=1 sample(s) · anthropic:claude-opus-4-7 → anthropic:claude-haiku-4-5
We ran the same 50 humaneval task(s) through both models and graded every answer with the same automated judge. The cheaper candidate (claude-haiku-4-5) cost 82% less than claude-opus-4-7, but scored 34 points lower on this suite — 90% of tasks passed before, 56% after.
The cheaper model was measurably worse on this suite — keep the original.
| Task | Original | Cheaper model | Why — the judge’s reason |
|---|---|---|---|
| 0 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 1 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 2 | pass | pass | ok |
| 3 | fail | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 4 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 5 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 6 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 7 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 8 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 9 | fail | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 10 | fail | fail | NameError: name 'is_palindrome' is not defined. Did you mean: 'make_palindrome'? |
| 11 | pass | pass | ok |
| 12 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 13 | pass | pass | ok |
| 14 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 15 | pass | pass | ok |
| 16 | pass | pass | ok |
| 17 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 18 | pass | pass | ok |
| 19 | pass | pass | ok |
| 20 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 21 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 22 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 23 | pass | pass | ok |
| 24 | pass | pass | ok |
| 25 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 26 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 27 | pass | pass | ok |
| 28 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 29 | pass | fail | NameError: name 'List' is not defined. Did you mean: 'list'? |
| 30 | pass | pass | ok |
| 31 | pass | pass | ok |
| 32 | fail | fail | NameError: name 'poly' is not defined |
| 33 | pass | pass | ok |
| 34 | pass | pass | ok |
| 35 | pass | pass | ok |
| 36 | pass | pass | ok |
| 37 | pass | pass | ok |
| 38 | fail | fail | NameError: name 'encode_cyclic' is not defined. Did you mean: 'decode_cyclic'? |
| 39 | pass | pass | ok |
| 40 | pass | pass | ok |
| 41 | pass | pass | ok |
| 42 | pass | pass | ok |
| 43 | pass | pass | ok |
| 44 | pass | pass | ok |
| 45 | pass | pass | ok |
| 46 | pass | pass | ok |
| 47 | pass | pass | ok |
| 48 | pass | pass | ok |
| 49 | pass | pass | ok |
McNemar’s test asks whether the difference between the two models is bigger than chance: a p-value above 0.05 means the change is not statistically significant. The 95% CI on the pass-rate delta is the range the true difference is likely in — if it crosses zero, the direction isn’t certain.