TokenJam proof — judged

tokenjam 0.5.1 n=5 tasks · k=1 sample(s) · deepseek:deepseek-reasoner → deepseek:deepseek-chat

Verdict: Insufficient evidence · McNemar p=1.000 (α=0.05) · candidate chosen by explicit --candidate override

-47.6%

cost delta (measured)

+20.0pp

pass-rate delta [95% CI -15.1, +55.1]

0 / 1

tasks broken / fixed by the swap

Pass rate (95% CI whiskers)

Original2/5 (40%)

Candidate3/5 (60%)

Original$0.001080

Candidate$0.000566

task	orig	cand	candidate detail
judged/refund-policy	0/1	0/1	deepeval:correctness@deepseek:deepseek-chat score=0.00 (>= 0.5) — The actual output does n
judged/capital	1/1	1/1	deepeval:correctness@deepseek:deepseek-chat score=1.00 (>= 0.5) — The actual output exactl
judged/retry-summary	1/1	1/1	deepeval:correctness@deepseek:deepseek-chat score=1.00 (>= 0.5) — The actual output adds u
judged/shipping	0/1	0/1	deepeval:correctness@deepseek:deepseek-chat score=0.00 (>= 0.5) — The actual output does n
judged/define-llm	0/1	1/1	deepeval:correctness@deepseek:deepseek-chat score=0.50 (>= 0.5) — The actual output includ

Accuracy is the pass-rate on THIS benchmark suite, not a general “quality preserved” claim. Confidence is the CI + p-value, not a single “safe %”.
Only n=5 tasks — too few for a significance verdict. Raise the task count for a defensible result.
A model had no TokenJam rate; cost used the $0.50/$2.00 default placeholder — savings are approximate.

Generated 2026-06-26 08:50 · tokenjam-bench · executable-accuracy proof