muteval — eval quality report card

1 of 6 probes flagged an issue — core = catches a real eval defect · validity = needs labels · hygiene = sanity check (a WARN here is a footnote) — no composite score (each signal stands on its own).

PASS judge_reliabilitycore

0/3 verdicts flipped across 3 runs = 0% flaky (Krippendorff alpha 1.00)

stable (<= 5% flip target; alpha near 1).

PASS discriminationcore

not assessed (no good/bad exemplars provided)

add per-case 'good' and 'bad' example outputs to enable this probe.

PASS human_agreementvalidity

not assessed (no human labels)

run `muteval label` to emit a worksheet, fill it, save as .muteval\labels.csv.

WARN statistical_adequacyhygiene

3/3 cases pass = 100% [95% CI 44-100%]

CI too wide (+/-28%) to trust the number — add cases (~9 for +/-15%).

PASS redundancyhygiene

not assessed (need >= 2 evals)

needs >= 2 evals and >= 3 cases whose scores vary.

PASS threshold_calibrationhygiene

not assessed (no good/bad exemplars provided)

add per-case 'good' and 'bad' example outputs to enable this probe.

Generated by muteval.