200 decisions · accuracy 100.0% [98.2, 100.0]† · confidence known 200/200 · ECE 0.0036 [0.0014, 0.0060] · ECE (equal-mass) 0.0036 [0.0014, 0.0060] · Brier 0.0003 [0.0001, 0.0005] · NLL 0.0037 [0.0015, 0.0063]
cost $0.0037 · p50 0.863s · p99 1.608s · slowest 8.188s
judge `jev` · model `typesafe-ai/jev` · backend `gateway` · recomputed 2026-09-19T09:09:12+00:00 (original run time not recorded) · judge-audit 0.2.0
regenerated 2026-09-24T15:48:07+00:00 from `docs/runs/audit-jev-real.ckpt.jsonl` by `scripts/runs_report.py` · judge-audit 0.4.0
Ground truth: GT-1 constructed — labels are true by construction of a seeded generator; suitable for calibration stress testing, not evidence of real-world accuracy; email categories are synthetic: seeded templates with item and number fills, not real mail; the label is the template's category by design; no human checked it; 100 % here is the floor a judge must clear, not evidence of production routing accuracy
Brackets are 95% percentile-bootstrap intervals over the dataset's distinct texts (2,000 resamples, seed 0): how far the number would move on another sample of n=200 drawn the same way.
**†** exact 95 % Clopper–Pearson (binomial) interval, published where the estimate is 0 % or 100 % and the bootstrap collapses to a point. It assumes independent rows, so where the dataset repeats texts it is a *lower bound* on the width the clustered interval would have had.
Zero observed errors through the most confident 100.0% [98.2, 100.0]† (200 decisions, confidence ≥ 0.89).
Retrospective on this dataset — not a production guarantee.
| coverage | accuracy | min confidence | n |
|---|---|---|---|
| 93.5% | 100.0% | 1.00 | 187 |
| 95.5% | 100.0% | 0.98 | 191 |
| 100.0% | 100.0% | 0.89 | 200 |
| bin | avg confidence | accuracy | n |
|---|---|---|---|
| 0.8-0.9 | 0.890 | 100.0% | 2 |
| 0.9-1.0 | 0.998 | 100.0% | 198 |
A perfectly honest judge sits on the diagonal: avg confidence == accuracy in every bin.
ECE uses ten equal-width bins; the equal-mass ECE cuts the rows into ten groups of about equal size (tied confidences never split), so it does not hinge on one crowded bin; Brier is the mean squared gap between confidence and outcome, needs no bins, and also rewards accuracy. Three separate numbers, never combined. No log-loss: one wrong answer at a declared confidence of 1.0 makes it infinite, and clipping the confidence would impute one.