How far this judge's confidence can be trusted on these labelled decisions, and how much it could decide alone.
† exact 95 % Clopper–Pearson (binomial) interval, published where the estimate is 0 % or 100 % and the bootstrap collapses to a point. It assumes independent rows, so where the dataset repeats texts it is a lower bound on the width the clustered interval would have had.
Safe automation rate at ≤ 5% error: 100.0% of the labelled decisions (200 of 200; 0 wrong, error bound 1.5%), deploying at confidence ≥ 0.89. Checked out of sample, with the threshold chosen on half the data: not checkable: a half holds at most 110 decisions with a confidence; the target needs 124.
This labelled set repeats texts (200 decisions, 161 distinct texts). The bound above treats the decisions as a random sample of your traffic, repeats included as they occur there. If the repeats were copied in instead, count each text once (at its highest confidence, wrong if any copy is: not always the more cautious reading when the copies of a text carry different confidences): 100.0% of the texts (161 of 161; 0 counted wrong, error bound 1.8%), at confidence ≥ 0.89.
Segments (true label): none has 30 automated decisions, so none is ranked. 10 segments with fewer than 30 automated decisions are not ranked.
| At most this error | In plain words | Safe automation rate | Deploy at confidence ≥ | Checked out of sample |
|---|---|---|---|---|
| 1% | at most 1 in 100 automated decisions wrong | none: needs 628 decisions with a confidence to bound the error at 1%; has 200 | — | — |
| 2% | at most 1 in 50 automated decisions wrong | none: needs 313 decisions with a confidence to bound the error at 2%; has 200 | — | — |
| 5% (primary) | at most 1 in 20 automated decisions wrong | 100.0% (200 of 200 decisions; 0 wrong, error bound 1.5%) | 0.89 | not checkable: a half holds at most 110 decisions with a confidence; the target needs 124 |
| 10% | at most 1 in 10 automated decisions wrong | 100.0% (200 of 200 decisions; 0 wrong, error bound 1.5%) | 0.89 | 100.0% automated, 0.0% wrong (seeds 0–9: 98.5%–100.0%) |
Valid only for: judge jev · model typesafe-ai/jev · measured 2026-09-19T09:09:12+00:00. Review by 2026-12-18, and measure again as soon as any of these changes.
With 95% confidence, the error rate among decisions at or above the threshold is at most the target — on traffic drawn like these labelled decisions (a random sample of it, each decision an independent draw), judged by the judge and prompt above, with the labels taken as right. Label errors cut both ways: a wrong label the judge disagrees with counts as a judge error, one it agrees with hides an error. The primary target is the one passed with --target (here 5%), to be chosen before the run (this report was regenerated from an archived checkpoint with the default target: the target was not fixed in advance); every row is its own 95% statement; the 4 rows hold together with probability at least 80%; keeping the best-looking row after reading them is neither. The rate is the share of the labelled decisions the threshold covers; the out-of-sample check chooses the threshold on half of them (split by distinct text, seed 0, spread over other seeds in brackets) and applies it to the other half, a conservative check rather than a forecast. A decision without a confidence is never automated. Exact one-sided binomial bound, fixed-sequence walk from the most confident down, starting at the cut that bounds the error with 2 errors (docs/judges.md § The safe automation rate). Statistical evidence on the labelled decisions, not a conformity assessment or a certification by a notified body.
Zero observed errors through the most confident 100.0% [98.2, 100.0]† (200 decisions, confidence ≥ 0.89).
Retrospective on this dataset — not a production guarantee.
A perfectly honest judge sits on the diagonal: in every bin, average confidence equals accuracy.
| Coverage | Accuracy | Min confidence | n |
|---|---|---|---|
| 93.5% | 100.0% | 1.00 | 187 |
| 95.5% | 100.0% | 0.98 | 191 |
| 100.0% | 100.0% | 0.89 | 200 |
| Bin | Avg confidence | Accuracy | n |
|---|---|---|---|
| 0.8-0.9 | 0.890 | 100.0% | 2 |
| 0.9-1.0 | 0.998 | 100.0% | 198 |
A perfectly honest judge sits on the diagonal: avg confidence == accuracy in every bin.
200 decisions · accuracy 100.0% [98.2, 100.0]† · confidence known 200/200 · ECE 0.0036 [0.0014, 0.0060] · ECE (equal-mass) 0.0036 [0.0014, 0.0060] · Brier 0.0003 [0.0001, 0.0005] · NLL 0.0037 [0.0015, 0.0063]
cost $0.0037 · p50 0.863s · p99 1.608s · slowest 8.188s
Brackets are 95% percentile-bootstrap intervals over the dataset's distinct texts (2,000 resamples, seed 0): how far the number would move on another sample of n=200 drawn the same way.
† exact 95 % Clopper–Pearson (binomial) interval, published where the estimate is 0 % or 100 % and the bootstrap collapses to a point. It assumes independent rows, so where the dataset repeats texts it is a lower bound on the width the clustered interval would have had.