judge-audit · Calibration auditStatistical evidence, not a conformity assessment

Audit report — jev

How far this judge's confidence can be trusted on these labelled decisions, and how much it could decide alone.

judge jev · model typesafe-ai/jev · backend gateway · recomputed 2026-09-19T09:09:12+00:00 (original run time not recorded) · judge-audit 0.2.0

regenerated 2026-10-07T06:59:06+00:00 from docs/runs/audit-jev-real.ckpt.jsonl by scripts/runs_report.py · judge-audit 0.5.1

Key figures

Safe automation rate at ≤ 5% error
100.0%
deploy at confidence ≥ 0.89 · out of sample: not checkable: a half holds at most 110 decisions with a confidence; the target needs 124
Accuracy
100.0%
95% interval [98.2, 100.0]† · n = 200
Calibration error (ECE)
0.0036
95% interval [0.0014, 0.0060] · 0 = confidence matches accuracy
Cost · latency
$0.0037
p50 0.863s · p99 1.608s

† exact 95 % Clopper–Pearson (binomial) interval, published where the estimate is 0 % or 100 % and the bootstrap collapses to a point. It assumes independent rows, so where the dataset repeats texts it is a lower bound on the width the clustered interval would have had.

Data and ground truthGround truth: GT-1 constructed — labels are true by construction of a seeded generator; suitable for calibration stress testing, not evidence of real-world accuracy; email categories are synthetic: seeded templates with item and number fills, not real mail; the label is the template's category by design; no human checked it; 100 % here is the floor a judge must clear, not evidence of production routing accuracy

Can I automate this?

Safe automation rate at ≤ 5% error: 100.0% of the labelled decisions (200 of 200; 0 wrong, error bound 1.5%), deploying at confidence ≥ 0.89. Checked out of sample, with the threshold chosen on half the data: not checkable: a half holds at most 110 decisions with a confidence; the target needs 124.

This labelled set repeats texts (200 decisions, 161 distinct texts). The bound above treats the decisions as a random sample of your traffic, repeats included as they occur there. If the repeats were copied in instead, count each text once (at its highest confidence, wrong if any copy is: not always the more cautious reading when the copies of a text carry different confidences): 100.0% of the texts (161 of 161; 0 counted wrong, error bound 1.8%), at confidence ≥ 0.89.

Segments (true label): none has 30 automated decisions, so none is ranked. 10 segments with fewer than 30 automated decisions are not ranked.

At most this errorIn plain wordsSafe automation rateDeploy at confidence ≥Checked out of sample
1%at most 1 in 100 automated decisions wrongnone: needs 628 decisions with a confidence to bound the error at 1%; has 200——
2%at most 1 in 50 automated decisions wrongnone: needs 313 decisions with a confidence to bound the error at 2%; has 200——
5% (primary)at most 1 in 20 automated decisions wrong100.0% (200 of 200 decisions; 0 wrong, error bound 1.5%)0.89not checkable: a half holds at most 110 decisions with a confidence; the target needs 124
10%at most 1 in 10 automated decisions wrong100.0% (200 of 200 decisions; 0 wrong, error bound 1.5%)0.89100.0% automated, 0.0% wrong (seeds 0–9: 98.5%–100.0%)

Valid only for: judge jev · model typesafe-ai/jev · measured 2026-09-19T09:09:12+00:00. Review by 2026-12-18, and measure again as soon as any of these changes.

With 95% confidence, the error rate among decisions at or above the threshold is at most the target — on traffic drawn like these labelled decisions (a random sample of it, each decision an independent draw), judged by the judge and prompt above, with the labels taken as right. Label errors cut both ways: a wrong label the judge disagrees with counts as a judge error, one it agrees with hides an error. The primary target is the one passed with --target (here 5%), to be chosen before the run (this report was regenerated from an archived checkpoint with the default target: the target was not fixed in advance); every row is its own 95% statement; the 4 rows hold together with probability at least 80%; keeping the best-looking row after reading them is neither. The rate is the share of the labelled decisions the threshold covers; the out-of-sample check chooses the threshold on half of them (split by distinct text, seed 0, spread over other seeds in brackets) and applies it to the other half, a conservative check rather than a forecast. A decision without a confidence is never automated. Exact one-sided binomial bound, fixed-sequence walk from the most confident down, starting at the cut that bounds the error with 2 errors (docs/judges.md § The safe automation rate). Statistical evidence on the labelled decisions, not a conformity assessment or a certification by a notified body.

Zero observed errors through the most confident 100.0% [98.2, 100.0]† (200 decisions, confidence ≥ 0.89).
Retrospective on this dataset — not a production guarantee.

Reliability diagram

A perfectly honest judge sits on the diagonal: in every bin, average confidence equals accuracy.

Reliability diagram: average stated confidence against observed accuracy in each confidence bin, with the diagonal of a perfectly honest judge

When it says ≥ 90 / 95 / 99 %, how often is it wrong?

confidencedecisionswrongerror rateupper bound (95%)
≥ 0.9019800.0%1.5%
≥ 0.9519400.0%1.5%
≥ 0.9918900.0%1.6%

0 decisions without a known confidence are left out of every row and counted here. A reading of the accuracy-vs-coverage curve at three fixed confidences, not a new score and not combined with any other number: the decisions declaring at least that confidence, how many were wrong, and an exact one-sided upper bound on their error rate (Clopper–Pearson, 95%). Rows are taken as independent: on a dataset that repeats texts, each copy counts, which can make the bound too tight. The cut is fixed in advance, not chosen on these rows, unlike the threshold an automation decision searches for.

Accuracy vs coverage

Accuracy against coverage: accuracy among the decisions above each confidence threshold, by the share of decisions above it
CoverageAccuracyMin confidencen
93.5%100.0%1.00187
95.5%100.0%0.98191
100.0%100.0%0.89200

Calibration bins

BinAvg confidenceAccuracyn
0.8-0.90.890100.0%2
0.9-1.00.998100.0%198

A perfectly honest judge sits on the diagonal: avg confidence == accuracy in every bin.

All metrics and method notes

200 decisions · accuracy 100.0% [98.2, 100.0]† · confidence known 200/200 · ECE 0.0036 [0.0014, 0.0060] · ECE (equal-mass) 0.0036 [0.0014, 0.0060] · Brier 0.0003 [0.0001, 0.0005] · NLL 0.0037 [0.0015, 0.0063]
cost $0.0037 · p50 0.863s · p99 1.608s · slowest 8.188s

Brackets are 95% percentile-bootstrap intervals over the dataset's distinct texts (2,000 resamples, seed 0): how far the number would move on another sample of n=200 drawn the same way.

† exact 95 % Clopper–Pearson (binomial) interval, published where the estimate is 0 % or 100 % and the bootstrap collapses to a point. It assumes independent rows, so where the dataset repeats texts it is a lower bound on the width the clustered interval would have had.