truescore — your judge is wrong, by a measurable amount
Statistics for LLM-judged evaluations: bias-corrected scores, real confidence intervals, and comparisons that hold up.
pip install truescore truescore doctor your_eval_results.csv
The number on your dashboard
A support assistant, 4000 questions graded by an LLM judge, 600 of them also graded by a
person. The data is simulated, so the real pass rate is known: 0.7140.
| estimate | value | 95% interval | contains the truth? |
|---|---|---|---|
| judge only | 0.8383 | [0.8265, 0.8493] | no |
| corrected | 0.7150 | [0.6836, 0.7463] | yes |
| human labels only | 0.7183 | [0.6810, 0.7528] | yes, but wider |
The judge is 12 points high and its interval doesn't reach the true value. The corrected estimate does, and it's tighter than using the 600 human labels alone.
Comparisons are worse
| v4 minus v3 | |
|---|---|
| judge says | +0.1700 |
| corrected | +0.0950 [+0.0505, +0.1394] |
| actual | +0.0900 |
v4 really is better. But 7 of those 17 points come from the judge preferring longer
answers, and v4's median answer is 197 tokens against v3's 99. Measured length bias:
+0.069 per 100 tokens, p = 1e-05.
Slice the same data by support segment and the judge says all three improved. One
regressed by 17 points, the one where answers got longest. Per-segment correction returns
-0.1659 against a planted truth of -0.1663.
Watching a dashboard doesn't help
A 95% interval is valid at one sample size you fixed in advance. Check it repeatedly as data arrives and it isn't 95% anymore. Over 300 simulated healthy streams:
| checked after every observation | false alarms |
|---|---|
| fixed-sample interval | 47.7% |
| truescore confidence sequence | 0.3% |
Start here
doctor reads your file, works out what your columns are, and tells you what
it can compute and what's blocked:
$ truescore doctor results.csv
results.csv: 4000 rows, 5 columns
column kind coverage detail
judge_passed verdict 100% pass/fail on every row
human_passed sparse_verdict 15% pass/fail on 600 of 4000 rows
response_tokens numeric 100% range 35 to 1016
what this file supports today:
- correct the score for judge error
- monitor a release for regressions, no human labels needed
what the judge appears to be biased by:
- response_tokens: the judge gets more generous as it rises
(+0.00069 per unit, adjusted p=2.23e-05)
In CI
- uses: SaifPunjwani/truescore@v0.5.0
with:
command: drift
args: anchor.csv --baseline judge_pinned --current judge_today --gold human
Exit codes are 0 for nothing found, 2 for a finding, 1 for couldn't run. A job can fail on judge drift without also failing on a bad filename.
How many human labels does this need?
Some, for the correction. You can't calibrate an instrument without something to calibrate it against.
About half the library needs none: regression monitoring, contamination testing, position-bias detection and label planning all run on judge output alone. For the rest it's a few hundred labels, you label once and reuse across releases, and you may already have human verdicts in your database (escalations, thumbs-down, refunds, repeat contacts).
The library prices it before you spend anything:
+/-0.05: 232 labels (without the judge: 317) +/-0.03: 670 labels (without the judge: 879) +/-0.02: 1644 labels (without the judge: 1978)
Why trust it
Every interval estimator has a coverage test: simulate from a known truth hundreds of times and check the 95% interval covers about 95% of the time. The confidence sequence is checked over whole trajectories.
The suite also fuzzes every public function and requires that nothing ever returns a NaN. Any input gives a finite result or raises. That found three real bugs before release, including a near-deterministic slice whose interval had 7.7% coverage.
602 tests. mypy --strict. numpy and scipy are the only dependencies. It never calls a model.