abeval — sample-size calculator

How many items does your eval need before the difference you care about is distinguishable from noise? Paired design: both runs scored on the same items. Same formulas as abeval power.

Baseline accuracy (%) where run A sits today

Difference to detect (points) the improvement you'd act on

Item correlation 0.5 typical for two variants of one system

Power (%) chance of detecting a real effect

– paired items needed

Smallest detectable difference at each eval size (log-scaled items axis)

Items in your eval	Smallest detectable difference

Two-sided test at 95% confidence on the mean of per-item differences, normal approximation. Verify locally: pip install abeval then abeval power --baseline 0.75 --delta 0.03. MIT-licensed — github.com/mohammadi-hadi/abeval.