Metadata-Version: 2.5
Name: evalpower
Version: 0.1.0
Summary: How many runs before your eval means anything? Repeat-run reliability statistics for stochastic evals: audit miss rates, exact Clopper-Pearson stochasticity, runs-needed, and pass^k / safe^k curves from per-cell run counts.
Author-email: Shiven Khurdi <shiven.khurdi@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: clopper-pearson,evaluation,flaky-tests,llm-evals,reliability,statistics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Mathematics
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.11
Provides-Extra: dev
Requires-Dist: hypothesis>=6.0; extra == 'dev'
Requires-Dist: mypy>=1.8; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff<0.17,>=0.16; extra == 'dev'
Description-Content-Type: text/markdown

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/shivenkk/evalpower/main/assets/banner-dark.svg">
  <img alt="evalpower: how many runs before your eval means anything" src="https://raw.githubusercontent.com/shivenkk/evalpower/main/assets/banner-light.svg">
</picture>

<p align="center">
  <a href="https://github.com/shivenkk/evalpower/actions/workflows/ci.yml"><img alt="CI" src="https://github.com/shivenkk/evalpower/actions/workflows/ci.yml/badge.svg"></a>
  <a href="https://pypi.org/project/evalpower/"><img alt="PyPI" src="https://img.shields.io/pypi/v/evalpower?color=4f46e5"></a>
  <img alt="Python 3.11+" src="https://img.shields.io/badge/python-3.11%2B-4f46e5">
  <a href="https://github.com/shivenkk/evalpower/blob/main/LICENSE"><img alt="Apache-2.0 license" src="https://img.shields.io/badge/license-Apache--2.0-22d3ee"></a>
  <a href="https://github.com/shivenkk/evalpower/tree/main/tests"><img alt="236 tests" src="https://img.shields.io/badge/tests-236-8b5cf6"></a>
  <a href="https://arxiv.org/abs/2608.15286"><img alt="arXiv 2608.15286" src="https://img.shields.io/badge/arXiv-2608.15286-b31b1b"></a>
</p>

**How many runs before your eval means anything?**

Your eval ran once and passed. What is the probability that means nothing?

Most eval harnesses answer a question nobody asked: *did it pass this time?* If
the thing you are measuring is stochastic — and agent behaviour, tool use, and
anything with a temperature above zero is stochastic — then a single green run
is a sample of size one, and a suite of green runs is a suite of samples of
size one.

This library takes the pass/fail counts you already have and tells you what
they actually support:

- **The k=1 audit miss rate.** Of the cells you *know* fail, because you watched
  them fail, how often does a single run look clean anyway? Above 0.5 and
  one-shot auditing is worse than a coin flip.
- **Whether a cell is demonstrably stochastic**, under an exact Clopper-Pearson
  criterion that requires the *interval endpoints* to clear the always-pass and
  always-fail traps — not just the point estimate to look intermediate.
- **How many runs you need** to reach a given confidence, and its inverse: what
  a clean sweep of n runs actually rules out.
- **pass^k and safe^k curves**, computed unbiasedly rather than by raising an
  observed rate to the k-th power.

No containers, no adapter, no environment reset, and nothing to integrate. It
runs against eval output you already have.

**This is not significance testing.** If your question is whether two models
differ, you want a significance tool, and good ones exist for LLM evals. This
one asks the prior question: whether a single model–task result is repeatable at
all. Significance compares between arms; reliability lives within a cell.

## Install

```
pip install evalpower
```

**Zero runtime dependencies.** Pure stdlib, and intended to stay that way: this
drops into any pipeline without touching its dependency resolution, and there
is no version of scipy that can break someone's build because they wanted a
confidence interval.

Typed (`py.typed`, checked under `mypy --strict`), Python 3.11+, tested on 3.11
through 3.14.

## Quickstart

```python
from evalpower import CellStats, audit_miss_rate, clopper_pearson, runs_needed

# A "cell" is whatever you repeat runs over — usually (model, task).
cells = {
    ("gpt-x", "refund-flow"):   CellStats(n=16, x=4),   # 4 of 16 runs failed
    ("gpt-x", "ledger-close"):  CellStats(n=16, x=0),   # clean sweep
    ("claude-y", "refund-flow"): CellStats(n=32, x=12),
}

audit_miss_rate(cells)                  # 0.6875 pair-weighted
audit_miss_rate(cells, "event")         # 0.65625 weighted by failure events
clopper_pearson(4, 16)                  # (0.0727, 0.5238) — exact, not Wald
runs_needed(0.25, confidence=0.95)      # 11
```

That clean sweep is the interesting one:

```python
from evalpower import zero_failure_upper_bound

zero_failure_upper_bound(16)   # 0.1707
```

Sixteen spotless runs do not show the failure rate is zero. They show it is
probably below 17%.

### From a file

```
$ evalpower results.csv

3 cells, 64 runs, 2 failure-producing, 16 failure events

  cell                    x/n    rate   95% CI (exact)  miss@1  stochastic  runs@95%
  ----------------------  -----  -----  --------------  ------  ----------  --------
  claude-y / refund-flow  12/32  0.375  (0.211, 0.563)  0.625   yes         7
  gpt-x / refund-flow     4/16   0.250  (0.073, 0.524)  0.750   yes         11
  gpt-x / ledger-close    0/16   0.000  (0.000, 0.206)  1.000   -           <=17.1%

k=1 audit miss rate over the 2 failure-producing cells
  pair-weighted   0.6875
  event-weighted  0.6562
  (the chance one run of a known-bad cell looks clean; above 0.5 is worse than a coin flip)

Demonstrably stochastic (exact 95% CI strictly inside (0.05, 0.95)): 2 of 3 cells
  carrying 100.0% of all failure events
  100.0% of failure events sit on cells with a rate in (0.1, 0.9)

  k    pass^k   safe^k
  ---  -------  -------
  1    0.79167  0.79167
  2    0.64435  0.64435
  3    0.54090  0.54090
  5    0.41944  0.41944
  10   0.33704  0.33704
```

The input is a table of per-cell counts, JSONL or CSV:

```csv
model,task,n,x
gpt-x,refund-flow,16,4
gpt-x,ledger-close,16,0
claude-y,refund-flow,32,12
```

Column names are matched leniently (`n`/`runs`, `x`/`failures`, …); see
`docs/concepts.md`.

### Or straight from per-run records

Harnesses emit one row per run, not per cell, so `--per-run` aggregates for you.
No pre-processing script, which is otherwise the first thing standing between you
and a report:

```csv
model,task,failed
gpt-x,refund-flow,true
gpt-x,refund-flow,false
gpt-x,refund-flow,false
gpt-x,refund-flow,false
```

```
$ evalpower --per-run runs.csv --k 1 2

1 cell, 4 runs, 1 failure-producing, 1 failure event

  cell                 x/n  rate   95% CI (exact)  miss@1  stochastic  runs@95%
  -------------------  ---  -----  --------------  ------  ----------  --------
  gpt-x / refund-flow  1/4  0.250  (0.006, 0.806)  0.750   -           11

k=1 audit miss rate over the 1 failure-producing cell
  pair-weighted   0.7500
  event-weighted  0.7500
```

Note what four runs buy: the interval is (0.006, 0.806), the cell does not clear
the demonstrably-stochastic bar, and it would take 11 runs to be 95% confident of
seeing that failure at all.

The outcome column can be named for either polarity — `failed`, `passed`,
`is_success` — or be a neutral `status`/`outcome`/`result` holding `pass`/`fail`.
Values may be `true`/`false`, `yes`/`no`, `y`/`n`, `1`/`0`. Anything that could
mean either thing (a neutral column holding `true`) is an error naming the flag
that resolves it, never a guess. Override discovery with `--cell-fields`,
`--failure-field`, `--success-field`.

### In a pipeline

`--json` emits the same report as data, so you can gate on it:

```
$ evalpower results.csv --json | jq '.audit_miss_rate.pair'
0.6875
```

There is deliberately no `--fail-under` and no exit-code policy. What counts as
an acceptable miss rate is a judgement about your system, not something this tool
should hold an opinion about — so it always exits 0 and hands you the numbers.

## Why exact intervals

Eval cells live where normal approximations fail: small n, proportions near 0
and 1. A Wald interval on 0 failures in 16 runs has **zero width** — it reports
that you have measured the failure rate to be exactly zero, with certainty.
Clopper-Pearson gives (0, 0.206). Every interval here is exact, computed by
bisection on the binomial tail, with no continuity correction and no normal
approximation anywhere.

The same discipline applies to the "how many runs" question. The rule of three
(`p <= 3/n` given zero failures) overstates the limit by about 8 percentage
points at n=7. This library uses the exact one-sided form, `1 - alpha^(1/n)`.

## Provenance

These estimators are the measurement core of a reliability study of LLM agents,
ported here with the domain-specific vocabulary generalized — that work's
"damage" is this library's outcome-neutral "failure". The math is unchanged.

`tests/test_paper_validation.py` reproduces that paper's published figures
exactly from its released run data — the audit miss rates, the trap-prevalence
bounds, and the beta-binomial ICCs — so "this is the same computation" is a
checkable claim rather than an assertion, and any future drift between the two
fails a test. The test skips cleanly when the source data is not present, which
is the normal case for everyone but its author.

**Citation.** The figures this library is validated against come from
[arXiv:2608.15286](https://arxiv.org/abs/2608.15286):

```bibtex
@misc{khurdi2026agentrelbench,
  title  = {No Task Fails Every Time: Why One-Shot Audits Are Structurally Blind to Agent Damage},
  author = {Shiven Khurdi},
  year   = {2026},
  eprint = {2608.15286},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url    = {https://arxiv.org/abs/2608.15286},
}
```

## Scope

This library does statistics on counts. It does not run your evals, sandbox
them, reset environments, or decide what "failure" means — that last one is
yours, and the library is deliberately agnostic to it. If your labeller emits a
conservative alternate count for ambiguous runs (crashes, timeouts), pass it as
`x_upper` and it stays a separately reported bound, never fused into the
headline.

## Licence

Apache-2.0.
