Metadata-Version: 2.5
Name: rankcheck
Version: 0.1.0
Summary: How many real ranks does a leaderboard carry, and how many independent signals does a model panel carry?
Project-URL: Homepage, https://github.com/probityai/rankcheck
Project-URL: Source, https://github.com/probityai/rankcheck
Author: Sankalp Gilda
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: rfc8785>=0.1.4
Provides-Extra: seal
Requires-Dist: cryptography>=42; extra == 'seal'
Requires-Dist: rfc3161-client>=1.0; extra == 'seal'
Requires-Dist: rfc8785>=0.1.4; extra == 'seal'
Provides-Extra: test
Requires-Dist: pytest>=8; extra == 'test'
Requires-Dist: scipy>=1.11; extra == 'test'
Description-Content-Type: text/markdown

# rankcheck

rankcheck counts how many real ranks a leaderboard has and how many independent signals a panel of models gives.

It is for people who read, publish or build leaderboards and model ensembles, and who want to know which "we beat
X" results would hold up under a paired test.

It accompanies the paper "How Many Independent Signals Do AI Leaderboards and AI Forecasters Carry?" (Apart
AI x Epistemics sprint, November 2026).

```
pip install rankcheck==0.1.0
```

```
$ rankcheck tiers --demo swebench-verified
board: SWE-bench Verified, population results (SWE-bench/experiments@40f164d5 ...)
systems: 135   items: 500   test: exact McNemar   correction: Holm (FWER 0.05)
adjacent ranks separable: 0 of 134 (Holm); 0 of 134 (Benjamini-Hochberg, q 0.05)
separable tiers: 12 for 135 systems (walk from the top, Bonferroni alpha 3.73e-04 per test)
  tier  1:  17 systems, head score 0.792: ...
records (a new best by date) separable from the previous record: 3 of 32 (Holm)
```

Status: 0.1.0, Python 3.10 or newer, depends on numpy and rfc8785. `verify-seal` also needs the `openssl` command.

## Commands

| Command | What it answers |
|---|---|
| `rankcheck tiers board.csv` | For a board with columns `model,item,score`: are neighbouring ranks distinguishable by a paired test over items (exact McNemar for 0/1 items, paired sign-flip otherwise, Holm correction), and how many separable tiers does the board hold? `--pairs` prints every pair, `--json` everything. |
| `rankcheck tiers --demo swebench-verified` | The same on the bundled per-instance SWE-bench Verified results (`--population results`, `all` or `bash-only`). |
| `rankcheck ranks board.csv` | Simultaneous 95% confidence sets for every system's rank (Mogstad, Romano, Shaikh and Wilhelm 2024, stepwise, item bootstrap; `--bootstrap gaussian` for the csranks parametric bootstrap). On 0/1 boards also the exact construction of Neuhof and Benjamini (Holm over all one-sided exact paired tests). `--check-examples` recomputes the paper's Tables 1 and 2 and compares. |
| `rankcheck indep forecasts.csv` | For columns `model,item,p` (optional `baseline`, `cluster`, `outcome`; re-asks as `model#r2`): the participation ratio and exchangeable effective number of forecasters with item-bootstrap intervals, and, when a reference forecast is given, the calibrated number of forecasters whose departures from the reference are independent. `--demo` runs a synthetic panel with a known answer. |
| `rankcheck verify-seal <bundle>` | For a sealed forecasting trial: rebuilds each wave's Merkle root, checks every RFC 3161 token against pinned authority roots (quorum of 2), and checks the wave's entry in the public append-only registry and its Rekor record. Prints PASS, FAIL or NOT RUN per check. |

## What the tests mean

Each system on a public per-item board is usually one run. The paired test covers item sampling only: if two
systems could not be told apart on these items, a rerun would not make them easier to tell apart.

## Data

The bundled SWE-bench Verified board is built from the public per-instance results in
[SWE-bench/experiments](https://github.com/SWE-bench/experiments) at commit `40f164d5`: an instance counts as
resolved when it is in the official `resolved` list.

The bundled PISA 2018 reading table (OECD countries, used by `--check-examples`) is the `pisa` dataset of the
[csranks](https://cran.r-project.org/package=csranks) R package 1.3.0, from OECD PISA 2018.

## Prior work

rankcheck was written during Apart Research's AI x Epistemics research sprint (13 to 15 November 2026). None of its
own code is preexisting. The files under `src/rankcheck/_sealed/` are byte-identical copies of estimator files sealed
during the sprint; their SHA-256 digests are in `src/rankcheck/_sealed/SEALED.json`.

Preexisting third-party code it depends on: numpy and rfc8785; optionally rfc3161-client, cryptography, scipy and
pytest. Bundled third-party data: SWE-bench/experiments at commit `40f164d5` and the csranks PISA 2018 table.

## License

[Apache-2.0](https://github.com/probityai/rankcheck/blob/main/LICENSE).
